diff --git a/.gitignore b/.gitignore index 100150be..31ec39cc 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,7 @@ src/kotlin/build/reports/jacoco/* !src/kotlin/build/reports/jacoco/test/ src/kotlin/build/reports/jacoco/test/* !src/kotlin/build/reports/jacoco/test/jacocoTestReport.xml + +# Benchmark artifacts +benchmarks/results/raw/ +benchmarks/results/run-report.json diff --git a/benchmarks/k6/README.md b/benchmarks/k6/README.md new file mode 100644 index 00000000..593e79cd --- /dev/null +++ b/benchmarks/k6/README.md @@ -0,0 +1,222 @@ +# Cloud Run Single-Instance Benchmark Harness + +This directory contains a benchmark harness for comparing the six language implementations with Cloud Run autoscaling neutralized (`max instances = 1`). + +## What it implements + +- Two benchmark passes: + - `memory` pass (runtime/framework signal) + - `db` pass (realistic signal) +- Fixed, fairness-first Cloud Run settings for all services +- Main ranking at fixed concurrency pressure with `concurrency=80` +- Separate cold-start appendix (not used in primary ranking) +- Optional non-ranking extreme run at `1000 RPS` (disabled by default) +- Sequential service execution (no parallel cross-service load) +- Raw k6 result exports and generated markdown summary + +## Files + +- `config.json`: benchmark parameters and Cloud Run parity settings +- `config.fast.json`: shorter benchmark profile for quicker p95-focused comparisons +- `services.json`: service URLs and per-service DB seed/reset hooks +- `scenarios.js`: k6 workload script (read-heavy CRUD mix) +- `run-benchmarks.js`: orchestration script for full memory+db benchmark execution +- `configure-cloud-run.js`: applies identical Cloud Run settings (dry-run by default) +- `generate-summary.js`: regenerates `benchmarks/results/summary.md` from `run-report.json` +- `summary.js`: shared summary rendering used by benchmark scripts + +## Prerequisites + +- `k6` installed and in `PATH` +- Node.js 20+ (uses built-in `fetch`) +- Network access from benchmark runner to all Cloud Run URLs +- If running configuration step: authenticated `gcloud` CLI and project access + +## 1) Fill service map + +Edit `benchmarks/k6/services.json`: + +- `memoryUrl`: Cloud Run URL for memory-backed deployment (no `/v1` suffix needed) +- `dbUrl`: Cloud Run URL for DB-backed deployment (no `/v1` suffix needed) +- `cloudRunService`: Cloud Run service name for settings updates +- `cloudRunRegion`: region (default `us-central1`) +- `memorySetupCommand`: optional command run before memory pass for this service +- `dbSetupCommand`: optional command run before DB pass for this service +- `dbSeedCommand`: optional per-service override shell command to reset+seed DB before each DB run + +Default seeding is configured once in `benchmarks/k6/config.json` as `defaultDbSeedCommand`: + +```bash +psql "$BENCHMARK_DATABASE_URL" -v ON_ERROR_STOP=1 -c "TRUNCATE TABLE lamps RESTART IDENTITY CASCADE; INSERT INTO lamps (id, is_on, created_at, updated_at, deleted_at) SELECT uuid_generate_v5('6ba7b810-9dad-11d1-80b4-00c04fd430c8', 'lamp-' || g), (g % 2 = 0), NOW() - ((10001 - g) * INTERVAL '1 second'), NOW() - ((10001 - g) * INTERVAL '1 second'), NULL FROM generate_series(1, 10000) AS g;" +``` + +Set `dbSeedCommand` in a service entry only when that service needs a custom seed/reset flow. + +If memory and DB use the same URL, run passes sequentially and toggle env vars via setup commands. +For this repository, DB mode is enabled by connection settings (for example `DATABASE_URL` or language-specific equivalents). +Example (TypeScript/Go/Python/Kotlin): + +```bash +gcloud run services update typescript-lamp-control-api --region europe-west1 --remove-env-vars DATABASE_URL +gcloud run services update typescript-lamp-control-api --region europe-west1 --update-env-vars DATABASE_URL="$BENCHMARK_DATABASE_URL" +``` + +Before running benchmarks, export required variables: + +```bash +export BENCHMARK_DATABASE_URL='postgresql://:@:5432/?sslmode=require' +export BENCHMARK_JDBC_DATABASE_URL='jdbc:postgresql://:5432/' +export BENCHMARK_DB_USER='' +export BENCHMARK_DB_PASSWORD='' +export BENCHMARK_CSHARP_CONNECTION_STRING='Host=;Port=5432;Database=;Username=;Password=' +``` + +`GOOGLE_CLOUD_PROJECT` is optional for `run-benchmarks.js`; if unset, it uses `cloudRun.projectId` from `benchmarks/k6/config.json`. + +## 2) Run from a GCP VM (recommended) + +Create a runner VM in the same region and install required tools: + +```bash +gcloud compute instances create lamp-bench-runner \ + --project= \ + --zone=europe-west1-b \ + --machine-type=e2-standard-4 \ + --image-family=ubuntu-2204-lts \ + --image-project=ubuntu-os-cloud \ + --boot-disk-size=30GB \ + --metadata=startup-script='#!/usr/bin/env bash +set -euxo pipefail +export DEBIAN_FRONTEND=noninteractive + +apt-get update +apt-get install -y ca-certificates curl gnupg git jq postgresql-client + +curl -fsSL https://deb.nodesource.com/setup_20.x | bash - +apt-get install -y nodejs + +curl -fsSL https://dl.k6.io/key.gpg | gpg --dearmor -o /usr/share/keyrings/k6-archive-keyring.gpg +echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" > /etc/apt/sources.list.d/k6.list +apt-get update +apt-get install -y k6 + +node --version +npm --version +k6 version +psql --version +' +``` + +Then connect: + +```bash +gcloud compute ssh lamp-bench-runner --project= --zone=europe-west1-b +``` + +## 3) Validate or apply Cloud Run parity settings + +Dry run (prints commands): + +```bash +node benchmarks/k6/configure-cloud-run.js +``` + +Apply settings: + +```bash +node benchmarks/k6/configure-cloud-run.js --execute +``` + +Settings come from `benchmarks/k6/config.json` under `cloudRun`. +Project resolution order is: `--project`, `cloudRun.projectId`, `cloudRun.projectNumber`, `GOOGLE_CLOUD_PROJECT`. +Startup probe applied to all services: + +```yaml +startupProbe: + timeoutSeconds: 1 + periodSeconds: 10 + failureThreshold: 3 + tcpSocket: + port: 8080 +``` + +For meaningful cold-start sampling, keep `cloudRun.minInstances=0`. + +## 4) Run benchmark + +Run both passes (`memory`,`db`) with settings from `config.json`: + +```bash +node benchmarks/k6/run-benchmarks.js +``` + +Run both passes with the faster profile: + +```bash +node benchmarks/k6/run-benchmarks.js --config benchmarks/k6/config.fast.json +``` + +Run only memory pass: + +```bash +node benchmarks/k6/run-benchmarks.js --passes memory +``` + +Run benchmark without running setup commands (`memorySetupCommand` / `dbSetupCommand`): + +```bash +node benchmarks/k6/run-benchmarks.js --passes memory --skip-setup +``` + +Run from macOS without sleep interruptions: + +```bash +caffeinate -i node benchmarks/k6/run-benchmarks.js +``` + +Fast profile on macOS: + +```bash +caffeinate -i node benchmarks/k6/run-benchmarks.js --config benchmarks/k6/config.fast.json +``` + +Disable cold-start appendix for quick local iterations: + +```bash +node benchmarks/k6/run-benchmarks.js --config benchmarks/k6/config.fast.json +# then set coldStart.enabled=false in the selected config +``` + +Runtime behavior: +- Cold-start probe runs before warmup/fixed/stress and is reported separately. +- Cold-start probe waits optional cooldown (`coldStart.cooldownSeconds`) to improve scale-to-zero likelihood. +- Precheck CRUD uses retry with exponential backoff (up to 7 attempts total). +- If an iteration still fails (precheck, k6, or setup error), the runner logs the error, records the failed iteration in `run-report.json`, and continues with the next iteration/service. + +Enable the extreme appendix run: + +```bash +# Set benchmarks/k6/config.json -> extreme.enabled to true +``` + +Outputs: + +- Raw k6 JSON: `benchmarks/results/raw//...` +- Structured run report: `benchmarks/results/run-report.json` +- Ranked markdown summary: `benchmarks/results/summary.md` +- Cold-start artifact per sampled iteration: `benchmarks/results/raw////iter-*/cold-start.json` + +## 5) Rebuild summary only + +```bash +node benchmarks/k6/generate-summary.js benchmarks/results/run-report.json benchmarks/results/summary.md +``` + +## Notes on fairness and interpretation + +- Keep Cloud Run settings identical across all six languages in the ranking run. +- Concurrency is a major factor even with `max instances=1`; it controls in-container contention. +- Use memory pass ranking to isolate runtime/framework signal. +- Use DB pass ranking to understand production-like behavior and DB bottleneck impact. +- Treat extreme `1000 RPS` run as saturation appendix, not primary ranking. +- Use `config.fast.json` for iterative checks and `config.json` for final publication-quality runs. diff --git a/benchmarks/k6/config.fast.json b/benchmarks/k6/config.fast.json new file mode 100644 index 00000000..013d01ab --- /dev/null +++ b/benchmarks/k6/config.fast.json @@ -0,0 +1,71 @@ +{ + "basePath": "/v1", + "passes": [ + "memory", + "db" + ], + "iterationsPerPass": 3, + "randomizeServiceOrder": true, + "warmup": { + "duration": "30s", + "rps": 20 + }, + "fixed": { + "duration": "90s", + "rps": 80 + }, + "stress": { + "stepDuration": "45s", + "rpsSteps": [ + 80, + 120, + 160 + ] + }, + "extreme": { + "enabled": false, + "duration": "60s", + "rps": 1000, + "runPerIteration": false + }, + "coldStart": { + "enabled": true, + "runPerIteration": false, + "cooldownSeconds": 900, + "maxWaitSeconds": 60, + "probeIntervalMs": 500, + "endpoint": "/lamps?pageSize=1", + "successStatus": 200 + }, + "slo": { + "p95Ms": 300, + "errorRate": 0.01 + }, + "workload": { + "listPercent": 50, + "getPercent": 20, + "createPercent": 20, + "updatePercent": 7, + "deletePercent": 3, + "pageSize": 25, + "seedFetchPages": 10, + "seedPageSize": 100 + }, + "defaultDbSeedCommand": "psql \"$BENCHMARK_DATABASE_URL\" -v ON_ERROR_STOP=1 -c \"TRUNCATE TABLE lamps RESTART IDENTITY CASCADE; INSERT INTO lamps (id, is_on, created_at, updated_at, deleted_at) SELECT uuid_generate_v5('6ba7b810-9dad-11d1-80b4-00c04fd430c8', 'lamp-' || g), (g % 2 = 0), NOW() - ((10001 - g) * INTERVAL '1 second'), NOW() - ((10001 - g) * INTERVAL '1 second'), NULL FROM generate_series(1, 10000) AS g;\"", + "cloudRun": { + "projectId": "lamp-control-469416", + "projectNumber": "827868544165", + "maxInstances": 1, + "minInstances": 0, + "concurrency": 80, + "cpu": "1", + "memory": "512Mi", + "timeout": "60s", + "startupProbe": { + "timeoutSeconds": 1, + "periodSeconds": 10, + "failureThreshold": 3, + "tcpPort": 8080 + } + } +} diff --git a/benchmarks/k6/config.json b/benchmarks/k6/config.json new file mode 100644 index 00000000..b4080897 --- /dev/null +++ b/benchmarks/k6/config.json @@ -0,0 +1,64 @@ +{ + "basePath": "/v1", + "passes": ["memory", "db"], + "iterationsPerPass": 2, + "randomizeServiceOrder": true, + "warmup": { + "duration": "60s", + "rps": 20 + }, + "fixed": { + "duration": "180s", + "rps": 80 + }, + "stress": { + "stepDuration": "60s", + "rpsSteps": [80, 120, 160, 200] + }, + "extreme": { + "enabled": false, + "duration": "60s", + "rps": 1000, + "runPerIteration": false + }, + "coldStart": { + "enabled": true, + "runPerIteration": false, + "cooldownSeconds": 900, + "maxWaitSeconds": 60, + "probeIntervalMs": 500, + "endpoint": "/lamps?pageSize=1", + "successStatus": 200 + }, + "slo": { + "p95Ms": 300, + "errorRate": 0.01 + }, + "workload": { + "listPercent": 50, + "getPercent": 20, + "createPercent": 20, + "updatePercent": 7, + "deletePercent": 3, + "pageSize": 25, + "seedFetchPages": 10, + "seedPageSize": 100 + }, + "defaultDbSeedCommand": "psql \"$BENCHMARK_DATABASE_URL\" -v ON_ERROR_STOP=1 -c \"TRUNCATE TABLE lamps RESTART IDENTITY CASCADE; INSERT INTO lamps (id, is_on, created_at, updated_at, deleted_at) SELECT uuid_generate_v5('6ba7b810-9dad-11d1-80b4-00c04fd430c8', 'lamp-' || g), (g % 2 = 0), NOW() - ((10001 - g) * INTERVAL '1 second'), NOW() - ((10001 - g) * INTERVAL '1 second'), NULL FROM generate_series(1, 10000) AS g;\"", + "cloudRun": { + "projectId": "lamp-control-469416", + "projectNumber": "827868544165", + "maxInstances": 1, + "minInstances": 0, + "concurrency": 80, + "cpu": "1", + "memory": "512Mi", + "timeout": "60s", + "startupProbe": { + "timeoutSeconds": 1, + "periodSeconds": 10, + "failureThreshold": 3, + "tcpPort": 8080 + } + } +} diff --git a/benchmarks/k6/configure-cloud-run.js b/benchmarks/k6/configure-cloud-run.js new file mode 100755 index 00000000..d9ee1f5b --- /dev/null +++ b/benchmarks/k6/configure-cloud-run.js @@ -0,0 +1,162 @@ +#!/usr/bin/env node +/* eslint-disable no-console */ +const fs = require('fs'); +const path = require('path'); +const { spawnSync } = require('child_process'); + +function parseArgs(argv) { + const args = { + services: path.join('benchmarks', 'k6', 'services.json'), + config: path.join('benchmarks', 'k6', 'config.json'), + project: '', + execute: false, + }; + + for (let i = 2; i < argv.length; i += 1) { + const token = argv[i]; + if (token === '--services') { + args.services = argv[++i]; + } else if (token === '--config') { + args.config = argv[++i]; + } else if (token === '--project') { + args.project = argv[++i]; + } else if (token === '--execute') { + args.execute = true; + } else if (token === '--help' || token === '-h') { + printHelp(); + process.exit(0); + } else { + throw new Error(`Unknown argument: ${token}`); + } + } + + return args; +} + +function printHelp() { + console.log(`Usage:\n node benchmarks/k6/configure-cloud-run.js [--project my-project] [--execute]\n\nProject is resolved in this order: --project, cloudRun.projectId, cloudRun.projectNumber, GOOGLE_CLOUD_PROJECT.\nBy default this script prints commands only. Add --execute to run them.`); +} + +function readJson(filePath) { + return JSON.parse(fs.readFileSync(filePath, 'utf8')); +} + +function runCommand(command, args) { + const rendered = `${command} ${args.join(' ')}`; + console.log(`\n$ ${rendered}`); + const result = spawnSync(command, args, { stdio: 'inherit' }); + if (result.status !== 0) { + throw new Error(`Command failed (${result.status}): ${rendered}`); + } +} + +function validateCloudRunConfig(cloudRun) { + const requiredKeys = [ + 'maxInstances', + 'minInstances', + 'concurrency', + 'cpu', + 'memory', + 'timeout', + ]; + + for (const key of requiredKeys) { + const value = cloudRun[key]; + if (value === undefined || value === null || String(value).trim() === '') { + throw new Error( + `Invalid cloudRun config: '${key}' is required in config.json (cloudRun.${key})` + ); + } + } + + const startupProbe = cloudRun.startupProbe || {}; + const requiredStartupProbeKeys = [ + 'timeoutSeconds', + 'periodSeconds', + 'failureThreshold', + 'tcpPort', + ]; + for (const key of requiredStartupProbeKeys) { + const value = startupProbe[key]; + if (value === undefined || value === null || String(value).trim() === '') { + throw new Error( + `Invalid cloudRun config: 'startupProbe.${key}' is required in config.json` + ); + } + } +} + +function resolveProject(argsProject, cloudRun) { + if (argsProject && argsProject.trim()) { + return argsProject.trim(); + } + if (cloudRun.projectId && String(cloudRun.projectId).trim()) { + return String(cloudRun.projectId).trim(); + } + if (cloudRun.projectNumber && String(cloudRun.projectNumber).trim()) { + return String(cloudRun.projectNumber).trim(); + } + if (process.env.GOOGLE_CLOUD_PROJECT && process.env.GOOGLE_CLOUD_PROJECT.trim()) { + return process.env.GOOGLE_CLOUD_PROJECT.trim(); + } + throw new Error( + "Missing project configuration. Set cloudRun.projectId (or projectNumber) in config.json, or pass --project, or set GOOGLE_CLOUD_PROJECT." + ); +} + +function main() { + const args = parseArgs(process.argv); + const services = readJson(args.services); + const config = readJson(args.config); + const cloudRun = config.cloudRun || {}; + validateCloudRunConfig(cloudRun); + const startupProbe = cloudRun.startupProbe; + const project = resolveProject(args.project, cloudRun); + + for (const service of services) { + if (!service.cloudRunService) { + console.log(`Skipping ${service.name}: cloudRunService is empty`); + continue; + } + + const region = service.cloudRunRegion || 'us-central1'; + const cmd = [ + 'run', + 'services', + 'update', + service.cloudRunService, + '--project', + project, + '--region', + region, + '--max-instances', + String(cloudRun.maxInstances), + '--min-instances', + String(cloudRun.minInstances), + '--concurrency', + String(cloudRun.concurrency), + '--cpu', + String(cloudRun.cpu), + '--memory', + String(cloudRun.memory), + '--timeout', + String(cloudRun.timeout), + '--cpu-throttling', + '--startup-probe', + `timeoutSeconds=${startupProbe.timeoutSeconds},periodSeconds=${startupProbe.periodSeconds},failureThreshold=${startupProbe.failureThreshold},tcpSocket.port=${startupProbe.tcpPort}`, + ]; + + if (!args.execute) { + console.log(`\n[dry-run] gcloud ${cmd.join(' ')}`); + continue; + } + + runCommand('gcloud', cmd); + } + + if (!args.execute) { + console.log('\nDry run complete. Re-run with --execute to apply updates.'); + } +} + +main(); diff --git a/benchmarks/k6/generate-summary.js b/benchmarks/k6/generate-summary.js new file mode 100755 index 00000000..887bcdf3 --- /dev/null +++ b/benchmarks/k6/generate-summary.js @@ -0,0 +1,27 @@ +#!/usr/bin/env node +/* eslint-disable no-console */ +const fs = require('fs'); +const path = require('path'); +const { writeSummary } = require('./summary'); + +function main() { + if (process.argv.includes('--help') || process.argv.includes('-h')) { + console.log( + 'Usage:\n node benchmarks/k6/generate-summary.js [report.json] [summary.md]' + ); + process.exit(0); + } + + const reportPath = process.argv[2] || path.join('benchmarks', 'results', 'run-report.json'); + const outputPath = process.argv[3] || path.join('benchmarks', 'results', 'summary.md'); + + if (!fs.existsSync(reportPath)) { + throw new Error(`Report not found: ${reportPath}`); + } + + const report = JSON.parse(fs.readFileSync(reportPath, 'utf8')); + writeSummary(report, outputPath); + console.log(`Wrote summary: ${outputPath}`); +} + +main(); diff --git a/benchmarks/k6/run-benchmarks.js b/benchmarks/k6/run-benchmarks.js new file mode 100755 index 00000000..8d17adfc --- /dev/null +++ b/benchmarks/k6/run-benchmarks.js @@ -0,0 +1,623 @@ +#!/usr/bin/env node +/* eslint-disable no-console */ +const fs = require('fs'); +const path = require('path'); +const { spawnSync } = require('child_process'); +const { writeSummary } = require('./summary'); + +function parseArgs(argv) { + const args = { + config: path.join('benchmarks', 'k6', 'config.json'), + services: path.join('benchmarks', 'k6', 'services.json'), + resultsDir: path.join('benchmarks', 'results'), + passes: null, + skipSetup: false, + }; + + for (let i = 2; i < argv.length; i += 1) { + const token = argv[i]; + if (token === '--config') { + args.config = argv[++i]; + } else if (token === '--services') { + args.services = argv[++i]; + } else if (token === '--results-dir') { + args.resultsDir = argv[++i]; + } else if (token === '--passes') { + args.passes = argv[++i].split(',').map((v) => v.trim()).filter(Boolean); + } else if (token === '--skip-setup') { + args.skipSetup = true; + } else if (token === '--help' || token === '-h') { + printHelp(); + process.exit(0); + } else { + throw new Error(`Unknown argument: ${token}`); + } + } + + return args; +} + +function printHelp() { + console.log(`Usage:\n node benchmarks/k6/run-benchmarks.js [--config path] [--services path] [--results-dir path] [--passes memory,db] [--skip-setup]\n`); +} + +function readJson(filePath) { + return JSON.parse(fs.readFileSync(filePath, 'utf8')); +} + +function ensureDir(dir) { + fs.mkdirSync(dir, { recursive: true }); +} + +function nowStamp() { + return new Date().toISOString().replace(/[:.]/g, '-'); +} + +function runCommand(command, args, options = {}) { + const rendered = `${command} ${args.join(' ')}`; + console.log(`\n$ ${rendered}`); + const result = spawnSync(command, args, { + stdio: 'inherit', + env: options.env || process.env, + shell: false, + }); + + if (result.status !== 0) { + throw new Error(`Command failed (${result.status}): ${rendered}`); + } +} + +function runShell(command, env) { + if (!command || !command.trim()) { + return; + } + + console.log(`\n$ ${command}`); + const result = spawnSync('bash', ['-lc', command], { + stdio: 'inherit', + env: env || process.env, + }); + + if (result.status !== 0) { + throw new Error(`Command failed (${result.status}): ${command}`); + } +} + +function sleep(ms) { + return new Promise((resolve) => { + setTimeout(resolve, ms); + }); +} + +async function httpJson(method, url, body, authHeader) { + const headers = {}; + if (authHeader) { + headers.Authorization = authHeader; + } + if (body !== undefined && body !== null) { + headers['Content-Type'] = 'application/json'; + } + + const response = await fetch(url, { + method, + headers, + body: body ? JSON.stringify(body) : undefined, + }); + + let parsed = null; + const text = await response.text(); + if (text) { + try { + parsed = JSON.parse(text); + } catch (_err) { + parsed = null; + } + } + + return { response, body: parsed, text }; +} + +async function runPrecheckWithRetry(baseUrl, basePath, authHeader, options = {}) { + const retries = Number.isFinite(options.retries) ? options.retries : 3; + const initialDelayMs = Number.isFinite(options.initialDelayMs) ? options.initialDelayMs : 1000; + const maxDelayMs = Number.isFinite(options.maxDelayMs) ? options.maxDelayMs : 5000; + let attempt = 0; + let delayMs = initialDelayMs; + let lastError = null; + + while (attempt <= retries) { + attempt += 1; + try { + await precheckCrud(baseUrl, basePath, authHeader); + return; + } catch (error) { + lastError = error; + const message = error && error.message ? error.message : String(error); + if (attempt > retries) { + break; + } + console.warn(`[precheck] attempt ${attempt}/${retries + 1} failed: ${message}. Retrying in ${delayMs}ms...`); + await sleep(delayMs); + delayMs = Math.min(delayMs * 2, maxDelayMs); + } + } + + throw lastError || new Error('Precheck failed after retries'); +} + +async function precheckCrud(baseUrl, basePath, authHeader) { + const prefix = `${baseUrl.replace(/\/$/, '')}${basePath}`; + + const create = await httpJson('POST', `${prefix}/lamps`, { status: true }, authHeader); + if (create.response.status !== 201 || !create.body || typeof create.body.id !== 'string') { + throw new Error(`Precheck create failed (${create.response.status})`); + } + + const lampId = create.body.id; + + const get = await httpJson('GET', `${prefix}/lamps/${lampId}`, null, authHeader); + if (get.response.status !== 200) { + throw new Error(`Precheck get failed (${get.response.status})`); + } + + const update = await httpJson('PUT', `${prefix}/lamps/${lampId}`, { status: false }, authHeader); + if (update.response.status !== 200) { + throw new Error(`Precheck update failed (${update.response.status})`); + } + + const list = await httpJson('GET', `${prefix}/lamps?pageSize=1`, null, authHeader); + if (list.response.status !== 200 || !list.body || !Array.isArray(list.body.data)) { + throw new Error(`Precheck list failed (${list.response.status})`); + } + + const del = await httpJson('DELETE', `${prefix}/lamps/${lampId}`, null, authHeader); + if (del.response.status !== 204) { + throw new Error(`Precheck delete failed (${del.response.status})`); + } +} + +function metricValues(metric) { + if (!metric) { + return null; + } + if (metric.values && typeof metric.values === 'object') { + return metric.values; + } + return metric; +} + +function parseMetric(summary, name) { + const metric = summary.metrics[name]; + const values = metricValues(metric); + if (!values) { + return null; + } + return { + avg: values.avg ?? null, + p95: values['p(95)'] ?? null, + p99: values['p(99)'] ?? null, + min: values.min ?? null, + max: values.max ?? null, + }; +} + +function parseRate(summary, name) { + const metric = summary.metrics[name]; + const values = metricValues(metric); + if (!values) { + return null; + } + if (values.rate != null) { + return values.rate; + } + return values.value ?? null; +} + +function parseScalar(summary, name) { + const metric = summary.metrics[name]; + const values = metricValues(metric); + if (!values) { + return null; + } + if (Number.isFinite(values.value)) { + return values.value; + } + if (Number.isFinite(values.avg)) { + return values.avg; + } + if (Number.isFinite(values.max)) { + return values.max; + } + return null; +} + +function median(numbers) { + const vals = numbers.filter((n) => Number.isFinite(n)).slice().sort((a, b) => a - b); + if (vals.length === 0) { + return null; + } + const mid = Math.floor(vals.length / 2); + return vals.length % 2 === 0 ? (vals[mid - 1] + vals[mid]) / 2 : vals[mid]; +} + +function maybeShuffle(list, enabled) { + const copy = list.slice(); + if (!enabled) { + return copy; + } + for (let i = copy.length - 1; i > 0; i -= 1) { + const j = Math.floor(Math.random() * (i + 1)); + [copy[i], copy[j]] = [copy[j], copy[i]]; + } + return copy; +} + +function validateService(service) { + if (!service.name) { + throw new Error('Each service must have a name'); + } +} + +function getPassSetupCommand(service, passName) { + if (passName === 'memory') { + return service.memorySetupCommand || ''; + } + if (passName === 'db') { + return service.dbSetupCommand || ''; + } + return ''; +} + +function getDbSeedCommand(config, service) { + if (service.dbSeedCommand && service.dbSeedCommand.trim()) { + return service.dbSeedCommand; + } + if (config.defaultDbSeedCommand && config.defaultDbSeedCommand.trim()) { + return config.defaultDbSeedCommand; + } + return ''; +} + +function buildSetupEnv(config) { + const env = { ...process.env }; + const projectId = config?.cloudRun?.projectId; + if ((!env.GOOGLE_CLOUD_PROJECT || !env.GOOGLE_CLOUD_PROJECT.trim()) && projectId) { + env.GOOGLE_CLOUD_PROJECT = String(projectId).trim(); + } + return env; +} + +function buildK6Env({ config, service, baseUrl, mode, targetRps, duration }) { + const env = { ...process.env }; + env.RUN_MODE = mode; + env.BASE_URL = baseUrl; + env.BASE_PATH = config.basePath; + env.TARGET_RPS = String(targetRps); + env.DURATION = duration; + env.PAGE_SIZE = String(config.workload.pageSize); + env.SEED_FETCH_PAGES = String(config.workload.seedFetchPages); + env.SEED_PAGE_SIZE = String(config.workload.seedPageSize); + env.LIST_WEIGHT = String(config.workload.listPercent); + env.GET_WEIGHT = String(config.workload.getPercent); + env.CREATE_WEIGHT = String(config.workload.createPercent); + env.UPDATE_WEIGHT = String(config.workload.updatePercent); + env.DELETE_WEIGHT = String(config.workload.deletePercent); + env.COLD_START_MAX_WAIT_SECONDS = String(config.coldStart?.maxWaitSeconds ?? 60); + env.COLD_START_PROBE_INTERVAL_MS = String(config.coldStart?.probeIntervalMs ?? 500); + env.COLD_START_ENDPOINT = String(config.coldStart?.endpoint || '/lamps?pageSize=1'); + env.COLD_START_SUCCESS_STATUS = String(config.coldStart?.successStatus ?? 200); + if (service.authHeader) { + env.AUTH_HEADER = service.authHeader; + } + return env; +} + +function runK6Phase({ scenarioPath, outputFile, env }) { + runCommand('k6', ['run', scenarioPath, '--summary-export', outputFile], { env }); + return readJson(outputFile); +} + +function aggregatePass(serviceRuns) { + const successfulRuns = serviceRuns.filter((r) => !r.failed); + const fixedP95 = median(successfulRuns.map((r) => r.fixed.duration?.p95)); + const fixedP99 = median(successfulRuns.map((r) => r.fixed.duration?.p99)); + const fixedAvg = median(successfulRuns.map((r) => r.fixed.duration?.avg)); + const fixedErrorRate = median(successfulRuns.map((r) => r.fixed.errorRate)); + const maxStableRps = median(successfulRuns.map((r) => r.stress.maxStableRps)); + + let extreme = null; + const extremeRuns = successfulRuns.filter((r) => r.extreme); + if (extremeRuns.length > 0) { + extreme = { + p95: median(extremeRuns.map((r) => r.extreme.duration?.p95)), + p99: median(extremeRuns.map((r) => r.extreme.duration?.p99)), + avg: median(extremeRuns.map((r) => r.extreme.duration?.avg)), + errorRate: median(extremeRuns.map((r) => r.extreme.errorRate)), + }; + } + + const coldSuccessfulRuns = successfulRuns.filter((r) => + r.coldStart && + !r.coldStart.failed && + Number.isFinite(r.coldStart.readyMs) + ); + const coldFailedRuns = successfulRuns.filter((r) => r.coldStart && r.coldStart.failed); + const coldStart = { + readyMs: median(coldSuccessfulRuns.map((r) => r.coldStart.readyMs)), + attempts: median(coldSuccessfulRuns.map((r) => r.coldStart.attempts)), + errorRate: median(coldSuccessfulRuns.map((r) => r.coldStart.errorRate)), + successfulColdSamples: coldSuccessfulRuns.length, + failedColdSamples: coldFailedRuns.length, + }; + + return { + successfulIterations: successfulRuns.length, + failedIterations: serviceRuns.length - successfulRuns.length, + coldStart, + fixed: { + p95: fixedP95, + p99: fixedP99, + avg: fixedAvg, + errorRate: fixedErrorRate, + }, + stress: { + maxStableRps, + }, + extreme, + }; +} + +async function main() { + const args = parseArgs(process.argv); + + const config = readJson(args.config); + const services = readJson(args.services); + + services.forEach(validateService); + + const configuredPasses = Array.isArray(config.passes) ? config.passes : ['memory', 'db']; + const passes = args.passes && args.passes.length > 0 ? args.passes : configuredPasses; + + const stamp = nowStamp(); + const scenarioPath = path.join('benchmarks', 'k6', 'scenarios.js'); + const rawRoot = path.join(args.resultsDir, 'raw', stamp); + const setupEnv = buildSetupEnv(config); + + ensureDir(rawRoot); + + runCommand('k6', ['version']); + + const report = { + generatedAt: new Date().toISOString(), + runId: stamp, + config, + passes, + rawRoot, + runs: {}, + aggregated: {}, + }; + + for (const passName of passes) { + if (!['memory', 'db'].includes(passName)) { + throw new Error(`Unsupported pass: ${passName}`); + } + + report.runs[passName] = {}; + const order = maybeShuffle(services, Boolean(config.randomizeServiceOrder)); + + for (const service of order) { + const baseUrl = passName === 'memory' ? service.memoryUrl : service.dbUrl; + if (!baseUrl) { + throw new Error(`Missing ${passName} URL for service ${service.name}`); + } + + const passSetupCommand = getPassSetupCommand(service, passName); + if (!args.skipSetup && passSetupCommand) { + runShell(passSetupCommand, setupEnv); + } + + const serviceRuns = []; + + for (let iteration = 1; iteration <= Number(config.iterationsPerPass || 1); iteration += 1) { + console.log(`\n=== ${passName.toUpperCase()} :: ${service.name} :: iteration ${iteration} ===`); + const iterDir = path.join(rawRoot, passName, service.name, `iter-${iteration}`); + ensureDir(iterDir); + + try { + if (passName === 'db') { + const dbSeedCommand = getDbSeedCommand(config, service); + if (dbSeedCommand) { + runShell(dbSeedCommand, setupEnv); + } + } + + let coldStart = null; + const shouldRunColdStart = Boolean(config.coldStart?.enabled) && + (Boolean(config.coldStart.runPerIteration) || iteration === 1); + + if (shouldRunColdStart) { + const cooldownSeconds = Number(config.coldStart?.cooldownSeconds || 0); + if (cooldownSeconds > 0) { + console.log(`Waiting ${cooldownSeconds}s cooldown before cold-start probe...`); + await sleep(cooldownSeconds * 1000); + } + + const coldStartFile = path.join(iterDir, 'cold-start.json'); + const coldStartSummary = runK6Phase({ + scenarioPath, + outputFile: coldStartFile, + env: buildK6Env({ + config, + service, + baseUrl, + mode: 'cold_start', + targetRps: 1, + duration: `${Number(config.coldStart?.maxWaitSeconds || 60)}s`, + }), + }); + + const readyMetric = parseMetric(coldStartSummary, 'cold_start_ready_ms'); + const readyMs = readyMetric?.avg ?? null; + coldStart = { + readyMs, + attempts: parseScalar(coldStartSummary, 'cold_start_attempts'), + firstSuccessStatus: parseScalar(coldStartSummary, 'cold_start_first_success_status'), + errorRate: parseRate(coldStartSummary, 'cold_start_error_rate') || 0, + duration: parseMetric(coldStartSummary, 'cold_start_req_duration'), + failed: !Number.isFinite(readyMs), + }; + + if (coldStart.failed) { + console.warn( + `[cold-start] ${passName}/${service.name}/iter-${iteration}: no successful response within max wait`, + ); + } + } + + await runPrecheckWithRetry(baseUrl, config.basePath, service.authHeader || '', { + retries: 6, + initialDelayMs: 1000, + maxDelayMs: 10000, + }); + + const warmupFile = path.join(iterDir, 'warmup.json'); + const warmupSummary = runK6Phase({ + scenarioPath, + outputFile: warmupFile, + env: buildK6Env({ + config, + service, + baseUrl, + mode: 'warmup', + targetRps: config.warmup.rps, + duration: config.warmup.duration, + }), + }); + + const fixedFile = path.join(iterDir, 'fixed.json'); + const fixedSummary = runK6Phase({ + scenarioPath, + outputFile: fixedFile, + env: buildK6Env({ + config, + service, + baseUrl, + mode: 'fixed', + targetRps: config.fixed.rps, + duration: config.fixed.duration, + }), + }); + + let maxStableRps = null; + const stressSteps = []; + for (const rps of config.stress.rpsSteps) { + const stressFile = path.join(iterDir, `stress-${rps}.json`); + const stressSummary = runK6Phase({ + scenarioPath, + outputFile: stressFile, + env: buildK6Env({ + config, + service, + baseUrl, + mode: 'stress', + targetRps: rps, + duration: config.stress.stepDuration, + }), + }); + + const dur = parseMetric(stressSummary, 'stress_req_duration'); + const err = parseRate(stressSummary, 'stress_error_rate') || 0; + const passed = + Number.isFinite(dur?.p95) && + dur.p95 <= Number(config.slo.p95Ms) && + err <= Number(config.slo.errorRate); + + stressSteps.push({ rps, duration: dur, errorRate: err, passed }); + + if (passed) { + maxStableRps = rps; + } else { + break; + } + } + + let extreme = null; + const shouldRunExtreme = Boolean(config.extreme?.enabled) && + (Boolean(config.extreme.runPerIteration) || iteration === 1); + + if (shouldRunExtreme) { + const extremeFile = path.join(iterDir, 'extreme-1000.json'); + const extremeSummary = runK6Phase({ + scenarioPath, + outputFile: extremeFile, + env: buildK6Env({ + config, + service, + baseUrl, + mode: 'extreme', + targetRps: config.extreme.rps, + duration: config.extreme.duration, + }), + }); + + extreme = { + duration: parseMetric(extremeSummary, 'extreme_req_duration'), + errorRate: parseRate(extremeSummary, 'extreme_error_rate') || 0, + }; + } + + const serviceRun = { + iteration, + failed: false, + coldStart, + warmup: { + duration: parseMetric(warmupSummary, 'warmup_req_duration'), + errorRate: parseRate(warmupSummary, 'warmup_error_rate') || 0, + }, + fixed: { + duration: parseMetric(fixedSummary, 'fixed_req_duration'), + errorRate: parseRate(fixedSummary, 'fixed_error_rate') || 0, + }, + stress: { + maxStableRps, + steps: stressSteps, + }, + extreme, + }; + + serviceRuns.push(serviceRun); + } catch (error) { + const message = error && error.message ? error.message : String(error); + console.warn(`[iteration failed] ${passName}/${service.name}/iter-${iteration}: ${message}. Continuing...`); + serviceRuns.push({ + iteration, + failed: true, + error: message, + coldStart: null, + }); + } + } + + report.runs[passName][service.name] = serviceRuns; + } + + report.aggregated[passName] = {}; + for (const [serviceName, serviceRuns] of Object.entries(report.runs[passName])) { + report.aggregated[passName][serviceName] = aggregatePass(serviceRuns); + } + } + + const reportPath = path.join(args.resultsDir, 'run-report.json'); + fs.writeFileSync(reportPath, JSON.stringify(report, null, 2), 'utf8'); + + const summaryPath = path.join(args.resultsDir, 'summary.md'); + writeSummary(report, summaryPath); + + console.log(`\nWrote report: ${reportPath}`); + console.log(`Wrote summary: ${summaryPath}`); +} + +main().catch((err) => { + console.error(err.message || err); + process.exit(1); +}); diff --git a/benchmarks/k6/scenarios.js b/benchmarks/k6/scenarios.js new file mode 100644 index 00000000..e8a4fa02 --- /dev/null +++ b/benchmarks/k6/scenarios.js @@ -0,0 +1,319 @@ +import http from 'k6/http'; +import { check } from 'k6'; +import { sleep } from 'k6'; +import { Gauge, Rate, Trend } from 'k6/metrics'; + +const RUN_MODE = (__ENV.RUN_MODE || 'fixed').trim(); +const BASE_URL = (__ENV.BASE_URL || '').replace(/\/$/, ''); +const BASE_PATH = __ENV.BASE_PATH || '/v1'; +const TARGET_RPS = Number(__ENV.TARGET_RPS || 1); +const DURATION = __ENV.DURATION || '60s'; +const PAGE_SIZE = Number(__ENV.PAGE_SIZE || 25); +const SEED_FETCH_PAGES = Number(__ENV.SEED_FETCH_PAGES || 10); +const SEED_PAGE_SIZE = Number(__ENV.SEED_PAGE_SIZE || 100); +const AUTH_HEADER = __ENV.AUTH_HEADER || ''; +const COLD_START_ENDPOINT = __ENV.COLD_START_ENDPOINT || '/lamps?pageSize=1'; +const COLD_START_SUCCESS_STATUS = Number(__ENV.COLD_START_SUCCESS_STATUS || 200); +const COLD_START_MAX_WAIT_SECONDS = Number(__ENV.COLD_START_MAX_WAIT_SECONDS || 60); +const COLD_START_PROBE_INTERVAL_MS = Number(__ENV.COLD_START_PROBE_INTERVAL_MS || 500); + +const LIST_WEIGHT = Number(__ENV.LIST_WEIGHT || 50); +const GET_WEIGHT = Number(__ENV.GET_WEIGHT || 20); +const CREATE_WEIGHT = Number(__ENV.CREATE_WEIGHT || 20); +const UPDATE_WEIGHT = Number(__ENV.UPDATE_WEIGHT || 7); +const DELETE_WEIGHT = Number(__ENV.DELETE_WEIGHT || 3); + +const PRE_ALLOCATED_VUS = Number( + __ENV.PRE_ALLOCATED_VUS || Math.max(10, Math.ceil(TARGET_RPS * 2)) +); +const MAX_VUS = Number(__ENV.MAX_VUS || Math.max(50, Math.ceil(TARGET_RPS * 4))); + +if (!BASE_URL) { + throw new Error('BASE_URL is required'); +} + +const requestDuration = new Trend(`${RUN_MODE}_req_duration`, true); +const errorRate = new Rate(`${RUN_MODE}_error_rate`); +const coldStartReadyMs = new Trend('cold_start_ready_ms', true); +const coldStartAttempts = new Gauge('cold_start_attempts'); +const coldStartFirstSuccessStatus = new Gauge('cold_start_first_success_status'); + +export const options = { + discardResponseBodies: false, + summaryTrendStats: ['avg', 'min', 'med', 'max', 'p(90)', 'p(95)', 'p(99)'], + scenarios: RUN_MODE === 'cold_start' + ? { + main: { + executor: 'per-vu-iterations', + vus: 1, + iterations: 1, + maxDuration: `${Math.max(10, COLD_START_MAX_WAIT_SECONDS + 10)}s`, + }, + } + : { + main: { + executor: 'constant-arrival-rate', + rate: TARGET_RPS, + timeUnit: '1s', + duration: DURATION, + preAllocatedVUs: PRE_ALLOCATED_VUS, + maxVUs: MAX_VUS, + }, + }, +}; + +const headers = {}; + +if (AUTH_HEADER) { + headers.Authorization = AUTH_HEADER; +} + +let vuOwnedIds = []; + +function url(pathAndQuery) { + return `${BASE_URL}${BASE_PATH}${pathAndQuery}`; +} + +function parseJson(resp) { + try { + return resp.json(); + } catch (_err) { + return null; + } +} + +function track(resp, ok) { + requestDuration.add(resp.timings.duration); + errorRate.add(!ok); +} + +function req(method, endpoint, body, expectedStatuses) { + const requestHeaders = { ...headers }; + if (body !== undefined && body !== null) { + requestHeaders['Content-Type'] = 'application/json'; + } + const response = http.request(method, url(endpoint), body, { headers: requestHeaders }); + const ok = check(response, { + [`${method} ${endpoint} status`]: (r) => expectedStatuses.includes(r.status), + }); + track(response, ok); + return { response, ok }; +} + +function randomBool() { + return Math.random() < 0.5; +} + +function randomFrom(list) { + return list[Math.floor(Math.random() * list.length)]; +} + +function listLamps() { + const { response } = req('GET', `/lamps?pageSize=${PAGE_SIZE}`, null, [200]); + const body = parseJson(response); + check(body, { + 'list lamps has data array': (b) => b && Array.isArray(b.data), + 'list lamps has hasMore': (b) => b && typeof b.hasMore === 'boolean', + }); +} + +function createLamp() { + const payload = JSON.stringify({ status: randomBool() }); + const { response, ok } = req('POST', '/lamps', payload, [201]); + if (!ok) { + return; + } + + const body = parseJson(response); + const hasId = check(body, { + 'create lamp has id': (b) => b && typeof b.id === 'string' && b.id.length > 0, + 'create lamp has status': (b) => b && typeof b.status === 'boolean', + }); + + if (hasId) { + vuOwnedIds.push(body.id); + } +} + +function pickLampId(data) { + if (vuOwnedIds.length > 0) { + return randomFrom(vuOwnedIds); + } + if (data.seedIds.length > 0) { + return randomFrom(data.seedIds); + } + return null; +} + +function getLamp(data) { + let lampId = pickLampId(data); + if (!lampId) { + createLamp(); + lampId = vuOwnedIds[vuOwnedIds.length - 1] || null; + } + if (!lampId) { + return; + } + + const { response } = req('GET', `/lamps/${lampId}`, null, [200]); + const body = parseJson(response); + check(body, { + 'get lamp has id': (b) => b && typeof b.id === 'string', + 'get lamp has status': (b) => b && typeof b.status === 'boolean', + }); +} + +function updateLamp(data) { + let lampId = pickLampId(data); + if (!lampId) { + createLamp(); + lampId = vuOwnedIds[vuOwnedIds.length - 1] || null; + } + if (!lampId) { + return; + } + + const payload = JSON.stringify({ status: randomBool() }); + const { response } = req('PUT', `/lamps/${lampId}`, payload, [200]); + const body = parseJson(response); + check(body, { + 'update lamp has id': (b) => b && typeof b.id === 'string', + 'update lamp has status': (b) => b && typeof b.status === 'boolean', + }); +} + +function deleteLamp() { + let lampId = null; + + if (vuOwnedIds.length > 0) { + lampId = vuOwnedIds.pop(); + } else { + const payload = JSON.stringify({ status: randomBool() }); + const { response, ok } = req('POST', '/lamps', payload, [201]); + if (!ok) { + return; + } + const body = parseJson(response); + if (!body || typeof body.id !== 'string') { + return; + } + lampId = body.id; + } + + if (!lampId) { + return; + } + + req('DELETE', `/lamps/${lampId}`, null, [204]); +} + +function pickOperation() { + const total = LIST_WEIGHT + GET_WEIGHT + CREATE_WEIGHT + UPDATE_WEIGHT + DELETE_WEIGHT; + const pick = Math.random() * total; + + if (pick < LIST_WEIGHT) { + return 'list'; + } + if (pick < LIST_WEIGHT + GET_WEIGHT) { + return 'get'; + } + if (pick < LIST_WEIGHT + GET_WEIGHT + CREATE_WEIGHT) { + return 'create'; + } + if (pick < LIST_WEIGHT + GET_WEIGHT + CREATE_WEIGHT + UPDATE_WEIGHT) { + return 'update'; + } + return 'delete'; +} + +export function setup() { + if (RUN_MODE === 'cold_start') { + return { seedIds: [] }; + } + + const seedIds = []; + let cursor = null; + + for (let i = 0; i < SEED_FETCH_PAGES; i += 1) { + const query = cursor + ? `/lamps?pageSize=${SEED_PAGE_SIZE}&cursor=${encodeURIComponent(cursor)}` + : `/lamps?pageSize=${SEED_PAGE_SIZE}`; + + const response = http.get(url(query), { headers }); + const ok = check(response, { + 'seed fetch status 200': (r) => r.status === 200, + }); + + if (!ok) { + break; + } + + const body = parseJson(response); + if (!body || !Array.isArray(body.data)) { + break; + } + + for (const lamp of body.data) { + if (lamp && typeof lamp.id === 'string') { + seedIds.push(lamp.id); + } + } + + if (!body.hasMore || !body.nextCursor) { + break; + } + + cursor = body.nextCursor; + } + + return { seedIds }; +} + +function runColdStartProbe() { + const startedAtMs = Date.now(); + const deadlineMs = startedAtMs + Math.max(1000, COLD_START_MAX_WAIT_SECONDS * 1000); + let attempts = 0; + + while (Date.now() <= deadlineMs) { + const response = http.get(url(COLD_START_ENDPOINT), { headers }); + attempts += 1; + + const ok = response.status === COLD_START_SUCCESS_STATUS; + requestDuration.add(response.timings.duration); + errorRate.add(!ok); + + if (ok) { + const readyMs = Date.now() - startedAtMs; + coldStartReadyMs.add(readyMs); + coldStartAttempts.add(attempts); + coldStartFirstSuccessStatus.add(response.status); + return; + } + + sleep(Math.max(10, COLD_START_PROBE_INTERVAL_MS) / 1000); + } + + coldStartAttempts.add(attempts); + coldStartFirstSuccessStatus.add(0); +} + +export default function (data) { + if (RUN_MODE === 'cold_start') { + runColdStartProbe(); + return; + } + + const operation = pickOperation(); + + if (operation === 'list') { + listLamps(); + } else if (operation === 'get') { + getLamp(data); + } else if (operation === 'create') { + createLamp(); + } else if (operation === 'update') { + updateLamp(data); + } else { + deleteLamp(); + } +} diff --git a/benchmarks/k6/services.json b/benchmarks/k6/services.json new file mode 100644 index 00000000..1d3633dc --- /dev/null +++ b/benchmarks/k6/services.json @@ -0,0 +1,62 @@ +[ + { + "name": "typescript", + "memoryUrl": "https://typescript-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://typescript-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "typescript-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update typescript-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars DATABASE_URL", + "dbSetupCommand": "gcloud run services update typescript-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars DATABASE_URL=\"$BENCHMARK_DATABASE_URL\"", + "dbSeedCommand": "" + }, + { + "name": "python", + "memoryUrl": "https://python-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://python-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "python-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update python-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars DATABASE_URL", + "dbSetupCommand": "gcloud run services update python-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars DATABASE_URL=\"$BENCHMARK_DATABASE_URL\"", + "dbSeedCommand": "" + }, + { + "name": "java", + "memoryUrl": "https://java-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://java-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "java-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update java-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars DATABASE_URL", + "dbSetupCommand": "gcloud run services update java-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars SPRING_DATASOURCE_URL=\"$BENCHMARK_JDBC_DATABASE_URL\" --update-env-vars DB_USER=\"$BENCHMARK_DB_USER\" --update-env-vars DB_PASSWORD=\"$BENCHMARK_DB_PASSWORD\"", + "dbSeedCommand": "" + }, + { + "name": "csharp", + "memoryUrl": "https://csharp-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://csharp-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "csharp-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update csharp-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars ConnectionStrings__LampControl", + "dbSetupCommand": "gcloud run services update csharp-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars ConnectionStrings__LampControl=\"$BENCHMARK_CSHARP_CONNECTION_STRING\"", + "dbSeedCommand": "" + }, + { + "name": "go", + "memoryUrl": "https://go-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://go-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "go-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update go-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars DATABASE_URL", + "dbSetupCommand": "gcloud run services update go-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars DATABASE_URL=\"$BENCHMARK_DATABASE_URL\"", + "dbSeedCommand": "" + }, + { + "name": "kotlin", + "memoryUrl": "https://kotlin-lamp-control-api-827868544165.europe-west1.run.app", + "dbUrl": "https://kotlin-lamp-control-api-827868544165.europe-west1.run.app", + "cloudRunService": "kotlin-lamp-control-api", + "cloudRunRegion": "europe-west1", + "memorySetupCommand": "gcloud run services update kotlin-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --remove-env-vars DATABASE_URL", + "dbSetupCommand": "gcloud run services update kotlin-lamp-control-api --project \"$GOOGLE_CLOUD_PROJECT\" --region europe-west1 --update-env-vars DATABASE_URL=\"$BENCHMARK_DATABASE_URL\"", + "dbSeedCommand": "" + } +] diff --git a/benchmarks/k6/summary.js b/benchmarks/k6/summary.js new file mode 100644 index 00000000..ae0cee5d --- /dev/null +++ b/benchmarks/k6/summary.js @@ -0,0 +1,106 @@ +#!/usr/bin/env node +const fs = require('fs'); + +function formatNumber(value, digits = 2) { + if (!Number.isFinite(value)) { + return 'n/a'; + } + return value.toFixed(digits); +} + +function writeSummary(report, outputFile) { + const lines = []; + lines.push('# Benchmark Summary'); + lines.push(''); + lines.push(`Generated: ${report.generatedAt}`); + lines.push(''); + + for (const passName of Object.keys(report.aggregated || {})) { + const rows = Object.entries(report.aggregated[passName] || {}); + rows.sort((a, b) => { + const ap = a[1].fixed?.p95; + const bp = b[1].fixed?.p95; + if (!Number.isFinite(ap) && !Number.isFinite(bp)) return 0; + if (!Number.isFinite(ap)) return 1; + if (!Number.isFinite(bp)) return -1; + return ap - bp; + }); + + lines.push(`## ${passName === 'memory' ? 'Memory Pass Ranking' : 'DB Pass Ranking'}`); + lines.push(''); + lines.push('| Rank | Service | p95 (ms) | p99 (ms) | Avg (ms) | Error Rate | Max Stable RPS |'); + lines.push('|---|---|---:|---:|---:|---:|---:|'); + + rows.forEach(([serviceName, metrics], idx) => { + lines.push( + `| ${idx + 1} | ${serviceName} | ${formatNumber(metrics.fixed?.p95)} | ${formatNumber(metrics.fixed?.p99)} | ${formatNumber(metrics.fixed?.avg)} | ${formatNumber((metrics.fixed?.errorRate ?? NaN) * 100, 3)}% | ${formatNumber(metrics.stress?.maxStableRps, 0)} |` + ); + }); + lines.push(''); + } + + const memory = report.aggregated?.memory || {}; + const db = report.aggregated?.db || {}; + const sharedServices = Object.keys(memory).filter((name) => db[name]); + + if (sharedServices.length > 0) { + lines.push('## Memory vs DB Delta'); + lines.push(''); + lines.push('| Service | Memory p95 (ms) | DB p95 (ms) | Delta (DB - Memory) |'); + lines.push('|---|---:|---:|---:|'); + + for (const serviceName of sharedServices) { + const mem = memory[serviceName]?.fixed?.p95; + const dbp = db[serviceName]?.fixed?.p95; + const delta = Number.isFinite(mem) && Number.isFinite(dbp) ? dbp - mem : null; + lines.push(`| ${serviceName} | ${formatNumber(mem)} | ${formatNumber(dbp)} | ${formatNumber(delta)} |`); + } + + lines.push(''); + } + + lines.push('## Cold Start Appendix'); + lines.push(''); + lines.push('| Pass | Service | Ready Time (ms) | Attempts | Error Rate | Samples (ok/failed) |'); + lines.push('|---|---|---:|---:|---:|---:|'); + for (const [passName, servicesMap] of Object.entries(report.aggregated || {})) { + for (const [serviceName, metrics] of Object.entries(servicesMap || {})) { + const cold = metrics.coldStart || {}; + const okSamples = Number.isFinite(cold.successfulColdSamples) ? cold.successfulColdSamples : 0; + const failedSamples = Number.isFinite(cold.failedColdSamples) ? cold.failedColdSamples : 0; + lines.push( + `| ${passName} | ${serviceName} | ${formatNumber(cold.readyMs)} | ${formatNumber(cold.attempts, 0)} | ${formatNumber((cold.errorRate ?? NaN) * 100, 3)}% | ${okSamples}/${failedSamples} |` + ); + } + } + lines.push(''); + + const extremeRps = report.config?.extreme?.rps; + const extremeLabel = Number.isFinite(extremeRps) + ? `Extreme Load Appendix (${extremeRps} RPS)` + : 'Extreme Load Appendix'; + lines.push(`## ${extremeLabel}`); + lines.push(''); + lines.push('| Pass | Service | p95 (ms) | p99 (ms) | Avg (ms) | Error Rate |'); + lines.push('|---|---|---:|---:|---:|---:|'); + + for (const [passName, servicesMap] of Object.entries(report.aggregated || {})) { + for (const [serviceName, metrics] of Object.entries(servicesMap || {})) { + if (!metrics.extreme) { + continue; + } + lines.push( + `| ${passName} | ${serviceName} | ${formatNumber(metrics.extreme?.p95)} | ${formatNumber(metrics.extreme?.p99)} | ${formatNumber(metrics.extreme?.avg)} | ${formatNumber((metrics.extreme?.errorRate ?? NaN) * 100, 3)}% |` + ); + } + } + + lines.push(''); + lines.push('Raw per-run k6 summaries are under `benchmarks/results/raw/`.'); + + fs.writeFileSync(outputFile, `${lines.join('\n')}\n`, 'utf8'); +} + +module.exports = { + writeSummary, +}; diff --git a/benchmarks/results/.gitkeep b/benchmarks/results/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/benchmarks/results/summary.md b/benchmarks/results/summary.md new file mode 100644 index 00000000..c115ab5d --- /dev/null +++ b/benchmarks/results/summary.md @@ -0,0 +1,11 @@ +# Benchmark Summary + +No benchmark runs yet. + +Run: + +```bash +node benchmarks/k6/run-benchmarks.js +``` + +Then this file will be replaced with ranked results. diff --git a/docs/POSTGRES_STARTUP_VARIABLES.md b/docs/POSTGRES_STARTUP_VARIABLES.md new file mode 100644 index 00000000..38867806 --- /dev/null +++ b/docs/POSTGRES_STARTUP_VARIABLES.md @@ -0,0 +1,139 @@ +# PostgreSQL Setup Guide (Per Language) + +Use this as a quick reference when you want each implementation to run with PostgreSQL instead of in-memory storage. + +## Quick Start + +1. Pick the language implementation you want to run. +2. Export the variables listed for that language. +3. Start the app in your usual mode (`serve-only`, `serve`, or `migrate` where supported). + +## TypeScript (`src/typescript`) + +Set: + +```bash +export DATABASE_URL='postgresql://:@:5432/?schema=public' +``` + +Notes: +- PostgreSQL is enabled when `DATABASE_URL` is set. +- If `DATABASE_URL` is unset, TypeScript uses in-memory storage. + +## Python (`src/python`) + +Set: + +```bash +export DATABASE_URL='postgresql://:@:5432/' +``` + +Optional pool tuning: + +```bash +export DB_POOL_MIN_SIZE='5' +export DB_POOL_MAX_SIZE='20' +``` + +Notes: +- PostgreSQL is enabled when `DATABASE_URL` is set. +- `DB_*` variables alone do not switch Python to PostgreSQL mode. + +## Java (`src/java`) + +Set: + +```bash +export SPRING_DATASOURCE_URL='jdbc:postgresql://:5432/' +export DB_USER='' +export DB_PASSWORD='' +``` + +Alternative: + +```bash +export DATABASE_URL='jdbc:postgresql://:5432/' +export DB_USER='' +export DB_PASSWORD='' +``` + +Notes: +- URL must be JDBC format (`jdbc:postgresql://...`). +- A plain `postgresql://...` URL will not work for Java datasource config. + +## C# (`src/csharp`) + +Set: + +```bash +export ConnectionStrings__LampControl='Host=;Port=5432;Database=;Username=;Password=' +``` + +Notes: +- PostgreSQL is enabled when `ConnectionStrings__LampControl` is set. +- `DATABASE_URL` is not used by C# runtime config. + +## Go (`src/go`) + +Recommended: + +```bash +export DATABASE_URL='postgres://:@:5432/?sslmode=disable' +``` + +Alternative (component vars): + +```bash +export DB_HOST='' +export DB_PORT='5432' +export DB_NAME='' +export DB_USER='' +export DB_PASSWORD='' +``` + +Optional pool tuning: + +```bash +export DB_POOL_MIN_SIZE='0' +export DB_POOL_MAX_SIZE='4' +``` + +Notes: +- `DATABASE_URL` takes precedence over component vars. + +## Kotlin (`src/kotlin`) + +Recommended: + +```bash +export DATABASE_URL='postgresql://:@:5432/' +``` + +Alternative (component vars): + +```bash +export DB_HOST='' +export DB_PORT='5432' +export DB_NAME='' +export DB_USER='' +export DB_PASSWORD='' +``` + +Optional pool/timeout tuning: + +```bash +export DB_POOL_MIN_SIZE='0' +export DB_POOL_MAX_SIZE='4' +export DB_MAX_LIFETIME_MS='3600000' +export DB_IDLE_TIMEOUT_MS='1800000' +export DB_CONNECTION_TIMEOUT_MS='30000' +``` + +Notes: +- If you use `DATABASE_URL`, keep it in standard `postgresql://...` or `postgres://...` form. + +## Common Gotchas + +- Java needs `jdbc:postgresql://...`; others usually use `postgresql://...` or `postgres://...`. +- C# uses `ConnectionStrings__LampControl`, not `DATABASE_URL`. +- If an app still runs in memory mode, first verify the exact variable name is exported in the same shell/session used to start the app.