feat: add single-instance Cloud Run benchmark harness#365
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a k6-based harness to benchmark single-instance Cloud Run deployments across the repo’s six language implementations, with scripts to run memory/DB passes and generate ranked summaries.
Changes:
- Add k6 workload script and Node.js orchestration to execute warmup/fixed/stress/extreme phases and export reports.
- Add Cloud Run parity configuration helper (gcloud command generator/executor) plus benchmark config/service maps.
- Add benchmark results directory scaffolding, placeholder summary, and ignore rules for generated artifacts.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| benchmarks/results/summary.md | Placeholder summary that gets overwritten by generated rankings. |
| benchmarks/results/.gitkeep | Keeps benchmarks/results/ tracked in git. |
| benchmarks/k6/services.json | Service map template for memory/DB URLs and optional DB seed command per language. |
| benchmarks/k6/scenarios.js | k6 constant-arrival-rate CRUD workload with weighted operation mix and seeding. |
| benchmarks/k6/run-benchmarks.js | Orchestrates benchmark passes/runs, exports raw k6 summaries, aggregates results, and writes run-report.json + summary.md. |
| benchmarks/k6/generate-summary.js | Regenerates benchmarks/results/summary.md from run-report.json. |
| benchmarks/k6/configure-cloud-run.js | Applies (or dry-runs) uniform Cloud Run settings via gcloud run services update. |
| benchmarks/k6/config.json | Default benchmark parameters, SLO gating, extreme appendix settings, and Cloud Run parity config. |
| benchmarks/k6/README.md | Usage guide and interpretation notes for running the benchmark harness. |
| .gitignore | Ignores generated raw benchmark exports and the JSON run report. |
| - `db` pass (realistic signal) | ||
| - Fixed, fairness-first Cloud Run settings for all services | ||
| - Main ranking at fixed concurrency pressure with `concurrency=80` | ||
| - Optional non-ranking extreme run at concurrency `1000` | ||
| - Sequential service execution (no parallel cross-service load) |
There was a problem hiding this comment.
README describes the extreme run as "concurrency 1000", but the harness actually drives load via k6 constant-arrival-rate at extreme.rps (1000) while Cloud Run concurrency remains whatever is configured in config.json (80). This is misleading for interpreting results; either rename this to "1000 RPS" (or similar) and clarify Cloud Run concurrency is unchanged, or update the tooling/config to also apply a cloudRun.concurrency=1000 setting for the extreme appendix.
| "extreme": { | ||
| "enabled": true, | ||
| "duration": "60s", | ||
| "rps": 1000, | ||
| "runPerIteration": false | ||
| }, |
There was a problem hiding this comment.
extreme.enabled is set to true by default, which contradicts the README calling the extreme run "optional" and will also trigger very large VU allocations for 1000 RPS in scenarios.js (preAllocatedVUs becomes ~2000). Consider defaulting this to false (or lowering the default RPS) and having users opt-in when they actually want the appendix run.
| '--max-instances', | ||
| String(cloudRun.maxInstances), | ||
| '--min-instances', | ||
| String(cloudRun.minInstances), | ||
| '--concurrency', | ||
| String(cloudRun.concurrency), | ||
| '--cpu', | ||
| String(cloudRun.cpu), | ||
| '--memory', | ||
| String(cloudRun.memory), | ||
| '--timeout', | ||
| String(cloudRun.timeout), |
There was a problem hiding this comment.
configure-cloud-run.js will happily pass "undefined" into gcloud flags if any cloudRun fields are missing from the provided config file (because of String(cloudRun.*)). Add validation with a clear error message (or safe defaults) for required keys like maxInstances/minInstances/concurrency/cpu/memory/timeout before building the command.
davideme
force-pushed
the
codex/benchmark-cloud-run-versions
branch
from
76dbdde to
2764dda
Compare
* feat(csharp): add DATABASE_URL fallback for postgres connection * test(csharp): fix file provider in connection resolution tests * chore(csharp): address PR review feedback
#368) * docs(go): document DATABASE_URL support for postgresql scheme * docs(go): address PR review comments on DB URL schemes
1 participant
Summary
Included files
Notes