diff --git a/backend/lined/docs/experiment-tasks.md b/backend/lined/docs/experiment-tasks.md index eac1f9b..22918f2 100644 --- a/backend/lined/docs/experiment-tasks.md +++ b/backend/lined/docs/experiment-tasks.md @@ -15,7 +15,7 @@ scientific experiment work. | `experiment/prometheus-telemetry-pipeline` | Task | Runtime infrastructure | Yes | Prometheus telemetry pipeline | Add a local Prometheus collection path for the kind backend, including scrape configuration and documentation for collecting Actuator runtime metrics. | Runtime metrics from the kind backend can be collected persistently enough for scenario comparison. | | `experiment/runtime-scenario-summaries` | Task | Runtime evidence | Partial / branch-only | Runtime scenario summaries | Add a repeatable workflow for running each deployment scenario under the selected k6 workload and producing sanitized `runtime-summary.json` artifacts. | Each scenario has comparable runtime summaries ready for collector ingestion. | | `experiment/runtime-provenance-manifest` | Task | Runtime evidence | Partial / branch-only | Runtime provenance manifest | Add provenance metadata to runtime fitness outputs, including commit SHA, image tag, workload, telemetry window, configuration hash, constraint version, and fitness vector. | Runtime fitness results are traceable and reproducible for audit and paper evidence. | -| `experiment/scenario-fixture-discipline` | Task | Runtime evidence | No | Scenario fixture discipline | Define explicit workload/context profiles and repeatable input setup for Lined experiment scenario runs. | Deployment/runtime comparisons use stable fixtures instead of manual setup. | +| `experiment/scenario-fixture-discipline` | Task | Runtime evidence | Yes | Scenario fixture discipline | Define explicit workload/context profiles and repeatable input setup for Lined experiment scenario runs. | Deployment/runtime comparisons use stable fixtures instead of manual setup. | | `experiment/slo-constraint-thresholds` | Task | Runtime evidence | Yes | SLO and constraint thresholds | Define initial latency, error-rate, availability, restart, readiness, and resource-efficiency thresholds for classifying valid experiment variants. | Runtime evidence can be evaluated against explicit constraints instead of ad hoc interpretation. | | `experiment/fitness-runtime-extension` | Task | Runtime scoring | Yes | Runtime fitness extension | Extend experiment documentation and/or collector design to include telemetry metrics. | Fixed CI fitness can be compared with runtime-aware adaptive fitness. | | `experiment/runtime-aware-scoring` | Task | Runtime scoring | No | Runtime-aware scoring | Add a versioned runtime fitness score that uses summarized runtime metrics while preserving the existing structural `fitnessScore`. | Runtime-aware scalar fitness can be computed without changing historical CI fitness semantics. | diff --git a/backend/lined/docs/load-test-baseline.md b/backend/lined/docs/load-test-baseline.md index 462b535..f076193 100644 --- a/backend/lined/docs/load-test-baseline.md +++ b/backend/lined/docs/load-test-baseline.md @@ -123,6 +123,26 @@ k6 run \ load-tests/k6/load-test-baseline.js ``` +## Fixture Profiles + +For deployment/runtime comparison runs, prefer the scenario-runner fixture +profiles in `load-tests/runtime-scenarios/fixture-profiles-v1.json` instead of +passing ad hoc k6 environment values. The profiles pin the workload and setup +inputs used by the k6 script, including user count, seeded task and event +counts, VU or stress settings, duration, and think time. + +Run a stable comparison fixture through the scenario runner: + +```bash +node load-tests/runtime-scenarios/scenario-runner-cli.mjs \ + --scenario fixed-medium \ + --fixture-profile comparison-baseline \ + --base-url http://localhost:8080 +``` + +Use direct `k6 run` commands for one-off local checks. Use fixture profiles for +experiment evidence that will be compared across deployment scenarios. + ## Run with Docker If k6 is not installed locally, use the official Grafana k6 image from the diff --git a/backend/lined/docs/runtime-scenario-summaries.md b/backend/lined/docs/runtime-scenario-summaries.md index 4f89f37..5be0152 100644 --- a/backend/lined/docs/runtime-scenario-summaries.md +++ b/backend/lined/docs/runtime-scenario-summaries.md @@ -59,6 +59,23 @@ smoke, baseline, read-heavy, write-heavy, mixed, stress, negative-smoke Use `smoke` for command validation and one of the longer non-negative profiles for scenario comparison. +Fixture profiles are versioned workload/context presets from +`load-tests/runtime-scenarios/fixture-profiles-v1.json`: + +| Fixture profile | Workload | Purpose | +|-----------------|----------|---------| +| `local-smoke` | `smoke` | Minimal local command validation. | +| `comparison-baseline` | `baseline` | Stable default runtime comparison fixture. | +| `comparison-read-heavy` | `read-heavy` | Read-oriented comparison over bounded setup data. | +| `comparison-write-heavy` | `write-heavy` | Write-oriented comparison with per-iteration cleanup. | +| `comparison-mixed` | `mixed` | Mixed reads, updates, and bounded writes. | +| `comparison-stress` | `stress` | Ramping-VU local stress comparison. | + +Profiles make the workload setup explicit by pinning allowed k6 inputs such as +`USER_COUNT`, `SEED_TASK_COUNT`, `SEED_EVENT_COUNT`, `VUS`, `DURATION`, +`STRESS_MAX_VUS`, `STRESS_STAGE_DURATION`, and `THINK_TIME_SECONDS`. +They do not change backend behavior or deployment manifests. + ## Run One Scenario Run the fixed-medium scenario with the smoke workload: @@ -70,15 +87,31 @@ node load-tests/runtime-scenarios/scenario-runner-cli.mjs \ --base-url http://localhost:8080 ``` -Run the same scenario with the default baseline workload: +Run the same scenario with the stable baseline fixture: ```bash node load-tests/runtime-scenarios/scenario-runner-cli.mjs \ --scenario fixed-medium \ - --workload baseline \ + --fixture-profile comparison-baseline \ --base-url http://localhost:8080 ``` +The fixture profile supplies default workload and k6 environment inputs. +Explicit CLI options still win: + +```bash +node load-tests/runtime-scenarios/scenario-runner-cli.mjs \ + --scenario fixed-medium \ + --fixture-profile comparison-baseline \ + --workload read-heavy \ + --k6-env VUS=2 \ + --base-url http://localhost:8080 +``` + +Use overrides only when the run intentionally differs from the named fixture; +the manifest records both the selected profile and the effective workload +environment. + The runner applies the selected scenario, waits for the backend rollout, runs k6 with summary export, collects summarized Kubernetes state, and writes: @@ -119,6 +152,7 @@ side effect. The runner keeps inputs narrow: - scenario and workload names are hardcoded allowlists; +- fixture profile names and profile k6 environment keys are allowlisted; - extra k6 environment variables are allowlisted; - `BASE_URL` must point to `localhost`, `127.0.0.1`, or `[::1]` unless `--allow-remote-base-url` is provided; @@ -179,8 +213,9 @@ the metric is omitted and listed in `missing`. `runtime-summary-manifest.json` is not collector input. It records sanitized provenance such as scenario path, workload variables, git commit, CLI version, -start/end timestamps, whether the scenario was applied, HPA cleanup status, -raw pre/post restart snapshots, the restart delta, and k6 exit code. +fixture profile metadata, start/end timestamps, whether the scenario was +applied, HPA cleanup status, raw pre/post restart snapshots, the restart delta, +and k6 exit code. ## Validate Locally diff --git a/backend/lined/load-tests/runtime-scenarios/command-runner.mjs b/backend/lined/load-tests/runtime-scenarios/command-runner.mjs new file mode 100644 index 0000000..1933375 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/command-runner.mjs @@ -0,0 +1,32 @@ +import { spawnSync } from 'node:child_process'; + +export const runCommand = ( + command, + args, + { allowFailure = false, capture = false, cwd, timeoutMs } = {} +) => { + const result = spawnSync(command, args, { + cwd, + encoding: capture ? 'utf-8' : undefined, + stdio: capture ? ['ignore', 'pipe', 'pipe'] : 'inherit', + timeout: timeoutMs, + }); + + if (isTimeout(result, timeoutMs) && !allowFailure) { + throw new Error(`${command} timed out after ${timeoutMs}ms`); + } + if (result.error && !allowFailure) { + throw result.error; + } + if (result.signal && !allowFailure) { + throw new Error(`${command} was killed by signal ${result.signal}`); + } + if (!allowFailure && result.status !== 0) { + throw new Error(`${command} ${args.join(' ')} failed with exit code ${result.status}`); + } + + return result; +}; + +const isTimeout = (result, timeoutMs) => timeoutMs !== undefined + && (result.error?.code === 'ETIMEDOUT' || result.signal === 'SIGTERM'); diff --git a/backend/lined/load-tests/runtime-scenarios/fixture-profiles-v1.json b/backend/lined/load-tests/runtime-scenarios/fixture-profiles-v1.json new file mode 100644 index 0000000..928f726 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/fixture-profiles-v1.json @@ -0,0 +1,75 @@ +{ + "schema_version": 1, + "profiles": { + "local-smoke": { + "description": "Minimal fixture for local command validation.", + "workload": "smoke", + "k6_env": { + "USER_COUNT": "2", + "SEED_TASK_COUNT": "2", + "SEED_EVENT_COUNT": "2", + "THINK_TIME_SECONDS": "0" + } + }, + "comparison-baseline": { + "description": "Standard fixture for stable baseline scenario comparison.", + "workload": "baseline", + "k6_env": { + "USER_COUNT": "4", + "SEED_TASK_COUNT": "12", + "SEED_EVENT_COUNT": "8", + "VUS": "5", + "DURATION": "2m", + "THINK_TIME_SECONDS": "1" + } + }, + "comparison-read-heavy": { + "description": "Read-oriented fixture over bounded users, lobby, tasks, and events.", + "workload": "read-heavy", + "k6_env": { + "USER_COUNT": "4", + "SEED_TASK_COUNT": "12", + "SEED_EVENT_COUNT": "8", + "VUS": "5", + "DURATION": "2m", + "THINK_TIME_SECONDS": "1" + } + }, + "comparison-write-heavy": { + "description": "Write-oriented fixture with bounded setup and per-iteration cleanup.", + "workload": "write-heavy", + "k6_env": { + "USER_COUNT": "4", + "SEED_TASK_COUNT": "12", + "SEED_EVENT_COUNT": "8", + "VUS": "5", + "DURATION": "2m", + "THINK_TIME_SECONDS": "1" + } + }, + "comparison-mixed": { + "description": "Mixed read, update, and bounded write fixture for scenario comparison.", + "workload": "mixed", + "k6_env": { + "USER_COUNT": "4", + "SEED_TASK_COUNT": "12", + "SEED_EVENT_COUNT": "8", + "VUS": "5", + "DURATION": "2m", + "THINK_TIME_SECONDS": "1" + } + }, + "comparison-stress": { + "description": "Ramping-VU fixture for local stress comparison.", + "workload": "stress", + "k6_env": { + "USER_COUNT": "4", + "SEED_TASK_COUNT": "12", + "SEED_EVENT_COUNT": "8", + "STRESS_MAX_VUS": "20", + "STRESS_STAGE_DURATION": "30s", + "THINK_TIME_SECONDS": "1" + } + } + } +} diff --git a/backend/lined/load-tests/runtime-scenarios/fixture-profiles.mjs b/backend/lined/load-tests/runtime-scenarios/fixture-profiles.mjs new file mode 100644 index 0000000..428ba93 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/fixture-profiles.mjs @@ -0,0 +1,131 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +export const FIXTURE_PROFILES_PATH = 'load-tests/runtime-scenarios/fixture-profiles-v1.json'; +export const K6_ENV_KEYS = new Set([ + 'RUN_ID', + 'USER_COUNT', + 'SEED_TASK_COUNT', + 'SEED_EVENT_COUNT', + 'VUS', + 'DURATION', + 'STRESS_MAX_VUS', + 'STRESS_STAGE_DURATION', + 'THINK_TIME_SECONDS', +]); + +const PROFILE_KEYS = new Set(['description', 'workload', 'k6_env']); + +export const fixtureProfileNames = ({ cwd = process.cwd(), file = FIXTURE_PROFILES_PATH } = {}) => { + const artifact = readFixtureArtifact(cwd, file); + return Object.keys(artifact.profiles).sort(); +}; + +export const loadFixtureProfile = ( + name, + { allowedWorkloads, cwd = process.cwd(), file = FIXTURE_PROFILES_PATH } = {} +) => { + const artifact = readFixtureArtifact(cwd, file); + const profile = artifact.profiles[name]; + if (!profile) { + throw new Error(`--fixture-profile must be one of: ${Object.keys(artifact.profiles).sort().join(', ')}`); + } + validateProfile(name, profile, allowedWorkloads); + return { + description: profile.description, + k6Env: { ...profile.k6_env }, + name, + schemaVersion: artifact.schema_version, + workload: profile.workload, + }; +}; + +export const applyFixtureProfileDefaults = ( + options, + { + cwd = process.cwd(), + explicitK6Env = {}, + fixtureFile = FIXTURE_PROFILES_PATH, + allowedWorkloads, + workloadExplicit = false, + } = {} +) => { + if (!options.fixtureProfile) { + return options; + } + + const profile = loadFixtureProfile(options.fixtureProfile, { + allowedWorkloads, + cwd, + file: fixtureFile, + }); + const merged = { + ...options, + fixtureProfileData: profile, + k6Env: { + ...profile.k6Env, + ...explicitK6Env, + }, + }; + if (!workloadExplicit) { + merged.workload = profile.workload; + } + return merged; +}; + +const readFixtureArtifact = (cwd, file) => { + const artifactPath = path.resolve(cwd, file); + const parsed = JSON.parse(fs.readFileSync(artifactPath, 'utf-8')); + if (!isRecord(parsed) || parsed.schema_version !== 1 || !isRecord(parsed.profiles)) { + throw new Error('fixture profile artifact must contain schema_version 1 and profiles'); + } + return parsed; +}; + +const validateProfile = (name, profile, allowedWorkloads) => { + requireRecord(`fixture profile ${name}`, profile); + requireKnownProfileKeys(name, profile); + requireWorkload(name, profile.workload, allowedWorkloads); + requireK6Env(name, profile.k6_env); +}; + +const requireRecord = (label, value) => { + if (!isRecord(value)) { + throw new Error(`${label} must be an object`); + } +}; + +const requireKnownProfileKeys = (name, profile) => { + const unknownKeys = Object.keys(profile).filter((key) => !PROFILE_KEYS.has(key)); + if (unknownKeys.length > 0) { + throw new Error(`fixture profile ${name} has unsupported keys: ${unknownKeys.join(', ')}`); + } +}; + +const requireWorkload = (name, workload, allowedWorkloads) => { + if (typeof workload !== 'string' || workload.length === 0) { + throw new Error(`fixture profile ${name} must define workload`); + } + if (allowedWorkloads && !allowedWorkloads.has(workload)) { + throw new Error( + `fixture profile ${name} has unsupported workload ${workload}; ` + + `allowed: ${Array.from(allowedWorkloads).join(', ')}` + ); + } +}; + +const requireK6Env = (name, k6Env) => { + requireRecord(`fixture profile ${name} k6_env`, k6Env); + Object.entries(k6Env).forEach(([key, value]) => requireK6EnvEntry(name, key, value)); +}; + +const requireK6EnvEntry = (name, key, value) => { + if (!K6_ENV_KEYS.has(key)) { + throw new Error(`fixture profile ${name} has unsupported k6 env ${key}`); + } + if (typeof value !== 'string') { + throw new Error(`fixture profile ${name} k6 env ${key} must be a string`); + } +}; + +const isRecord = (value) => typeof value === 'object' && value !== null && !Array.isArray(value); diff --git a/backend/lined/load-tests/runtime-scenarios/k6-adapter.mjs b/backend/lined/load-tests/runtime-scenarios/k6-adapter.mjs new file mode 100644 index 0000000..7522c87 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/k6-adapter.mjs @@ -0,0 +1,85 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +import { runCommand } from './command-runner.mjs'; +import { parseK6Summary } from './runtime-summary.mjs'; + +export const SUMMARY_TREND_STATS = 'p(95),p(99),avg,min,max'; +export const K6_PREFLIGHT_TIMEOUT_MS = 30_000; +export const K6_RUN_TIMEOUT_MS = 600_000; + +export const assertK6Available = ( + k6Bin, + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + const result = commandRunner(k6Bin, ['version'], { + allowFailure: true, + capture: true, + cwd, + timeoutMs: K6_PREFLIGHT_TIMEOUT_MS, + }); + + if (result.error?.code === 'ENOENT') { + throw new Error( + `k6 executable not found: ${k6Bin}. ` + + 'Install k6 and make it available in PATH, or pass --k6-bin /absolute/path/to/k6. ' + + 'On macOS with Homebrew: brew install k6.' + ); + } + if (result.error) { + throw result.error; + } + if (result.signal) { + throw new Error(`k6 preflight was killed by signal ${result.signal}`); + } + if (result.status !== 0) { + throw new Error(`k6 preflight failed with exit code ${result.status}`); + } +}; + +export const runK6 = ( + options, + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-k6-summary-')); + const summaryPath = path.join(tempDir, 'summary.json'); + const args = [ + 'run', + '--summary-export', + summaryPath, + '--summary-trend-stats', + SUMMARY_TREND_STATS, + '-e', + `WORKLOAD=${options.workload}`, + '-e', + `BASE_URL=${options.baseUrl}`, + ]; + + for (const [key, value] of Object.entries(options.k6Env)) { + args.push('-e', `${key}=${value}`); + } + if (options.allowRemoteBaseUrl) { + args.push('-e', 'ALLOW_REMOTE_BASE_URL=true'); + } + args.push(options.script); + + const result = commandRunner(options.k6Bin, args, { + allowFailure: true, + cwd, + timeoutMs: K6_RUN_TIMEOUT_MS, + }); + + try { + return { + args, + exitCode: result.signal ? null : result.status, + signal: result.signal ?? undefined, + summary: fs.existsSync(summaryPath) + ? parseK6Summary(fs.readFileSync(summaryPath, 'utf-8')) + : undefined, + }; + } finally { + fs.rmSync(tempDir, { force: true, recursive: true }); + } +}; diff --git a/backend/lined/load-tests/runtime-scenarios/kubernetes-adapter.mjs b/backend/lined/load-tests/runtime-scenarios/kubernetes-adapter.mjs new file mode 100644 index 0000000..0da16a2 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/kubernetes-adapter.mjs @@ -0,0 +1,287 @@ +import { runCommand } from './command-runner.mjs'; + +export const NAMESPACE = 'lined'; +export const BACKEND_DEPLOYMENT = 'lined-backend'; +export const BACKEND_LABEL = 'app.kubernetes.io/name=lined-backend'; +export const KUBECTL_TIMEOUT_MS = 60_000; +export const KUBECTL_TOP_TIMEOUT_MS = 30_000; +export const KUBECTL_ROLLOUT_TIMEOUT_MS = 600_000; + +export const cleanupHpaIfNeeded = ( + options, + scenario, + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + if (!scenario.fixedReplicas || options.skipHpaCleanup) { + return false; + } + + commandRunner('kubectl', [ + '-n', + NAMESPACE, + 'delete', + 'hpa', + BACKEND_DEPLOYMENT, + '--ignore-not-found', + ], { cwd, timeoutMs: KUBECTL_TIMEOUT_MS }); + + return true; +}; + +export const applyScenarioIfNeeded = ( + options, + scenario, + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + if (!options.apply) { + return false; + } + + commandRunner('kubectl', ['apply', '-k', scenario.path], { + cwd, + timeoutMs: KUBECTL_TIMEOUT_MS, + }); + return true; +}; + +export const waitForRollout = ( + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + commandRunner('kubectl', [ + '-n', + NAMESPACE, + 'rollout', + 'status', + `deployment/${BACKEND_DEPLOYMENT}`, + ], { cwd, timeoutMs: KUBECTL_ROLLOUT_TIMEOUT_MS }); +}; + +export const collectKubernetesState = ( + scenarioName, + { commandRunner = runCommand, cwd = process.cwd() } = {} +) => { + const deployment = readKubectlJson(commandRunner, [ + '-n', + NAMESPACE, + 'get', + 'deployment', + BACKEND_DEPLOYMENT, + '-o', + 'json', + ], cwd, KUBECTL_TIMEOUT_MS); + const pods = readKubectlJson(commandRunner, [ + '-n', + NAMESPACE, + 'get', + 'pods', + '-l', + BACKEND_LABEL, + '-o', + 'json', + ], cwd, KUBECTL_TIMEOUT_MS); + const hpa = readOptionalKubectlJson(commandRunner, [ + '-n', + NAMESPACE, + 'get', + 'hpa', + BACKEND_DEPLOYMENT, + '-o', + 'json', + ], cwd, KUBECTL_TIMEOUT_MS); + const top = readOptionalKubectlText(commandRunner, [ + '-n', + NAMESPACE, + 'top', + 'pods', + '-l', + BACKEND_LABEL, + '--no-headers', + ], cwd, KUBECTL_TOP_TIMEOUT_MS); + + return { + ...summarizeKubernetesState({ + deployment, + hpa, + pods, + topOutput: top, + }), + scenario: scenarioName, + }; +}; + +export const summarizeKubernetesState = ({ deployment, hpa, pods, topOutput }) => { + const backendContainer = findBackendContainer(deployment); + const cpuRequest = parseCpuQuantity(backendContainer?.resources?.requests?.cpu); + const memoryLimit = parseMemoryQuantity(backendContainer?.resources?.limits?.memory); + const backendPods = Array.isArray(pods?.items) ? pods.items : []; + const podCount = Math.max(backendPods.length, 1); + const usage = parseTopPods(topOutput); + + const cpuUsage = sumPodUsage(usage, 'cpuMillicores'); + const memoryUsage = sumPodUsage(usage, 'memoryBytes'); + + return { + cpuUtilization: ratioOrUndefined( + cpuUsage, + cpuRequest === undefined ? undefined : cpuRequest * podCount + ), + hpa: summarizeHpa(hpa), + memoryUtilization: ratioOrUndefined( + memoryUsage, + memoryLimit === undefined ? undefined : memoryLimit * podCount + ), + metricsServerAvailable: usage.length > 0, + replicas: deployment?.status?.replicas, + restartCount: sumRestartCount(backendPods), + }; +}; + +export const parseCpuQuantity = (value) => { + if (value === undefined) { + return undefined; + } + const raw = String(value).trim(); + if (/^\d+(\.\d+)?m$/.test(raw)) { + return Number.parseFloat(raw.slice(0, -1)); + } + if (/^\d+(\.\d+)?$/.test(raw)) { + return Number.parseFloat(raw) * 1000; + } + if (/^\d+(\.\d+)?u$/.test(raw)) { + return Number.parseFloat(raw.slice(0, -1)) / 1000; + } + if (/^\d+(\.\d+)?n$/.test(raw)) { + return Number.parseFloat(raw.slice(0, -1)) / 1000000; + } + throw new Error(`Unsupported CPU quantity: ${value}`); +}; + +export const parseMemoryQuantity = (value) => { + if (value === undefined) { + return undefined; + } + const raw = String(value).trim(); + const match = /^(\d+(?:\.\d+)?)([A-Za-z]+)?$/.exec(raw); + if (!match) { + throw new Error(`Unsupported memory quantity: ${value}`); + } + + const amount = Number.parseFloat(match[1]); + const suffix = match[2] ?? ''; + const multipliers = { + '': 1, + K: 1000, + Ki: 1024, + M: 1000 ** 2, + Mi: 1024 ** 2, + G: 1000 ** 3, + Gi: 1024 ** 3, + T: 1000 ** 4, + Ti: 1024 ** 4, + }; + const multiplier = multipliers[suffix]; + if (multiplier === undefined) { + throw new Error(`Unsupported memory quantity: ${value}`); + } + + return amount * multiplier; +}; + +export const parseTopPods = (content = '') => content + .split(/\r?\n/) + .map((line) => line.trim()) + .filter(Boolean) + .map((line) => { + const parts = line.split(/\s+/); + if (parts.length < 3) { + throw new Error(`Malformed kubectl top pods line: ${line}`); + } + const [name, cpu, memory] = parts; + return { + cpuMillicores: parseCpuQuantity(cpu), + memoryBytes: parseMemoryQuantity(memory), + name, + }; + }); + +const readKubectlJson = (commandRunner, args, cwd, timeoutMs) => JSON.parse(commandRunner( + 'kubectl', + args, + { + capture: true, + cwd, + timeoutMs, + } +).stdout); + +const readOptionalKubectlJson = (commandRunner, args, cwd, timeoutMs) => { + const result = commandRunner('kubectl', args, { + allowFailure: true, + capture: true, + cwd, + timeoutMs, + }); + const output = String(result.stdout ?? ''); + if (result.error || result.status !== 0 || output.trim() === '') { + return undefined; + } + return JSON.parse(output); +}; + +const readOptionalKubectlText = (commandRunner, args, cwd, timeoutMs) => { + const result = commandRunner('kubectl', args, { + allowFailure: true, + capture: true, + cwd, + timeoutMs, + }); + return result.error || result.status !== 0 ? '' : String(result.stdout ?? ''); +}; + +const findBackendContainer = (deployment) => { + const containers = deployment?.spec?.template?.spec?.containers; + if (!Array.isArray(containers)) { + return undefined; + } + return containers.find((container) => container.name === 'backend') ?? containers[0]; +}; + +const summarizeHpa = (hpa) => { + if (!hpa) { + return undefined; + } + const currentReplicas = hpa.status?.currentReplicas; + const desiredReplicas = hpa.status?.desiredReplicas; + if (typeof currentReplicas !== 'number' || typeof desiredReplicas !== 'number') { + return undefined; + } + return { + currentReplicas, + desiredReplicas, + }; +}; + +const sumRestartCount = (pods) => pods + .flatMap((pod) => Array.isArray(pod.status?.containerStatuses) + ? pod.status.containerStatuses + : []) + .filter((status) => status.name === 'backend' || pods.length === 1) + .reduce((total, status) => total + (status.restartCount ?? 0), 0); + +const sumPodUsage = (usage, field) => { + if (usage.length === 0) { + return undefined; + } + return usage.reduce((total, pod) => total + pod[field], 0); +}; + +const ratioOrUndefined = (numerator, denominator) => { + if ( + !Number.isFinite(numerator) + || !Number.isFinite(denominator) + || denominator <= 0 + ) { + return undefined; + } + return Number((numerator / denominator).toFixed(6)); +}; diff --git a/backend/lined/load-tests/runtime-scenarios/runtime-summary.mjs b/backend/lined/load-tests/runtime-scenarios/runtime-summary.mjs new file mode 100644 index 0000000..271d95a --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/runtime-summary.mjs @@ -0,0 +1,151 @@ +export const CLI_VERSION = 1; +export const SOURCE = 'local-kind'; + +export const parseK6Summary = (content) => { + const parsed = JSON.parse(content); + if (!isRecord(parsed) || !isRecord(parsed.metrics)) { + throw new Error('k6 summary export must contain metrics'); + } + return parsed; +}; + +export const buildRuntimeSummary = ({ k6Summary, kubernetes, scenario, workload }) => { + const missing = new Set(['availability']); + const summary = { + latency_p95_ms: requiredMetric(k6Summary, 'http_req_duration', 'p(95)'), + latency_p99_ms: requiredMetric(k6Summary, 'http_req_duration', 'p(99)'), + error_rate: requiredMetric(k6Summary, 'http_req_failed', 'rate', ['value']), + throughput_rps: requiredMetric(k6Summary, 'http_reqs', 'rate'), + restart_count: kubernetes.restartCount, + }; + + if (kubernetes.cpuUtilization === undefined) { + missing.add('cpu_utilization'); + } else { + summary.cpu_utilization = kubernetes.cpuUtilization; + } + + if (kubernetes.memoryUtilization === undefined) { + missing.add('memory_utilization'); + } else { + summary.memory_utilization = kubernetes.memoryUtilization; + } + + if (kubernetes.hpa) { + summary.hpa_current_replicas = kubernetes.hpa.currentReplicas; + summary.hpa_desired_replicas = kubernetes.hpa.desiredReplicas; + } else if (scenario === 'hpa-cpu') { + missing.add('hpa_current_replicas'); + missing.add('hpa_desired_replicas'); + } + + return { + schema_version: 1, + scenario, + workload, + source: SOURCE, + summary, + missing: Array.from(missing).sort(), + }; +}; + +export const buildWindowKubernetesState = (before, after) => ({ + ...after, + restartCount: restartDelta(before?.restartCount, after?.restartCount), + restartCountAfter: after?.restartCount, + restartCountBefore: before?.restartCount, +}); + +export const buildManifest = ({ + appliedScenario, + finishedAt, + git, + hpaCleanup, + k6, + kubernetes, + options, + scenario, + startedAt, + summaryExported, + summaryWritten, +}) => ({ + schema_version: 1, + artifact: 'runtime-scenario-summary', + cli_version: CLI_VERSION, + source: SOURCE, + scenario: options.scenario, + scenario_path: scenario.path, + workload: options.workload, + fixture_profile: options.fixtureProfileData ? { + name: options.fixtureProfileData.name, + schema_version: options.fixtureProfileData.schemaVersion, + workload: options.fixtureProfileData.workload, + k6_env: options.fixtureProfileData.k6Env, + } : undefined, + workload_env: { + BASE_URL: options.baseUrl, + WORKLOAD: options.workload, + ...options.k6Env, + }, + started_at: startedAt, + finished_at: finishedAt, + git, + k6: { + exit_code: k6.exitCode, + executable: options.k6Bin, + script: options.script, + signal: k6.signal, + summary_exported: summaryExported, + summary_trend_stats: k6.summaryTrendStats, + }, + collector_summary_written: summaryWritten, + kubernetes: { + applied_scenario: appliedScenario, + deployment: 'lined-backend', + hpa_cleanup: hpaCleanup, + metrics_server_available: kubernetes.metricsServerAvailable, + namespace: 'lined', + replicas: kubernetes.replicas, + restart_count_after: kubernetes.restartCountAfter, + restart_count_before: kubernetes.restartCountBefore, + restart_count_delta: kubernetes.restartCount, + }, +}); + +const requiredMetric = (k6Summary, metric, valueName, fallbackValueNames = []) => { + const metricValues = k6Summary.metrics?.[metric]; + const value = readK6MetricValue(metricValues, [valueName, ...fallbackValueNames]); + if (typeof value !== 'number' || !Number.isFinite(value)) { + throw new Error(`k6 summary missing numeric ${metric}.${valueName}`); + } + return value; +}; + +const readK6MetricValue = (metricValues, valueNames) => { + if (!metricValues) { + return undefined; + } + for (const valueName of valueNames) { + const nested = metricValues.values?.[valueName]; + if (nested !== undefined) { + return nested; + } + const flat = metricValues[valueName]; + if (flat !== undefined) { + return flat; + } + } + return undefined; +}; + +const restartDelta = (before, after) => { + if (typeof after !== 'number' || !Number.isFinite(after)) { + return undefined; + } + if (typeof before !== 'number' || !Number.isFinite(before)) { + return after; + } + return Math.max(0, after - before); +}; + +const isRecord = (value) => typeof value === 'object' && value !== null && !Array.isArray(value); diff --git a/backend/lined/load-tests/runtime-scenarios/scenario-runner-cli.mjs b/backend/lined/load-tests/runtime-scenarios/scenario-runner-cli.mjs new file mode 100644 index 0000000..ec49d18 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/scenario-runner-cli.mjs @@ -0,0 +1,27 @@ +#!/usr/bin/env node +import { + ScenarioRunError, + parseArgs, + printHelp, + runScenario, +} from './scenario-runner.mjs'; + +try { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + console.log(printHelp()); + process.exit(0); + } + + const result = runScenario(options); + console.log(`Wrote collector summary: ${result.summaryPath}`); + console.log(`Wrote summary manifest: ${result.manifestPath}`); +} catch (error) { + if (error instanceof ScenarioRunError && error.result?.manifestPath) { + console.error(error.message); + console.error(`Wrote summary manifest: ${error.result.manifestPath}`); + process.exit(1); + } + console.error(error instanceof Error ? error.message : error); + process.exit(1); +} diff --git a/backend/lined/load-tests/runtime-scenarios/scenario-runner.mjs b/backend/lined/load-tests/runtime-scenarios/scenario-runner.mjs new file mode 100644 index 0000000..3963db4 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/scenario-runner.mjs @@ -0,0 +1,373 @@ +import fs from 'node:fs'; +import path from 'node:path'; + +import { + applyScenarioIfNeeded, + cleanupHpaIfNeeded, + collectKubernetesState, + waitForRollout, +} from './kubernetes-adapter.mjs'; +import { assertK6Available, runK6, SUMMARY_TREND_STATS } from './k6-adapter.mjs'; +import { + buildManifest, + buildRuntimeSummary, + buildWindowKubernetesState, +} from './runtime-summary.mjs'; +import { runCommand } from './command-runner.mjs'; +import { + FIXTURE_PROFILES_PATH, + K6_ENV_KEYS, + applyFixtureProfileDefaults, + fixtureProfileNames, +} from './fixture-profiles.mjs'; + +export const SCENARIOS = Object.freeze({ + 'fixed-small': { + fixedReplicas: true, + path: 'k8s/kind/scenarios/fixed-small', + }, + 'fixed-medium': { + fixedReplicas: true, + path: 'k8s/kind/scenarios/fixed-medium', + }, + 'replicas-2': { + fixedReplicas: true, + path: 'k8s/kind/scenarios/replicas-2', + }, + 'hpa-cpu': { + fixedReplicas: false, + path: 'k8s/kind/scenarios/hpa-cpu', + }, +}); + +export const WORKLOADS = Object.freeze([ + 'smoke', + 'baseline', + 'read-heavy', + 'write-heavy', + 'mixed', + 'stress', + 'negative-smoke', +]); + +const DEFAULT_BASE_URL = 'http://localhost:8080'; +const DEFAULT_K6_BIN = 'k6'; +const DEFAULT_OUTPUT_ROOT = 'load-tests/runtime-scenarios/output'; +const DEFAULT_SCRIPT = 'load-tests/k6/load-test-baseline.js'; +const DEFAULT_WORKLOAD = 'baseline'; +const HELP_OPTIONS = new Set(['--help', '-h']); + +const OPTION_HANDLERS = Object.freeze({ + '--scenario': readOptionInto('scenario'), + '--workload': (state, option) => { + state.options.workload = readNextOptionValue(state, option); + state.workloadExplicit = true; + }, + '--base-url': readOptionInto('baseUrl'), + '--fixture-profile': readOptionInto('fixtureProfile'), + '--fixture-profile-file': readOptionInto('fixtureProfileFile'), + '--output-root': readOptionInto('outputRoot'), + '--script': readOptionInto('script'), + '--k6-bin': readOptionInto('k6Bin'), + '--k6-env': (state, option) => addK6Env( + state.explicitK6Env, + readNextOptionValue(state, option) + ), + '--skip-apply': (state) => { + state.options.apply = false; + }, + '--skip-hpa-cleanup': (state) => { + state.options.skipHpaCleanup = true; + }, + '--allow-remote-base-url': (state) => { + state.options.allowRemoteBaseUrl = true; + }, +}); + +export class ScenarioRunError extends Error { + constructor(message, result) { + super(message); + this.name = 'ScenarioRunError'; + this.result = result; + } +} + +export const defaultOptions = () => ({ + allowRemoteBaseUrl: false, + apply: true, + baseUrl: DEFAULT_BASE_URL, + fixtureProfile: undefined, + fixtureProfileFile: FIXTURE_PROFILES_PATH, + k6Bin: DEFAULT_K6_BIN, + k6Env: {}, + outputRoot: DEFAULT_OUTPUT_ROOT, + script: DEFAULT_SCRIPT, + skipHpaCleanup: false, + workload: DEFAULT_WORKLOAD, +}); + +export const parseArgs = (argv) => { + const state = { + argv, + explicitK6Env: {}, + index: 0, + options: defaultOptions(), + workloadExplicit: false, + }; + + for (; state.index < argv.length; state.index += 1) { + const arg = argv[state.index]; + if (HELP_OPTIONS.has(arg)) { + return { ...state.options, help: true }; + } + const handler = OPTION_HANDLERS[arg]; + if (!handler) { + throw new Error(`Unknown option: ${arg}`); + } + handler(state, arg); + } + + state.options.k6Env = { ...state.explicitK6Env }; + const resolvedOptions = applyFixtureProfileDefaults(state.options, { + allowedWorkloads: new Set(WORKLOADS), + explicitK6Env: state.explicitK6Env, + fixtureFile: state.options.fixtureProfileFile, + workloadExplicit: state.workloadExplicit, + }); + validateOptions(resolvedOptions); + return resolvedOptions; +}; + +export const printHelp = () => `Usage: + node load-tests/runtime-scenarios/scenario-runner-cli.mjs --scenario [options] + +Options: + --scenario ${Object.keys(SCENARIOS).join(', ')} + --workload ${WORKLOADS.join(', ')} (default: ${DEFAULT_WORKLOAD}) + --fixture-profile ${fixtureProfileNames().join(', ')} + --fixture-profile-file Fixture profile artifact (default: ${FIXTURE_PROFILES_PATH}) + --base-url Backend URL (default: ${DEFAULT_BASE_URL}) + --output-root Output root (default: ${DEFAULT_OUTPUT_ROOT}) + --script k6 script path (default: ${DEFAULT_SCRIPT}) + --k6-bin k6 executable (default: ${DEFAULT_K6_BIN}) + --k6-env KEY=value Extra k6 env; repeatable for ${Array.from(K6_ENV_KEYS).join(', ')} + --skip-apply Do not apply the selected kustomize scenario + --skip-hpa-cleanup Do not delete HPA before fixed-replica scenarios + --allow-remote-base-url Allow non-local BASE_URL and pass ALLOW_REMOTE_BASE_URL=true to k6 +`; + +export const runScenario = ( + options, + { + clock = () => new Date().toISOString(), + commandRunner = runCommand, + cwd = process.cwd(), + kubernetesAdapter = defaultKubernetesAdapter(commandRunner, cwd), + k6Adapter = defaultK6Adapter(commandRunner, cwd), + gitReader = defaultGitReader(commandRunner, cwd), + } = {} +) => { + const runOptions = resolveFixtureOptions(options); + validateOptions(runOptions); + const scenario = SCENARIOS[runOptions.scenario]; + k6Adapter.assertAvailable(runOptions.k6Bin); + + const startedAt = clock(); + const outputDir = outputDirectory(runOptions, startedAt, cwd); + const hpaCleanup = kubernetesAdapter.cleanupHpaIfNeeded(runOptions, scenario); + const appliedScenario = kubernetesAdapter.applyScenarioIfNeeded(runOptions, scenario); + kubernetesAdapter.waitForRollout(); + + const beforeWorkload = kubernetesAdapter.collectState(runOptions.scenario); + const k6Result = k6Adapter.run(runOptions); + const afterWorkload = kubernetesAdapter.collectState(runOptions.scenario); + const kubernetes = buildWindowKubernetesState(beforeWorkload, afterWorkload); + const finishedAt = clock(); + const manifest = buildManifest({ + appliedScenario, + finishedAt, + git: gitReader(), + hpaCleanup, + k6: { + exitCode: k6Result.exitCode, + signal: k6Result.signal, + summaryTrendStats: SUMMARY_TREND_STATS, + }, + kubernetes, + options: runOptions, + scenario, + startedAt, + summaryExported: k6Result.summary !== undefined, + summaryWritten: k6Result.exitCode === 0 && k6Result.summary !== undefined, + }); + + const manifestPath = path.join(outputDir, 'runtime-summary-manifest.json'); + fs.mkdirSync(outputDir, { recursive: true }); + writeJson(manifestPath, manifest); + + if (k6Result.signal) { + throw new ScenarioRunError( + `k6 was killed by signal ${k6Result.signal}; ` + + `wrote manifest ${manifestPath} but did not write collector summary`, + { + manifest, + manifestPath, + } + ); + } + + if (k6Result.exitCode !== 0) { + throw new ScenarioRunError( + `k6 failed with exit code ${k6Result.exitCode}; ` + + `wrote manifest ${manifestPath} but did not write collector summary`, + { + manifest, + manifestPath, + } + ); + } + + if (k6Result.summary === undefined) { + throw new Error('k6 completed without a summary export; collector summary was not written'); + } + + const summary = buildRuntimeSummary({ + k6Summary: k6Result.summary, + kubernetes, + scenario: runOptions.scenario, + workload: runOptions.workload, + }); + const summaryPath = path.join(outputDir, 'runtime-summary.json'); + writeJson(summaryPath, summary); + + return { + manifest, + manifestPath, + summary, + summaryPath, + }; +}; + +const resolveFixtureOptions = (options) => { + if (!options.fixtureProfile || options.fixtureProfileData) { + return options; + } + return applyFixtureProfileDefaults( + { + ...options, + fixtureProfileFile: options.fixtureProfileFile ?? FIXTURE_PROFILES_PATH, + k6Env: options.k6Env ?? {}, + }, + { + allowedWorkloads: new Set(WORKLOADS), + explicitK6Env: options.k6Env ?? {}, + fixtureFile: options.fixtureProfileFile ?? FIXTURE_PROFILES_PATH, + workloadExplicit: isProgrammaticWorkloadExplicit(options), + } + ); +}; + +const isProgrammaticWorkloadExplicit = (options) => options.workloadExplicit === true + || (options.workload !== undefined && options.workload !== DEFAULT_WORKLOAD); + +export const ensureLocalBaseUrl = (baseUrl, allowRemoteBaseUrl) => { + if (allowRemoteBaseUrl) { + return; + } + const local = /^https?:\/\/(localhost|127\.0\.0\.1|\[::1\])(?::|\/|$)/.test(baseUrl); + if (!local) { + throw new Error( + 'BASE_URL must point to localhost, 127.0.0.1, or [::1]. ' + + 'Use --allow-remote-base-url only for an intentional controlled target.' + ); + } +}; + +const defaultKubernetesAdapter = (commandRunner, cwd) => ({ + applyScenarioIfNeeded: (options, scenario) => applyScenarioIfNeeded( + options, + scenario, + { commandRunner, cwd } + ), + cleanupHpaIfNeeded: (options, scenario) => cleanupHpaIfNeeded( + options, + scenario, + { commandRunner, cwd } + ), + collectState: (scenarioName) => collectKubernetesState( + scenarioName, + { commandRunner, cwd } + ), + waitForRollout: () => waitForRollout({ commandRunner, cwd }), +}); + +const defaultK6Adapter = (commandRunner, cwd) => ({ + assertAvailable: (k6Bin) => assertK6Available(k6Bin, { commandRunner, cwd }), + run: (options) => runK6(options, { commandRunner, cwd }), +}); + +const defaultGitReader = (commandRunner, cwd) => () => ({ + branch: readOptionalGit(commandRunner, ['branch', '--show-current'], cwd), + commit: readOptionalGit(commandRunner, ['rev-parse', 'HEAD'], cwd), +}); + +const readOptionValue = (argv, index, option) => { + const value = argv[index]; + if (value === undefined || value.startsWith('--')) { + throw new Error(`${option} requires a value`); + } + return value; +}; + +function readOptionInto(property) { + return (state, option) => { + state.options[property] = readNextOptionValue(state, option); + }; +} + +function readNextOptionValue(state, option) { + state.index += 1; + return readOptionValue(state.argv, state.index, option); +} + +const addK6Env = (k6Env, assignment) => { + const separator = assignment.indexOf('='); + if (separator < 1) { + throw new Error('--k6-env requires KEY=value'); + } + const key = assignment.slice(0, separator); + const value = assignment.slice(separator + 1); + if (!K6_ENV_KEYS.has(key)) { + throw new Error(`Unsupported k6 env ${key}; allowed: ${Array.from(K6_ENV_KEYS).join(', ')}`); + } + k6Env[key] = value; +}; + +const validateOptions = (options) => { + if (!options.scenario || SCENARIOS[options.scenario] === undefined) { + throw new Error(`--scenario must be one of: ${Object.keys(SCENARIOS).join(', ')}`); + } + if (!WORKLOADS.includes(options.workload)) { + throw new Error(`--workload must be one of: ${WORKLOADS.join(', ')}`); + } + ensureLocalBaseUrl(options.baseUrl, options.allowRemoteBaseUrl); +}; + +const outputDirectory = (options, startedAt, cwd) => { + const safeTimestamp = startedAt.replaceAll(/[:.]/g, '-'); + const outputName = `${options.scenario}-${options.workload}-${safeTimestamp}`; + return path.resolve(cwd, options.outputRoot, outputName); +}; + +const readOptionalGit = (commandRunner, args, cwd) => { + const result = commandRunner('git', args, { + allowFailure: true, + capture: true, + cwd, + }); + return result.status === 0 ? result.stdout.trim() : undefined; +}; + +const writeJson = (file, value) => { + fs.writeFileSync(file, `${JSON.stringify(value, null, 2)}\n`, 'utf-8'); +}; diff --git a/backend/lined/load-tests/runtime-scenarios/scenario-runner.test.mjs b/backend/lined/load-tests/runtime-scenarios/scenario-runner.test.mjs new file mode 100644 index 0000000..04372f0 --- /dev/null +++ b/backend/lined/load-tests/runtime-scenarios/scenario-runner.test.mjs @@ -0,0 +1,988 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { describe, it } from 'node:test'; + +import { runCommand } from './command-runner.mjs'; +import { assertK6Available, runK6 } from './k6-adapter.mjs'; +import { + cleanupHpaIfNeeded, + parseCpuQuantity, + parseMemoryQuantity, + parseTopPods, + summarizeKubernetesState, +} from './kubernetes-adapter.mjs'; +import { + ScenarioRunError, + ensureLocalBaseUrl, + parseArgs, + runScenario, +} from './scenario-runner.mjs'; +import { buildManifest, buildRuntimeSummary } from './runtime-summary.mjs'; +import { loadFixtureProfile } from './fixture-profiles.mjs'; + +const nestedK6Summary = { + metrics: { + http_req_duration: { + values: { + 'p(95)': 250.5, + 'p(99)': 550.25, + }, + }, + http_req_failed: { + values: { + rate: 0.002, + }, + }, + http_reqs: { + values: { + rate: 42.1, + }, + }, + }, +}; + +const flatK6Summary = { + metrics: { + http_req_duration: { + 'p(95)': 150.25, + 'p(99)': 275.5, + }, + http_req_failed: { + fails: 0, + passes: 100, + value: 0, + }, + http_reqs: { + count: 100, + rate: 25.5, + }, + }, +}; + +const deployment = { + spec: { + template: { + spec: { + containers: [{ + name: 'backend', + resources: { + limits: { + memory: '1Gi', + }, + requests: { + cpu: '500m', + }, + }, + }], + }, + }, + }, + status: { + replicas: 2, + }, +}; + +const pods = { + items: [{ + status: { + containerStatuses: [{ + name: 'backend', + restartCount: 1, + }], + }, + }, { + status: { + containerStatuses: [{ + name: 'backend', + restartCount: 2, + }], + }, + }], +}; + +const TEXTS = Object.freeze({ + env: { + stressThinkTime: 'THINK_TIME_SECONDS', + token: 'TOKEN', + userCount: 'USER_COUNT', + vus: 'VUS', + }, + fixture: { + baseline: 'comparison-baseline', + readHeavy: 'comparison-read-heavy', + unknown: 'unknown', + unsafe: 'unsafe', + }, + scenario: { + fixedMedium: 'fixed-medium', + hpaCpu: 'hpa-cpu', + unknown: 'unknown', + }, + workload: { + baseline: 'baseline', + typo: 'basline', + readHeavy: 'read-heavy', + smoke: 'smoke', + unknown: 'unknown', + }, +}); + +const VALUES = Object.freeze({ + events: { + baselineSeedCount: '8', + }, + tasks: { + baselineSeedCount: '12', + }, + thinkTime: { + none: '0', + }, + users: { + baselineCount: '4', + }, + vus: { + baseline: '5', + override: '2', + }, +}); + +describe('parseArgs', () => { + it('accepts valid scenario, workload, and allowlisted k6 env options', (t) => { + t.plan(3); + const options = parseArgs([ + '--scenario', + TEXTS.scenario.fixedMedium, + '--workload', + TEXTS.workload.smoke, + '--k6-env', + `${TEXTS.env.vus}=${VALUES.vus.override}`, + ]); + + t.assert.equal(options.scenario, TEXTS.scenario.fixedMedium); + t.assert.equal(options.workload, TEXTS.workload.smoke); + t.assert.equal(options.k6Env.VUS, VALUES.vus.override); + }); + + it('applies a fixture profile as workload and k6 env defaults', (t) => { + t.plan(5); + const options = parseArgs([ + '--scenario', + TEXTS.scenario.fixedMedium, + '--fixture-profile', + TEXTS.fixture.baseline, + ]); + + t.assert.equal(options.fixtureProfileData.name, TEXTS.fixture.baseline); + t.assert.equal(options.workload, TEXTS.workload.baseline); + t.assert.equal(options.k6Env.USER_COUNT, VALUES.users.baselineCount); + t.assert.equal(options.k6Env.SEED_TASK_COUNT, VALUES.tasks.baselineSeedCount); + t.assert.equal(options.k6Env.VUS, VALUES.vus.baseline); + }); + + it('lets explicit workload and k6 env override fixture defaults', (t) => { + t.plan(4); + const options = parseArgs([ + '--scenario', + TEXTS.scenario.fixedMedium, + '--fixture-profile', + TEXTS.fixture.baseline, + '--workload', + TEXTS.workload.readHeavy, + '--k6-env', + `${TEXTS.env.vus}=${VALUES.vus.override}`, + '--k6-env', + `${TEXTS.env.stressThinkTime}=${VALUES.thinkTime.none}`, + ]); + + t.assert.equal(options.workload, TEXTS.workload.readHeavy); + t.assert.equal(options.k6Env.USER_COUNT, VALUES.users.baselineCount); + t.assert.equal(options.k6Env.VUS, VALUES.vus.override); + t.assert.equal(options.k6Env.THINK_TIME_SECONDS, VALUES.thinkTime.none); + }); + + it('rejects unknown scenarios and workloads', (t) => { + t.plan(2); + t.assert.throws( + () => parseArgs(['--scenario', TEXTS.scenario.unknown]), + /--scenario must be one of/ + ); + t.assert.throws( + () => parseArgs(['--scenario', TEXTS.scenario.fixedMedium, '--workload', TEXTS.workload.unknown]), + /--workload must be one of/ + ); + }); + + it('rejects unknown fixture profiles', (t) => { + t.plan(1); + t.assert.throws( + () => parseArgs(['--scenario', TEXTS.scenario.fixedMedium, '--fixture-profile', TEXTS.fixture.unknown]), + /--fixture-profile must be one of/ + ); + }); + + it('rejects unsupported k6 env keys so secrets are not forwarded', (t) => { + t.plan(1); + t.assert.throws( + () => parseArgs(['--scenario', TEXTS.scenario.fixedMedium, '--k6-env', `${TEXTS.env.token}=secret`]), + /Unsupported k6 env TOKEN/ + ); + }); + + it('rejects unsupported fixture k6 env keys', (t) => { + t.plan(1); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-fixtures-')); + const fixtureFile = path.join(directory, 'fixtures.json'); + fs.writeFileSync(fixtureFile, JSON.stringify({ + schema_version: 1, + profiles: { + [TEXTS.fixture.unsafe]: { + workload: TEXTS.workload.baseline, + k6_env: { + [TEXTS.env.token]: 'secret', + }, + }, + }, + }), 'utf-8'); + + try { + t.assert.throws( + () => loadFixtureProfile(TEXTS.fixture.unsafe, { file: fixtureFile }), + /unsupported k6 env TOKEN/ + ); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('rejects unsupported fixture workloads before option validation', (t) => { + t.plan(1); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-fixtures-')); + const fixtureFile = path.join(directory, 'fixtures.json'); + fs.writeFileSync(fixtureFile, JSON.stringify({ + schema_version: 1, + profiles: { + [TEXTS.fixture.unsafe]: { + workload: TEXTS.workload.typo, + k6_env: {}, + }, + }, + }), 'utf-8'); + + try { + t.assert.throws( + () => parseArgs([ + '--scenario', + TEXTS.scenario.fixedMedium, + '--fixture-profile', + TEXTS.fixture.unsafe, + '--fixture-profile-file', + fixtureFile, + ]), + /fixture profile unsafe has unsupported workload basline/ + ); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); +}); + +describe('runCommand', () => { + it('fails with a timeout-specific message', (t) => { + t.plan(1); + t.assert.throws( + () => runCommand(process.execPath, ['-e', 'setTimeout(() => {}, 1000)'], { + capture: true, + timeoutMs: 10, + }), + /timed out after 10ms/ + ); + }); +}); + +describe('ensureLocalBaseUrl', () => { + it('accepts local targets by default', (t) => { + t.plan(3); + t.assert.doesNotThrow(() => ensureLocalBaseUrl('http://localhost:8080', false)); + t.assert.doesNotThrow(() => ensureLocalBaseUrl('http://127.0.0.1:8080', false)); + t.assert.doesNotThrow(() => ensureLocalBaseUrl('http://[::1]:8080', false)); + }); + + it('rejects remote targets unless explicitly allowed', (t) => { + t.plan(2); + t.assert.throws( + () => ensureLocalBaseUrl('http://example.com', false), + /BASE_URL must point to localhost/ + ); + t.assert.doesNotThrow(() => ensureLocalBaseUrl('http://example.com', true)); + }); +}); + +describe('runK6', () => { + it('reports a clear install hint when k6 is missing', (t) => { + t.plan(1); + t.assert.throws( + () => assertK6Available('missing-k6', { + commandRunner: () => ({ + error: Object.assign(new Error('spawn missing-k6 ENOENT'), { + code: 'ENOENT', + }), + status: null, + }), + }), + /Install k6/ + ); + }); + + it('builds argv arrays instead of shell command strings', (t) => { + t.plan(6); + const calls = []; + const commandRunner = (command, args, options) => { + calls.push({ args, command, options }); + return { status: 0 }; + }; + + runK6( + { + allowRemoteBaseUrl: true, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: { + VUS: '2', + }, + script: 'load-tests/k6/load-test-baseline.js', + workload: 'smoke', + }, + { commandRunner } + ); + + t.assert.equal(calls[0].command, 'k6'); + t.assert.ok(Array.isArray(calls[0].args)); + t.assert.ok(calls[0].args.includes('--summary-export')); + t.assert.ok(calls[0].args.includes('VUS=2')); + t.assert.ok(calls[0].args.includes('ALLOW_REMOTE_BASE_URL=true')); + t.assert.equal(typeof calls[0].options.timeoutMs, 'number'); + }); + + it('reports a signal separately from exit code', (t) => { + t.plan(2); + const result = runK6( + { + allowRemoteBaseUrl: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + script: 'load-tests/k6/load-test-baseline.js', + workload: 'smoke', + }, + { + commandRunner: () => ({ + signal: 'SIGTERM', + status: null, + }), + } + ); + + t.assert.equal(result.exitCode, null); + t.assert.equal(result.signal, 'SIGTERM'); + }); +}); + +describe('Kubernetes state adapter helpers', () => { + it('deletes stale HPA for fixed scenarios unless skipped', (t) => { + t.plan(2); + const calls = []; + const commandRunner = (command, args) => { + calls.push({ args, command }); + return { status: 0 }; + }; + + const cleaned = cleanupHpaIfNeeded( + { skipHpaCleanup: false }, + { fixedReplicas: true }, + { commandRunner } + ); + + t.assert.equal(cleaned, true); + t.assert.deepEqual(calls[0], { + command: 'kubectl', + args: [ + '-n', + 'lined', + 'delete', + 'hpa', + 'lined-backend', + '--ignore-not-found', + ], + }); + }); + + it('skips HPA cleanup when requested', (t) => { + t.plan(2); + const calls = []; + const cleaned = cleanupHpaIfNeeded( + { skipHpaCleanup: true }, + { fixedReplicas: true }, + { + commandRunner: (command, args) => { + calls.push({ args, command }); + return { status: 0 }; + }, + } + ); + + t.assert.equal(cleaned, false); + t.assert.deepEqual(calls, []); + }); + + it('parses Kubernetes resource quantities and pod top output', (t) => { + t.plan(6); + t.assert.equal(parseCpuQuantity('500m'), 500); + t.assert.equal(parseCpuQuantity('1'), 1000); + t.assert.equal(parseCpuQuantity('250u'), 0.25); + t.assert.equal(parseMemoryQuantity('1Gi'), 1024 ** 3); + t.assert.equal(parseMemoryQuantity('512Mi'), 512 * 1024 ** 2); + t.assert.deepEqual(parseTopPods('lined-backend-a 250m 512Mi\n'), [{ + cpuMillicores: 250, + memoryBytes: 512 * 1024 ** 2, + name: 'lined-backend-a', + }]); + }); + + it('rejects malformed pod top output', (t) => { + t.plan(1); + t.assert.throws( + () => parseTopPods('lined-backend-a 250m\n'), + /Malformed kubectl top pods line/ + ); + }); + + it('summarizes Kubernetes utilization and restarts', (t) => { + t.plan(4); + const result = summarizeKubernetesState({ + deployment, + pods, + topOutput: 'lined-backend-a 250m 512Mi\nlined-backend-b 250m 512Mi\n', + }); + + t.assert.equal(result.cpuUtilization, 0.5); + t.assert.equal(result.memoryUtilization, 0.5); + t.assert.equal(result.restartCount, 3); + t.assert.equal(result.metricsServerAvailable, true); + }); + + it('omits utilization when metrics-server data is missing', (t) => { + t.plan(3); + const result = summarizeKubernetesState({ + deployment, + pods, + topOutput: '', + }); + + t.assert.equal(result.cpuUtilization, undefined); + t.assert.equal(result.memoryUtilization, undefined); + t.assert.equal(result.metricsServerAvailable, false); + }); +}); + +describe('buildRuntimeSummary', () => { + it('builds a collector-compatible summary and records missing optional metrics', (t) => { + t.plan(1); + const summary = buildRuntimeSummary({ + k6Summary: nestedK6Summary, + kubernetes: { + metricsServerAvailable: false, + restartCount: 0, + }, + scenario: 'fixed-medium', + workload: 'smoke', + }); + + t.assert.deepEqual(summary, { + schema_version: 1, + scenario: 'fixed-medium', + workload: 'smoke', + source: 'local-kind', + summary: { + latency_p95_ms: 250.5, + latency_p99_ms: 550.25, + error_rate: 0.002, + throughput_rps: 42.1, + restart_count: 0, + }, + missing: [ + 'availability', + 'cpu_utilization', + 'memory_utilization', + ], + }); + }); + + it('reads flat k6 v2 summary exports', (t) => { + t.plan(4); + const summary = buildRuntimeSummary({ + k6Summary: flatK6Summary, + kubernetes: { + metricsServerAvailable: false, + restartCount: 0, + }, + scenario: 'fixed-medium', + workload: 'smoke', + }); + + t.assert.equal(summary.summary.latency_p95_ms, 150.25); + t.assert.equal(summary.summary.latency_p99_ms, 275.5); + t.assert.equal(summary.summary.error_rate, 0); + t.assert.equal(summary.summary.throughput_rps, 25.5); + }); + + it('marks missing HPA fields for the HPA scenario when no HPA state is present', (t) => { + t.plan(2); + const summary = buildRuntimeSummary({ + k6Summary: nestedK6Summary, + kubernetes: { + metricsServerAvailable: false, + restartCount: 0, + }, + scenario: 'hpa-cpu', + workload: 'baseline', + }); + + t.assert.ok(summary.missing.includes('hpa_current_replicas')); + t.assert.ok(summary.missing.includes('hpa_desired_replicas')); + }); + + it('uses measurement-window restart deltas instead of cumulative snapshots', (t) => { + t.plan(4); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + restartCounts: [4, 6], + }) + ); + + t.assert.equal(result.summary.summary.restart_count, 2); + t.assert.equal(result.manifest.kubernetes.restart_count_before, 4); + t.assert.equal(result.manifest.kubernetes.restart_count_after, 6); + t.assert.equal(result.manifest.kubernetes.restart_count_delta, 2); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('does not emit a negative restart delta when pod counters reset', (t) => { + t.plan(3); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + restartCounts: [4, 1], + }) + ); + + t.assert.equal(result.summary.summary.restart_count, 0); + t.assert.equal(result.manifest.kubernetes.restart_count_before, 4); + t.assert.equal(result.manifest.kubernetes.restart_count_after, 1); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('fails without writing a collector summary when k6 omits summary export', (t) => { + t.plan(5); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + let thrown; + t.assert.throws( + () => runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: undefined, + }) + ), + (error) => { + thrown = error; + return /summary export/.test(error.message); + } + ); + + const runDirs = fs.readdirSync(directory); + const runDir = path.join(directory, runDirs[0]); + const manifest = JSON.parse( + fs.readFileSync(path.join(runDir, 'runtime-summary-manifest.json'), 'utf-8') + ); + + t.assert.equal(thrown instanceof Error, true); + t.assert.equal(fs.existsSync(path.join(runDir, 'runtime-summary-manifest.json')), true); + t.assert.equal(fs.existsSync(path.join(runDir, 'runtime-summary.json')), false); + t.assert.equal(manifest.collector_summary_written, false); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); +}); + +describe('manifest and runScenario', () => { + it('records sanitized provenance in the manifest', (t) => { + t.plan(6); + const manifest = buildManifest({ + appliedScenario: true, + finishedAt: '2026-06-01T10:00:10.000Z', + git: { + branch: 'bug/scenario-runner-seam', + commit: 'abc123', + }, + hpaCleanup: true, + k6: { + exitCode: 0, + signal: undefined, + summaryTrendStats: 'p(95),p(99),avg,min,max', + }, + kubernetes: { + metricsServerAvailable: false, + replicas: 1, + }, + options: { + baseUrl: 'http://localhost:8080', + fixtureProfileData: { + k6Env: { + USER_COUNT: '4', + VUS: '2', + }, + name: 'comparison-baseline', + schemaVersion: 1, + workload: 'baseline', + }, + k6Bin: 'k6', + k6Env: { + VUS: '2', + }, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + workload: 'smoke', + }, + scenario: { + path: 'k8s/kind/scenarios/fixed-medium', + }, + startedAt: '2026-06-01T10:00:00.000Z', + summaryExported: true, + summaryWritten: true, + }); + + t.assert.equal(manifest.kubernetes.applied_scenario, true); + t.assert.equal(manifest.kubernetes.hpa_cleanup, true); + t.assert.equal(manifest.collector_summary_written, true); + t.assert.deepEqual(manifest.fixture_profile, { + name: 'comparison-baseline', + schema_version: 1, + workload: 'baseline', + k6_env: { + USER_COUNT: '4', + VUS: '2', + }, + }); + t.assert.equal(manifest.workload_env.VUS, '2'); + t.assert.equal(manifest.git.branch, 'bug/scenario-runner-seam'); + }); + + it('writes a summary and manifest for successful runs', (t) => { + t.plan(5); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + }) + ); + + t.assert.equal(fs.existsSync(result.summaryPath), true); + t.assert.equal(fs.existsSync(result.manifestPath), true); + t.assert.equal(result.summary.summary.latency_p95_ms, 250.5); + t.assert.equal(result.summary.fixture_profile, undefined); + t.assert.equal(result.manifest.collector_summary_written, true); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('applies fixture profiles when runScenario is called directly', (t) => { + t.plan(5); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + fixtureProfile: TEXTS.fixture.readHeavy, + k6Bin: 'k6', + k6Env: { + VUS: VALUES.vus.override, + }, + outputRoot: directory, + scenario: TEXTS.scenario.fixedMedium, + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: TEXTS.workload.baseline, + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + }) + ); + + t.assert.equal(result.summary.workload, TEXTS.workload.readHeavy); + t.assert.equal(result.manifest.fixture_profile.name, TEXTS.fixture.readHeavy); + t.assert.equal(result.manifest.workload_env.WORKLOAD, TEXTS.workload.readHeavy); + t.assert.equal(result.manifest.workload_env.USER_COUNT, VALUES.users.baselineCount); + t.assert.equal(result.manifest.workload_env.VUS, VALUES.vus.override); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('lets direct runScenario workload overrides win over fixture defaults', (t) => { + t.plan(4); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + fixtureProfile: TEXTS.fixture.baseline, + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: TEXTS.scenario.fixedMedium, + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: TEXTS.workload.readHeavy, + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + }) + ); + + t.assert.equal(result.summary.workload, TEXTS.workload.readHeavy); + t.assert.equal(result.manifest.fixture_profile.name, TEXTS.fixture.baseline); + t.assert.equal(result.manifest.workload_env.WORKLOAD, TEXTS.workload.readHeavy); + t.assert.equal(result.manifest.workload_env.USER_COUNT, VALUES.users.baselineCount); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('lets direct runScenario default workload overrides win when marked explicit', (t) => { + t.plan(4); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + const result = runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + fixtureProfile: TEXTS.fixture.readHeavy, + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: TEXTS.scenario.fixedMedium, + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: TEXTS.workload.baseline, + workloadExplicit: true, + }, + fakeAdapters({ + k6ExitCode: 0, + k6Summary: nestedK6Summary, + }) + ); + + t.assert.equal(result.summary.workload, TEXTS.workload.baseline); + t.assert.equal(result.manifest.fixture_profile.name, TEXTS.fixture.readHeavy); + t.assert.equal(result.manifest.workload_env.WORKLOAD, TEXTS.workload.baseline); + t.assert.equal(result.manifest.workload_env.USER_COUNT, VALUES.users.baselineCount); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('uses signal-specific errors when k6 is killed', (t) => { + t.plan(2); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + t.assert.throws( + () => runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: null, + k6Signal: 'SIGTERM', + k6Summary: nestedK6Summary, + }) + ), + /k6 was killed by signal SIGTERM/ + ); + + const runDir = path.join(directory, fs.readdirSync(directory)[0]); + const manifest = JSON.parse( + fs.readFileSync(path.join(runDir, 'runtime-summary-manifest.json'), 'utf-8') + ); + t.assert.equal(manifest.k6.signal, 'SIGTERM'); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); + + it('writes only a manifest when k6 fails', (t) => { + t.plan(4); + const directory = fs.mkdtempSync(path.join(os.tmpdir(), 'lined-runner-')); + + try { + t.assert.throws( + () => runScenario( + { + allowRemoteBaseUrl: false, + apply: false, + baseUrl: 'http://localhost:8080', + k6Bin: 'k6', + k6Env: {}, + outputRoot: directory, + scenario: 'fixed-medium', + script: 'load-tests/k6/load-test-baseline.js', + skipHpaCleanup: false, + workload: 'smoke', + }, + fakeAdapters({ + k6ExitCode: 1, + k6Summary: nestedK6Summary, + }) + ), + ScenarioRunError + ); + + const runDirs = fs.readdirSync(directory); + t.assert.equal(runDirs.length, 1); + const runDir = path.join(directory, runDirs[0]); + t.assert.equal(fs.existsSync(path.join(runDir, 'runtime-summary-manifest.json')), true); + t.assert.equal(fs.existsSync(path.join(runDir, 'runtime-summary.json')), false); + } finally { + fs.rmSync(directory, { force: true, recursive: true }); + } + }); +}); + +const fakeAdapters = ({ + k6ExitCode, + k6Signal, + k6Summary, + restartCounts = [0, 0], +}) => ({ + clock: fakeClock(), + gitReader: () => ({ + branch: 'bug/scenario-runner-seam', + commit: 'abc123', + }), + k6Adapter: { + assertAvailable: () => {}, + run: () => ({ + exitCode: k6ExitCode, + signal: k6Signal, + summary: k6Summary, + }), + }, + kubernetesAdapter: { + applyScenarioIfNeeded: () => false, + cleanupHpaIfNeeded: () => true, + collectState: () => { + const restartCount = restartCounts.shift() ?? restartCounts.at(-1) ?? 0; + return { + metricsServerAvailable: false, + replicas: 1, + restartCount, + }; + }, + waitForRollout: () => {}, + }, +}); + +const fakeClock = () => { + const times = [ + '2026-06-01T10:00:00.000Z', + '2026-06-01T10:00:10.000Z', + ]; + return () => times.shift() ?? '2026-06-01T10:00:10.000Z'; +};