Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/lined/docs/experiment-tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ scientific experiment work.
| `experiment/prometheus-telemetry-pipeline` | Task | Runtime infrastructure | Yes | Prometheus telemetry pipeline | Add a local Prometheus collection path for the kind backend, including scrape configuration and documentation for collecting Actuator runtime metrics. | Runtime metrics from the kind backend can be collected persistently enough for scenario comparison. |
| `experiment/runtime-scenario-summaries` | Task | Runtime evidence | Partial / branch-only | Runtime scenario summaries | Add a repeatable workflow for running each deployment scenario under the selected k6 workload and producing sanitized `runtime-summary.json` artifacts. | Each scenario has comparable runtime summaries ready for collector ingestion. |
| `experiment/runtime-provenance-manifest` | Task | Runtime evidence | Partial / branch-only | Runtime provenance manifest | Add provenance metadata to runtime fitness outputs, including commit SHA, image tag, workload, telemetry window, configuration hash, constraint version, and fitness vector. | Runtime fitness results are traceable and reproducible for audit and paper evidence. |
| `experiment/scenario-fixture-discipline` | Task | Runtime evidence | No | Scenario fixture discipline | Define explicit workload/context profiles and repeatable input setup for Lined experiment scenario runs. | Deployment/runtime comparisons use stable fixtures instead of manual setup. |
| `experiment/scenario-fixture-discipline` | Task | Runtime evidence | Yes | Scenario fixture discipline | Define explicit workload/context profiles and repeatable input setup for Lined experiment scenario runs. | Deployment/runtime comparisons use stable fixtures instead of manual setup. |
| `experiment/slo-constraint-thresholds` | Task | Runtime evidence | Yes | SLO and constraint thresholds | Define initial latency, error-rate, availability, restart, readiness, and resource-efficiency thresholds for classifying valid experiment variants. | Runtime evidence can be evaluated against explicit constraints instead of ad hoc interpretation. |
| `experiment/fitness-runtime-extension` | Task | Runtime scoring | Yes | Runtime fitness extension | Extend experiment documentation and/or collector design to include telemetry metrics. | Fixed CI fitness can be compared with runtime-aware adaptive fitness. |
| `experiment/runtime-aware-scoring` | Task | Runtime scoring | No | Runtime-aware scoring | Add a versioned runtime fitness score that uses summarized runtime metrics while preserving the existing structural `fitnessScore`. | Runtime-aware scalar fitness can be computed without changing historical CI fitness semantics. |
Expand Down
20 changes: 20 additions & 0 deletions backend/lined/docs/load-test-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,26 @@ k6 run \
load-tests/k6/load-test-baseline.js
```

## Fixture Profiles

For deployment/runtime comparison runs, prefer the scenario-runner fixture
profiles in `load-tests/runtime-scenarios/fixture-profiles-v1.json` instead of
passing ad hoc k6 environment values. The profiles pin the workload and setup
inputs used by the k6 script, including user count, seeded task and event
counts, VU or stress settings, duration, and think time.

Run a stable comparison fixture through the scenario runner:

```bash
node load-tests/runtime-scenarios/scenario-runner-cli.mjs \
--scenario fixed-medium \
--fixture-profile comparison-baseline \
--base-url http://localhost:8080
```

Use direct `k6 run` commands for one-off local checks. Use fixture profiles for
experiment evidence that will be compared across deployment scenarios.

## Run with Docker

If k6 is not installed locally, use the official Grafana k6 image from the
Expand Down
43 changes: 39 additions & 4 deletions backend/lined/docs/runtime-scenario-summaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,23 @@ smoke, baseline, read-heavy, write-heavy, mixed, stress, negative-smoke
Use `smoke` for command validation and one of the longer non-negative profiles
for scenario comparison.

Fixture profiles are versioned workload/context presets from
`load-tests/runtime-scenarios/fixture-profiles-v1.json`:

| Fixture profile | Workload | Purpose |
|-----------------|----------|---------|
| `local-smoke` | `smoke` | Minimal local command validation. |
| `comparison-baseline` | `baseline` | Stable default runtime comparison fixture. |
| `comparison-read-heavy` | `read-heavy` | Read-oriented comparison over bounded setup data. |
| `comparison-write-heavy` | `write-heavy` | Write-oriented comparison with per-iteration cleanup. |
| `comparison-mixed` | `mixed` | Mixed reads, updates, and bounded writes. |
| `comparison-stress` | `stress` | Ramping-VU local stress comparison. |

Profiles make the workload setup explicit by pinning allowed k6 inputs such as
`USER_COUNT`, `SEED_TASK_COUNT`, `SEED_EVENT_COUNT`, `VUS`, `DURATION`,
`STRESS_MAX_VUS`, `STRESS_STAGE_DURATION`, and `THINK_TIME_SECONDS`.
They do not change backend behavior or deployment manifests.

## Run One Scenario

Run the fixed-medium scenario with the smoke workload:
Expand All @@ -70,15 +87,31 @@ node load-tests/runtime-scenarios/scenario-runner-cli.mjs \
--base-url http://localhost:8080
```

Run the same scenario with the default baseline workload:
Run the same scenario with the stable baseline fixture:

```bash
node load-tests/runtime-scenarios/scenario-runner-cli.mjs \
--scenario fixed-medium \
--workload baseline \
--fixture-profile comparison-baseline \
--base-url http://localhost:8080
```

The fixture profile supplies default workload and k6 environment inputs.
Explicit CLI options still win:

```bash
node load-tests/runtime-scenarios/scenario-runner-cli.mjs \
--scenario fixed-medium \
--fixture-profile comparison-baseline \
--workload read-heavy \
--k6-env VUS=2 \
--base-url http://localhost:8080
```

Use overrides only when the run intentionally differs from the named fixture;
the manifest records both the selected profile and the effective workload
environment.

The runner applies the selected scenario, waits for the backend rollout, runs
k6 with summary export, collects summarized Kubernetes state, and writes:

Expand Down Expand Up @@ -119,6 +152,7 @@ side effect.
The runner keeps inputs narrow:

- scenario and workload names are hardcoded allowlists;
- fixture profile names and profile k6 environment keys are allowlisted;
- extra k6 environment variables are allowlisted;
- `BASE_URL` must point to `localhost`, `127.0.0.1`, or `[::1]` unless
`--allow-remote-base-url` is provided;
Expand Down Expand Up @@ -179,8 +213,9 @@ the metric is omitted and listed in `missing`.

`runtime-summary-manifest.json` is not collector input. It records sanitized
provenance such as scenario path, workload variables, git commit, CLI version,
start/end timestamps, whether the scenario was applied, HPA cleanup status,
raw pre/post restart snapshots, the restart delta, and k6 exit code.
fixture profile metadata, start/end timestamps, whether the scenario was
applied, HPA cleanup status, raw pre/post restart snapshots, the restart delta,
and k6 exit code.

## Validate Locally

Expand Down
32 changes: 32 additions & 0 deletions backend/lined/load-tests/runtime-scenarios/command-runner.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import { spawnSync } from 'node:child_process';

export const runCommand = (
command,
args,
{ allowFailure = false, capture = false, cwd, timeoutMs } = {}
) => {
const result = spawnSync(command, args, {
cwd,
encoding: capture ? 'utf-8' : undefined,
stdio: capture ? ['ignore', 'pipe', 'pipe'] : 'inherit',
timeout: timeoutMs,
});

if (isTimeout(result, timeoutMs) && !allowFailure) {
throw new Error(`${command} timed out after ${timeoutMs}ms`);
}
if (result.error && !allowFailure) {
throw result.error;
}
if (result.signal && !allowFailure) {
throw new Error(`${command} was killed by signal ${result.signal}`);
}
if (!allowFailure && result.status !== 0) {
throw new Error(`${command} ${args.join(' ')} failed with exit code ${result.status}`);
}

return result;
};
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No timeout on spawnSync.

kubectl rollout status can block for minutes (or indefinitely if a pod never becomes Ready), and a stuck k6 run will also block forever. Consider threading a timeout option through the adapters:

export const runCommand = (
  command, args,
  { allowFailure = false, capture = false, cwd, timeoutMs } = {}
) => {
  const result = spawnSync(command, args, {
    cwd,
    encoding: capture ? 'utf-8' : undefined,
    stdio: capture ? ['ignore', 'pipe', 'pipe'] : 'inherit',
    timeout: timeoutMs,   // undefined = no limit
  });
  // spawnSync sets signal='SIGTERM', status=null on timeout
  if (result.signal === 'SIGTERM' && !allowFailure) {
    throw new Error(`${command} timed out after ${timeoutMs}ms`);
  }
  ...

Even a generous ceiling (e.g. 10 min for rollout, 5 min for k6 smoke) prevents a CI job from hanging until its runner wall-clock limit kills it with no useful error message.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


const isTimeout = (result, timeoutMs) => timeoutMs !== undefined
&& (result.error?.code === 'ETIMEDOUT' || result.signal === 'SIGTERM');
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
{
"schema_version": 1,
"profiles": {
"local-smoke": {
"description": "Minimal fixture for local command validation.",
"workload": "smoke",
"k6_env": {
"USER_COUNT": "2",
"SEED_TASK_COUNT": "2",
"SEED_EVENT_COUNT": "2",
"THINK_TIME_SECONDS": "0"
}
},
"comparison-baseline": {
"description": "Standard fixture for stable baseline scenario comparison.",
"workload": "baseline",
"k6_env": {
"USER_COUNT": "4",
"SEED_TASK_COUNT": "12",
"SEED_EVENT_COUNT": "8",
"VUS": "5",
"DURATION": "2m",
"THINK_TIME_SECONDS": "1"
}
},
"comparison-read-heavy": {
"description": "Read-oriented fixture over bounded users, lobby, tasks, and events.",
"workload": "read-heavy",
"k6_env": {
"USER_COUNT": "4",
"SEED_TASK_COUNT": "12",
"SEED_EVENT_COUNT": "8",
"VUS": "5",
"DURATION": "2m",
"THINK_TIME_SECONDS": "1"
}
},
"comparison-write-heavy": {
"description": "Write-oriented fixture with bounded setup and per-iteration cleanup.",
"workload": "write-heavy",
"k6_env": {
"USER_COUNT": "4",
"SEED_TASK_COUNT": "12",
"SEED_EVENT_COUNT": "8",
"VUS": "5",
"DURATION": "2m",
"THINK_TIME_SECONDS": "1"
}
},
"comparison-mixed": {
"description": "Mixed read, update, and bounded write fixture for scenario comparison.",
"workload": "mixed",
"k6_env": {
"USER_COUNT": "4",
"SEED_TASK_COUNT": "12",
"SEED_EVENT_COUNT": "8",
"VUS": "5",
"DURATION": "2m",
"THINK_TIME_SECONDS": "1"
}
},
"comparison-stress": {
"description": "Ramping-VU fixture for local stress comparison.",
"workload": "stress",
"k6_env": {
"USER_COUNT": "4",
"SEED_TASK_COUNT": "12",
"SEED_EVENT_COUNT": "8",
"STRESS_MAX_VUS": "20",
"STRESS_STAGE_DURATION": "30s",
"THINK_TIME_SECONDS": "1"
}
}
}
}
131 changes: 131 additions & 0 deletions backend/lined/load-tests/runtime-scenarios/fixture-profiles.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
import fs from 'node:fs';
import path from 'node:path';

export const FIXTURE_PROFILES_PATH = 'load-tests/runtime-scenarios/fixture-profiles-v1.json';
export const K6_ENV_KEYS = new Set([
'RUN_ID',
'USER_COUNT',
'SEED_TASK_COUNT',
'SEED_EVENT_COUNT',
'VUS',
'DURATION',
'STRESS_MAX_VUS',
'STRESS_STAGE_DURATION',
'THINK_TIME_SECONDS',
]);

const PROFILE_KEYS = new Set(['description', 'workload', 'k6_env']);

export const fixtureProfileNames = ({ cwd = process.cwd(), file = FIXTURE_PROFILES_PATH } = {}) => {
const artifact = readFixtureArtifact(cwd, file);
return Object.keys(artifact.profiles).sort();
};

export const loadFixtureProfile = (
name,
{ allowedWorkloads, cwd = process.cwd(), file = FIXTURE_PROFILES_PATH } = {}
) => {
const artifact = readFixtureArtifact(cwd, file);
const profile = artifact.profiles[name];
if (!profile) {
throw new Error(`--fixture-profile must be one of: ${Object.keys(artifact.profiles).sort().join(', ')}`);
}
validateProfile(name, profile, allowedWorkloads);
return {
description: profile.description,
k6Env: { ...profile.k6_env },
name,
schemaVersion: artifact.schema_version,
workload: profile.workload,
};
};

export const applyFixtureProfileDefaults = (
options,
{
cwd = process.cwd(),
explicitK6Env = {},
fixtureFile = FIXTURE_PROFILES_PATH,
allowedWorkloads,
workloadExplicit = false,
} = {}
) => {
if (!options.fixtureProfile) {
return options;
}

const profile = loadFixtureProfile(options.fixtureProfile, {
allowedWorkloads,
cwd,
file: fixtureFile,
});
const merged = {
...options,
fixtureProfileData: profile,
k6Env: {
...profile.k6Env,
...explicitK6Env,
},
};
if (!workloadExplicit) {
merged.workload = profile.workload;
}
return merged;
};

const readFixtureArtifact = (cwd, file) => {
const artifactPath = path.resolve(cwd, file);
const parsed = JSON.parse(fs.readFileSync(artifactPath, 'utf-8'));
if (!isRecord(parsed) || parsed.schema_version !== 1 || !isRecord(parsed.profiles)) {
throw new Error('fixture profile artifact must contain schema_version 1 and profiles');
}
return parsed;
};

const validateProfile = (name, profile, allowedWorkloads) => {
requireRecord(`fixture profile ${name}`, profile);
requireKnownProfileKeys(name, profile);
requireWorkload(name, profile.workload, allowedWorkloads);
requireK6Env(name, profile.k6_env);
};

const requireRecord = (label, value) => {
if (!isRecord(value)) {
throw new Error(`${label} must be an object`);
}
};

const requireKnownProfileKeys = (name, profile) => {
const unknownKeys = Object.keys(profile).filter((key) => !PROFILE_KEYS.has(key));
if (unknownKeys.length > 0) {
throw new Error(`fixture profile ${name} has unsupported keys: ${unknownKeys.join(', ')}`);
}
};

const requireWorkload = (name, workload, allowedWorkloads) => {
if (typeof workload !== 'string' || workload.length === 0) {
throw new Error(`fixture profile ${name} must define workload`);
}
if (allowedWorkloads && !allowedWorkloads.has(workload)) {
throw new Error(
`fixture profile ${name} has unsupported workload ${workload}; `
+ `allowed: ${Array.from(allowedWorkloads).join(', ')}`
);
}
};

const requireK6Env = (name, k6Env) => {
requireRecord(`fixture profile ${name} k6_env`, k6Env);
Object.entries(k6Env).forEach(([key, value]) => requireK6EnvEntry(name, key, value));
};

const requireK6EnvEntry = (name, key, value) => {
if (!K6_ENV_KEYS.has(key)) {
throw new Error(`fixture profile ${name} has unsupported k6 env ${key}`);
}
if (typeof value !== 'string') {
throw new Error(`fixture profile ${name} k6 env ${key} must be a string`);
}
};

const isRecord = (value) => typeof value === 'object' && value !== null && !Array.isArray(value);
Loading
Loading