diff --git a/docs/docs/dev-manuals/architecture/component-diagram.mdx b/docs/docs/dev-manuals/architecture/component-diagram.mdx index cdbc315..95ce61f 100644 --- a/docs/docs/dev-manuals/architecture/component-diagram.mdx +++ b/docs/docs/dev-manuals/architecture/component-diagram.mdx @@ -26,3 +26,11 @@ Explanation: - **solid line**: asynchronous communication (dapr pub/sub) + +--- + +## Component Diagram of the MiSArch Experiment Tool + +For the MiSArch Experiment Tool, the component diagram is as follows: + + diff --git a/docs/docs/dev-manuals/architecture/experiment-configuration.mdx b/docs/docs/dev-manuals/architecture/experiment-configuration.mdx index f371f63..915d0d7 100644 --- a/docs/docs/dev-manuals/architecture/experiment-configuration.mdx +++ b/docs/docs/dev-manuals/architecture/experiment-configuration.mdx @@ -2,6 +2,8 @@ sidebar_position: 3 --- +> Note that this page is still viable, but the MiSArch Experiment Tool has way more functionality by now and can be used instead. + import CustomImage from "@site/src/components/CustomImage"; # Experiment configuration diff --git a/docs/docs/dev-manuals/other-repositories/experiment-executor-frontend.mdx b/docs/docs/dev-manuals/other-repositories/experiment-executor-frontend.mdx index 3e403e1..aa990d7 100644 --- a/docs/docs/dev-manuals/other-repositories/experiment-executor-frontend.mdx +++ b/docs/docs/dev-manuals/other-repositories/experiment-executor-frontend.mdx @@ -19,10 +19,10 @@ The repository is structured as follows:
- `/src/`: The source code - - `components/`: [Single-File Components](https://vuejs.org/glossary/#single-file-component) that can be reused -- this is basically an internal UI components library. - - `model/`: Data models used in the frontend. - - `types/`: TypeScript types. - - `util/`: Utility source code. + - `components/`: [Single-File Components](https://vuejs.org/glossary/#single-file-component) that can be reused -- this is basically an internal UI components library. + - `model/`: Data models used in the frontend. + - `types/`: TypeScript types. + - `util/`: Utility source code.
diff --git a/docs/docs/dev-manuals/other-repositories/experiment-executor.mdx b/docs/docs/dev-manuals/other-repositories/experiment-executor.mdx index 1cbaae8..4eb7cde 100644 --- a/docs/docs/dev-manuals/other-repositories/experiment-executor.mdx +++ b/docs/docs/dev-manuals/other-repositories/experiment-executor.mdx @@ -87,7 +87,57 @@ The following steps describe the workflow that is executed when an experiment is ### Synchronization & Registration -TODO mermaid +```mermaid +sequenceDiagram + participant EE as Experiment Executor + participant MEC as MiSArch Experiment Config + participant CTE as Chaos Toolkit Executor + participant GE as Gatling Executor + participant GR as Grafana + + EE->>MEC: Reset All Configurations + EE->>EE: Register MiSArch Experiment Config as Ready + EE->>CTE: Send Failure Injection Configuration + CTE->>CTE: Load Failure Configuration and Prepare Execution + EE->>GE: Send Load Test Configuration + GE->>GE: Load Load Test Configuration and Prepare Execution + + loop Poll Experiment Trigger + EE->>EE: Check Trigger Condition + end + + CTE->>EE: Register Chaos Toolkit as Ready + loop Poll Experiment Trigger + CTE->>EE: Check Trigger Condition + end + + + GE->>EE: Register Gatling as Ready + loop Poll Experiment Trigger + GE->>EE: Check Trigger Condition + end + + EE->>CTE: Start Chaos Toolkit Execution + EE->>GE: Start Load Execution + EE->>EE: Start MiSArch Experiment Execution + + loop Configure Failure + EE->>MEC: Apply Failure Configuration + end + + loop Configure Failure + CTE->>CTE: Apply Failure Configuration + end + + + loop Execute Workload + GE->>GE: Execute Workload against SuT + end + + GE->>EE: Load Test Metrics + EE->>EE: Aggregate and Store Results + EE->>GR: Forward Result Dashboard +``` - Endpoints Provided: 1. Register component as ready. diff --git a/docs/docs/dev-manuals/other-repositories/gatling-executor.mdx b/docs/docs/dev-manuals/other-repositories/gatling-executor.mdx index 3831ecf..bccf5e0 100644 --- a/docs/docs/dev-manuals/other-repositories/gatling-executor.mdx +++ b/docs/docs/dev-manuals/other-repositories/gatling-executor.mdx @@ -29,12 +29,13 @@ The Gatling Executor exposes the following REST API endpoints to manage and exec
- `/gatling-server/src`: Source code of the web server Spring Boot component - - `controller/`: Package that includes the REST controller - - `service/`: Package for all service classes containing the actual business logic + + - `controller/`: Package that includes the REST controller + - `service/`: Package for all service classes containing the actual business logic - `/gatling-test/src`: Source code of the Gatling load testing component - - `kotlin`: Kotlin source files that are compiled on experiment start by Gradle, which will be extend by the dynamic Kotlin files received from the API - - `resources`: Template files for reference + - `kotlin`: Kotlin source files that are compiled on experiment start by Gradle, which will be extend by the dynamic Kotlin files received from the API + - `resources`: Template files for reference
diff --git a/docs/docs/user-manuals/how-to-use-experiment-tool.mdx b/docs/docs/user-manuals/how-to-use-experiment-tool.mdx new file mode 100644 index 0000000..a285c4c --- /dev/null +++ b/docs/docs/user-manuals/how-to-use-experiment-tool.mdx @@ -0,0 +1,315 @@ +import CustomImage from "@site/src/components/CustomImage"; + +# How to Use the MiSArch Experiment Tool + +> This page walks you through **every step** of designing, executing, and analyzing experiments with the MiSArch Experiment Tool. + +> Note, that the MiSArch Experiment Tool itself contains several helpful documentation in the help pages, helping you to get your experiments running. + +--- + +## Contents + +1. [What is an Experiment?](#what-is-an-experiment) +2. [Before You Start (Artifacts & Concepts)](#before-you-start-artifacts--concepts) +3. [Create or Load an Experiment](#create-or-load-an-experiment) +4. [Tour of the User Interface](#tour-of-the-user-interface) +5. [Configure Work (Gatling Kotlin DSL)](#configure-work-gatling-kotlin-dsl) +6. [Configure Load (Open-Load Model)](#configure-load-open-load-model) +7. [Optional: Configure a Warm-Up](#optional-configure-a-warm-up) +8. [Configure Failures](#configure-failures) + - [Chaos Toolkit Failures](#chaos-toolkit-failures) + - [MiSArch Experiment Configuration Failures](#misarch-experiment-configuration-failures) +9. [Define Experiment Goals](#define-experiment-goals) + - [Manual Goals](#manual-goals) + - [Automated Goal Definition](#automated-goal-definition) +10. [Visualize with the Experiment Graph](#visualize-with-the-experiment-graph) +11. [Run, Stop, Save, and Version Experiments](#run-stop-save-and-version-experiments) +12. [Analyze Results in Grafana Dashboards](#analyze-results-in-grafana-dashboards) +13. [Tips, Constraints, and Gotchas](#tips-constraints-and-gotchas) +14. [Storage Structure & Files](#appendix-storage-structure--files) + +--- + +## What is an Experiment? + +An **experiment** is a versioned bundle of configurations identified by **Test-UUID** and **Test-Version**. It can contain: + +- **Work** (the user/session _behavior_ scripted in Gatling Kotlin DSL). +- **Load** (the _arrival pattern_ of users for each work scenario, modeled as open load). +- **Failures** via: + - **Chaos Toolkit** (standard experiments/probes/actions), + - **MiSArch Experiment Configuration** (service-level deterioration and resource stress). +- **Goals** (thresholds for client-facing performance metrics). +- An optional **warm-up** phase. + +Experiments live as files in the Experiment Executor’s storage, grouped by Test-UUID and version. + + + +--- + +## Before You Start (Artifacts & Concepts) + +You’ll work with these pieces: + +- **Gatling Work**: Kotlin DSL scenarios describing the sequence of requests and expected data, plus pauses to mimic user behavior. +- **Gatling Load**: Open-load arrivals (users per second) for each scenario; duration is in seconds. +- **Chaos Toolkit**: Optional steady-state hypothesis (probes) and a sequence of actions/probes (with pauses) to inject failures. +- **MiSArch Experiment Configuration**: Time-ordered sets of failures for specific services: + - Pub-sub deterioration (delay/error on dapr pub-sub), + - Service-invocation deterioration (delay/error on HTTP), + - Artificial **memory** or **CPU** usage. + - Failures **persist until overwritten** by the next failure set. +- **Goals & Warm-Up**: + - **Goals**: e.g., max/mean response time thresholds. + - **Warm-up**: constant _closed_ load per scenario for a configured time before the actual experiment. + +--- + +## Create or Load an Experiment + +When opening the tool you can either **load** an existing experiment version or **generate** a new one. + +![Experiment Selection](/images/experiment-executor-frontend-screenshots/Frontend_select.png) + +**Generating a new experiment** provides initial presets: + +- Choose a **load pattern**: + 1. **Realistic** (sampled from a real web-shop day pattern, scaled by your max-arrivals), + 2. **Scalability** (monotonic increase), + 3. **Elasticity** (three spikes with cool-downs), + 4. **Resilience** (slow ramp, cool-down, big spike, cool-down, high plateau). +- Default **work scenarios** (e.g., `buyProcessScenario`, `abortedBuyProcessScenario`) for realistic/resilience; `buyProcessScenario` for resilience/scalability. +- Default **failures** (example): + - Chaos Toolkit: check gateway healthy, then terminate it at experiment start (Kubernetes/Docker variants). + - MiSArch Config: HTTP 404 with 1000 ms delay on **catalog** service for 1/6 of the experiment after the Chaos failure. +- Default **goals**: max response time **2000 ms**, mean **1000 ms** (adjust as needed). + +![Load Patterns](/images/experiment-executor-frontend-screenshots/load_patterns.svg) + +--- + +## Tour of the User Interface + +The single-page UI has these segments: + +1. **Header** (save, delete, load/new, new version, execute/stop, global help). +2. **Experiment Graph** (arrivals/requests plus failure timelines; toggle between load and warm-up view). +3. **Work Configuration** (code editor with tabs per scenario). +4. **Chaos Toolkit Configuration** (visual editor or raw JSON editor). +5. **MiSArch Experiment Configuration** (visual editor or raw JSON editor). +6. **Goal Configuration** (manual thresholds or auto goal definition). + +![Entire User Interface](/images/experiment-executor-frontend-screenshots/Frontend_Entire.png) + +--- + +## Configure Work (Gatling Kotlin DSL) + +Open **Work** in the top-right editor. Each **scenario** is a tab: + +- Add / rename / delete scenarios via the UI controls. +- Script _requests_, _checks_, and _pauses_ (Durations) to approximate realistic behavior. +- Keep work and load **separate** (load is configured elsewhere). + +> **Note:** Only the **Gatling Kotlin DSL** is supported for work scripts. + +**Example code:** + +```kotlin +val abortedBuyProcessScenario = scenario("abortedBuyProcessScenario") + .exec { session -> + session.set("targetUrl", "http://misarch-gateway.misarch.svc.cluster.local:8080/graphql") + } + .exec( + http("Get Access Token") + .post("http://keycloak.misarch.svc.cluster.local:80/keycloak/realms/Misarch/protocol/openid-connect/token") + .check(jsonPath("$.access_token").saveAs("accessToken")) + ) + .pause(Duration.ofMillis(0), Duration.ofMillis(0)) +``` + +## Configure Load (Open-Load Model) + +Click the **gear** icon in the graph to open the Load Configuration. + + + +For each scenario: + +- **Duration (s):** total time during which users arrive. +- **Arrivals (users/s):** specify how many new users start the scenario each second. + > **Example (10 s):** 3, 4, 0, 0, 0, 0, 0, 0, 0, 5 + > This does not imply no requests in “zero” seconds—users that arrived earlier are still completing work. + +--- + +## Optional: Configure a Warm-Up + +Open the **hamburger menu** to configure a warm-up: + +- **Duration** +- **Constant users per scenario** (closed load) + +This helps stabilize the system so your actual experiment measurements start under steady conditions. + + + +Replace with your warm-up dialog screenshot. + +--- + +## Configure Failures + +You can define failures in two places. Both offer: + +- A **simplified form** to add/remove/edit failures quickly. +- A **raw JSON** editor if you prefer direct DSL editing. +- Both views stay in sync. + +### Chaos Toolkit Failures + +- Optional **steady-state hypothesis** (probes) to assert system health before failure. +- **Method** actions and/or probes that inject failures; pauses available before/after. +- Examples: check a Kubernetes deployment is healthy, then terminate pods matching a label; or Docker container termination and restart. + +```json +{ + "title": "your-uuid:vX", + "steady-state-hypothesis": { + /* probes */ + }, + "method": [ + { + "type": "action", + "name": "Your Action", + "provider": { + /* module/func + args */ + }, + "pauses": { "before": 0, "after": 0 } + } + ] +} +``` + +### MiSArch Experiment Configuration Failures + +- Define **failure sets** for one or more services at specific times, with optional pauses between sets: + - **pubSubDeterioration** (delay/error probability), + - **serviceInvocationDeterioration** (delay/error code/probabilities), + - **artificialMemoryUsage**, + - **artificialCPUUsage**. +- Important: A failure **persists** until an ensuing set overwrites it—this enables overlapping, time-boxed failure compositions. + +```json +[ + { + "failures": [ + { + "name": "catalog", + "pubSubDeterioration": null, + "serviceInvocationDeterioration": [ + { "path": "/", "delay": 1000, "delayProbability": 1, "errorProbability": 1, "errorCode": 404 } + ], + "artificialMemoryUsage": null, + "artificialCPUUsage": null + } + ], + "pauses": { "before": 10, "after": 5 } + }, + { + "failures": [{ "name": "catalog", "serviceInvocationDeterioration": null }], + "pauses": { "before": 0, "after": 0 } + } +] +``` + +--- + +## Define Experiment Goals + +You can set goals manually or automatically. + +### Manual Goals + +- For each goal, specify: + - **Metric** (dropdown), + - **Threshold** (number), + - **Highlight color** for violations in dashboards. + +### Automated Goal Definition + +- Provide: + - Duration of the steady-state measurement, + - Static users during measurement, + - Factor applied to measured metrics to derive thresholds (e.g., mean response time 1000 ms × 1.5 → threshold 1500 ms). + + + +Replace with your auto-goal UI screenshot. + +--- + +## Visualize with the Experiment Graph + +The Experiment Graph shows: + +- **Arriving users** and **approximate total requests** over time per scenario. +- Vertical markers when Chaos Toolkit and MiSArch failures occur (sequential timing with configured pauses). +- **Buttons** to switch to **Load** and **Warm-Up** views. + > Note: Request/arrival estimates assume no delays/failures; actual execution may stretch scenarios when the system degrades. Use this graph to + > reason about when failures strike relative to load. + +--- + +## Run, Stop, Save, and Version Experiments + +In the **Header**: + +- Save pending changes. +- Delete the current experiment or version. +- Load an existing experiment or generate a new one. +- Create a new version (makes a copy to tweak safely). +- Execute or Stop the experiment. +- Access General Help and context help for each section. + +--- + +## Analyze Results in Grafana Dashboards + +After each run, a version-specific Grafana dashboard is created/updated. Features include: + +- Run comparison: every metric appears twice—Now vs. N seconds ago—aligning the previous run’s start time to the latest run for apples-to-apples + viewing. +- Sections: + +1. Gatling Total: active users per scenario; responses (success/failed); key stats (min/mean/max latency, throughput, request rate). Goal breaches + are highlighted. +2. Gatling Errors: concrete error breakdowns (timeouts, server errors, failed checks). +3. Gatling Requests: per-request percentiles, throughput (RPS), success ratio—pinpoint bottlenecks. +4. Service Metrics: CPU/memory and internal latencies per MiSArch service (from cAdvisor, Dapr, and service OpenTelemetry), selectable by component. + +--- + +## Tips, Constraints, and Gotchas + +- Work DSL: Only **Gatling Kotlin DSL** for scenarios. +- **Open load only** for experiments (arrivals model). +- **Time unit** for load durations is seconds. +- **Failure persistence:** MiSArch failure settings persist until overwritten by a later failure set. +- **Chaos Toolkit extensions:** Custom extensions not pre-installed on the Executor may be unavailable—prefer common, pre-installed ones. +- **Approximations:** Graphed requests/arrivals are estimates—real runs with delays/failures will deviate. +- **Realistic pattern:** For long runs, realistic load is sampled from a real 24-hour dataset subset and scaled by your configured max arrivals. + +--- + +## Storage Structure & Files + +- Experiments are stored by **Test-UUID** and **Test-Version**. +- Each experiment bundles: + - Work scripts (Kotlin DSL), + - Per-scenario load configuration, + - Chaos Toolkit JSON, + - MiSArch Experiment Config JSON, + - General config JSON (goals, optional warm-up, Test-UUID/version). diff --git a/static/diagrams/comDiagDocker-dark.svg b/static/diagrams/comDiagDocker-dark.svg new file mode 100644 index 0000000..d2871da --- /dev/null +++ b/static/diagrams/comDiagDocker-dark.svg @@ -0,0 +1,4 @@ + + + +
Network Traffic
:GatlingExecutor
:ExperimentExecutor
Frontend
:Experiment Executor
:ExperimentConfig
:ChaosToolkitExecutor
:DaprSidecar
:ExperimentConfig Sidecar
:MiSArchComponent
:OpenTelemetryCollector
:Prometheus
:Grafana
:InfluxDB
:Cadvisor
:DockerSocket
UI
API
Start Experiment
Gatling Metrics
MiSArch WebShop Backend
MiSArch
Experiment
Config
Start Experiment
Metrics
Ingestion API
Container 
Metrics
:DaprPubSub
Events
Query API
Metrics
Ingestion
Prometheus
API
Prometheus API
Quey API
Container
Management
\ No newline at end of file diff --git a/static/diagrams/comDiagDocker-light.svg b/static/diagrams/comDiagDocker-light.svg new file mode 100644 index 0000000..d2871da --- /dev/null +++ b/static/diagrams/comDiagDocker-light.svg @@ -0,0 +1,4 @@ + + + +
Network Traffic
:GatlingExecutor
:ExperimentExecutor
Frontend
:Experiment Executor
:ExperimentConfig
:ChaosToolkitExecutor
:DaprSidecar
:ExperimentConfig Sidecar
:MiSArchComponent
:OpenTelemetryCollector
:Prometheus
:Grafana
:InfluxDB
:Cadvisor
:DockerSocket
UI
API
Start Experiment
Gatling Metrics
MiSArch WebShop Backend
MiSArch
Experiment
Config
Start Experiment
Metrics
Ingestion API
Container 
Metrics
:DaprPubSub
Events
Query API
Metrics
Ingestion
Prometheus
API
Prometheus API
Quey API
Container
Management
\ No newline at end of file diff --git a/static/diagrams/usingMiSArchExperimentBPMN-dark.svg b/static/diagrams/usingMiSArchExperimentBPMN-dark.svg new file mode 100644 index 0000000..0ec5989 --- /dev/null +++ b/static/diagrams/usingMiSArchExperimentBPMN-dark.svg @@ -0,0 +1,4 @@ + + + +
Researcher
Create Experiment
Modify Experiment
Execute Experiment
Inspect Grafana Experiment Dashbaord
Experiment Executor
Generate And Store Experiment
Update And Store Experiment
Run Stored Experiment
Reset MiSArch
Experiment Config
Register MiSArch Experiment Config
Start Experiment
Collect And Transform Gatling Metrics
Create Grafana Dashboard Template
Gatling Executor
Load and Prepare Experiment
no
Warm-Up
Configured?
Execute 
Warm-Up
yes
Steady-State
Collection
Configured?
Execute 
Steady-State
Collection
Register
Gatling
yes
no
Execute 
Gatling Workload
Execute MiSArch Experiment Config
Failure
Execute 
Chaos Toolkit Failure
Experiment
Experiment Modifcation
Chaos Toolkit Executor
Load and Prepare Experiment
Register Chaos Toolkit
Gatling Configuration
Chaos Toolkit Configuration
Gatling Metrics
\ No newline at end of file diff --git a/static/diagrams/usingMiSArchExperimentBPMN-light.svg b/static/diagrams/usingMiSArchExperimentBPMN-light.svg new file mode 100644 index 0000000..0ec5989 --- /dev/null +++ b/static/diagrams/usingMiSArchExperimentBPMN-light.svg @@ -0,0 +1,4 @@ + + + +
Researcher
Create Experiment
Modify Experiment
Execute Experiment
Inspect Grafana Experiment Dashbaord
Experiment Executor
Generate And Store Experiment
Update And Store Experiment
Run Stored Experiment
Reset MiSArch
Experiment Config
Register MiSArch Experiment Config
Start Experiment
Collect And Transform Gatling Metrics
Create Grafana Dashboard Template
Gatling Executor
Load and Prepare Experiment
no
Warm-Up
Configured?
Execute 
Warm-Up
yes
Steady-State
Collection
Configured?
Execute 
Steady-State
Collection
Register
Gatling
yes
no
Execute 
Gatling Workload
Execute MiSArch Experiment Config
Failure
Execute 
Chaos Toolkit Failure
Experiment
Experiment Modifcation
Chaos Toolkit Executor
Load and Prepare Experiment
Register Chaos Toolkit
Gatling Configuration
Chaos Toolkit Configuration
Gatling Metrics
\ No newline at end of file diff --git a/static/images/experiment-executor-frontend-screenshots/Frontend_Configure_Load.png b/static/images/experiment-executor-frontend-screenshots/Frontend_Configure_Load.png new file mode 100644 index 0000000..4db0d5e Binary files /dev/null and b/static/images/experiment-executor-frontend-screenshots/Frontend_Configure_Load.png differ diff --git a/static/images/experiment-executor-frontend-screenshots/Frontend_Entire.png b/static/images/experiment-executor-frontend-screenshots/Frontend_Entire.png new file mode 100644 index 0000000..0ac24a4 Binary files /dev/null and b/static/images/experiment-executor-frontend-screenshots/Frontend_Entire.png differ diff --git a/static/images/experiment-executor-frontend-screenshots/Frontend_select.png b/static/images/experiment-executor-frontend-screenshots/Frontend_select.png new file mode 100644 index 0000000..b6e4ed2 Binary files /dev/null and b/static/images/experiment-executor-frontend-screenshots/Frontend_select.png differ diff --git a/static/images/experiment-executor-frontend-screenshots/Frontend_steady_state.png b/static/images/experiment-executor-frontend-screenshots/Frontend_steady_state.png new file mode 100644 index 0000000..67e08c3 Binary files /dev/null and b/static/images/experiment-executor-frontend-screenshots/Frontend_steady_state.png differ diff --git a/static/images/experiment-executor-frontend-screenshots/Frontend_warmup.png b/static/images/experiment-executor-frontend-screenshots/Frontend_warmup.png new file mode 100644 index 0000000..76bfc3f Binary files /dev/null and b/static/images/experiment-executor-frontend-screenshots/Frontend_warmup.png differ diff --git a/static/images/experiment-executor-frontend-screenshots/load_patterns.svg b/static/images/experiment-executor-frontend-screenshots/load_patterns.svg new file mode 100644 index 0000000..8f82796 --- /dev/null +++ b/static/images/experiment-executor-frontend-screenshots/load_patterns.svg @@ -0,0 +1 @@ +0500100015001214161Arriving Users / sTime (s)Scalability050010001214161Arriving Users / sTime (s)Elasticity01000200030001214161Arriving Users / sTime (s)Resilience02468101214161Arriving Users / sTime (s)Realistic \ No newline at end of file