You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -83,30 +102,155 @@ On top of the generic CLI, CodSpeed provides first-class integrations for multip
83
102
84
103
Need to bench another language or framework? Open [an issue](https://github.com/CodSpeedHQ/codspeed/issues) or let us know on [Discord](https://codspeed.io/discord)!
85
104
86
-
## Advanced usage
105
+
### CLI Harness: `codspeed.yml` configuration
106
+
107
+
The CLI also offers a built-in harness that allows you to define benchmarks directly.
108
+
109
+
You can define multiple `codspeed exec` benchmark targets and configure options in a `codspeed.yml` file.
110
+
This is useful when you want to benchmark several commands with different configurations.
111
+
112
+
Create a `codspeed.yml` file in your project root:
113
+
114
+
```yaml
115
+
# Global options applied to all benchmarks
116
+
options:
117
+
warmup-time: "0.2s"
118
+
max-time: 1s
119
+
120
+
# List of benchmarks to run
121
+
benchmarks:
122
+
- name: "Fast operation"
123
+
exec: ./my_binary --mode fast
124
+
options:
125
+
max-rounds: 20
126
+
127
+
- name: "Slow operation"
128
+
exec: ./my_binary --mode slow
129
+
options:
130
+
max-time: 200ms
131
+
132
+
- name: "Script benchmark"
133
+
exec: python scripts/benchmark.py
134
+
```
135
+
136
+
Then run all benchmarks with:
137
+
138
+
```bash
139
+
codspeed run --mode walltime
140
+
```
141
+
142
+
> [!TIP]
143
+
> For more details on configuration options, see the [CLI documentation](https://codspeed.io/docs/cli).
87
144
88
-
### Installing tools before running
145
+
##Performance Instruments
89
146
90
-
You can install executors and instruments before running the benchmark with the `setup` command:
147
+
CodSpeed provides multiple instruments to measure different aspects of your code's performance. Choose the one that best fits your use case:
148
+
149
+
### CPU Simulation
150
+
151
+
Simulates CPU behavior for **<1% variance** regardless of system load. Hardware-agnostic measurements with automatic flame graphs.
152
+
153
+
**Best for:** CPU-intensive code, CI regression detection, cross-platform comparison
91
154
92
155
```bash
93
-
codspeed setup
156
+
codspeed exec --mode simulation -- ./my-binary
94
157
```
95
158
96
-
This is especially useful when configuring environments with tools such as docker.
By default, the runner will run the benchmark in the `simulation` mode. You can specify the mode with the `--mode` flag of the `run` command:
165
+
**Supported:** Rust, C/C++ with libc, jemalloc, mimalloc
101
166
102
167
```bash
103
-
# Run in the `simulation` mode
104
-
codspeed run --mode simulation <my-benchmark-command>
168
+
codspeed exec --mode memory -- ./my-binary
169
+
```
170
+
171
+
### Walltime
172
+
173
+
Measures real-world execution time including I/O, system calls, and multi-threading effects.
105
174
106
-
# Run in the `walltime` mode
107
-
codspeed run --mode walltime <my-benchmark-command>
175
+
**Best for:** API tests, I/O-heavy workloads, multi-threaded applications
176
+
177
+
```bash
178
+
codspeed exec --mode walltime -- ./my-api-test
108
179
```
109
180
110
181
> [!WARNING]
111
182
> Using the `walltime` mode on traditional VMs/Hosted Runners will lead to inconsistent data. For the best results, we recommend using CodSpeed Hosted Macro Runners, which are fine-tuned for performance measurement consistency.
112
183
> Check out the [Walltime Instrument Documentation](https://docs.codspeed.io/instruments/walltime/) for more details.
184
+
185
+
> [!TIP]
186
+
> For detailed information on each instrument, see the [Instruments documentation](https://codspeed.io/docs/instruments).
187
+
188
+
## Usage In CI environments
189
+
190
+
Running CodSpeed in CI allows you to automatically detect performance regressions on every pull request and track performance evolution over time.
191
+
192
+
### GitHub Actions
193
+
194
+
We recommend using our official GitHub Action: [@CodSpeedHQ/action](https://github.com/CodSpeedHQ/action).
195
+
196
+
Here is a sample `.github/workflows/codspeed.yml` workflow for Python:
197
+
198
+
```yaml
199
+
name: CodSpeed Benchmarks
200
+
201
+
on:
202
+
push:
203
+
branches:
204
+
- "main"# or "master"
205
+
pull_request:
206
+
workflow_dispatch:
207
+
208
+
permissions:
209
+
contents: read
210
+
id-token: write
211
+
212
+
jobs:
213
+
benchmarks:
214
+
runs-on: ubuntu-latest
215
+
steps:
216
+
- uses: actions/checkout@v4
217
+
218
+
# Set up your language/environment here
219
+
# For Python:
220
+
- uses: actions/setup-python@v5
221
+
with:
222
+
python-version: "3.12"
223
+
- run: pip install -r requirements.txt
224
+
225
+
# Run benchmarks with CodSpeed
226
+
- uses: CodSpeedHQ/action@v4
227
+
with:
228
+
mode: instrumentation
229
+
run: pytest tests/ --codspeed
230
+
```
231
+
232
+
### GitLab CI
233
+
234
+
Here is a sample `.gitlab-ci.yml` configuration for Python:
- codspeed run --mode simulation -- pytest tests/ --codspeed
253
+
```
254
+
255
+
> [!TIP]
256
+
> For more CI integration examples and advanced configurations, check out the [CI Integration Documentation](https://codspeed.io/docs/integrations/ci/).
0 commit comments