ASV Benchmarks

[Avinash-Raj]

ASV (Air Speed Velocity) is a benchmarking tool which helps us to find performance regressions in queries across a range of Git commits.

To run TPC-H benchmarks against a range of recent commits on your Velox fork:

Checkout to the target branch (e.g.,BM-techpreview or merged-prs) on your local Velox repository.

Run the benchmark script with the --commits argument, specifying the range of commits you wish to compare:

cd velox/scripts
./run_asv_commit_range.sh --commits HEAD~3..HEAD 2>&1 | tee asv_commit_range.log

This command will perform a Git checkout for each commit within the specified range (HEAD3, HEAD2, HEAD~1, and HEAD) and run the TPC-H benchmarks for each one.

The output is the comparison of performance metrics (like query latency) between the commits, helping you identify which specific commit introduced a performance change (regression or improvement).

Sample

[Copilot]

Pull Request Overview

This PR introduces an ASV (Airspeed Velocity) benchmarking framework for tracking Velox TPC-H query performance over time, with comprehensive tooling for single-commit and commit-range benchmarking.

Key Changes:

• Added ASV benchmark infrastructure with Docker-based execution environment
• Implemented automated commit-range benchmarking with two modes (range/endpoints)
• Created Bash parameter expansion compatibility fix for older Bash versions

Reviewed Changes

Copilot reviewed 17 out of 20 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
velox/scripts/build_velox.sh	Fixed Bash 4.0+ syntax for case-insensitive comparison to work with older Bash versions
velox/scripts/run_asv_benchmarks.sh	Script for benchmarking single commits with configurable options and environment variables
velox/scripts/run_asv_commit_range.sh	Automated multi-commit benchmarking with Git operations, Docker builds, and cleanup
velox/scripts/build_asv_image.sh	Docker image builder for ASV benchmarks with dependency checking
velox/scripts/apply_velox_patches.sh	Idempotent patch application for TPC-H Python bindings
velox/scripts/run_asv_server.sh	Preview server launcher for existing benchmark results
velox/scripts/clear_asv_results.sh	Permission-aware results cleanup utility
velox/docker/docker-compose.adapters.benchmark.yml	Docker Compose configuration for ASV benchmark container
velox/docker/adapters_build.dockerfile	Added simdjson as bundled dependency
velox/asv_benchmarks/benchmarks/tpch_benchmarks.py	ASV benchmark definitions for all 22 TPC-H queries
velox/asv_benchmarks/asv.conf.json	ASV configuration with virtualenv setup and custom build commands
velox/asv_benchmarks/entrypoint.sh	Container entrypoint handling ASV execution, machine config, and preview modes
velox/asv_benchmarks/Dockerfile	Multi-stage build for ASV image with Cython bindings
velox/asv_benchmarks/machine.json	Default machine metadata template
velox/asv_benchmarks/README.md	Comprehensive documentation covering all workflows and troubleshooting
velox/asv_benchmarks/.dockerignore	Docker build context exclusions
velox/patches/01-modified-files.patch	Velox patches for TPC-H Python bindings support

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[simoneves]

@Avinash-Raj can you freshen this one up from main, please, to flatten out (e.g.) the simdjson change.

Not familiar with ASV so have some background reading to do.