diff --git a/README.md b/README.md index 094c292..7e45765 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,394 @@ -User documentation generated by claude. -To use -1) enter your claude directory -2) Grab a README.md from an existing wrapper repo -3) Bring in your new wrapper -4) Go to your new wrapper repo -5) Enter claude -6) Tell claue to write a user document for the wrapper, using the README.md pulled in #2 as the template. -7) Review, update/cleanup README.md +# Valkey Benchmark Wrapper + +## Description + +This wrapper facilitates the automated execution of the Valkey benchmark. Valkey is an open-source, in-memory data store forked from Redis, providing high-performance key-value storage with sub-millisecond latency. This benchmark measures throughput (requests per second) across various operations to evaluate server performance. + +The wrapper provides: +- Automated Valkey service management and benchmark execution. +- Customizable connection counts, request volumes, and thread configurations. +- Multi-iteration testing with best-result selection. +- Result collection, processing, and verification. +- CSV and JSON output formats. +- System configuration metadata capture. +- Integration with test_tools framework. +- Optional Performance Co-Pilot (PCP) integration. + +## Command-Line Options + +``` +Valkey Options: + --connections : Number of parallel connections to use during the benchmark. + Defaults to 50 connections. + --threads : Comma-separated list of thread counts to test with (e.g., "2,4,8"). + If not specified, automatically generates intervals from 1 to number of CPUs + (up to 8 intervals). Each thread count is tested separately. + --total_requests : Total number of requests to execute during the benchmark. + Defaults to 400,000 requests. + +General test_tools options: + --home_parent : Parent home directory. If not set, defaults to current working directory. + --host_config : Host configuration name, defaults to current hostname. + --iterations : Number of times to run the test, defaults to 1. + --run_user: User that is actually running the test on the test system. Defaults to current user. + --sys_type: Type of system working with (aws, azure, hostname). Defaults to hostname. + --sysname: Name of the system running, used in determining config files. Defaults to hostname. + --tuned_setting: Used in naming the results directory. For RHEL, defaults to current active tuned profile. + For non-RHEL systems, defaults to 'none'. + --use_pcp: Enable Performance Co-Pilot monitoring during test execution. + --tools_git : Git repo to retrieve the required tools from. + Default: https://github.com/redhat-performance/test_tools-wrappers + --usage: Display this usage message. +``` + +## What the Script Does + +The `valkey.sh` script performs the following workflow: + +1. **Environment Setup**: + - Clones the test_tools-wrappers repository if not present (default: ~/test_tools). + - Sources error codes and general setup utilities. + - Gathers system hardware information. + +2. **Package Installation**: + - Installs required dependencies via package_tool based on valkey.json configuration. + - Dependencies vary by OS variant (RHEL, Ubuntu, SLES, Amazon Linux). + +3. **Thread Configuration**: + - If --threads not specified, automatically determines thread counts: + - For systems with < 8 CPUs: tests with intervals up to CPU count + - For systems with ≥ 8 CPUs: tests with up to 8 intervals + - Generates evenly distributed thread counts for comprehensive testing. + +4. **Test Execution**: + - For each iteration (--iterations count): + - For each thread count: + - Starts Valkey service via systemctl. + - Waits for service to become active. + - Runs valkey-benchmark with specified parameters: + - Host: 127.0.0.1 + - Port: 6379 + - Requests: --total_requests (default 400,000) + - Connections: --connections (default 50) + - Threads: current thread count being tested + - Output: CSV format + - Captures start and end timestamps. + - Stops Valkey service after test completion. + - Optionally records PCP performance data during execution. + +5. **Data Collection**: + - Captures raw output from each iteration/thread combination in CSV files. + - Extracts performance metrics for each test type: + - PING_INLINE: Simple PING command performance + - PING_MBULK: PING command with bulk protocol + - SET: Write performance for SET operations + - GET: Read performance for GET operations + - INCR: Increment operation performance + - LPUSH: List push operation performance + - RPUSH: Right-side list push performance + - LPOP: List pop operation performance + - RPOP: Right-side list pop performance + - SADD: Set add operation performance + - HSET: Hash set operation performance + - SPOP: Set pop operation performance + - ZADD: Sorted set add performance + - ZPOPMIN: Sorted set pop minimum performance + - LPUSH (variadic): Variadic list push performance + - LRANGE_100: Range query with 100 elements + - LRANGE_300: Range query with 300 elements + - LRANGE_500: Range query with 500 elements + - LRANGE_600: Range query with 600 elements + - MSET: Multiple SET operations performance + +6. **Result Processing**: + - For each test type, selects the highest RPS (requests per second) across all thread counts. + - Generates CSV files with configuration and performance data. + - Creates JSON output for verification. + - Validates results against Pydantic schema (results_schema.py). + +7. **Result Aggregation** (Multi-iteration runs): + - When --iterations > 1: + - Collects results from all iterations. + - For each test type, computes average RPS: + - If ≥ 5 iterations: excludes highest and lowest values, averages the rest. + - If < 5 iterations: averages all values. + - Generates combined report in valkey_results.csv. + +8. **Verification**: + - Validates results against Pydantic schema ensuring: + - All test names match expected values. + - All performance values are positive, finite numbers. + - Timestamps are valid datetime objects. + - Uses csv_to_json and verify_results from test_tools. + +9. **Output**: + - Creates timestamped results directory in `${HOME}/export_results/valkey_` + - Saves all raw output files, processed CSV/JSON, and system metadata. + - Optionally saves PCP performance data. + - Archives results to configured storage location. + +## Dependencies + +Location of underlying workload: Valkey (open-source Redis fork) + +**General packages required**: Valkey server and valkey-benchmark tool (specific packages defined in valkey.json) + +To run: +```bash +git clone https://github.com/redhat-performance/valkey-wrapper +cd valkey-wrapper/valkey +./valkey.sh +``` + +The script will automatically detect your CPU configuration and run benchmarks across multiple thread counts. + +## The Valkey Benchmark + +Valkey is an open-source, in-memory data store that provides high-performance key-value storage. The benchmark tests various data structure operations to measure throughput and performance characteristics. + +### Operations Tested + +1. **PING_INLINE**: Simple PING command using inline protocol. + +2. **PING_MBULK**: PING command using bulk/binary protocol. + +3. **SET**: Write performance for string SET operations. + +4. **GET**: Read performance for string GET operations. + +5. **INCR**: Atomic increment operation performance. + +6. **LPUSH**: List data structure - push to left side. + +7. **RPUSH**: List data structure - push to right side. + +8. **LPOP**: List data structure - pop from left side. + +9. **RPOP**: List data structure - pop from right side. + +10. **SADD**: Set data structure - add member. + +11. **HSET**: Hash data structure - set field. + +12. **SPOP**: Set data structure - pop random member. + +13. **ZADD**: Sorted set - add member with score. + +14. **ZPOPMIN**: Sorted set - pop member with minimum score. + +15. **LPUSH (variadic)**: List push with multiple values. + +16. **LRANGE_100**: List range query retrieving 100 elements. + +17. **LRANGE_300**: List range query retrieving 300 elements. + +18. **LRANGE_500**: List range query retrieving 500 elements. + +19. **LRANGE_600**: List range query retrieving 600 elements. + +20. **MSET**: Multiple SET operations in single command. + +### Performance Metrics + +Each operation reports: + +1. **RPS (Requests Per Second)**: Throughput measurement showing how many operations can be completed per second. + - Higher values indicate better performance. + - Measured separately for each thread count configuration. + +2. **Best Thread Performance**: The script selects the highest RPS achieved across all tested thread counts for each operation. + +3. **Timestamps**: Start and end time for each test run, enabling duration calculation and temporal analysis. + +## Output Files + +The results directory contains: + +- **valkey_iter_\*.csv**: CSV files with performance metrics from each iteration +- **valkey_verify.json**: JSON file with validated results data +- **valkey_iter_\*\_\*_threads.csv**: Raw benchmark output for each iteration/thread combination +- **valkey_results.csv**: Aggregated results across all iterations (when --iterations > 1) +- **test_results_report**: Detailed test execution report +- **meta_data\*.yml**: System metadata (CPU info, memory, kernel version) +- **PCP data** (if --use_pcp option used): Performance Co-Pilot monitoring data + +## Examples + +### Basic run with defaults +```bash +./valkey.sh +``` +This runs with: +- 400,000 total requests +- 50 concurrent connections +- Auto-detected thread counts (1 to number of CPUs, up to 8 intervals) +- 1 iteration +- Best result selected for each operation across all thread counts + +### Run with specific thread counts +```bash +./valkey.sh --threads 2,4,8,16 +``` +Tests with exactly 2, 4, 8, and 16 threads, selecting best performance for each operation. + +### Run multiple iterations for consistency +```bash +./valkey.sh --iterations 5 +``` +Runs the complete test suite 5 times and averages the results (excluding highest and lowest values). + +### Run with higher request volume +```bash +./valkey.sh --total_requests 1000000 +``` +Executes 1 million requests for more stable long-running performance measurements. + +### Run with more connections +```bash +./valkey.sh --connections 100 +``` +Tests with 100 parallel connections instead of the default 50. + +### Run with PCP monitoring +```bash +./valkey.sh --use_pcp +``` +Collects Performance Co-Pilot data during the run for detailed performance analysis. + +### Combination example +```bash +./valkey.sh --iterations 3 --threads 4,8,16 --total_requests 500000 --connections 75 --use_pcp +``` +Runs 3 iterations with specific thread counts, 500k requests, 75 connections, and PCP monitoring. + +## How Thread Selection Works + +### Automatic Mode (Default) +When --threads is not specified: +1. Detects CPU count using `nproc`. +2. Determines interval count: + - Systems with < 8 CPUs: interval = CPU count + - Systems with ≥ 8 CPUs: interval = 8 +3. Generates evenly distributed thread counts from 1 to CPU count. +4. Tests each thread count separately. +5. Selects best RPS for each operation across all thread counts. + +### Manual Mode +When --threads is specified: +- Tests only the specified thread counts (comma-separated list). +- Useful for targeted testing or comparing specific configurations. +- Still selects best result across specified thread counts. + +## How Result Selection Works + +For each operation: +1. The operation is tested with each thread count. +2. Each thread count produces an RPS value. +3. The script selects the **highest RPS** across all thread counts. +4. This best-case performance is recorded in the final results. + +This approach ensures: +- Optimal thread count for each operation is automatically found. +- Results represent peak achievable performance. +- Different operations may achieve best performance at different thread counts. + +## How Result Averaging Works + +When running multiple iterations (--iterations > 1): + +1. Each iteration produces a complete set of results for all operations. +2. Raw results are saved in separate valkey_iter_*.csv files. +3. The wrapper collects RPS values from each iteration. +4. Final results calculation: + - **If iterations ≥ 5**: Excludes highest and lowest values, averages the remaining samples. + - **If iterations < 5**: Averages all values. +5. Combined results saved in valkey_results.csv. + +This approach: +- Reduces impact of transient system effects. +- Provides more reliable performance measurements. +- For ≥5 iterations, removes outliers automatically. +- Helps identify result variance across runs. + +## Return Codes + +The script uses standardized error codes from test_tools error_codes: +- **0** (E_SUCCESS): Success +- **101**: Git clone failure +- **E_GENERAL**: General execution errors (service failures, benchmark failures, validation failures) +- **E_USAGE**: Invalid usage/arguments + +Exit codes indicate specific failure points for automated testing workflows. + +## Notes + +### Supported Platforms +- **Linux**: x86_64 and aarch64 architectures +- **OS Support**: RHEL, Ubuntu, SLES, Amazon Linux +- **Tested Distributions**: RHEL 8/9/10, Ubuntu 20.04+, SLES 15+, Amazon Linux 2/2023 + +### Performance Considerations +- Valkey is memory-intensive; ensure sufficient RAM is available. +- Different operations have different performance characteristics: + - **Fastest**: Simple operations (PING, SET, GET, INCR) + - **Moderate**: List and Set operations (LPUSH, SADD, etc.) + - **Slower**: Range queries (LRANGE_*) due to data retrieval overhead +- Thread scaling varies by operation: + - Some operations scale linearly with threads. + - Others may plateau or show diminishing returns beyond certain thread counts. +- Run multiple iterations (--iterations 3 or higher) for reliable results. +- Ensure system is idle during testing for best consistency. + +### Thread Count Tuning +The wrapper's default thread count generation is optimized for: +- Modern multi-core processors (8+ cores) +- Finding optimal thread count per operation automatically +- Comprehensive coverage without excessive test time + +For different requirements: +- **Targeted testing**: Use --threads with specific values +- **Quick validation**: Use --threads 1,$(nproc) to test single-thread and maximum +- **Detailed scaling analysis**: Use --threads with many intermediate values + +### Connection Count Considerations +The default 50 connections provides a balance between: +- Realistic concurrent load simulation +- Avoiding connection overhead domination +- Stable, reproducible results + +Adjust based on your use case: +- **Higher connections** (100-200): Simulate high-concurrency scenarios +- **Lower connections** (10-25): Focus on per-connection efficiency +- Match your production workload characteristics when possible + +### Performance Tips +- Run multiple test iterations (--iterations 3+) to verify consistency +- Ensure system is idle (no other workloads) for best results +- Disable CPU frequency scaling (use performance governor) for reproducible results +- Consider the active tuned profile on RHEL systems +- For production benchmarking, allow system to warm up with a test run first +- PCP monitoring (--use_pcp) adds minimal overhead but provides detailed metrics +- Ensure Valkey service is properly configured before running benchmarks + +### Troubleshooting +- If Valkey service fails to start, check systemctl status and logs +- If benchmark fails, verify valkey-benchmark tool is installed and in PATH +- If results seem inconsistent, run more iterations and check system load during testing +- Use --use_pcp to collect detailed performance counters for analysis +- Check valkey_iter_* files for detailed error messages +- Verify sufficient memory is available (Valkey is in-memory) +- For connection errors, check Valkey configuration (maxclients setting) + +### Comparison with Redis +- **Valkey**: Open-source fork of Redis with community-driven development +- **Redis**: Original project, now with different licensing +- **Compatibility**: Valkey maintains protocol compatibility with Redis +- **Performance**: Generally equivalent for most operations +- Use Valkey for open-source deployments without licensing concerns + +## References + +- Valkey Official Site: https://valkey.io/ +- Valkey GitHub: https://github.com/valkey-io/valkey +- Redis Benchmark Documentation: https://redis.io/docs/management/optimization/benchmarks/ +- test_tools Framework: https://github.com/redhat-performance/test_tools-wrappers diff --git a/valkey/valkey.sh b/valkey/valkey.sh index dfef781..0f075a2 100755 --- a/valkey/valkey.sh +++ b/valkey/valkey.sh @@ -20,7 +20,7 @@ # along with this program; if not, write to the Free Software # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. # -# This script automates the execution of coremark. It will determine the +# This script automates the execution of valkey. It will determine the # set of default run parameters based on the system configuration. # @@ -34,8 +34,7 @@ arguments="$@" script_dir=$(realpath $(dirname $0)) pcpdir="" -#total_requests=600000 -total_requests=100000 +total_requests=400000 connections=50 threads="" @@ -84,7 +83,11 @@ export TOOLS_BIN usage() { echo "Usage $1:" - echo add usage + echo " --connections : Number of parallel connections to use during the benchmark. Defaults to 50 connections." + echo " --threads : Comma-separated list of thread counts to test with (e.g., "2,4,8")." + echo " If not specified, automatically generates intervals from 1 to number of CPUs" + echo " (up to 8 intervals). Each thread count is tested separately." + echo " --total_requests : Total number of requests to execute during the benchmark. Defaults to 400,000 requests." source $TOOLS_BIN/general_setup --usage exit $E_USAGE } @@ -178,19 +181,23 @@ test_list="" execute_valkey() { out_file=valkey_iter_${1} + iter=${1} test_list="" run_threads=$(echo $threads | sed "s/,/ /g") - run_threads=1 for th in $run_threads; do if [[ $to_use_pcp -eq 1 ]]; then start_pcp_subset fi + systemctl start valkey + systemctl wait valkey + systemctl status valkey start_time=$(retrieve_time_stamp) valkey-benchmark -h 127.0.0.1 -p 6379 -n $total_requests -c $connections --threads $th --csv | sed "s/\"//g" | sed "s/ /_/g" | sed "s/)//g" | sed "s/(//g" > ${out_file}_${th}_threads.csv end_time=$(retrieve_time_stamp) + systemctl stop valkey if [[ $test_list == "" ]]; then separ="" echo file "${out_file}_${th}_threads.csv" @@ -226,25 +233,57 @@ execute_valkey() $TOOLS_BIN/test_header_info --front_matter --results_file ${out_file}.csv --host $to_configuration --sys_type $to_sys_type --tuned $to_tuned_setting --results_version $valkey_version --test_name $test_name --field_header "Test,RPS" for tst in $test_list do - value=$(grep "^$tst", *csv | cut -d',' -f2 | sed "s/\"//g" | sort -n | tail -1) + value=$(grep "^$tst", valkey_iter_${iter}_*csv | cut -d',' -f2 | sed "s/\"//g" | sort -n | tail -1) echo ${tst},${value},${start_time},${end_time} >> ${out_file}.csv done - pwd echo ${TOOLS_BIN}/csv_to_json $to_json_flags --csv_file ${out_file}.csv --output_file valkey_verify.json ${TOOLS_BIN}/csv_to_json $to_json_flags --csv_file ${out_file}.csv --output_file valkey_verify.json test_rtc=$? if [[ $test_rtc -ne 0 ]]; then - exit_out "${TOOLS_BIN}/csv_to_json $to_json_flags --csv_file ${out_file}.csv --output_file valkey_verify.json returned an an error" $test_rtc + exit_out "${TOOLS_BIN}/csv_to_json $to_json_flags --csv_file ${out_file}.csv --output_file valkey_verify.json returned an error" $test_rtc fi ${TOOLS_BIN}/verify_results $to_verify_flags --schema_file $script_dir/results_schema.py --class_name valkey_Results --file valkey_verify.json test_rtc=$? if [[ $test_rtc -ne 0 ]]; then - echo Test failure detected: $out_File + echo Test failure detected: $out_file fi } +generate_combine_report() +{ + rm -f $results_file + for tst in $test_list + do + high=0 + low="" + sum=0 + for iter in $(seq 1 1 $to_times_to_run); do + file="valkey_iter_${iter}.csv" + rval=$(grep ^${tst}, $file | cut -d, -f2 |cut -d. -f1) + if [[ -z $rval ]]; then + echo Error in parsing alkey_iter_${iter}.csv + continue + fi + let "sum=${sum}+${rval}" + if [[ $rval -gt $high ]]; then + high=$rval + fi + if [[ $low == "" ]] || [[ $rval -lt $low ]]; then + low=$rval + fi + done + if [[ $to_times_to_run -gt 4 ]]; then + let "samples=${to_times_to_run}-2" + let "sum=${sum}-${high}-${low}" + else + samples=${to_times_to_run} + fi + avg=$(echo $sum/$samples | bc) + echo ${tst},$avg >> $results_file + done +} + ARGUMENT_LIST=( - "commit" "connections" "threads" "total_requests" @@ -267,10 +306,6 @@ eval set --$opts while [[ $# -gt 0 ]]; do case "$1" in - --commit) - commit=$2 - shift 2 - ;; --connections) connections=$2 shift 2 @@ -309,6 +344,7 @@ if [[ $threads == "" ]]; then threads=$(${TOOLS_BIN}/generate_intervals --interval $intervals --max_value $cpus) fi +chmod 755 /etc/rc.d/rc.local package_tool --no_packages $to_no_pkg_install --wrapper_config $curdir/../valkey.json # Get PCP setup if we're using it @@ -327,19 +363,11 @@ rm -rf *csv for iter in $(seq 1 1 $to_times_to_run); do execute_valkey $iter done +generate_combine_report # Shutdown PCP and clean up after ourselves if [[ $to_use_pcp -eq 1 ]]; then shutdown_pcp fi -#$TOOLS_BIN/csv_to_json $to_json_flags --csv_file ${results_file} --output_file results_.json -#$TOOLS_BIN/verify_results $to_verify_flags --schema_file $script_dir/../result_schema.py --class_name --file results_.json - - -#${TOOLS_BIN}/save_results --curdir $curdir --home_root $to_home_root - -#example - -#${TOOLS_BIN}/save_results --curdir $curdir --home_root $to_home_root --other_files "*_summary,run*log,test_results_report,${pcpdir}" --results $results_file --test_name coremark --tuned_setting=$to_tuned_setting --version $_version --user $to_user -#================================================= +${TOOLS_BIN}/save_results --curdir $curdir --home_root $to_home_root --other_files "*_summary,run*log,test_results_report,${pcpdir},*csv" --results $results_file --test_name valkey --tuned_setting=$to_tuned_setting --version $valkey_version --user $to_user exit $E_SUCCESS diff --git a/valkey/valkey_verify.json b/valkey/valkey_verify.json deleted file mode 100644 index a4f3082..0000000 --- a/valkey/valkey_verify.json +++ /dev/null @@ -1,128 +0,0 @@ -[ - { - "Test":"XADD", - "RPS":54495.91, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"MSET_10_keys", - "RPS":56561.09, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LRANGE_600_first_600_elements", - "RPS":8062.57, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LRANGE_500_first_500_elements", - "RPS":9525.62, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LRANGE_300_first_300_elements", - "RPS":13629.55, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LRANGE_100_first_100_elements", - "RPS":29368.58, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LPUSH_needed_to_benchmark_LRANGE", - "RPS":54674.69, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"ZPOPMIN", - "RPS":53390.28, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"ZADD", - "RPS":53937.43, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"SPOP", - "RPS":53304.9, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"HSET", - "RPS":53908.36, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"SADD", - "RPS":54318.3, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"RPOP", - "RPS":53705.69, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LPOP", - "RPS":53333.33, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"RPUSH", - "RPS":54259.36, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"LPUSH", - "RPS":52910.05, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"INCR", - "RPS":53219.8, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"GET", - "RPS":52301.26, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"SET", - "RPS":45269.35, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"PING_MBULK", - "RPS":53078.56, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - }, - { - "Test":"PING_INLINE", - "RPS":53163.21, - "Start_Date":"2026-06-05T21:44:50Z", - "End_Date":"2026-06-05T21:45:56Z" - } -] \ No newline at end of file