Skip to content

feat(bench): add map-reduce chunking to NIAH benchmark#865

Merged
starpit merged 1 commit into
IBM:mainfrom
starpit:feat/niah-map-reduce-chunking
Feb 17, 2026
Merged

feat(bench): add map-reduce chunking to NIAH benchmark#865
starpit merged 1 commit into
IBM:mainfrom
starpit:feat/niah-map-reduce-chunking

Conversation

@starpit
Copy link
Copy Markdown
Member

@starpit starpit commented Feb 17, 2026

Summary

Implement chunk-based map-reduce structure in niah.rs benchmark, matching the pattern from haystack.rs. This allows testing retrieval performance with different chunking strategies.

Changes

  • Add chunk parameter to run_niah_test() function
  • Implement map-reduce logic: split context into chunks, process each chunk separately (map), then combine results (reduce)
  • Add BENCH_CHUNK_SIZES environment variable (default: 0,2,4)
  • Always include chunk size in benchmark IDs for proper filtering
  • Update documentation with chunking examples and filtering guide

Usage Examples

# Run with defaults: chunk=0, chunk=2, and chunk=4
cargo bench --bench niah --features tok

# Filter to run only chunk=0 (non-chunked) benchmarks
cargo bench --bench niah --features tok -- "chunk=0"

# Filter to run only chunk=2 benchmarks
cargo bench --bench niah --features tok -- "chunk=2"

# Run only chunked benchmarks (chunk > 0)
cargo bench --bench niah --features tok -- "chunk=(2|4)"

Made with Bob

@starpit starpit added the made with bob PR created with assistance from Bob AI label Feb 17, 2026
Implement chunk-based map-reduce structure in niah.rs benchmark, matching
the pattern from haystack.rs. This allows testing retrieval performance
with different chunking strategies.

Changes:
- Add chunk parameter to run_niah_test() function
- Implement map-reduce logic: split context into chunks, process each
  chunk separately (map), then combine results (reduce)
- Add BENCH_CHUNK_SIZES environment variable (default: 0,2,4)
- Always include chunk size in benchmark IDs for proper filtering
- Update documentation with chunking examples and filtering guide

Made with Bob

Signed-off-by: Nick Mitchell <nickm@us.ibm.com>
@starpit starpit force-pushed the feat/niah-map-reduce-chunking branch from cef4583 to e445b34 Compare February 17, 2026 15:36
@starpit starpit merged commit c76e8d0 into IBM:main Feb 17, 2026
36 checks passed
@starpit starpit deleted the feat/niah-map-reduce-chunking branch February 17, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

made with bob PR created with assistance from Bob AI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant