🔗 Live results page → https://tanmay0401.github.io/code-mixed-search-benchmark/
A reproducible benchmark for evaluating product-search retrieval on Hinglish
and code-mixed shopping queries (e.g. lal running shoes, gym ke liye shoes),
mapped onto Amazon's ESCI relevance judgments.
One-line pitch: A BEIR-style, zero-shot IR benchmark for the underserved setting of code-mixed e-commerce search, comparing BM25, multilingual dense retrieval, and LLM reranking with nDCG and MRR, plus a structured failure analysis.
Keyword search sees zero overlap between lal and red, so it fails on Hinglish.
Over a billion people search in code-mixed language, yet nearly all public IR
benchmarks are monolingual English. This project quantifies how much semantic
and LLM methods recover — and where they still break.
| Phase | Deliverable | Status |
|---|---|---|
| 1 | Problem understanding | ✅ |
| 2 | Literature review | ✅ |
| 3 | Architecture & planning | ✅ |
| 4 | Benchmark construction (348 queries) | ✅ |
| 5 | Amazon ESCI integration (qrels + corpus) | ✅ |
| 6 | BM25 baseline (from scratch) | ✅ |
| 7 | Dense retrieval (multilingual e5 + FAISS) | ✅ |
| 8 | Reranking (cross-encoder + LLM pipeline) | ✅ |
| 9 | Evaluation (nDCG, MRR from scratch) | ✅ |
| 10 | Failure analysis (12 modes, per-category) | ✅ |
| 11 | Engineering standards (pyproject, license, pins, 33 tests) | ✅ |
| 12 | Documentation (technical report + dataset card) | ✅ |
| 13–14 | Teaching notes + interview prep | ✅ |
Read the writeups: reports/technical_report.md ·
reports/failure_analysis.md ·
reports/interview_prep.md ·
DATASET.md
- 348 hand-authored queries across 10 domains and all 13 challenge categories.
- Languages: English, Devanagari Hindi, Roman Hindi, and code-mixed.
- Each query carries an English gloss — the bridge used to map onto ESCI (Phase 5).
- Source of truth:
src/data/query_pool.py; schema:config/queries_schema.yaml.
- Each query's English gloss is mapped onto the large US ESCI query pool
(1.82M judgments / 97,345 queries): exact + fuzzy (rapidfuzz
token_sort_ratio) matching first, then a multilingual-embedding (E5) semantic fallback for anything the string matcher misses. - All 348 queries (100%) map to
qrels.jsonl, each tagged with a confidence tier (match_type): exact 106 · fuzzy 81 · manual 6 · semantic_high 117 · semantic_med 36 · semantic_low 2. The 2semantic_loware best-effort maps for items essentially absent from the US catalog — flagged, not hidden. Filter bymatch_typeto evaluate on a high-confidence subset. corpus.jsonl= the 6,991 products judged for the mapped queries (the searchable set). ESCI labels E/S/C/I → gains 3/2/1/0 for nDCG.
Environment: use Python 3.11 via conda. Python 3.14 has no prebuilt wheels for the ML stack (pyarrow/faiss/torch) and fails to compile from source.
# 1) Create the environment (once)
conda create -y -n esci python=3.11
conda activate esci
# 2) Install dependencies
pip install -r requirements.txt
# 3) Prepare data: download Amazon ESCI + build corpus/qrels (Phase 5)
python scripts/01_prepare_data.py
# 4) Build & validate the query benchmark (writes data/benchmark/queries.jsonl)
python scripts/02_build_benchmark.py
# 5) Run the tests that prove the resume claims (>=300 queries, >=10 categories, ...)
python -m pytest tests/ -vNot activating conda? Call the env's Python directly, e.g.
D:\ANACONDA\envs\esci\python.exe scripts\02_build_benchmark.py.
| Method | Overall | English | Hinglish/Hindi |
|---|---|---|---|
| BM25 | 0.468 | 0.549 | 0.288 |
| BM25 + cross-encoder | 0.496 | 0.584 | 0.297 |
| Dense (e5-small) | 0.473 | 0.612 | 0.161 |
| Dense + cross-encoder | 0.477 | 0.606 | 0.186 |
Key finding: on code-mixed Hinglish queries, lexical BM25 (+reranking) beats
multilingual dense retrieval — the opposite of the naive expectation. Dense wins on
clean English (0.612) but collapses on Hinglish (0.161) due to romanised-Hindi drift.
Every method drops ~0.26–0.45 nDCG from English to Hinglish: the vernacular search gap,
quantified. (Metrics implemented from scratch in src/evaluation/metrics.py.)
Live results & charts → https://tanmay0401.github.io/code-mixed-search-benchmark/
config/ settings (config.yaml) + query taxonomy (queries_schema.yaml)
data/ raw ESCI (Phase 5), processed corpus/qrels, authored queries
src/ importable library: data, retrieval, reranking, evaluation, analysis
scripts/ thin numbered CLI entry points (the reproducible pipeline)
tests/ unit/integration tests
results/ generated runs, metrics, figures (regenerable, git-ignored)
reports/ technical report
Single random seed in config/config.yaml, pinned requirements.txt,
read-only data/raw/, and a fixed sequence of numbered scripts. Same inputs +
same commands → identical outputs.