Evaluation Report: Code Search Improvements and CodeRankEmbed Analysis

[tomohiro-owada]

Summary

We investigated several approaches to improve source code search in DevRag, including:

• Code indexing with Tree-sitter AST parsing
• Dictionary feature for Japanese→English translation
• CodeRankEmbed (code-specialized embedding model) integration

Conclusion: Claude Code's standard tools (Grep/Glob) are sufficiently powerful for most code search tasks. DevRag should be positioned as a complementary tool for semantic/conceptual search rather than a replacement.

---

Key Findings

1. DevRag vs Grep/Glob Comparison

Test Case	DevRag	Grep	Winner
Find vector search function	Irrelevant results	Found Search(), SearchWithFilter()	Grep
Find error logging code	Irrelevant results	Found 10 entries	Grep
Find callers of function	0 results (before fix)	Found 5 locations	Grep

2. CodeRankEmbed Evaluation

Aspect	CodeRankEmbed	e5-small (current)
Dimensions	768	384
Model Size	522MB	~90MB
English code search	△ On par with e5-small
Conceptual understanding	△ e5-small is better
Japanese support	✗ Not supported	○ Supported

Result: No significant advantage over current e5-small model.

3. Implemented Improvements

• ✅ Code indexing with Tree-sitter (Go, Python, TypeScript, JavaScript, PHP, Rust, Vue)
• ✅ Dictionary feature for Japanese→English word mapping
• ✅ Fixed relation search bug (target_chunk_id NULL issue)
• ❌ CodeRankEmbed integration (postponed - no clear benefit)

---

Recommendations

1. Keep e5-small - Multilingual support, lightweight, decent conceptual understanding
2. Use DevRag for:
   • Conceptual search when keywords are unknown
   • Japanese document search
   • Exploratory search across large document sets
3. Use Grep/Glob for:
   • When keywords are clear
   • Finding specific call sites
   • Pattern-based search

---

Full Report

See the detailed evaluation reports in the docs/ directory:

• English Report: docs/evaluation-report-2026-01-13-en.md
• Japanese Report: docs/evaluation-report-2026-01-13.md

---

Related Commit

commit 7af5f6f
Add code indexing with AST parsing and dictionary feature

- Add Tree-sitter based code parsing for Go, Python, TypeScript, JavaScript, PHP, Rust, Vue
- Extract code relations (calls, imports, inherits) for knowledge graph
- Add dictionary feature for Japanese→English word mapping
- Auto-build dictionary when search has no translations
- Fix relation search query to properly find incoming/outgoing relations
- Add search_relations and build_dictionary MCP tools