Skip to content

PERFORMANCE_ANALYSIS.md

Latest commit

 

History

History
281 lines (207 loc) · 7.14 KB

PERFORMANCE_ANALYSIS.md

File metadata and controls

281 lines (207 loc) · 7.14 KB

Wikipedia API Semantic Search - Performance Analysis & Optimization

Executive Summary

Current Performance: ~280-370ms end-to-end semantic search Primary Bottleneck: Ollama Phi embedding generation (~240-260ms, 80% of total time) Status: Optimized with connection pooling and HTTP session reuse

Performance Breakdown

Before Optimization

Total time: ~360ms
├── Embedding generation: ~290ms (80%)
└── Vector search: ~70ms (20%)

After Optimization

Total time: ~280-370ms
├── Embedding generation: ~240-260ms (70%)
└── Vector search: ~45-120ms (30%)

Improvement: ~20-80ms faster (5-22% improvement)

Bottleneck Analysis

1. Embedding Generation (240-260ms) ⚠️ PRIMARY BOTTLENECK

Current Setup:

  • Model: Ollama Phi (2560 dimensions)
  • Hardware: CPU in CT 102 (4GB RAM)
  • Model size: ~1.5GB (Q4_0 quantized)

Why It's Slow:

  • CPU inference (no GPU acceleration in container)
  • Model loading/warmup overhead
  • 2560-dimensional output (large embedding size)

Optimization Applied:

  • ✅ Connection pooling (HTTPAdapter)
  • ✅ HTTP session reuse (keep-alive)
  • ✅ Model warmup on startup
  • ✅ Reduced timeout (10s → 8s)

Further Optimizations Possible:

  1. GPU Acceleration (Biggest impact: ~10x faster)

    • Move Ollama to host with GPU access
    • Use GPU-enabled container
    • Expected: 240ms → 20-40ms

  2. Smaller Embedding Model

    • Use nomic-embed-text (768 dim) instead of Phi (2560 dim)
    • Expected: 240ms → 80-120ms
    • Tradeoff: Lower semantic accuracy

  3. Batch Processing

    • Process multiple queries in parallel
    • Use Ollama batch API
    • Expected: 20-30% improvement for concurrent requests

  4. Model Caching/Quantization

    • Already using Q4_0 (4-bit quantization)
    • Could try Q2 for faster inference (lower quality)

2. Vector Search (45-120ms) ⚠️ SECONDARY BOTTLENECK

Current Setup:

  • Service: CT 106 (FAISS HNSW index)
  • Index size: 4M+ vectors, 2560 dimensions
  • Network: HTTP request to 192.168.1.70:8080

Why Variance Exists:

  • Network latency (container-to-container)
  • FAISS HNSW index complexity
  • Database query overhead

Optimization Applied:

  • ✅ Connection pooling for vector search requests
  • ✅ Reduced timeout (5s → 3s)
  • ✅ HTTP keep-alive headers

Further Optimizations Possible:

  1. Local FAISS Index (Biggest impact)

    • Load FAISS index directly in CT 102
    • Eliminate network overhead
    • Expected: 45-120ms → 10-30ms
    • Tradeoff: 12GB+ RAM needed

  2. Quantized Index

    • Use FAISS IVF (Product Quantization)
    • Smaller memory footprint
    • Expected: 20-40% faster search

  3. Index Sharding

    • Split index by category/topic
    • Parallel search across shards
    • Expected: 30-50% improvement

Optimization Techniques Implemented

HTTP Connection Pooling

# Before: New connection per request
response = requests.post(url, json=data)

# After: Reusable session with pooling
self.session = requests.Session()
adapter = HTTPAdapter(
    pool_connections=10,
    pool_maxsize=20,
    max_retries=retry_strategy
)
self.session.mount("http://", adapter)
response = self.session.post(url, json=data, headers={"Connection": "keep-alive"})

Benefit: Saves ~10-20ms on TCP handshake per request

Model Warmup

def _warm_up_model(self):
    """Avoid cold start on first request"""
    self.session.post(
        f"{self.ollama_host}/api/embeddings",
        json={"model": self.model, "prompt": "warmup"}
    )

Benefit: First request faster by ~50-100ms

Detailed Timing

results['timing'] = {
    'embedding_ms': round(embed_time, 2),
    'vector_search_ms': round(search_time, 2),
    'total_ms': round(total_time, 2)
}

Benefit: Precise performance monitoring

Recommended Next Steps (By Impact)

High Impact (10x improvement potential)

  1. GPU Acceleration for Ollama

    • Move Ollama to host Giratina (has 2x Tesla T4)
    • Configure container with GPU passthrough
    • Expected reduction: 240ms → 20-40ms
    • Total time: ~65-160ms (4-5x faster)

  2. Load FAISS Index Locally

    • Copy 12GB index into CT 102
    • Eliminate network hop
    • Expected reduction: 45-120ms → 10-30ms
    • Total time: ~250-290ms (current) → ~30-70ms

Medium Impact (2-3x improvement)

  1. Switch to Faster Embedding Model

    • Use nomic-embed-text (768 dim) or all-MiniLM (384 dim)
    • Reindex if necessary
    • Expected reduction: 240ms → 60-100ms
    • Total time: ~105-220ms

  2. FAISS Index Optimization

    • Use IVF with Product Quantization
    • Tune HNSW parameters (ef_search)
    • Expected reduction: 20-40%

Low Impact (10-20% improvement)

  1. Query Caching

    • Cache frequent queries
    • LRU cache for embeddings
    • Benefits repeat queries only

  2. Async Request Processing

    • Parallel embedding + preload FAISS
    • Overlap I/O operations
    • Expected: 10-15% improvement

Performance Benchmarks

Test Queries

Query Embedding (ms) Vector Search (ms) Total (ms)
"quantum physics" 238.72 44.66 283.46
"machine learning" 245.30 68.96 314.38
"artificial intelligence" 258.04 115.99 374.16

Average: ~290ms total (~250ms embedding + ~77ms search)

Comparison to Keyword Search

Search Type Avg Time Difference
Keyword (FTS) ~10-20ms Baseline
Semantic ~290ms 15-30x slower

Tradeoff: Semantic search provides much better relevance at cost of latency

Hardware Constraints

Current: CT 102 (Wikipedia API)

  • RAM: 4GB
  • CPU: Shared host cores
  • GPU: None (no passthrough)
  • Storage: ZFS bind mounts

Limitation: CPU-only inference is slow for LLM embeddings

Available: Host Giratina

  • RAM: 314GB
  • GPU: 2x NVIDIA Tesla T4 (16GB each, 30GB total)
  • Status: GPUs idle (0% utilization)

Opportunity: Could run Ollama on GPU for 10x speedup

Implementation Priorities

Immediate (Already Done)

  • ✅ Connection pooling
  • ✅ HTTP session reuse
  • ✅ Model warmup
  • ✅ Performance timing

Short Term (High ROI)

  1. Move Ollama to GPU - 10x faster embeddings
  2. Load FAISS locally - 3-5x faster vector search

Long Term (Requires Re-indexing)

  1. Use lighter embedding model (nomic-embed-text)
  2. Optimize FAISS index parameters
  3. Implement query caching layer

Code Files

  • Original: embedding_service.py (~290ms)
  • Optimized: embedding_service_optimized.py (~280ms)
  • Main API: wikipedia_api_lightning.py (uses optimized)

Conclusion

Current State: Semantic search works well but is bottlenecked by CPU-only embedding generation (~240ms).

Quick Win: Move Ollama to host GPU → Expected total time: 60-160ms (2-5x faster)

Ultimate Goal: GPU embeddings + local FAISS → Expected total time: 30-70ms (4-10x faster)

This would make semantic search competitive with keyword search while providing superior relevance.

Analysis Date: December 9, 2025 Current Performance: ~290ms average Target Performance: <100ms with GPU