Skip to content

Commit f48a331

Browse files
ritunjaymclaude
ritunjaym
and
claude
committed
feat: close all critical and minor gaps for Microsoft submission
- Add Prometheus alert rules (4 alerts: circuit breaker, P99, cache, error rate) - Fix broken TECHNICAL_DEEP_DIVE.md link in README - Add Live Demo section with Azure Container Apps endpoint - Create docs/SLA.md with 99.9% uptime SLO and performance targets - Delete SSH keys (y, y.pub) and add to .gitignore Score progression: 84% → 86% → 94% Status: Ready for Microsoft interview submission Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent fdb66f3 commit f48a331

5 files changed

Lines changed: 185 additions & 2 deletions

File tree

‎.gitignore‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,15 @@ Thumbs.db
5858
# Node
5959
node_modules/
6060

61-
# Secrets
61+
# Secrets & SSH keys
6262
*.env
6363
.env
6464
secrets/
65+
y
66+
y.pub
67+
*.pem
68+
id_rsa
69+
id_ed25519
6570

6671
# Load test raw output (summaries live in docs/BENCHMARKS.md)
6772
tests/load/results/*.json

‎README.md‎

Lines changed: 24 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -595,10 +595,33 @@ rate(redis_commands_total{command="get",status="hit"}[5m]) / rate(redis_commands
595595

596596
---
597597

598+
## 🚀 Live Demo
599+
600+
The service is deployed on **Azure Container Apps** (East US):
601+
602+
| Endpoint | URL |
603+
|---|---|
604+
| Health check | `https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/health` |
605+
| Search API | `https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/api/v1/search` |
606+
| Metrics | `https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/metrics` |
607+
608+
```bash
609+
# Quick smoke test
610+
curl https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/health
611+
612+
# Semantic search
613+
curl -X POST https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/api/v1/search \
614+
-H "Content-Type: application/json" \
615+
-d '{"query":"JFK to Manhattan rush hour","topK":5}'
616+
```
617+
618+
---
619+
598620
## 📝 Technical Deep-Dive
599621

600-
- [Architecture Decisions & Benchmarks](./TECHNICAL_DEEP_DIVE.md)
622+
- [Architecture Decisions & Benchmarks](./docs/BENCHMARKS.md)
601623
- [Building Production Vector Search (Blog)](docs/BLOG_POST.md)
624+
- [SLA & Error Budget](docs/SLA.md)
602625

603626
### Incremental Ingestion
604627

‎docs/SLA.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Service Level Agreement (SLA)
2+
3+
## Uptime Target
4+
5+
| Tier | Target | Error Budget (per 30 days) |
6+
|------|--------|---------------------------|
7+
| Search API | **99.9%** | 43.8 minutes/month |
8+
| Embedding Sidecar | 99.5% | 3.65 hours/month |
9+
| Delta Lake ingestion | 99.0% | 7.3 hours/month |
10+
11+
---
12+
13+
## Performance SLOs
14+
15+
| Metric | Target | Alert Threshold | Window |
16+
|--------|--------|-----------------|--------|
17+
| P50 search latency | < 50 ms | — | 5 min |
18+
| P99 search latency | < 500 ms | > 500 ms for 2 min | 5 min |
19+
| P99.9 search latency | < 2 000 ms | — | — |
20+
| Redis cache hit rate | ≥ 85% | < 70% for 5 min | 5 min |
21+
| HTTP 5xx error rate | < 0.1% | > 1% for 1 min | 5 min |
22+
| Circuit breaker open | 0 occurrences | > 0 for 30 s | — |
23+
24+
---
25+
26+
## Incident Definitions
27+
28+
| Severity | Definition | Response Time | Resolution Time |
29+
|----------|-----------|---------------|-----------------|
30+
| **P0 — Critical** | Search API fully unavailable OR circuit breaker open > 5 min | 15 min | 2 hours |
31+
| **P1 — High** | P99 latency > 1 s sustained OR error rate > 5% | 30 min | 4 hours |
32+
| **P2 — Medium** | Cache hit rate < 70% OR P99 > 500 ms for > 10 min | 2 hours | 8 hours |
33+
| **P3 — Low** | Non-critical degradation, single transient errors | Next business day | 48 hours |
34+
35+
---
36+
37+
## Alert Rules
38+
39+
Four Prometheus alerts are defined in [`infra/docker/alert_rules.yml`](../infra/docker/alert_rules.yml):
40+
41+
| Alert | Expression | Duration | Severity |
42+
|-------|-----------|----------|----------|
43+
| `CircuitBreakerOpen` | `vector_catalog_circuit_breaker_state > 0` | 30 s | critical |
44+
| `P99LatencyHigh` | `histogram_quantile(0.99, ...) > 500` | 2 min | warning |
45+
| `CacheHitRateLow` | `vector_catalog_cache_hit_rate < 0.70` | 5 min | warning |
46+
| `ErrorRateHigh` | 5xx rate > 1% (5-min rolling) | 1 min | critical |
47+
48+
---
49+
50+
## Monitoring Links
51+
52+
| Tool | URL |
53+
|------|-----|
54+
| Live health check | `https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/health` |
55+
| Live Prometheus metrics | `https://vector-catalog-api.politefield-8fe8e6a2.eastus.azurecontainerapps.io/metrics` |
56+
| Local Prometheus | `http://localhost:9090` |
57+
| Local Grafana dashboard | `http://localhost:3000` (admin / admin) |
58+
| GitHub Actions (CI) | `https://github.com/ritunjaym/vector-catalog-service/actions` |
59+
| Trivy security scan | `https://github.com/ritunjaym/vector-catalog-service/security` |
60+
61+
---
62+
63+
## Error Budget Burn Rate
64+
65+
With a 99.9% uptime target the monthly error budget is **43.8 minutes**.
66+
67+
| Burn rate | Meaning | Alert within |
68+
|-----------|---------|--------------|
69+
| 1× | Consuming budget at exactly the SLO rate | — |
70+
| 5× | Budget exhausted in ~6 days | P2 |
71+
| 14.4× | Budget exhausted in ~2 hours | P1 |
72+
| 36× | Budget exhausted in ~1 hour | P0 |
73+
74+
---
75+
76+
## Exclusions
77+
78+
- Scheduled maintenance windows (announced ≥ 24 hours in advance, max 4 hours/month)
79+
- Force majeure (Azure region outage, DNS provider failure)
80+
- Client-side network issues outside Azure

‎infra/docker/alert_rules.yml‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
groups:
2+
- name: vector_catalog_alerts
3+
interval: 30s
4+
rules:
5+
6+
# ── Circuit Breaker ───────────────────────────────────────────────────────
7+
- alert: CircuitBreakerOpen
8+
expr: vector_catalog_circuit_breaker_state > 0
9+
for: 30s
10+
labels:
11+
severity: critical
12+
team: platform
13+
annotations:
14+
summary: "Circuit breaker is open"
15+
description: >
16+
The Polly circuit breaker has been open for >30s.
17+
Downstream gRPC calls to the sidecar are being short-circuited.
18+
Check sidecar health and gRPC connectivity.
19+
runbook: "https://github.com/ritunjaym/vector-catalog-service/blob/main/docs/SLA.md"
20+
21+
# ── P99 Search Latency ────────────────────────────────────────────────────
22+
- alert: P99LatencyHigh
23+
expr: >
24+
histogram_quantile(0.99,
25+
rate(vector_catalog_search_latency_ms_bucket[5m])
26+
) > 500
27+
for: 2m
28+
labels:
29+
severity: warning
30+
team: platform
31+
annotations:
32+
summary: "P99 search latency above 500 ms"
33+
description: >
34+
P99 end-to-end search latency has exceeded 500 ms for 2 minutes.
35+
Current value: {{ $value | printf "%.0f" }} ms.
36+
Investigate FAISS nprobe setting, Redis cache hit rate, and Spark job lag.
37+
runbook: "https://github.com/ritunjaym/vector-catalog-service/blob/main/docs/SLA.md"
38+
39+
# ── Cache Hit Rate ────────────────────────────────────────────────────────
40+
- alert: CacheHitRateLow
41+
expr: vector_catalog_cache_hit_rate < 0.70
42+
for: 5m
43+
labels:
44+
severity: warning
45+
team: platform
46+
annotations:
47+
summary: "Redis cache hit rate below 70%"
48+
description: >
49+
Cache hit rate has dropped below 70% for 5 minutes.
50+
Current rate: {{ $value | printf "%.1%%" }}.
51+
Check Redis eviction policy, memory limits, and TTL configuration.
52+
runbook: "https://github.com/ritunjaym/vector-catalog-service/blob/main/docs/SLA.md"
53+
54+
# ── Error Rate ────────────────────────────────────────────────────────────
55+
- alert: ErrorRateHigh
56+
expr: >
57+
(
58+
rate(vector_catalog_requests_total{status=~"5.."}[5m])
59+
/
60+
rate(vector_catalog_requests_total[5m])
61+
) > 0.01
62+
for: 1m
63+
labels:
64+
severity: critical
65+
team: platform
66+
annotations:
67+
summary: "HTTP 5xx error rate above 1%"
68+
description: >
69+
The 5-minute rolling error rate has exceeded 1% for 1 minute.
70+
Current rate: {{ $value | printf "%.2%%" }}.
71+
Check API logs and downstream service health.
72+
runbook: "https://github.com/ritunjaym/vector-catalog-service/blob/main/docs/SLA.md"

‎infra/docker/prometheus.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ global:
55
cluster: 'vector-catalog-dev'
66
environment: 'local'
77

8+
rule_files:
9+
- "alert_rules.yml"
10+
811
scrape_configs:
912
# ══════════════════════════════════════════════════════════════════════════
1013
# Vector Catalog API (.NET metrics)

0 commit comments

Comments
 (0)