URL Shortener

High-performance URL shortening service with Go microservices, master-replica DB, and async batch processing.

Features

• Microservices: API Gateway, Shorten, Redirect, Batch Writer
• Master-Replica DB: Read-write splitting with PostgreSQL streaming replication
• Redis Cache: 95%+ hit rate, sub-100ms response
• Async Writes: RabbitMQ + batch processing (100x reduced DB load)
• Monitoring: Prometheus + Grafana dashboards
• Rate Limiting: IP-based protection (60 req/min)

Quick Start

# Start all services (includes master-replica DB)
docker-compose up -d

# Test replication
./scripts/test-replication.sh

# Access services
# API: http://localhost:8080
# Grafana: http://localhost:3000 (admin/admin)
# Prometheus: http://localhost:9090

API Usage

Create short URL:

curl -X POST http://localhost:8080/shorten \
  -H "Content-Type: application/json" \
  -d '{"long_url": "https://example.com"}'

Use short URL:

curl -L http://localhost:8080/abc123

Endpoints:

• POST /shorten - Create short URL
• GET /:code - Redirect to original URL
• GET /info/:code - Get URL statistics
• GET /health - Health check
• GET /metrics - Prometheus metrics

Architecture

User Request
    ↓
API Gateway (:8080) → Rate Limiting
    ↓
┌───────────┬────────────┐
│           │            │
Shorten    Redirect    Health
Service    Service     Check
(:8081)    (:8082)
    │          │
    ↓          ↓
 Redis ←──────┘ (Cache 95%+ hits)
    │
    ↓
RabbitMQ (Async queue)
    │
    ↓
Batch Writer (100 URLs/batch)
    │
    ↓
PostgreSQL Master ──replication──> Replica
(All writes)                       (All reads)

Read-Write Splitting

• Master (postgres-master:5432): URL creation, visit updates
• Replica (postgres-replica:5433): Redirects, queries
• Replication lag: < 100ms
• Auto-failover: Falls back to master if replica unavailable

See docs/MASTER_REPLICA_SETUP.md for details.

Performance

• QPS: 5000+ redirects/sec
• Latency: P99 < 50ms
• Cache Hit Rate: > 95%
• DB Load: Master 20-30%, Replica 40-50%
• Read/Write Ratio: 100:1

Tech Stack

• Backend: Go 1.21, Gin framework, GORM
• Database: PostgreSQL 13 (Master-Replica)
• Cache: Redis 6
• Queue: RabbitMQ 3
• Monitoring: Prometheus + Grafana
• Deployment: Docker Compose / Kubernetes

Monitoring

Access Grafana dashboard at http://localhost:3000 (admin/admin)

Metrics:

• HTTP: QPS, latency (P50/P95/P99), error rate
• Cache: Hit/miss rate
• Database: Query duration, connection pool
• Business: URLs created, redirects performed

See monitoring/README.md for setup.

Project Structure

api-gateway/          # Entry point, routing, rate limiting
shorten-service/      # Generate short URLs
redirect-service/     # Resolve URLs, cache-first
batch-writer-service/ # Async DB writes
common/
  ├── db/            # Master-replica connection
  ├── metrics/       # Prometheus metrics
  ├── models/        # Shared data models
  └── redis/         # Redis client
k8s/                 # Kubernetes configs
monitoring/          # Prometheus + Grafana
scripts/             # Deployment & test scripts
docs/                # Documentation

Development

make help          # Show all commands
make docker-up     # Start with Docker Compose
make k8s-deploy    # Deploy to Kubernetes
make test          # Run tests

Each service is a separate Go module. Run go mod tidy after changes.

Configuration

See .env.example for all options.

Key settings:

# Rate limiting
RATE_LIMIT_PER_MINUTE=60

# Database
DB_HOST=postgres-master
DB_REPLICA_HOST=postgres-replica

# Batch processing
BATCH_SIZE=100
FLUSH_INTERVAL=5s