🗄️ Distributed Database System

A distributed database system with a Go API Gateway coordinating three polyglot worker shards.

Architecture · Quick Start · API Reference · Demo Guide

---

Overview

This project implements a sharded, distributed database where:

• A Go Master node acts as an API gateway — it validates requests, routes shards, and coordinates scatter/gather reads and MapReduce operations.
• Three Worker nodes (Go, Python, Node.js) each own a physical SQL Server database shard and execute all actual SQL.
• Clients talk only to the Master on port 8080. Workers reject direct access with 401 Unauthorized.

Architecture

Client / Postman / GUI
         │
         ▼
┌─────────────────────────────┐
│   Master API Gateway (Go)   │  :8080
│  • Request validation        │
│  • Shard routing             │
│  • Scatter/gather reads      │
│  • MapReduce coordination    │
│  • Health monitoring         │
└──────────┬──────────────────┘
           │   X-Master-Token
     ┌─────┼──────────┐
     ▼     ▼          ▼
 Worker  Worker    Worker
   Go    Python   Node.js
  :8081  :8082    :8083
  SchoolDBGO  SchoolDBPY  SchoolDBNODE

Sharding Strategy

Request Type	Shard Key Source	Routing
INSERT	shard_key → student_id / user_id → row hash	One worker
SELECT (with key)	shard_key or student_id = X in condition	One worker
SELECT (no key)	—	Scatter/gather all workers
UPDATE	shard_key required	One worker
DELETE	shard_key required	One worker

Physical Databases

Logical Name	Worker	Physical DB
SchoolDB	Go	SchoolDBGO
SchoolDB	Python	SchoolDBPY
SchoolDB	Node.js	SchoolDBNODE

---

Project Structure

distributed-db/
├── master/               # Go API Gateway — port 8080
│   ├── main.go           # HTTP server, routes, CORS
│   ├── handlers.go       # All HTTP handler functions
│   ├── replication.go    # Shard routing, fan-out, MapReduce
│   └── go.mod
│
├── slave-go/             # Go worker shard — port 8081 — suffix GO
│   ├── main.go
│   ├── handlers.go
│   ├── db.go             # SQL Server connection + SwitchDB
│   └── go.mod
│
├── slave-python/         # Python Flask worker shard — port 8082 — suffix PY
│   ├── app.py
│   └── requirements.txt
│
├── slave-node/           # Node.js Express worker shard — port 8083 — suffix NODE
│   ├── server.js
│   └── package.json
│
└── docs/
    ├── postman-guide.md
    └── Distributed_DB_Postman_Collection.json

---

Requirements

Requirement	Version
Go	1.21+
Python	3.9+
Node.js	18+
SQL Server	Any with Windows Authentication
ODBC Driver	17 for SQL Server

Install ODBC Driver 17: Microsoft Download

---

Quick Start

Open four separate terminals, one per service.

1. Master (API Gateway)

cd master
go run .
# Listening on :8080

2. Worker — Go

cd slave-go
go run .
# Listening on :8081

3. Worker — Python

cd slave-python
pip install -r requirements.txt
python app.py
# Listening on :8082

4. Worker — Node.js

cd slave-node
npm install
node server.js
# Listening on :8083

Optional: Backup Master (Failover Demo)

cd master
$env:MASTER_PORT="8084"
$env:MASTER_ROLE="backup"
go run .

---

Demo Workflow

Import the Postman collection and follow the step-by-step guide:

1. Import docs/Distributed_DB_Postman_Collection.json into Postman.
2. Follow docs/postman-guide.md to demonstrate all features.

Minimal flow (cURL)

# Create databases on all shards
curl -X POST http://localhost:8080/create-db \
  -H "Content-Type: application/json" \
  -d '{"db_name": "SchoolDB"}'

# Create the table schema on all shards
curl -X POST http://localhost:8080/create-table \
  -H "Content-Type: application/json" \
  -d '{
    "db_name": "SchoolDB",
    "table": "Students",
    "columns": {
      "student_id": "INT",
      "name": "VARCHAR(100)",
      "grade": "VARCHAR(5)"
    }
  }'

# Insert — routed to one shard
curl -X POST http://localhost:8080/insert \
  -H "Content-Type: application/json" \
  -d '{
    "db_name": "SchoolDB",
    "table": "Students",
    "shard_key": "1001",
    "data": {"student_id": "1001", "name": "Ali", "grade": "A"}
  }'

# Select — single shard (key provided)
curl -X POST http://localhost:8080/select \
  -H "Content-Type: application/json" \
  -d '{
    "db_name": "SchoolDB",
    "table": "Students",
    "shard_key": "1001",
    "condition": "student_id = 1001"
  }'

# Select — scatter/gather all shards (no key)
curl -X POST http://localhost:8080/select \
  -H "Content-Type: application/json" \
  -d '{"db_name": "SchoolDB", "table": "Students"}'

# MapReduce count
curl -X POST http://localhost:8080/map-reduce-count \
  -H "Content-Type: application/json" \
  -d '{"db_name": "SchoolDB", "table": "Students"}'

---

API Reference

All requests go to http://localhost:8080.

Schema / Database Operations (broadcast to all workers)

Method	Endpoint	Body fields
POST	/create-db	db_name
POST	/drop-db	db_name
POST	/create-table	db_name, table, columns (object)
POST	/delete-table	db_name, table

Data Operations (routed by shard key)

Method	Endpoint	Body fields	Notes
POST	/insert	db_name, table, data, shard_key?	Routed to one shard
POST	/select	db_name, table, condition?, shard_key?	One shard or scatter/gather
POST	/update	db_name, table, set, condition, shard_key?	Requires shard key
POST	/delete	db_name, table, condition, shard_key?	Requires shard key

Cluster Operations

Method	Endpoint	Description
GET	/metadata	Master role, worker list, routing info
GET	/health	Gateway + all worker health
POST	/map-reduce-count	Count rows across all shards
POST	/security-check	Prove workers reject tokenless requests

---

Security Model

External Client ──► Master (port 8080)
                         │
                         │  adds X-Master-Token: master-secret-token-2025
                         ▼
                    Worker Nodes (8081–8083)

• Workers expose no public interface. Every non-health endpoint checks X-Master-Token.
• Missing or wrong token → 401 Unauthorized.
• The GUI/Postman never needs to know the token — the Master injects it on forwarding.

---

Features Demonstrated

Feature	How it works
Sharding	FNV-1a hash of shard key % 3 selects the worker
Scatter/Gather	SELECT without key fans out to all 3 workers concurrently
MapReduce	Workers map (return rows), Master reduces (sums counts)
API Gateway security	Workers reject all requests without X-Master-Token
Request validation	Master rejects non-POST, missing fields, unsafe updates
Health monitoring	Master polls workers concurrently and reports status
Master failover	Backup master on :8084 — stateless, same routing config
Polyglot workers	Go · Python Flask · Node.js Express — same REST contract

---

Master Failover

Because the Master stores only routing metadata (no application data), failover is seamless:

1. Start backup master: MASTER_PORT=8084 MASTER_ROLE=backup go run .
2. Kill primary master (:8080).
3. Change your Postman baseUrl to http://localhost:8084.
4. All operations continue — workers are unchanged.

---

License

MIT — see LICENSE.