Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
474 lines (422 loc) · 16.5 KB

ARCHITECTURE.md

File metadata and controls

474 lines (422 loc) · 16.5 KB

WhoVoted Architecture

System Overview

┌─────────────────────────────────────────────────────────────────┐
│                         WhoVoted System                          │
└─────────────────────────────────────────────────────────────────┘

┌──────────────────┐         ┌──────────────────┐
│   Public Users   │         │  Admin Users     │
│                  │         │                  │
│  • View Map      │         │  • Upload CSV    │
│  • Search        │         │  • Monitor       │
│  • Geolocation   │         │  • Manage Data   │
└────────┬─────────┘         └────────┬─────────┘
         │                            │
         │ HTTP                       │ HTTP + Auth
         │                            │
         ▼                            ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Flask Backend (Port 5000)                   │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   app.py     │  │   auth.py    │  │  upload.py   │          │
│  │              │  │              │  │              │          │
│  │ • Routes     │  │ • Sessions   │  │ • Validate   │          │
│  │ • Static     │  │ • Tokens     │  │ • Save       │          │
│  │ • CORS       │  │ • Decorator  │  │ • Cleanup    │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │ processor.py │  │ geocoder.py  │  │  config.py   │          │
│  │              │  │              │  │              │          │
│  │ • Validate   │  │ • Cache      │  │ • Settings   │          │
│  │ • Clean      │  │ • Nominatim  │  │ • Logging    │          │
│  │ • Geocode    │  │ • Rate Limit │  │ • Paths      │          │
│  │ • Generate   │  │ • Retry      │  │              │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
└─────────────────────────────────────────────────────────────────┘
         │                            │
         │ File I/O                   │ HTTP API
         │                            │
         ▼                            ▼
┌──────────────────┐         ┌──────────────────┐
│  File System     │         │  Nominatim API   │
│                  │         │  (OpenStreetMap) │
│  • uploads/      │         │                  │
│  • data/         │         │  • Geocoding     │
│  • logs/         │         │  • Search        │
│  • public/       │         │  • Reverse       │
└──────────────────┘         └──────────────────┘
         │
         │ Serve Static
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Frontend (public/)                          │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │  index.html  │  │    map.js    │  │  search.js   │          │
│  │              │  │              │  │              │          │
│  │ • Layout     │  │ • Leaflet    │  │ • Nominatim  │          │
│  │ • Controls   │  │ • Markers    │  │ • Autocomplete│         │
│  │ • UI         │  │ • Clusters   │  │ • Reverse    │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
│                                                                   │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐          │
│  │   data.js    │  │  config.js   │  │  styles.css  │          │
│  │              │  │              │  │              │          │
│  │ • Load JSON  │  │ • Map Center │  │ • Responsive │          │
│  │ • Parse      │  │ • Zoom       │  │ • Layout     │          │
│  │ • Error      │  │ • Bounds     │  │ • Colors     │          │
│  └──────────────┘  └──────────────┘  └──────────────┘          │
└─────────────────────────────────────────────────────────────────┘
         │
         │ Tiles
         │
         ▼
┌──────────────────┐
│  OpenStreetMap   │
│  Tile Servers    │
│                  │
│  • Map Tiles     │
│  • Attribution   │
└──────────────────┘

Data Flow

Public User Flow

User Opens Browser
       │
       ▼
Load index.html
       │
       ▼
Load JavaScript (map.js, search.js, data.js)
       │
       ▼
Fetch map_data.json
       │
       ▼
Load OpenStreetMap Tiles
       │
       ▼
Render Markers on Map
       │
       ▼
User Interacts (search, click, zoom)

Admin Upload Flow

Admin Logs In
       │
       ▼
POST /admin/login (username, password)
       │
       ▼
Create Session Token
       │
       ▼
Set Cookie
       │
       ▼
Access Dashboard
       │
       ▼
Upload CSV File
       │
       ▼
POST /admin/upload (file)
       │
       ▼
Validate File (extension, size)
       │
       ▼
Save to uploads/
       │
       ▼
Create Processing Job
       │
       ▼
Start Background Thread
       │
       ├─► Step 1: Validate CSV
       │   └─► Check columns, format
       │
       ├─► Step 2: Clean Addresses
       │   └─► Normalize, standardize
       │
       ├─► Step 3: Geocode
       │   ├─► Check Cache
       │   ├─► Call Nominatim (if not cached)
       │   ├─► Rate Limit (1 req/sec)
       │   └─► Store in Cache
       │
       ├─► Step 4: Generate Output
       │   ├─► Create map_data.json (GeoJSON)
       │   ├─► Create metadata.json
       │   └─► Create error CSV (if failures)
       │
       └─► Step 5: Deploy
           └─► Copy to public/data/
       │
       ▼
Processing Complete
       │
       ▼
Admin Views Results
       │
       ▼
Public Map Updates Automatically

Component Responsibilities

Backend Components

app.py

  • HTTP routing
  • Static file serving
  • Request handling
  • Error handling
  • Security headers

auth.py

  • Credential validation
  • Session management
  • Token generation
  • Authentication decorator
  • Session cleanup

upload.py

  • File validation
  • File storage
  • Metadata extraction
  • Old file cleanup

processor.py

  • CSV validation
  • Address cleaning
  • Geocoding orchestration
  • Output generation
  • Error reporting
  • Progress tracking

geocoder.py

  • Cache management
  • Nominatim API calls
  • Rate limiting
  • Retry logic
  • Error handling

config.py

  • Environment variables
  • Path management
  • Logging setup
  • Configuration validation

Frontend Components

index.html

  • Page structure
  • UI layout
  • Controls
  • Search box
  • Map container

map.js

  • Leaflet initialization
  • Marker rendering
  • Cluster management
  • Geolocation
  • Map controls

search.js

  • Nominatim search
  • Autocomplete
  • Result display
  • Reverse geocoding
  • Error handling

data.js

  • JSON loading
  • Data parsing
  • Error handling
  • Coordinate validation

config.js

  • Map center
  • Zoom levels
  • Bounds
  • API settings

styles.css

  • Responsive layout
  • Colors and fonts
  • Control styling
  • Mobile support

Security Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Security Layers                          │
└─────────────────────────────────────────────────────────────────┘

Layer 1: Network
├─► HTTPS (production)
├─► CORS restrictions
└─► Rate limiting

Layer 2: Authentication
├─► Session tokens (32-byte random)
├─► HttpOnly cookies
├─► 24-hour expiration
└─► Secure flag (production)

Layer 3: Authorization
├─► @require_auth decorator
├─► Token validation
├─► Session expiration check
└─► Redirect to login

Layer 4: Input Validation
├─► File extension check
├─► File size limit (100MB)
├─► CSV structure validation
└─► Address format validation

Layer 5: Output Sanitization
├─► JSON encoding
├─► Error message filtering
└─► Log sanitization

Layer 6: Security Headers
├─► X-Content-Type-Options: nosniff
├─► X-Frame-Options: DENY
├─► X-XSS-Protection: 1; mode=block
├─► Strict-Transport-Security
└─► Content-Security-Policy

Caching Strategy

┌─────────────────────────────────────────────────────────────────┐
│                      Geocoding Cache Flow                        │
└─────────────────────────────────────────────────────────────────┘

Address Input
       │
       ▼
Normalize Address
(uppercase, trim, standardize)
       │
       ▼
Check Cache
       │
       ├─► Cache Hit
       │   └─► Return Cached Result (instant)
       │
       └─► Cache Miss
           │
           ▼
           Call Nominatim API
           (rate limited: 1 req/sec)
           │
           ▼
           Store in Cache
           │
           ▼
           Save to geocoding_cache.json
           │
           ▼
           Return Result

Cache Benefits:
• 100% hit rate on repeat uploads
• Reduces API calls to zero (for cached addresses)
• Speeds up processing dramatically
• Persists across server restarts

File Structure

WhoVoted/
├── backend/                    # Flask application
│   ├── app.py                 # Main application (200 lines)
│   ├── auth.py                # Authentication (150 lines)
│   ├── upload.py              # File handling (100 lines)
│   ├── processor.py           # Data processing (400 lines)
│   ├── geocoder.py            # Nominatim integration (250 lines)
│   ├── config.py              # Configuration (100 lines)
│   ├── requirements.txt       # Python dependencies
│   ├── .env                   # Environment variables
│   └── admin/                 # Admin panel
│       ├── login.html         # Login page (150 lines)
│       ├── dashboard.html     # Dashboard (200 lines)
│       └── dashboard.js       # Dashboard logic (300 lines)
│
├── public/                     # Static frontend
│   ├── index.html             # Main page (150 lines)
│   ├── map.js                 # Map management (200 lines)
│   ├── search.js              # Search functionality (150 lines)
│   ├── data.js                # Data loading (100 lines)
│   ├── config.js              # Frontend config (20 lines)
│   ├── styles.css             # Styles (300 lines)
│   └── data/                  # Generated data
│       ├── map_data.json      # GeoJSON features
│       ├── metadata.json      # Processing metadata
│       └── progress.json      # Progress tracking
│
├── data/                       # Backend data
│   ├── geocoding_cache.json   # Persistent cache
│   └── processing_errors.csv  # Error reports
│
├── uploads/                    # Temporary uploads
│   └── *.csv                  # Uploaded CSV files
│
├── logs/                       # Application logs
│   └── app.log                # Server logs
│
└── Documentation
    ├── START_HERE.md          # Quick start
    ├── QUICK_START.md         # Reference guide
    ├── COMPLETION_SUMMARY.md  # Feature list
    ├── OVERNIGHT_WORK_SUMMARY.md  # Work summary
    ├── README_MODERNIZATION.md    # Full docs
    ├── IMPLEMENTATION_SUMMARY.md  # Technical details
    ├── PRODUCTION_CHECKLIST.md    # Deployment guide
    └── ARCHITECTURE.md        # This file

Total Lines of Code: ~2,500
Total Files: 25+

Technology Stack

Backend

  • Framework: Flask 2.3+
  • Language: Python 3.8+
  • Authentication: Session tokens (in-memory)
  • Geocoding: Nominatim API
  • Caching: JSON file-based
  • Logging: Python logging module

Frontend

  • Map Library: Leaflet.js 1.9+
  • Clustering: Leaflet.markercluster
  • Tiles: OpenStreetMap
  • Search: Nominatim API
  • Styling: Custom CSS (responsive)

External Services

  • OpenStreetMap: Map tiles (free)
  • Nominatim: Geocoding API (free)

Development Tools

  • Testing: Python unittest, custom test scripts
  • Logging: File and console logging
  • Error Handling: Try-catch, graceful degradation

Performance Characteristics

Backend

  • Request Handling: < 100ms (static files)
  • Authentication: < 50ms (token validation)
  • CSV Validation: < 1s (for 1000 rows)
  • Geocoding: ~1s per address (first time)
  • Cached Geocoding: < 0.1s per address
  • Output Generation: < 1s (for 1000 features)

Frontend

  • Initial Load: < 1s (with cached assets)
  • Map Rendering: < 1s (for 100 markers)
  • Search Response: < 500ms (Nominatim)
  • Marker Click: < 50ms (popup display)

Scalability

  • Max Upload Size: 100MB (configurable)
  • Max Records: 150k+ (tested)
  • Concurrent Users: 100+ (single instance)
  • Cache Size: Unlimited (JSON file)

Deployment Options

Local Development (Current)

  • Flask development server
  • Port 5000
  • HTTP (not HTTPS)
  • In-memory sessions
  • File-based cache

Production (Recommended)

  • Backend: Railway, Heroku, or DigitalOcean
  • Frontend: GitHub Pages, Netlify, or Vercel
  • Server: Gunicorn or Waitress (WSGI)
  • Protocol: HTTPS (required)
  • Sessions: Redis or database
  • Cache: Redis or database
  • Monitoring: Sentry, Datadog, or similar

Last Updated: 2026-02-20 Version: 1.0 Status: Production-ready for local use