Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
580 lines (485 loc) · 27.8 KB

ARCHITECTURE.md

File metadata and controls

580 lines (485 loc) · 27.8 KB

IronDrop Architecture Documentation v2.7.0

Overview

IronDrop is a file server written in Rust. It uses only the standard library for networking and file I/O (no external HTTP framework). This document provides an overview of the system architecture, component interactions, and implementation details.

System Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   CLI Parser    │───▶│   Server Init   │───▶│Custom Thread Pool│
│   (cli.rs)      │    │   (main.rs)     │    │   (server.rs)   │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                                                        ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│ Template Engine │◀───│   HTTP Handler  │◀───│  Request Router │
│ (templates.rs)  │    │  (response.rs)  │    │   (http.rs)     │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Static Assets  │    │   File System   │    │Upload & Multipart│
│ (templates/*)   │    │    (fs.rs)      │    │upload.rs+multipart│
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                │                       │
                                ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Downloads     │    │     Uploads     │    │   Search Engine │
│ Range Requests  │    │ Direct Streaming │    │Ultra-compact    │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                                        │
                                                        ▼
                                             ┌─────────────────┐
                                             │Security & Monitor│
                                             │ Rate Limit+Stats │
                                             └─────────────────┘

Core Modules

1. Entry Point & Configuration

  • main.rs (6 lines): Simple entry point that calls irondrop::run()
  • lib.rs (56 lines): Library initialization, logging setup, and server bootstrap
  • cli.rs (200+ lines): Command-line interface with comprehensive validation
  • config/mod.rs: Configuration system with hierarchical precedence (CLI > INI > defaults)
  • config/ini_parser.rs: Zero-dependency INI parser for configuration files

2. HTTP Processing Layer

  • server.rs: Custom thread pool implementation with rate limiting and connection management
  • http.rs: HTTP request parsing, routing, connection handling, and static asset serving
  • response.rs: HTTP response building, MIME type detection, and error page generation
  • router.rs: Simple HTTP router with exact and prefix path matching
  • handlers.rs: Internal route handlers for health checks, status, uploads, and monitoring
  • middleware.rs: Authentication middleware with Basic Auth support

3. File Operations

  • fs.rs: Directory listing generation, file details, and file system interactions
  • upload.rs: Direct upload handler with memory/disk streaming and atomic operations

4. Search System

  • search.rs: Search system; ultra-compact mode with hierarchical path storage and string pooling

5. Template System

  • templates.rs: Native template engine with embedded assets and variable interpolation
  • templates/directory/: Directory listing templates (HTML, CSS, JS)
  • templates/upload/: File upload templates (HTML, CSS, JS)
  • templates/error/: Error page templates (HTML, CSS, JS)
  • templates/monitor/: Monitoring dashboard templates

6. Support Systems

  • error.rs: Comprehensive error types including upload-specific errors
  • utils.rs: Utility functions for path handling, URL parsing, and encoding

Request Processing Flow

                           HTTP Request
                                │
                                ▼
                     ┌─────────────────┐
                     │  Rate Limiting  │───[Fail]───▶ 429 Too Many Requests
                     │     Check       │
                     └─────────────────┘
                                │ [Pass]
                                ▼
                     ┌─────────────────┐
                     │ Authentication  │───[Fail]───▶ 401 Unauthorized
                     │  Middleware     │
                     └─────────────────┘
                                │ [Pass]
                                ▼
                     ┌─────────────────┐
                     │   HTTP Router   │
                     │  (router.rs)    │
                     └─────────────────┘
                                │
        ┌───────────┬───────────┼───────────┬───────────┬───────────┐
        │           │           │           │           │           │
        ▼           ▼           ▼           ▼           ▼           ▼
  [Static Assets] [Health]  [Upload API]  [File Sys]  [Search API] [Monitor]
   /_irondrop/    /_irondrop/health   /_irondrop/   Directory   /_irondrop/  /monitor
    /static/*               /upload       Listing     /search
        │           │           │           │           │           │
        ▼           ▼           ▼           ▼           ▼           ▼
   Serve CSS/JS  JSON Status Process Upload Path Check Search Engine Dashboard
                                  │           │           │
                         [Pass]   │   [Fail]  │           ▼
                                  ▼           ▼      JSON Results
                            Resource Type  403 Forbidden
                             Detection
                                  │
                    ┌─────────────┼─────────────┐
                    │             │             │
                    ▼             ▼             ▼
              [Directory]       [File]      [Upload UI]
                    │             │             │
                    ▼             ▼             ▼
           Template-based    Stream File   Upload Interface
             Listing          Content        Processing

File Structure

src/
├── main.rs              # Entry point (6 lines)
├── lib.rs               # Library initialization (56 lines)
├── cli.rs               # CLI interface with validation (200+ lines)
├── config/              # Configuration system
│   ├── mod.rs           # Config struct and loading logic
│   └── ini_parser.rs    # Zero-dependency INI parser
├── server.rs            # Thread pool + rate limiting (400+ lines)
├── http.rs              # HTTP parsing + connection handling (600+ lines)
├── router.rs            # HTTP routing system (100+ lines)
├── handlers.rs          # Internal route handlers (200+ lines)
├── middleware.rs        # Authentication middleware (100+ lines)
├── templates.rs         # Template engine with embedded assets (300+ lines)
├── fs.rs                # File system operations (200+ lines)
├── response.rs          # Response building + MIME detection (400+ lines)
├── upload.rs            # Direct upload handler with streaming (500+ lines)
├── search.rs            # Ultra-compact search engine (400+ lines)
├── error.rs             # Comprehensive error types (100+ lines)
└── utils.rs             # Utility functions for paths and encoding

templates/
├── directory/           # Directory listing UI
│   ├── index.html       # HTML structure with search
│   ├── styles.css       # Professional dark theme
│   └── script.js        # Interactive browsing + search
├── upload/              # Upload interface
│   ├── page.html        # Standalone upload page
│   ├── form.html        # Reusable upload component
│   ├── styles.css       # Upload UI styling
│   └── script.js        # Direct binary upload logic
├── error/               # Error pages
│   ├── page.html        # Error page template
│   └── styles.css       # Error page styling
└── monitor/             # Monitoring dashboard
    ├── page.html        # Dashboard template
    ├── styles.css       # Dashboard styling
    └── script.js        # Real-time metrics updates
└── error/               # Error pages
    ├── page.html        # Error page structure
    ├── styles.css       # Error styling
    └── script.js        # Error page enhancements

tests/
├── integration_test.rs     # Core server tests (19 tests)
├── integration_test.rs       # Auth + security tests (6 tests)
├── edge_case_test.rs         # Upload edge cases (10 tests)
├── memory_optimization_test.rs # Memory efficiency (6 tests)
├── performance_test.rs       # Upload performance (5 tests)
├── stress_test.rs           # Stress testing (4 tests)
├── multipart_test.rs        # Multipart parser tests (7 tests)
├── ultra_compact_test.rs    # Search engine tests (4 tests)
├── template_embedding_test.rs # Template system tests (3 tests)
├── test_upload.sh           # End-to-end upload testing
├── test_1gb_upload.sh       # Large file upload testing
└── test_executable_portability.sh # Portability validation

Total: 199 tests across 16 test files

Search System Architecture

Overview

IronDrop features a sophisticated dual-mode search system designed for both efficiency and scalability, with support for directories containing millions of files while maintaining low memory usage.

Search Implementation Modes

1. Standard Search Engine (search.rs)

  • Target: Directories with up to 100K files
  • Memory Usage: ~10MB for 10K files
  • Features:
    • LRU cache with 5-minute TTL
    • Thread-safe operations with Arc<Mutex<>>
    • Fuzzy search with relevance scoring
    • Real-time indexing with background updates
    • Full-text search with token matching

2. Ultra-Compact Search (ultra_compact_search.rs)

  • Target: Directories with 10M+ files
  • Memory Usage: <100MB for 10M files (11 bytes per entry)
  • Features:
    • Hierarchical path storage with parent references
    • Unified string pool with binary search
    • Bit-packed metadata (size, timestamps, flags)
    • Cache-aligned structures for CPU optimization
    • Radix-accelerated indexing

Memory Optimization Techniques

Standard Entry (24 bytes):     Ultra-Compact Entry (11 bytes):
┌────────────────────┐        ┌─────────────────┐
│ Full Path (String) │        │ Name Offset (3) │
│ Name (String)      │        │ Parent ID (3)   │
│ Size (u64)         │        │ Size Log2 (1)   │
│ Modified (u64)     │        │ Packed Data (4) │
│ Flags (u32)        │        └─────────────────┘
└────────────────────┘        58% memory reduction

Search Performance Characteristics

Directory Size Standard Mode Ultra-Compact Mode
1K files <1ms <1ms
10K files 2-5ms 1-3ms
100K files 10-20ms 5-10ms
1M files N/A 20-50ms
10M files N/A 100-200ms

Search API Integration

The search system integrates with the HTTP layer through dedicated endpoints:

  • GET /_irondrop/search?q=query: Primary search interface
  • Frontend Integration: Real-time search with 300ms debouncing
  • Result Pagination: Configurable limits and offsets
  • JSON Response Format: Structured results with metadata

Caching Strategy

Request → Cache Check → Hit: Return Cached Results
             │
             └─ Miss → Index Search → Cache Store → Return Results
  • LRU Eviction: Least recently used entries removed first
  • TTL Expiration: 5-minute automatic cache invalidation
  • Memory Bounds: Maximum 1000 cached queries
  • Thread Safety: Concurrent read/write operations supported

HTTP Layer Streaming Architecture

Overview

IronDrop v2.7.0 provides advanced HTTP layer streaming for efficient handling of large file uploads. The system automatically switches between memory-based and disk-based processing based on content size, providing optimal performance and resource utilization.

RequestBody Architecture

pub enum RequestBody {
    Memory(Vec<u8>),           // Small uploads (≤64MB)
    File(PathBuf),             // Large uploads (>64MB)
}

The RequestBody enum provides a unified interface for handling HTTP request bodies of varying sizes:

  • Memory Variant: Stores small uploads directly in memory for fast processing
  • File Variant: Streams large uploads to temporary files to prevent memory exhaustion
  • Automatic Selection: Transparent switching based on configurable size threshold
  • Resource Management: Automatic cleanup of temporary files with error recovery

Streaming Decision Flow

HTTP Request → Content-Length Check → Size Threshold Comparison
                                              │
                    ┌─────────────────────────┼─────────────────────────┐
                    │ ≤64MB                    │                    >64MB │
                    ▼                         ▼                         ▼
            Memory Processing           Disk Streaming              Disk Streaming
            ┌─────────────────┐        ┌─────────────────┐        ┌─────────────────┐
            │ Read to Vec<u8> │        │ Create Temp File│        │ Stream to Disk  │
            │ Fast Processing │        │ Stream Chunks   │        │ Memory Efficient│
            │ Low Latency     │        │ Auto Cleanup    │        │ Large File Safe │
            └─────────────────┘        └─────────────────┘        └─────────────────┘
                    │                         │                         │
                    └─────────────────────────┼─────────────────────────┘
                                              ▼
                                    Unified Processing
                                    (Multipart Parser)

Performance Characteristics

Upload Size Processing Mode Memory Usage Disk I/O Latency
<1KB Memory ~1KB None <1ms
1KB-64MB Memory ~Size None <10ms
64MB-100MB Disk Streaming ~64KB Sequential <100ms
100MB-1GB Disk Streaming ~64KB Sequential <1s
1GB-10GB Disk Streaming ~64KB Sequential <10s

Security and Resource Protection

Temporary File Management

  • Secure Creation: Uses tempfile::NamedTempFile for secure temporary file creation
  • Automatic Cleanup: Files automatically deleted when RequestBody is dropped
  • Error Recovery: Cleanup guaranteed even during error conditions
  • Permission Control: Temporary files created with restricted permissions

Resource Limits

  • Memory Protection: Prevents memory exhaustion from large uploads
  • Disk Space Monitoring: Checks available disk space before streaming
  • Concurrent Upload Limits: Configurable limits on simultaneous uploads
  • Size Validation: Enforces maximum upload size limits

Integration with Upload System

The HTTP streaming layer integrates seamlessly with the existing upload infrastructure:

impl UploadHandler {
    pub fn handle_upload(&self, request_body: RequestBody) -> Result<(), AppError> {
        match request_body {
            RequestBody::Memory(data) => {
                // Fast path for small uploads
                let cursor = Cursor::new(data);
                self.process_multipart(cursor)
            }
            RequestBody::File(path) => {
                // Streaming path for large uploads
                let file = File::open(path)?;
                self.process_multipart(file)
            }
        }
    }
}

Monitoring and Observability

The streaming system provides comprehensive monitoring capabilities:

  • Upload Metrics: Track memory vs. disk processing ratios
  • Performance Monitoring: Measure processing times by upload size
  • Resource Usage: Monitor temporary file creation and cleanup
  • Error Tracking: Log streaming failures and recovery actions

Configuration Options

pub struct StreamingConfig {
    pub memory_threshold: usize,      // Default: 64MB
    pub chunk_size: usize,           // Default: 64KB
    pub temp_dir: Option<PathBuf>,   // Default: system temp
    pub max_concurrent: usize,       // Default: 10
}

Testing Infrastructure

Dedicated HTTP streaming tests verify correct behavior:

  • http_streaming_test.rs: Comprehensive streaming functionality tests
  • Size Threshold Testing: Verify correct mode selection
  • Resource Cleanup Testing: Ensure temporary files are properly cleaned up
  • Error Condition Testing: Validate error recovery and resource cleanup
  • Performance Testing: Measure streaming performance across size ranges

Security Architecture

Defense in Depth

  1. Input Validation Layer

    • CLI parameter validation with bounds checking
    • HTTP header parsing with malformed request rejection
    • Multipart boundary validation and size limits
    • Filename sanitization and path traversal prevention

  2. Access Control Layer

    • Optional Basic Authentication with secure credential handling
    • Rate limiting (120 requests/minute per IP, configurable)
    • Connection limiting (10 concurrent per IP, configurable)
    • Extension filtering with glob pattern support

  3. Resource Protection Layer

    • Request timeouts to prevent resource exhaustion
    • Memory-efficient streaming for large file operations
    • Disk space checking before upload operations
    • Thread pool management with configurable limits

  4. Audit and Monitoring Layer

    • Comprehensive request logging with unique IDs
    • Performance metrics collection and statistics
    • Health check endpoints (/_irondrop/health, /_irondrop/status)
    • Unified monitoring dashboard (/monitor, /_irondrop/monitor?json=1)
    • Error tracking and security event logging

Security Features by Component

Component Security Features
CLI Input validation, path traversal prevention, size bounds
HTTP Layer Header validation, method restrictions, rate limiting
Upload System File validation, atomic operations, extension filtering
Multipart Parser Boundary validation, size limits, malformed data rejection
Template System Path restrictions, variable escaping, static asset control
File System Canonicalization, directory traversal prevention

Performance Characteristics

Memory Usage

  • Baseline: ~3MB + (thread_count × 8KB stack)
  • Template Cache: In-memory storage for frequently accessed templates
  • Upload Buffer: HTTP streaming with automatic memory/disk switching
  • Small Uploads (≤64MB): Direct memory processing for optimal performance
  • Large Uploads (>64MB): Disk streaming with ~64KB memory footprint
  • File Operations: Configurable chunk sizes (default: 1KB)

Concurrent Processing

  • Thread Pool: Custom implementation (default: 8 threads)
  • Upload Handling: Supports multiple concurrent uploads
  • Rate Limiting: Per-IP tracking with automatic cleanup
  • Connection Management: Efficient file descriptor usage

Request Latency

Operation Typical Latency Notes
Static Assets <0.5ms CSS/JS with caching headers
Directory Listing <2ms Template rendering with file sorting
Health Checks <0.1ms JSON status endpoints
File Downloads Variable Depends on file size and network
File Uploads Variable Includes validation and atomic writing
Error Pages <1ms Template-based professional pages

Scalability Limits

  • File Size: Up to 10GB uploads supported
  • Concurrent Users: Limited by thread pool and system resources
  • Directory Size: Efficient handling of large directories
  • Template Complexity: Sub-millisecond variable interpolation

Template System Architecture

Template Engine Design

The native template engine provides:

  • Variable Interpolation: {{VARIABLE}} syntax with HTML escaping
  • Static Asset Serving: Organized CSS/JS delivery via /_irondrop/static/ routes
  • Modular Templates: Separated concerns (HTML structure, CSS styling, JS behavior)
  • Caching: In-memory template storage for performance

Template Organization

Each template module follows consistent patterns:

  • HTML: Clean semantic structure with accessibility features
  • CSS: Professional design with CSS custom properties
  • JavaScript: Progressive enhancement with graceful degradation

Asset Pipeline

Template Request → Template Engine → Variable Interpolation → HTML Response
                       ↓
Static Asset Request → Asset Router → Direct File Serving → CSS/JS Response

Testing Architecture

Test Organization

  • Unit Tests: Individual component testing
  • Integration Tests: End-to-end functionality verification
  • Security Tests: Boundary and vulnerability testing
  • Performance Tests: Load and stress testing scenarios

Test Coverage by Component

Test File Component Coverage Test Count
integration_test.rs Core server functionality 19
integration_test.rs Authentication and security 6
upload_integration_test.rs Upload system 29
multipart_test.rs Multipart parser 7
debug_upload_test.rs Edge cases and debugging Variable
Others Template system, HTTP handling 40+

Custom Test Infrastructure

  • Native HTTP Client: Pure Rust implementation for testing
  • Mock File Systems: Temporary directories and file operations
  • Concurrent Testing: Multi-threaded test scenarios
  • Security Validation: Path traversal and injection testing

Configuration System

CLI Configuration

pub struct Cli {
    directory: PathBuf,           // Required: directory to serve
    listen: String,               // Default: "127.0.0.1"
    port: u16,                    // Default: 8080
    allowed_extensions: String,    // Default: "*.zip,*.txt"
    threads: usize,               // Default: 8
    chunk_size: usize,            // Default: 1024
    verbose: bool,                // Default: false
    detailed_logging: bool,       // Default: false
    username: Option<String>,     // Optional: basic auth
    password: Option<String>,     // Optional: basic auth
    enable_upload: bool,          // Default: false
    max_upload_size: u32,         // Default: 10240 (10GB)
    upload_dir: Option<PathBuf>,  // Optional: custom upload dir
}

Validation Pipeline

  1. Parse-time Validation: Clap value parsers and constraints
  2. Runtime Validation: Additional checks during server initialization
  3. Operation Validation: Per-request validation and security checks

Error Handling System

Error Types

pub enum AppError {
    Io(std::io::Error),
    InvalidMultipart(String),
    InvalidFilename(String),
    PayloadTooLarge(u64),
    UnsupportedMediaType(String),
    InvalidConfiguration(String),
    // ... additional error variants
}

Error Propagation

  • Result Types: Consistent error handling throughout the application
  • Error Context: Detailed error messages for debugging
  • User-Friendly Messages: Professional error pages with guidance
  • Logging Integration: Error events logged for monitoring

Deployment Considerations

Single Binary Deployment

  • Zero Dependencies: Pure Rust implementation
  • Embedded Assets: Templates compiled into binary
  • Cross-Platform: Supports Linux, macOS, and Windows
  • Portable: Self-contained executable with no external dependencies

Production Hardening

  • Security Defaults: Safe defaults with explicit feature activation
  • Resource Limits: Configurable bounds to prevent abuse
  • Monitoring Integration: Health endpoints for infrastructure monitoring
  • Graceful Degradation: Error recovery and fallback mechanisms

Scalability Options

  • Reverse Proxy: Can be deployed behind nginx/Apache for additional features
  • Load Balancing: Multiple instances can serve the same content
  • Resource Scaling: Configurable thread pool and memory limits
  • Container Deployment: Docker-friendly single binary

Future Architecture Considerations

Potential Enhancements

  1. Database Integration: Optional metadata storage for advanced features
  2. Plugin System: Extensible architecture for custom functionality
  3. WebSocket Support: Real-time features like upload progress
  4. Distributed Storage: Support for cloud storage backends
  5. Advanced Auth: OAuth2/OIDC integration for enterprise deployments

Performance Optimizations

  1. HTTP/2 Support: Enhanced protocol capabilities
  2. Async I/O: Tokio integration for improved concurrency
  3. Compression: Built-in gzip/brotli compression
  4. CDN Integration: Edge caching and global distribution
  5. Database Caching: Redis integration for session management

This architecture documentation reflects the current state of IronDrop v2.7.0 and serves as a foundation for understanding the system's design principles, implementation details, and operational characteristics.