Skip to content

Latest commit

 

History

History
1006 lines (828 loc) · 38.1 KB

File metadata and controls

1006 lines (828 loc) · 38.1 KB

py_rt System Architecture: Python-Rust Separation

Executive Summary

The py_rt algorithmic trading system is designed with a clear separation between:

  • Python (Offline): Research, backtesting, analysis, and strategy development
  • Rust (Online): Low-latency order execution, market data processing, and risk management

This architecture maximizes Python's productivity for research while leveraging Rust's performance for production trading.


1. Python Offline Architecture

1.1 Backtesting Framework

python-trading/
├── backtesting/
│   ├── engine.py           # Core backtesting engine
│   ├── historical_replay.py # Time-series data replay
│   ├── event_processor.py  # Event-driven simulation
│   ├── slippage_models.py  # Transaction cost models
│   └── performance.py      # Strategy performance metrics

Components:

  • Historical Data Replay: Tick-by-tick or bar-based replay with configurable time compression
  • Event-Driven Simulation: Order matching, fills, and market impact simulation
  • Slippage Models: Realistic transaction cost modeling (spread, market impact, latency)
  • Performance Analytics: Sharpe ratio, max drawdown, win rate, profit factor

Key Features:

  • Vectorized operations using NumPy/Pandas for speed
  • Out-of-sample validation with walk-forward analysis
  • Multi-asset, multi-strategy backtesting
  • Custom commission and slippage models

1.2 Strategy Parameter Optimization

python-trading/
├── optimization/
│   ├── grid_search.py      # Exhaustive grid search
│   ├── genetic.py          # Genetic algorithms
│   ├── bayesian.py         # Bayesian optimization
│   ├── walk_forward.py     # Walk-forward optimization
│   └── objectives.py       # Objective functions (Sharpe, Calmar, etc.)

Optimization Methods:

  • Grid Search: Exhaustive parameter sweep (parallelized)
  • Genetic Algorithms: Population-based optimization for complex parameter spaces
  • Bayesian Optimization: Sample-efficient optimization using Gaussian processes
  • Walk-Forward: Rolling window optimization to prevent overfitting

Libraries:

  • optuna for Bayesian optimization
  • scipy.optimize for gradient-based methods
  • multiprocessing for parallel grid search

1.3 Statistical Analysis and Visualization

python-trading/
├── analysis/
│   ├── statistics.py       # Statistical tests and metrics
│   ├── attribution.py      # Performance attribution
│   ├── risk_metrics.py     # VaR, CVaR, beta, correlation
│   └── visualization.py    # Interactive charts (Plotly/Bokeh)

Analysis Tools:

  • Time Series Analysis: Stationarity tests, autocorrelation, seasonality
  • Risk Metrics: Value at Risk (VaR), Conditional VaR, beta, Sharpe ratio
  • Attribution Analysis: Factor exposure, alpha/beta decomposition
  • Monte Carlo Simulation: Distribution of outcomes, stress testing

Visualization:

  • Equity curves with drawdown overlays
  • Trade distribution heatmaps
  • Parameter sensitivity surfaces
  • Correlation matrices
  • Portfolio composition over time

1.4 Machine Learning Pipelines

python-trading/
├── ml/
│   ├── features/
│   │   ├── technical.py    # Technical indicators
│   │   ├── fundamental.py  # Fundamental features
│   │   ├── sentiment.py    # NLP sentiment features
│   │   └── alternative.py  # Alternative data features
│   ├── models/
│   │   ├── classifiers.py  # Direction prediction (XGBoost, LightGBM)
│   │   ├── regressors.py   # Price/return prediction
│   │   ├── reinforcement.py # RL agents (PPO, DQN)
│   │   └── ensemble.py     # Model ensembling
│   └── pipeline.py         # End-to-end training pipeline

ML Stack:

  • Feature Engineering: 200+ technical/fundamental/alternative features
  • Models: XGBoost, LightGBM, neural networks (PyTorch), RL agents
  • Cross-Validation: Time-series CV with purging/embargo
  • Model Export: ONNX format for Rust integration

1.5 Research and Experimentation

python-trading/
├── research/
│   ├── notebooks/          # Jupyter notebooks for exploration
│   ├── factor_analysis.py  # Factor discovery and testing
│   ├── regime_detection.py # Market regime identification
│   └── correlation.py      # Cross-asset correlation studies

Research Tools:

  • Jupyter notebooks for interactive exploration
  • Factor backtesting framework (Alphalens-style)
  • Market regime detection (HMM, clustering)
  • Hypothesis testing and statistical validation

1.6 Data Preprocessing and Feature Engineering

python-trading/
├── data/
│   ├── ingestion/
│   │   ├── market_data.py  # Price/volume data
│   │   ├── fundamental.py  # Earnings, ratios
│   │   └── alternative.py  # News, social media, web scraping
│   ├── cleaning.py         # Outlier detection, missing data
│   ├── normalization.py    # Scaling, standardization
│   └── storage.py          # Parquet/HDF5 storage

Data Pipeline:

  • Ingestion: REST APIs, web scraping, database connections
  • Cleaning: Outlier detection, missing value imputation, survivorship bias correction
  • Normalization: Z-score, min-max, robust scaling
  • Storage: Efficient columnar storage (Parquet, HDF5)

2. Rust Online Architecture

2.1 Market Data Ingestion (WebSocket)

rust/
├── market_data/
│   ├── websocket.rs        # WebSocket client (tokio-tungstenite)
│   ├── parser.rs           # Message parsing (JSON/binary)
│   ├── orderbook.rs        # L2/L3 order book reconstruction
│   ├── aggregator.rs       # Multi-source data aggregation
│   └── feed_handler.rs     # Feed normalization

Architecture:

Exchange WebSocket → Parser → Order Book → Signal Generator → Order Manager
                                    ↓
                            Market Data Cache (Shared Memory)

Key Features:

  • Async I/O: Tokio-based async runtime for concurrent WebSocket connections
  • Zero-Copy Parsing: Efficient binary parsing with serde and custom parsers
  • Order Book Management: Fast L2/L3 order book with FIFO/priority queues
  • Data Normalization: Unified interface across exchanges (Binance, Coinbase, Kraken)

Performance Targets:

  • WebSocket message processing: < 10 μs
  • Order book update: < 5 μs
  • End-to-end latency: < 100 μs (message receipt to signal)

2.2 Order Execution Engine

rust/
├── execution/
│   ├── order_manager.rs    # Order lifecycle management
│   ├── router.rs           # Smart order routing
│   ├── algo_execution.rs   # TWAP, VWAP, iceberg
│   ├── fill_simulator.rs   # Latency/slippage simulation
│   └── exchange_api.rs     # REST/WebSocket API clients

Order Flow:

Strategy Signal → Order Manager → Router → Exchange API
                        ↓
                  Risk Manager (pre-trade checks)
                        ↓
                  Position Tracker (post-fill)

Execution Algorithms:

  • Market Orders: Immediate execution with slippage control
  • Limit Orders: Passive order placement with time-in-force
  • TWAP/VWAP: Time/volume-weighted average price execution
  • Iceberg: Hidden order splitting for large positions
  • Smart Routing: Best execution across multiple venues

Safety Features:

  • Pre-trade risk checks (position limits, concentration)
  • Order rate limiting (exchange API limits)
  • Fill reconciliation (expected vs actual)
  • Emergency stop mechanisms

2.3 Risk Management System

rust/
├── risk/
│   ├── pre_trade.rs        # Pre-trade risk checks
│   ├── post_trade.rs       # Post-trade monitoring
│   ├── var_calculator.rs   # Value at Risk
│   ├── position_limits.rs  # Position/concentration limits
│   └── margin_monitor.rs   # Margin requirements

Risk Checks:

Pre-Trade:

  • Position limit enforcement (per symbol, per sector, portfolio-wide)
  • Order size validation (max notional, max % of volume)
  • Margin availability check
  • Concentration risk (max % of portfolio in single asset)

Post-Trade:

  • Real-time P&L monitoring
  • VaR calculation (parametric, historical, Monte Carlo)
  • Drawdown tracking (stop trading on breach)
  • Correlation risk (portfolio-wide)

Configuration:

struct RiskLimits {
    max_position_size: f64,        // Max position in USD
    max_portfolio_concentration: f64, // Max % in single asset
    max_drawdown_pct: f64,         // Stop trading threshold
    var_limit_1d_95: f64,          // 1-day 95% VaR limit
    margin_buffer: f64,            // Required margin buffer
}

2.4 Position Tracking

rust/
├── position/
│   ├── tracker.rs          # Real-time position tracking
│   ├── pnl.rs              # Mark-to-market P&L
│   ├── reconciliation.rs   # Position reconciliation
│   └── reporting.rs        # Position reports

Position State:

struct Position {
    symbol: String,
    quantity: f64,              // Signed (positive = long)
    avg_entry_price: f64,
    current_price: f64,
    unrealized_pnl: f64,
    realized_pnl: f64,
    last_update: Instant,
}

Features:

  • Real-time position updates on fills
  • Mark-to-market P&L calculation
  • Multi-exchange position aggregation
  • Position reconciliation (system vs exchange)

2.5 Real-Time Signal Processing

rust/
├── signals/
│   ├── processor.rs        # Signal computation
│   ├── indicators.rs       # Technical indicators (Rust impl)
│   ├── ml_inference.rs     # ONNX model inference
│   └── aggregator.rs       # Multi-signal aggregation

Signal Pipeline:

Market Data → Indicators → ML Model → Signal Aggregation → Order Decision
                              ↓
                        Signal Cache (Ring Buffer)

Technical Indicators (Rust):

  • Moving averages (SMA, EMA, WMA)
  • Momentum (RSI, MACD, Stochastic)
  • Volatility (ATR, Bollinger Bands)
  • Volume (OBV, VWAP)

ML Inference:

  • ONNX runtime for model inference
  • Feature vector construction from market data
  • Sub-millisecond inference latency

2.6 Message Broker Integration (ZeroMQ)

rust/
├── messaging/
│   ├── zmq_publisher.rs    # Publish market data, signals, fills
│   ├── zmq_subscriber.rs   # Subscribe to commands, config updates
│   ├── protocol.rs         # Message serialization (Protocol Buffers)
│   └── router.rs           # Message routing logic

ZeroMQ Topology:

Rust (Publisher) → ZMQ PUB socket → Python (Subscriber)
                                     ↓
                            Market data, signals, fills

Python (Publisher) → ZMQ PUB socket → Rust (Subscriber)
                                      ↓
                             Strategy updates, commands

Message Types:

  • Market Data: Tick data, order book snapshots, trades
  • Signals: Trading signals, strategy state
  • Orders: Order requests, modifications, cancellations
  • Fills: Execution reports
  • Commands: Start/stop trading, update parameters

Serialization: Protocol Buffers for efficient binary serialization


3. Python-Rust Integration Layer

3.1 PyO3 Bindings

rust/
├── python_bindings/
│   ├── lib.rs              # PyO3 module definition
│   ├── backtesting.rs      # Expose backtesting functions
│   ├── indicators.rs       # Expose technical indicators
│   ├── risk.rs             # Expose risk calculations
│   └── data.rs             # Expose data structures

Exposed Functions:

#[pyfunction]
fn calculate_sma(prices: Vec<f64>, window: usize) -> PyResult<Vec<f64>> {
    // Fast SMA calculation in Rust
}

#[pyfunction]
fn backtest_strategy(
    data: &PyDataFrame,
    strategy: &PyStrategy,
    config: BacktestConfig
) -> PyResult<BacktestResults> {
    // High-performance backtesting
}

#[pymodule]
fn py_rt_core(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(calculate_sma, m)?)?;
    m.add_function(wrap_pyfunction!(backtest_strategy, m)?)?;
    Ok(())
}

Python Usage:

import py_rt_core

# Use Rust-accelerated functions from Python
sma = py_rt_core.calculate_sma(prices, window=20)
results = py_rt_core.backtest_strategy(df, strategy, config)

3.2 Shared Memory IPC

rust/
├── ipc/
│   ├── shared_memory.rs    # Shared memory allocator
│   ├── ring_buffer.rs      # Lock-free ring buffer
│   └── atomic_flags.rs     # Atomic synchronization

Architecture:

Python Process                 Rust Process
      ↓                              ↓
Shared Memory Region (mmap)
      ↓                              ↓
Ring Buffer (lock-free)

Implementation:

  • Memory-Mapped Files: POSIX shared memory (/dev/shm)
  • Lock-Free Data Structures: Ring buffer with atomic read/write pointers
  • Zero-Copy Transfer: Direct memory access without serialization

Use Cases:

  • High-frequency market data streaming (Rust → Python)
  • Real-time P&L updates (Rust → Python)
  • Strategy parameter updates (Python → Rust)

3.3 Protocol Buffers for Data Exchange

proto/
├── market_data.proto       # Market data messages
├── orders.proto            # Order messages
├── positions.proto         # Position snapshots
└── signals.proto           # Signal messages

Example Protocol Buffer Definition:

syntax = "proto3";

message MarketTick {
    string symbol = 1;
    double price = 2;
    double volume = 3;
    int64 timestamp_us = 4;
    string exchange = 5;
}

message OrderRequest {
    string order_id = 1;
    string symbol = 2;
    enum Side { BUY = 0; SELL = 1; }
    Side side = 3;
    double quantity = 4;
    double price = 5;  // 0 for market orders
    enum OrderType { MARKET = 0; LIMIT = 1; }
    OrderType order_type = 6;
}

Benefits:

  • Compact binary encoding (smaller than JSON)
  • Schema evolution (backward/forward compatibility)
  • Code generation for Rust and Python
  • Fast serialization/deserialization

3.4 FFI Safety Boundaries

Safety Principles:

  1. No Raw Pointers Across FFI: Use PyO3's smart pointer wrappers
  2. Error Handling: Rust errors converted to Python exceptions
  3. Memory Management: Rust owns memory, Python gets views/copies
  4. Thread Safety: Synchronization for concurrent access

Example Safe FFI:

#[pyfunction]
fn get_position_snapshot(symbol: &str) -> PyResult<PositionSnapshot> {
    POSITION_TRACKER
        .lock()
        .map_err(|e| PyErr::new::<PyRuntimeError, _>(format!("Lock error: {}", e)))?
        .get_position(symbol)
        .ok_or_else(|| PyErr::new::<PyValueError, _>("Position not found"))
        .map(|pos| pos.clone())  // Clone to avoid lifetime issues
}

3.5 Performance Monitoring

rust/
├── monitoring/
│   ├── metrics.rs          # Prometheus metrics
│   ├── latency.rs          # Latency tracking
│   ├── profiler.rs         # CPU/memory profiling
│   └── dashboard.rs        # Real-time dashboard (Grafana)

Metrics Collected:

  • Latency: End-to-end, per component (p50, p95, p99)
  • Throughput: Messages/second, orders/second
  • Resource Usage: CPU, memory, network bandwidth
  • Error Rates: Failed orders, disconnections, exceptions

Monitoring Stack:

  • Prometheus: Metrics collection and storage
  • Grafana: Real-time dashboards
  • Jaeger: Distributed tracing (optional)

4. Architecture Diagrams

4.1 Component Interaction Diagram

┌─────────────────────────────────────────────────────────────────────┐
│                         PYTHON OFFLINE                               │
│                                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │
│  │  Backtesting │  │ Optimization │  │   Analysis   │              │
│  │    Engine    │  │   (Optuna)   │  │ (Stats/Viz)  │              │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘              │
│         │                  │                  │                      │
│  ┌──────▼──────────────────▼──────────────────▼───────┐             │
│  │           ML Pipeline (Feature Eng + Training)      │             │
│  └──────────────────────────┬──────────────────────────┘             │
│                             │                                         │
│                    ┌────────▼────────┐                               │
│                    │  ONNX Model     │                               │
│                    │  Export         │                               │
│                    └────────┬────────┘                               │
└─────────────────────────────┼──────────────────────────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │  Protocol Buffers  │ (Model weights, config)
                    │  ZeroMQ / PyO3     │
                    └─────────┬─────────┘
                              │
┌─────────────────────────────▼──────────────────────────────────────┐
│                          RUST ONLINE                                │
│                                                                      │
│  ┌─────────────────┐     ┌─────────────────┐                       │
│  │  Market Data    │     │  ML Inference   │                       │
│  │  WebSocket      │────▶│  (ONNX Runtime) │                       │
│  └────────┬────────┘     └────────┬────────┘                       │
│           │                       │                                 │
│           │              ┌────────▼────────┐                        │
│           │              │  Signal         │                        │
│           └─────────────▶│  Processor      │                        │
│                          └────────┬────────┘                        │
│                                   │                                 │
│                          ┌────────▼────────┐                        │
│                          │  Risk Manager   │                        │
│                          │  (Pre-Trade)    │                        │
│                          └────────┬────────┘                        │
│                                   │                                 │
│                          ┌────────▼────────┐                        │
│                          │  Order Manager  │                        │
│                          │  & Execution    │                        │
│                          └────────┬────────┘                        │
│                                   │                                 │
│                          ┌────────▼────────┐                        │
│                          │  Position       │                        │
│                          │  Tracker        │                        │
│                          └─────────────────┘                        │
│                                   │                                 │
│                          ┌────────▼────────┐                        │
│                          │  ZMQ Publisher  │                        │
│                          │  (Fills, P&L)   │                        │
│                          └─────────────────┘                        │
└─────────────────────────────────────────────────────────────────────┘

4.2 Data Flow Diagram (Offline → Online)

RESEARCH PHASE (Python)
  │
  ├─▶ Historical Data → Feature Engineering → ML Training
  │                                               │
  │                                               ▼
  │                                         ONNX Model Export
  │                                               │
  ├─▶ Backtesting → Parameter Optimization ───────┘
  │        │                                      │
  │        ▼                                      │
  │   Performance Analysis                       │
  │   (Accept/Reject Strategy)                   │
  │                                               │
  └─────────────────────────────────────────────▶│
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Strategy Config   │
                                        │  + ONNX Model      │
                                        └─────────┬─────────┘
                                                  │
                                                  ▼
                                        PRODUCTION (Rust)
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Load Model/Config│
                                        └─────────┬─────────┘
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Market Data      │
                                        │  Stream           │
                                        └─────────┬─────────┘
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Signal Generation│
                                        └─────────┬─────────┘
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Order Execution  │
                                        └─────────┬─────────┘
                                                  │
                                        ┌─────────▼─────────┐
                                        │  Real-time P&L    │
                                        │  Monitoring       │
                                        └───────────────────┘

4.3 Latency-Critical Paths

CRITICAL PATH (Target: < 100 μs end-to-end)

WebSocket Message ────▶ Parser ────▶ Order Book ────▶ Signal ────▶ Order
    Arrival          (< 10 μs)      Update        Computation   Submission
                                   (< 5 μs)        (< 50 μs)     (< 30 μs)
                                                      │
                                                      ▼
                                              ┌───────────────┐
                                              │ Risk Check    │
                                              │ (< 10 μs)     │
                                              └───────────────┘

OPTIMIZATIONS:
- Zero-copy message parsing
- Lock-free order book updates
- Pre-compiled ONNX models
- Inline risk checks
- Connection pooling (WebSocket)
- CPU pinning (critical threads)

4.4 Deployment Topology

┌─────────────────────────────────────────────────────────────────┐
│                        TRADING SERVER                            │
│                                                                  │
│  ┌──────────────────┐              ┌──────────────────┐         │
│  │  Python Process  │◀──ZMQ IPC───▶│  Rust Process    │         │
│  │                  │              │                  │         │
│  │  - Research      │              │  - Market Data   │         │
│  │  - Backtesting   │              │  - Execution     │         │
│  │  - Monitoring    │              │  - Risk Mgmt     │         │
│  │  - Dashboard     │              │  - Position Mgmt │         │
│  └──────────────────┘              └──────────────────┘         │
│         │                                    │                  │
│         │                                    │                  │
│         ▼                                    ▼                  │
│  ┌──────────────────────────────────────────────────┐          │
│  │          Shared Memory (Market Data Cache)        │          │
│  └──────────────────────────────────────────────────┘          │
│                                                                  │
└──────────────────────────────┬───────────────────────────────────┘
                               │
                               ▼
                     ┌─────────────────┐
                     │  Internet       │
                     └─────────┬───────┘
                               │
         ┌─────────────────────┼─────────────────────┐
         │                     │                     │
         ▼                     ▼                     ▼
┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
│  Exchange A     │  │  Exchange B     │  │  Data Provider  │
│  (Binance)      │  │  (Coinbase)     │  │  (Historical)   │
│  WebSocket      │  │  WebSocket      │  │  REST API       │
└─────────────────┘  └─────────────────┘  └─────────────────┘

DEPLOYMENT CONFIGURATION:
- Colocation: Server in same datacenter as exchange (minimize latency)
- CPU Pinning: Critical Rust threads pinned to specific cores
- NUMA Awareness: Memory allocated close to processing cores
- Network: 10 Gbps low-latency network interface
- OS: Linux with real-time kernel patches (PREEMPT_RT)

5. Technology Stack

5.1 Python Stack

Component Technology Purpose
Data Analysis Pandas, NumPy Data manipulation, vectorized ops
Visualization Plotly, Matplotlib Interactive charts, analysis
ML Framework PyTorch, XGBoost Neural networks, gradient boosting
Optimization Optuna, Scipy Hyperparameter tuning
Backtesting Custom engine Strategy validation
IPC PyZMQ, PyO3 Inter-process communication

5.2 Rust Stack

Component Technology Purpose
Async Runtime Tokio Async I/O, WebSocket handling
WebSocket tokio-tungstenite Exchange WebSocket clients
Serialization serde, prost JSON/Protocol Buffer parsing
ML Inference ort (ONNX Runtime) Model inference
IPC zmq, shared_memory Inter-process communication
Metrics prometheus Performance monitoring

5.3 Cross-Language

Component Technology Purpose
FFI Bindings PyO3 Python-Rust bindings
Serialization Protocol Buffers Data exchange format
Messaging ZeroMQ Pub/sub messaging
Model Format ONNX ML model interchange

6. Performance Characteristics

6.1 Latency Budget

Component Target Latency Notes
WebSocket message receipt < 10 μs Network + parsing
Order book update < 5 μs Lock-free data structure
Signal computation < 50 μs ONNX inference + indicators
Risk check < 10 μs Pre-computed limits
Order submission < 30 μs REST API call
End-to-end < 100 μs Message → Order

6.2 Throughput Targets

Metric Target Notes
Market data msgs/sec 100,000+ Multi-exchange aggregation
Orders/sec 1,000+ Peak order submission rate
Backtesting speed 1M ticks/sec Vectorized Python + Rust
ML inference < 1 ms ONNX optimized models

6.3 Resource Requirements

Resource Development Production
CPU 4 cores 8+ cores (pinned)
RAM 16 GB 32+ GB
Network 100 Mbps 10 Gbps (low latency)
Storage 256 GB SSD 1+ TB NVMe SSD

7. Key Design Decisions

7.1 Why Python for Offline Tasks?

Advantages:

  • Rich ecosystem for data analysis (Pandas, NumPy, Scikit-learn)
  • Fast prototyping and iteration
  • Excellent visualization libraries
  • Large ML/data science community

Acceptable Trade-offs:

  • Slower execution (mitigated by vectorization and Rust acceleration)
  • GIL limitations (not critical for offline batch processing)

7.2 Why Rust for Online Tasks?

Advantages:

  • Guaranteed memory safety without garbage collection
  • Zero-cost abstractions (no runtime overhead)
  • Fearless concurrency (compile-time race detection)
  • Predictable performance (no GC pauses)

Critical for Trading:

  • Sub-millisecond latency requirements
  • 24/7 operation without crashes
  • Efficient resource usage (CPU, memory, network)

7.3 Integration Strategy

Choice: Hybrid (PyO3 + ZeroMQ + Shared Memory)

Method Use Case Latency Complexity
PyO3 Calling Rust functions from Python Low Medium
ZeroMQ Async messaging (fills, signals) Medium Low
Shared Memory High-frequency data streaming Very Low High

Decision: Use all three for different scenarios

  • PyO3 for function calls (backtesting acceleration)
  • ZeroMQ for event-driven messaging (fills, commands)
  • Shared memory for ultra-low-latency data (market data feed)

8. Development Workflow

8.1 Strategy Development Cycle

1. Research (Python)
   ├─▶ Hypothesis formulation
   ├─▶ Data exploration (Jupyter)
   ├─▶ Feature engineering
   └─▶ Preliminary backtesting

2. Optimization (Python)
   ├─▶ Parameter grid search
   ├─▶ Walk-forward validation
   └─▶ Out-of-sample testing

3. Production (Rust)
   ├─▶ Model export (ONNX)
   ├─▶ Strategy configuration
   ├─▶ Paper trading validation
   └─▶ Live deployment

4. Monitoring (Python + Rust)
   ├─▶ Real-time P&L tracking
   ├─▶ Performance analytics
   └─▶ Anomaly detection

8.2 Testing Strategy

Python Tests:

  • Unit tests for backtesting engine
  • Integration tests for data pipeline
  • Validation tests for ML models (out-of-sample)

Rust Tests:

  • Unit tests for order book, execution logic
  • Integration tests for WebSocket handling
  • Fuzz testing for message parsing
  • Benchmark tests for latency validation

End-to-End Tests:

  • Simulated trading scenarios (Python → Rust)
  • Paper trading with live data
  • Stress testing (high message rate)

9. Deployment Checklist

9.1 Pre-Production

  • Backtest strategy on 3+ years of data
  • Walk-forward optimization completed
  • Out-of-sample validation passed (Sharpe > 1.5)
  • Risk limits configured and tested
  • Paper trading results verified (1+ month)
  • Latency benchmarks met (< 100 μs)
  • Failover mechanisms tested
  • Monitoring dashboards configured

9.2 Production Launch

  • Colocation server provisioned
  • Exchange API keys configured (separate for prod)
  • Risk limits enabled (conservative at start)
  • Position size limits reduced (10% of target)
  • Real-time monitoring active
  • On-call engineer available
  • Emergency stop procedures documented

9.3 Post-Launch

  • Daily P&L reconciliation
  • Weekly performance review
  • Monthly strategy retraining
  • Quarterly risk model validation

10. Future Enhancements

10.1 Short-Term (3-6 months)

  • Multi-strategy portfolio optimization
  • Advanced execution algorithms (POV, adaptive TWAP)
  • GPU acceleration for ML training
  • Reinforcement learning for execution

10.2 Long-Term (6-12 months)

  • Multi-asset class support (equities, futures, options)
  • High-frequency market making strategies
  • Distributed backtesting (Kubernetes cluster)
  • Alternative data integration (sentiment, satellite imagery)

11. Contact and Resources

Architecture Owner: System Architecture Team Last Updated: 2025-10-14 Version: 1.0.0

Related Documents:

  • /docs/python-offline-setup.md - Python development environment
  • /docs/rust-online-setup.md - Rust production system
  • /docs/integration-guide.md - Python-Rust integration details
  • /docs/deployment-guide.md - Production deployment procedures

External Resources:


Appendix A: Message Flow Example

Scenario: Market data arrives, signal generated, order submitted

T+0 μs:   WebSocket message arrives (BTC/USD price update)
          ↓
T+10 μs:  Message parsed, order book updated
          ↓
T+15 μs:  Signal processor triggered
          ↓
T+25 μs:  ONNX model inference (features → prediction)
          ↓
T+55 μs:  Signal aggregation (combine with other signals)
          ↓
T+60 μs:  Order decision made (BUY 0.1 BTC @ market)
          ↓
T+65 μs:  Pre-trade risk check (position limits, margin)
          ↓
T+70 μs:  Order submitted to exchange WebSocket
          ↓
T+500 μs: Exchange ACK received
          ↓
T+2 ms:   Fill confirmation received
          ↓
T+2.1 ms: Position updated, P&L calculated
          ↓
T+2.2 ms: Fill published to Python via ZMQ

Total latency: 70 μs (decision) + 430 μs (network) = 500 μs (well under 1 ms target)


Appendix B: Configuration Examples

B.1 Risk Limits Configuration

# config/risk_limits.toml

[position_limits]
max_position_usd = 100000.0          # $100k max position
max_portfolio_concentration = 0.25    # 25% max in single asset
max_leverage = 3.0                    # 3x max leverage

[risk_metrics]
var_limit_1d_95 = 5000.0             # $5k max 1-day VaR (95%)
max_drawdown_pct = 0.15               # 15% max drawdown (stop trading)

[order_limits]
max_order_size_usd = 10000.0         # $10k max single order
max_order_rate_per_min = 60          # Max 60 orders/min

[margin]
margin_buffer = 0.20                  # 20% margin buffer
margin_call_threshold = 0.05          # 5% to margin call

B.2 Strategy Configuration

# config/strategy_btc_momentum.toml

[strategy]
name = "BTC Momentum"
version = "1.2.0"
asset = "BTC/USD"
exchange = "binance"

[parameters]
lookback_period = 20                  # 20-bar lookback
entry_threshold = 0.65                # ML model confidence threshold
exit_threshold = 0.45
position_size_pct = 0.10              # 10% of portfolio

[ml_model]
path = "models/btc_momentum_v1.2.onnx"
features = ["rsi_14", "macd", "volume_ratio", "bb_pct"]

[execution]
order_type = "limit"
limit_offset_bps = 5                  # 5 bps from mid
timeout_seconds = 30

End of Architecture Document