FARP (Financial Analytics & Risk Pipeline) is designed as a modular, production-grade financial data pipeline that transforms raw trade and market data into actionable analytics.
┌─────────────────────────────────────────────────────────────────────────────┐
│ FARP Architecture │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────────────────────────────────┐ │
│ │ Data Sources │ │ FARP Core │ │
│ ├─────────────────┤ ├──────────────────────────────────────────────┤ │
│ │ │ │ │ │
│ │ • Yahoo Finance │────▶│ ┌─────────────┐ ┌───────────────────┐ │ │
│ │ • CoinGecko │ │ │ Ingestion │───▶│ Analytics Engine │ │ │
│ │ • Trade API │ │ │ Layer │ │ │ │ │
│ │ │ │ └─────────────┘ │ • Position Calc │ │ │
│ └─────────────────┘ │ │ │ • PnL Engine │ │ │
│ │ ▼ │ • Risk Metrics │ │ │
│ ┌─────────────────┐ │ ┌─────────────┐ │ • Exposure Calc │ │ │
│ │ Consumers │ │ │ PostgreSQL │ │ • FX Service │ │ │
│ ├─────────────────┤ │ │ Database │ └───────────────────┘ │ │
│ │ │ │ │ │ │ │ │
│ │ • REST API │◀────│ │ ┌─────────┐ │◀───────────┘ │ │
│ │ • Streamlit UI │ │ │ │ Raw │ │ │ │
│ │ • Scheduler │ │ │ ├─────────┤ │ │ │
│ │ │ │ │ │ Staging │ │ │ │
│ └─────────────────┘ │ │ ├─────────┤ │ │ │
│ │ │ │Analytics│ │ │ │
│ │ │ └─────────┘ │ │ │
│ │ └─────────────┘ │ │
│ └──────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
The ingestion layer is responsible for receiving and normalizing data from external sources.
| Module | Purpose | Source |
|---|---|---|
trade_receiver.py |
Accepts trade submissions via API | User/Systems |
market_data.py |
Fetches prices and FX rates | Yahoo Finance, CoinGecko |
reference_data.py |
Manages instrument/symbol master | Internal |
Key Features:
- Raw data preservation for audit trails
- Data validation and normalization
- Automatic instrument creation
- Deduplication handling
The analytics engine processes raw data into decision-grade metrics.
| Module | Purpose | Key Calculations |
|---|---|---|
position_calc.py |
Position tracking | FIFO cost basis, MTM valuation |
pnl_engine.py |
P&L calculations | Realized/unrealized split, attribution |
risk_metrics.py |
Risk analytics | VaR, Sharpe, volatility, drawdown, beta |
exposure_calc.py |
Exposure analysis | By asset class, currency, sector |
fx_service.py |
Currency conversion | Multi-currency aggregation |
RESTful API built with FastAPI providing:
- Trade management endpoints
- Position and PnL queries
- Risk metrics retrieval
- Market data access
Interactive Streamlit dashboard with:
- Portfolio overview
- PnL visualization
- Risk metrics display
- Position details
Background job scheduler using APScheduler:
- Price updates (every 5 min)
- FX rate updates (every 15 min)
- Position snapshots (every 30 min)
- End-of-day processing
FARP uses a three-layer data architecture that separates concerns and ensures data integrity.
┌─────────────────────────────────────────────────────────────────────┐
│ Database Architecture │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ RAW LAYER │ │
│ │ (Immutable, Sacred) │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ • raw_trades - Original trade records as received │ │
│ │ • raw_market_data - Original market data feeds │ │
│ │ │ │
│ │ ✓ Never modified after creation │ │
│ │ ✓ Complete audit trail │ │
│ │ ✓ Enables data replay │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ STAGING LAYER │ │
│ │ (Financial Hygiene) │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ • instruments - Symbol/asset master │ │
│ │ • trades - Cleaned, deduplicated trades │ │
│ │ • prices - OHLCV price history │ │
│ │ • fx_rates - Currency exchange rates │ │
│ │ • corporate_actions - Splits, dividends │ │
│ │ │ │
│ │ ✓ Normalized and validated │ │
│ │ ✓ Linked via foreign keys │ │
│ │ ✓ Query-optimized indexes │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ANALYTICS LAYER │ │
│ │ (Decision Tables) │ │
│ ├─────────────────────────────────────────────────────────────┤ │
│ │ • positions - Point-in-time position snapshots │ │
│ │ • pnl_snapshots - PnL history (intraday and EOD) │ │
│ │ • risk_metrics - Calculated risk metrics │ │
│ │ • exposures - Exposure breakdowns │ │
│ │ • job_runs - Scheduler job history │ │
│ │ │ │
│ │ ✓ Pre-calculated metrics │ │
│ │ ✓ Time-series friendly │ │
│ │ ✓ Dashboard-ready │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
| Layer | Purpose | Key Principle |
|---|---|---|
| Raw | Audit & Replay | Never modify, preserve everything |
| Staging | Data Quality | Normalize, deduplicate, validate |
| Analytics | Performance | Pre-calculate, snapshot, optimize |
User/System
│
▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Trade API │────▶│ Raw Layer │────▶│ Staging │
│ Endpoint │ │ (raw_trades)│ │ (trades) │
└─────────────┘ └─────────────┘ └─────────────┘
│
┌──────────────────────────┘
▼
┌─────────────┐
│ Analytics │
│ (positions) │
└─────────────┘
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Trades │────▶│ Position │────▶│ PnL │
│ (staging) │ │ Calculator │ │ Engine │
└─────────────┘ └─────────────┘ └─────────────┘
│ │
┌─────────────────────┴───────────────────┘
▼ ▼ ▼
┌─────────┐ ┌─────────────┐ ┌───────────┐
│ FX │◀───────▶│ Risk │────▶│ Exposure │
│ Service │ │ Metrics │ │ Calc │
└─────────┘ └─────────────┘ └───────────┘
| Layer | Technology | Purpose |
|---|---|---|
| Language | Python 3.11+ | Core runtime |
| API | FastAPI | REST API framework |
| Database | PostgreSQL 15+ | Data persistence |
| ORM | SQLAlchemy 2.0+ | Database abstraction |
| Dashboard | Streamlit | Interactive UI |
| Scheduler | APScheduler | Background jobs |
| Data Processing | Pandas, NumPy | Analytics calculations |
| Market Data | yfinance, pycoingecko | External data feeds |
| Configuration | pydantic-settings | Type-safe configuration |
| Logging | structlog | Structured logging |
| Containerization | Docker, Docker Compose | Deployment |
Raw data is never modified. Corrections are stored as new records, enabling full audit trails and data replay.
Each layer has a distinct purpose, making the system easier to maintain, test, and extend.
Analytics are pre-calculated and stored as snapshots, enabling fast queries for dashboards and APIs.
All financial calculations support multi-currency portfolios with automatic FX conversion to a base currency.
New asset classes, data sources, and analytics can be added without modifying existing components.
- Database Schema - Detailed table definitions
- API Reference - API endpoints
- Analytics Engine - Calculation details
- Configuration - System settings