Troubleshooting Guide

Common issues and solutions for the py_rt trading system.

Connection Issues
Build Errors
Runtime Errors
Performance Issues
Data Issues
API Issues
Debugging Tools

Connection Issues

ZMQ Connection Failed

Symptom: Services fail to connect via ZeroMQ

Error: Connection refused (os error 111)
Failed to connect to tcp://localhost:5555

Solutions:

Check if publisher is running:

# Check if market data service is running
sudo systemctl status py_rt-market-data

# Or check process
ps aux | grep market-data

Verify port is accessible:

# Check if port is listening
netstat -tlnp | grep 5555
# or
ss -tlnp | grep 5555

Check firewall rules:

# Allow ZMQ ports
sudo ufw allow 5555/tcp
sudo ufw allow 5556/tcp
sudo ufw allow 5557/tcp
sudo ufw allow 5558/tcp

Verify bind address in config:

{
  "zmq_pub_address": "tcp://*:5555",  // Bind to all interfaces
  "zmq_sub_address": "tcp://localhost:5555"  // Connect to localhost
}

Alpaca WebSocket Connection Failed

Symptom: Market data service cannot connect to Alpaca

WebSocket error: tungstenite::Error(Http(Response { status: 401 }))

Solutions:

Verify API credentials:

# Check environment variables
echo $ALPACA_API_KEY
echo $ALPACA_SECRET_KEY

# Test credentials
curl -u $ALPACA_API_KEY:$ALPACA_SECRET_KEY \
  https://paper-api.alpaca.markets/v2/account

Check API URL:

# Paper trading (default)
ALPACA_BASE_URL=https://paper-api.alpaca.markets

# Live trading (caution!)
ALPACA_BASE_URL=https://api.alpaca.markets

Verify account status:

# Check account in Alpaca dashboard
# Ensure trading is enabled

Build Errors

Rust Compilation Errors

Symptom: Cargo build fails

error[E0599]: no method named `send` found for type `Socket`

Solutions:

Update dependencies:

cd rust
cargo update
cargo build --release

Clean build cache:

cargo clean
cargo build --release

Check Rust version:

rustc --version
# Should be 1.70+

# Update if needed
rustup update

Missing ZMQ Library

Symptom: Linker error about libzmq

= note: /usr/bin/ld: cannot find -lzmq

Solutions:

# Ubuntu/Debian
sudo apt-get install libzmq3-dev

# RHEL/CentOS
sudo yum install zeromq-devel

# macOS
brew install zeromq

# Then rebuild
cargo clean
cargo build --release

Python Package Installation Errors

Symptom: uv sync fails

error: Failed to download package

Solutions:

Update uv:

pip install --upgrade uv

Clear cache:

uv cache clean
uv sync --reinstall

Check Python version:

python --version
# Should be 3.11+

# Use specific version
uv python install 3.11

Runtime Errors

Panic in Rust Component

Symptom: Service crashes with panic

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value'

Solutions:

Enable backtrace:

RUST_BACKTRACE=1 ./market-data
# or
RUST_BACKTRACE=full ./market-data

Check logs:

# Systemd journal
sudo journalctl -u py_rt-market-data -n 100 --no-pager

# Log files
tail -f /var/log/py_rt/market-data.log

Run in debug mode:

RUST_LOG=debug cargo run --bin market-data

Memory Leak

Symptom: Memory usage grows continuously

Solutions:

Monitor memory:

# Check current usage
ps aux | grep market-data

# Monitor over time
watch -n 5 'ps aux | grep py_rt'

Profile with valgrind:

cargo build
valgrind --leak-check=full ./target/debug/market-data

Use heaptrack:

heaptrack ./target/release/market-data
heaptrack_gui heaptrack.market-data.*.gz

Deadlock

Symptom: Service hangs, no progress

Solutions:

Get stack trace:

# Find PID
ps aux | grep market-data

# Send SIGQUIT to print stack trace
kill -3 <PID>

# Check logs for stack trace

Use GDB:

# Attach to running process
gdb -p <PID>

# Get backtrace
(gdb) thread apply all bt

Performance Issues

High Latency

Symptom: Orders execute slowly (>100ms)

Solutions:

Check system load:

# CPU usage
top

# I/O wait
iostat -x 1

# Network latency
ping data.alpaca.markets

Optimize Rust build:

[profile.release]
opt-level = 3
lto = "fat"
codegen-units = 1

Tune ZMQ buffers:

socket.set_sndhwm(10000)?;
socket.set_rcvhwm(10000)?;
socket.set_tcp_nodelay(true)?;

Low Throughput

Symptom: Processing <1000 messages/second

Solutions:

Profile with perf:

# Record
sudo perf record -g ./target/release/market-data

# Analyze
sudo perf report

Use flamegraph:

cargo install flamegraph
cargo flamegraph --bin market-data

Check async runtime:

// Increase worker threads
tokio::runtime::Builder::new_multi_thread()
    .worker_threads(8)
    .enable_all()
    .build()?

Data Issues

Missing Market Data

Symptom: No updates received

Solutions:

Check subscription:

// Verify symbols are subscribed
let symbols = vec!["AAPL", "MSFT"];
ws_client.subscribe(&symbols).await?;

Check market hours:

# Market data only during trading hours
# NYSE: 9:30 AM - 4:00 PM ET

Verify data plan:

# Check Alpaca account
# Ensure data subscription is active

Stale Data

Symptom: Data is delayed or old

Solutions:

Check timestamp:

let age = SystemTime::now() - bar.timestamp;
if age > Duration::from_secs(60) {
    warn!("Stale data: {:?}", age);
}

Monitor WebSocket connection:

// Add heartbeat
tokio::spawn(async move {
    let mut interval = tokio::time::interval(Duration::from_secs(30));
    loop {
        interval.tick().await;
        ws.send_heartbeat().await?;
    }
});

API Issues

Rate Limit Exceeded

Symptom: API requests rejected with 429

ApiError: Rate limit exceeded (429)

Solutions:

Check rate limiter:

use governor::{Quota, RateLimiter};

let limiter = RateLimiter::direct(
    Quota::per_minute(nonzero!(200u32))
);

// Before each request
limiter.until_ready().await;

Reduce request frequency:

// Batch orders
let orders = vec![order1, order2, order3];
let results = execute_batch(orders).await?;

Authentication Errors

Symptom: 401 Unauthorized

Solutions:

Verify credentials:

# Test with curl
curl -v \
  -H "APCA-API-KEY-ID: $ALPACA_API_KEY" \
  -H "APCA-API-SECRET-KEY: $ALPACA_SECRET_KEY" \
  https://paper-api.alpaca.markets/v2/account

Check credentials format:

# Should be alphanumeric strings
echo $ALPACA_API_KEY | wc -c
# Should be 20+ characters

Order Rejection

Symptom: Orders rejected by broker

Solutions:

Check order parameters:

// Ensure valid order type
let order = Order {
    symbol: "AAPL".to_string(),
    side: Side::Buy,
    quantity: 100,
    order_type: OrderType::Limit,
    limit_price: Some(150.0),  // Required for Limit orders
};

Verify buying power:

let account = client.get_account().await?;
if order_value > account.buying_power {
    return Err(Error::InsufficientFunds);
}

Check position limits:

let position = positions.get(&symbol);
if position.quantity + order.quantity > max_position_size {
    return Err(Error::PositionLimit);
}

Debugging Tools

Logging

// Enable debug logging
use tracing::{debug, info, warn, error};

#[instrument(skip(data))]
async fn process_trade(trade: Trade) {
    debug!("Processing trade: {:?}", trade);
    // ...
    info!("Trade processed successfully");
}

// Run with debug logs
RUST_LOG=debug cargo run

Metrics

use prometheus::{Counter, Histogram, register_counter, register_histogram};

lazy_static! {
    static ref TRADES_PROCESSED: Counter = register_counter!(
        "trades_processed_total",
        "Total number of trades processed"
    ).unwrap();

    static ref ORDER_LATENCY: Histogram = register_histogram!(
        "order_latency_seconds",
        "Order execution latency"
    ).unwrap();
}

// Use in code
TRADES_PROCESSED.inc();
let timer = ORDER_LATENCY.start_timer();
// ... execute order ...
timer.observe_duration();

Distributed Tracing

use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

tracing_subscriber::registry()
    .with(tracing_subscriber::fmt::layer())
    .with(tracing_subscriber::EnvFilter::from_default_env())
    .init();

Health Checks

# Check service health
curl http://localhost:9090/health

# Check metrics
curl http://localhost:9090/metrics

# Check component status
systemctl status py_rt-*

Testing

# Run tests with output
cargo test -- --nocapture --test-threads=1

# Run specific test
cargo test test_order_execution

# Run with logging
RUST_LOG=debug cargo test

Getting Help

If you can't resolve your issue:

Search GitHub Issues
Check GitHub Discussions
Create a new issue with:
- Error message
- Log output
- System information
- Steps to reproduce

Next Steps

Performance Tuning - Optimize system
Monitoring Guide - Set up monitoring
Contributing - Fix issues and contribute

Last Updated: 2025-10-14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Troubleshooting Guide

Table of Contents

Connection Issues

ZMQ Connection Failed

Alpaca WebSocket Connection Failed

Build Errors

Rust Compilation Errors

Missing ZMQ Library

Python Package Installation Errors

Runtime Errors

Panic in Rust Component

Memory Leak

Deadlock

Performance Issues

High Latency

Low Throughput

Data Issues

Missing Market Data

Stale Data

API Issues

Rate Limit Exceeded

Authentication Errors

Order Rejection

Debugging Tools

Logging

Metrics

Distributed Tracing

Health Checks

Testing

Getting Help

Next Steps

FilesExpand file tree

troubleshooting.md

Latest commit

History

troubleshooting.md

File metadata and controls

Troubleshooting Guide

Table of Contents

Connection Issues

ZMQ Connection Failed

Alpaca WebSocket Connection Failed

Build Errors

Rust Compilation Errors

Missing ZMQ Library

Python Package Installation Errors

Runtime Errors

Panic in Rust Component

Memory Leak

Deadlock

Performance Issues

High Latency

Low Throughput

Data Issues

Missing Market Data

Stale Data

API Issues

Rate Limit Exceeded

Authentication Errors

Order Rejection

Debugging Tools

Logging

Metrics

Distributed Tracing

Health Checks

Testing

Getting Help

Next Steps