Ottawa Falcons FC Stats Tracker

Overview

A Python-based web scraping and API project that:

• Automatically collects soccer statistics from OCSL website
• Stores data in a SQLite database
• Provides stats via API endpoints for Ottawa Falcons soccer club teams

Note: This project is currently under active development. Some features may change or be incomplete.

Tech Stack

• Python
• Flask
• SQLite
• AWS EC2
• Nginx
• Poetry
• Terraform
• GitHub Actions

Local Setup

Initial Setup

1. Ensure poetry is installed. See system requirements/installation guide

2. Install dependencies

poetry install

3. Create local /instance/config.py based on /instance/config.example.py

4. Initialize database

poetry run init_db

5. Seed database with test data

poetry run seed_dev_db

6. Start development server

poetry run dev

Debugging

Use iPdb for interactive debugging:

import ipdb; ipdb.set_trace()

Production Deployment

Automated via GitHub Actions:

• terraform-plan.yml runs on pull requests to main
• terraform-apply.yml runs on merges to main, updates infrastucture
• deploy.yml runs on merges to main, updates production code

Initial Server Setup

1. Install dependencies

sudo dnf install git python3
pip install poetry
sudo dnf install nginx

2. Setup api and scheduler services with systemd

/etc/systemd/system/falcons-stats-api.service

[Unit]
Description=Gunicorn service for Falcons Stats Flask API
After=network.target

[Service]
User=ssm-user
WorkingDirectory=/home/ssm-user/falcons-stats
ExecStart=/home/ssm-user/.local/bin/poetry run gunicorn --workers 2 --bind 0.0.0.0:8080 'falcons_stats:create_app()'
Restart=always

[Install]
WantedBy=multi-user.target

/etc/systemd/system/falcons-stats-scheduler.service

[Unit]
Description=Falcons Stats Scheduler Service
After=network.target

[Service]
User=ssm-user
WorkingDirectory=/home/ssm-user/falcons-stats
ExecStart=/home/ssm-user/.local/bin/poetry run scheduler
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

3. Nginx Reverse Proxy Configuration

sudo apt update
sudo apt install -y nginx

/etc/nginx/conf.d/falcons-stats-api.conf

server {
    listen 80;
    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

4. Copy instance/config.example.py to instance/config.py and update with production values

5. Run these commands to start services:

Enable and Start Services

sudo systemctl daemon-reload
sudo systemctl enable falcons-stats-api falcons-stats-scheduler nginx
sudo systemctl start falcons-stats-api falcons-stats-scheduler nginx

Now, services are handled by systemd and should restart whenever server restarts/crashes.

Check Service Status

sudo systemctl status falcons-stats-api
sudo systemctl status falcons-stats-scheduler

Note: Most deployment tasks are now automated through GitHub Actions

Logging

• Structured, machine-parsable JSON logging (mostly followed best practices from this guide)
• CloudWatch support in production

API Endpoints

Access leading scorers via:

/leading-scorers

Access leading kepers via:

/leading-keepers

Project Roadmap

Current TODOs

• Replace mock data with actual data in scrapers (waiting on season to start for HTML tables to be present)
• Finish adding seeds for all Falcons teams/divs (waiting on all teams to register)
• Better error handling and logging (scrapers, api endpoints)
• Add tests, and include in pipeline

Future Enhancements

• Add support for database migrations
• Scrape more data (schedules, team standings, etc)
• Capture stats from all divisions/teams
• Run scheduler service as more granular background jobs
• Enhance observability with performance metrics, execution tracking, and resource monitoring for scheduled tasks