Skip to content

README.md

Latest commit

 

History

History
171 lines (140 loc) · 3.45 KB

README.md

File metadata and controls

171 lines (140 loc) · 3.45 KB

DevQuest.IO - Analytics Service

Overview

This analytics service microservice is a Python-based backend service for the Coding Progress Dashboard, responsible for processing, aggregating, and analyzing coding activity data.

Features

  • LeetCode data scraping and analysis
  • Redis-based caching and task management
  • Celery for asynchronous task processing
  • Comprehensive logging and monitoring
  • Pytest-based testing suite

Tech Stack

  • Python 3.10+
  • FastAPI
  • Celery
  • Redis
  • pytest
  • Docker
  • GraphQL integration

Project Structure

.
├── Dockerfile
├── README.md
├── docker-compose.yaml
├── config.py
├── main.py
├── requirements.txt
├── filebeat/
│   └── filebeat.yml
├── logstash/
│   ├── config/
│   └── pipeline/
├── src/
│   ├── services/
│   ├── models/
│   └── utils/
├── tests/
│   └── test_analytics_service.py
└── pytest.ini

Prerequisites

  • Python 3.10+
  • Docker
  • Docker Compose
  • Redis
  • Celery
  • FastAPI

Environment Setup

Virtual Environment

# Create virtual environment
python -m venv latest

# Activate virtual environment
# On Unix or MacOS
source latest/bin/activate
# On Windows
latest\Scripts\activate

Installation

  1. Clone the repository
git clone https://github.com/your-org/coding-progress-analytics-service.git
cd coding-progress-analytics-service
  1. Install dependencies
pip install -r requirements.txt

Configuration

Create a .env file with the following configurations:

# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379

# Logging Configuration
LOG_LEVEL=INFO
LOG_FILE=devquest.log

# Celery Configuration
CELERY_BROKER_URL=redis://localhost:6379/0
CELERY_RESULT_BACKEND=redis://localhost:6379/0

Running the Service

Development Mode

# Start Redis
docker-compose up redis

# Run main application
python main.py

# Run Celery worker
celery -A tasks worker --loglevel=info

Docker Deployment

# Build docker image
docker-compose build

# Start all services
docker-compose up

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=.

# Generate HTML coverage report
pytest --cov=. --cov-report=html

Key Components

Modules

  • main.py: Primary application entry point
  • tasks.py: Celery background tasks
  • redis_service.py: Redis interaction utilities
  • leetcode_service.py: LeetCode data scraping and processing
  • leetcode_graphql.py: GraphQL interactions
  • models.py: Data models and schemas
  • logging_config.py: Logging configuration

Logging

  • Configured log file: devquest.log
  • Supports different log levels (DEBUG, INFO, WARNING, ERROR)
  • Integrated with Filebeat for log management

Task Processing

  • Celery for asynchronous task management
  • Background tasks for data scraping and processing
  • Redis as message broker and result backend

Monitoring and Observability

  • Filebeat for log shipping
  • Logstash for log processing
  • Configurable logging levels
  • Comprehensive error tracking

Continuous Integration

  • pytest for unit and integration testing
  • Code coverage reporting
  • GitHub Actions workflow (recommended)

Troubleshooting

  • Ensure Redis is running
  • Check .env file configurations
  • Verify Python and dependency versions
  • Review Celery worker logs
  • Check network connectivity

License

Distributed under the MIT License. See LICENSE for more information.