BetFullX

API Consumption and BigQuery Integration

This project is responsible for consuming data from an API and integrating it with Google BigQuery. It includes functionalities for data extraction, processing, and incremental loading, using Python and libraries such as requests, pandas, and google-cloud-bigquery.

System Requirements

• Python 3.8+
• Google Cloud Platform (GCP) account
• google-cloud-bigquery library
• requests library
• pandas library
• python-dotenv library
• GCP JSON key file for authentication

Setup

1. Clone the repository:

   git clone git@github.com:ricardodev10/BetFullX.git
   cd <repository-name>

2. Install dependencies:

   pip install -r requirements.txt

3. Configure environment variables:
   Create a .env file in the root directory with the following variables:

   API_KEY=<your-api-key>
   API_URL=https://v3.football.api-sports.io
   PROJECT_ID=<gcp-project-id>
   DATASET_NAME=<dataset-name>
   FULL_LOAD_DATE=2023-01-01
   LEAGUE=2
   SEASON=2023

4. Set up the GCP service key:
   Place the JSON key file in the key/ directory and adjust the path in the code:

   key_path = "./key/your-service-key-file.json"

Project Structure

• main.py: Main file for executing the project.
• requirements.txt: Project dependencies list.
• key/: Directory for the GCP JSON key file.
• .env: File for environment variables.

Key Features

1. API Consumption

• Data is extracted from configured endpoints, including:
  • Past fixtures (past_fixtures)
  • Future fixtures (future_fixtures)
  • Player data (players)

2. Data Processing

• Uses pandas for normalizing and processing JSON data.
• Supports nested fields and repeated entries.

3. BigQuery Integration

• Checks for the existence of datasets and tables before loading.
• Supports different write modes:
  • WRITE_APPEND
  • WRITE_TRUNCATE

4. Incremental Loading

• Updates based on the last recorded load date.
• Logs updates in BigQuery.

Execution

To run the project, use the following command:

python main.py

🙂 Feedback

If you have any feedback, please send it to me at ricardodev10@yahoo.com

💛 Author

Made with ♥ by Ricardo Junior 👋

Learning is continuous and there will always be a next level.

    

⬆️ Back to top