Weather API Data Analysis Using Kafka

A real-time weather data streaming and analysis pipeline built with Python and Apache Kafka that fetches weather information from an online API, streams it to Kafka topics, and processes it for analytics or storage.

🧠 Project Overview

This project demonstrates how to build a real-time data pipeline that:

• Collects weather data from a weather API (like OpenWeatherMap)
• Streams incoming data using Apache Kafka messaging
• Processes the streamed data using Kafka consumers
• Enables further analysis or storage of weather insights

The architecture is inspired by common real-time weather data pipelines where sensors/APIs stream data, Kafka acts as a message broker, and consumers process the data downstream. :contentReference[oaicite:0]{index=0}

📁 Repository Contents

• weather-producer.py — Kafka producer that fetches and publishes weather data
• weather-consumer.py — Kafka consumer that reads and processes weather data
• app.py — Entry point script (e.g., to start producer/consumer logic)
• ML model.ipynb — Notebook for any machine learning or analysis on the streamed data
• docker-compose.yaml / Dockerfile — Docker setup for running Kafka and services
• requirements.txt — Python dependencies

📦 Dependencies

Install Python packages:

pip install -r requirements.txt
Key libraries typically include:

confluent_kafka (Kafka client for Python)

requests (for API calls)

pandas (for data processing)

Also make sure Apache Kafka (and Zookeeper if required) is installed and running locally or in Docker.

🚀 How to Run
Start Kafka broker (locally or via Docker Compose):

bash
Copy code
docker-compose up
Run the Kafka producer to fetch & stream weather data:

bash
Copy code
python weather-producer.py
Run the Kafka consumer to read and process the incoming weather events:

bash
Copy code
python weather-consumer.py
(Optional) Use ML model.ipynb for analysis or forecasting.