This was AlexU CSE 493: Designing Data Intensive Applications course project.
Distributed Weather Stations Monitoring is a distributed system that fetches weather information from multiple sources, archives and visualizes it.
- Reading from multiple stations using kafka streams.
- Archiving the latest values from each station efficiently using Riak Bitcask.
- Archiving and partitioning the data depending on the weather stations in Parquet files.
- Indexing and Visualizing the archived data using Elastic search and Kibana.
The system is fully deployed using Docker and K8s (Kubernetes) but it might need a decent node/machine to be capable of deploying all the services.