Skip to content

Latest commit

 

History

History
75 lines (52 loc) · 3.44 KB

README.md

File metadata and controls

75 lines (52 loc) · 3.44 KB

Streaming with Java

A project for comparing blocking versus reactive Java when processing streaming ADS-B (Automatic Dependent Surveillance-Broadcast) data from Kafka. The applications consume messages from a Kafka topic, enrich them with aircraft registration data from MongoDB, and write the results to MongoDB.

Project Structure

Directory Description
streamer/ Go application that streams historical Stratux ADS-B messages to Kafka at a configurable rate
blocking_java/ Blocking Java implementation using Kafka consumer and MongoDB sync driver
reactive_java/ Reactive Java implementation using Reactor Kafka and MongoDB reactive streams driver
run_benchmark.sh Script to run a coordinated benchmark of both Java applications

Prerequisites

  • Java 21+ – for the Java applications
  • Maven – to build the Java applications
  • MongoDB – running locally with:
    • stratux_raw collection (source data for the streamer)
    • registered_aircraft collection (lookup: n_number, manufacturer, model)
    • Sample ADS-B and aircraft registration data is available in streamer/collection_data/ – see streamer/README.md for import instructions and details.
  • Docker – for Kafka (started automatically by the streamer)

Quick Start

  1. Streamer: Build from source (cd streamer && go build -o stratux-simulator .) or use a pre-built executable from streamer/distributions/ (see streamer/README.md).

  2. Sample data (optional): Import sample ADS-B and aircraft registration data from streamer/collection_data/ – see streamer/README.md for details.

  3. Ensure MongoDB is running with the required collections populated.

  4. Run the benchmark script (builds Java apps, starts streamer, runs blocking then reactive for 60 seconds each):

    ./run_benchmark.sh 60

  5. Or run components manually:

    • Start the streamer (after building or using a pre-built binary): cd streamer && ./stratux-simulator --rate 1000
    • Run blocking app: java -jar blocking_java/target/blocking-java-adsb-processor-1.0.0.jar
    • Run reactive app: java -jar reactive_java/target/reactive-java-adsb-processor-1.0.0.jar

Processing Pipeline

Both Java applications implement the same pipeline:

  1. Consume JSON messages from the Kafka stratux topic
  2. Discard messages without a reg (registration) field
  3. Look up registered_aircraft by n_number (matching reg, with optional N prefix stripped)
  4. Add manufacturer and model to the message when a match is found
  5. Write enriched messages to a configurable MongoDB collection

Configuration

Each application loads configuration from a .env file in its directory. Key variables:

Variable Description Default
MONGO_URI MongoDB connection string mongodb://localhost:27017
MONGO_DATABASE MongoDB database Stratux
KAFKA_BROKER Kafka bootstrap servers localhost:9092
KAFKA_TOPIC Kafka topic name stratux
OUTPUT_COLLECTION MongoDB collection for enriched output enriched_adsb

Statistics

On shutdown (Ctrl+C or SIGTERM), both applications print:

  • Messages received from Kafka
  • Messages written to MongoDB
  • Messages rejected (missing reg)

License

This project is licensed under the Apache License, Version 2.0. See LICENSE for the full text.