Docker Compose for Apache Flink and Kafka

This docker-compose.yml file sets up an environment with Apache Flink and Kafka. The setup includes a Flink JobManager, TaskManager, and a Kafka broker configured to communicate within the same Docker network. This environment is suitable for streaming applications where Flink processes data from Kafka topics.

Services

1. JobManager

• Image: flink-java17
  A custom image for Apache Flink with Java 17.
• Container Name: flink-jobmanager
• Ports: Exposes port 8081 for accessing the Flink Web UI.
• Command: Runs the Flink JobManager in foreground mode.
• Environment Variables:
  • JOB_MANAGER_RPC_ADDRESS: Sets the JobManager RPC address.
  • TASK_MANAGER_NUMBER_OF_TASK_SLOTS: Configures the number of task slots per TaskManager.
  • JAVA_HOME and PATH: Specifies the path for Java 17.
  • FLINK_JVM_OPTIONS: Adds JVM options to avoid Java 17 module access issues.
• Volumes:
  • flink-conf.yaml: Configures Flink's runtime environment.
  • application.properties: Custom application configurations.
• Networks: Connected to flink-kafka-network.

2. TaskManager

• Image: flink-java17
  Uses the same custom Flink image as JobManager.
• Container Name: flink-taskmanager
• Depends On: Starts after JobManager.
• Command: Runs the Flink TaskManager in foreground mode.
• Environment Variables:
  • Similar to JobManager to ensure compatibility.
• Volumes:
  • Shares the same configuration files as JobManager.
• Networks: Connected to flink-kafka-network.

3. Kafka

• Image: bitnami/kafka
  A Kafka image configured for Kraft (Kafka Raft).
• Container Name: kraft-kafka
• Ports: Exposes port 9092 for Kafka clients.
• Environment Variables:
  • Configures Kafka in Kraft mode with roles as broker and controller.
  • Sets up listeners for PLAINTEXT communication.
  • Advertises the Kafka broker as kraft-kafka:9092.
  • Creates a topic named quickstart-events with 1 partition and a replication factor of 1.
• Networks: Connected to flink-kafka-network.

Network

• flink-kafka-network: A bridge network that allows Flink and Kafka to communicate.

Usage Instructions

1. Start the Environment: Run the following command to start the services:

   docker-compose up -d