Skip to content

Shiffi ports and ....... #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
shiffiH wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Shiffi ports and ....... #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
39ebf77
Modify service configurations and ports in docker-compose
shiffiH Nov 13, 2025
a97c01e
Update API base URL for devices updater
shiffiH Nov 13, 2025
17cec9b
Update DEVICES_API_BASE environment variable
shiffiH Nov 13, 2025
2e8f1b0
Update API base URL for token retrieval
shiffiH Nov 13, 2025
1faa9ce
Update API base URL for devices
shiffiH Nov 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 55 additions & 35 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

# ==========================
# Docker Compose - AG Cloud
# ==========================
Expand All @@ -10,8 +11,6 @@ networks:
ag_cloud:
name: ag_cloud
driver: bridge
flink-net:
driver: bridge

# --------------------------
# Volumes
Expand Down Expand Up @@ -87,7 +86,7 @@ services:
dockerfile: Dockerfile
environment:
- ADDR=0.0.0.0
- PORT=8005
- PORT=8003
- USE_UTC=false
- WINDOW_MIN=1
- STABLE_SEC=1
Expand All @@ -101,7 +100,7 @@ services:

command: [ "python", "-u", "src/metrics.py" ]
ports:
- "8005:8005"
- "8003:8003"
depends_on:
- minio-hot
networks:
Expand All @@ -115,8 +114,10 @@ services:

plant_stress_daily:
build: ./services/plant_stress
env_file:
- ./services/plant_stress/.env
restart: "no"
environment:
TZ: "Asia/Jerusalem"
MODEL_DIR: /models
CONFIDENCE_THRESHOLD: "0.60"
TF_CPP_MIN_LOG_LEVEL: "2"
Expand All @@ -128,17 +129,20 @@ services:
MINIO_BUCKET: sound
MINIO_PREFIX: plants/
MINIO_SECURE: "false"

DEFAULT_AREA: unknown
DEFAULT_LAT: 0.0
DEFAULT_LON: 0.0
DEFAULT_IMAGE_URL: https://example.com/placeholder.jpg
DEFAULT_VOD: https://example.com/placeholder.mp4
DEFAULT_HLS: https://example.com/placeholder.m3u8
# ==== Alerts → Kafka ====
ENABLE_ALERTS: "true"
KAFKA_BOOTSTRAP: "kafka:9092"
ALERT_TOPIC: "alerts"
ALERT_TYPE: "plant_drought_detected"
KAFKA_CLIENT_ID: "plant-stress-producer"
command: ["python","-u","/app/predict_minio_daily.py"]
volumes:
- "./services/plant_stress/models:/models:ro"
depends_on:
Expand All @@ -151,7 +155,6 @@ services:
kafka:
condition: service_healthy
networks: [ag_cloud]
restart: unless-stopped

# -------------------------
# MQTT + Kafka + MQTT-router
Expand Down Expand Up @@ -391,7 +394,7 @@ services:
MINIO_NOTIFY_KAFKA_ENABLE_aerial: "on"
MINIO_NOTIFY_KAFKA_BROKERS_aerial: "kafka:9092"
MINIO_NOTIFY_KAFKA_TOPIC_aerial: "image.new.aerial"

MINIO_NOTIFY_KAFKA_ENABLE_air: "on"
MINIO_NOTIFY_KAFKA_BROKERS_air: "kafka:9092"
MINIO_NOTIFY_KAFKA_TOPIC_air: "image.new.air"
Expand Down Expand Up @@ -620,7 +623,7 @@ services:
- mqtt_ingest
networks:
- ag_cloud

mqtt_gateway:
build:
context: ./services/mqtt_gateway
Expand Down Expand Up @@ -670,20 +673,18 @@ services:
- HEAD=/app/classification/models/head/head_cnn14_rf.joblib
- HEAD_META=/app/classification/models/head/head_cnn14_rf.joblib.meta.json

# DB
# DB
- WRITE_DB=false
- DB_URL=postgresql://missions_user:pg123@postgres:5432/missions_db
- DB_SCHEMA=agcloud_audio
- DB_RUN_ID=api-default
- FILES_SCHEMA=public
- FILES_TABLE=sound_new_sounds_connections

# Kafka
# Kafka
- KAFKA_BROKERS=kafka:9092
- ALERTS_TOPIC=alerts
- ENABLE_ALERTS=true

# MinIO
# MinIO
- MINIO_ENDPOINT=minio-hot:9000
- MINIO_ACCESS_KEY=minioadmin
- MINIO_SECRET_KEY=minioadmin123
Expand All @@ -694,7 +695,7 @@ services:
- ALLOWED_CONTENT_TYPES=audio/wav,audio/x-wav,audio/mpeg,audio/flac,audio/ogg,audio/mp4
- MAX_BYTES=104857600

# Tuning params
# Tuning params
- UNKNOWN_THRESHOLD=0.4
- WINDOW_SEC=2.0
- HOP_SEC=0.5
Expand All @@ -708,11 +709,11 @@ services:
mc-bootstrap:
condition: service_started
ports:
- "8088:8088"
- "8056:8056"
networks:
- ag_cloud
healthcheck:
test: [ "CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8088/health').read()" ]
test: [ "CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8056/health').read()" ]
interval: 45s
timeout: 5s
retries: 10
Expand Down Expand Up @@ -813,7 +814,7 @@ services:
ports:
- "8091:8088"
restart: unless-stopped

# --------------------------
# Flink JobManager & TaskManager
# --------------------------
Expand Down Expand Up @@ -844,7 +845,7 @@ services:
python.executable: /usr/bin/python3
- HTTP_INFER_URL=http://fruit-inference-http:8004/infer_json
volumes:
- ./streaming/flink/jobs:/opt/flink/jobs:ro
- ./streaming/flink/jobs:/opt/flink/jobs
Copy link
Copy Markdown

@greptile-apps greptile-apps Bot Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: Jobs volume mount changed from read-only (:ro) to read-write - could allow unintended modifications. Is read-write access needed for the Flink jobs directory?

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

- ./streaming/flink/connectors/flink-json-1.18.1.jar:/opt/flink/lib/flink-json-1.18.1.jar:ro
- ./streaming/flink/connectors/flink-sql-connector-kafka-3.2.0-1.18.jar:/opt/flink/lib/flink-sql-connector-kafka-3.2.0-1.18.jar:ro
- ./streaming/flink/connectors/flink-connector-kafka-3.2.0-1.18.jar:/opt/flink/lib/flink-connector-kafka-3.2.0-1.18.jar:ro
Expand Down Expand Up @@ -911,9 +912,9 @@ services:
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
parallelism.default: 2
taskmanager.numberOfTaskSlots: 2
taskmanager.numberOfTaskSlots: 4
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m
taskmanager.memory.process.size: 2048m
s3.endpoint: http://minio-hot:9000
s3.path.style.access: true
s3.access.key: minioadmin
Expand Down Expand Up @@ -1056,7 +1057,7 @@ services:
networks: [ ag_cloud ]
environment:
- KAFKA_BOOTSTRAP=kafka:9092
- INPUT_TOPIC=imagery.new.fruit
- INPUT_TOPIC=inference.dispatched.camera
- TEAM=fruit
- HTTP_URL=http://fruit-inference-http:8004/infer_json
- DLQ_TOPIC=dlq.inference.http
Expand All @@ -1071,9 +1072,9 @@ services:
- ./streaming/flink/connectors/kafka-clients-3.2.3.jar:/opt/flink/lib/kafka-clients-3.2.3.jar:ro
- ./streaming/flink/connectors/lz4-java-1.8.0.jar:/opt/flink/lib/lz4-java-1.8.0.jar:ro
- ./streaming/flink/connectors/snappy-java-1.1.10.5.jar:/opt/flink/lib/snappy-java-1.1.10.5.jar:ro
command: [ "bash", "-lc", "set -e; echo 'Waiting for JobManager to accept commands...'; until /opt/flink/bin/flink list --jobmanager flink-jobmanager:8081 >/dev/null 2>&1; do echo 'still waiting...'; sleep 3; done; echo 'JobManager is ready!'; /opt/flink/bin/flink run -Dpython.client.executable=/usr/bin/python3 -Dpython.executable=/usr/bin/python3 -Dpipeline.jars=file:///opt/flink/lib/flink-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/flink-sql-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/flink-json-1.18.1.jar --jobmanager flink-jobmanager:8081 --detached --python /opt/flink/jobs/http_dispatcher.py -- --bootstrap kafka:9092 --input-topic imagery.new.fruit --team fruit --http-url http://fruit-inference-http:8004/infer_json --group-id http-dispatcher-fruit --dlq-topic dlq.inference.http; tail -f /dev/null" ]
command: [ "bash", "-lc", "set -e; echo 'Waiting for JobManager to accept commands...'; until /opt/flink/bin/flink list --jobmanager flink-jobmanager:8081 >/dev/null 2>&1; do echo 'still waiting...'; sleep 3; done; echo 'JobManager is ready!'; /opt/flink/bin/flink run -Dpython.client.executable=/usr/bin/python3 -Dpython.executable=/usr/bin/python3 -Dpipeline.jars=file:///opt/flink/lib/flink-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/flink-sql-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/flink-json-1.18.1.jar --jobmanager flink-jobmanager:8081 --detached --python /opt/flink/jobs/http_dispatcher.py -- --bootstrap kafka:9092 --input-topic inference.dispatched.camera --team fruit --http-url http://fruit-inference-http:8004/infer_json --group-id http-dispatcher-fruit --dlq-topic dlq.inference.http; tail -f /dev/null" ]
restart: always

flink-dispatcher-camera:
image: agcloud-flink-py:1.18
container_name: flink-dispatcher-camera
Expand All @@ -1084,7 +1085,7 @@ services:
networks: [ag_cloud]
environment:
- KAFKA_BOOTSTRAP=kafka:9092
- INPUT_TOPIC=imagery.new.camera
- INPUT_TOPIC=image.new.fruits
- TEAM=camera
- HTTP_URL=http://camera-inference-http:8004/infer_json
- DLQ_TOPIC=dlq.inference.http
Expand All @@ -1094,7 +1095,7 @@ services:
volumes:
- ./streaming/flink/jobs:/opt/flink/jobs:ro
- ./streaming/flink/connectors:/opt/flink/lib/connectors:ro
command: [ "bash", "-lc", "set -e; echo 'Waiting for JobManager to accept commands...'; until /opt/flink/bin/flink list --jobmanager flink-jobmanager:8081 >/dev/null 2>&1; do echo 'still waiting...'; sleep 3; done; echo 'JobManager is ready!'; /opt/flink/bin/flink run -Dpython.client.executable=/usr/bin/python3 -Dpython.executable=/usr/bin/python3 -Dpipeline.jars=file:///opt/flink/lib/connectors/flink-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/connectors/flink-sql-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/connectors/flink-json-1.18.1.jar --jobmanager flink-jobmanager:8081 --detached --python /opt/flink/jobs/http_dispatcher.py -- --bootstrap kafka:9092 --input-topic imagery.new.camera --team camera --http-url http://camera-inference-http:8004/infer_json --group-id http-dispatcher-camera --dlq-topic dlq.inference.http; tail -f /dev/null" ]
command: [ "bash", "-lc", "set -e; echo 'Waiting for JobManager to accept commands...'; until /opt/flink/bin/flink list --jobmanager flink-jobmanager:8081 >/dev/null 2>&1; do echo 'still waiting...'; sleep 3; done; echo 'JobManager is ready!'; /opt/flink/bin/flink run -Dpython.client.executable=/usr/bin/python3 -Dpython.executable=/usr/bin/python3 -Dpipeline.jars=file:///opt/flink/lib/connectors/flink-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/connectors/flink-sql-connector-kafka-3.2.0-1.18.jar,file:///opt/flink/lib/connectors/flink-json-1.18.1.jar --jobmanager flink-jobmanager:8081 --detached --python /opt/flink/jobs/http_dispatcher.py -- --bootstrap kafka:9092 --input-topic image.new.fruits --team camera --http-url http://camera-inference-http:8004/infer_json --group-id http-dispatcher-camera --dlq-topic dlq.inference.http; tail -f /dev/null" ]
restart: always

flink-dispatcher-soil:
Expand All @@ -1117,7 +1118,7 @@ services:
- ./streaming/flink/jobs:/opt/flink/jobs:ro
- ./streaming/flink/connectors:/opt/flink/lib/connectors:ro
command: [ "bash", "-lc", "set -e; echo 'Waiting...'; until /opt/flink/bin/flink list --jobmanager flink-jobmanager:8081 >/dev/null 2>&1; do echo 'still waiting...'; sleep 3; done; echo 'JobManager is ready!'; /opt/flink/bin/flink run -Dpython.client.executable=/usr/bin/python3 -Dpython.executable=/usr/bin/python3 -Dpipeline.jars=file:///opt/flink/lib/connectors/... --jobmanager flink-jobmanager:8081 --detached --python /opt/flink/jobs/http_dispatcher.py -- --bootstrap kafka:9092 --input-topic image.new.ground --team soil_moisture --http-url http://soil-inference-http:8004/infer_json --group-id http-dispatcher-soil --dlq-topic dlq.inference.http; tail -f /dev/null" ]


flink-alerts-job:
build:
Expand Down Expand Up @@ -1180,18 +1181,18 @@ services:
container_name: alerts_gateway
command: [ "uvicorn", "gateway:app", "--host", "0.0.0.0", "--port", "8000" ]
ports:
- "8010:8000"
- "8010:8000"
networks:
- ag_cloud

image-linker-jobmanager:
build:
context: ./services/image-linker
dockerfile: Dockerfile.flink
container_name: image-linker-jobmanager
command: jobmanager
ports:
- "8084:8081"
- "8084:8081"
environment:
- JOB_MANAGER_RPC_ADDRESS=image-linker-jobmanager
- KAFKA_BROKERS=kafka:9092
Expand Down Expand Up @@ -1238,7 +1239,7 @@ services:
container_name: flink-sounds-http-jobmanager
command: jobmanager
ports:
- "8083:8081"
- "8083:8081"
environment:
JOB_MANAGER_RPC_ADDRESS: flink-sounds-http-jobmanager
KAFKA_BROKERS: kafka:9092
Expand Down Expand Up @@ -1340,13 +1341,13 @@ services:
sleep 5;
done &&
echo '✅ Reports ready, submitting Flink job...' &&
flink run -m localhost:8091 -py /opt/app/sensorAnomalyPro/app.py &&
flink run -m localhost:8099 -py /opt/app/sensorAnomalyPro/app.py &&
tail -f /dev/null"
depends_on:
- sensor_anomaly_pro
- kafka
ports:
- "8091:8091"
- "8099:8099"
environment:
- JOB_MANAGER_RPC_ADDRESS=jobmanager
- KAFKA_BROKERS=kafka:9092
Expand Down Expand Up @@ -1381,7 +1382,26 @@ services:
- ag_cloud



edge-sensors:
build:
context: ./mqtt_and_kafka/Sensor_edge_device
dockerfile: Dockerfile.edge
container_name: edge-sensors
depends_on:
mosquitto:
condition: service_healthy
environment:
- BROKER=mosquitto
- PORT=1883
- TOPIC=dev-robot-telemetry-raw
networks:
- ag_cloud
restart: unless-stopped





vector_service:
build: ./services/vector_service
container_name: vector_service
Expand Down Expand Up @@ -1409,14 +1429,14 @@ services:
dockerfile: Dockerfile.flink
container_name: sensorguard-jobmanager
ports:
- "8081:8081"
- "8999:8999"
command: >
bash -c "
/docker-entrypoint.sh jobmanager &
echo 'Waiting for Flink JobManager startup...' &&
sleep 15 &&
echo 'Submitting sensorGuard Flink job...' &&
flink run -m localhost:8081 -py /opt/app/main.py &&
flink run -m localhost:8999 -py /opt/app/main.py &&
tail -f /dev/null"
environment:
- JOB_MANAGER_RPC_ADDRESS=sensorguard-jobmanager
Expand Down
2 changes: 1 addition & 1 deletion services/sensorGuard/sensorGuard/flink_app/api/devices_updater.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ def update_device_last_seen(device_id: str):
Updates the 'last_seen' field for a specific device in the devices_sensor table.
Uses PATCH /api/tables/devices_sensor
"""
api_base = "http://host.docker.internal:8001"
api_base = "http://db_api_service:8001"
token = get_access_token(api_base)
headers = {
"X-Service-Token": token,
Expand Down
2 changes: 1 addition & 1 deletion services/sensorGuard/sensorGuard/flink_app/config/settings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

class Settings:
# --- API Configuration ---
DEVICES_API_BASE = os.getenv("DEVICES_API_BASE", "http://host.docker.internal:8001")
DEVICES_API_BASE = os.getenv("DEVICES_API_BASE", "http://db_api_service:8001")
DEVICES_API_TOKEN = os.getenv("DEVICES_API_TOKEN", None)

# --- Kafka Configuration ---
Expand Down
2 changes: 1 addition & 1 deletion services/sensorGuard/sensorGuard/flink_app/core/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def __init__(self, cfg, writer, state: StateStore | None = None):
self.state = state or StateStore()

# --- API info & persistent token ---
self.api_base = "http://host.docker.internal:8001"
self.api_base = "http://db_api_service:8001"
self.token = get_access_token(self.api_base)
if self.token:
print("[ENGINE] Access token acquired successfully.")
Expand Down
2 changes: 1 addition & 1 deletion services/sensorGuard/sensorGuard/flink_app/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ def main():
cfg_path = base_dir / "config" / "rules.yaml"

# --- Load sensors from API ---
api_base = os.getenv("DEVICES_API_BASE", "http://host.docker.internal:8001")
api_base = os.getenv("DEVICES_API_BASE", "http://db_api_service:8001")
print(f"[INIT] Fetching active sensors from {api_base}...")

token = get_access_token(api_base)
Expand Down