Remove OTel agent from Docker images; use compose volume mounts instead (linkedin#478)

## Summary
Fixes the Docker build failure introduced by the OTel agent COPY (linkedin#474's
`ja[r]` glob
trick still fails when `build/otel/` directory doesn't exist in the
build context).

The OTel Java agent should not be baked into production Docker images —
it is only
needed for ad-hoc local integration testing. This PR removes the agent
from Docker
images entirely and instead provides docker-compose override files to
volume-mount
it when needed.

## How builds work after this change

| Scenario | OTel agent | How |
|----------|-----------|-----|
| **`./gradlew build`** (regular build) | Not downloaded | No OTel in
build or images |
| **CI** (`build-run-tests.yml`, `build-tag-publish.yml`) | Not
downloaded | Docker images build cleanly without OTel |
| **`./gradlew dockerPrereqs`** (local integration) | Downloaded to
`build/otel/` | Available for volume-mounting into containers |
| **Integration testing via docker-compose** | Volume-mounted at runtime
| Use otel-profile compose overrides (see below) |

### Ad-hoc integration testing with tracing

```bash
# Build JARs + download OTel agent
./gradlew dockerPrereqs

# Start services with OTel agent volume-mounted
docker compose \
  -f infra/recipes/docker-compose/oh-only/docker-compose.yml \
  -f infra/recipes/docker-compose/common/otel-profile/tables.yml \
  -f infra/recipes/docker-compose/common/otel-profile/housetables.yml \
  -f infra/recipes/docker-compose/common/otel-profile/jobs.yml \
  up -d --build
```

`run.sh` already detects the agent at
`./otel/opentelemetry-javaagent.jar` and
attaches it automatically. Without the otel-profile overrides there is
no JAR
mounted, so `run.sh` skips it.

## Changes
- Remove `COPY` of OTel agent JAR from all three service Dockerfiles
- Remove `downloadOtelAgent` from CI `build-run-tests` workflow
- Add `infra/recipes/docker-compose/common/otel-profile/` compose
overrides
  that volume-mount the agent into each service container
- Restore `otel_annotations_version` ext property and OTel annotation
dependencies as `compileOnly` (needed at compile time for `@WithSpan`,
not
  shipped in fat JARs)

## Test Plan
- [ ] CI `build-run-tests` workflow passes (no OTel download, no Docker
COPY)
- [ ] CI `build-tag-publish` workflow passes (Docker images build
without OTel)
- [ ] Local `./gradlew dockerPrereqs && docker compose ... -f
otel-profile/tables.yml up` attaches agent
- [ ] Services start correctly without otel-profile overrides (no agent,
no error)

Co-authored-by: Vibe Kanban <noreply@vibekanban.com>

Author: cbb330

.github/workflows/build-run-tests.yml

@@ -21,7 +21,7 @@ jobs:
         uses: gradle/actions/setup-gradle@v5
 
       - name: Build with Gradle
-        run: ./gradlew clean build downloadOtelAgent
+        run: ./gradlew clean build
 
       - name: Start Docker Containers
         run: docker compose -f infra/recipes/docker-compose/oh-only/docker-compose.yml up -d --build

build.gradle

@@ -33,6 +33,7 @@ ext {
   iceberg_1_2_version = "1.2.0.6"
   iceberg_1_5_version = "1.5.2.7"
   otel_agent_version = "2.12.0" // Bundles OTel SDK 1.47.0
+  otel_annotations_version = "2.12.0" // Match agent version
 }
 
 // OpenTelemetry Java Agent for distributed tracing.

housetables-service.Dockerfile

@@ -35,9 +35,6 @@ RUN ls $APP_NAME.jar
 
 COPY run.sh .
 
-# OpenTelemetry Java Agent for distributed tracing (optional; run.sh attaches it only if present).
-# The [r] glob trick makes COPY a no-op when the file is absent, avoiding a hard build failure.
-COPY build/otel/opentelemetry-javaagent.ja[r] otel/
 
 # Ensure that everything in $USER_HOME is owned by openhouse user
 RUN chown -R openhouse:openhouse $USER_HOME

iceberg/openhouse/internalcatalog/build.gradle

@@ -12,6 +12,8 @@ plugins {
 dependencies {
   implementation 'com.github.spotbugs:spotbugs-annotations:4.8.1'
   api 'org.springframework.retry:spring-retry:1.3.3'
+  compileOnly "io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:${otel_annotations_version}"
+  api 'io.opentelemetry:opentelemetry-api:1.47.0'
   api project(':client:hts')
   api project(':cluster:storage')
   api project(':cluster:metrics')

infra/recipes/docker-compose/common/otel-profile/housetables.yml

@@ -0,0 +1,6 @@
+services:
+  openhouse-housetables:
+    volumes:
+      - ../../../../build/otel/opentelemetry-javaagent.jar:/home/openhouse/otel/opentelemetry-javaagent.jar:ro
+    environment:
+      - OTEL_SERVICE_NAME=openhouse-housetables

infra/recipes/docker-compose/common/otel-profile/jobs.yml

@@ -0,0 +1,6 @@
+services:
+  openhouse-jobs:
+    volumes:
+      - ../../../../build/otel/opentelemetry-javaagent.jar:/home/openhouse/otel/opentelemetry-javaagent.jar:ro
+    environment:
+      - OTEL_SERVICE_NAME=openhouse-jobs

infra/recipes/docker-compose/common/otel-profile/tables.yml

@@ -0,0 +1,6 @@
+services:
+  openhouse-tables:
+    volumes:
+      - ../../../../build/otel/opentelemetry-javaagent.jar:/home/openhouse/otel/opentelemetry-javaagent.jar:ro
+    environment:
+      - OTEL_SERVICE_NAME=openhouse-tables

jobs-service.Dockerfile

@@ -33,9 +33,6 @@ RUN ls $APP_NAME.jar
 
 COPY run.sh .
 
-# OpenTelemetry Java Agent for distributed tracing (optional; run.sh attaches it only if present).
-# The [r] glob trick makes COPY a no-op when the file is absent, avoiding a hard build failure.
-COPY build/otel/opentelemetry-javaagent.ja[r] otel/
 
 # Ensure that everything in $USER_HOME is owned by openhouse user
 RUN chown -R openhouse:openhouse $USER_HOME

services/tables/build.gradle

@@ -31,6 +31,7 @@ openApi {
 }
 
 dependencies {
+  compileOnly "io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:${otel_annotations_version}"
   api project(':services:common')
   api project(':iceberg:openhouse:internalcatalog')
   api project(':client:hts')

tables-service.Dockerfile

@@ -35,9 +35,6 @@ RUN ls $APP_NAME.jar
 
 COPY run.sh .
 
-# OpenTelemetry Java Agent for distributed tracing (optional; run.sh attaches it only if present).
-# The [r] glob trick makes COPY a no-op when the file is absent, avoiding a hard build failure.
-COPY build/otel/opentelemetry-javaagent.ja[r] otel/
 
 # Ensure that everything in $USER_HOME is owned by openhouse user
 RUN chown -R openhouse:openhouse $USER_HOME