Skip to content

Unify protobuf libraries to remove protobuf version pin #4018

@alco

Description

@alco

Problem

We pin protobuf to ~> 0.13.0 in packages/sync-service/mix.exs to avoid a duplicate module conflict at mix release time. The conflict is between Google.Protobuf.* well-known type modules shipped by two different protobuf libraries in our dependency tree:

  • protobuf (>= 0.14.0) — used by otel_metric_exporter for OTLP metric/log encoding. Starting with v0.14.0, it bundles Google.Protobuf.* modules (previously in the separate google_protos package).
  • protox — used by pg_query_ex for Postgres parse tree (de)serialization. Its well-known type source files (lib/google/protobuf/*.ex) define Protox.Google.Protobuf.* container modules, but the use Protox.Define macro inside them generates bare Google.Protobuf.* modules at compile time (via DefineMessage.define/2 which does defmodule unquote(msg_schema.name) where name: is Google.Protobuf.Any, not Protox.Google.Protobuf.Any).

The conflict only manifests during mix release (not mix compile), because that's when OTP checks for duplicate modules across applications.

Neither side's Google.Protobuf.* modules are actually used by our codeotel_metric_exporter's generated .pb.ex files are namespaced under OtelMetricExporter.*, and pg_query_ex's proto schema has no well-known type imports.

Approaches explored

1. Remove conflicting source files in Dockerfile (quick fix)

Add RUN rm -rf deps/protox/lib/google between mix deps.get and mix deps.compile in the Dockerfile. This prevents protox from compiling the well-known type modules, and Mix auto-generates a clean .app file without them.

Verified locally: protox compiles fine (32 files instead of 50) and mix release passes.

Pros: minimal, no upstream changes needed.
Cons: only applies to Docker builds; local mix release would still fail without a similar step.

2. Fork protox and delete lib/google/

Fork ahamez/protox, remove the lib/google/ directory, point pg_query_ex at the fork.

Pros: clean, works everywhere (not just Docker).
Cons: fork maintenance burden.

3. Post-compile cleanup in mix.exs

Add a Mix alias that removes Google.Protobuf.*.beam files from protox's ebin/ and patches its .app file after compilation.

Pros: works for all build contexts.
Cons: fragile, modifies compiled artifacts.

4. Fix protox upstream

Change protox's well-known type schemas to use Protox.Google.Protobuf.* as the module name: so DefineMessage generates prefixed modules. Requires updating parse.ex internals and the code generator's handling of well-known type references. This is a breaking change for protox.

Pros: proper fix at the source.
Cons: breaking change, significant scope, depends on upstream maintainer.

5. Unify on a single protobuf library (preferred direction)

Use either protobuf or protox in both otel_metric_exporter and pg_query_ex, eliminating the conflict entirely.

  • Option A: Migrate pg_query_ex to use protobuf — regenerate the proto files with protoc-gen-elixir (from the protobuf package) instead of protox. pg_query_ex is our own package so we control this.
  • Option B: Migrate otel_metric_exporter to use protox — regenerate the OTLP proto files with protox. Also our own package.

Pros: eliminates the root cause, no workarounds needed, one fewer dependency in the tree.
Cons: requires regenerating proto code and updating encode/decode call sites.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions