Dead Letter Queue (DLQ) for Connectors

[kriti-sc]

Overview

When a connector fails to process one or more messages from a batch, those messages are currently lost forever. The following is a proposal for a Dead Letter Queue (DLQ) that routes failed messages to a dedicated Iggy topic so they can be inspected, debugged, and replayed.

---

Outline

1. Design defines the DLQ semantics and the high-level design for routing failures to DLQ.
2. Connector to DLQ Message Path describes the control flow to route a failed message to the DLQ.
3. ABI Impact evaluates alternatives for implementing the required changes to the iggy_sink_open FFI.

---

Design

DLQ Semantics

1. DLQ to contain failing messages only, not the entire batch

When a batch partially fails, the connector must separate successful messages from failed ones and only send the failed messages to the DLQ. The sucessful messages must be written to the target.

Writing the entire batch to DLQ on any single failure wastes storage, makes debugging and replay harder, and obscures the actual failure rate.

2. DLQ is an Iggy topic

Failed messages are published to a user-defined Iggy topic. These are fully user-defined in the sink config. The runtime reads them at startup and sets up the DLQ producer accordingly.

High-level Design

The design of the DLQ follows directly from the principles above.

1. Since DLQ operates at message granularity and only the connector has visibility into message-level failures, it is the connector's resposibility to route messages to the DLQ. This also means that DLQ is opt-in per connector. If a connector does not implement DLQ, failed messages are silently lost.

2. DLQ writes are bridged via a DlqCallback function pointer injected into the plugin at open() time, following the same pattern as LogCallback. This is needed because connectors do not have access to the runtime's IggyClient.

---

Connector to DLQ Message Path

This section describes the flow of a failed message from detection in the sink/source to its eventual persistence in the DLQ.

Sink

1. The runtime injects a DlqWriter into the sink at open() time via a callback
2. During consume(), the sink segregates messages into two structures — successful and failed — as it processes each message
3. At the end of consume(), successful messages are written to the target in a single batch write
4. The sink calls dlq.send(topic_metadata, partition_id, msg, reason) for each failed message — the SDK constructs the DlqEnvelope, serializes it to bytes, and invokes the callback
5. The runtime's callback receives the serialized bytes and produces them to the DLQ stream and topic specified in config

Source

1. The runtime injects a DlqWriter into the source at open() time via a callback
2. During poll(), the source fetches a batch from the external system and converts each record into a ProducedMessage
3. Records that fail conversion or transformation are sent to DLQ via dlq.send() with the failure reason
4. Successful records are returned in ProducedMessages — the runtime writes them to the target Iggy topic
5. The runtime's callback receives the serialized bytes and produces them to the DLQ stream and topic specified in config

---

ABI Impact

Introducing DlqCallback requires changing the iggy_sink_open (and iggy_source_open) FFI signature — the entrypoint symbol every plugin must export. This is a breaking ABI change: any plugin compiled against the old signature and loaded by a new runtime will misinterpret the call arguments, regardless of whether the plugin uses DLQ. A forced rebuild of all plugins is required.

Three approaches to handle this:

Option 1: New versioned entrypoint symbol

Add iggy_sink_open_v2 with the new signature alongside the existing iggy_sink_open. The runtime checks for iggy_sink_open_v2 first (dlopen symbol lookup), and falls back to iggy_sink_open for older plugins. Old plugins continue to work without recompile; new plugins opt in by exporting the v2 symbol.

• Pro: fully backwards compatible, no forced rebuild
• Con: runtime must carry fallback logic indefinitely; adds complexity to the plugin loading path

Option 2: Separate iggy_sink_set_dlq_callback symbol (recommended)

Keep iggy_sink_open unchanged. Add a new optional symbol iggy_sink_set_dlq_callback(id, dlq_callback) that the runtime calls after open() if DLQ is configured for that sink. Plugins that don't export this symbol simply don't get DLQ wired up.

// new optional symbol — only connectors that support DLQ need to export this
unsafe extern "C" fn iggy_sink_set_dlq_callback(id: u32, dlq_callback: DlqCallback) -> i32

• Pro: purely additive, no signature change, no versioning logic, cleanly expresses DLQ as optional capability
• Con: two-step initialization (open, then set callback); runtime must handle the case where the symbol is absent

Option 3: Accept the forced rebuild

Since all official sink and source plugins live in the same repository, a coordinated rebuild across all of them is low friction. The only plugins affected without a rebuild would be third-party plugins compiled outside the repo.

• Pro: simplest — one clean signature, no compatibility shims
• Con: breaks any out-of-tree plugins; requires discipline to rebuild everything together

[numinnex]

I've been thinking about it for a while and concluded that DLQ is a feature that can be trivialy implemented
using our existing SDK (both a "deep copy" of the message or just a "pointer" to the log through offset).

In the future we would like to implement an KVP store (for each partition) that is accessible by users through our SDK. It could be used to implement the "pointer" variant of DLQ even simpler than through dedicated topic, by simply storing "key" - "dlq-{name}", value - [integer, ....], for the "deep copy" variant a dedicated topic would still be the best solution.

Shortly, I think DLQ as a functionality is very simple to implement and relies heavily on user defined business logic, which in order to support would require changes to our connector infrastructure in a way where I don't see the justification for it.

[kriti-sc]

this is a valid point. However, having DLQ support as a first class citizen supported by the runtime will improve user experience. For ex. runtime can track DLQ message counts, surface them in the /stats API, alert on elevated DLQ rates, etc.

Adding DLQ doesn't have to be a breaking ABI change. Option 2 suggests an additive change which doesn't impact the existing interface.

[spetz]

I agree with @numinnex in a sense that DLQ or more like a dead letter topic (w/e we call this one), should be more like a global feature that could be used for any purpose (stream/topic in general), and not just for the connectors runtime, which relies on the same topics anyway :)

Once we have an embedded KV store, it becomes rather easy to implement for both sides - server and client SDK.
Thus, I'm not certain if it's worth spending too much effort right now other than maybe implementing an optional connectors runtime DLQ (a dedicated topic per connector) to which the sink would append any failed/unprocessable messages in the custom envelope containing the source offset, headers, payload, etc.