Hi,
Description
To improve the developer experience when writing and testing complex pipelines, one of my most requested features is the ability to properly debug a pipeline.
This would ideally include the ability to:
- Step through each processor's execution
- Run to a specific pipeline step and halt (Breakpoints)
- Inspect the input message before a processor executes
- Inspect the output message after a processor executes (very helpful for branch processors!)
Currently, there doesn't appear to be a public API that allows for this level of introspection, which is essential for building a step-through debugger or a visual pipeline tracer.
Alternatives Considered
My first approach was to try and implement this externally:
- Parse an existing pipeline configuration
- Manually wrap every single processor in a custom "debug" processor (which would record I/O or halt execution)
- Feed this new, enriched pipeline configuration into a StreamBuilder.
This approach failed for two main reasons:
- Complexity: Programmatically parsing and reliably wrapping all processors, especially nested ones (like in branch or switch processors), is extremely difficult and fragile.
- Context Loss: This wrapping "mangles" the original processor paths (e.g.,
processors.0 becomes processors.0.debug_wrapper.original or something similar). This makes it very difficult to map the execution step back to the original pipeline configuration for visualization.
Proposed Solution
I request a new public API to inject a wrapper or hook around each processor during its construction, without altering the user-defined configuration path.
This could be conceptually similar to the existing BuildTraced() function on the StreamBuilder, but allows for custom pre- and post hook functions.
The pre- and post functions would provide informations about the wrapped processor (name, type, path, config - basically a WalkedComponent) and access to the currect message batch.
Using the return value of the hooks, one could influence the execution and stop the pipeline if desired.
Hi,
Description
To improve the developer experience when writing and testing complex pipelines, one of my most requested features is the ability to properly debug a pipeline.
This would ideally include the ability to:
Currently, there doesn't appear to be a public API that allows for this level of introspection, which is essential for building a step-through debugger or a visual pipeline tracer.
Alternatives Considered
My first approach was to try and implement this externally:
This approach failed for two main reasons:
processors.0becomesprocessors.0.debug_wrapper.originalor something similar). This makes it very difficult to map the execution step back to the original pipeline configuration for visualization.Proposed Solution
I request a new public API to inject a wrapper or hook around each processor during its construction, without altering the user-defined configuration path.
This could be conceptually similar to the existing BuildTraced() function on the StreamBuilder, but allows for custom pre- and post hook functions.
The pre- and post functions would provide informations about the wrapped processor (name, type, path, config - basically a WalkedComponent) and access to the currect message batch.
Using the return value of the hooks, one could influence the execution and stop the pipeline if desired.