Skip to content

fix(workflow): replace NUL in webhook output and trigger fields - #9242

Draft
ThePumpingLemma wants to merge 1 commit into
mainfrom
dgrochowski/webhook-body-nul
Draft

ThePumpingLemma wants to merge 1 commit into
mainfrom
dgrochowski/webhook-body-nul

Conversation

@ThePumpingLemma

@ThePumpingLemma ThePumpingLemma commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

PostgreSQL jsonb rejects \^@ (unsupported Unicode escape sequence). call_webhook decodes the response body with String::from_utf8_lossy, which keeps NUL, and the body goes into the run's execution_trace. When a webhook returns a NUL byte, the final status update fails (Failed to update run to Failed: … unsupported Unicode escape sequence) and the run stays running forever.

Webhook-trigger bodies and workflow-command payloads have the same problem: JSON "\^@" escapes parse into NUL in webhook_fields, so writing workflow_runs.trigger_context fails and the webhook endpoint returns a 500.

Change

  • call_webhook now builds its output with webhook_output, which decodes lossily and replaces U+0000 with U+FFFD. That's the same substitution from_utf8_lossy already makes for invalid UTF-8, and the run keeps going. I chose this over failing the step: the body is already a best-effort rendering, so failing a workflow over bytes a remote server sent would be inconsistent.
  • The webhook endpoint (api/bridge.rs) and the workflow command handler (handlers/command_executor.rs) had duplicate loops that flatten JSON into trigger fields. Both now call TriggerContext::extend_webhook_fields, which applies the same replacement to keys and values.

Testing

Ran against an isolated postgres:17-alpine (port 5481) with the schema applied by bin/pgschema and reconcile-schema-after-pgschema.sql, plus redis:7-alpine (port 6391). I stopped both afterwards.

Before the fix (with storable temporarily made the identity function):

---- executor::tests::webhook_output_replaces_nul_like_invalid_utf8 stdout ----
trace is not jsonb-storable: [{"output":{"body":"ok\^@tail�","status":200},"status":"completed","step_id":"hook"}]

---- executor::tests::extend_webhook_fields_replaces_nul_in_keys_and_values stdout ----
context is not jsonb-storable: {...,"webhook_fields":{"k\^@":"1","nested":"{\"x\":\"\\^@\"}","note":"a\^@b"}}

---- postgres_tests::webhook_output_with_nul_completes_run stdout ----
assertion `left == right` failed: a NUL in the webhook body must not strand the run
  left: Running
 right: Completed

---- api::bridge::workflow_webhook_postgres_tests::webhook_body_field_with_nul_creates_run stdout ----
assertion `left == right` failed: response: {"error":"internal server error"}
  left: 500
 right: 202

After the fix:

  • cargo test -p buzz-workflow --lib --features reqwest -- --include-ignored: 180 passed
  • cargo test -p buzz-relay --lib -- --include-ignored workflow_webhook_postgres_tests: 2 passed
  • cargo test -p buzz-relay --lib command_executor: passed
  • cargo clippy --all-targets -- -D warnings for buzz-workflow (with and without reqwest) and buzz-relay: clean
  • cargo fmt --all: clean

Not covered

  • add_reaction_impl still stores the relay's own reaction API response (resp.text(), which keeps NUL) in the trace. That response comes from our own relay, not from an arbitrary remote server.
  • workflow_runs.error_message is TEXT, which also rejects NUL. A WorkflowError message carrying NUL would still fail the Failed update; for example, the AddReaction error embeds the relay's response body.
  • Trigger fields built from stored Nostr events (build_trigger_context) aren't changed. events.content is TEXT, so a NUL can't get past ingest.

Review

Agent review, no blockers. Scores: minimalism 9, elegance 9, correctness 9.

  • Clean cutover: the only places outside tests that write webhook_fields are api/bridge.rs and handlers/command_executor.rs, and both now call TriggerContext::extend_webhook_fields. No copy of the old loop is left.
  • No working behavior changes. Before this PR, each of these inputs failed: a 500, a failed create_workflow_run, or a run stuck in running. A condition can't have matched on NUL, because workflow definitions are stored as jsonb and can't contain one.
  • Non-string values are stored as JSON text, where NUL is already escaped as \u0000, so they're unchanged.
  • storable is private to executor.rs, next to its two callers. Response headers aren't stored, and HeaderValue rejects NUL anyway.
  • Optional:
    • After replacement, "k\u0000" and "k\uFFFD" in the same body collapse to one key, and the last one in map order wins. The sender controls both keys, so there's no cross-tenant effect.
    • The call_webhook_impl call site has no test of its own. The SSRF guard blocks loopback, so the test goes through webhook_output and the run below covers the call end to end.

Exercised on a local relay

Isolated harness relay (start-isolated-test-relay.sh --profile dev), workflows owned by a channel admin, each triggered with POST /hooks/{id}:

Workflow Input Result
send_message 'trigger note: {{trigger.note}}' body {"note":"a\u0000b"} 202; run completed; trigger_context.webhook_fields.note = a�b; message bytes …61 efbfbd 62
call_webhook → httpbin.org/base64/YQBiIFdGTlVM (a\0b WFNUL), then send_message {} 202; run completed; trace {"body":"a�b WFNUL","status":200}; message hook said: a�b WFNUL
call_webhook → clean httpbin body, then send_message {"note":"plain"} 202; run completed; body clean webhook body, message posted
same as the first row {"k\u0000":"nul","k\ufffd":"fffd"} 202; run completed; one key k� = fffd

Postgres-gated tests against the harness (DATABASE_URL/REDIS_URL pointing at it):

  • cargo test -p buzz-workflow --lib --features reqwest -- --include-ignored: 180 passed
  • cargo test -p buzz-relay --lib -- --include-ignored workflow_webhook_postgres_tests: 2 passed

Afterwards I tore the harness down with docker compose -p buzz-harness -f docker-compose.harness.yml down -v.

PostgreSQL jsonb rejects \u0000 ("unsupported Unicode escape sequence").
call_webhook decoded the response body with String::from_utf8_lossy,
which keeps NUL, and the body lands in the run's execution_trace. The
final status update then failed ("Failed to update run to Failed: ...")
and the run stayed `running` forever. Webhook-trigger and command
payload fields parsed from JSON ("\u0000" escapes) hit the same wall in
workflow_runs.trigger_context, failing run creation with a 500.

Replace U+0000 with U+FFFD at decode, the same substitution
from_utf8_lossy already makes for invalid UTF-8. Refusing the step
instead would fail a workflow over bytes a remote server chose to send,
and the body is already a lossy, best-effort rendering; keeping the run
going with a visible replacement character is consistent with that.

Trigger fields from the webhook endpoint and the workflow command
handler shared a duplicated flatten loop; both now go through
TriggerContext::extend_webhook_fields, which applies the same
replacement to keys and values.

Signed-off-by: David Grochowski <dgrochowski@squareup.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant