Skip to content

fix(relay): apply ingest's content checks to workflow SendMessage - #9229

Draft
ThePumpingLemma wants to merge 1 commit into
dgrochowski/reject-nul-and-close-stalled-backfillfrom
dgrochowski/workflow-message-content-checks
Draft

ThePumpingLemma wants to merge 1 commit into
dgrochowski/reject-nul-and-close-stalled-backfillfrom
dgrochowski/workflow-message-content-checks

Conversation

@ThePumpingLemma

@ThePumpingLemma ThePumpingLemma commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #9180 (which adds reject_nul_bytes); retarget to main once it merges.

Problem

Ingest refuses event content over 256 KB and content or tags with a NUL byte, because Postgres text cannot store \0. The workflow sink's SendMessage builds a relay-signed kind:9 and inserts it directly, so neither check runs. Its text comes from templates, and a template can include a call_webhook response body. That body can be up to 1 MiB and is decoded with String::from_utf8_lossy, which keeps \0. As a result:

  • A webhook body between 256 KB and 1 MiB gets stored as a message that ingest would have refused.
  • A body with a NUL byte fails at insert, and the run step fails with ActionSinkError::Database instead of an error that names the bad input.

Change

  • handlers/ingest.rs: the size limit and the content NUL check are now shared pub(crate) pieces: MAX_EVENT_CONTENT_BYTES, check_content_size, check_content_nul, and a StoredContentError whose Display leaves off the invalid: prefix.
    • ingest_event_inner keeps its current order. The size check runs first. The NUL check (content and tags, via reject_nul_bytes) runs after auth, scope and the write restriction, before command routing.
    • Ingest's rejection messages are unchanged.
  • workflow_sink.rs: send_message runs both checks on the rendered text before any database work. A failure becomes ActionSinkError::InvalidInput. The failed run then records code webhook_failed with the message webhook error: invalid input: <reason> (every sink error already maps this way).
    • The sink builds its tags from parsed or stored values (channel UUID, pubkeys, resolved event ids), so the text is the only value that needs the check.

Testing

  • New test workflow_sink::tests::send_message_refuses_unstorable_text_before_touching_the_database. It drives RelayActionSink::send_message against an unreachable database:
    • NUL text and text one byte over the limit must come back as InvalidInput.
    • Text exactly at the limit must get past the checks and reach the database, where it fails as Database.
  • Before the fix: cargo test -p buzz-relay --lib send_message_refuses_unstorable_text failed with
    content contains a NUL byte: must be refused as invalid input, got Err(Database("database error: pool timed out while waiting for an open connection"))
    test result: FAILED. 0 passed; 1 failed
    
  • After the fix:
    • cargo test -p buzz-relay --lib -- workflow_sink nul_bytes: 23 passed, 5 ignored (Postgres).
    • cargo test -p buzz-relay --lib -- handlers::ingest: 187 passed, 22 ignored (Postgres).
  • cargo fmt --all and cargo clippy -p buzz-relay --all-targets -- -D warnings are clean.
  • The pre-push rust-tests hook passed.

Not covered

  • The Postgres-gated workflow_sink tests were not run (no local database).
  • moderation_notices.rs: the notice body is relay-formatted text around an operator-authored public_reason, which is already stored in Postgres, so no NUL can be in it. No webhook or other untrusted text reaches it. Left unchanged.
  • Other relay-signed inserts (system and membership messages in side_effects.rs, git announcements, audio) build their content from relay or stored values, not caller-supplied text. Left unchanged.
  • The open PR that adds more sink send methods will need the same check when it lands.

Review

Blockers: none. Checked the following:

  • Every tag the sink builds comes from parsed or stored values, so only the text needs the check.
  • The size check measures the same event.content bytes against the same constant as ingest.
  • ActionSink has no other relay-signed send method.

Open (optional):

  • Text over the limit now shows up in the failed run with a readable reason. A NUL byte that comes from a call_webhook body does not yet show up in the run record. That webhook step's output is already in the execution trace, and Postgres JSONB refuses NUL escapes, so writing the run's final status fails and the run stays running. This happened before this PR as well. For now the new NUL check is a defensive guard. Follow-up: make the webhook body storable when it is decoded (for example, replace \0 the way invalid UTF-8 is already replaced) or refuse it at the webhook step.

Exercised on a local relay

Commit tested: f4dc6ddf3 (stacked on #9180). Dev profile, isolated harness relay on :3030. The sender is the channel owner. Workflows were created with buzz workflows create, each with a message_posted trigger, and triggered with buzz messages send.

A local HTTP server cannot be used. call_webhook runs check_ssrf, which rejects loopback and private addresses, and there is no override. The workflow pointing at http://127.0.0.1:8099 failed with webhook error: SSRF blocked: '127.0.0.1' resolved to private/reserved address 127.0.0.1. So the content cases used public httpbin endpoints:

  • /base64/… returns the body a\0b WFNUL
  • /drip?numbytes=300000 returns a 300,000-byte body

In each workflow the body was templated into send_message as {{steps.hook.output.body}}.

Workflow Run result (workflow_runs) Message stored?
call_webhook body over 256 KB → send_message failed, webhook_failed: webhook error: invalid input: content exceeds maximum size of 262144 bytes (got 300005) no
call_webhook body with NUL → send_message step refused (log: Workflow run failed … webhook_failed), but the run row stays running with an empty trace (see defect) no
clean call_webhook body → send_message completed yes, hook said: clean webhook body
plain send_message completed yes, plain workflow message
call_webhook to 127.0.0.1 failed (SSRF blocked) no

Defect (not introduced here): with RUST_LOG=buzz_workflow=debug, the NUL run logs the sequence below.

  1. SendMessage → …: hook said: a\u0000b WFNUL
  2. Workflow run failed … webhook_failed
  3. Failed to update run to Failed: database error: … unsupported Unicode escape sequence

The completed hook step's output, which is the webhook body containing \u0000, is written into the execution_trace jsonb column, and Postgres jsonb rejects \u0000. So the failure is never recorded, the run stays running indefinitely, and the content error this PR adds is never visible on the run. The same would happen for any step output containing NUL, even without a send_message.

Postgres-gated tests: I pointed DATABASE_URL, BUZZ_TEST_DATABASE_URL and BUZZ_TEST_REDIS_URL at the harness.

  • cargo test -p buzz-relay --lib -- --ignored --test-threads=1 workflow_sink nul_bytes: 5 passed, 0 failed.
  • The new send_message_refuses_unstorable_text_before_touching_the_database: 1 passed.

Not exercised:

  • A local webhook server, because the SSRF guard blocks it.
  • A body between 256 KB and 1 MiB other than 300,000 bytes.
  • NUL in tags; the sink builds its tags from parsed values.

Problem: ingest refuses event content over 256 KB and content or tags
containing a NUL byte (Postgres `text` cannot store `\0`). The workflow
sink's SendMessage builds and inserts a relay-signed kind:9 directly,
so it skipped both checks. Its text is rendered from templates that can
include a `call_webhook` response body (up to 1 MiB, decoded with
`String::from_utf8_lossy`, which keeps `\0`).

How it fails: a webhook body between 256 KB and 1 MiB is stored as a
message ingest would have refused; a body with a NUL byte fails at
insert and the run step fails with `ActionSinkError::Database` instead
of naming the bad input.

Change: the size limit and the content NUL check become shared
`pub(crate)` primitives in ingest (`MAX_EVENT_CONTENT_BYTES`,
`check_content_size`, `check_content_nul`, with a `StoredContentError`
whose Display omits the `invalid:` prefix). `ingest_event_inner` keeps
its ordering: the size check first, the NUL check (content and tags)
after auth, scope and the write restriction, before command routing.
Its rejection messages are unchanged. The sink runs both checks on the
rendered text before any database work and maps a failure to
`ActionSinkError::InvalidInput`. Tags in the sink are built from parsed
or stored values (UUID, pubkeys, resolved event ids), so the text is
the only value that needs checking there.

Why safe: ingest behaviour and messages are unchanged; the sink only
refuses text that ingest would refuse or that could not be stored.

Tests: `send_message_refuses_unstorable_text_before_touching_the_database`
drives `RelayActionSink::send_message` against an unreachable database.
Before the fix NUL text returned `Database("... pool timed out ...")`;
after it NUL and over-limit text return `InvalidInput`, and text at the
limit still reaches the database.

Signed-off-by: David Grochowski <dgrochowski@squareup.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant