Skip to content

fix(relay): keep database error text off WebSocket replies - #9234

Draft
ThePumpingLemma wants to merge 1 commit into
mainfrom
dgrochowski/no-raw-db-text-on-websocket
Draft

ThePumpingLemma wants to merge 1 commit into
mainfrom
dgrochowski/no-raw-db-text-on-websocket

Conversation

@ThePumpingLemma

@ThePumpingLemma ThePumpingLemma commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Two WebSocket paths sent the database driver's error text to clients. HTTP already avoids this: api::db_read_error logs only DbError::kind(), because DB error text can repeat a bound value.

  • COUNT: req::db_read_closed_reason returned format!("error: {e}") for any failure that wasn't a timeout. A failed COUNT read therefore closed with the raw sqlx message, e.g. ["CLOSED","c","error: database error: encountered unexpected or invalid data: <text>"]. Its sibling history_read_closed_reason, used for REQ, already returned the fixed error: database error.
  • Channel write gate: ingest::check_channel_membership returned Result<(), String> and built "error: database error: {e}" when a lookup failed. Callers turned every Err into IngestError::Rejected, whose text is forwarded verbatim and counted as invalid. A channel ephemeral event whose membership lookup failed got OK false "error: database error: database error: pool timed out while waiting for an open connection". Two callers (edit ownership, report readability) told denials from DB errors apart with starts_with("error:").

Change

  • REQ history and COUNT now share one req::db_read_closed_reason(&DbError) -> &'static str. A statement cancel keeps error: query timed out; anything else gets error: database error. All four COUNT failure sites go through count::close_failed_count, which logs only the error kind.
  • check_channel_membership returns ChannelWriteRefusal (Denied(&'static str) | Db(DbError)). From<ChannelWriteRefusal> for IngestError turns a denial into Rejected with the same text and a failed lookup into Internal. The string sniffing in validate_edit_ownership and report::reporter_can_read is gone; both match on the variant.
  • New IngestError::db(&DbError) builds the existing Internal("error: database error: <kind>") text. It replaces the one existing copy (the event insert) and is used by the gate, edit ownership and the report handler. validate_edit_ownership and handle_report_event now return IngestError, so their own lookup failures are internal too. Before, those also became Rejected("…: <driver text>").

Every client-facing refusal text is byte-identical: restricted: not a channel member, restricted: only members can post in this channel, and the edit/report invalid: reasons. Only DB failures change:

  • Persistent writes answer error: internal server error (HTTP 500) instead of invalid/400 with driver text.
  • Ephemeral writes answer error: database error: <kind>.
  • COUNT closes with the same fixed reason as REQ.

Two cases keep their behaviour: a non-member whose channel lookup fails is still refused as a non-member, and a denial for a token-scoped target still hides the target. A COUNT read that can't get a connection closes with error: database error, exactly like a REQ history read today (the bridge shed-read test pins that for REQ). A separate WebSocket "unavailable" reason would have to change REQ too, so it is not part of this PR.

Testing

Regression tests, run before the fix (only the tests were added; production code unchanged):

$ cargo test -p buzz-relay --lib -- db_read_closed_reason_never_carries_driver_text channel_ephemeral_membership_lookup_failure_is_internal
test handlers::req::tests::db_read_closed_reason_never_carries_driver_text ... FAILED
test handlers::event::tests::channel_ephemeral_membership_lookup_failure_is_internal ... FAILED

---- handlers::req::tests::db_read_closed_reason_never_carries_driver_text stdout ----
assertion `left == right` failed: Sqlx(Protocol("bound value 'secret'"))
  left: "error: database error: encountered unexpected or invalid data: bound value 'secret'"
 right: "error: database error"

---- handlers::event::tests::channel_ephemeral_membership_lookup_failure_is_internal stdout ----
a failed membership lookup must be internal, got Err(Rejected("error: database error: database error: pool timed out while waiting for an open connection"))

test result: FAILED. 0 passed; 2 failed

After the fix:

  • cargo test -p buzz-relay --lib -- handlers::req:: handlers::count:: handlers::event:: handlers::report:: handlers::ingest:: → 297 passed; 0 failed; 40 ignored, including both regression tests. The ephemeral test now gets Internal("error: database error: pool_timed_out").
  • cargo clippy -p buzz-relay --all-targets -- -D warnings → clean (this also compiles the Postgres-only tests).
  • cargo fmt --all → no changes.
  • Pre-push hooks passed on push, including rust-tests and branch-skew.

The Postgres-only routed_read_cancel_maps_to_timeout_contract_on_count_and_search used to accept any error: prefix for the ordinary-error WS COUNT frame. It now pins the exact ["CLOSED","c","error: database error"]. It compiles but was not run here (no local Postgres).

Not covered

These handlers still put format!("database error: {e}") (driver text) into a String that becomes IngestError::Rejected. Each needs a typed error or IngestError::db; that is a separate sweep:

  • handlers/relay_admin.rs lines 258, 277, 352, 402, 409, 488, 496 (via map_relay_admin_error)
  • handlers/identity_archive.rs lines 82, 88, 243, 281 (mapped to Rejected("invalid: …") in ingest.rs)
  • handlers/moderation_commands.rs lines 114, 202, 285, 331, 392, 460
  • handlers/product_feedback.rs line 55
  • handlers/ingest.rs thread-parent lookups (around lines 1186, 1198, 1377, 1386) and the forum vote target lookup (around line 1568)

These are already Internal, so clients get sanitised text, but they still put raw driver text into the HTTP error log: load_channel_for_write (ingest.rs around line 1104), the relay-member removal (around line 3061), and map_serving_fence_state.

Review

No blockers, and no code changes from review. Checked:

  • Refusal texts other than DB failures are byte-identical: the membership constants, the edit invalid: reasons, and the report invalid: reasons.
  • target_denied still hides the target for every gate refusal, including a failed lookup.
  • IngestError::db carries only DbError::kind() labels.
  • Internal maps to the existing error metric label, HTTP 500, and the conformance server_error.
  • No desktop, mobile, CLI, SDK, or ACP client matches the old DB texts. Edit DB failures move from terminal invalid: to retryable error:.

Open optional items:

  • The per-caller DB arms in validate_edit_ownership and report::reporter_can_read have no test. An unreachable-DB unit test, like the ephemeral one, would pin them.
  • The REQ history failure path still sets the outcome by hand where access.db_failed(&e) does the same thing. This predates the PR.

Exercised on a local relay

Commit tested: a0987ab65. Dev profile, isolated harness relay on :3030, authenticated channel member. I broke the database two ways, each with a WS connection already open:

  • Rename: renamed the events and channel_members tables, so queries fail with SQLSTATE 42P01.
  • Pause: ran docker pause on the harness Postgres, so pool acquire times out.
Request Baseline Tables renamed Postgres paused
WS COUNT COUNT {count:1} CLOSED "error: database error" CLOSED "error: database error" (≈3 s)
WS REQ EOSE CLOSED "error: database error" CLOSED "error: database error"
WS EVENT kind 9 (channel write) OK true OK false "error: internal server error" OK false "error: internal server error"
WS EVENT kind 20001 (channel ephemeral) OK true OK false "error: database error: database:42P01" OK false "error: internal error checking restriction state"
HTTP /count 200 500 internal server error 404 relay: no community is configured for this host
HTTP /events kind 9 200 500 internal server error same 404

No WebSocket reply contained driver text: no relation names, no sqlx messages, no "pool timed out". The ephemeral reply carries only the error kind (database:42P01). After the tables were restored and Postgres unpaused, the same requests succeeded again without restarting the relay.

Observation (outside this PR): while Postgres is unreachable, HTTP requests get 404 no community is configured for this host. That is because tenant resolution treats a failed lookup as an unknown host. A 503 would describe the outage better.

Postgres-gated tests: I pointed DATABASE_URL, BUZZ_TEST_DATABASE_URL and BUZZ_TEST_REDIS_URL at the harness.

  • --ignored routed_read_cancel_maps_to_timeout_contract_on_count_and_search handlers::count:: handlers::report::: 2 passed, including the modified routed_read_cancel… test.
  • The two new non-ignored regression tests: 2 passed.
  • The broader --ignored --test-threads=1 routed_read_cancel handlers::req:: handlers::count:: handlers::event:: handlers::report:: handlers::ingest::: 40 passed, 1 failed. The failure is ingest_write_fence_follows_community_deletion_lifecycle. It runs sqlx migrations, which fail against the harness schema that pgschema already applied (type "channel_type" already exists). That is a test-environment problem.

Not exercised:

  • The edit-ownership and report lookup failure paths.
  • A statement-timeout cancel (error: query timed out).

Problem: two WebSocket paths sent the database driver's error text to
the client. HTTP already avoids this (`api::db_read_error` logs only
`DbError::kind()`, because DB error text can repeat a bound value).

- COUNT: `req::db_read_closed_reason` returned `format!("error: {e}")`
  for any non-timeout failure, so a failed COUNT read closed with the
  raw sqlx message, e.g. `error: database error: encountered unexpected
  or invalid data: <text>`. The sibling `history_read_closed_reason`
  already returned the fixed `DATABASE_ERROR_CLOSED`.
- Channel write gate: `ingest::check_channel_membership` returned
  `Result<(), String>` and built `"error: database error: {e}"` for a
  failed lookup. Callers mapped every `Err` to `IngestError::Rejected`,
  whose text is forwarded verbatim and counted as `invalid`. A channel
  ephemeral event whose membership lookup failed answered
  `OK false "error: database error: database error: pool timed out
  while waiting for an open connection"`. Two callers told the cases
  apart by `starts_with("error:")`.

Change:
- One `req::db_read_closed_reason(&DbError) -> &'static str` for REQ
  history and COUNT: a statement cancel keeps `error: query timed out`,
  anything else is `error: database error`. The four COUNT failure
  sites go through `count::close_failed_count`, which logs only the
  error kind.
- `check_channel_membership` returns `ChannelWriteRefusal`
  (`Denied(&'static str)` | `Db(DbError)`). `From<ChannelWriteRefusal>
  for IngestError` maps a denial to `Rejected` with the same text and a
  failed lookup to `Internal`. The string sniffing in
  `validate_edit_ownership` and `report::reporter_can_read` is deleted;
  both now match on the variant.
- `IngestError::db(&DbError)` builds the existing
  `Internal("error: database error: <kind>")` text. It replaces the one
  existing copy (the event insert) and is used by the gate, edit
  ownership and the report handler. `validate_edit_ownership` and
  `handle_report_event` return `IngestError`, so their own lookup
  failures (which also became `Rejected("…: <driver text>")`) are
  internal too.

Why it is safe: every client-facing refusal text stays byte-identical
(`restricted: not a channel member`, `restricted: only members can post
in this channel`, the edit/report `invalid:` reasons). Only DB failures
change: persistent writes now answer `error: internal server error`
(HTTP 500) instead of `invalid`/400 with driver text, ephemeral writes
answer `error: database error: <kind>`, and COUNT closes with the same
fixed reason as REQ. A connection-unavailable COUNT read gets
`error: database error`, like a REQ history read; a separate WS
"unavailable" reason would have to change REQ too. A non-member whose
channel lookup fails is still refused as a non-member, and a
token-scoped target denial still hides the target on any refusal.

Tests:
- `handlers::req::tests::db_read_closed_reason_never_carries_driver_text`
  failed before (`left: "error: database error: encountered unexpected
  or invalid data: bound value 'secret'"`), passes after.
- `handlers::event::tests::channel_ephemeral_membership_lookup_failure_is_internal`
  drives `publish_channel_ephemeral` against an unreachable database.
  Failed before (`Err(Rejected("error: database error: database error:
  pool timed out while waiting for an open connection"))`), passes
  after with `Internal("error: database error: pool_timed_out")`.
- The Postgres-only bridge COUNT test now pins the exact ordinary-error
  CLOSED frame instead of accepting any `error: ` prefix.

Signed-off-by: David Grochowski <dgrochowski@squareup.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant