Search query timeouts (~300ms)

[disconsented]

Search query timeouts (~300 ms)

Goal

No read query should be allowed to run longer than ~300 ms. A query that exceeds
it is aborted and the endpoint returns an error rather than holding a DB worker
(and, today, an actor mailbox) open indefinitely. This is both a UX guarantee
and an abuse mitigation — it caps the cost of any single request, including
pathological filter combinations.

Where queries run today

Two read paths build a SelectStatement by hand and run it against the embedded
SurrealDB engine:

• Search / list — query_inner in src/web/query.rs:67, executed at
  src/web/query.rs:229 (db.query(stmt)). This is the expensive one: optional
  title substring CONTAINS, tags CONTAINSALL, language CONTAINSANY,
  ordering, plus two graph-traversal sub-selects (workshop_item_properties,
  tags filter).
• Item detail — get_item in src/web/item.rs:114, executed at
  src/web/item.rs:284. Two graph traversals each direction
  (item_dependencies in/out) plus the properties lookup.

Neither sets a timeout, so both inherit the engine default (effectively
unbounded for our purposes).

Approach: two layers

Layer 1 — SurrealQL TIMEOUT clause (primary)

SelectStatement already has a timeout field. In surrealdb-core 3.1.2:

// src/sql/statements/select.rs
pub struct SelectStatement {
    ...
    pub timeout: Expr,   // default Expr::Literal(Literal::None) → no clause emitted
    ...
}

When timeout is anything other than Literal::None, the statement renders
TIMEOUT <expr> (see select.rs:69). So we set it to a duration literal:

use std::time::Duration;
use surrealdb_core::sql::{Expr, Literal};
// PublicDuration is the duration newtype already re-exported from surrealdb_core::sql
// (used as Literal::Duration(PublicDuration) — confirm the exact path when wiring this up)

const QUERY_TIMEOUT: Duration = Duration::from_millis(300);

stmt.timeout = Expr::Literal(Literal::Duration(QUERY_TIMEOUT.into()));

This makes the engine itself stop executing and return an error once the budget
is blown — the cleanest option because the engine stops doing work, it isn't
just abandoned.

Apply at both sites:

• src/web/query.rs — set stmt.timeout before the debug!(sql = ...) line at :228.
• src/web/item.rs — set stmt.timeout before the debug!(sql = ...) at :283.

Pull the constant into one place (e.g. a const QUERY_TIMEOUT in src/web/mod.rs)
so both paths share it and it's tunable in one spot.

Verify when implementing: confirm the concrete type inside
Literal::Duration(...) for the sql module (vs the expr module) and that
Duration::into() resolves to it. literal.rs:29 shows the variant; grep for
PublicDuration in surrealdb-core for the alias. The to-SQL output should be
TIMEOUT 300ms; assert this in a unit test via stmt.to_sql().

Layer 2 — tokio::time::timeout wrapper (belt-and-suspenders)

The TIMEOUT clause governs query execution, but the surrounding .await
(result decoding via take(0), the try_from conversion loop in
query_inner, actor round-trip latency) is not covered. Wrap the whole DB
interaction so a hang anywhere still bounds the request:

let results = tokio::time::timeout(
    QUERY_TIMEOUT + Duration::from_millis(50), // small grace over the SQL budget
    db.query(stmt),
)
.await
.map_err(|_elapsed| /* InnerError::Unavailable / 503 or a 504-style error */)?
.whatever_context("querying")?;

Use a slightly larger outer budget than the SQL TIMEOUT so that, in the normal
slow case, the engine's own clean abort fires first and we get a precise error;
the tokio wrapper only trips on a genuine hang.

Error semantics

A timeout is not a 500. Map it to a distinct status so it's observable and
so the UI can react:

• Search (query.rs) currently returns the generic web::Error (500). Add a
  timeout branch returning 503 Service Unavailable (or 504) with a short body.
• Item (item.rs) already has an InnerError enum (item.rs:34) — add a
  Timeout variant mapping to 503/504, alongside NotFound / InternalError.

Log timeouts at warn with the rendered SQL and the bound filter values so slow
filter combinations can be found and indexed.

Decisions to make

1. 300 ms for everything, or per-endpoint? The item-detail query is usually
   cheaper than a full filtered search. A single 300 ms cap is simplest; consider
   a slightly higher cap for search if 300 ms turns out to reject legitimate
   large-result queries. Start with one shared constant.
2. Status code: 503 vs 504. 504 (Gateway Timeout) is semantically closer but
   the service isn't a gateway; 503 with Retry-After is also defensible. Pick one
   and use it in both paths.

Testing

• Unit: build the statement, assert stmt.to_sql() contains TIMEOUT 300ms.
• Unit: existing get_item tests in src/web/item.rs:397 stand up an in-memory
  DB — extend with a case asserting the timeout clause is present on the emitted
  SQL (cheap and doesn't need a genuinely slow query).
• Manual: point at a large dataset, issue a broad title substring search with no
  other filters, confirm it returns a timeout error rather than hanging.

Effort

Small — roughly half a day including tests. Mostly mechanical once the
Literal::Duration construction is confirmed to compile.