docs: introduce Connect section, connect-string reference, QWP stubs#444
Open
bluestreak01 wants to merge 99 commits into
Open
docs: introduce Connect section, connect-string reference, QWP stubs#444bluestreak01 wants to merge 99 commits into
bluestreak01 wants to merge 99 commits into
Conversation
Consolidate how-applications-talk-to-QuestDB content under a single Connect supersection. Rewrite the ingestion overview as QWP-native "Connect to QuestDB", add a comprehensive connect-string reference at documentation/client-configuration/, and scaffold the Wire Protocols sub-section under documentation/protocols/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Component Converter ReminderA component in We are creating markdown correspondents of every path (e.g. questdb.com/docs/quick-start/ → questdb.com/docs/quick-start.md) for LLM consumption. Quick Check
💡 This is a friendly reminder, not a blocker. Ignore if not applicable. |
|
🚀 Build success! Latest successful preview: https://preview-444--questdb-documentation.netlify.app/docs/ Commit SHA: 74ffded
|
UDP documentation is deferred. Removes the stub page, sidebar entry, and links from ingress, egress, overview, and connect-string pages.
Closed
- Add "Why implement a QWP client" pitch and "Client lifecycle" narrative so new implementers can orient before diving into encoding details. - Spell out sequence numbering (server-assigned by receive order, not in the wire header), Gorilla first-DoD anchor, decimal scale formula (value = unscaled / 10^scale), and VARCHAR offset endianness — closes silent-wrong-guess risks for one-shot client generation. - Collapse Symbol section to WebSocket-only (per-table dict is UDP) and drop the now-stranded per-table example. - Document the practical WebSocket frame cap: http.recv.buffer.size (default 2 MiB) is the real ceiling, not the 16 MB protocol limit; exceeding it returns close code 1009 MESSAGE_TOO_BIG. - Fill out durable-ack semantics: watermark trails OK, empty messages trivially durable, reconnects discard in-flight tracking. - Note X-QWP-Client-Id may influence version selection. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/protocols/qwp-ingress-websocket.md
- Add "Why implement a QWP query client" pitch and "Client lifecycle" narrative paralleling the ingress doc; surfaces the java-questdb-client reference impl link upfront. - Document the practical WebSocket frame cap on /read/v1: client-to-server frames (QUERY_REQUEST in particular) are bounded by http.recv.buffer.size (default 2 MiB), not the 16 MiB protocol limit; oversized frames are rejected with close code 1009 MESSAGE_TOO_BIG. - Clarify X-QWP-Max-Batch-Rows only asks for smaller batches than the server default (clamps to server's hard limit). - Tighten NULL sentinel docs: FLOAT/DOUBLE sentinel is *any* NaN (incl. 0.0/0.0); IPv4 0.0.0.0 and all-ones GEOHASH cannot round-trip as non-null. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…sage Brokers up to Connect - Add a "Use QWP for new clients" tip callout to ilp/overview.md naming QWP's wins (binary, type-rich, faster, failover, store-and-forward) and framing ILP as the path for InfluxDB / Telegraf / Kafka / Flink users who already emit ILP. - Shorter callout on ilp/columnset-types.md reframing the page as "extensions on top of the InfluxDB type model" and noting QWP exposes the full QuestDB type system natively (no suffix encoding, no casts). - Operator-facing callout on ilp/advanced-settings.md flagging this page as the legacy ILP tuning surface and pointing new deployments at QWP. - Sidebar: lift Java Embedded and Message Brokers out from under ILP (they're protocol-agnostic delivery mechanisms, not ILP sub-pages). Final Connect order: Overview, Connect string, Date to Timestamp, Client Libraries, Message Brokers, Compatibility Protocols, Java Embedded, Wire Protocols. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eader) The egress endpoint /read/v1 is asymmetric on the wire: server-to-client frames carry the 12-byte QWP header, but client-to-server frames start directly with msg_kind — no QWP header. Including the header makes the server read 0x51 (the ASCII 'Q' of "QWP1") as an unknown msg_kind and close the WebSocket with code 1006, partway through send. Verified against server (QwpEgressUpgradeProcessor.dispatchEgressMessage calls peekMsgKind at offset 0 of the WS frame body) and Java reference client (QwpEgressIoThread.sendQueryRequest writes msg_kind as the first byte, no header). The upstream wire-egress.md spec is wrong on this point and should be filed separately. - Rewrite Message structure section with two ASCII diagrams (server-to- client with header, client-to-server without) and a warning callout naming the symptom and the reason (server keeps the header for RESULT_BATCH's flags + payload_length; client control frames have no analogous need). - Fix Example 1: drop the bogus 12-byte header from the QUERY_REQUEST hex dump; RESULT_BATCH / RESULT_END below unchanged. - Client lifecycle step 4: inline note that the binary frame body starts directly with msg_kind for client-to-server frames. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
New page (documentation/connect/agents.md) covering how AI agents operate QuestDB across three angles: protocols, tooling, and operational practices. Positioning: - QWP egress is the recommended path for SQL execution (DDL + streaming SELECT), with native client libraries when available and the protocol spec for clean-room implementations. - QWP ingress is the recommended path for all writes (bulk and sustained), including local-file uploads — the recipe explicitly calls out the failure mode where agents reach for read_parquet/read_csv/COPY, which require server-side filesystem access. - REST is positioned for schema discovery and small ad-hoc queries that fit in a single HTTP response. - PGWire and /imp are intentionally not recommended (superseded by QWP). - No MCP framing: an MCP server would just wrap REST + QWP without adding capability, so the page tells agents to use the underlying protocols directly. Includes a Recipes section seeded with the local-file upload recipe; links to the existing Getting Started > AI Coding Agents page for the tooling quickstart and the QuestDB / TSBS Claude skills. Sidebar: - Add Connect > Agents (between Client Libraries and Message Brokers). - Move Date to Timestamp inside Client Libraries (cross-cutting reference for all language clients). - Move Connect string inside Client Libraries as the first item (config schema shared by every QWP client). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/questdb/documentation into docs/connect-and-qwp-scaffolding
2. adjust c/c++ api docs
…/questdb/documentation into docs/connect-and-qwp-scaffolding
Rewrite all 11 <Tabs> blocks from the multi-line
<Tabs defaultValue="..." values={[ ... ]}> JSX-expression form to the
modern <Tabs defaultValue="..."> with per-TabItem label= attributes.
Docusaurus 3.8.1 auto-derives the values list from TabItem children, so
the rendered output is identical.
WebStorm's HTML parser does not understand the values={[ ... ]} JSX
expression. With the old syntax every Tabs opening was treated as an
unrecognised tag, cascading into "Closing tag matches nothing" errors on
every </TabItem> and </Tabs> below. The IDE MCP confirms those errors
disappear entirely with the new syntax.
No content change. The diff is structural only (-33 lines, three
mechanical replace_all operations: <Tabs ... values={[...]}> -> <Tabs ...>,
<TabItem value="c"> -> <TabItem value="c" label="C">,
<TabItem value="cpp"> -> <TabItem value="cpp" label="C++">).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add OIDC/mTLS/token-rotation "unsupported" table mirroring the .NET pattern, document null/omit equivalence with the first-row schema- inference caveat, explain what surfaces when the reconnect budget elapses (Rust has no ConnectionListener so the budget-exhaustion path must be spelled out), align the arrays example table name with sibling clients (fx_order_book -> book), and fix the production full example: remove tls_verify=unsafe_off from the production-labelled snippet, and rewrite the inline "// comment" text inside the multi-line Rust string literal that was actually being concatenated into the connect string at runtime. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quick-start ingest now catches LineSenderServerException (inspecting SenderError for status / category / server message) and LineSenderException, instead of bubbling untyped exceptions. Quick-start query aligns column names with the ingest example (sym -> symbol) and catches IllegalStateException for terminal-client state. amount is now DOUBLE consistently across ingest, full example, DDL, and bind-parameters SQL -- previously drifted between LONG and DOUBLE on the same page. New "Awaiting acknowledgements" subsection documents flushAndGetSequence / awaitAckedFsn / getAckedFsn and the FSN that correlates with SenderError.getFromFsn / getToFsn. Add request-id-aware onError / onEnd / onExecDone / onFailoverReset overloads with a "Diagnostics surface" table covering request id, failing SQL availability, bind-index policy, message stability, and PII safety. Full example: typed IllegalStateException / LineSenderException catches instead of bare catch (Exception e). Also tighten connect-string.md buffer_pool_size type to "int (>= 1)" to match the runtime minimum (resolves the pull-time conflict).
…/questdb/documentation into docs/connect-and-qwp-scaffolding # Conflicts: # documentation/connect/clients/connect-string.md
This reverts commit 63c24aa.
adjusted for the latest behavioural changes
Address agent-fit findings on the Rust QWP client page: - refute set_null / column_null / column_*_null inference trap - align book table to use ticker column (matches Java/Go/C-C++) - restate Sender single-owner contract in every section that constructs a Sender (Quick start, Builder, Handler callback, Progress modes, Full example) - annotate QwpWsSenderError status / message / message_sequence as Option, including None semantics for WebSocket protocol violations - recommend category (typed enum) over status (raw byte) for programmatic dispatch; link the QWP wire-protocol status codes - inline the Sender construction in the Polling example so the section is self-contained for one-shot agent retrieval - reframe the qwpws / qwpwss alias mention as long-form symmetry Correct a misleading claim caught in follow-up review: column_*_opt(name, None) is a no-op equivalent to omitting the setter, not a wire-level null write at the call site. The wire-level null is gap-filled by the columnar encoder at flush time, only for rows that skipped a column some other row in the same batch did set. This matches the implementation at buffer.rs:996 and the test_buffer_opt_none_skips_columns assertion in tests/buffer_opt.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Rust full example used book(ticker, price, size), conflicting with the array-ingest section earlier on the same page that defines book as (ticker, bids[][], asks[][]). Match Java's full example schema so the page is internally consistent and an agent retrieving any one section generates compatible code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Rust quickstart used bare `?` propagation and never surfaced the async error model. An agent copy-pasting it would silently swallow server-side rejections, since `flush()` only returns local errors. Mirror Java's quickstart by draining `poll_qwp_ws_error()` after `close_drain()`, with a pointer to the handler-callback alternative for long-running senders. Validated end-to-end against questdb-http2: the schema-mismatch path is observed by the poll loop as advertised. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The qwp.max.schemas.per.connection server property is removed as part of the QWP schema-reference simplification, so its configuration reference section no longer describes anything real. The prose also documented the deleted schema registry: distinct schemas registered per connection, rejection of full-schema messages, and a CACHE_RESET that restarts the registry. Drop the whole section. qwp.max.tables.per.connection is a separate limit and stays. Also tidy the SERVER_INFO row in the egress message-kind table, whose trailing whitespace drifted out of column alignment in the Phase 0 edit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The QWP wire docs described only the non-null array row layout. Document the distinction explicitly: a NULL array is carried in the column null bitmap (never inline; nDims==0 is invalid), while a non-null empty array is written inline with nDims>=1 and a 0-length dimension (product 0, no value bytes). Empty arrays of different shapes are distinct values, and decoders must accept a 0-length dimension. - qwp-ingress-websocket.md: add the rule to the Array types section (the canonical encoding). - qwp-egress-websocket.md: expand the array-null note with the empty-array case, the decoder requirement, and a cross-link to the ingress spec. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…scaffolding # Conflicts: # documentation/changelog.mdx # documentation/sidebars.js
The connect/QWP scaffolding rename sweep missed two files, leaving 8 internal link references pointing at moved pages and breaking the build (onBrokenLinks: throw). - changelog.mdx: ilp/overview and telegraf -> /docs/connect/... - concepts/deep-dive/sql-extensions.md: six pgwire links -> /docs/connect/compatibility/pgwire/... yarn build now passes (365 pages, 0 broken links).
The connect-string error-handling section linked to questdb/questdb-enterprise (a private repo, and the referenced sf-client.md path does not even exist there), leaving the QWP error-policy precedence rules inaccessible to public readers. Replace the dead private link with the precedence model inlined and a pointer to the public spec in questdb/java-questdb-client. Cross-checked against the client source: - on_*_error knobs and errorPolicy/errorPolicyResolver are NOT yet implemented (parser does not recognise them, no builder methods); they exist only as the SenderError.Policy spec. Reword accordingly. - errorHandler(...) and error_inbox_capacity are the only wired error-handling surfaces today (WebSocket/QWP only). - Document accepted values + defaults per category and the forced-halt behaviour for PROTOCOL_VIOLATION/UNKNOWN.
The reference client (java-questdb-client) accepts only [http, https, tcp, tcps, ws, wss, udp] and rejects qwpws/qwpwss, so the "long-form alias" claims would break any connect string a reader copies. Standardize on ws/wss across connect-string, rust, and c-and-cpp docs (C/C++ API symbol names like line_sender_qwpws_* are left intact). Also ship the wire-protocols overview as a clean page: remove the "Page in draft" admonition and both TODO stubs, fill Versioning with the actual X-QWP-Max-Version / min(clientMax, serverMax) negotiation, and fix stale "Protocols section" naming. Confirmed wss defaults to 9000 (not 443) against the parser; the explicit call-out stays.
Add a deeper, multi-agent review pass to the docs review-pr skill, modeled on the questdb repo's adversarial reviewer, in both harness variants (.pi uses subagent, .claude uses the Agent tool). - Add review levels (0-2) and a gh pr-based target step that checks out a PR or falls back to the current branch. - Add a change surface map (inbound/outbound links, sidebar neighbors, shared schema/vocabulary, reading paths) as input for the agents. - Add six review agents: structured agents for correctness, placement, information flow, and cross-page navigation/coherence (human + LLM), plus two fresh-context adversarial agents (skeptical reader and LLM one-shot consumer) that run without checklists to escape anchoring. - Add a verification pass that moves false positives to a Downgraded section before reporting.
Member
Author
Review: Connect section, connect-string reference, QWP specsReviewed at level 2 (flagship restructure: new top-level section + 25 page moves + marquee reference pages). Build run against the PR branch; findings verified directly (build, grep, link/anchor resolution, source cross-checks). Verdict: request-changesThe content is genuinely strong — the QWP wire specs and the connect-string reference are some of the best dual-audience material in the repo, the build is clean across 365 pages, and the cross-client schema drift flagged earlier has been properly fixed. But the PR ships with a missing-redirects regression on 25+ indexed pages, an internal self-contradiction in the flagship reference page, several unaddressed review items, unverified performance claims, and non-docs scope creep. These block merge. Blocking issuesCross-cutting1. [Critical] 25+ moved pages have no redirects in this repo's 2. [Major] Scope creep — non-docs artifacts bundled into a docs-content PR.
None are documentation content. Split into separate PR(s); remove 3. [Major] PR description is stale and understates scope. It calls the wire-protocol pages "stubs" and lists client-failover and store-and-forward concepts as out of scope — but the PR ships a ~1,200-line ingress spec, a full egress spec, and complete
|
The .NET client no longer accepts the `auth` key (a pre-built Authorization header value); credentials are supplied as `username`/`password` or `token`. Drop `auth` from the egress query client's Auth key table and update the mutual-exclusivity note to `username`/`password` versus `token`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restructures java.md around QuestDB.connect / Query / Completion / pooled senders, with the existing Sender and QwpQueryClient sections retained as low-level primitives. Switches connect-string examples to ws::/wss:: to match the page's stated QWP focus. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates how-applications-talk-to-QuestDB content under a single Connect supersection. The native QWP clients, the compatibility protocols (ILP / PGWire / REST), and the wire-protocol specifications now share one home. The connect-string reference at
documentation/client-configuration/connect-string.mdanchors the section as the comprehensive knob doc.What's new
Sidebar restructure
Connectsection absorbs Client Libraries, Compatibility Protocols, and Wire Protocols.Connect → Client Libraries.Connect → Compatibility Protocols.Connect → Wire Protocolsfor the client-implementer audience.Connect → Overview (
documentation/ingestion/overview.md)Connect → Connect string reference (
documentation/client-configuration/connect-string.md, new)*Applies to: ingress.*/*Applies to: ingress and egress.*).sf_dirpath handling (no shell expansion, parent must exist).sender_idvalidation (allowed characters, lock-collision semantics).Wire Protocols scaffolding (
documentation/protocols/, new)Touch-ups
#first-party-clients→#client-librariesincapacity-planning.md,monitoring-alerting.md,datatypes/overview.md, andsrc/components/Resources/index.tsx.date-to-timestamp-conversionfrontmatter description from a bare language list to an actual summary.Project handoff (
ONBOARDING.md)Bundle B / A / C team handoff document captured at repo root — assignments, coordination rules, first-prompt templates for the remaining work streams (Wire-protocol content, Client failover, Store-and-forward concepts). Reviewers can opt to keep or remove from this PR.
Out of scope (intentionally)
Connect → Compatibility Protocols → ILP. The connect-string reference is QWP-native; ILP-specific knobs (protocol_version,retry_timeout,request_timeout,request_min_throughput) are not duplicated.Known follow-ups
sf_durability=flush|append, theon_*_errorfamily) so LLM auto-completion is less likely to surface them as available.Add per-language file moves and HTTP-side URL redirects in a separate restructure PR.Redirects now handled in questdb/questdb.io#2917.Test plan
yarn build— 353 pages generated, no broken links / anchorstarget/zoneframing matches the QuestDB Enterprise feature storyONBOARDING.mdshould remain at repo root🤖 Generated with Claude Code