Skip to content

Payloadbuilder Query Engine

Jump to bottom
Marcus Henriksson edited this page Apr 2, 2026 · 2 revisions

Payloadbuilder Query Engine

Payloadbuilder is Queryeer's built-in query engine. It implements a SQL-like query language that can federate queries across multiple heterogeneous data sources — called catalogs — in a single query. Unlike the JDBC engine which connects to a single relational database, Payloadbuilder can join data from an Elasticsearch index with a local CSV file and an HTTP API endpoint in one statement.

Overview

Payloadbuilder is a separate open-source project (github.com/kuseman/payloadbuilder) that is embedded in Queryeer. It provides:

  • A SQL-like query language parsed by an ANTLR grammar
  • A catalog plugin API for adding new data source types
  • An execution engine that evaluates queries against one or more catalogs
  • A schema/metadata API consumed by Queryeer's code completion

Catalogs

A catalog in Payloadbuilder is a named connector to a specific data source type. Multiple catalog instances can be configured and referenced in a single query using a <catalog>.<table> syntax.

Queryeer ships with the following built-in catalogs:

Elasticsearch Catalog

Query Elasticsearch indices using SQL-like syntax.

Example

SELECT *
FROM es#"index/my_index" i
WHERE i.status = 'active'
  AND i.created_at > '2024-01-01'
LIMIT 100

Features

  • Full-text search predicates mapped to Elasticsearch match and term queries
  • Aggregation support
  • Nested field access with dot notation
  • Configurable Elasticsearch host, port, and authentication
  • Index metadata crawled for code completion (field names and types)

Configuration Open Options → Payloadbuilder → Elasticsearch to set:

  • Hostname and port
  • Authentication (basic auth or API key)
  • Default index prefix

Filesystem Catalog

Query files and directories on the local filesystem as if they were database tables.

Example — list files in a directory

SELECT name, size, lastModified
FROM fs#"file/C:/data/exports"
WHERE extension = 'csv'
ORDER BY lastModified DESC

Example — read lines from a text file

SELECT line, lineNumber
FROM fs#"line/C:/logs/app.log"
WHERE line LIKE '%ERROR%'

Features

  • Browse directory contents (file name, size, dates, extension)
  • Read file contents line by line
  • Pattern-based file filtering
  • Cross-catalog joins between file data and other sources

HTTP Catalog

Query HTTP/REST endpoints and treat the JSON response as a table.

Example

SELECT id, name, status
FROM http#"endpoint/https://api.example.com/users"
WHERE status = 'active'

Features

  • GET requests with optional headers and query parameters
  • JSON response automatically flattened into rows and columns
  • Configurable base URL, headers, and authentication
  • Supports paginated APIs through catalog-level configuration

Joining Catalogs

A key feature of Payloadbuilder is the ability to join data from different catalog types in a single query:

-- Join Elasticsearch data with a local CSV file
SELECT u.id, u.name, f.region
FROM es#"index/users" u
INNER JOIN fs#"line/C:/data/regions.csv" f
  ON u.region_id = f.id
WHERE u.active = true

The Payloadbuilder execution engine handles cross-catalog joins by pulling data from each source and performing the join in memory.

Query Language

Payloadbuilder's query language is a subset of SQL extended with catalog-specific table reference syntax:

<catalog>#"<type>/<path or name>"

Supported SQL constructs:

  • SELECT with column aliases and expressions
  • FROM with single and multi-table sources
  • WHERE with comparison, logical, and LIKE operators
  • JOIN (INNER, LEFT, CROSS) across tables and catalogs
  • GROUP BY and aggregate functions (COUNT, SUM, MIN, MAX, AVG)
  • ORDER BY with ASC / DESC
  • LIMIT / TOP for result truncation
  • Subqueries and CTEs (WITH ... AS (...))
  • Built-in scalar functions (string, date, math, type conversion)

Code Completion

When a Payloadbuilder query is open, the editor's completion engine is aware of:

  • Configured catalog names (as table prefixes)
  • Field/column names from connected catalogs (Elasticsearch field names, filesystem columns, etc.)
  • Built-in Payloadbuilder functions
  • Table aliases defined earlier in the same query

Adding Custom Catalogs

New catalog types can be added via the plugin system by implementing the ICatalogExtensionFactory interface. A custom catalog must provide:

  • A Catalog implementation with table/schema metadata
  • An operator that executes scan/filter/projection against the data source
  • Optional: a configuration panel shown in the Options dialog

Once packaged as a JAR and placed in plugins/, the new catalog type is available to all Payloadbuilder queries.

Configuration

Payloadbuilder and its catalogs are configured under Options → Payloadbuilder. Each catalog instance has its own sub-section where connection details, credentials, and catalog-specific options are set.

The IPayloadbuilderService is also available as an injectable service for use by other plugins that want to execute Payloadbuilder queries programmatically.

Clone this wiki locally