Skip to content

Add single-slot backend-local ZSTD_DDict cache for repeated dict usage#4

Open
brianr2600 wants to merge 1 commit into
grahamedgecombe:masterfrom
brianr2600:ddict-cache
Open

Add single-slot backend-local ZSTD_DDict cache for repeated dict usage#4
brianr2600 wants to merge 1 commit into
grahamedgecombe:masterfrom
brianr2600:ddict-cache

Conversation

@brianr2600

Copy link
Copy Markdown

This patch introduces a single-slot backend-local ZSTD_DDict* cache to avoid re-parsing the dictionary bytes on every row when a query passes a dictionary by column reference.

ZSTD_decompress_usingDict re-parses the dictionary on every invocation. For queries fetching thousands of rows with a large (e.g., 1MB) dictionary passed per-row, this memory parse cost can dominate execution.

Implementation

  • Computes a fingerprint (dict_len, first-16-bytes, last-16-bytes) from the dictionary argument (the size_t length plus 32 bytes of content).
  • On a cache hit, switches to ZSTD_decompress_usingDDict to bypass the parse phase.
  • On a cache miss, frees the stale DDict and reallocates ZSTD_createDDict().
  • Unloads gracefully via _PG_fini.
  • Preserves the upstream path for dictionaries under the 32-byte threshold.

In local synthetic benchmarking on 1000 rows with a 256KB dict, this dropped the per-row overhead of zstd_decompress from ~11.3 µs to ~2.8 µs (a 4.2x speedup).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant