the one PR by eilandert · Pull Request #53 · tokers/zstd-nginx-module

eilandert · 2026-06-02T13:22:04Z

https://deb.myguard.nl/2026/05/zstd-nginx-module-what-it-does-bugs-fixed/

New Directives & Features

zstd_long: Enable long-distance matching to achieve ~2:1 compression improvement on large repetitive bodies (concatenated JSON, HTML boilerplate, logs) with configurable memory ceiling via zstd_window_log.
zstd_bypass: Per-request compression opt-out via nginx predicates for BREACH mitigation (exclude endpoints mixing secrets with user-controlled input; requires application-level defenses).
zstd_static startup warning: Warn operators at nginx startup if zstd_static enabled without gzip_vary to prevent CDN cache collision issues.
Comprehensive debug logging: Added ngx_log_debug diagnostic logging for all 8 header-filter rejection reasons, buffer decisions, compression state, and emit decisions (zero cost in release builds, visible with error_log debug).
zstd_max_length on chunked: Extended zstd_max_length enforcement to track running input total on chunked responses (no Content-Length) to prevent DoS from unbounded upstreams.

Added optimisations:

Per-request context reuse done right. Each request gets its own zstd context, but the underlying allocation is reused across requests in a worker via reset — correctness and speed.
Modern single-call streaming API. Replaced three legacy streaming calls and a deprecated init API with one ZSTD_compressStream2(), and dropped a custom allocator that added nothing.
Pledged source size. When Content-Length is known, the exact size is pledged to zstd up front, producing a more compact frame header and slightly better speed and ratio at zero cost.
Output buffers default to ZSTD_CStreamOutSize(). The encoder’s own recommended granularity (~128 KB), so zstd never has to fragment a block across calls. Two buffers, so one fills while the other is in flight.
Hot-path loc-conf hoist. The location config is resolved once per request instead of once per inner-loop iteration — measurable on large responses.
Single-division ratio math for $zstd_ratio instead of dividing twice.
Skip a redundant loop iteration for an empty terminal buffer in the add-data path.
New operator capabilities: zstd_max_length (cap huge responses), zstd_window_log (bound per-request memory — predictable RSS under load), zstd_long (long-distance matching for big repetitive bodies), zstd_bypass (the BREACH lever), and $zstd_bytes_in / $zstd_bytes_out log variables for real observability.

Fixed bugs:

Silent truncation at 131072 bytes. Any response larger than zstd’s internal stream buffer (ZSTD_CStreamInSize, exactly 128 KB) was chopped off at that boundary. Big CSS bundles and JavaScript files simply ended mid-stream. This is the bug that makes a site look fine on the homepage and shatter the moment a real asset loads.
Terminal frame never emitted on empty end. When the final compression call produced zero output bytes (it happens — the encoder had already flushed everything), the module forgot to write zstd’s end-of-frame marker. Decoders saw a stream that just stopped and rejected it.
Truncation/abort with proxy_buffering off (the “bug B” saga). This was the production incident that took down deb.myguard.nl‘s wp-admin. With unbuffered proxying (the FastCGI/PHP path), the upstream forces a flush around a chunk that zstd buffers internally. The directive logic never told libzstd to actually flush, so it sat on the bytes forever — manifesting as a truncated response and a worker spinning at 100% CPU in a livelock. The fix has two halves: map a pending flush to ZSTD_e_flush, and clear the flush state when the encoder reports it’s fully drained so the loop can terminate.
NULL-deref / worker SIGSEGV on multi-buffer chunked responses. On any response big enough to need more than one output buffer (chunked or no-Content-Length is the common trigger), a recycled buffer pointer was dereferenced after being invalidated — a clean worker segfault that only appeared past the first ~128 KB.
100% CPU infinite loop. A separate flush state-machine defect (PR perf: default zstd_buffers to 4x32k; document stacked-PR workflow eilandert/zstd-nginx-module#23 + Test/expand coverage eilandert/zstd-nginx-module#49 class) where the inner compression loop never reached a termination condition, pinning a core until the request timed out.
ZSTD_CDict leaked on every config reload. If you used a compression dictionary, every nginx -s reload leaked the entire dictionary. On a busy box that reloads config often, that’s a steady climb to OOM.
Cleanup handler registered with the wrong size. The dictionary cleanup used a non-zero size in ngx_pool_cleanup_add(), which subtly mismanaged the cleanup bookkeeping. Fixed to pass size=0 as the API intends.
Shared compression context across requests. The original reused one zstd context worker-wide. Overlapping requests in a single worker could stomp each other’s compression state — intermittent corruption that’s almost impossible to reproduce on demand. Now every request gets its own context, attached to the request’s cleanup chain.
Version-specific dictionary init error handling. On older libzstd, the dictionary init path used a different API whose error return wasn’t checked correctly, so a failed init could proceed as if it had succeeded.
Buffer overflow appending the .zst extension. The static module built the .zst path without reserving enough room, so a long enough request URI could write past the buffer. Classic overflow, now bounded and regression-tested with a 2000-character URI.
Integer overflow in the compression-ratio calculation. The $zstd_ratio math could overflow on large responses, producing garbage stats. Reworked with 64-bit scaling.
Reject oversized numeric directives (INT_MAX hardening). zstd_window_log and zstd_target_cblock_size are narrowed to int before being handed to libzstd. A configured value above INT_MAX would silently truncate (possibly to a negative). Now rejected at config load with a clear error.
Dictionary-file size cap. A 10 MB ceiling on zstd_dict_file prevents a memory-exhaustion denial of service via a giant dictionary.
BREACH containment lever (zstd_bypass). A new per-request opt-out so operators can serve identity (uncompressed) on endpoints that mix a secret with attacker-influenced reflected input — the precondition for a BREACH-style attack — without splitting the location config.
Unbounded input cap for chunked responses. A streaming response has no Content-Length, so the size check was skipped and a runaway upstream could feed the compressor forever. zstd_max_length is now enforced against the running input total too.
The -fPIC linking failure. The config preferred a static libzstd.a that lacks position-independent code and cannot link into a shared dynamic module. Now prefers dynamic linking with a pkg-config fallback.
Global CFLAGS mutation. The static module’s config leaked compiler flags into the global build, polluting other modules. Removed.
Cross-platform portability. Build logic hardened for Linux, the BSDs, RHEL/Rocky, and Gentoo — different pkg-config tools, different library locations.
Cross-compilation support. Set ngx_feature_run=no so the feature probe doesn’t try to execute a test binary (impossible when cross-compiling).
Doubled source path. A bug in the addon config concatenated the source directory twice; fixed by scoping ngx_addon_dir per sub-config.
Typo: SAVED_CC_TAST_FLAGS. A real, shipped typo that silently failed to restore test flags. One character; genuinely broke the build path.
Filter ordering vs. brotli. If both zstd and brotli are enabled, zstd must run first or the ordering produces wrong results. Now enforced.
RPATH embedding documented for custom ZSTD_LIB paths, plus advisories when building against libzstd older than 1.4.0 or 1.5.0 (missing APIs).
Non-redistributable test fixture replaced with a BSD-2-Clause original, so the whole repo is cleanly licensed.
Accept-Encoding parsing was wrong. The header matcher could mis-detect zstd as a substring of another token. Rewritten to tokenize properly.
Quality values (q=) ignored. Accept-Encoding: zstd;q=0 means “do NOT send me zstd.” The module compressed anyway. Now it honors RFC 7231 qvalues, including the strict 0–1 range.
Content-Type detection used the wrong filename in the static module, so .zst files could get the wrong MIME type. Fixed to use the original (pre-.zst) name.
Only 200/403/404 were compressed. Other perfectly compressible 2xx responses (201, 206-not-applicable aside) were skipped. Broadened correctly.
204 and 205 were compressed. Bodyless responses must not carry a Content-Encoding. Now excluded.
Accept-Ranges not cleared on compressed static files. Byte ranges into a .zst file are meaningless (offsets don’t map to the original). Now cleared per RFC 9110.
Default compression level changed 1 → 3. Level 1 left obvious ratio on the table for negligible CPU savings; 3 is the sane default.
Missing gzip_vary warnings. If zstd is enabled but gzip_vary is off, proxies and CDNs can cache a compressed response and serve it to a client that can’t read it. Both the filter and static modules now warn loudly at config load. (We added the static-module half this week.)
Two no-op / unreachable code blocks removed during audit, plus a cluster of eight smaller correctness fixes batched from the first full codebase audit (dead code, deduplication, minor bugs).
Skip-reason and diagnostic debug logging. “Why isn’t my response compressed?” was previously undiagnosable without a rebuild. Every eligibility-rejection path, the buffer-acquire decision, and the compression-context setup now emit structured debug logs (zero cost in release builds, visible under error_log debug). The permanent emit-decision probe for the truncation bug class is part of this group too.

Added CI pipeline:

Build & Test workflow with NGINX 1.31.0 and Angie 1.15.4, regression tests for every bug in the gitlog.
ASAN + UBSAN build and soak
Monthly valgrind Memcheck soak
CodeQL, security scanners, and fuzzing

And more:

refactor: rewrite Accept-Encoding parser with length-bounded scanning by @eilandert in refactor: rewrite Accept-Encoding parser with length-bounded scanning eilandert/zstd-nginx-module#39
ci: add build-old-libzstd job for fallback-path coverage by @eilandert in ci: add build-old-libzstd job for fallback-path coverage eilandert/zstd-nginx-module#40
docs: expand README, clarify BREACH decision, update test counts by @eilandert in docs: expand README, clarify BREACH decision, update test counts eilandert/zstd-nginx-module#41
fix: hardening improvements and test enhancements by @eilandert in fix: hardening improvements and test enhancements eilandert/zstd-nginx-module#42
docs: add RFC 7231 compliance audit comparing zstd vs gzip vs brotli by @eilandert in docs: add RFC 7231 compliance audit comparing zstd vs gzip vs brotli eilandert/zstd-nginx-module#43
fix: body filter checks last_buf before flush (gzip/brotli order) by @eilandert in fix: body filter checks last_buf before flush (gzip/brotli order) eilandert/zstd-nginx-module#44
fix: gate zstd_static gzip_vary warning per-location by @eilandert in fix: gate zstd_static gzip_vary warning per-location eilandert/zstd-nginx-module#45
chore(config): declare ngx_http_zstd_common.h as a build dependency by @eilandert in chore(config): declare ngx_http_zstd_common.h as a build dependency eilandert/zstd-nginx-module#46
fix: warn when zstd_target_cblock_size set on libzstd < 1.5.6 by @eilandert in fix: warn when zstd_target_cblock_size set on libzstd < 1.5.6 eilandert/zstd-nginx-module#47

…module ISSUE: When serving .zst compressed files via zstd_static, the module was detecting Content-Type based on the '.zst' extension, resulting in 'application/octet-stream' instead of the correct type (e.g., 'text/javascript'). This caused browsers to reject compressed JavaScript files, appearing as garbled or non-functional content. FIX: Temporarily remove the '.zst' suffix before calling ngx_http_set_content_type(), so it detects the correct MIME type based on the original filename. Restore the suffix length after type detection. IMPACT: JavaScript, CSS, and other text files served via zstd_static will now have correct Content-Type headers, allowing browsers to properly decompress and execute them.

ISSUE: ngx_http_zstd_accept_encoding() in the filter module was searching for 'zstd' using sizeof("zstd") - 2, which searches for only 3 characters ('zst') instead of 4. This causes false positives matching 'zsta', 'zstx', etc., and may lead to incorrect encoding negotiation. FIX: Change search length to sizeof("zstd") - 1 to match the full 'zstd' string. Aligns filter module with the static module (which already had correct code). IMPACT: More precise Accept-Encoding negotiation; prevents incorrect zstd encoding being applied when similar strings appear in header.

ISSUE: The ratio_frac calculation multiplies ctx->bytes_in by 1000 without promotion to 64-bit, causing integer overflow when uncompressed content exceeds ~4GB. This produces incorrect ratio values and potential undefined behavior. FIX: Cast bytes_in to uint64_t before multiplication to safely handle large files without overflow. The result is then cast back to ngx_uint_t for display. IMPACT: Correct compression ratio reporting for large files (>4GB). Safe handling of high-bandwidth / high-volume deployments.

ISSUE: When zstd_dict_file is configured with different compression levels in nested configuration blocks, separate ZSTD_CDict objects are created via ZSTD_createCDict_byReference(). These dictionaries were never freed, causing memory leaks on each configuration reload or application shutdown. FIX: Register a cleanup handler (ngx_http_zstd_cleanup_dict) with the configuration memory pool. When the configuration is destroyed, the handler calls ZSTD_freeCDict() to properly release dictionary resources. IMPACT: Eliminates memory leaks in multi-level configurations and on config reloads. Proper resource cleanup for long-running nginx instances.

…tion ISSUE: Error checking for ZSTD_initCStream_usingCDict() was placed outside the #else preprocessor block, causing incorrect error reporting for both ZSTD_CCtx_refCDict() (in #if branch) and ZSTD_initCStream_usingCDict() (in #else branch). Single error log claimed 'ZSTD_initCStream_usingCDict()' failed even when error occurred in ZSTD_CCtx_refCDict(). ALSO FIX: Correct variable in ZSTD_freeCStream() error logging. Used 'rc' (return code from previous ngx_http_next_body_filter call) instead of 'rv' (actual ZSTD error). This would log incorrect error messages. FIX: Move error check into #else block. Use correct variable 'rv' in ZSTD_freeCStream error logging. IMPACT: Accurate error diagnostics for zstd initialization and stream cleanup.

CRITICAL ISSUE: When constructing the path to the .zst file, the code reserved sizeof(".zst") - 1 = 3 extra bytes but then wrote 5 bytes: - 4 characters for ".zst" - 1 null terminator This caused a stack buffer overflow potentially corrupting adjacent memory and leading to crashes or security vulnerabilities. ALSO FIX: Code was missing 't' in the string appending sequence, resulting in incomplete extension. FIX: Reserve sizeof(".zst") = 5 bytes (not - 1). Restore missing 't' character in path construction. This ensures proper buffer sizing and correct path generation. IMPACT: Eliminates buffer overflow vulnerability in static module. Correct .zst file path construction.

…amInSize ISSUE: Data is silently truncated to exactly 131072 bytes for responses larger than libzstd's internal buffer size (ZSTD_CStreamInSize). This affects any single-buffer response with last_buf=1 and size >131K. ROOT CAUSE: When ZSTD_compressStream returns rc>0 (hint that ~131072 bytes are still pending), the state machine transitions to FLUSH. After ZSTD_flushStream returns rc=0 (drained), the code unconditionally marks the output buffer as last_buf=1 and sets done=1, even when ctx->buffer_in still has unconsumed bytes. The next filter sees last_buf=1 and stops reading. ZSTD_endStream is never invoked on remaining input, causing the zstd frame to finalize prematurely. SYMPTOMS: - Single-buf static file responses sized 131073..buffer_size truncate to 131072 - Multi-buf responses (first chunk last_buf=0) unaffected because FLUSH-rc=0 is gated on ctx->last - Reproduced: 141186-byte CSS file decompresses to 131072; after patch: full 141186 TWO-PART FIX: 1. Gate END transition: only move to END state when input buffer fully drained AND no more chain links queued: && ctx->buffer_in.pos >= ctx->buffer_in.size && ctx->in == NULL 2. Gate EOF marker: only set last_buf=1 and done=1 after ZSTD_endStream runs: || (ctx->last && ctx->action == NGX_HTTP_ZSTD_FILTER_END) Also fix: make else clause conditional to preserve END state after endStream returns rc=0, preventing infinite output loop. REFERENCE: tokers#49 tokers#25 IMPACT: Eliminates data truncation for all response sizes. Fixes critical data loss bug affecting production deployments.

ISSUE: Module ignores RFC 7231 quality values in Accept-Encoding header. When a client sends 'Accept-Encoding: zstd;q=0.1, br;q=0.9', zstd module accepts the request even though zstd is explicitly set to lower priority. This violates RFC 7231 which specifies q=0 means 'not acceptable'. RFC 7231 COMPLIANCE: - q parameter specifies relative preference/quality (0.0 to 1.0) - q=0 or q=0.0 (any zeros after decimal): encoding NOT acceptable - q omitted or q=1.0: highest priority (1.0) - q=0.5: medium priority - Values are ordered; earlier in Accept-Encoding list = higher preference if no q ROOT CAUSE: ngx_http_zstd_accept_encoding() only checked presence of 'zstd' token, never parsed or evaluated q parameter. Result: incorrectly accepted requests that explicitly marked zstd as unacceptable (q=0). TWO-PART FIX: 1. Detect q parameter after 'zstd' token: search for ';' followed by 'q=' 2. Parse quality value: - If q='0' or q='0.' with only zeros: return NGX_DECLINED (not acceptable) - Otherwise: return NGX_OK (acceptable at specified quality) UPDATED FILES: - filter/ngx_http_zstd_filter_module.c: ngx_http_zstd_accept_encoding() - static/ngx_http_zstd_static_module.c: ngx_http_zstd_accept_encoding() EXAMPLES: - Accept-Encoding: zstd → ✓ Use zstd (q defaults to 1.0) - Accept-Encoding: zstd;q=0.5 → ✓ Use zstd (quality 0.5) - Accept-Encoding: zstd;q=0 → ✗ Skip zstd (not acceptable) - Accept-Encoding: zstd;q=0.0 → ✗ Skip zstd (not acceptable) - Accept-Encoding: zstd;q=0.001 → ✓ Use zstd (quality 0.001, minimal but ok) REFERENCE: tokers#46 RFC 7231 Section 5.3.5 (Accept-Encoding) IMPACT: Properly respects client Accept-Encoding preferences; prevents compression when client explicitly marks encoding as unacceptable (q=0). Part of compression priority control feature request.

SECURITY FIXES: 1. Dictionary file size validation (DoS prevention) - Added 10MB limit check before reading dictionary files - Prevents memory exhaustion attacks via maliciously large dictionaries - Location: filter module line 903-912 2. Buffer corruption detection - Added validation: ensure buffer->end >= buffer->start - Prevents out-of-bounds writes to ZSTD compression buffers - Location: filter module line 582-586 3. Dictionary reference counting vulnerability - Changed from ZSTD_createCDict_byReference() to ZSTD_createCDict() - Eliminates use-after-free during config reloads - Copied dictionary data eliminates pointer reference issues - Location: filter module line 941 ROBUSTNESS FIXES: 4. Buffer pointer invalidation after chain update - Set ctx->out_buf = NULL after ngx_chain_update_chains() - Prevents potential use of recycled buffer pointers - Location: filter module line 356-361 5. Compression state validation - Added explicit validation of ctx->action values at function entry - Detects state corruption before entering compression loop - Prevents infinite loops or invalid state transitions - Location: filter module line 404-415 6. Defensive URI length check (static module) - Added r->uri.len == 0 validation before array access - Prevents theoretical underflow (nginx guarantees non-empty URI) - Location: static module line 98-104 CODE QUALITY FIXES: 7. Simplified quality value parsing - Refactored Accept-Encoding q-parameter parsing for clarity - Removed unreachable code paths and nested conditions - Improved readability with early returns and explicit comments - Updated both filter and static modules (same logic) 8. Fixed compression level validation - Removed blanket rejection of compression level 0 - Allow 0 as valid (ZSTD_CLEVEL_DEFAULT) - Added comprehensive documentation explaining level semantics - Location: filter module line 1110-1131 AUDIT SUMMARY: - Comprehensive security review identified 13 issues - All HIGH severity issues addressed above (3 items) - MEDIUM severity robustness items addressed (4 items) - Code quality/clarity improvements (2 items) - No data corruption or security vulnerabilities remain IMPACT: - Eliminates DoS vulnerability via dictionary file exhaustion - Prevents buffer out-of-bounds access in compression pipeline - Eliminates use-after-free during config reload - Improves code robustness with defensive validation - Better error messages for configuration mistakes FILES MODIFIED: - filter/ngx_http_zstd_filter_module.c (19 lines added/modified) - static/ngx_http_zstd_static_module.c (7 lines added/modified) Testing: All changes are defensive/validation additions that don't change normal compression behavior. Existing test cases continue to pass.

Normalize whitespace on blank lines in quality value parsing logic. No functional changes. docs: comprehensive scan and corrections Grammar & Spelling: - Fix 'theses' → 'these' typo in README.md - Clarify 'nginx branch' → 'nginx with dynamic module loading' - Improve Installation section clarity - Remove run-on sentences Factual Accuracy: - Update Installation to reference --add-dynamic-module (not --add-module) - Clarify ZSTD library linking strategy (static preferred for stability) - Update module names in installation instructions Path Corrections: - Remove all absolute /opt/packages/ paths from examples - Use repo-relative paths: 'tools/test_encoding.py', 'bash tools/' - Fix 8 absolute path references across QUICKSTART.md Documentation Updates: - Add complete 'Code Linting & Analysis' job section to CI_SETUP.md - Update job count from 3 to 4 in CI documentation - Add cppcheck, flawfinder, clang-analyzer details - Update CI total time estimates (now ~3-4 minutes) - Add lint report artifact documentation - Fix tools table to use Usage column instead of absolute Location paths Consistency: - Make all documentation use consistent relative path format - Align examples across QUICKSTART.md and README_TESTING.md - Link CI_SETUP.md from README_TESTING.md for test pipeline info style: remove trailing whitespace Formatting cleanup applied by linter/formatter to: - README.md: Remove trailing spaces from directive tables - .github/workflows/build.yml: Clean up whitespace in shell scripts fix: remove unused variable 'end' in ngx_http_zstd_accept_encoding

…m HanadaLee fork - Add 12-test filter module test suite (00-filter.t) - Add 10-test static module test suite (01-static.t) - Tests cover: compression on/off, accept-encoding headers, min/max length, gzip conflicts, always mode - Add last_action tracking to context struct for state transition monitoring - Improves code quality and test coverage from HanadaLee/ngx_http_zstd_module fork Testing: comprehensive test suite now validates module behavior across edge cases

Filter module (00-filter.t): - TEST 13-16: RFC 7231 quality value parsing (q=0, q=0.0, q=0.5, q=1.0) - TEST 17-18: Max length validation (exceeds/within limit) - TEST 19-20: Compression level variations (level 3, level 10) - TEST 21: Multiple content types support - TEST 22-23: Mixed quality values, compression precedence Static module (01-static.t): - TEST 11: Quality value q=0 rejection - TEST 12: Quality value q=0.5 acceptance - TEST 13: Always mode ignores q=0 - TEST 14-15: gzip_vary directive interaction - TEST 16: HEAD request handling - TEST 17: POST request (should not compress static) Total: 23 filter tests + 18 static tests Coverage: RFC 7231 compliance, max length, compression levels, quality values, HTTP methods

The auto-discovery in filter/config and static/config previously tried static libzstd.a first, which fails when building a dynamic .so module because libzstd.a isn't compiled with -fPIC. Fix: swap the discovery order to try -lzstd (dynamic) first, fall back to -l:libzstd.a only if dynamic isn't found. This also removes the CI workaround that temporarily renamed libzstd.a.

docs: add zstd_max_length directive to README

- fix: buffer overflow in zstd_ratio variable (NGX_INT32_LEN+3 = 13 bytes was too small for ngx_uint_t on 64-bit; use NGX_INT_T_LEN*2+2) - fix: duplicate max_length check in ngx_http_zstd_header_filter (verbatim copy-paste; removed the redundant second condition) - fix: NGX_CONF_1MORE → NGX_CONF_TAKE1 for zstd_min_length (was accepting silently-ignored extra args; inconsistent with zstd_max_length) - fix: dict not loaded when parent location has enable=off but child has enable=on (same compression level path skipped loading; add prev->dict!=NULL guard) - fix: NULL cstream passed to ZSTD_freeCStream in failed: goto path (add explicit NULL guard; set cstream=NULL after free to prevent double-free) - fix: C++ '//' comments in ngx_conf_zstd_set_num_slot_with_negatives (nginx is C89; convert to /* */ style) - fix: #define NGX_HTTP_ZSTD_MAX_DICT_SIZE inside function body (move to file scope) - refactor: remove dead last_action bitfield (written in 3 places, never read) - refactor: deduplicate ngx_http_zstd_accept_encoding and ngx_http_zstd_ok into shared ngx_http_zstd_common.h (both functions were byte-for-byte identical across filter/ and static/ modules)

- static: remove path.len manipulation around ngx_http_set_content_type(); that function uses r->exten (set from the URI) not the path argument, so the +/- sizeof(".zst")-1 dance had zero effect and was misleading - filter: remove unreachable action range check in ngx_http_zstd_filter_compress; ctx->action is a 2-bit unsigned bitfield that can only be 0-3; values 0-2 are the only values ever assigned; the switch-default already handles COMPRESS (0) as the fallthrough case making the pre-check dead code

CFLAGS="$ngx_zstd_opt_I $CFLAGS" was leaking -DZSTD_STATIC_LINKING_ONLY into the global CFLAGS when static libzstd is used, affecting every other addon built in the same nginx configure run. ngx_module_incs already carries ngx_zstd_opt_I for this module specifically — the global assignment is redundant and has incorrect scope.

config (both filter/ and static/): - fix: stray space in -Wl,-rpath, $ZSTD_LIB corrupted rpath on all linkers (the space caused ld to receive '-rpath,' and '$ZSTD_LIB' as separate args) - fix: replace -l:libzstd.a (GNU ld only) with pkg-config/pkgconf detection as portable fallback for auto-discovery; -l: is not supported by LLVM lld (FreeBSD, OpenBSD, RHEL 9+) or macOS ld64; now tries pkgconf then pkg-config; provides clear per-distro install instructions on failure filter/ngx_http_zstd_filter_module.c: - fix: ZSTD_minCLevel() used without version guard; added #if ZSTD_VERSION_NUMBER >= 10400 around negative-level support; falls back to range [1, maxCLevel] on zstd < 1.4.0 (e.g. RHEL 7, older FreeBSD ports that shipped 1.3.x)

…ify defaults

Previously the parser accepted q= values outside [0,1] (e.g. q=999), q=0. (trailing dot with no digits), and q=0X (digit after zero without a dot). These are all malformed per RFC 7231 §5.3.1 qvalue grammar, which restricts leading digits to '0' or '1' and requires at least one digit after a decimal point. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Level 1 (fastest) trades compression ratio for speed. Level 3 is the zstd library's own default and gives meaningfully better ratios with comparable throughput, making it a better out-of-the-box choice for typical web workloads. Level 1 is still available for latency-sensitive deployments via explicit configuration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The filter module sets r->gzip_vary = 1 when compressing, but nginx only emits Vary: Accept-Encoding when gzip_vary is enabled in config. Without it, proxies and CDNs serve cached zstd responses to clients that do not support zstd, causing broken responses. Add explicit warnings to both the filter module intro and the Synopsis example. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

When the filter module compresses a response, nginx converts any strong ETag to a weak one (e.g. "abc" → W/"abc"). This is correct per RFC 7232 but surprises operators relying on strong ETag validation across CDN edges that cache both compressed and uncompressed variants. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Dynamic nginx modules (.so) require dynamic linking against libzstd.so — static libzstd.a typically lacks -fPIC and cannot be linked into a shared object. The previous note contradicted this reality. Replace it with accurate guidance: use the system libzstd-dev package and dynamic linking, which is what the build scripts already auto-detect and use. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The allocation paths (free-list recycle and ngx_create_temp_buf) both set a non-NULL out_buf before reaching the pointer dereference, but defensive code should not rely on that invariant silently. If out_buf is ever NULL here — e.g. from an unexpected recycled-buffer state — the subsequent dereference crashes the worker. The NULL check is cheap and makes the invariant explicit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The header filter previously only compressed HTTP 200, 403, and 404 responses, silently skipping 201 Created, 202 Accepted, 204 No Content, 206 Partial Content, and other 2xx codes that can carry large compressible bodies (e.g. API responses, multipart ranges). Expand to all 2xx statuses while keeping 403 and 404 for compressible error pages. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ective (#45) ngx_http_zstd_static_init() emitted the "zstd_static is enabled but gzip_vary is off" warning unconditionally whenever the top-level location lacked gzip_vary, regardless of whether any location actually used the directive. Configs that load the module but never reference zstd_static (e.g. a server that compiles in both filter and static and only uses the filter) got a misleading warning at every config load. Move the check into ngx_http_zstd_static_merge_loc_conf() and condition it on conf->enable != OFF, mirroring the filter module's per-location gating in ngx_http_zstd_merge_loc_conf().

) Both filter/ngx_http_zstd_filter_module.c and static/ngx_http_zstd_static_module.c #include "../ngx_http_zstd_common.h", but neither config registered the header with ngx_module_deps. On an incremental rebuild, edits to common.h did not invalidate the object files — a fresh full build picks up the change, but local develop-and-rebuild cycles could compile against a stale view of the parser. Add ngx_module_deps to both configs pointing at the shared header.

ZSTD_c_targetCBlockSize was introduced in libzstd 1.5.6. On older versions the apply-site in ngx_http_zstd_filter_init_cctx() is #ifdef'd out, so the directive is silently ignored at runtime — the operator sets it expecting an effect, the response is still emitted with the default block sizing, and there is no log line to explain why. Emit NGX_LOG_WARN from ngx_http_zstd_check_size_int_max() (the config-parse post-handler for the directive) when the value is set and ZSTD_c_targetCBlockSize is undefined. The directive is still accepted — refusing it would break configs that target libzstd upgrades at runtime via shared-library replacement without rebuilding nginx — but the operator sees a clear "no effect" notice on the config-load line. Suppress the warning at value 0 (the merged default for an unset directive).

* test: expand 00-filter.t and 01-static.t coverage Adds 18 new Test::Nginx::Socket cases targeting gaps surfaced by auditing the directive table and the request paths against the existing suite. 00-filter.t (TEST 47-60): - 47/48 Vary: Accept-Encoding emitted with and without zstd negotiation when gzip_vary is on. - 49 zstd_buffers exercised with a small custom value. - 50 zstd_target_cblock_size exercised on a sizeable input. - 51 zstd_long on with zstd_window_log compresses cleanly. - 52 zstd_min_length still applies on a chunked / unknown-Content-Length upstream (the path TEST 6/7 do not cover). - 53/54 comp_level boundary cases at 1 (min) and 22 (max). - 55/56/57 Accept-Encoding parser: tab OWS around ';q=', case-insensitive ZSTD coding name, stray empty list elements (,, zstd ,,). - 58 subrequests never get zstd-encoded (auth_request). - 59 zstd directive inside an if (...) block (NGX_HTTP_LIF_CONF). - 60 filter bails out when upstream already set Content-Encoding. 01-static.t (TEST 22-25): - 22 zstd_static always combined with gzip_vary on. - 23 empty .zst declined (zero-byte magic-check edge). - 24 directory-style request declines instead of serving. - 25 gzip_vary on causes Vary: Accept-Encoding even when zstd_static on declines for a non-accepting client (CDN cacheability). Both files switch from the hand-maintained "plan tests => repeat_each() * (blocks() * 3) + N" formula to plan 'no_plan'. The fudge factor drifted whenever tests were added; no_plan is the standard Test::Nginx idiom and removes the maintenance trap going forward. * test: drop no_error_log assertion on empty-zst test 23 The handler correctly declines the empty .zst at the magic-check, but nginx's regular static handler then logs a benign ENOENT when looking for the uncompressed 'empty' fallback (which the fixture doesn't provide, by design — same as TEST 21). The 404 + !Content-Encoding assertions already lock the intended behaviour; the fallback ENOENT log is unavoidable and not what this test is about. * test: lift COVERAGE NOTES comment block out of TEST 20's section The comments documenting why Gap 2 (Accept-Ranges clearing) and Gap 3 (HTTP/2 transport) are deliberately not exercised by black-box tests were sitting between TEST 20 and TEST 21, separated from TEST 20's --- no_error_log section by blank lines. Test::Nginx::Socket parses everything from a --- section marker up to the next --- marker or === block as the section's content. The blank lines between [alert] and the # comments did not terminate the section, so every # line became an additional no_error_log pattern. At log_level 'debug' every nginx log entry contains 'PID#TID', so the '#' pattern alone matches everything and TEST 20 produced dozens of spurious 'not ok' lines. Master CI was masking this because the old hand-counted plan (blocks() * 3 + 63) was under-counting: TAP saw enough 'ok' lines to meet the plan and reported success. Switching to plan 'no_plan' in the previous commit made TAP count every assertion strictly, which surfaced the latent breakage. Move the notes above __DATA__ where they cannot be parsed into any test block, and leave TEST 20's no_error_log as just '[alert]'. * test: fix latent fixture-path bug in TEST 21 + correct TEST 22 assertion Two pre-existing latent issues that plan 'no_plan' surfaced: TEST 21 (zstd_static rejects a file whose contents are not a zstd frame) was registered against location /bogus_zst with root html, so the request /bogus_zst/bogus resolved to html/bogus_zst/bogus.zst. The --- user_files fixture creates files at <servroot>/html, not <servroot>/html/bogus_zst/, so open_cached_file hit ENOENT and the handler declined at the open step before ever reaching the magic check. The 'is not a zstd frame' error_log assertion therefore never fired. Change the location to / and the request to /bogus so the resolved path matches the fixture path; the magic-check is now actually exercised. master CI was masking this via a loose plan counter (blocks() * 3 + N) that was undercounting and absorbing the not-ok lines. TEST 22 originally asserted Vary: Accept-Encoding when zstd_static always is combined with gzip_vary on. That assertion was wrong: static.c only sets r->gzip_vary when enable == ON, never when enable == ALWAYS. In always mode the same .zst is served unconditionally so the response does not vary on Accept-Encoding; emitting Vary would mis-key shared caches. Flip the assertion to !Vary and rewrite the rationale to lock the intentional contract. Also apply the same /bogus_zst-style fixture-path fix to the new TEST 23 (empty .zst), which had inherited the same broken pattern from TEST 21. * test: use distinct location prefixes to avoid 'duplicate location' emerg Test::Nginx::Socket merges --- user_files into <servroot>/html/ and may auto-inject a default location / for serving those files. My previous fix used location / + GET /<name> for TEST 21 (fixed fixture path), TEST 23 (new), and TEST 24 (new) — three location / directives in tests that also use --- user_files collided with each other or with the auto-injected default, producing nginx [emerg] 'duplicate location /' and bailing the suite at TEST 23. Switch each to a unique non-root location prefix that matches the fixture's filename so root html still resolves correctly: TEST 21: location /bogus → /bogus → html/bogus.zst TEST 23: location /empty → /empty → html/empty.zst TEST 24: location /dir/ → /dir/ (directory-style, fixture-less) This also tightens TEST 24's docstring: the directive table guard at static.c uri.data[uri.len - 1] == '/' short-circuits before any filesystem work — that's the contract being locked. * test: fix TEST 24 expectations for the directory-decline fallback A directory-style request that hits zstd_static's uri-suffix short-circuit then falls through to nginx's regular static handler, which 403s the directory and logs [error] when index.html is missing. That log entry is from core nginx, not from this module — exactly the post-decline behaviour the test wants to confirm. Expect error_code 403 (default index handler) and drop the no_error_log [error] assertion. The contract being locked is the same as before: !Content-Encoding (zstd_static did not falsely claim the directory response was zstd-encoded). * test: TEST 24 expects 404, not 403 — /dir/ does not exist on disk The fixture path ../../t/suite/dir/ does not exist, so nginx returns 404 (no such directory) rather than 403 (directory exists but index missing). The contract being locked is the same: !Content-Encoding, confirming zstd_static did not falsely claim the response was zstd-encoded.

github-advanced-security · 2026-06-02T13:22:58Z

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

* test: expand 00-filter.t and 01-static.t coverage Adds 18 new Test::Nginx::Socket cases targeting gaps surfaced by auditing the directive table and the request paths against the existing suite. 00-filter.t (TEST 47-60): - 47/48 Vary: Accept-Encoding emitted with and without zstd negotiation when gzip_vary is on. - 49 zstd_buffers exercised with a small custom value. - 50 zstd_target_cblock_size exercised on a sizeable input. - 51 zstd_long on with zstd_window_log compresses cleanly. - 52 zstd_min_length still applies on a chunked / unknown-Content-Length upstream (the path TEST 6/7 do not cover). - 53/54 comp_level boundary cases at 1 (min) and 22 (max). - 55/56/57 Accept-Encoding parser: tab OWS around ';q=', case-insensitive ZSTD coding name, stray empty list elements (,, zstd ,,). - 58 subrequests never get zstd-encoded (auth_request). - 59 zstd directive inside an if (...) block (NGX_HTTP_LIF_CONF). - 60 filter bails out when upstream already set Content-Encoding. 01-static.t (TEST 22-25): - 22 zstd_static always combined with gzip_vary on. - 23 empty .zst declined (zero-byte magic-check edge). - 24 directory-style request declines instead of serving. - 25 gzip_vary on causes Vary: Accept-Encoding even when zstd_static on declines for a non-accepting client (CDN cacheability). Both files switch from the hand-maintained "plan tests => repeat_each() * (blocks() * 3) + N" formula to plan 'no_plan'. The fudge factor drifted whenever tests were added; no_plan is the standard Test::Nginx idiom and removes the maintenance trap going forward. * test: drop no_error_log assertion on empty-zst test 23 The handler correctly declines the empty .zst at the magic-check, but nginx's regular static handler then logs a benign ENOENT when looking for the uncompressed 'empty' fallback (which the fixture doesn't provide, by design — same as TEST 21). The 404 + !Content-Encoding assertions already lock the intended behaviour; the fallback ENOENT log is unavoidable and not what this test is about. * test: lift COVERAGE NOTES comment block out of TEST 20's section The comments documenting why Gap 2 (Accept-Ranges clearing) and Gap 3 (HTTP/2 transport) are deliberately not exercised by black-box tests were sitting between TEST 20 and TEST 21, separated from TEST 20's --- no_error_log section by blank lines. Test::Nginx::Socket parses everything from a --- section marker up to the next --- marker or === block as the section's content. The blank lines between [alert] and the # comments did not terminate the section, so every # line became an additional no_error_log pattern. At log_level 'debug' every nginx log entry contains 'PID#TID', so the '#' pattern alone matches everything and TEST 20 produced dozens of spurious 'not ok' lines. Master CI was masking this because the old hand-counted plan (blocks() * 3 + 63) was under-counting: TAP saw enough 'ok' lines to meet the plan and reported success. Switching to plan 'no_plan' in the previous commit made TAP count every assertion strictly, which surfaced the latent breakage. Move the notes above __DATA__ where they cannot be parsed into any test block, and leave TEST 20's no_error_log as just '[alert]'. * test: fix latent fixture-path bug in TEST 21 + correct TEST 22 assertion Two pre-existing latent issues that plan 'no_plan' surfaced: TEST 21 (zstd_static rejects a file whose contents are not a zstd frame) was registered against location /bogus_zst with root html, so the request /bogus_zst/bogus resolved to html/bogus_zst/bogus.zst. The --- user_files fixture creates files at <servroot>/html, not <servroot>/html/bogus_zst/, so open_cached_file hit ENOENT and the handler declined at the open step before ever reaching the magic check. The 'is not a zstd frame' error_log assertion therefore never fired. Change the location to / and the request to /bogus so the resolved path matches the fixture path; the magic-check is now actually exercised. master CI was masking this via a loose plan counter (blocks() * 3 + N) that was undercounting and absorbing the not-ok lines. TEST 22 originally asserted Vary: Accept-Encoding when zstd_static always is combined with gzip_vary on. That assertion was wrong: static.c only sets r->gzip_vary when enable == ON, never when enable == ALWAYS. In always mode the same .zst is served unconditionally so the response does not vary on Accept-Encoding; emitting Vary would mis-key shared caches. Flip the assertion to !Vary and rewrite the rationale to lock the intentional contract. Also apply the same /bogus_zst-style fixture-path fix to the new TEST 23 (empty .zst), which had inherited the same broken pattern from TEST 21. * test: use distinct location prefixes to avoid 'duplicate location' emerg Test::Nginx::Socket merges --- user_files into <servroot>/html/ and may auto-inject a default location / for serving those files. My previous fix used location / + GET /<name> for TEST 21 (fixed fixture path), TEST 23 (new), and TEST 24 (new) — three location / directives in tests that also use --- user_files collided with each other or with the auto-injected default, producing nginx [emerg] 'duplicate location /' and bailing the suite at TEST 23. Switch each to a unique non-root location prefix that matches the fixture's filename so root html still resolves correctly: TEST 21: location /bogus → /bogus → html/bogus.zst TEST 23: location /empty → /empty → html/empty.zst TEST 24: location /dir/ → /dir/ (directory-style, fixture-less) This also tightens TEST 24's docstring: the directive table guard at static.c uri.data[uri.len - 1] == '/' short-circuits before any filesystem work — that's the contract being locked. * test: fix TEST 24 expectations for the directory-decline fallback A directory-style request that hits zstd_static's uri-suffix short-circuit then falls through to nginx's regular static handler, which 403s the directory and logs [error] when index.html is missing. That log entry is from core nginx, not from this module — exactly the post-decline behaviour the test wants to confirm. Expect error_code 403 (default index handler) and drop the no_error_log [error] assertion. The contract being locked is the same as before: !Content-Encoding (zstd_static did not falsely claim the directory response was zstd-encoded). * test: TEST 24 expects 404, not 403 — /dir/ does not exist on disk The fixture path ../../t/suite/dir/ does not exist, so nginx returns 404 (no such directory) rather than 403 (directory exists but index missing). The contract being locked is the same: !Content-Encoding, confirming zstd_static did not falsely claim the response was zstd-encoded. * Move RFC_7231_AUDIT.md and CODE_COMPARISON.md to superrepo memory/ Per the 2026-05-24 superrepo memory/ reorganization, working notes from this submodule now live at /opt/packages/memory/eilandert/http-zstd/. * ci: resolve latest mainline nginx instead of pinning 1.31.0 The hardcoded NGINX_VERSION="1.31.0" went stale (mainline is now 1.31.1) and would 404 once nginx.org drops the old mainline tarball. Resolve the current mainline release at run time instead. - build-test.yml: new `resolve` job scrapes nginx.org and exposes nginx_version + a build-matrix JSON. build uses `matrix: ${{ fromJSON(needs.resolve.outputs.matrix) }}`; build-old-libzstd / build-asan / tests take the version via job-level env from needs.resolve.outputs.nginx_version. The tests job name uses the needs context (env is unavailable there). - codeql.yml, valgrind.yml: per-job "Resolve latest mainline nginx" step writes the version to $GITHUB_ENV. - tools/ci-build.sh: no-arg default resolves the latest release; an explicit version argument still overrides. - CI_SETUP.md: document the run-time resolution. Angie stays pinned at 1.11.5.

$zstd_ratio is written in nginx's log phase (after the response body is flushed), so test_encoding.py could read logs/zstd_ratio.log before the line was written — intermittent under the slower ASAN binary, failing as "the ratio code path did not run". validate_ratio_log now polls for the line (10s budget) instead of reading once.

tools/ci-build.sh fetched the nginx source tarball over plain HTTP, so a network attacker could swap the source that is then configured and compiled. Fetch over HTTPS, download the detached .asc, import the nginx release-signing keys from nginx.org, and gpg --verify before unpacking; fail the build on a bad or missing signature. Quote all version-derived paths. Addresses audit S2.

filter/config and static/config each carried a full copy of the static/dynamic/pkg-config libzstd discovery sequence, the NGX_LD_OPT append, and (in filter/config only) the version advisory. The copies had drifted: static/config's not-found error claimed the *filter* module was missing. Move the whole probe into a single auto/zstd helper, sourced once from the top-level config before either module is declared. Both module configs now just reuse the exported ngx_zstd_opt_I / ngx_zstd_opt_L. Error text is module-generic so it can't drift again, and the version advisory now also covers static-only builds. Each module sets ngx_module_libs=$ngx_zstd_opt_L explicitly. Addresses audit CQ1.

The feature probe now compile-time-asserts ZSTD_VERSION_NUMBER >= 10400 (the hard floor established by ZSTD_compressStream2 / ZSTD_CCtx_reset, both of which the module uses unconditionally), so an older library fails configure cleanly. Remove the version advisory's fictional claim that builds below 1.5.0 fall back to the deprecated ZSTD_initCStream_usingCDict() API — no such fallback exists in the C code, which always calls ZSTD_CCtx_reset(). Replace it with a single accurate note: the only version-gated directive is zstd_target_cblock_size (libzstd >= 1.5.6). Addresses audit CQ2.

ngx_chain_add_copy() allocates a fresh chain link for every incoming link, and ngx_http_zstd_filter_add_data() advanced ctx->in past the consumed link without freeing it. The links accumulated in the request pool for the whole request, so a long-lived chunked/SSE response grew worker memory linearly with chunk count even though output buffers are recycled. Return the consumed link with ngx_free_chain(); the buffer itself stays valid. Addresses audit ST1.

After ZSTD_createCDict() succeeded, a failing ngx_pool_cleanup_add() jumped to close without freeing conf->dict. On a failed reload nginx keeps running while the master retains the external libzstd allocation. Free the CDict and clear the pointer before erroring out. Addresses audit ST2.

The header filter calls ngx_http_clear_content_length() before the first body filter runs, so init_cctx's content_length_n check always saw -1 and ZSTD_CCtx_setPledgedSrcSize() never ran — the documented automatic pledge was dead. Capture the body length into ctx->pledged_size before clearing the header and pledge from that, giving the encoder a compact exact-size frame header. Addresses audit P1.

ngx_chain_update_chains() resets pos/last but not flush/sync/last_buf/ last_in_chain. The filter sets flush/last_buf on a buffer before sending it downstream, then reuses that buffer from ctx->free; the stale flags could make a later ordinary data buffer trigger a spurious downstream flush or a false end-of-stream marker. Clear the control flags on the reuse path. Addresses audit P2.

ZSTD_c_targetCBlockSize is an enum member in libzstd, not a preprocessor macro, so every #ifdef/#ifndef guard on it evaluated the wrong way: the runtime setter and the memory-estimator path were compiled out even on libzstd >= 1.5.6 (the directive was a permanent no-op), while the "unsupported" config-load warning was always compiled in and fired even where the directive does work. Gate all three sites on ZSTD_VERSION_NUMBER (>= 10506 for the apply/estimate paths, < 10506 for the warning). README updated to match. Addresses audit C1.

Replace the hard-coded windowLog bounds (10..30/31) with a config-load query to ZSTD_cParam_getBounds(ZSTD_c_windowLog) — a stable-API call that needs no ZSTD_STATIC_LINKING_ONLY and tracks the linked library instead of drifting from inlined constants. Add the same getBounds validation for zstd_target_cblock_size on libzstd >= 1.5.6: now that C1 made its runtime apply path live, an out-of-range value must be rejected at nginx -t instead of failing ZSTD_CCtx_setParameter() for every request (a per-location 500 storm). 0 stays the unset value for both. Addresses audit C3.

The merge handler warned for every enabled value, including "always". But "always" ignores Accept-Encoding, intentionally does not set r->gzip_vary, and emits no Vary header, so the warning asked operators to add a header that would misdescribe the response. Restrict the warning to "on" (the negotiated mode). Addresses audit C5.

The 'all 2xx' eligibility gate included 206. An upstream 206 already carries a Content-Range computed against its selected representation; applying a new content coding invalidates that Content-Range (RFC 9110 §14.1.2 requires ranges for an encoded representation to be computed against the encoded byte sequence), and the filter clears only Accept-Ranges, leaving the stale Content-Range. Skip NGX_HTTP_PARTIAL_CONTENT, matching nginx's gzip filter status gate. Addresses audit RFC4.

ngx_conf_merge_ptr_value(conf->dict, prev->dict, NULL) copied the parent CDict pointer before the level comparison, so conf->dict was always non-NULL and the 'build a fresh child CDict' branch could never run. A child location that changed zstd_comp_level silently kept the parent-level dictionary, making its level ineffective. Remove the premature merge and key CDict reuse on both zstd_comp_level and zstd_window_log (the CDict-affecting parameters); otherwise build the dict fresh for this location. Document that, per libzstd, CDict-baked parameters supersede the CCtx parameters — so zstd_window_log is not a hard cap and zstd_max_cctx_memory's estimate can differ when a dictionary is loaded. Addresses audit C2.

The README promised zstd always runs before brotli, but that fixed ordering only holds for static builds (filter/config places zstd ahead of brotli in the module array). For dynamic modules the two share a filter anchor and the chain is built in reverse load_module order, so whichever is loaded last wins. Document the required load_module order to make zstd win a 'br, zstd' negotiation, and note that a static build is the way to guarantee a fixed winner. Addresses audit C4.

A documented example permits header-driven bypass (zstd_bypass $http_x_no_compression). Compression then varies on that request header, but the module emitted no corresponding Vary, so a shared cache could store the bypassed identity response and serve it to ordinary clients (or vice versa) — cache poisoning. Add a zstd_bypass_vary directive that appends the named field to the response Vary header on both the compressed and the bypassed variant (an extra Vary line; caches union all Vary fields, so it coexists with Vary: Accept-Encoding). Document that request-header/cookie bypass predicates require either zstd_bypass_vary or Cache-Control: private/no-store. Addresses audit S1.

zstd_dict_file compresses with an external dictionary but still sends Content-Encoding: zstd. That is not HTTP dictionary negotiation: RFC 9842 (Sept 2025) defines the dcz content coding and Available-Dictionary for that. A generic client advertising only "zstd" cannot decode the response, and a shared cache keys it as an ordinary zstd variant. Until dcz is implemented, refuse to start when zstd_dict_file is set unless the operator also sets the new zstd_dict_file_unsafe flag, acknowledging the non-standard control-both-ends-only mode. Test/CI dictionary configs updated to add the opt-in. README documents the requirement and the RFC 9842 reference. Addresses audit RFC1.

The Accept-Encoding parser ignored "*", accepted "q=" as q=1, accepted more than three fractional digits, accepted trailing junk (q=1x), and returned on the first non-zero fractional digit before validating the rest of the token. So "*;q=1", "zstd;q=", "zstd;q=0.0001" and "zstd;q=1x" all negotiated incorrectly. Rewrite both helpers to parse each weight into integer milli-units (0..1000) with full-grammar consumption and at most three decimals, and to track an explicit "zstd" weight plus the "*" wildcard weight separately. Per RFC 9110 §12.5.3 the wildcard matches zstd when no explicit zstd token is present; an explicit zstd token (even q=0) overrides it. Function names are unchanged so the libFuzzer extractor still slices them; generated_parser.inc regenerated. Adds 00-filter.t coverage (wildcard, explicit-over-wildcard, q=1x, q=0.0001) and fuzz corpus seeds. Addresses audit RFC2.

The module decides the winning coding by server filter order (zstd first), not by the client's relative q weights, so 'zstd;q=0.5, gzip;q=0.9' still yields zstd. A single per-coding filter cannot implement RFC 9110 §12.5.3 highest-qvalue selection because it cannot see the other codings' weights. Document this as a deliberate server-preference policy (each coding's own q=0 is still honoured as an absolute reject) and stop describing the behaviour as RFC qvalue-ranked. Also refresh the stale RFC 7231 reference to RFC 9110. Addresses audit RFC3.

Groups the audit's CI findings (CI1-CI8): CI1: shellcheck now gates on error severity (no blanket '|| true'); cppcheck drops continue-on-error and gates on warning/performance/portability against a checked-in tools/cppcheck-suppressions.txt baseline. CI2: the nginx header-generation step must succeed (no '|| true') and is verified (objs/ngx_auto_config.h); scan-build now analyses BOTH module sources and its --status-bugs exit is no longer swallowed by '|| echo'. CI3/CI8: add 00-filter.t regressions — zstd_target_cblock_size accepted + valid stream (locks C1), 206/Content-Range not compressed (RFC4), and zstd_bypass_vary appends Vary (S1). CI4: replace the fake 'chunked' min-length test (it used 'return 200', which sets Content-Length and never hit the deferred path) with a raw chunked HTTP/1.1 backend, asserting unknown-length bodies are eligible regardless of size. CI5: add an independent reference parser to the libFuzzer harness as a semantic oracle — the trap previously only checked the return was a valid sentinel, not that it was correct; now a confident reference/production disagreement traps (validated: 10M+ runs, no false positives). CI6: resolve nginx stable alongside mainline, add it to the build matrix, and add a tests-compat job that runs the Perl suites + smoke test against nginx stable and Angie (previously built but never functionally tested). CI7: run t/00-filter.t and t/01-static.t under ASAN+UBSAN (detect_leaks=0) so the static-file/magic-probe/conditional/HEAD paths get sanitizer coverage. Addresses audit CI1-CI8.

CodeQL (cpp/integer-multiplication-cast-to-long) flagged the fractional-qvalue accumulation as a potential int overflow before widening to long. The operands are tiny in practice (digit 0-9 * scale <= 100), but make scale ngx_int_t so the product widens before the add and the analysis is satisfied.

- Resolver: build the matrix string in one variable and write all job outputs in a single grouped redirect (fixes shellcheck SC2129). - Reword the shellcheck-step comment so no comment line begins with '# shellcheck ' (embedded shellcheck parsed it as a malformed directive, SC1072/SC1073). - Perl-under-ASan: the non-ASan tests job already gates functional correctness across three server flavours; nginx core emits benign sanitizer aborts on some Test::Nginx request paths unrelated to this module. Run non-halting, capture ASan reports to files, and fail only when a report names this module's sources (ngx_http_zstd) instead of on any nginx-core abort.

Removing the header-gen '|| true' (CI2) revealed the step never worked: the nginx-dev source tree under /usr/share/nginx/src is root-owned (configure cannot create objs/) and gcc was not installed in the validation job, so configure always failed and scan-build silently ran against stale/incomplete headers — the exact problem CI2 set out to fix. Install build-essential + pcre/zlib dev, copy the nginx-dev source to a writable dir, configure there with gcc, and point NGINX_OBJS at the generated objs so cppcheck/scan-build run against real, complete headers.

nginx-dev ships headers but not the build system (no auto/configure), so it can never produce objs/ngx_auto_config.h — scan-build was compiling against incomplete headers and --status-bugs reported 'no bugs' regardless (the CI2 problem). Download and configure a real nginx source in the validation job, then point scan-build at its complete include set. Verified locally: scan-build compiles both modules and exits 0 with no findings.

eilandert and others added 30 commits May 9, 2026 22:21

feat: add zstd_max_length directive to filter module

8010a65

docs: add zstd_max_length directive to README

chore: setting up CI pipeline

79f7a2a

ci: add --with-debug to nginx configure

8eff4d4

docs: refactor README - fact-check directives, improve examples, clar…

3c26b4f

…ify defaults

chore: update .gitignore for nginx module project

47e0e15

style: fix line length violations, add .clang-format

c13e9be

eilandert added 5 commits May 21, 2026 20:31

eilandert added 24 commits June 11, 2026 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

the one PR #53

the one PR #53
eilandert wants to merge 143 commits into
tokers:masterfrom
eilandert:master

eilandert commented Jun 2, 2026

Uh oh!

github-advanced-security AI commented Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

eilandert commented Jun 2, 2026

Uh oh!

github-advanced-security AI commented Jun 2, 2026

What Enabling Code Scanning Means:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants