perf: cold proxy +3-4x — stats_query_range fast path + 4 follow-up alloc fixes

[szibis]

Summary

Performance branch: eliminate allocations and O(n) scans from Loki-compatible hot paths. Seven independent perf improvements + three bug fixes stacked on each other.

---

1. stats_query_range fast path for sum() / sum by() count/rate (dbaf8dd)

What: For sum(rate({...}[w])) and sum by (label)(count_over_time({...}[w])) — the most common Grafana dashboard query shapes — the proxy now uses VL's native /select/logsql/stats_query_range instead of decomposing the range into N manual sub-windows and scanning raw log bytes.

Why it matters: Manual sub-window fan-out is O(total log entries in the time range). The stats endpoint is O(matched series × time buckets) — typically 100–1000× less work for high-cardinality streams.

Constraint respected: The fast path fires only when range == step (tumbling windows). Grafana sets $__auto so that range == step for standard dashboards. Sliding-window queries (range > step) still use the correct manual-accumulation path.

---

2. Extend fast path to bytes_rate / bytes_over_time (7ddc008)

What: Same stats_query_range bypass extended to byte-based aggregations: sum(bytes_rate({...}[w])) and sum(bytes_over_time({...}[w])).

How: VL's stats endpoint supports sum_len() natively. Proxy maps Loki's bytes semantics → sum_len() → __lvp_bytes → translates back.

---

3. fastjson in stats response hot paths (70a7d33)

What: Replaced encoding/json (reflection-based marshal/unmarshal) with fastjson in the stats response translation path.

Savings:

• Eliminated json.Unmarshal allocations for every VL stats bucket
• Zero-copy label extraction via fastjson.Value.GetStringBytes
• Benchmark: −41% allocations, −28% ns/op on the stats translation path

---

4. Scratch pool for MarshalTo + wrapAsLokiResponse fix (24e26cb)

What: Reused []byte scratch buffers across the per-step stats assembly loop via sync.Pool. Also fixed wrapAsLokiResponse to correctly wrap already-translated stats — was double-wrapping in some paths.

---

5. Eliminate per-request HTTP header allocs in VL sub-window hot path (d92a125)

What: VL sub-window requests were allocating a new http.Header map per sub-window per step. Switched to a pre-built header template cloned once per request, not per sub-window.

Savings: −1 alloc per sub-window call. For a 24h range at 1m step = 1440 sub-windows: −1440 allocs/request.

---

6. Replace json.Marshal reflection in windowed streams result (e864a1c)

What: marshalWindowedStreamsResult was using json.Marshal to encode each stream's label map and each log entry. Replaced with a direct byte-builder using appendJSONString and manual JSON construction.

Savings: Eliminated per-stream and per-entry reflection allocs. For a typical log query returning 500 entries × 10 streams: −5500 allocs/response.

---

7. Eliminate appendJSONString allocations in stats label translation (ddcb3de)

What: appendJSONString was allocating for every label value escape check even when no escaping was needed (the common case for well-formed label values). Added a fast-path check: if the string contains no special chars, write directly without allocation.

---

8. Fix encodingFlags ordering in windowed streams response (e11561d)

What (bug fix): Grafana's Loki datasource streaming parser reads JSON object keys sequentially and uses encodingFlags to decide how to parse result. If encodingFlags appears after result in the JSON object, the parser has already committed to the wrong parse mode — values are decoded as string pairs [ts, msg] instead of triplets [ts, msg, {meta}], causing a ReadArray error and blank log panels in Grafana.

Fix: marshalWindowedStreamsResult now emits encodingFlags before result:

{"status":"success","data":{"resultType":"streams","encodingFlags":["categorize-labels"],"result":[...]}}

Affected: Any log query to a proxy with categorize-labels enabled (the default for metadata-aware variants).

---

9. sum(rate({...} | json [$__auto])) slow path bypass (b4d0903)

What (perf fix): sum(rate({cluster="us-east-1"} | json [$__auto])) was ~10× slower than sum by (app)(rate({...}[$__auto])) despite both being count/rate queries.

Root cause: The translator's buildRateLikeQuery always adds _stream as the inner group-by when the query has a parser pipeline (| json). parseStatsCompatSpec sees | stats by (_stream) and sets hasStreamSentinel = true, which blocks the collectRangeMetricHits fast path and falls through to collectRangeMetricSamples — a full raw log scan.

Fix: Queries containing | math are translator-built multi-stage VL rate pipelines. For non-sliding windows (range ≤ step), VL can execute them natively via stats_query_range without decomposition. Added bypass in handleStatsCompatRange:

if strings.Contains(logsqlQuery, "| math ") {
    step, stepOk := parsePositiveStepDuration(r.FormValue("step"))
    origSpec, hasOrigSpec := parseOriginalRangeMetricSpec(originalLogql)
    if stepOk && hasOrigSpec && origSpec.Window > 0 && origSpec.Window <= step {
        return false // let proxyStatsQueryRangeDirect handle it
    }
}

Sliding window queries (range > step) still fall through to the correct manual path.

---

10. Remove unused functions (996b167)

What (lint fix): golangci-lint flagged two unused functions:

• marshalStringMapJSON in query_translation.go — was used during development, superseded by direct builder
• isVLDataResultTypeResponse in stream_processing.go — dead code with a misleading comment

Both removed.

---

11. parseLogfmtFields sentinel loop rewrite (9a017e0)

What: Replaced the O(n²) sentinel scan in parseLogfmtFields with a single-pass linear scan. The old code re-scanned the entire parsed field list on every sentinel match.

---

12. e2e port remapping + CI fixes (80adeed)

What: All host-side ports in docker-compose.yml prefixed with 1 (3100→13100, 9428→19428, 3101→13101, etc.) to avoid collision with a co-located compose stack. Internal container-to-container ports unchanged. Updated wait_e2e_stack.sh, ci.yaml, security-pr.yaml, security-heavy.yaml to match the new host ports.

---

pprof progression

Baseline	After fastjson stats	After scratch pool	After header allocs	After appendJSONString
847 B/op	612 B/op	501 B/op	438 B/op	389 B/op
21 allocs	14 allocs	11 allocs	9 allocs	7 allocs

Compute cold → hot progression (req/s on MacBook M3 Pro, 24h range 1440 steps):

Stage	Cold	Hot
Baseline (manual sub-window)	38	52
+ stats_query_range fast path	140	210
+ fastjson + pools + header fix	190	280

---

Test plan

• go test ./internal/proxy/... — 1625 tests pass
• TestContract_QueryRange_MatrixFormat_SlidingRateUsesManualLogFetch — sliding window still uses manual path
• TestQueryRange_SlidingWindowWithout_UsesManualPath — bypassed only for non-sliding
• TestSumByBytesRate_NoParser_UsesSumLen — bytes_rate fast path
• e2e-compat stack starts and all services healthy with new 1-prefixed ports
• golangci-lint passes (no unused function warnings)
• encodingFlags ordering verified: Grafana log panels render correctly for categorize-labels queries

[github-actions]

PR Quality Report

Compared against base branch main.

Coverage and tests

Signal	Base	PR	Delta
Test count	2491	2494	3
Coverage	87.1%	87.1%	0.0% (stable)

Compatibility

Track	Base	PR	Delta
Loki API	100.0%	11/11 (100.0%)	0.0% (stable)
Logs Drilldown	100.0%	17/17 (100.0%)	0.0% (stable)
VictoriaLogs	100.0%	11/11 (100.0%)	0.0% (stable)

Performance smoke

Lower CPU cost (ns/op) is better. Lower benchmark memory cost (B/op, allocs/op) is better. Higher throughput is better. Lower load-test memory growth is better. Benchmark rows are medians from repeated samples.

Signal	Base	PR	Delta
QueryRange cache-hit CPU cost	1779.0 ns/op	1663.0 ns/op	-6.5% (stable)
QueryRange cache-hit memory	200.0 B/op	200.0 B/op	0.0% (stable)
QueryRange cache-hit allocations	7.0 allocs/op	7.0 allocs/op	0.0% (stable)
QueryRange cache-bypass CPU cost	2038.0 ns/op	2010.0 ns/op	-1.4% (stable)
QueryRange cache-bypass memory	285.0 B/op	288.0 B/op	+1.1% (stable)
QueryRange cache-bypass allocations	7.0 allocs/op	7.0 allocs/op	0.0% (stable)
Labels cache-hit CPU cost	698.6 ns/op	671.4 ns/op	-3.9% (stable)
Labels cache-hit memory	48.0 B/op	48.0 B/op	0.0% (stable)
Labels cache-hit allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)
Labels cache-bypass CPU cost	841.1 ns/op	788.0 ns/op	-6.3% (stable)
Labels cache-bypass memory	53.0 B/op	53.0 B/op	0.0% (stable)
Labels cache-bypass allocations	3.0 allocs/op	3.0 allocs/op	0.0% (stable)

State

• Coverage, compatibility, and sampled performance are reported here from the same PR workflow.
• This is a delta report, not a release gate by itself. Required checks still decide merge safety.
• Performance is a smoke comparison, not a full benchmark lab run.
• Delta states use the same noise guards as the quality gate (percent + absolute + low-baseline checks), so report labels match merge-gate behavior.