Track compressed resource state sizes in deploy telemetry (direct engine)#5608
Draft
shreyas-goenka wants to merge 1 commit into
Draft
Track compressed resource state sizes in deploy telemetry (direct engine)#5608shreyas-goenka wants to merge 1 commit into
shreyas-goenka wants to merge 1 commit into
Conversation
…ine)
Deploy telemetry already reports per-resource-type raw state-size statistics
(state_size_{max,mean,median}_bytes). The deployment metadata service stores
that same per-resource state compressed, so this adds compressed-size
counterparts to gauge how much resource state shrinks under compression rather
than just the raw sizes:
- state_compressed_size_max_bytes
- state_compressed_size_mean_bytes
- state_compressed_size_median_bytes
The compressed length is computed once per resource at state-export time
(alongside the existing raw length) using the standard library's compress/flate.
This is a deliberately rough proxy for the server side (which uses zstd): flate
keeps the dependency/supply-chain surface small while still giving useful signal
on compressibility. Only the direct engine is measured, matching the existing
raw-size behavior.
Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
Integration test reportCommit: ece87d5
24 interesting tests: 15 SKIP, 7 KNOWN, 2 flaky
Top 30 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bundle deploy telemetry already reports per-resource-type raw state-size statistics (
state_size_{max,mean,median}_bytesinResourceMetadata). The same per-resource state is stored compressed downstream, so this adds the compressed-size counterparts to gauge how much resource state shrinks under compression, not just the raw sizes:state_compressed_size_max_bytesstate_compressed_size_mean_bytesstate_compressed_size_median_bytesHow
compressedStateSize(inbundle/direct/dstate) compresses each resource's serialized state once at export time, alongside the existinglen(state)raw measurement.compress/flaterather than a dedicated zstd package: a deliberately rough proxy for the server side (which uses zstd), chosen to keep the dependency/supply-chain surface small while still giving useful signal on compressibility. No new dependency is added.ResourceStatecarries the newStateCompressedSizeBytes;resourceMetadataFromStateaggregates max/mean/median per resource type, mirroring the raw stats.Performance
flateruns at state-export time over individually small resource states (each well under the server's per-resource limit), not in a tight loop, so even large bundles compress in a few milliseconds — negligible next to a deploy's network I/O. No background goroutine is warranted.Tests
TestCompressedStateSize(empty → 0, compressible blob shrinks) plus updatedresources_metadataaggregation tests.flate, so they are asserted directly (no redaction).Notes
This pull request and its description were written by Isaac.