Skip to content

HBASE-30242 Add Region server level HFile Compression Metrics#8389

Open
Umeshkumar9414 wants to merge 1 commit into
apache:masterfrom
Umeshkumar9414:HBASE-30242
Open

HBASE-30242 Add Region server level HFile Compression Metrics#8389
Umeshkumar9414 wants to merge 1 commit into
apache:masterfrom
Umeshkumar9414:HBASE-30242

Conversation

@Umeshkumar9414

@Umeshkumar9414 Umeshkumar9414 commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Local testing
image

Note : In case of compression NONE, ratio goes above 1 becuase storeFileSize includes HFile trailer/index overhead that isn't counted in the uncompressed data blocks.

Co-authored-by: Opus 4.6 <noreply@anthropic.com>
}

@Override
public double getStoreFileCompressionRatio() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally for compression factor, higher is better. This is what command line compression tools like xz or zstd or etc. typically report. And for compression savings, higher is also better, e.g 1 − compressed / uncompressed, which is easy to read as a percentage. This method provides the inverse of that so it makes the desired direction "down", which is awkward for Prometheus or Grafana dashboards. Operators reading a dashboard tile labeled storeFileCompressionRatio with a value e.g. of 0.33 will routinely interpret it the opposite of what is intended.

Recommendation: either rename to storeFileCompressedToUncompressedRatio so the direction is clearly established by the name of the metric, or reimplement as uncompressed / compressed, or storeFileCompressionSavingsRatio as (1 − compressed/uncompressed).

@Override
public double getStoreFileCompressionRatio() {
long uncompressed = aggregate.storeFileUncompressedSize;
if (uncompressed == 0) {

@apurtell apurtell Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A value of "1.0" will generally look identical to an operator like somehow compression has stopped working, and will generate false positive alerts.

(Claude calls this a "footgun for dashboards", lol. He means well...)

Recommendation: Return 0.0 instead, and update the description to something like "Returns 0.0 when there is no data."

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds RegionServer-level metrics for total uncompressed HFile bytes and an overall store-file compression ratio, surfaced through the existing MetricsRegionServer wrapper/source plumbing and validated via unit tests.

Changes:

  • Extend MetricsRegionServerWrapper with getStoreFileUncompressedSize() and getStoreFileCompressionRatio().
  • Compute/aggregate uncompressed store bytes in MetricsRegionServerWrapperImpl and export both gauges via MetricsRegionServerSourceImpl.
  • Update RegionServer metrics unit tests and wrapper stubs to cover the new gauges.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java Asserts the two new gauges are emitted by the metrics source.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperStub.java Adds stub implementations returning deterministic values for the new metrics.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java Aggregates uncompressed bytes from stores and computes RS-level compression ratio.
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapper.java Extends wrapper interface with uncompressed size and compression ratio accessors.
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java Exports the new wrapper values as gauges in the metrics record.
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java Defines metric names and descriptions for the two new gauges.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +124 to +128
/**
* Get the compression ratio of store files on this region server. This is the ratio of compressed
* on-disk size to uncompressed data size. Returns 1.0 when there is no data.
*/
double getStoreFileCompressionRatio();
Comment on lines +249 to +252
String STOREFILE_UNCOMPRESSED_SIZE_DESC = "Total uncompressed size of storefiles being served.";
String STOREFILE_COMPRESSION_RATIO_DESC =
"Compression ratio of storefiles (compressed/uncompressed). Lower values indicate better"
+ " compression. Returns 1.0 when there is no data.";
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants