[feat][monitor] PIP-223: Add metrics for all rest endpoints.

[dao-jun]

PIP: #18560

Motivation

#18560

Modifications

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: dao-jun#6

[dao-jun]

refers to: #18836

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 86.79245% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 73.59%. Comparing base (bbc6224) to head (e139404).
Report is 19 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21772      +/-   ##
============================================
+ Coverage     73.57%   73.59%   +0.01%     
+ Complexity    32624    32153     -471     
============================================
  Files          1877     1878       +1     
  Lines        139502   139603     +101     
  Branches      15299    15321      +22     
============================================
+ Hits         102638   102737      +99     
+ Misses        28908    28878      -30     
- Partials       7956     7988      +32     

Flag	Coverage Δ
inttests	26.43% <3.77%> (+1.85%)	⬆️
systests	24.31% <3.77%> (-0.02%)	⬇️
unittests	72.84% <86.79%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...org/apache/pulsar/broker/ServiceConfiguration.java	99.39% <100.00%> (+<0.01%)	⬆️
.../java/org/apache/pulsar/broker/web/WebService.java	94.21% <100.00%> (+0.06%)	⬆️
...e/pulsar/broker/web/RestEndpointMetricsFilter.java	85.71% <85.71%> (ø)

... and 74 files with indirect coverage changes

[asafm]

Thanks for implementing this.
Please note that we have already started implementing PIP-264, which is Open Telemetry. We have added the infrastructure needed to define metrics, and we are underway of adding the first metrics.
Once we finish the first PR, can we ping you to add those metrics using OTel? Once that PR we fill finally know the naming conventions to use.

[asafm]

Save yourself this code and the intimate knowledge of how it is implemented:
Create a client like this against the local broker.

            prometheusMetricsClient = new PrometheusMetricsClient("127.0.0.1", pulsar.getListenPortHTTP().get());

Get the Metrics:

            Metrics metrics = prometheusMetricsClient.getMetrics();

and assert example:

            Metric backlogAgeMetric =
                    metrics.findSingleMetricByNameAndLabels("pulsar_storage_backlog_age_seconds",
                            Pair.of("topic", topic1));
            assertThat(backlogAgeMetric.tags).containsExactly(
                    entry("cluster", CLUSTER_NAME),
                    entry("namespace", namespace),
                    entry("topic", topic1));
            assertThat((long) backlogAgeMetric.value).isCloseTo(expectedMessageAgeSeconds, within(2L));

Add the method you lack at Metrics

[asafm]

Please look at #22058 to understand how to do:
Instrument name, description, unit.

[asafm]

Still relevant :)