HDDS-13014. Improve PrometheusMetricsSink#normalizeName performance

[ivandika3]

What changes were proposed in this pull request?

From the 5 minutes flamegraph of S3G, we see that nearly 40% of the CPU usage are spent on PrometheusMetrics#putMetrics.
Moreover, 30% of the CPU usage are attributed only for PrometheusMetricsSinkUtil#normalizeName. We see that most of the time are spent on regex matching (probably due to the lookbehind and the lookahead mechanisms).

We need to find a way to improve the performance
Two possible improvements

• Optimize the regex matching or replace it entirely: Perhaps we can take some logic from Prometheus JMX exporter (https://github.com/prometheus/jmx_exporter) or replace it with jmx_exporter implementation.
• Add a name conversion cache between the Hadoop metrics name and the Prometheus metrics name

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13014

How was this patch tested?

Clean CI: https://github.com/ivandika3/ozone/actions/runs/15001669302

Simple microbenchmark. See: https://issues.apache.org/jira/secure/attachment/13076478/TestPrometheusMetricsSinkUtilPerformance.java

# With cache

Warming up...

Running performance test...

Performance Test Results:
Total test cases: 5
Total iterations: 100000
Total operations: 500000
Total time: 47.00 ms
Average time per operation: 0.000 ms

Testing cache hit performance...
Cache hit results:
Total time with cache: 39.00 ms
Average time per operation with cache: 0.000 ms


# Without cache

Warming up...

Running performance test...

Performance Test Results:
Total test cases: 5
Total iterations: 100000
Total operations: 500000
Total time: 1236.00 ms
Average time per operation: 0.002 ms

Testing cache hit performance...
Cache hit results:
Total time with cache: 1167.00 ms
Average time per operation with cache: 0.002 ms

[ivandika3]

Currently for datanode, if datanode still uses the schema V2 (i.e. one RocksDB per container), there might be high memory overhead due to the RocksDB metrics.

I think to reduce the number of metrics stored in the cache, we might need to change the metrics to use tag instead
So instead of

rocksdb_ds_4545195c_120e_47c0_832a_25c4e618250e_container_db_write_raw_block_micros_percentile95{hostname="<redacted>"}

We use

rocksdb_container_db_write_raw_block_micros_percentile95{storageid="ds_4545195c_120e_47c0_832a_25c4e618250", hostname="<redacted>"}

This applies to other similar metrics

[adoroszlai]

Thanks @ivandika3 for the patch. Can we use some kind of size-limited cache?

we might need to change the metrics to use tag
instead of rocksdb_ds_4545195c_120e_47c0_832a_25c4e618250e_container_db_write_raw_block_micros_percentile95{hostname="<redacted>"}
We use rocksdb_container_db_write_raw_block_micros_percentile95{storageid="ds_4545195c_120e_47c0_832a_25c4e618250", hostname="<redacted>"}

This applies to other similar metrics

I think that would also improve usability of metrics, regardless of the cache size problem.

[ivandika3]

@adoroszlai Thanks for the review. I have updated the patch to use a fixed cache size with max size 100,000. Please let me know what you think.

[adoroszlai]

I have updated the patch to use a fixed cache size with max size 100,000

Thanks @ivandika3 for updating the patch, LGTM.

[adoroszlai]

nit (change only if you need to update the patch for any other reason): add import.

[ivandika3]

Apologies, previously I was using Guava Cache interface that conflicts with our internal Cache class.

[ivandika3]

FYI, raised a similar patch to Hadoop as HADOOP-19571 (apache/hadoop#7692)

[adoroszlai]

Thanks @ivandika3 for the patch.

[ivandika3]

Thanks @adoroszlai for the review.