Native Histogram Support for Kubernetes Metrics

Kubernetes components can expose histogram metrics in Prometheus Native Histogram format, alongside the classic histogram format. Native histograms use exponential bucket boundaries instead of fixed boundaries, providing significant storage efficiency, improved query performance, and finer-grained visibility into distributions.

Before you begin

To use native histograms, you need:

• Kubernetes v1.36 or later with the NativeHistograms feature gate enabled.
• Prometheus 2.40 or later to scrape and store native histograms. Prometheus 3.0+ is recommended for per-job configuration.

What are native histograms?

Classic Prometheus histograms use fixed bucket boundaries (for example, [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10] seconds). Each bucket creates a separate time series (_bucket, _count, _sum), which can lead to:

• High storage costs at scale, because each histogram generates many time series.
• Accuracy issues, because data points within a wide bucket range are indistinguishable. For example, a request completing in 1µs and one completing in 4ms both fall into the same le="0.005" bucket.

Native histograms address these limitations by using exponential bucket boundaries that automatically adjust to the data distribution. Benefits include:

• ~10x reduction in time series count per histogram metric, significantly reducing Prometheus storage and improving query performance.
• Finer-grained resolution for detecting performance regressions and setting precise SLO thresholds.

How it works

When the NativeHistograms feature gate is enabled, Kubernetes components expose histogram metrics in both classic and native formats simultaneously (dual exposition). The format returned depends on the Accept header in the HTTP request (Prometheus content negotiation). Prometheus sets this header automatically based on your scrape configuration; you only need to be aware of it when querying the /metrics endpoint directly.

• Text format (Accept: text/plain, OpenMetrics 1.0): Returns only classic histogram buckets. Backward compatible with all existing tooling.

  # Classic histogram buckets (always present)
  apiserver_request_duration_seconds_bucket{le="0.005"} 1000
  apiserver_request_duration_seconds_bucket{le="0.01"} 2000
  ...
  apiserver_request_duration_seconds_bucket{le="+Inf"} 10000
  apiserver_request_duration_seconds_count 10000
  apiserver_request_duration_seconds_sum 450.5
  

• Protobuf format (Accept: application/vnd.google.protobuf): Contains both classic buckets and native histogram data. Prometheus automatically requests this format when scrape_native_histograms: true is set in the Prometheus scrape configuration for the corresponding scrape job.

This dual exposition strategy ensures:

• Existing dashboards and alerts continue to work without changes.
• Users can migrate queries to native histograms at their own pace.
• Prometheus stores whichever format it is configured to ingest.

Enabling native histograms

Enabling native histograms is a two-step process: enable the feature gate on Kubernetes components, and configure Prometheus to scrape native histograms.

Step 1: Enable the Kubernetes feature gate

Enable the NativeHistograms feature gate on the Kubernetes components you want to expose native histograms from:

--feature-gates=NativeHistograms=true

This feature gate applies to the following components:

• kube-apiserver
• kube-controller-manager
• kube-scheduler
• kubelet
• kube-proxy

Each component's metrics are independent; you can enable or disable the feature gate per component.

Step 2: Configure Prometheus

The Prometheus configuration depends on your Prometheus version.

For Prometheus 3.x, use per-job configuration for fine-grained control:

scrape_configs:
  - job_name: 'kubernetes-apiservers'
    scrape_native_histograms: true            # Ingest native histograms
    always_scrape_classic_histograms: true     # Keep classic format during migration

Set both options to true during the migration period. This allows you to ingest native histograms while keeping classic histograms for existing dashboards.

Migrating dashboards and alerts

When migrating from classic to native histogram queries, follow this workflow:

1. Enable both formats: Set scrape_native_histograms: true and always_scrape_classic_histograms: true in your Prometheus scrape config.

2. Migrate queries: Update dashboard queries and alert expressions from classic histogram functions to native histogram equivalents.

   Classic query:

   histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
   

   Native histogram query:

   histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))
   

3. Verify in staging: Test all dashboards and alerts with native histogram queries before rolling out to production.

4. Disable classic scraping: Once migration is complete and verified, set always_scrape_classic_histograms: false to reduce storage overhead.

Disabling native histograms

You can disable native histograms at any time using either of two approaches:

• Prometheus-side (fastest, no Kubernetes restart needed; Prometheus 3.x only): Set scrape_native_histograms: false per scrape job. Prometheus resumes scraping classic format on the next scrape interval.

• Kubernetes feature gate: Restart the component with --feature-gates=NativeHistograms=false. Only classic histogram format is exposed after restart.

When native histograms are disabled, the metrics endpoint reverts to classic histogram format only. Historical native histogram data in Prometheus remains queryable.

Troubleshooting

• Dashboards show no data after enabling native histograms : This occurs when Prometheus is configured with scrape_native_histograms: true but always_scrape_classic_histograms: false (the default), and your dashboards still use classic histogram queries (for example, histogram_quantile(..._bucket...)).

  Fix: Set always_scrape_classic_histograms: true to restore classic format ingestion while you migrate dashboards.

• Memory usage increase after enabling native histograms : A small memory increase is expected for native histogram bucket storage, bounded by a maximum of 160 buckets per histogram. Monitor process_resident_memory_bytes for unexpected increases.

  Fix: If memory pressure is severe, disable native histogram ingestion in Prometheus (scrape_native_histograms: false) or disable the Kubernetes feature gate.

• Prometheus logs errors about unknown metric format : Your Prometheus version is too old to understand native histograms.

  Fix: Upgrade Prometheus to 2.40+ or disable native histograms in Kubernetes.

• Not sure if native histograms are being exposed : Check the feature gate status by querying kubernetes_feature_enabled{name="NativeHistograms"} in Prometheus. A value of 1 indicates the feature is enabled. You can also query the metrics endpoint directly with protobuf format:

  curl -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
    https://<component-address>/metrics
  

  The response should contain native histogram encoding for histogram metrics.

References

• Read the Prometheus Native Histograms documentation for details on the native histogram format and query functions.
• See the Kubernetes metrics reference for the full list of metrics exposed by Kubernetes components.