# Traffic Metrics

Kuma facilitates consistent traffic metrics across all data plane proxies in your mesh.

You add metrics to a mesh configuration, or to an individual Dataplane configuration if you need, for example, to override the default metrics port that's already in use on the specified machine.

Kuma provides full integration with Prometheus:

  • Each proxy can expose its metrics in Prometheus format.
  • Because metrics are part of the mesh configuration, Prometheus can automatically find every proxy in the mesh.

To collect metrics from Kuma, you need to first expose metrics from proxies and then configure Prometheus to collect them.

# Expose metrics from data plane proxies

To expose metrics from every proxy in the mesh, configure the Mesh resource:

    This tells Kuma to configure every proxy in the default mesh to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

    The metrics endpoint is forwarded to the standard Envoy Prometheus metrics endpoint (opens new window) and supports the same query parameters. You can pass the filter query parameter to limit the results to metrics whose names match a given regular expression. By default all available metrics are returned.

    # Override Prometheus settings per data plane proxy

      # Configure Prometheus

      Although proxy metrics are now exposed, you still need to let Prometheus discover them.

      In Prometheus version 2.29 and later, you can add Kuma metrics to your prometheus.yml:

      scrape_configs:
          - job_name: 'kuma-dataplanes'
            scrape_interval: "5s"
            relabel_configs:
            - source_labels:
              - __meta_kuma_mesh
              regex: "(.*)"
              target_label: mesh
            - source_labels:
              - __meta_kuma_dataplane
              regex: "(.*)"
              target_label: dataplane
            - source_labels:
              - __meta_kuma_service
              regex: "(.*)"
              target_label: service
            - action: labelmap
              regex: __meta_kuma_label_(.+)
            kuma_sd_configs:
            - server: "http://kuma-control-plane.kuma-system.svc:5676"
      
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20

      For more information, see the Prometheus documentation (opens new window).

      For earlier versions of Prometheus, Kuma provides the kuma-prometheus-sd tool, which runs alongside your Prometheus instance. This tool fetches a list of current data plane proxies from the Kuma control plane and saves the list in Prometheus-compatible format to a file on disk. Prometheus watches for changes to the file and updates its scraping configuration accordingly.

        Check the Targets page in the Prometheus dashboard. You should see a list of data plane proxies from your mesh. For example:

        A screenshot of Targets page on Prometheus UI

        # Secure data plane proxy metrics

        Kuma lets you expose proxy metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the mesh for this feature to work, which is the default deployment model when kumactl install metrics is used on Kubernetes.

          # Expose metrics from applications

          In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies.

            To consume paths protected by mTLS, you need Traffic Permission that lets Prometheus consume applications.

            # Grafana Dashboards

            Kuma ships with four default dashboards that are available to import from the Grafana Labs repository (opens new window).

            # Kuma Dataplane

            This dashboards lets you investigate the status of a single dataplane in the mesh.

            Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard

            # Kuma Mesh

            This dashboard lets you investigate the aggregated statistics of a single mesh.

            Kuma Mesh dashboard

            # Kuma Service to Service

            This dashboard lets you investigate aggregated statistics from dataplanes of given source service to dataplanes of given destination service.

            Kuma Service to Service dashboard Kuma Service to Service HTTP

            # Kuma CP

            This dashboard lets you investigate statistics of the control plane.

            Kuma CP dashboard Kuma CP dashboard Kuma CP dashboard

            # Kuma Service

            This dashboard lets you investigate aggregated statistics for each service.

            Kuma Service dashboard

            # Grafana Datasource

            The Grafana Datasource is a datasource specifically built to relate information from the control-plane with prometheus metrics.

            Current features include:

            To use the plugin you'll need to add the binary to your grafana instance by following the installation instructions (opens new window).

            To make things simpler the datasource is installed and configured when using kumactl install metrics.

            Last Updated: 11/22/2021, 4:36:48 PM