Careful!

You are browsing documentation for a version of Kuma that is not the latest release.

Looking for even older versions? Learn more.

Traffic Metrics

Kuma facilitates consistent traffic metrics across all dataplanes in your mesh.

A user can enable traffic metrics by editing a Mesh resource and providing the desired Mesh-wide configuration. If necessary, metrics configuration can be customized for each Dataplane individually, e.g. to override the default metrics port that might be already in use on that particular machine.

Out-of-the-box, Kuma provides full integration with Prometheus:

  • if enabled, every dataplane will expose its metrics in Prometheus format
  • furthemore, Kuma will make sure that Prometheus can automatically find every dataplane in the mesh

To collect metrics from Kuma, you need to first expose metrics from Dataplanes and then configure Prometheus to collect them.

Expose metrics from Dataplanes

To expose Prometheus metrics from every dataplane in the mesh, configure a Mesh resource as follows:

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus

which is a convenient shortcut for

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        skipMTLS: false
        port: 5670
        path: /metrics
        tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS  
          kuma.io/service: dataplane-metrics

Both snippets from above instruct Kuma to configure every dataplane in the mesh default to expose an HTTP endpoint with Prometheus metrics on port 5670 and URI path /metrics.

Override Prometheus settings per Dataplane

To override Mesh-wide defaults for a particular Pod, use Kuma-specific annotations:

  • prometheus.metrics.kuma.io/port - to override Mesh-wide default port
  • prometheus.metrics.kuma.io/path - to override Mesh-wide default path

E.g.,

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kuma-example
  name: kuma-tcp-echo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        prometheus.metrics.kuma.io/port: "1234"               # override Mesh-wide default port
        prometheus.metrics.kuma.io/path: "/non-standard-path" # override Mesh-wide default path
    spec:
      containers:
      ...

As a result, dataplane for this particular Pod will expose an HTTP endpoint with Prometheus metrics on port 1234 and URI path /non-standard-path.

Configure Prometheus

Although dataplane metrics are now exposed, Prometheus doesn’t know anything about it just yet.

To help Prometheus to automatically discover dataplanes, Kuma provides a tool - kuma-prometheus-sd. kuma-prometheus-sd is meant to run alongside Prometheus instance. It knows location of Kuma Control Plane is and can fetch an up-to-date list of dataplanes from it. It then transforms that information into a format that Prometheus can understand, and saves it into a file on disk. Prometheus watches for changes to that file and updates its scraping configuration accordingly.

Use kumactl install metrics | kubectl apply -f - to deploy configured Prometheus with Grafana.

If you’ve got Prometheus deployment already, you can use Prometheus federation to bring Kuma metrics to your main Prometheus cluster.

Now, if you check Targets page on Prometheus UI, you should see a list of dataplanes from your mesh, e.g.

A screenshot of Targets page on Prometheus UI

Secure Dataplane metrics

Kuma lets you expose Dataplane metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the Mesh for this feature to work, which is the default deployment model when kumactl install metrics is used on Kubernetes.

Make sure that mTLS is enabled in the Mesh.

apiVersion: kuma.io/v1alpha1
kind: Mesh
metadata:
  name: default
spec:
  mtls:
    enabledBackend: ca-1
    backends:
    - name: ca-1
      type: builtin
  metrics:
    enabledBackend: prometheus-1
    backends:
    - name: prometheus-1
      type: prometheus
      conf:
        port: 5670
        path: /metrics
        skipMTLS: false
        tags: # tags that can be referred in Traffic Permission  
          kuma.io/service: dataplane-metrics

Allow the traffic from Grafana to Prometheus Server and from Prometheus Server to Dataplane metrics and for other Prometheus components:

apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  name: metrics-permissions
spec:
  sources:
    - match:
       kuma.io/service: prometheus-server_kuma-metrics_svc_80
  destinations:
    - match:
       kuma.io/service: dataplane-metrics
    - match:
       kuma.io/service: "prometheus-alertmanager_kuma-metrics_svc_80"
    - match:
       kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_80"
    - match:
       kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_81"
    - match:
       kuma.io/service: "prometheus-pushgateway_kuma-metrics_svc_9091"
---
apiVersion: kuma.io/v1alpha1
kind: TrafficPermission
mesh: default
metadata:
  name: grafana-to-prometheus
spec:
   sources:
   - match:
      kuma.io/service: "grafana_kuma-metrics_svc_80"
   destinations:
   - match:
      kuma.io/service: "prometheus-server_kuma-metrics_svc_80"

Expose metrics from applications

In addition to exposing metrics from Dataplane, you may want to expose metrics from application next to Kuma DP.

Use standard prometheus.io annotations either on Pod or Service

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kuma-example
  name: kuma-tcp-echo
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "1234"
        prometheus.io/path: "/non-standard-path"
    spec:
      containers:
      ...

Remember that in order to consume paths protected by mTLS, you need Traffic Permission that lets Prometheus consume applications.

Grafana Dashboards

Kuma ships with 4 default dashboards that are available to import from Grafana Labs repository.

Kuma Dataplane

This dashboards lets you investigate the status of a single dataplane in the mesh.

Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard Kuma Dataplane dashboard

Kuma Mesh

This dashboard lets you investigate the aggregated statistics of a single mesh.

Kuma Mesh dashboard

Kuma Service to Service

This dashboard lets you investigate aggregated statistics from dataplanes of given source service to dataplanes of given destination service.

Kuma Service to Service dashboard Kuma Service to Service HTTP

Kuma CP

This dashboard lets you investigate statistics of the control plane.

Kuma CP dashboard Kuma CP dashboard Kuma CP dashboard

Kuma Service

This dashboard lets you investigate aggregated statistics for each service.

Kuma Service dashboard
Last Updated: 1/16/2023, 13:14:43 PM