MeshMetric

Kuma facilitates consistent traffic metrics across all data plane proxies in your mesh.

You can define metrics configuration for a whole Mesh, and optionally tweak certain parts for individual data plane proxies. For example, you might need to override the default metrics port if it’s already in use on the specified machine.

Kuma provides full integration with Prometheus:

• Each proxy can expose its metrics in Prometheus format.
• Kuma exposes an API called the monitoring assignment service (MADS) which exposes proxies configured by MeshMetric.

Moreover, Kuma provides experimental integration with OpenTelemetry:

• Each proxy can publish its metrics to OpenTelemetry collector.

To collect metrics from Kuma, you need to expose metrics from proxies and applications.

TargetRef support matrix

To learn more about the information in this table, see the matching docs.

Configuration

There are three main sections of the configuration: sidecar, applications, backends. The first two define how to scrape parts of the mesh (sidecar and underlying applications), the third one defines what to do with the data (in case of Prometheus instructs to scrape specific address, in case of OpenTelemetry defines where to push data).

Sidecar

This part of the configuration applies to the data plane proxy scraping. In case you don’t want to retrieve all Envoy’s metrics, it’s possible to filter them.

Regex

You are able to specify regex which causes the metrics endpoint to return only matching metrics.

Unused metrics

By default, metrics that were not updated won’t be published. You can set the includeUnused flag that returns all metrics from Envoy.

Examples

Include unused metrics and filter them by regex

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: metrics-default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    sidecar:
      regex: http2_act.*
      includeUnused: true
    backends:
    - type: Prometheus
      prometheus:
        port: 5670
        path: "/metrics"

type: MeshMetric
mesh: default
name: metrics-default
spec:
  targetRef:
    kind: Mesh
  default:
    sidecar:
      regex: http2_act.*
      includeUnused: true
    backends:
    - type: Prometheus
      prometheus:
        port: 5670
        path: "/metrics"

Applications

In addition to exposing metrics from the data plane proxies, you might want to expose metrics from applications running next to the proxies. Kuma allows scraping Prometheus metrics from the applications endpoint running in the same Pod or VM. Later those metrics are aggregated and exposed at the same port/path as data plane proxy metrics. It is possible to configure it at the Mesh level, for all the applications in the Mesh, or just for specific applications.

Here are reasons where you’d want to use this feature:

• Application metrics are labelled with your mesh parameters (tags, mesh, data plane name…), this means that in mixed Universal and Kubernetes mode metrics are reported with the same types of labels.
• Both application and sidecar metrics are scraped at the same time. This makes sure they are coherent (with 2 different scrapers they can end up scraping at different intervals and make metrics harder to correlate).
• If you disable passthrough and your mesh uses mTLS and Prometheus is outside the mesh this is the only way to retrieve these metrics as the app is completely hidden behind the sidecar.

Example section of the configuration:

applications:
  - name: "backend" # application name used for logging and to scope OpenTelemetry metrics (optional)
    path: "/metrics/prometheus" # application metrics endpoint path
    address: # optional custom address if the underlying application listens on a different address than the Data Plane Proxy
    port: 8888 # port on which application is listening

Backends

Prometheus

backends:
  - type: Prometheus
    prometheus: 
      port: 5670
      path: /metrics

This tells Kuma to expose an HTTP endpoint with Prometheus metrics on port 5670 and uri path /metrics.

The metrics endpoint is forwarded to the standard Envoy Prometheus metrics endpoint and supports the same query parameters. You can pass the filter query parameter to limit the results to metrics whose names match a given regular expression. By default, all available metrics are returned.

Secure metrics with TLS

Kuma allows configuring metrics endpoint with TLS.

backends:
  - type: Prometheus
    prometheus: 
      port: 5670
      path: /metrics
      tls:
        mode: ProvidedTLS

In addition to the MeshMetric configuration, kuma-sidecar requires a provided certificate and key for its operation.

apiVersion: kuma.io/v1alpha1
kind: ContainerPatch
metadata:
  name: container-patch-1
  namespace: kuma-system
spec:
  sidecarPatch:
    - op: add
      path: /env/-
      value: '{
          "name": "KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH",
          "value": "/kuma/server.crt"
        }'
    - op: add
      path: /env/-
      value: '{
          "name": "KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH",
          "value": "/kuma/server.key"
        }'

* `KUMA_DATAPLANE_RUNTIME_METRICS_CERT_PATH`
* `KUMA_DATAPLANE_RUNTIME_METRICS_KEY_PATH`

activeMTLSBackend

We no longer support activeMTLSBackend, if you need to encrypt and authorize the metrics use Secure metrics with TLS with a combination of one of the authorization methods.

Running multiple Prometheus deployments

If you need to run multiple instances of Prometheus and want to target different set of Data Plane Proxies you can do this by using Client ID setting on both MeshMetric (clientId) and Prometheus configuration (client_id).

Example Prometheus configuration

Let’s assume we have two prometheus deployments main and secondary. We would like to use each of them to monitor different sets of data plane proxies, with different tags.

We can start with configuring each Prometheus deployments to use Kuma SD. Prometheus’s deployments will be differentiated by client_id parameter.

Main Prometheus config:

scrape_configs:
  - job_name: 'kuma-dataplanes'
    # ...
    kuma_sd_configs:
    - server: http://kuma-control-plane.kuma-system:5676
      refresh_interval: 60s # different from prometheus-secondary
      client_id: "prometheus-main" # Kuma will use this to pick proper data plane proxies

Secondary Prometheus config:

scrape_configs:
  - job_name: 'kuma-dataplanes'
    # ...
    kuma_sd_configs:
      - server: http://kuma-control-plane.kuma-system:5676
        refresh_interval: 20s # different from prometheus-main
        client_id: "prometheus-secondary"

Now we can configure first MeshMetric policy to pick data plane proxies with tag prometheus: main for main Prometheus discovery. clientId in policy should be the same as client_id in Prometheus configuration.

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: prometheus-one
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      prometheus: main
  default:
    backends:
    - type: Prometheus
      prometheus:
        clientId: prometheus-main
        port: 5670
        path: "/metrics"

type: MeshMetric
name: prometheus-one
mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      prometheus: main
  default:
    backends:
    - type: Prometheus
      prometheus:
        clientId: prometheus-main
        port: 5670
        path: "/metrics"

And policy for secondary Prometheus deployment that will pick data plane proxies with tag prometheus: secondary.

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: prometheus-two
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      prometheus: secondary
  default:
    backends:
    - type: Prometheus
      prometheus:
        clientId: prometheus-secondary
        port: 5670
        path: "/metrics"

type: MeshMetric
name: prometheus-two
mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      prometheus: secondary
  default:
    backends:
    - type: Prometheus
      prometheus:
        clientId: prometheus-secondary
        port: 5670
        path: "/metrics"

OpenTelemetry (experimental)

backends:
  - type: OpenTelemetry
    openTelemetry: 
      endpoint: otel-collector.observability.svc:4317

This configuration tells Kuma data plane proxy to push metrics to OpenTelemetry collector. Dataplane Proxy will scrape metrics from Envoy and other applications in a Pod/VM. and push them to configured OpenTelemetry collector.

When you configure application scraping make sure to specify application.name to utilize OpenTelemetry scoping

Limitations

• You cannot configure scraping interval for OpenTelemetry. By default, it will publish metrics to collector every 30 seconds. Ability to configure scraping interval in policy will be added in the future.

• Right now Kuma supports configuring only one OpenTelemetry backend.

• OpenTelemetry integration does not take sidecar configuration into account. This support will be added in the next release.

• Application must expose metrics in Prometheus format for this integration to work

Examples

With custom port, path, clientId, application aggregation and service override

The first policy defines a default MeshMetric policy for the default mesh. The second policy creates an override for workloads tagged with framework: example-web-framework. That web framework exposes metrics under /metrics/prometheus and port 8888.

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: metrics-default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    sidecar:
      includeUnused: false
    backends:
    - type: Prometheus
      prometheus:
        clientId: main-backend
        port: 5670
        path: "/metrics"
        tls:
          mode: ProvidedTLS

type: MeshMetric
mesh: default
name: metrics-default
spec:
  targetRef:
    kind: Mesh
  default:
    sidecar:
      includeUnused: false
    backends:
    - type: Prometheus
      prometheus:
        clientId: main-backend
        port: 5670
        path: "/metrics"
        tls:
          mode: ProvidedTLS

apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
  name: metrics-for-mesh-service
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      framework: example-web-framework
  default:
    applications:
    - path: "/metrics/prometheus"
      port: 8888

type: MeshMetric
mesh: default
name: metrics-for-mesh-service
spec:
  targetRef:
    kind: MeshSubset
    tags:
      framework: example-web-framework
  default:
    applications:
    - path: "/metrics/prometheus"
      port: 8888

All policy options