You are browsing documentation for a version of Kuma that is not the latest release.
Looking for even older versions? Learn more.
Kuma facilitates consistent traffic metrics across all dataplanes in your mesh.
A user can enable traffic metrics by editing a
Mesh resource and providing the desired
Mesh-wide configuration. If necessary, metrics configuration can be customized for each
Dataplane individually, e.g. to override the default metrics port that might be already in use on that particular machine.
Kuma provides full integration with
- if enabled, every dataplane will expose its metrics in
Kumawill make sure that
Prometheuscan automatically find every dataplane in the mesh
To collect metrics from Kuma, you need to first expose metrics from Dataplanes and then configure Prometheus to collect them.
Expose metrics from Dataplanes
Prometheus metrics from every dataplane in the mesh, configure a
Mesh resource as follows:
apiVersion: kuma.io/v1alpha1 kind: Mesh metadata: name: default spec: metrics: enabledBackend: prometheus-1 backends: - name: prometheus-1 type: prometheus
which is a convenient shortcut for
apiVersion: kuma.io/v1alpha1 kind: Mesh metadata: name: default spec: metrics: enabledBackend: prometheus-1 backends: - name: prometheus-1 type: prometheus conf: skipMTLS: false port: 5670 path: /metrics tags: # tags that can be referred in Traffic Permission when metrics are secured by mTLS kuma.io/service: dataplane-metrics
Both snippets from above instruct
Kuma to configure every dataplane in the mesh
default to expose an HTTP endpoint with
Prometheus metrics on port
5670 and URI path
Override Prometheus settings per Dataplane
Mesh-wide defaults for a particular
prometheus.metrics.kuma.io/port- to override
Mesh-wide default port
prometheus.metrics.kuma.io/path- to override
Mesh-wide default path
apiVersion: apps/v1 kind: Deployment metadata: namespace: kuma-example name: kuma-tcp-echo spec: ... template: metadata: ... annotations: prometheus.metrics.kuma.io/port: "1234" # override Mesh-wide default port prometheus.metrics.kuma.io/path: "/non-standard-path" # override Mesh-wide default path spec: containers: ...
As a result, dataplane for this particular
Pod will expose an HTTP endpoint with
Prometheus metrics on port
1234 and URI path
Although dataplane metrics are now exposed,
Prometheus doesn’t know anything about it just yet.
Prometheus to automatically discover dataplanes,
Kuma provides a tool -
kuma-prometheus-sd is meant to run alongside
It knows location of
Kuma Control Plane is and can fetch an up-to-date list of dataplanes from it.
It then transforms that information into a format that
Prometheus can understand, and saves it into a file on disk.
Prometheus watches for changes to that file and updates its scraping configuration accordingly.
kumactl install metrics | kubectl apply -f - to deploy configured Prometheus with Grafana.
If you’ve got Prometheus deployment already, you can use Prometheus federation to bring Kuma metrics to your main Prometheus cluster.
Now, if you check
Targets page on
Prometheus UI, you should see a list of dataplanes from your mesh, e.g.
Secure Dataplane metrics
Kuma lets you expose Dataplane metrics in a secure way by leveraging mTLS. Prometheus needs to be a part of the Mesh for this feature to work, which is the default deployment model when
kumactl install metrics is used on Kubernetes.
Make sure that mTLS is enabled in the Mesh.
apiVersion: kuma.io/v1alpha1 kind: Mesh metadata: name: default spec: mtls: enabledBackend: ca-1 backends: - name: ca-1 type: builtin metrics: enabledBackend: prometheus-1 backends: - name: prometheus-1 type: prometheus conf: port: 5670 path: /metrics skipMTLS: false tags: # tags that can be referred in Traffic Permission kuma.io/service: dataplane-metrics
Allow the traffic from Grafana to Prometheus Server and from Prometheus Server to Dataplane metrics and for other Prometheus components:
apiVersion: kuma.io/v1alpha1 kind: TrafficPermission mesh: default metadata: name: metrics-permissions spec: sources: - match: kuma.io/service: prometheus-server_kuma-metrics_svc_80 destinations: - match: kuma.io/service: dataplane-metrics - match: kuma.io/service: "prometheus-alertmanager_kuma-metrics_svc_80" - match: kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_80" - match: kuma.io/service: "prometheus-kube-state-metrics_kuma-metrics_svc_81" - match: kuma.io/service: "prometheus-pushgateway_kuma-metrics_svc_9091" --- apiVersion: kuma.io/v1alpha1 kind: TrafficPermission mesh: default metadata: name: grafana-to-prometheus spec: sources: - match: kuma.io/service: "grafana_kuma-metrics_svc_80" destinations: - match: kuma.io/service: "prometheus-server_kuma-metrics_svc_80"
Expose metrics from applications
In addition to exposing metrics from Dataplane, you may want to expose metrics from application next to Kuma DP.
prometheus.io annotations either on
apiVersion: apps/v1 kind: Deployment metadata: namespace: kuma-example name: kuma-tcp-echo spec: ... template: metadata: ... annotations: prometheus.io/scrape: "true" prometheus.io/port: "1234" prometheus.io/path: "/non-standard-path" spec: containers: ...
Remember that in order to consume paths protected by mTLS, you need Traffic Permission that lets Prometheus consume applications.
Kuma ships with 4 default dashboards that are available to import from Grafana Labs repository.
This dashboards lets you investigate the status of a single dataplane in the mesh.
This dashboard lets you investigate the aggregated statistics of a single mesh.
Kuma Service to Service
This dashboard lets you investigate aggregated statistics from dataplanes of given source service to dataplanes of given destination service.
This dashboard lets you investigate statistics of the control plane.
This dashboard lets you investigate aggregated statistics for each service.