Collect metrics with OpenTelemetry
Kuma provides integration with OpenTelemetry. You can collect and push dataplane proxy and application metrics to OpenTelemetry collector. Which opens up lots of possibilities of processing and exporting metrics to multiple ecosystems like Datadog, Grafana cloud, Honeycomb and more.
Prerequisites
- Completed quickstart to set up a zone control plane with demo application
Install Kuma observability stack
To start we need to install Kuma observability stack which is build on top of Prometheus and Grafana.
kumactl install observability | kubectl apply -f-
We will use it to scrape metrics from OpenTelemetry collector and visualise them on Kuma dashboards.
Install OpenTelemetry collector
First we need an OpenTelemetry collector configuration. Save it by running:
echo "
mode: deployment
config:
exporters:
prometheus:
endpoint: \${env:MY_POD_IP}:8889
extensions:
health_check:
endpoint: \${env:MY_POD_IP}:13133
memory_ballast: {}
processors:
batch: {}
memory_limiter: null
receivers:
otlp:
protocols:
grpc:
endpoint: \${env:MY_POD_IP}:4317
service:
extensions:
- health_check
- memory_ballast
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
ports:
otlp:
enabled: true
containerPort: 4317
servicePort: 4317
hostPort: 4317
protocol: TCP
appProtocol: grpc
prometheus:
enabled: true
containerPort: 8889
servicePort: 8889
protocol: TCP
image:
repository: 'otel/opentelemetry-collector-contrib'
" > values-otel.yaml
This is the Helm chart configuration we will be using. This will configure OpenTelemetry collector to listen on grpc port 4317
for metrics
pushed by dataplane proxy, process and expose collected metrics in Prometheus format on port 8889
. In the next step we
will configure Prometheus to scrape these metrics. Our configuration relies on the contrib distribution of opentelemetry-collector so we set this in the values.
Most important in this configuration is pipelines
section:
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
In this basic guide we will focus only on collecting metrics, but this can be also easily configured to collect traces and logs.
We use otlp
receiver to accept metrics pushed from dataplane proxies.
Then we have basic recommended processors to limit memory usage and to process metrics in batch. You can filter, modify and do more with available processors.
Last part is exporters
section. You can export metrics to multiple destination like Prometheus, Datadog, Grafana Cloud and more. Full list of available exporters can be found here.
We will use Prometheus exporter for now.
With configuration in place we can install OpenTelemetry collector:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install --namespace mesh-observability opentelemetry-collector open-telemetry/opentelemetry-collector -f values-otel.yaml
Configure Prometheus to scrape metrics from OpenTelemetry collector
We need to update prometheus-server
ConfigMap and add scrape_configs
entry:
- job_name: "opentelemetry-collector"
scrape_interval: 15s
static_configs:
- targets: ["opentelemetry-collector.mesh-observability.svc:8889"]
Prometheus will automatically pick up this config and start scraping OpenTelemetry collector. To check if config was applied properly you can go to Prometheus GUI:
kubectl port-forward svc/prometheus-server -n mesh-observability 9090:80
Now go to http://localhost:9090/targets. You should see new target opentelemetry-collector
:
Enable OpenTelemetry metrics and check results
By now we have installed and configured all needed observability tools: OpenTelemetry collector, Prometheus and Grafana. We can now apply MeshMetric policy:
echo '
apiVersion: kuma.io/v1alpha1
kind: MeshMetric
metadata:
name: otel-metrics
namespace: kuma-system
labels:
kuma.io/mesh: default
spec:
targetRef:
kind: Mesh
default:
backends:
- type: OpenTelemetry
openTelemetry:
endpoint: opentelemetry-collector.mesh-observability.svc:4317
' | kubectl apply -f -
This policy will configure all dataplane proxies in default
Mesh to collect and push metrics to OpenTelemetry collector.
To check results we need to log into Grafana. First enable port forward to Grafana GUI:
kubectl port-forward svc/grafana -n mesh-observability 3000:80
Then navigate to browser http://localhost:3000
and check Dataplane
dashboard. You should see something similar to:
Next steps
- Further explore MeshMetric policy
- Explore MeshAccessLog and MeshTrace policies that work with OpenTelemetry
- Explore features of OpenTelemetry collector for metrics filtering/processing and exporting
- Checkout tutorials on how to push metrics to SaaS solutions Grafana cloud, Datadog, Honeycomb