Kuma Star

MeshTrace

This policy uses new policy matching algorithm. Do not combine with TrafficTrace.

This policy enables publishing traces to a third party tracing solution.

Tracing is supported over HTTP, HTTP2, and gRPC protocols. You must explicitly specify the protocol for each service and data plane proxy you want to enable tracing for.

Kuma currently supports the following trace exposition formats:

Services still need to be instrumented to preserve the trace chain across requests made across different services.

You can instrument with a language library of your choice (for Zipkin and for Datadog). For HTTP you can also manually forward the following headers:

  • x-request-id
  • x-b3-traceid
  • x-b3-parentspanid
  • x-b3-spanid
  • x-b3-sampled
  • x-b3-flags

TargetRef support matrix

targetRef Allowed kinds
targetRef.kind Mesh, Dataplane, MeshSubset(deprecated)

To learn more about the information in this table, see the matching docs.

Configuration

Sampling

Most of the time setting only overall is sufficient. random and client are for advanced use cases.

You can configure sampling settings equivalent to Envoy’s:

The value is always a percentage and is between 0 and 100.

Example:

sampling:
  overall: 80
  random: 60
  client: 40

Tags

You can add tags to trace metadata by directly supplying the value (literal) or by taking it from a header (header).

Example:

tags:
  - name: team
    literal: core
  - name: env
    header:
      name: x-env
      default: prod
  - name: version
    header:
      name: x-version

If a value is missing for header, default is used. If default isn’t provided, then the tag won’t be added.

Delegated Gateway

When configuring MeshTrace with a delegated gateway you have to add these tags to the policy to properly aggregate traces:

tags:
  - name: b3-traceid
    header:
      name: x-b3-traceid
  - name: b3-parent-span-id
    header:
      name: x-b3-parent-span-id

Backends

Datadog

You can configure a Datadog backend with a url and splitService.

Example:

datadog:
  url: http://my-agent:8080 # Required. The url to reach a running datadog agent
  splitService: true # Default to false. If true, it will split inbound and outbound requests in different services in Datadog

The splitService property determines if Datadog service names should be split based on traffic direction and destination. For example, with splitService: true and a backend service that communicates with a couple of databases, you would get service names like backend_INBOUND, backend_OUTBOUND_db1, and backend_OUTBOUND_db2 in Datadog.

Zipkin

In most cases the only field you’ll want to set is url.

Example:

zipkin:
  url: http://jaeger-collector:9411/api/v2/spans # Required. The url to a zipkin collector to send traces to 
  traceId128bit: false # Default to false which will expose a 64bits traceId. If true, the id of the trace is 128bits
  apiVersion: httpJson # Default to httpJson. It can be httpJson, httpProto and is the version of the zipkin API
  sharedSpanContext: false # Default to true. If true, the inbound and outbound traffic will share the same span.

OpenTelemetry

The only field you can set is endpoint.

Example:

openTelemetry:
  endpoint: otel-collector:4317 # Required. Address of OpenTelemetry collector

Examples

Zipkin

Simple example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

Full example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    - name: b3-traceid
      header:
        name: x-b3-traceid
    - name: b3-parent-span-id
      header:
        name: x-b3-parent-span-id
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Zipkin
      zipkin:
        url: http://jaeger-collector:9411/api/v2/spans
        apiVersion: httpJson

Datadog

This assumes a Datadog agent is configured and running. If you haven’t already check the Datadog observability page.

Simple example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126

Full example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    - name: b3-traceid
      header:
        name: x-b3-traceid
    - name: b3-parent-span-id
      header:
        name: x-b3-parent-span-id
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: Datadog
      datadog:
        url: http://127.0.0.1:8126
        splitService: true

OpenTelemetry

This assumes a OpenTelemetry collector is configured and running. If you haven’t already check the OpenTelementry operator.

Simple example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

Full example:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  default:
    tags:
    - name: team
      literal: core
    - name: env
      header:
        name: x-env
        default: prod
    - name: version
      header:
        name: x-version
    - name: b3-traceid
      header:
        name: x-b3-traceid
    - name: b3-parent-span-id
      header:
        name: x-b3-parent-span-id
    sampling:
      overall: 80
      random: 60
      client: 40
    backends:
    - type: OpenTelemetry
      openTelemetry:
        endpoint: otel-collector.com:4317

Targeting parts of the infrastructure

While usually you want all the traces to be sent to the same tracing backend, you can target parts of a Mesh by using a finer-grained targetRef and a designated backend to trace different paths of our service traffic. This is especially useful when you want traces to never leave a world region, or a cloud, for example.

In this example, we have two zones east and west, each of these with their own Zipkin collector: east.zipkincollector:9411/api/v2/spans and west.zipkincollector:9411/api/v2/spans. We want data plane proxies in each zone to only send traces to their local collector.

To do this, we use a TargetRef kind value of MeshSubset to filter which data plane proxy a policy applies to.

West only policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: trace-west
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: west
  default:
    backends:
    - type: Zipkin
      zipkin:
        url: http://west.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

East only policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTrace
metadata:
  name: trace-east
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: east
  default:
    backends:
    - zipkin:
        url: http://east.zipkincollector:9411/api/v2/spans
        apiVersion: httpJson

See also

All policy options

targetRef object
TargetRef is a reference to the resource the policy takes an effect on. The resource could be either...
kind enum required
Kind of the referenced resource
Values: Mesh | MeshSubset | MeshGateway | MeshService | MeshExternalService | MeshMultiZoneService | MeshServiceSubset | MeshHTTPRoute | Dataplane
name string
Name of the referenced resource. Can only be used with kinds: `MeshService`, `MeshServiceSubset` and...
namespace string
Namespace specifies the namespace of target resource. If empty only resources in policy namespace wi...
labels object
Labels are used to select group of MeshServices that match labels. Either Labels or Name and Namespa...
sectionName string
SectionName is used to target specific section of resource. For example, you can target port from Me...
mesh string
Mesh is reserved for future use to identify cross mesh resources.
proxyTypes array
ProxyTypes specifies the data plane types that are subject to the policy. When not specified, all da...
tags object
Tags used to select a subset of proxies by tags. Can only be used with kinds `MeshSubset` and `MeshS...
default object
MeshTrace configuration.
A one element array of backend definition. Envoy allows configuring only 1 backend, so the natural w...
Datadog backend configuration.
splitService boolean
Determines if datadog service name should be split based on traffic direction and destination. For e...
url string required
Address of Datadog collector, only host and port are allowed (no paths, fragments etc.)
OpenTelemetry backend configuration.
endpoint string required
Address of OpenTelemetry collector.
type enum required
Values: Zipkin | Datadog | OpenTelemetry
Zipkin backend configuration.
apiVersion enum
Version of the API. https://github.com/envoyproxy/envoy/blob/v1.22.0/api/envoy/config/trace/v3/zipki...
Values: httpJson | httpProto
Default: "httpJson"
sharedSpanContext boolean
Determines whether client and server spans will share the same span context. https://github.com/envo...
Default: true
traceId128bit boolean
Generate 128bit traces.
url string required
Address of Zipkin collector.
Sampling configuration. Sampling is the process by which a decision is made on whether to process/ex...
client any
Target percentage of requests that will be force traced if the 'x-client-trace-id' header is set. Mi...
overall any
Target percentage of requests will be traced after all other sampling checks have been applied (clie...
random any
Target percentage of requests that will be randomly selected for trace generation, if not requested ...
Custom tags configuration. You can add custom tags to traces based on headers or literal values.
Tag taken from a header.
default string
Default value to use if header is missing. If the default is missing and there is no value the tag w...
name string required
Name of the header.
literal string
Tag taken from literal value.
name string required
Name of the tag.