Kuma Star

Careful!

You are browsing documentation for the next version of Kuma. Use this version at your own risk.

Policies

What is a policy?

A policy is a set of configuration that will be used to generate the data plane proxy configuration. Kuma combines policies with the Dataplane resource to generate the Envoy configuration of a data plane proxy.

What do policies look like?

Like all resources in Kuma, there are two parts to a policy: the metadata and the spec.

Metadata

Metadata identifies the policies by its name, type and what mesh it’s part of:

In Kubernetes all our policies are implemented as custom resource definitions (CRD) in the group kuma.io/v1alpha1.

apiVersion: kuma.io/v1alpha1
kind: ExamplePolicy
metadata:
  name: my-policy-name
  namespace: kuma-system
spec: ... # spec data specific to the policy kind

By default the policy is created in the default mesh. You can specify the mesh by using the kuma.io/mesh label.

For example:

apiVersion: kuma.io/v1alpha1
kind: ExamplePolicy
metadata:
  name: my-policy-name
  namespace: kuma-system
  labels:
    kuma.io/mesh: "my-mesh"
spec: ... # spec data specific to the policy kind

Spec

The spec field contains the actual configuration of the policy.

Some policies apply to only a subset of the configuration of the proxy.

  • Inbound policies apply only to incoming traffic. Most inbound policies now use spec.rules[] to define their configuration. However, MeshTrafficPermission and MeshFaultInjection still use the spec.from[].targetRef field, which defines the subset of clients that are going to be impacted by this policy.
  • Outbound policies apply only to outgoing traffic. The spec.to[].targetRef field defines the outbounds that are going to be impacted by this policy

The actual configuration is defined under the default field.

For example:

apiVersion: kuma.io/v1alpha1
kind: ExampleOutboundPolicy
metadata:
  name: my-example
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: MeshService
      name: my-service_svc
    default:
      key: value
---
apiVersion: kuma.io/v1alpha1
kind: ExampleInboundPolicy
metadata:
  name: my-example
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: my-app
    sectionName: httpport
  rules:
  - default:
      key: value

Some policies are not directional and will not have to and rules. Some examples of such policies are MeshTrace or MeshProxyPatch. For example

apiVersion: kuma.io/v1alpha1
kind: NonDirectionalPolicy
metadata:
  name: my-example
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  default:
    key: value

All specs have a top level targetRef which identifies which proxies this policy applies to. In particular, it defines which proxies have their Envoy configuration modified.

One of the benefits of targetRef policies is that the spec is always the same between Kubernetes and Universal.

This means that converting policies between Universal and Kubernetes only means rewriting the metadata.

Writing a targetRef

targetRef is a concept borrowed from Kubernetes Gateway API. Its goal is to reference resources in a cluster.

It looks like:

targetRef:
  kind: Mesh | Dataplane | MeshService | MeshExternalService | MeshMultiZoneService | MeshGateway
  name: my-name # On Kubernetes resources can be selected by name/namespace
  namespace: ns
  labels: # Alternative to name/namespace, labels can be used to select a group of resources
    key: value
  sectionName: ASection # This is used when trying to attach to a specific part of a resource (for example an inbound port of a `Dataplane`)
  tags: # Only for kind MeshGateway to select a set of listeners
    key: value
  proxyTypes: [ Sidecar, Gateway ] # Only for kind Mesh to apply to all proxies of a specific type

Consider the two example policies below:

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: example-outbound
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web-frontend
  to:
  - targetRef:
      kind: MeshService
      name: web-backend_kuma-demo_svc_8080
    default:
      backends:
      - file:
          format:
            plain: '{"start_time": "%START_TIME%"}'
          path: "/tmp/logs.txt"

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: example-outbound
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web-frontend
  to:
  - targetRef:
      kind: MeshService
      name: web-backend
      namespace: kuma-demo
      sectionName: httpport
    default:
      backends:
      - file:
          format:
            plain: '{"start_time": "%START_TIME%"}'
          path: "/tmp/logs.txt"

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: example-inbound
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web-frontend
  rules:
  - default:
      backends:
      - file:
          format:
            plain: '{"start_time": "%START_TIME%"}'
          path: "/tmp/logs.txt"

Using spec.targetRef, this policy targets all proxies that have a label app:web-frontend. It defines the scope of this policy as applying to traffic either from or to data plane proxies with the tag app:web-frontend.

The spec.to[].targetRef section enables logging for any traffic going to web-backend. The spec.rules[] section enables logging for any traffic coming on inbound listeners of the web-frontend proxies.

Using a sectionName

The targetRef.sectionName field helps select specific sections within certain resource kinds:

  • Dataplane – selects an inbound port
  • MeshService – selects a port of the matching services
  • MeshMultiZoneService – selects a port

To resolve sectionName, the following steps are applied:

  1. Look for a section where the name matches sectionName.
  2. If no match is found, try interpreting sectionName as a number and find a port with the same value—only if the port’s name is unset.

Examples

Given a Dataplane resource with several inbound listeners:

type: Dataplane
mesh: default
name: backend
labels:
  app: backend
networking:
  address: 192.168.0.2
  inbound:
    - name: backend-api
      port: 8080
      tags:
        kuma.io/service: backend
    - name: admin-api
      port: 5000
      tags:
        kuma.io/service: backend-admin

Inbound policies can attach to all inbound listeners:

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: all-inbounds
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: backend
  rules:
  - default:
      backends:
      - file:
          format:
            plain: '{"start_time": "%START_TIME%"}'
          path: "/tmp/logs.txt"

or just some inbound listeners:

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: only-backend-api-inbound
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: backend
    sectionName: backend-api
  rules:
  - default:
      backends:
      - file:
          format:
            plain: '{"start_time": "%START_TIME%"}'
          path: "/tmp/logs.txt"

Omitting targetRef

When a spec.targetRef is not present, it is semantically equivalent to spec.targetRef.kind: Mesh and refers to everything inside the Mesh.

Applying to specific proxy types

The top level targetRef field can select a specific subset of data plane proxies. The field named proxyTypes can restrict policies to specific types of data plane proxies:

  • Sidecar: Targets data plane proxies acting as sidecars to applications (including delegated gateways).
  • Gateway: Applies to data plane proxies operating in built-in Gateway mode.
  • Empty list: Defaults to targeting all data plane proxies.

Example

The following policy will only apply to gateway data-planes:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: gateway-only-timeout
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
    proxyTypes:
    - Gateway
  to:
  - targetRef:
      kind: Mesh
    default:
      idleTimeout: 10s

Targeting gateways

Given a MeshGateway:

apiVersion: kuma.io/v1alpha1
kind: MeshGateway
mesh: default
metadata:
  name: edge
  namespace: kuma-system
conf:
  listeners:
    - port: 80
      protocol: HTTP
      tags:
        port: http-80
    - port: 443
      protocol: HTTPS
      tags:
        port: https-443

Policies can attach to all listeners:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout-all
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshGateway
    name: edge
  to:
  - targetRef:
      kind: Mesh
    default:
      idleTimeout: 10s

so that requests to either port 80 or 443 will have an idle timeout of 10 seconds, or just some listeners:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout-8080
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshGateway
    name: edge
    tags:
      port: http-80
  to:
  - targetRef:
      kind: Mesh
    default:
      idleTimeout: 10s

So that only requests to port 80 will have the idle timeout.

Note that depending on the policy, there may be restrictions on whether or not specific listeners can be selected.

Routes

Read the MeshHTTPRoute docs and MeshTCPRoute docs for more on how to target gateways for routing traffic.

Target kind support for different policies

Not every policy supports to and rules levels. Additionally, not every resource can appear at every supported level. The specified top level resource can also affect which resources can appear in to or rules.

To help users, each policy documentation includes tables indicating which targetRef kinds is supported at each level. For each type of proxy, sidecar or builtin gateway, the table indicates for each targetRef level, which kinds are supported.

Example tables

These are just examples, remember to check the docs specific to your policy.

targetRef Allowed kinds
targetRef.kind Mesh, Dataplane, MeshGateway
to[].targetRef.kind Mesh, MeshService, MeshExternalService, MeshMultiZoneService

The table above show that we can select sidecar proxies via Mesh, Dataplane, MeshGateway

We can use the policy as an outbound policy with:

  • to[].targetRef.kind: Mesh which will apply to all traffic originating at the sidecar to anywhere
  • to[].tagerRef.kind: MeshService which will apply to all traffic to specific services
  • to[].tagerRef.kind: MeshExternalService which will apply to all traffic to specific external services
  • to[].tagerRef.kind: MeshMultiZoneService which will apply to all traffic to specific multi-zone services

Merging configuration

A proxy can be targeted by multiple targetRef’s, to define how policies are merged together the following strategy is used:

We define a total order of policy priority. The table below defines the sorting order for resources in the cluster. Sorting is applied sequentially by attribute, with ties broken using the next attribute in the list.

  Attribute Order
1 spec.targetRef * Mesh (less priority)
* MeshGateway
* Dataplane
* Dataplane with labels
* Dataplane with labels/sectionName
* Dataplane with name/namespace
* Dataplane with name/namespace/sectionName
2 Origin
Label kuma.io/origin		 * global (less priority)
* zone
3 Policy Role
Label kuma.io/policy-role		 * system (less priority)
* producer
* consumer
* workload-owner
4 Display Name
Label kuma.io/display-name		 Inverted lexicographical order, i.e;
* zzzzz (less priority)
* aaaaa1
* aaaaa
* aaa

For to and rules policies we concatenate the array for each matching policies. For to policies we sort concatenated arrays again based on the spec.to[].targetRef field:

  Attribute Order
1 spec.to[].targetRef * Mesh (less priority)
* MeshService
* MeshService with sectionName
* MeshExternalService
* MeshMultiZoneService

We then build configuration by merging each level using JSON patch merge.

For example if I have 2 default ordered this way:

default:
  conf: 1
  sub:
    array: [ 1, 2, 3 ]
    other: 50
    other-array: [ 3, 4, 5 ]
---
default:
  sub:
    array: [ ]
    other-array: [ 5, 6 ]
    extra: 2

The merge result is:

default:
  conf: 1
  sub:
    array: [ ]
    other: 50
    other-array: [ 5, 6 ]
    extra: 2

Using policies with MeshService, MeshMultizoneService and MeshExternalService.

MeshService is a feature to define services explicitly in Kuma. It can be selectively enabled and disable depending on the value of meshServices.mode on your Mesh object.

When using explicit services, MeshServiceSubset is no longer a valid kind and MeshService can only be used to select an actual MeshService resource (it can no longer select a kuma.io/service).

In the following example we’ll assume we have a MeshService:

apiVersion: kuma.io/v1alpha1
kind: MeshService
metadata:
  name: my-service
  namespace: kuma-demo
  labels:
    k8s.kuma.io/namespace: kuma-demo
    kuma.io/zone: my-zone
    app: redis
spec:
  selector:
    dataplaneTags:
      app: redis
      k8s.kuma.io/namespace: kuma-demo
  ports:
  - port: 6739
    targetPort: 6739
    appProtocol: tcp

There are 2 ways to select a MeshService:

If you are in the same namespace (or same zone in Universal) you can select one specific service by using its explicit name:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout-to-redis
  namespace: kuma-demo
spec:
  to:
  - targetRef:
      kind: MeshService
      name: redis
    default:
      connectionTimeout: 10s

Selecting all matching MeshServices by labels:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: all-in-my-namespace 
  namespace: kuma-demo
spec:
  to:
  - targetRef:
      kind: MeshService
      labels:
        k8s.kuma.io/namespace: kuma-demo
    default:
      connectionTimeout: 10s

In this case this is equivalent to writing a specific policy for each service that matches this label (in our example for each service in this namespace in each zones).

When MeshService have multiple ports, you can use sectionName to restrict policy to a single port.

Global, zonal, producer and consumer policies

Policies can be applied to a zone or to a namespace when using Kubernetes. Policies will always impact at most the scope at which they are defined. In other words:

  1. a policy applied to the global control plane will apply to all proxies in all zones.
  2. a policy applied to a zone will only apply to proxies inside this zone. It is equivalent to having: 
    spec:
  targetRef: 
    kind: Dataplane
    labels:
      kuma.io/zone: "my-zone"
  3. a policy applied to a namespace will only apply to proxies inside this namespace. It is equivalent to having: 
    spec:
  targetRef: 
    kind: Dataplane
    labels:
      kuma.io/zone: "my-zone"
      kuma.io/namespace: "my-ns"

There is however, one exception to this when using MeshService with outbound policies (policies with spec.to[].targetRef). In this case, if you define a policy in the same namespace as the MeshService it is defined in, that policy will be considered a producer policy. This means that all clients of this service (even in different zones) will be impacted by this policy.

An example of a producer policy is:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout-to-redis
  namespace: kuma-demo
spec:
  to:
    - targetRef:
        kind: MeshService
        name: redis
      default:
        connectionTimeout: 10s

The other type of policy is a consumer policy which most commonly use labels to match a service.

An example of a consumer policy which would override the previous producer policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: timeout-to-redis-consumer
  namespace: kuma-demo
spec:
  to:
    - targetRef:
        kind: MeshService
        labels:
          k8s.kuma.io/service-name: redis
      default:
        connectionTimeout: 10s

Remember that labels on a MeshService applies to each matching MeshService. To communicate to services named the same way in different namespaces or zones with different configuration use a more specific set of labels.

Kuma adds a label kuma.io/policy-role to identify the type of the policy. The values of the label are:

  • system: Policies defined on global or in the zone’s system namespace
  • workload-owner: Policies defined in a non system namespaces that do not have spec.to entries, or have only spec.rules
  • consumer: Policies defined in a non system namespace that have spec.to which either do not use name or have a different namespace
  • producer: Policies defined in the same namespace as the services identified in the spec.to[].targetRef

Example

We have 2 clients client1 and client2 they run in different namespaces respectively ns1 and ns2.

 
flowchart LR
subgraph ns1
    client1(client)
end
subgraph ns2
  client2(client)
  server(MeshService: server)
end
client1 --> server
client2 --> server

We’re going to define a producer policy first:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
    name: producer-policy
    namespace: ns2
spec:
  to:
    - targetRef:
        kind: MeshService
        name: server
      default:
        idleTimeout: 20s

We know it’s a producer policy because it is defined in the same namespace as the MeshService: server and names this server in its spec.to[].targetRef. So both client1 and client2 will receive the timeout of 20 seconds.

We now create a consumer policy:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: consumer-policy
  namespace: ns1
spec:
  to:
    - targetRef:
        kind: MeshService
        labels:
          k8s.kuma.io/service-name: server
      default:
        idleTimeout: 30s

Here the policy only impacts client1 as client2 doesn’t run in ns1. As consumer policies have a higher priority over producer policies, client1 will have a idleTimeout: 30s.

We can define another policy to impact client2:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: consumer-policy
  namespace: ns2
spec:
  to:
    - targetRef:
        kind: MeshService
        labels:
          k8s.kuma.io/service-name: server
      default:
        idleTimeout: 40s

Note that the only different here is the namespace, we now define a consumer policy inside ns2.

Use labels for consumer policies and name for producer policies. It will be easier to differentiate between producer and consumer policies.

Examples

Applying a global default

type: ExamplePolicy
name: example
mesh: default
spec:
  targetRef:
    kind: Mesh
  to:
    - targetRef:
        kind: Mesh
      default:
        key: value

All traffic from any proxy (top level targetRef) going to any proxy (to targetRef) will have this policy applied with value key=value.

Recommending to users

type: ExamplePolicy
name: example
mesh: default
spec:
  targetRef:
    kind: Mesh
  to:
    - targetRef:
        kind: MeshService
        name: my-service
      default:
        key: value

All traffic from any proxy (top level targetRef) going to the service “my-service” (to targetRef) will have this policy applied with value key=value.

This is useful when a service owner wants to suggest a set of configurations to its clients.

Configuring all proxies of a team

type: ExamplePolicy
name: example
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      team: "my-team"
  rules:
    - default:
        key: value

All traffic that’s going to any proxy with the tag team=my-team (top level targetRef) will have this policy applied with value key=value.

This is a useful way to define coarse-grained rules for example.

Configuring all proxies in a zone

type: ExamplePolicy
name: example
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      kuma.io/zone: "east"
  default:
    key: value

All proxies in zone east (top level targetRef) will have this policy configured with key=value.

This can be very useful when observability stores are different for each zone for example.

Configuring all gateways in a Mesh

type: ExamplePolicy
name: example
mesh: default
spec:
  targetRef:
    kind: Mesh
    proxyTypes: ["Gateway"]
  default:
    key: value

All gateway proxies in mesh default will have this policy configured with key=value.

This can be very useful when timeout configurations for gateways need to differ from those of other proxies.

Applying policies in shadow mode

Overview

Shadow mode allows users to mark policies with a specific label to simulate configuration changes without affecting the live environment. It enables the observation of potential impact on Envoy proxy configurations, with a risk-free method to test, validate, and fine-tune settings before actual deployment.

It’s not necessary but CLI tools like jq and jd can greatly improve working with Kuma resources.

How to use shadow mode

  1. Before applying the policy, add a kuma.io/effect: shadow label.

  2. Check the proxy config with shadow policies taken into account through the Kuma API. By using HTTP API: 
     curl http://localhost:5681/meshes/${mesh}/dataplane/${dataplane}/_config?shadow=true

    or by using kumactl:

     kumactl inspect dataplane ${name} --type=config --shadow
  3. Check the diff in JSONPatch format through the Kuma API. By using HTTP API: 
     curl http://localhost:5681/meshes/${mesh}/dataplane/${dataplane}/_config?shadow=true&include=diff

    or by using kumactl:

     kumactl inspect dataplane ${name} --type=config --shadow --include=diff

Limitations and Considerations

Currently, the Kuma API mentioned above works only on Zone CP. Attempts to use it on Global CP lead to 405 Method Not Allowed. This might change in the future.

Examples

Apply policy with kuma.io/effect: shadow label:

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: frontend-timeouts
  namespace: kuma-system
  labels:
    kuma.io/effect: shadow
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/service: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_3001
    default:
      idleTimeout: 23s

apiVersion: kuma.io/v1alpha1
kind: MeshTimeout
metadata:
  name: frontend-timeouts
  namespace: kuma-system
  labels:
    kuma.io/effect: shadow
    kuma.io/mesh: default
spec:
  targetRef:
    kind: MeshSubset
    tags:
      kuma.io/service: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: httpport
    default:
      idleTimeout: 23s

Check the diff using kumactl:

$ kumactl inspect dataplane frontend-dpp --type=config --include=diff --shadow | jq '.diff' | jd -t patch2jd
@ ["type.googleapis.com/envoy.config.cluster.v3.Cluster","backend_kuma-demo_svc_3001","typedExtensionProtocolOptions","envoy.extensions.upstreams.http.v3.HttpProtocolOptions","commonHttpProtocolOptions","idleTimeout"]
- "3600s"
@ ["type.googleapis.com/envoy.config.cluster.v3.Cluster","backend_kuma-demo_svc_3001","typedExtensionProtocolOptions","envoy.extensions.upstreams.http.v3.HttpProtocolOptions","commonHttpProtocolOptions","idleTimeout"]
+ "23s"

The output not only identifies the exact location in Envoy where the change will occur, but also shows the current timeout value that we’re planning to replace.