Progressively rolling in strict mTLS

The MeshTLS policy allows you to gradually migrate services to mutual TLS without dropping a packet.

Prerequisites

• Completed quickstart to set up a zone control plane with demo application.
• jq - a command-line JSON processor

helm upgrade \
  --install \
  --create-namespace \
  --namespace kuma-system \
  kuma kuma/kuma
kubectl wait -n kuma-system --for=condition=ready pod --selector=app=kuma-control-plane --timeout=90s
kubectl apply -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/refs/heads/main/k8s/001-with-mtls.yaml

Basic setup

To make sure that traffic works in our examples let’s configure MeshTrafficPermission to allow all traffic:

echo "apiVersion: kuma.io/v1alpha1
kind: MeshTrafficPermission
metadata:
  namespace: kuma-system
  name: mtp
spec:
  targetRef:
    kind: Mesh
  from:
    - targetRef:
        kind: Mesh
      default:
        action: Allow" | kubectl apply -f -

Gradually bring another service into the mesh

Start a second demo app

First we start a new demo setup in another namespace called kuma-demo-migration. The below command installs the demo one more time in another namespace:

kubectl apply -f https://raw.githubusercontent.com/kumahq/kuma-counter-demo/refs/heads/main/k8s/003-migration-demo.yaml

Below diagram shows which applications are inside the mesh and which are not. Purple links indicate that the communication is encrypted, gray ones are plaintext.

 
---
title: service graph of the second demo app
---
flowchart LR
    subgraph meshed
        subgraph kuma-demo
            direction LR
            demo-app(demo-app :5000)
            kv(kv :6379)
            demo-app --> kv
        end
    end
    subgraph non-meshed
        subgraph kuma-demo-migration
            direction LR
            kv2(kv :6379)
            demo-app2(demo-app :5000)
            demo-app2 --> kv2
        end
    end
    linkStyle 0 stroke:#d25585, stroke-width:2px;
    linkStyle 1 stroke:#555a5d, stroke-width:2px;
  

Enable port forwarding for both demo-apps

kubectl port-forward svc/demo-app -n kuma-demo 5051:5050

And in separate terminal window

kubectl port-forward svc/demo-app -n kuma-demo-migration 5052:5050

Open up both apps’ GUI and turn on auto incrementing.

Enable permissive mode on kv

We begin with preparing kv to start in permissive mode when deployed inside the mesh. To enable permissive mode we define this MeshTLS policy:

echo "apiVersion: kuma.io/v1alpha1
kind: MeshTLS
metadata:
  name: kv
  namespace: kuma-demo-migration
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: kv
  from:
  - targetRef:
      kind: Mesh
    default:
      mode: Permissive" | kubectl apply -f -

Migrate kv to mesh

We need to start by labeling kuma-demo-migration namespace with kuma.io/sidecar-injection=true label:

kubectl label namespace kuma-demo-migration kuma.io/sidecar-injection=enabled --overwrite

Then we need to restart the kv deployment

kubectl rollout restart deployment kv -n kuma-demo-migration

After this kv will be receiving plaintext traffic from non-meshed client. You can check the stats for kv data plane:

kubectl port-forward svc/kuma-control-plane -n kuma-system 5681:5681

export KV_DPP_NAME=$(curl -s http://localhost:5681/meshes/default/dataplanes/_overview\?name\=kv | jq -r '.items[0].name')
curl -s http://localhost:5681/meshes/default/dataplanes/$KV_DPP_NAME/stats | grep cluster.localhost_5050.upstream_cx_total

You should see metrics increment after running this curl command multiple times. Metrics will look like:

cluster.localhost_5050.upstream_cx_total: 9

The below diagram shows that the second kv was moved to be inside the mesh:

 
---
title: service graph when kv is inside the mesh
---
flowchart LR
    subgraph meshed
        subgraph kuma-demo
            direction LR
            demo-app(demo-app :5000)
            kv(kv :6379)
            demo-app --> kv
        end
        
        subgraph kuma-demo-migration 
            direction LR
            kv2(kv :6379)
        end
    end
    subgraph non-meshed
        subgraph kuma-demo-migration
            direction LR
            demo-app2(demo-app :5000)
            demo-app2 --> kv2
        end
    end
    linkStyle 0 stroke:#d25585, stroke-width:2px;
    linkStyle 1 stroke:#555a5d, stroke-width:2px;
  

Migrate client to mesh

Next we do the same to the client so the traffic is encrypted:

kubectl rollout restart deployment demo-app -n kuma-demo-migration

After this is done, you’ll have to re-enable the port-forward, and then you can go to the Kuma GUI, check the Stats tab for the kv Dataplane in the kuma-demo-migration namespace, and you should see this metric increment:

inbound_POD_IP_5050.rbac.allowed

The below diagram shows that all services are now in the mesh:

 
---
title: service graph when both client and kv are inside the mesh
---
flowchart LR
    subgraph meshed
        subgraph kuma-demo
            direction LR
            demo-app(demo-app :5000)
            kv(kv :6379)
            demo-app --> kv
        end
        subgraph kuma-demo-migration
            direction LR
            demo-app2(demo-app :5000)
            kv2(kv :6379)
            demo-app2 --> kv2
        end
    end
    linkStyle 0,1 stroke:#d25585, stroke-width:2px;
  

Set strict mode on kv

Finally, to set strict mode you can either edit the policy or remove it (the default is taken from Mesh object which is STRICT).

kubectl delete meshtlses -n kuma-demo-migration kv

What you’ve learned

With a couple of easy steps we were able to gradually bring a service into the mesh without dropping a packet and encrypting the traffic whenever it’s possible.

Next steps

Read more about MeshTLS policy.