Kuma Star

Deploy a multi-zone global control plane

Prerequisites

To set up a multi-zone deployment we will need to:

Usage

Set up the global control plane

The global control plane must run on a dedicated cluster (unless using “Universal on Kubernetes” mode), and cannot be assigned to a zone.

The global control plane on Kubernetes must reside on its own Kubernetes cluster, to keep its resources separate from the resources the zone control planes create during synchronization.

Run:

kumactl install control-plane \
  --set "controlPlane.mode=global" \
  | kubectl apply -f -

Find the external IP and port of the kuma-global-zone-sync service in the kuma-system namespace:

kubectl get services -n kuma-system

NAMESPACE     NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                  AGE
kuma-system   kuma-global-zone-sync     LoadBalancer   10.105.9.10     35.226.196.103   5685:30685/TCP                                                           89s
kuma-system   kuma-control-plane     ClusterIP      10.105.12.133   <none>           5681/TCP,443/TCP,5676/TCP,5677/TCP,5678/TCP,5679/TCP,5682/TCP,5653/UDP   90s

By default, it’s exposed on port 5685. In this example the value is 35.226.196.103:5685. You pass this as the value of <global-kds-address> when you set up the zone control planes.

Set up the zone control planes

You need the following values to pass to each zone control plane setup:

  • zone – the zone name. An arbitrary string. This value registers the zone control plane with the global control plane.
  • kds-global-address – the external IP and port of the global control plane.

For every individual zone create an own cluster and for each run:

kumactl install control-plane \
  --set "controlPlane.mode=zone" \
  --set "controlPlane.zone=<zone-name>" \
  --set "ingress.enabled=true" \
  --set "controlPlane.kdsGlobalAddress=grpcs://<global-kds-address>:5685" \
  --set "controlPlane.tls.kdsZoneClient.skipVerify=true" \
  | kubectl apply -f -

where controlPlane.zone is the same value for all zone control planes in the same zone.

Add --set egress.enabled=true to list of arguments if you want to deploy optional Zone Egress.

Set --set controlPlane.tls.kdsZoneClient.skipVerify=true because the default global control plane’s certificate is self-signed. For production use a certificate signed by a trusted CA. See Secure access across services page for more information.

After installing a zone control plane, make sure to restart the application pods that are already running such that the data plane proxies can be connected.

Verify control plane connectivity

If your global control plane runs on Kubernetes, you’ll need to configure your kumactl like so:

# forward traffic from local pc into global control plane in the cluster
kubectl -n kuma-system port-forward svc/kuma-control-plane 5681:5681 &

# configure control plane for kumactl
kumactl config control-planes add \
  --name global-control-plane \
  --address http://localhost:5681 \
  --skip-verify

You can run kumactl get zones, or check the list of zones in the web UI for the global control plane, to verify zone control plane connections.

When a zone control plane connects to the global control plane, the Zone resource is created automatically in the global control plane.

The Zone Ingress tab of the web UI also lists zone control planes that you deployed with zone ingress.

Ensure mTLS is enabled on the multi-zone meshes

mTLS is mandatory to enable cross-zone service communication. mTLS can be configured in your mesh configuration as indicated in the mTLS section. This is required because Kuma uses the Server Name Indication field, part of the TLS protocol, as a way to pass routing information cross zones.

Cross-zone communication details

For this example we will assume we have a service running in a Kubernetes zone exposing a kuma.io/service with value echo-server_echo-example_svc_1010. The following examples are running in the remote zone trying to access the previously mentioned service.

To view the list of service names available, run:

kubectl get serviceinsight all-services-default -oyaml

apiVersion: kuma.io/v1alpha1
kind: ServiceInsight
mesh: default
metadata:
  name: all-services-default
spec:
  services:
    echo-server_echo-example_svc_1010:
      dataplanes:
        online: 1
        total: 1
      issuedBackends:
        ca-1: 1
      status: online

The following are some examples of different ways to address echo-server in the echo-example Namespace in a multi-zone mesh.

To send a request in the same zone, you can rely on Kubernetes DNS and use the usual Kubernetes hostnames and ports:

curl http://echo-server:1010

Requests are distributed round-robin between zones. You can use locality-aware load balancing to keep requests in the same zone.

To send a request to any zone, you can use the generated kuma.io/service and Kuma DNS:

curl http://echo-server_echo-example_svc_1010.mesh:80

Kuma DNS also supports RFC 1123 compatible names, where underscores are replaced with dots:

curl http://echo-server.echo-example.svc.1010.mesh:80

For security reasons it’s not possible to customize the kuma.io/service in Kubernetes.

If you want to have the same service running on both Universal and Kubernetes make sure to align the Universal’s data plane inbound to have the same kuma.io/service as the one in Kubernetes or leverage MeshHTTPRoute and MeshTCPRoute.

Delete a zone

To delete a Zone we must first shut down the corresponding Kuma zone control plane instances. As long as the Zone CP is running this will not be possible, and Kuma returns a validation error like:

zone: unable to delete Zone, Zone CP is still connected, please shut it down first

When the Zone CP is fully disconnected and shut down, then the Zone can be deleted. All corresponding resources (like Dataplane and DataplaneInsight) will be deleted automatically as well.

kubectl delete zone zone-1

Disable a zone

Change the enabled property value to false in the global control plane:

apiVersion: kuma.io/v1alpha1
kind: Zone
metadata:
  name: zone-1
spec:
  enabled: false

With this setting, the global control plane will stop exchanging configuration with this zone. As a result, the zone’s ingress from zone-1 will be deleted from other zone and traffic won’t be routed to it anymore. The zone will show as Offline in the GUI and CLI.