Kuma Star

MeshExternalService

This resource is experimental!

This resource allows services running inside the mesh to consume services that are not part of the mesh. The MeshExternalService resource allows you to declare external resources instead of relying on MeshPassthrough or passthrough mode.

What is the difference between MeshPassthrough and MeshExternalService?

The main difference is that MeshExternalService is assigned a custom domain and can be targeted by policies. MeshPassthrough, on the other hand, does not alter the address of the original host and cannot be targeted by policies.

Currently MeshExternalService resource only supports targeting by MeshTrafficPermission with Zone Egress. This limitation will be lifted in the next release.

Configuration

Match

This section specifies the rules for matching traffic that will be routed to external resources defined in endpoints section. The only type supported is HostnameGenerator (this field is optional so can be omitted) and it means that it will match traffic directed to a hostname created by the hostname generator. The port field when omitted means that all traffic will be matched. Protocols that are supported are: tcp, grpc, http, http2.

match:
  type: HostnameGenerator # optional
  port: 4244
  protocol: tcp

Endpoints

This section specifies the destination of the matched traffic. It’s possible to define IPs, DNS names and unix domain sockets.

endpoints:
  - address: 1.1.1.1
    port: 12345
  - address: example.com
    port: 80
  - address: unix:///tmp/example.sock

TLS

This section describes the TLS and verification behaviour. TLS origination happens on the sidecar, so if your application is already using TLS you might want to use MeshPassthrough. You can define TLS version requirements, option to allow renegotiation, verification of SNI, SAN, custom CA and client certificate and key for server verification. To disable parts of the verification you can set different mode - SkipSAN, SkipCA, SkipAll, Secured (default).

tls:
  version:
    min: TLS12
    max: TLS13
  allowRenegotiation: false
  verification:
    mode: SkipCA
    serverName: "example.com"
    subjectAltNames:
      - type: Exact
        value: example.com
      - type: Prefix
        value: "spiffe://example.local/ns/local"
    caCert:
      inline: dGVzdA==
    clientCert:
      secret: "123"
    clientKey:
      secret: "456"

When TLS is enabled but caCert is not set, the sidecar uses the autodetected OS-specific CA. The user can override the default CA by setting the path in the environment variable KUMA_DATAPLANE_RUNTIME_DYNAMIC_SYSTEM_CA_PATH for the sidecar.

DNS setup

To be able to access MeshExternalService via a hostname you need to define a HostnameGenerator with a meshExternalService selector. In the future release a default HostnameGenerator will be provided.

Once a HostnameGenerator and a MeshExternalService is in place the following will happen:

  • a hostname (or multiple hostnames if there are many HostnameGenerators matching) are generated using the specified templates
  • a VIP is allocated from 242.0.0.0/8 range (can be changed by KUMA_IPAM_MESH_EXTERNAL_SERVICE_CIDR environment variable)
  • Envoy cluster is created which will use endpoints defined in spec.endpoints as the cluster endpoints

Do not hijack original addresses like httpbin.com (the way it was done with External Service). Hijacking the original address is like performing a man-in-the-middle attack so there is a high chance of something breaking. If you need to transparently pass traffic through the Mesh without modifying it use MeshPassthrough.

For accessing entire subdomains, take a look at Wildcard DNS matching in MeshPassthrough.

Universal mode without Transparent Proxy

MeshExternalService works on Universal mode without Transparent Proxy, but you need to manually define an outbound that targets the correct MeshExternalService:

networking:
  outbound:
    - port: 8080
      backendRef:
        kind: MeshExternalService
        name: mes-http

The whole command will look something like this:

./kuma-dp run \
  --cp-address=https://localhost:5678/ \
  --dns-enabled=false \
  --dataplane-token-file=token-file \
  --dataplane="
type: Dataplane
mesh: default
name: example
networking:
  address: 127.0.0.1
  inbound:
    - port: 16379
      servicePort: 26379
      serviceAddress: 127.0.0.1
      tags:
        kuma.io/service: example
        kuma.io/protocol: tcp
  outbound:
    - port: 8080
      backendRef:
        kind: MeshExternalService
        name: mes-http
  admin:
    port: 9901"

Examples

TCP examples use https://tcpbin.com/ service which is a TCP echo service, check out the website for more details. HTTP examples use https://httpbin.org/ service which is a website for inspecting and debugging HTTP requests. GRPC examples use https://grpcbin.test.k6.io/ service which is a gRPC Request & Response Service. You can use grpcurl as a client, it is available in netshoot debug image alongside other tools used in later sections.

For the examples below we’re using a single-zone deployment and the following HostnameGenerator:

apiVersion: kuma.io/v1alpha1
kind: HostnameGenerator
metadata:
  name: example
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  selector:
    meshExternalService:
      matchLabels:
        kuma.io/origin: zone
  template: "{{ .DisplayName }}.svc.meshext.local"

If you’re in multi-zone deployment and you’re applying resources on the global control plane you’d need a second HostnameGenerator with matchLabels: kuma.io/origin: global for resources applied on the global Control Plane and to adjust the URLs accordingly to match the template.

TCP

This is a simple example of accessing tcpbin.com service without TLS that echos back bytes sent to it.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-tcp
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 4242
    protocol: tcp
  endpoints:
  - address: tcpbin.com
    port: 4242

Running this should result in printing ‘echo this’ in the terminal:

echo 'echo this' | nc -q 3 mes-tcp.svc.meshext.local 4242

TCP with TLS

This example builds up on the previous example adding TLS verification with default system CA. Notice that we’re using a TLS port 4243.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-tcp-tls
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 4243
    protocol: tcp
  endpoints:
  - address: tcpbin.com
    port: 4243
  tls:
    enabled: true
    verification:
      serverName: tcpbin.com

Running this should result in printing ‘echo this’ in the terminal:

echo 'echo this' | nc -q 3 mes-tcp-tls.svc.meshext.local 4243

TCP with mTLS

This example builds up on the previous example adding client cert and key. Notice that we’re using an mTLS port 4244.

In a real world scenario you should use secret and refer to it through it’s name and store sensitive information as a Kubernetes secret instead of using inline. This example is purposefully simplified to make it easy to try out.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-tcp-mtls
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 4244
    protocol: tcp
  endpoints:
  - address: tcpbin.com
    port: 4244
  tls:
    enabled: true
    verification:
      serverName: tcpbin.com
      clientCert:
        inline: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURaVENDQWsyZ0F3SUJBZ0lCS2pBTkJna3Foa2lHOXcwQkFRc0ZBRENCaXpFTE1Ba0dBMVVFQmhNQ1ZWTXgKQ3pBSkJnTlZCQWdNQWtOQk1SWXdGQVlEVlFRSERBMVRZVzRnUm5KaGJtTnBjMk52TVE4d0RRWURWUVFLREFaMApZM0JpYVc0eEREQUtCZ05WQkFzTUEyOXdjekVUTUJFR0ExVUVBd3dLZEdOd1ltbHVMbU52YlRFak1DRUdDU3FHClNJYjNEUUVKQVJZVWFHRnljbmxpWVdka2FVQm5iV0ZwYkM1amIyMHdIaGNOTWpRd05qRTBNRGt4T1RVMldoY04KTWpRd05qRTFNRGt4T1RVMldqQWNNUm93R0FZRFZRUUREQkYwWTNCaWFXNHVZMjl0TFdOc2FXVnVkRENDQVNJdwpEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBS3VZVHo0UzRIcEtvc05FRWcxbDFOT3ZORjZtCkxEN0o4d1ZsdFVSd2JXNDNaM2JJVmNOUFVuNjdkTVpEUUpRcVU2Z05kRTg0eXBvQTUxbXpqcC9IeGZZY2c1cEMKZXFyK1RtVDV1S3UyME8ycDZmYXhrZDlYUHpmWXUybE1QM0tROU9DbmpNbEIwQkFiNUNpTklRTmIwMUtheDhCOAphT0Z1a3VYKzdheWhRRXFxRHJ5d0d6Q0hJc2ppU3lDVHdRRXhiQVJTY1BGOW5XaDlHZVRERGlRSEtKUXFVOEpnClVoRGMzQUdjdlI2c0pEZFZ1RXl2UndrNVcrTFhoNktPQVAwNHEwaGF2OXFiaXNaZitXNVNRMDFvbXIzRGhzSTQKZkx2c05oN0I5UGpqaVNnV09hQ3F4bEdCSDNJdUJtWXgxR1k5RCtia0VJazVsVkZ4WC92d0N1SmhzRlVDQXdFQQpBYU5DTUVBd0hRWURWUjBPQkJZRUZGSDRnN3V3OFdzTUIwYlRKNld2NFV5NGtlME1NQjhHQTFVZEl3UVlNQmFBCkZPTHVNb3dCU0FaZlY1djgyTG1sYUlJT3ZVL0RNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUJmYzlxQWFNUjcKbTEzMlZoSk52M0dVOXNZMytMUHN6dkJMMkYyeGw5U1AwVlA3cG9Hd3ZvSnJ0cXdGR0FFaGtibzNCUkhvZTBTMwpVUkdtTlNjWlpTMVdkQllBSVdvdkJKbFNhakxqVDVWNGNrc0FXVjZ3djlmaE9XcXFTSmpVaEwwZ3FkWkp2NDNoCkFnVnZGeXQrUHF4NmpZMHBhUkp6TS9OM1pkTFBDTkZiMVlIMzE3Q0FyQlV1R2xWQzJzRDNJd0lEK0YzczdhOEcKQ2FSb0VnWGpNdEZVcnBJN2RqaTVSeHlENE9Ma2o3bmw0aXk3anJDWlAvRitKbys4dEo3NVI4VW8xeU8zSXorOApTanFkYWdwT3RKQzdEV1pyNlNYOW85a0lsTUl4SEhITmN0RElteXB6SytmWFpaZUFnTVVTSE1kWCsrd0xNTVdrCnZ0Tm1vUFZVVHJvNwotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCgo=
      clientKey:
        inline: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ3JtRTgrRXVCNlNxTEQKUkJJTlpkVFRyelJlcGl3K3lmTUZaYlZFY0cxdU4yZDJ5RlhEVDFKK3UzVEdRMENVS2xPb0RYUlBPTXFhQU9kWgpzNDZmeDhYMkhJT2FRbnFxL2s1aytiaXJ0dER0cWVuMnNaSGZWejgzMkx0cFREOXlrUFRncDR6SlFkQVFHK1FvCmpTRURXOU5TbXNmQWZHamhicExsL3Uyc29VQktxZzY4c0Jzd2h5TEk0a3NnazhFQk1Xd0VVbkR4Zloxb2ZSbmsKd3c0a0J5aVVLbFBDWUZJUTNOd0JuTDBlckNRM1ZiaE1yMGNKT1Z2aTE0ZWlqZ0Q5T0t0SVdyL2FtNHJHWC9sdQpVa05OYUpxOXc0YkNPSHk3N0RZZXdmVDQ0NGtvRmptZ3FzWlJnUjl5TGdabU1kUm1QUS9tNUJDSk9aVlJjVi83CjhBcmlZYkJWQWdNQkFBRUNnZ0VBRTRiM3JaYTBXUFpaWTJOQnNxaWQrYUQ4a3JEU1pDclRMeEFOK3NYWWppeGIKNTlhUWUvTnc3ZDhqUU5TeWFxb09ieGRvM3dNVmUwVVREdEF5TU5pcEhJTE9MeVhWazlQdzAramZMUnRXMTFUNAp2UXdrRDRoOE56ekF4eERZUDQ5amJwVmluaHlSTXVRWnFNdTJzQTBwRlVOcjYrbThmYnI1bUpiVU1Vc0FaLzZXCmJzRjYyOEZxcVk3S0ZDSkY0RmNZaEdoREV2WTZiaWFUc3l2aUhSRWFhSkl1Mm5qQ3EwUnAza1UwSlhGamlXVS8KZSt4WHpoeXlKbkFsejNzRmJycWZzV3I1UHJ5NTJ3ZUlFOVBDSUxINzhldld1dXIxc0NSSHJ6SVAxS3JHWU1XMApkcXVJcllFbzZVUm1SU2ZuTzBjNFVjK2ltZzBLQUovWTFZeWM1c0MzQVFLQmdRRGhWZGt5VkRjWGUydFREZlFxCnBsU3FyWFRaZCtLRlhTdzBwNDNwMmltN2hzSmxmNUlHN3AwYWFEdUJlZUdsVmhGT3E5dU5CRmgxMWN4ajgza2gKZVI1Q1RWZXpjV1NhWExLN3pMUjdqNUFjdTlyaU1tWGFEUUdJaWQrNU83cGFnRFNtNHpOSENZdnFKYm1XcU5DWApOcXVsNFI3L3NoUGd3MjQ0R0J6TmxSdjJZUUtCZ1FEQzhrWGM4UWt1amtwS0pkVm8wWFZFd1MzdWFmb3lVeGNxClBmb2dWOEpsUE1GRCs5VjJjNkhSMkpJWktKMzU0RFhsby8ydWllZzhQT0lUMXBpVlVCQk55eElQY1FxMUcweFcKQmtPMkpyMmdyZlN6KzNwM2RzTGR5dXU5bWc4eVBiNWhwRVdjTjFtUDFiWko3YUs0Qjk5SFArczR2TXd1c2xWOApzZDI2Uk1YV2RRS0JnRnRqeWhkVGVKU1poY25GbXdYQk9BMlJGQmN2UER3Q3NlOFpGY0dHcmU1VWxYczg1aWpSCmxmNGowQjZQSkNrK1l2NlpUUTVBZVBBeHFoZlBvNDBqNWxYVnNJQWl1VDZ4NGZ1dzVuSkdvNWhEeUY1OU9qbloKbEltZ0FaREszS1hmNFhyZUl1bm93VXBSeXBlRUdEVjhBdG5nR0FaMFh3T0Z2Nm9ZZlhZVHg2ZUJBb0dBRlJQNAo5ZENoKzRTckI2VmJrNy9CL0RNZThqNUhMUlhLMVdocUdRRWtKYW9TQTNYQk9OTjcxYUtpK1ZGbzgxR0l3bEdlCjVqWkhBK3haVFdmUWk2UmlmdWJNQnh0ajJ2MGVuZGFEajdoVW5JRHlpbHRRZklZOHY1cG5MdEx2ZmJFcldvZFcKZDNPTW5YNnYvUUpTcTY4K053ZjBPT2hBODNPWXhxaThucDA4L3RrQ2dZRUFqYkNUcHhIM2hmd2p2Y2t2TFIxVQpHWVdUaVkzMWJWVDJEVTNFZjlLb09KQXJQSlRuRlNyQ0VuNEpESGk0S1NrencvRHpQRmtMRzdsSXZFV21qcTJTCnRyUGdHMEtCcWxQa1pCRkRYUy9tdWFWVzQyUFhFeVhLV1kzdFFaRnVzQk5XdVZXOFNvZW8zdTd1YWlXbHRPNzAKTnZOSW83U21MNVJ0UEF4d1NpUTJzOGc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0KCg==

Running this should result in printing ‘echo this’ in the terminal:

echo 'echo this' | nc -q 3 mes-tcp-mtls.svc.meshext.local 4244

HTTP

This is a simple example using plaintext HTTP.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-http
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 80
    protocol: http
  endpoints:
  - address: httpbin.org
    port: 80

Running this should result in printing httpbin.org HTML in the terminal:

curl -s http://mes-http.svc.meshext.local

HTTPS

This example builds up on the previous example adding TLS verification with default system CA.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-https
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 80
    protocol: http
  endpoints:
  - address: httpbin.org
    port: 443
  tls:
    enabled: true
    verification:
      serverName: httpbin.org

Running this should result in printing httpbin.org HTML in the terminal:

curl http://mes-https.svc.meshext.local

gRPC

This is a simple example using plaintext gRPC.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-grpc
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 9000
    protocol: grpc
  endpoints:
  - address: grpcbin.test.k6.io
    port: 9000

Running this should result in printing grpcbin.test.k6.io available methods:

grpcurl -plaintext -v mes-grpc.svc.meshext.local:9000 list

gRPCS

This example builds up on the previous example adding TLS verification with default system CA. Notice that we’re using a different port 9001.

apiVersion: kuma.io/v1alpha1
kind: MeshExternalService
metadata:
  name: mes-grpcs
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  match:
    type: HostnameGenerator
    port: 9001
    protocol: grpc
  endpoints:
  - address: grpcbin.test.k6.io
    port: 9001
  tls:
    enabled: true
    verification:
      serverName: grpcbin.test.k6.io

Running this should result in printing grpcbin.test.k6.io available methods:

grpcurl -plaintext -v mes-grpcs.svc.meshext.local:9001 list # this is using plaintext because Envoy is doing TLS origination

All policy configuration settings