This policy allows to configure low-level Envoy resources(opens new window) directly in those situations where Kuma-native policies do not expose the Envoy functionality we are looking for.
Please open a new issue on GitHub(opens new window) describing what missing functionality couldn't be found as a Kuma-native policy and we will make sure to prioritize it in the roadmap for future versions of Kuma.
Specifically by using the ProxyTemplate policy we can provide custom definitions of:
By default Kuma uses the following default ProxyTemplate resource for every data plane proxy (kuma-dp, which embeds Envoy) that is being added to a Mesh. With a custom ProxyTemplate resource it is possible to extend or replace the default Envoy configuration that Kuma provides to every data plane proxy.
The default ProxyTemplate resource that by default Kuma applies to every data plane proxy looks like:
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service:'*'conf:# `imports` allows us to reuse the dataplane configuration that Kuma# generates automatically and add more customizations on top of itimports:# `default-proxy` is a reference name for the default# data plane proxy configuration generated by Kuma- default-proxy
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service:'*'conf:# `imports` allows us to reuse the dataplane configuration that Kuma# generates automatically and add more customizations on top of itimports:# `default-proxy` is a reference name for the default# data plane proxy configuration generated by Kuma- default-proxy
1 2 3 4 5 6 7 8 9 10 11 12 13
In the examples described above, please note that:
The selectors object allows us to determine what data plane proxies will be targeted by the ProxyTemplate resource (accordingly to the Kuma Tags specified).
The imports object allows us to reuse the configuration that Kuma generates automatically so that it can be extended by our own custom configuration.
The only available builtin configuration that can be used inside the imports section are:
default-proxy - default configuration for regular dataplanes.
ingress - default configuration for Ingress dataplanes.
In order to customize the configuration of a particular data plane proxy (or a group of data plane proxies), we can apply modifications.
You can combine many modifications of any type within one ProxyTemplate. Each modification consists of the following sections:
operation - operation that will be applied on generated config (e.g. add, remove, patch).
match - some operation can be applied on matched resources (e.g. remove only resource of given name, patch all outbound resources).
value - raw Envoy xDS configuration. Can be partial if operation is patch.
All resources generated by Kuma are marked with origin value, so you can match resources by it. Examples: add new filters but only on inbound listeners, set timeouts on outbound clusters etc.
Available origins:
inbound - resources generated for incoming traffic.
outbound - resources generated for outgoing traffic.
transparent - resources generated for transparent proxy functionality.
prometheus - resources generated when Prometheus metrics are enabled.
direct-access - resources generated for Direct Access functionality.
ingress - resources generated for Ingress Dataplane.
add - add a new cluster or replace existing if the name is the same.
remove - remove a cluster.
patch - patch a part of cluster definition.
Available matchers:
name - name of the cluster.
origin - origin of the cluster.
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-cluster:operation: add
value:|
name: test-cluster
connectTimeout: 5s
type: STATIC-cluster:operation: patch
match:# optional: if absent, all clusters will be patchedname: test-cluster # optional: if absent, all clusters regardless of name will be patchedorigin: inbound # optional: if absent, all clusters regardless of its origin will be patchedvalue:|# you can specify only part of cluster definition that will be merged into existing clusterconnectTimeout: 5s
-cluster:operation: remove
match:# optional: if absent, all clusters will be removedname: test-cluster # optional: if absent, all clusters regardless of name will be removedorigin: inbound # optional: if absent, all clusters regardless of its origin will be removed
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-cluster:operation: add
value:|
name: test-cluster
connectTimeout: 5s
type: STATIC-cluster:operation: patch
match:# optional: if absent, all clusters will be patchedname: test-cluster # optional: if absent, all clusters regardless of name will be patchedorigin: inbound # optional: if absent, all clusters regardless of its origin will be patchedvalue:|# you can specify only part of cluster definition that will be merged into existing clusterconnectTimeout: 5s
-cluster:operation: remove
match:# optional: if absent, all clusters will be removedname: test-cluster # optional: if absent, all clusters regardless of name will be removedorigin: inbound # optional: if absent, all clusters regardless of its origin will be removed
add - add a new listener or replace existing if the name is the same.
remove - remove a listener.
patch - patch a part of listener definition.
Available matchers:
name - name of the listener.
origin - origin of the listener.
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-listener:operation: add
value:|
name: test-listener
address:
socketAddress:
address: 192.168.0.1
portValue: 8080-listener:operation: patch
match:# optional: if absent, all listeners will be patchedname: test-listener # optional: if absent, all listeners regardless of name will be patchedorigin: inbound # optional: if absent, all listeners regardless of its origin will be patchedvalue:|# you can specify only part of listener definition that will be merged into existing listenercontinueOnListenerFiltersTimeout:true-listener:operation: remove
match:# optional: if absent, all listeners will be removedname: test-listener # optional: if absent, all listeners regardless of name will be removedorigin: inbound # optional: if absent, all listeners regardless of its origin will be removed
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-listener:operation: add
value:|
name: test-listener
address:
socketAddress:
address: 192.168.0.1
portValue: 8080-listener:operation: patch
match:# optional: if absent, all listeners will be patchedname: test-listener # optional: if absent, all listeners regardless of name will be patchedorigin: inbound # optional: if absent, all listeners regardless of its origin will be patchedvalue:|# you can specify only part of listener definition that will be merged into existing listenercontinueOnListenerFiltersTimeout:true-listener:operation: remove
match:# optional: if absent, all listeners will be removedname: test-listener # optional: if absent, all listeners regardless of name will be removedorigin: inbound # optional: if absent, all listeners regardless of its origin will be removed
addFirst - add a new filter as a first filter in Filter Chain.
addLast - add a new filter as a last filter in Filter Chain.
addAfter - add a new filter after other filter in Filter Chain that is matched using match section.
addBefore - add a new filter before other filter in Filter Chain that is matched using match section.
patch - patch a matched filter in Filter Chain.
remove - remove a filter in Filter Chain.
Available matchers:
name - name of the network filter.
listenerName - name of the listener.
origin - origin of the listener.
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-networkFilter:operation: addFirst
match:# optional: if absent, filter will be added to all listenerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addLast
match:# optional: if absent, filter will be added to all listenerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addBefore
match:name: envoy.tcp_proxy # a new filter (Local RateLimit) will be added before existing (TcpProxy). If there is no TcpProxy filter, Local RateLimit won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addAfter
match:name: envoy.tcp_proxy # a new filter (Local RateLimit) will be added after existing (TcpProxy). If there is no TcpProxy filter, Local RateLimit won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: patch
match:name: envoy.tcp_proxy
listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be patched within all listeners regardless of nameorigin: inbound # optional: if absent, filter will be patched within all listeners regardless of its originvalue:|# you can specify only part of filter definition that will be merged into existing filtername: envoy.tcp_proxy
typedConfig:'@type': type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
idleTimeout: 10s
-networkFilter:operation: remove
match:# optional: if absent, all filters from all listeners will be removedname: envoy.tcp_proxy # optional: if absent, all filters regardless of name will be removedlistenerName: inbound:127.0.0.0:80# optional: if absent, all filters regardless of the listener name will be removedorigin: inbound # optional: if absent, all filters regardless of its origin will be removed
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-networkFilter:operation: addFirst
match:# optional: if absent, filter will be added to all listenerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addLast
match:# optional: if absent, filter will be added to all listenerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addBefore
match:name: envoy.tcp_proxy # a new filter (Local RateLimit) will be added before existing (TcpProxy). If there is no TcpProxy filter, Local RateLimit won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: addAfter
match:name: envoy.tcp_proxy # a new filter (Local RateLimit) will be added after existing (TcpProxy). If there is no TcpProxy filter, Local RateLimit won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.network.local_ratelimit
typedConfig:
'@type': type.googleapis.com/config.filter.network.local_rate_limit.v2alpha.LocalRateLimit
statPrefix: rateLimit
tokenBucket:
fillInterval: 1s-networkFilter:operation: patch
match:name: envoy.tcp_proxy
listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be patched within all listeners regardless of nameorigin: inbound # optional: if absent, filter will be patched within all listeners regardless of its originvalue:|# you can specify only part of filter definition that will be merged into existing filtername: envoy.tcp_proxy
typedConfig:'@type': type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
idleTimeout: 10s
-networkFilter:operation: remove
match:# optional: if absent, all filters from all listeners will be removedname: envoy.tcp_proxy # optional: if absent, all filters regardless of name will be removedlistenerName: inbound:127.0.0.0:80# optional: if absent, all filters regardless of the listener name will be removedorigin: inbound # optional: if absent, all filters regardless of its origin will be removed
addFirst - add a new filter as a first filter in HTTP Connection Manager.
addLast - add a new filter as a last filter in HTTP Connection Manager.
addAfter - add a new filter after other filter in HTTP Connection Manager that is matched using match section.
addBefore - add a new filter before other filter in HTTP Connection Manager that is matched using match section.
patch - patch a matched filter in HTTP Connection Manager.
remove - remove a filter in HTTP Connection Manager.
Available matchers:
name - name of the network filter
listenerName - name of the listener
origin - origin of the listener
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-httpFilter:operation: addFirst
match:# optional: if absent, filter will be added to all HTTP Connection ManagerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addLast
match:# optional: if absent, filter will be added to all HTTP Connection ManagerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addBefore
match:name: envoy.router # a new filter (Gzip) will be added before existing (Router). If there is no Router filter, Gzip won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addAfter
match:name: envoy.router # a new filter (Gzip) will be added after existing (Router). If there is no Router filter, Gzip won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: patch
match:name: envoy.filters.http.router
listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be patched within all listeners regardless of nameorigin: inbound # optional: if absent, filter will be patched within all listeners regardless of its originvalue:|# you can specify only part of filter definition that will be merged into existing filtername: envoy.filters.http.router
typedConfig:'@type': type.googleapis.com/envoy.config.filter.http.router.v2.Router
dynamicStats:false-httpFilter:operation: remove
match:# optional: if absent, all filters from all listeners will be removedname: envoy.filters.http.gzip # optional: if absent, all filters regardless of name will be removedlistenerName: inbound:127.0.0.0:80# optional: if absent, all filters regardless of the listener name will be removedorigin: inbound # optional: if absent, all filters regardless of its origin will be removed
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-httpFilter:operation: addFirst
match:# optional: if absent, filter will be added to all HTTP Connection ManagerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addLast
match:# optional: if absent, filter will be added to all HTTP Connection ManagerslistenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addBefore
match:name: envoy.router # a new filter (Gzip) will be added before existing (Router). If there is no Router filter, Gzip won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: addAfter
match:name: envoy.router # a new filter (Gzip) will be added after existing (Router). If there is no Router filter, Gzip won't be added.listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be added to all listeners regardless of nameorigin: inbound # optional: if absent, filter will be added to all listeners regardless of its originvalue:|
name: envoy.filters.http.gzip
typedConfig:
'@type': type.googleapis.com/envoy.config.filter.http.gzip.v2.Gzip
memoryLevel: 9-httpFilter:operation: patch
match:name: envoy.filters.http.router
listenerName: inbound:127.0.0.0:80# optional: if absent, filter will be patched within all listeners regardless of nameorigin: inbound # optional: if absent, filter will be patched within all listeners regardless of its originvalue:|# you can specify only part of filter definition that will be merged into existing filtername: envoy.filters.http.router
typedConfig:'@type': type.googleapis.com/envoy.config.filter.http.router.v2.Router
dynamicStats:false-httpFilter:operation: remove
match:# optional: if absent, all filters from all listeners will be removedname: envoy.filters.http.gzip # optional: if absent, all filters regardless of name will be removedlistenerName: inbound:127.0.0.0:80# optional: if absent, all filters regardless of the listener name will be removedorigin: inbound # optional: if absent, all filters regardless of its origin will be removed
apiVersion: kuma.io/v1alpha1
kind: ProxyTemplate
mesh: default
metadata:name: custom-template-1spec:selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-virtualHost:operation: add
value:|
name: backend
domains:
- "*"
routes:
- match:
prefix: /
route:
cluster: backend-virtualHost:operation: patch
match:# optional: if absent, all listeners will be patchedname: backend # optional: if absent, all virtual hosts regardless of name will be patchedorigin: inbound # optional: if absent, all virtual hosts regardless of its origin will be patchedrouteConfigurationName: outbound:backend # optional: if absent, all virtual hosts in all route configurations will be patchedvalue:|# you can specify only part of virtual host definition that will be merged into existing virtual hostretryPolicy:retryOn: 5xx
numRetries:3-virtualHost:operation: remove
match:# optional: if absent, all virtual hosts will be removedname: test-listener # optional: if absent, all virtual hsots regardless of name will be removedorigin: inbound # optional: if absent, all virtual hosts regardless of its origin will be removed
type: ProxyTemplate
mesh: default
name: custom-template-1selectors:-match:kuma.io/service: backend
conf:imports:- default-proxy
modifications:-virtualHost:operation: add
value:|
name: backend
domains:
- "*"
routes:
- match:
prefix: /
route:
cluster: backend-virtualHost:operation: patch
match:# optional: if absent, all listeners will be patchedname: backend # optional: if absent, all virtual hosts regardless of name will be patchedorigin: inbound # optional: if absent, all virtual hosts regardless of its origin will be patchedrouteConfigurationName: outbound:backend # optional: if absent, all virtual hosts in all route configurations will be patchedvalue:|# you can specify only part of virtual host definition that will be merged into existing virtual hostretryPolicy:retryOn: 5xx
numRetries:3-virtualHost:operation: remove
match:# optional: if absent, all virtual hosts will be removedname: test-listener # optional: if absent, all virtual hsots regardless of name will be removedorigin: inbound # optional: if absent, all virtual hosts regardless of its origin will be removed
At runtime, whenever kuma-cp generates the configuration for a given data plane proxy, it will proceed as follows:
Kuma will search for all the ProxyTemplates resources that have been defined in the specified Mesh.
Then, it will load in memory those ProxyTemplates resources whose selectorsmatch either an inbound or a gateway definition of any data plane proxy accordingly to the Kuma Tags selected.
Every matching ProxyTemplate will be then ranked. The ProxyTemplate resource with the highest ranking will be used to generate the configuration for that specific data plane proxy (or proxies).
If the ProxyTemplate resource specifies an imports object, these resources will be generated first.
If a ProxyTemplate defines a modification object, all modifications will be applied, one by one in order defined in modification section.