Kuma Star

MeshRetry

This policy uses new policy matching algorithm. Do not combine with Retry.

This policy enables Kuma to know how to behave if there are failed requests which could be retried.

TargetRef support matrix

targetRef Allowed kinds
targetRef.kind Mesh, Dataplane, MeshSubset(deprecated)
to[].targetRef.kind Mesh, MeshService, MeshExternalService, MeshHTTPRoute

To learn more about the information in this table, see the matching docs.

Configuration

The policy let you configure retry behaviour for HTTP, GRPC and TCP protocols. The protocol is selected by picking the most specific protocol.

Each protocol has a separate section under default in the policy yaml. Some sections are common between protocols or have similar meaning.

Retry on

The field retryOn is a list of conditions which will cause a retry.

For HTTP these are related to the response status code or method (5xx, 429, HttpMethodGet). For gRPC these are status codes in response headers (canceled, deadline-exceeded, etc.). There is no equivalent for TCP.

One or more conditions can be specified, for example:

retryOn:
  - "429"
  - "503"

means that it the policy will retry on a status code 429 or 503.

Full list of available HTTP conditions:

retryOn:
  - 5XX
  - GatewayError
  - Reset
  - Retriable4xx
  - ConnectFailure
  - EnvoyRatelimited
  - RefusedStream
  - Http3PostConnectFailure
  - HttpMethodConnect
  - HttpMethodDelete
  - HttpMethodGet
  - HttpMethodHead
  - HttpMethodOptions
  - HttpMethodPatch
  - HttpMethodPost
  - HttpMethodPut
  - HttpMethodTrace
  - "429" # any HTTP status code
  - "503"

Full list of available gRPC conditions:

retryOn:
  - Canceled
  - DeadlineExceeded
  - Internal
  - ResourceExhausted
  - Unavailable

Back off

This parameter is applicable to both HTTP and GRPC.

It consists of BaseInterval (the amount of time between retries) and MaxInterval (the maximal amount of time taken between retries).

We use an exponential back-off algorithm with jitter for retries. Given a base interval B and retry number N, the back-off for the retry is in the range [0, (2N - 1) × B).

For example, given a 25 ms interval, the first retry will be delayed randomly by 0-24 ms, the second by 0-74 ms, the third by 0-174 ms, and so on.

The interval is capped at a MaxInterval, which defaults to 10 times the BaseInterval.

Rate limited back off

This parameter is applicable to both HTTP and GRPC.

MeshRetry can be configured in such a way that when the upstream server rate limits the request and responds with a header like retry-after or x-ratelimit-reset it uses the value from the header to determine when to send the retry request instead of the back off algorithm.

Example

Given this configuration:

retryOn:
  - "503"
rateLimitedBackOff:
  resetHeaders:
    - name: retry-after
      format: Seconds
    - name: x-ratelimit-reset
      format: UnixTimestamp

and an HTTP response:

HTTP/1.1 503 Service Unavailable
retry-after: 15

The retry request will be issued after 15 seconds.

If the response is as follows:

HTTP/1.1 503 Service Unavailable
x-ratelimit-reset: 1706096119

The request will be retried at Wed Jan 24 2024 11:35:19 GMT+0000.

If the response does not contain retry-after or x-ratelimit-reset header (with valid integer value) then the amount of time to wait before issuing a request is determined by back off algorithm.

Examples

HTTP frontend to backend on 5xx

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-http
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-http
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      http:
        numRetries: 10
        backOff:
          baseInterval: 15s
          maxInterval: 20m
        retryOn:
        - 5xx

gRPC frontend to backend on DeadlineExceeded

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-grpc
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-grpc
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    tags:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      grpc:
        numRetries: 5
        backOff:
          baseInterval: 5s
          maxInterval: 1m
        retryOn:
        - DeadlineExceeded

TCP frontend to backend

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-tcp
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      tcp:
        maxConnectAttempt: 5

apiVersion: kuma.io/v1alpha1
kind: MeshRetry
metadata:
  name: frontend-to-backend-retry-tcp
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: frontend
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      tcp:
        maxConnectAttempt: 5

See also

All policy options

targetRef object
TargetRef is a reference to the resource the policy takes an effect on. The resource could be either...
kind enum required
Kind of the referenced resource
Values: Mesh | MeshSubset | MeshGateway | MeshService | MeshExternalService | MeshMultiZoneService | MeshServiceSubset | MeshHTTPRoute | Dataplane
name string
Name of the referenced resource. Can only be used with kinds: `MeshService`, `MeshServiceSubset` and...
namespace string
Namespace specifies the namespace of target resource. If empty only resources in policy namespace wi...
labels object
Labels are used to select group of MeshServices that match labels. Either Labels or Name and Namespa...
sectionName string
SectionName is used to target specific section of resource. For example, you can target port from Me...
mesh string
Mesh is reserved for future use to identify cross mesh resources.
proxyTypes array
ProxyTypes specifies the data plane types that are subject to the policy. When not specified, all da...
tags object
Tags used to select a subset of proxies by tags. Can only be used with kinds `MeshSubset` and `MeshS...
to array
To list makes a match between the consumed services and corresponding configurations
TargetRef is a reference to the resource that represents a group of destinations.
kind enum required
Kind of the referenced resource
Values: Mesh | MeshSubset | MeshGateway | MeshService | MeshExternalService | MeshMultiZoneService | MeshServiceSubset | MeshHTTPRoute | Dataplane
name string
Name of the referenced resource. Can only be used with kinds: `MeshService`, `MeshServiceSubset` and...
namespace string
Namespace specifies the namespace of target resource. If empty only resources in policy namespace wi...
labels object
Labels are used to select group of MeshServices that match labels. Either Labels or Name and Namespa...
sectionName string
SectionName is used to target specific section of resource. For example, you can target port from Me...
mesh string
Mesh is reserved for future use to identify cross mesh resources.
proxyTypes array
ProxyTypes specifies the data plane types that are subject to the policy. When not specified, all da...
tags object
Tags used to select a subset of proxies by tags. Can only be used with kinds `MeshSubset` and `MeshS...
Default is a configuration specific to the group of destinations referenced in 'targetRef'
GRPC defines a configuration of retries for GRPC traffic
BackOff is a configuration of durations which will be used in an exponential backoff strategy betwee...
baseInterval string
BaseInterval is an amount of time which should be taken between retries. Must be greater than zero. ...
maxInterval string
MaxInterval is a maximal amount of time which will be taken between retries. Default is 10 times the...
numRetries integer
NumRetries is the number of attempts that will be made on failed (and retriable) requests. If not se...
perTryTimeout string
PerTryTimeout is the maximum amount of time each retry attempt can take before it times out. If not ...
RateLimitedBackOff is a configuration of backoff which will be used when the upstream returns one of...
maxInterval string
MaxInterval is a maximal amount of time which will be taken between retries. If not specified then t...
ResetHeaders specifies the list of headers (like Retry-After or X-RateLimit-Reset) to match against ...
format enum required
The format of the reset header.
Values: Seconds | UnixTimestamp
name string required
The Name of the reset header.
retryOn array
RetryOn is a list of conditions which will cause a retry.
HTTP defines a configuration of retries for HTTP traffic
BackOff is a configuration of durations which will be used in exponential backoff strategy between r...
baseInterval string
BaseInterval is an amount of time which should be taken between retries. Must be greater than zero. ...
maxInterval string
MaxInterval is a maximal amount of time which will be taken between retries. Default is 10 times the...
HostSelection is a list of predicates that dictate how hosts should be selected when requests are re...
predicate enum required
Type is requested predicate mode.
Values: OmitPreviousHosts | OmitHostsWithTags | OmitPreviousPriorities
tags object
Tags is a map of metadata to match against for selecting the omitted hosts. Required if Type is Omit...
updateFrequency integer
UpdateFrequency is how often the priority load should be updated based on previously attempted prior...
Default: 2
hostSelectionMaxAttempts integer
HostSelectionMaxAttempts is the maximum number of times host selection will be reattempted before gi...
numRetries integer
NumRetries is the number of attempts that will be made on failed (and retriable) requests. If not s...
perTryTimeout string
PerTryTimeout is the amount of time after which retry attempt should time out. If left unspecified, ...
RateLimitedBackOff is a configuration of backoff which will be used when the upstream returns one of...
maxInterval string
MaxInterval is a maximal amount of time which will be taken between retries. If not specified then t...
ResetHeaders specifies the list of headers (like Retry-After or X-RateLimit-Reset) to match against ...
format enum required
The format of the reset header.
Values: Seconds | UnixTimestamp
name string required
The Name of the reset header.
RetriableRequestHeaders is an HTTP headers which must be present in the request for retries to be at...
name string required
Name is the name of the HTTP Header to be matched. Name MUST be lower case as they will be handled w...
type enum
Type specifies how to match against the value of the header.
Values: Exact | Present | RegularExpression | Absent | Prefix
Default: "Exact"
value string
Value is the value of HTTP Header to be matched.
RetriableResponseHeaders is an HTTP response headers that trigger a retry if present in the response...
name string required
Name is the name of the HTTP Header to be matched. Name MUST be lower case as they will be handled w...
type enum
Type specifies how to match against the value of the header.
Values: Exact | Present | RegularExpression | Absent | Prefix
Default: "Exact"
value string
Value is the value of HTTP Header to be matched.
retryOn array
RetryOn is a list of conditions which will cause a retry. Available values are: [5XX, GatewayError, ...
TCP defines a configuration of retries for TCP traffic
maxConnectAttempt integer
MaxConnectAttempt is a maximal amount of TCP connection attempts which will be made before giving up