MeshCircuitBreaker

This policy will look for errors in the live traffic being exchanged between our data plane proxies. It will mark a data proxy as unhealthy if certain conditions are met. The policy will ensure that no additional traffic can reach an unhealthy data plane proxy until it is healthy again.

Circuit breakers - unlike active MeshHealthChecks - do not send additional traffic to our data plane proxies but they rather inspect the existing service traffic. They are also commonly used to prevent cascading failures.

The conditions that determine when a circuit breaker is closed or open are being configured on connection limits or outlier detection basis. For outlier detection to open circuit breaker you can configure what we call detectors. This policy provides 5 different types of detectors, and they are triggered on some deviations in the upstream service behavior. All detectors could coexist on the same outbound interface.

Once one of the detectors has been triggered the corresponding data plane proxy is ejected from the set of the load balancer for a period equal to baseEjectionTime. Every further ejection of the same data plane proxy will further extend the baseEjectionTime multiplied by the number of ejections: for example the fourth ejection will be lasting for a period of time of 4 * baseEjectionTime.

This policy provides passive checks. If you want to configure active checks, please utilize the MeshHealthCheck policy. Data plane proxies with passive checks won’t explicitly send requests to other data plane proxies to determine if target proxies are healthy or not.

TargetRef support matrix

To learn more about the information in this table, see the matching docs.

Configuration

Connection limits

• maxConnections - (optional) The maximum number of connections allowed to be made to the upstream Envoy Cluster. If not specified then equal to 1024.
• maxConnectionPools - (optional) The maximum number of connection pools per Envoy Cluster that are concurrently supported at once. Set this for Envoy Clusters which create a large number of connection pools. If not specified, the default is unlimited.
• maxPendingRequests - (optional) The maximum number of pending requests that are allowed to the upstream Envoy Cluster. This limit is applied as a connection limit for non-HTTP traffic. If not specified then equal to 1024.
• maxRetries - (optional) The maximum number of parallel retries that will be allowed to the upstream Envoy Cluster. If not specified then equal to 3.
• maxRequests - (optional) The maximum number of parallel requests that are allowed to be made to the upstream Envoy Cluster. This limit does not apply to non-HTTP traffic. If not specified then equal to 1024.

Outlier detection

Outlier detection can be configured for HTTP, TCP or gRPC traffic.

• disabled - (optional) When set to true, outlierDetection configuration won’t take any effect.
• interval - (optional) The time interval between ejection analysis sweeps. This can result in both new ejections and hosts being returned to service.
• baseEjectionTime - (optional) The base time that a host is ejected for. The real time is equal to the base time multiplied by the number of times the host has been ejected.
• maxEjectionPercent - (optional) The maximum % of an upstream Envoy Clusters that can be ejected due to outlier detection. Defaults to 10% but will eject at least one host regardless of the value.
• splitExternalAndLocalErrors - (optional) Determines whether to distinguish local origin failures from external errors. If set to true the following configuration parameters are taken into account: detectors.localOriginFailures.consecutive.

• detectors - Contains configuration for supported outlier detectors. At least one detector needs to be configured when policy is configured for outlier detection.

• healthyPanicThreshold - (optional) Allows to configure panic threshold for Envoy cluster. If not specified, the default is 50%. To disable panic mode, set to 0%. The panic threshold is a percentage that determines when Envoy enters panic mode due to too few healthy hosts in an upstream cluster’s priority level. If the percentage of healthy hosts falls below the threshold (for example 50%), Envoy ignores the unhealthy status and sends traffic to all hosts in that priority level (by default). If the threshold is set to 0, panic mode is disabled for that priority, and if all hosts are unhealthy, Envoy fails to select a host, returning a ‘503 - no healthy upstream’ error.

Detectors configuration

Configuration for supported outlier detectors. At least one detector needs to be configured when policy is configured for outlier detection.

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        detectors:
          totalFailures:
            consecutive: 10

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        splitExternalAndLocalErrors: true
        detectors:
          totalFailures:
            consecutive: 10

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        detectors:
          gatewayFailures:
            consecutive: 10

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        splitExternalAndLocalErrors: true
        detectors:
          gatewayFailures:
            consecutive: 10

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        splitExternalAndLocalErrors: true
        detectors:
          localOriginFailures:
            consecutive: 10

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        splitExternalAndLocalErrors: true
        detectors:
          successRate:
            minimumHosts: 5
            requestVolume: 10
            standardDeviationFactor: "1.9"

type: MeshCircuitBreaker
mesh: default
name: circuit-breaker
spec:
  targetRef:
    kind: Mesh
  to:
  - targetRef:
      kind: Mesh
    default:
      outlierDetection:
        splitExternalAndLocalErrors: true
        detectors:
          failurePercentage:
            requestVolume: 10
            minimumHosts: 5
            threshold: 85

Examples

Basic circuit breaker for outbound traffic from web, to backend service

apiVersion: kuma.io/v1alpha1
kind: MeshCircuitBreaker
metadata:
  name: web-to-backend-circuit-breaker
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  to:
  - targetRef:
      kind: MeshService
      name: backend_kuma-demo_svc_8080
    default:
      connectionLimits:
        maxConnections: 2
        maxPendingRequests: 8
        maxRetries: 2
        maxRequests: 2

apiVersion: kuma.io/v1alpha1
kind: MeshCircuitBreaker
metadata:
  name: web-to-backend-circuit-breaker
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  to:
  - targetRef:
      kind: MeshService
      name: backend
      namespace: kuma-demo
      sectionName: http
    default:
      connectionLimits:
        maxConnections: 2
        maxPendingRequests: 8
        maxRetries: 2
        maxRequests: 2

type: MeshCircuitBreaker
name: web-to-backend-circuit-breaker
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  to:
  - targetRef:
      kind: MeshService
      name: backend
    default:
      connectionLimits:
        maxConnections: 2
        maxPendingRequests: 8
        maxRetries: 2
        maxRequests: 2

type: MeshCircuitBreaker
name: web-to-backend-circuit-breaker
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  to:
  - targetRef:
      kind: MeshService
      name: backend
      sectionName: http
    default:
      connectionLimits:
        maxConnections: 2
        maxPendingRequests: 8
        maxRetries: 2
        maxRequests: 2

Outlier detection for inbound traffic to backend service

apiVersion: kuma.io/v1alpha1
kind: MeshCircuitBreaker
metadata:
  name: backend-inbound-outlier-detection
  namespace: kuma-demo
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  rules:
  - default:
      outlierDetection:
        interval: 5s
        baseEjectionTime: 30s
        maxEjectionPercent: 20
        splitExternalAndLocalErrors: true
        detectors:
          totalFailures:
            consecutive: 10
          gatewayFailures:
            consecutive: 10
          localOriginFailures:
            consecutive: 10
          successRate:
            minimumHosts: 5
            requestVolume: 10
            standardDeviationFactor: 1.9
          failurePercentage:
            requestVolume: 10
            minimumHosts: 5
            threshold: 85

type: MeshCircuitBreaker
name: backend-inbound-outlier-detection
mesh: default
spec:
  targetRef:
    kind: Dataplane
    labels:
      app: web
  rules:
  - default:
      outlierDetection:
        interval: 5s
        baseEjectionTime: 30s
        maxEjectionPercent: 20
        splitExternalAndLocalErrors: true
        detectors:
          totalFailures:
            consecutive: 10
          gatewayFailures:
            consecutive: 10
          localOriginFailures:
            consecutive: 10
          successRate:
            minimumHosts: 5
            requestVolume: 10
            standardDeviationFactor: 1.9
          failurePercentage:
            requestVolume: 10
            minimumHosts: 5
            threshold: 85

See also

• MeshHealthCheck - Active health checking for services
• MeshTimeout - Configure timeouts that trigger circuit breakers
• MeshRetry - Retry logic for failed requests

All policy options