Known limitations

This page is the canonical place to land if a fault “applied successfully” but didn’t appear to do anything, or if a SUT pod refuses to come up after enabling Envoy injection.

Chaos Mesh’s NetworkChaos may or may not work on GKE Dataplane V2 — measure it

Historically it did not. Chaos Mesh installs a netem qdisc on the pod’s eth0, which we verified is present at the kernel level; Dataplane V2 routed pod-to-pod traffic through eBPF maps that bypassed the tc qdisc layer, so the latency / loss never got applied and the Sent ... pkt counter on the qdisc stayed flat.

References: chaos-mesh#3302, cilium#19975.

That no longer reproduces on current GKE. Measured 2026-09-04 on a Standard cluster with datapathProvider: ADVANCED_DATAPATH, GKE 1.36.3-gke.1537000, Cilium v1.19.4-gke.49 (anetd), Chaos Mesh v2.8.2, COS nodes on kernel 6.12.94+:

Both landed. The delay figure is much larger than the injected 250ms because one Online Boutique page load fans out into a dozen internal round trips and each one pays the delay twice — a useful reminder that injected latency and observed latency are not the same number.

Treat this as version-dependent rather than settled either way. Cilium, the GKE release, and the node image all move, and the failure mode when it does not work is that the fault applies cleanly and does nothing. Simian’s default efficacy gate is what makes that safe to be uncertain about: on a cluster where the qdisc is bypassed, the delay or partition fails its Settle probe and is rolled back rather than reported as applied. To find out where your own cluster stands, apply one gated NetworkChaos and read the fault.efficacy audit record — the GKE bring-up page walks through exactly that.

Alternatives, still shipped and still useful — they do not depend on the dataplane at all, which is the point:

• The network-policy engine handles partition-style chaos. Works on DPv2.
• The envoy-fault engine handles HTTP-layer delay + abort via an injected Envoy sidecar. Works on DPv2 (subject to the limitation immediately below).
• The word silently no longer applies to any of it. NetworkChaos carries a default efficacy gate: a partition or delay that the qdisc never applied fails at inject time with a named probe and is rolled back, instead of being reported as a fault the agent under test then gets scored on. Same for NetworkPolicy on a CNI that does not enforce it.

For non-network chaos, PodChaos / StressChaos / TimeChaos / IOChaos / JVMChaos continue to work fine on Dataplane V2. See DPv2-compatible chaos engines for the full design rationale.

Envoy injection breaks gRPC kubelet probes

This is why the chart default is sutInjection.envoyFaults: false.

The current Envoy injection model intercepts ALL inbound TCP on the SUT-declared service ports via iptables PREROUTING REDIRECT to Envoy’s listener (port 15006). Envoy speaks HTTP at the L7 layer; it does not understand gRPC health-probe payloads.

For Online Boutique specifically, --no-envoy-faults=false (i.e. injection on) leaves 9 of 12 deployments crash-looping. Until probe rewriting (Istio’s pilot-agent style) or an outbound-only redirect mode is implemented, the rule of thumb is: only enable Envoy injection for SUTs whose probes you’ve audited as HTTP-only or TCP-only.

Cheap-escape-hatch: exclude probe ports from interception

If a workload’s probe port is different from its service port (e.g. a service on 8080 with a probe on 8081), you can exempt the probe port from the iptables redirect — kubelet’s probe traffic bypasses Envoy entirely while service traffic still goes through:

# SUT-wide: exclude port 8081 from interception for every Deployment
simian sut deploy --namespace boutique-1 --no-envoy-faults=false \
                  --envoy-exclude-port=8081

# Per-workload: only this Deployment exempts the listed ports
metadata:
  template:
    metadata:
      annotations:
        simian.chaos/envoy-exclude-ports: "8081,9090"

Or declare it on the SUT itself by implementing the EnvoyExcludePortsProvider interface (see pkg/sut/sut.go). The three layers merge.

Caveat: when probe port equals service port (Online Boutique’s situation for most workloads), exempting the port also disables fault injection against that workload. Trade-off: “no CrashLoopBackOff” vs “no fault injection on this workload.” For SUTs that need both, the full probe-rewriter (forthcoming) is the proper fix.

Workaround for arbitrary workloads

Deploy the SUT with the default (--no-envoy-faults=true), then hand-author a small Deployment whose probes you control (HTTP httpGet or TCP tcpSocket), add the Envoy sidecar + iptables init + bootstrap ConfigMap from pkg/sut/envoy/ to it, apply the EnvoyHttpDelay / EnvoyHttpAbort chaos against that pod’s label selector, and measure with curl through the Envoy listener port (15006). See Using the chaos engines for the simian chaos invocation pattern.

Chaos Mesh on GKE Standard with Node Auto-Provisioning

The chaos-daemon DaemonSet won’t land on NAP-provisioned nodes without (a) the right default-compute-class-non-daemonset label on the chaos-mesh namespace and (b) a cloud.google.com/compute-class:NoSchedule toleration. Without both, NetworkChaos / IOChaos reconciliation fails with cannot find daemonIP on node ....

This is an install-time concern, not a Simian bug — but it affects every chaos-mesh-using install on GKE NAP. Documented in the README’s “Known cluster-side gotchas” section.

NodeUnready cannot be synthesized on a cloud-managed cluster

The kube-state engine synthesizes objects rather than damaging existing ones, which is what makes it safe to point at a namespace you care about. The obvious way to extend that to a node-tier fault is a phantom Node: create a Node object with no kubelet behind it, let the node-lifecycle-controller notice nothing is heartbeating, and get Ready=Unknown / NodeStatusNeverUpdated after the grace period — non-destructive, and revertible by deleting the object.

It does not survive contact with a managed cluster. Measured on GKE 1.36.3-gke.1537000: a phantom Node, cordoned and tainted, was deleted within 10 seconds. The cloud-node-controller reconciles Node objects against the provider’s instance list and garbage-collects any node with no VM behind it, which is exactly the property the fault depends on.

So NodeUnready has no synthesize-mode implementation. The remaining mechanisms — cordon, drain, or an unreachable:NoExecute taint on a real node — all mutate infrastructure that was there before Simian arrived, which is mutate mode’s problem (revert-on-lease-expiry, #59) rather than this engine’s, and a genuine blast radius on a shared cluster rather than a bounded one. The kind is deliberately not in the catalog: an engine that cannot produce a fault should not list it.

IOChaos and DNSChaos need a writable root filesystem

Chaos Mesh IOChaos moves the target volume aside next to its mount point, and DNSChaos backs up /etc/resolv.conf. Both write to the container’s root filesystem, so both fail against a container with readOnlyRootFilesystem: true, which Online Boutique sets on every service. IOChaos also needs volumePath to be one of the container’s mount points exactly.

Simian reads the target pods before applying either kind and refuses an incompatible one with executor[precheck:target-incompatible], naming the pod, the container and the reason. Nothing is applied, so nothing is left behind a finalizer. The autonomous planner sees the same facts in its topology (readonly_rootfs=<containers>, mounts=<container>:<path>) and is told not to choose those kinds there. It did anyway on the 2026-10-01 trial, so a plan step the topology rules out now fails plan validation, and the planner’s corrective retry has to replace it before anything reaches the executor. A restarted controller rebuilds the last 24 hours of faults and refusals from its audit file (--audit-file) — including faults applied and then backed out because they did not inject or could not be verified — so the planner’s “recent faults” and “recently refused” lists and the loop’s repeated-refusal limit survive a restart. Targets without labels, or whose pods cannot be listed, are not judged, and the engine’s own injection check catches them as before.

Zone and node outages are arena-only pod failures

ZoneOutage and NodeOutage take the arena’s pods in a zone, or on a node, out of service by failing them in place. That is a real zone outage as the workloads see it for the first few minutes: replicas there stop answering, and whatever is not spread across zones goes fully down (driver.applied names those under fully_down). It is not the whole of one:

• Nothing is evicted or rescheduled. The pods stay bound to their nodes, so no replacement starts elsewhere and no PodDisruptionBudget or autoscaler is involved. A real outage gets there after tolerationSeconds (300s by default). Simian’s duration ceiling is five minutes, so the fault ends at about the point where Kubernetes would start evicting.
• Only the arena is affected. Other namespaces’ pods in the zone, the nodes themselves, and everything outside Kubernetes keep running: node readiness, the zone’s network, its persistent disks, its load-balancer backends. A cluster-wide cordon-and-drain of a node would be a node-tier fault, and is deferred.
• Pods created after the fault starts are not failed. A pod scheduled into the zone during the fault, by a rollout or an HPA, runs normally.
• Zones are read from node labels. ZoneOutage needs nodes labelled topology.kubernetes.io/zone, which every managed cloud cluster sets and kind does not (the e2e cluster labels its two workers zone-a and zone-b). On a single-zone cluster there is no second zone to survive an outage of the first, so there is nothing for the fault to test. Use NodeOutage there.

Autonomous LLM bias toward chaos-mesh

Without --hypothesis-hint, the LLM almost never picks the new network-policy or envoy-fault engines because chaos-mesh has 12+ catalog entries vs 1+2. Possible mitigations: (a) tier-policy filtering, (b) explicit per-engine “weight” in the catalog, (c) prompt rule that encourages cross-engine plans. Not blocking; the hypothesis-hint workaround is reliable.

Metrics provider deferred

get_metrics returns {"configured":false,"reason":"metrics provider not configured (deferred); see roadmap.md M3 risks."}. The hook is wired; a real provider lands in a later milestone.