GKE bring-up

Getting started gets you a fault on whatever cluster your kubeconfig names. This page is about the first run against a real GKE cluster: what to check before you start, what changes on Dataplane V2, and how to establish — by measurement, not by assumption — which fault kinds land there.

Every command below was run on 2026-09-04 against a Standard GKE cluster in us-central1:

GKE          1.36.3-gke.1537000 (RAPID)
dataplane    ADVANCED_DATAPATH (Dataplane V2 — Cilium v1.19.4-gke.49 via anetd)
nodes        4 × COS, kernel 6.12.94+, containerd 2.2.5
chaos-mesh   v2.8.2

The numbers quoted are the ones that run produced. Yours will differ; the point is the shape of the check, not the figure.

Before you start

# Dataplane: ADVANCED_DATAPATH means Dataplane V2. Anything else is kube-proxy
# + your chosen CNI, and Chaos Mesh behaves the way its own docs describe.
gcloud container clusters describe "$CLUSTER" --region "$REGION" \
  --format="value(networkConfig.datapathProvider,autopilot.enabled)"

# Chaos Mesh, with chaos-daemon on every node you plan to inject into.
kubectl -n chaos-mesh get pods
kubectl get crd -o name | grep chaos-mesh.org | wc -l

Two GKE-specific constraints worth knowing before you spend cluster time:

• Autopilot will not run Chaos Mesh’s chaos-daemon. It needs a privileged pod with host PID and the container runtime socket. On Autopilot, use the network-policy and envoy-fault engines, both of which are ordinary workloads and API calls.

• Node-tier faults do more than you think on a managed cluster. KernelChaos and PhysicalMachineChaos act on the node, and a GKE node pool with autorepair will react to what you did to it. Start with --permitted-tiers namespace, which is the fence that keeps the executor from applying them at all:

  bin/simian serve --permitted-tiers namespace
  

  An unrecognised tier name stops the controller starting rather than falling back to the default, so a typo here fails loudly. See Helm values.

Run the controller in-cluster, not on your laptop

Local simian serve against a remote cluster works — arena creation, fault apply and clear are all API calls — but the efficacy probes dial pod IPs directly. Off-cluster, that measures your workstation’s round trip to the VPC instead of an in-cluster one.

On the cluster above, the same frontend pod answered in ~40ms from inside the cluster and 170–330ms from a workstation. The SOT half of the delay gate requires the target to be faster than a quarter of the injected latency before it will accept the fault as gateable, so an off-cluster controller fails prechecks that an in-cluster one passes, on a fault that is perfectly fine.

Use local serve to get your first fault out. Move to the Helm chart before you trust the verdicts.

Arena and SUT

bin/simian sut deploy --namespace simian-gke-1 --create-arena --use-controller

1m43s on a warm 4-node cluster: namespace, RBAC, twelve Online Boutique deployments, and a 30s steady-state baseline window. --use-controller routes the deploy through the running controller’s establish_baseline tool so its get_baseline cache is populated — without it the baseline is captured client-side and an agent asking the controller for it gets nothing. Envoy injection is off by default and should stay off for Online Boutique — its gRPC kubelet probes do not survive the current interception model (known limitations).

Find out what actually lands

This is the part that matters on GKE, and the reason the efficacy gates exist. Apply one fault per engine and read the fault.efficacy record rather than the exit code.

# Chaos Mesh, dataplane-independent: a pod kill is self-evident, so it has no gate.
bin/simian chaos --kind PodChaos --namespace simian-gke-1 --workload recommendationservice \
  --duration 60s \
  --spec '{"action":"pod-kill","mode":"one","selector":{"labelSelectors":{"app":"recommendationservice"}}}'

# Chaos Mesh, dataplane-dependent: this is the one to be suspicious of on DPv2.
bin/simian chaos --kind NetworkChaos --namespace simian-gke-1 --workload frontend \
  --duration 2m \
  --spec '{"action":"delay","mode":"all","selector":{"labelSelectors":{"app":"frontend"}},"delay":{"latency":"250ms","correlation":"0","jitter":"0ms"}}'

# Node-resource pressure: no dataplane involvement at all.
bin/simian chaos --kind StressChaos --namespace simian-gke-1 --workload productcatalogservice \
  --duration 90s \
  --spec '{"mode":"one","selector":{"labelSelectors":{"app":"productcatalogservice"}},"stressors":{"cpu":{"workers":2,"load":80}}}'

# Dataplane-independent partition: a plain NetworkPolicy, enforced by Cilium.
bin/simian chaos --engine network-policy --kind NetworkPolicy \
  --api-version networking.k8s.io/v1 \
  --namespace simian-gke-1 --workload cartservice --duration 90s \
  --spec '{"labelSelectors":{"app":"cartservice"},"directions":["ingress","egress"]}'

# Declarative state, no dataplane at all: a workload synthesized broken.
# Every field of the spec is optional; run each of the thirteen kinds. NoOp is the
# control — it synthesizes a *healthy* workload, and its gate passing is what
# tells you a later empty finding means "nothing was wrong" and not "the probe
# never worked here".
for kind in ImageUnresolvable ContainerExitLoop MemoryLimitSqueeze Unschedulable \
            JobFailure SelectorDrift BackendCrashLoop UnboundClaim DependencyStall \
            PDBGridlock CertExpiry NoOp; do
  bin/simian chaos --engine kube-state --kind "$kind" --api-version apps/v1 \
    --namespace simian-gke-1 --duration 4m
done

# RolloutStuck is held out of the loop because it needs a longer lease than the
# rest: Apply waits for a healthy revision to come up before wedging the next
# one, and the gate then waits out the Deployment's progress deadline.
bin/simian chaos --engine kube-state --kind RolloutStuck --api-version apps/v1 \
  --namespace simian-gke-1 --duration 10m

What the run above produced:

The bundle rows were measured a day later, 2026-09-05/06, on the same cluster and in a scratch namespace; everything above them came from the single run described at the top. Efficacy rate across them was 1.00.

The multi-gate kinds are worth a second look. SelectorDrift and UnboundClaim each prove their fault in two steps, in order, because the second step’s evidence is an absence — no endpoint addresses, no schedulable pod — and an absence on its own is also what you get when nothing was created at all. Settle probes run in sequence and stop at the first failure, so the first gate (“the workload is Ready”, “the claim exists and is Pending”) is what makes the second one mean something. See efficacy probes.

DependencyStall is the inverse case and takes three. Its first two gates assert the workload is healthy — Ready pods, ready endpoints — and only then does the third read the log. Without them a gate that just grepped the log would pass against a crash-looping pod that printed the line on its way down; with them, the finding means “and only the log is wrong”, which is the whole point of the kind. It is also the one kind where kubectl get pods, kubectl get svc and kubectl describe deploy all report a healthy namespace.

RolloutStuck is the one that only a live cluster could have taught. The first GKE run took the arena down: a container with no readiness probe is Ready for as long as it is running, and a container that exits after 200ms is running for 200ms — long enough that the kubelet reported the broken pods Ready, the Deployment controller declared the new ReplicaSet available and scaled the working revision to zero. A completed rollout does not un-complete, so the progress-deadline clock had already stopped when the pods began to crash, and the gate correctly refused the fault: ProgressDeadlineExceeded never arrived. Adding minReadySeconds fixed the outage but not the timing — the deadline now reset on every restart’s readiness flicker, so the condition first appeared at 159s and then flipped back to ReplicaSetUpdated. The kind now ships with a readiness probe on the broken revision that cannot pass, and the condition lands at 61.8s and stays. None of this is visible against a fake clientset, where status is whatever the test writes.

BackendCrashLoop and SelectorDrift are the pair worth running in the same namespace, because the only thing that separates them is the EndpointSlice:

SLICE                       ADDRS                           READY
orders-api-ymjvzs12-p5gxn   [10.13.128.55],[10.13.128.58]   false,false   # BackendCrashLoop
storefront-sr92vf56-qtm4t   <none>                          <none>        # SelectorDrift

Both Services answer nothing. One has no backends because its selector misses, the other because its backends are dead, and neither kind’s gate can pass against the other’s fault. BackendCrashLoop took the RolloutStuck lesson as a starting condition rather than learning it again: its container carries a readiness probe that cannot pass, so the endpoint condition is monotone. The counterfactual was measured rather than assumed — the same bundle without the probe read ready: true on 2 of the first 45 polls, both inside the first ninety seconds, which is exactly the window a subject would be triaging in.

NetworkChaos landing on Dataplane V2 contradicts what this project documented for the last year. It is a real measurement, not a correction of a mistake: the bypass was verified at the time on an older Cilium. Treat it as version-dependent and re-check per cluster — see known limitations. Reading the audit record is the check:

# passed: true means the probe saw the fault; the fault is live.
# passed: false means it was rolled back and nothing is running.
grep fault.efficacy controller.log | jq '.payload | {probe, passed, expected, observed}'

Unschedulable and Node Auto-Provisioning

The Unschedulable kind defaults to a CPU request of 1000, which looks absurd until you consider what a merely large request does on GKE. 64 CPU is unschedulable on today’s nodes but perfectly satisfiable by a bigger one, so the cluster autoscaler or Node Auto-Provisioning reads it as a provisioning signal: it adds a node, the pod schedules, and the fault heals partway through the experiment — with a machine on the bill. A request nothing can satisfy is declared unschedulable and left alone. On the run above the node count stayed at 4 and no TriggeredScaleUp event was emitted.

If your scenario is about placement rather than capacity, use node_selector instead; the two are mutually exclusive, and setting both would make the FailedScheduling message name whichever predicate the scheduler checked first.

CrashLoopBackOff is not a state you can poll for

The obvious gate for ContainerExitLoop is state.waiting.reason == CrashLoopBackOff. It passed in 6.5s on the first run here and then missed entirely on the second — 44 polls over 91s, every one of them reading empty, against a pod the event log showed was visibly backing off.

A container that exits immediately spends almost all of its time with the previous termination showing in state.terminated; the kubelet flips to waiting: CrashLoopBackOff only in a narrow window around each restart decision. Polling the same pod by hand every 10s caught it once in six:

last=[Error] restarts=[4] wait=[]
last=[Error] restarts=[4] wait=[]
last=[Error] restarts=[4] wait=[]
last=[Error] restarts=[4] wait=[]
last=[Error] restarts=[4] wait=[CrashLoopBackOff]
last=[Error] restarts=[5] wait=[]

lastState.terminated.reason is stable from the first restart on, so that is what the gate reads. Four consecutive runs against the same cluster passed in 2.2s, 4.4s, 2.2s and 2.3s — two or three polls each, no spread worth the name. If you write your own probe against a restarting workload, read lastState, not state.

…and stable from the first restart is too early to ask anybody

Read the poll trace above again, though, and the fix has a hole in it. Stable from the first restart on means the gate is satisfied at the first restart. On this cluster it passed 2.3s after apply, with the container having died exactly once — and the harness then handed the namespace to k8s-lookout and scored it on whether it could see a crash loop. It could not, because there was not one yet. Recall 0.00 against a fault the scorecard called landed.

That is a Simian bug of the precise kind the deterministic subject exists to surface, and no agent subject would have found it: with an agent you cannot tell a harness that asked too early from a subject that answered badly.

Both crash-loop kinds now carry a second gate on restartCount >= 5. The number is the backoff schedule read off — 10s, 20s, 40s, 80s, 160s — and the poll trace above is the evidence for it: the one window that caught CrashLoopBackOff was at restarts=4, which is the 80-second backoff. From the fifth restart the pod sits in waiting: CrashLoopBackOff for 160 seconds out of every 160, which is when a crash loop stops being a state you have to sample luckily.

Measured: the second gate passed in 165.5s over 79 polls, observing 5, and the same scenario’s recall went 0.00 → 1.00. It costs two and a half minutes per crash-loop scenario, and the parity pack’s crash-loop and cascade leases grew to 10m and 12m to cover it. A gate that passes early is not faster; it is wrong sooner.

That left one residual, and the deterministic subject found that too. The harness asks its question about a second after the counter ticks, which is the moment a loop looks least like one. Three consecutive runs scored severity 0.67, 1.00, 1.00: on the first, lookout caught the container between restarts and reported ExcessiveRestarts at warning; on the other two it saw CrashLoopBackOff and said critical. Identical inputs, identical subject, different number.

The tempting fix is to move the scenario’s ground truth to warning so the score comes out at 1.00. That would be calibrating a scenario against one subject’s thresholds, which is the opposite of what the pack is for — and it would leave the flapping in place, just below the resolution of this particular measure.

The actual fix is a third gate, state.waiting.reason contains CrashLoopBackOff, run after the restart count. That is the criterion this section spent two pages explaining you cannot poll for, and past the fifth restart it stops being one: the backoff window widens to 160s, and inside a window that long the pod reliably enters the state and stays. It took 65.6s, 78.4s and 82.4s across three runs — 32 to 40 polls, not the poll or two the first draft of this section predicted, so the timeout is set above the slowest with room rather than trimmed to it.

The first two gates prove the fault landed; the third makes sure the subject is looking at a steady state rather than a transient. With all three in place, three consecutive runs of the lookout pack’s three-scenario slice scored identically on every measure — recall, severity and hallucination all 1.00 — which is the property a deterministic subject is in the suite to establish.

What the MemoryLimitSqueeze shakedown found

The first implementation wrote into a medium: Memory emptyDir, on the correct theory that tmpfs pages are charged to the writing container’s cgroup. On GKE that produced StartError, not OOMKilled, at every limit tried — because a tmpfs emptyDir belongs to the pod, not the container. Its pages outlive the OOM kill, so the restarted container’s runc init is killed against a cgroup that is already full (container init was OOM-killed (memory limit too low?)) before any of the workload’s own code runs.

The gate caught it: probe "simian-oom-killed" never passed in 2m0s (57 polls): wanted "OOMKilled" in output, last saw "StartError", and the fault was rolled back rather than reported as applied. The kind now allocates anonymous memory, which is freed with the process, so every restart cycle reproduces the same clean OOMKilled. This is the failure mode the whole efficacy story exists for — without the gate it would have shipped as a fault that “worked”.

Sizing a delay so the gate can see it

The delay gate is deliberately conservative: SOT demands the target answer in under a quarter of the injected latency, and Settle demands at least half of it. That is a 4× signal-to-noise requirement, and it is what stops “the app was always slow” from being reported as a fault that landed.

Online Boutique’s frontend answers in 40–240ms depending on what its downstreams are doing. A 250ms delay against it is right at the edge — the SOT threshold is 62.5ms, so the precheck passes or fails on which sample it happens to take. Injecting 2s instead puts the SOT threshold at 500ms, clear of the noise.

Pick the injected latency relative to the workload’s own baseline, not to the number that sounds dramatic.

Cleaning up

bin/simian chaos --list-active
bin/simian chaos --clear f-<uid>
bin/simian sut destroy --namespace simian-gke-1 --with-arena

sut destroy refuses while Simian-managed faults are still leased. Leases also expire on their own, and the reaper sweeps every 30s by default, so a resource can outlive its deadline by up to one sweep.