Efficacy probes

A fault that the cluster accepts and then quietly does nothing is the worst outcome Simian can produce. It is not a missing measurement — it is a wrong one. The eval result reads “the agent missed a network partition” when there was no partition, and it averages in with the real data points.

Probes close that gap. A fault carrying them is not reported as applied until Simian has observed it land.

Every fault kind Simian can inject dataplane-side carries one by default — the operator does not have to remember, and the planner cannot forget.

Efficacy, not outcome

Simian checks that the fault landed. It does not check what the fault caused.

• Efficacy — is the pod genuinely in CrashLoopBackOff? Does the NetworkPolicy exist and select the right pods? Simian’s business.
• Outcome — did latency rise, did the SLO burn, did the agent notice? The agent under test’s business, and what it is scored on.

Measuring outcome here would let the harness grade its own experiment, so the probe types are deliberately limited to things that answer “did it land”.

Two phases: SOT, then Settle

Probes hang off FaultManifest.probes and carry a mode. Simian schedules two of them; the other Litmus modes round-trip untouched.

The SOT phase exists because half the interesting proofs are differential. “The target does not answer” is not evidence of a partition against a workload that was not answering in the first place — it is the same vacuous pass as "expect_contains": "", one layer up. So an unreachability gate is only allowed to mean anything if a reachability check passed first, and Simian attaches the pair together.

Running SOT before Apply rather than after is deliberate: a fault whose preconditions are already broken should never have been injected, and rejecting it costs the arena nothing.

Writing a probe

{
  "engine": "chaos-mesh",
  "api_version": "chaos-mesh.org/v1alpha1",
  "resource_kind": "PodChaos",
  "duration": "2m",
  "spec": {"action": "pod-failure", "mode": "one",
           "selector": {"labelSelectors": {"app": "payments"}}},
  "targets": [{"namespace": "boutique", "name": "payments"}],
  "probes": [{
    "name": "payments crash-loops",
    "type": "k8s",
    "mode": "Settle",
    "spec": {
      "resource": "pods",
      "jsonpath": "{.items[*].status.containerStatuses[*].state.waiting.reason}",
      "expect_contains": "CrashLoopBackOff",
      "timeout": "90s",
      "interval": "2s"
    }
  }]
}

Submit it with simian chaos --manifest ./fault.json, or through the submit_manifest MCP tool.

The k8s probe type

kubectl get <resource> -n <namespace> -o jsonpath=<expr> in a loop. That correspondence is deliberate: settle conditions already written for kubectl port over without translation.

When no name is given the probe lists, and the jsonpath sees the whole list object — so {.items[*]...} works just as it does with kubectl.

expect_empty exists because some faults have no string to match on. A Service with no ready endpoints is an absence, and "expect_contains": "" would look like a check while passing unconditionally. Simian rejects a probe that declares no condition rather than accepting one that cannot fail.

expect_at_least exists because some faults are a repetition, and no string match can tell one apart from a single failure. A container’s lastState.terminated.reason reads Error from its first restart onwards, so a gate built on it passes about two seconds after apply — with nothing looping yet. "expect_at_least": 5 over {...restartCount} says what the ground truth actually claims.

Three rules follow from what the gates are for, and all three are enforced:

• Every value, not any. The expression yields one number per container, and a workload where one replica of two has got there is a fault that half landed.
• An empty render is not a pass. It is what a pod that does not exist yet produces.
• A bound of zero is rejected. Every counter Kubernetes writes is at least zero, so it is a check that passes before the object exists.

The http probe type

The k8s probe reads a field. Dataplane faults do not have one: a partition, a netem delay and an injected 503 leave nothing on any Kubernetes object to match. Chaos Mesh will happily report a NetworkChaos as Injected on a cluster whose datapath never traversed the qdisc it installed.

So the http probe dials the target pods directly — pod IP, no Service, no ingress — and asserts on what comes back, or on the fact that nothing does.

Every pod the selector resolves must satisfy every stated expectation, and a round that resolves no pods is a failure rather than a pass — “nothing to dial” is the same vacuous success expect_empty exists to refuse. expect_unreachable cannot be combined with any other expectation, because there is no response to assert on.

Two details are load-bearing and worth knowing about:

• Every attempt dials a fresh connection. A partition drops new flows; conntrack lets an already-established one through. A probe that reused the socket its SOT check opened would go on getting 200s in 1ms through a partition that really did land, and reject a working fault. This was not theoretical — it is exactly what the first live run did.
• Latency is measured through the body, not to the first byte. A delay fault can hold the body rather than the headers.

A request that never came back is slow

A min_latency gate has to decide what a request that timed out means. The literal reading — no response, no measurement, no pass — is wrong here, and wrong in the expensive direction: it takes a delay fault that landed harder than asked and reports it as a fault that did nothing. The SUT ate the chaos and the audit record says it did not.

That is not hypothetical. An injected 250ms delay on Online Boutique’s frontend produced a 3.9s page load, because one page fans out into a dozen internal round trips and each pays the delay twice. Any per-request deadline sized against the injected number will expire.

So a timeout satisfies min_latency, under four conditions, all of which have to hold:

• min_latency is the only expectation on the probe. A status code, a body match or a reachability check cannot be satisfied by a response that never arrived, and a timeout fails them.
• The request ran at least min_latency before giving up. A connection refused in 2ms is not slowness, it is a dead pod.
• The failure is genuinely a timeout — context.DeadlineExceeded or a net.Error reporting Timeout(). A connection reset at 120ms is a broken target, not a slow one, and is not counted.
• request_timeout >= min_latency, so the deadline could not have fired before the threshold was reachable.

The caller’s own deadline is not evidence either: a cancelled probe fails rather than passing on the way out.

The SOT half is what makes the inference safe. simian-fast-before has already proved this pod answers well inside the threshold, so “it stopped answering within the deadline” is a change Simian caused, not a property of the workload. The Expected string says so out loud — latency >= 125ms (or no response within 1s) — so the audit record never claims a measurement it does not have.

The logs probe type

The k8s probe reads a field and the http probe dials a port. Neither can see a fault that leaves every field on every object correct: a workload whose dependency has stopped answering is Ready, Available and endpointed, and says so only in what it writes about itself. The logs probe reads that — kubectl logs in a loop, over the pods the fault’s own selector resolves.

A round passes if any pod matches, and the Observed string names which one — checkout-api-x-5bcf5f5cd5-s7btf: level=error msg="upstream request failed" …. It reads at most ten pods and 256KiB per pod, and one pod whose log cannot be read does not sink a round the others can answer; an error surfaces only when nothing was readable. Unlike the http probe’s lister it does not filter on phase or pod IP — a Pending or terminated pod still has a log, and is often the one worth reading.

There is deliberately no expect_empty, and it is the one asymmetry with the k8s probe. “The log does not say X” is satisfied by a container that never started, by a pod that was never created, and by a typo in the expectation — the vacuous pass with no way left to tell it from a real one. For the same reason an empty or whitespace-only expect_contains is rejected at parse time rather than at poll time: every log contains "", and almost every log contains " ".

The probe needs pods/log on the arena Role, which simian arena create, the manifests and the Helm chart already grant.

Why not cmd

An exec-into-the-pod probe would read tc -s qdisc and settle a whole class of question directly. It is deliberately not implemented: it needs pods/exec on the chaos controller’s ServiceAccount, which is a real blast-radius increase and deserves its own decision rather than arriving as a side effect of a probe type. Everything the default gates need is reachable over HTTP.

Default gates

Probes only help if they are present, and the component that writes the manifest is the component being evaluated. A planner that can omit its own gate will eventually omit it. So the gates are not a manifest concern: the executor attaches them, per fault kind, from a table the manifest does not get a vote on.

The table is keyed by (engine, kind), not by cluster. The Chaos Mesh catalog is derived from live CRD discovery, so anything hand-listed per installation would go stale the moment a cluster shipped a different set of CRDs. The gate attaches by kind, which means a CRD Simian has never seen on this cluster before still arrives gated. Each entry’s description also rides along on the catalog as efficacy_gate, so a planner can tell a verified fault kind from an unverified one before it picks.

A manifest overrides a default only by naming it — declaring a probe called simian-partitioned replaces that one and leaves the rest. Everything else is additive, so a manifest cannot dissolve its gate by declaring something unrelated. The names are reserved and prefixed: simian-reachable-before, simian-partitioned, simian-fast-before, simian-delayed, simian-envoy-runtime, simian-image-pull-failed, simian-crash-looping, simian-oom-killed, simian-unschedulable, simian-job-failed, simian-workload-ready, simian-workload-rolled-out, simian-no-endpoints, simian-claim-pending, simian-endpoints-ready, simian-dependency-stalled, simian-restarts-climbing, simian-crash-loop-visible.

A synthesized fault has no SOT half, and that is not an oversight

Every kube-state gate is Settle-only. Every dataplane gate above needs a precheck because its Settle assertion is differential: “the target does not answer” proves nothing about a workload that was not answering beforehand.

A synthesized workload did not exist before Apply. “These pods are in ImagePullBackOff” cannot be a pre-existing condition, because there was no pre-existing anything — there is nothing for a precheck to rule out. When the engine’s mutate mode lands, which patches a workload that was already running, the SOT half comes back with it.

That the gate can name pods that do not exist yet is why the synthesized workload’s name is derived from the fault UID rather than from a fresh random value: the executor builds the probe before it calls Apply, so both sides have to compute the same name from the manifest alone.

Each gate asserts the narrowest stable field it can, and the second word is the one that cost a debugging session. The obvious gate for ContainerExitLoop is state.waiting.reason == CrashLoopBackOff, and it is a coin flip: a container that exits immediately spends almost all of its time with the previous termination showing in state.terminated, and the kubelet flips to waiting: CrashLoopBackOff only in a narrow window around each restart decision. Measured on GKE 1.36 that window caught one poll in six — one run passed in 6.5s, the next missed it across 44 polls and rolled back a fault that had visibly landed. Both crash-loop kinds now read lastState.terminated.reason, which is stable from the first restart on: Error for a non-zero exit, OOMKilled for a memory kill.

Stable is not the same as landed

That fix bought a second bug, and the deterministic subject found it. Stable from the first restart on means the gate passes at the first restart — 2.3 seconds after apply on GKE 1.36, with the container having died exactly once. The harness then handed the scenario to k8s-lookout and scored it on whether it could see a crash loop, which at that point had not happened. Recall 0.00 on a fault reported as landed. Nothing was wrong with the detector.

So both crash-loop kinds carry a second gate, simian-restarts-climbing, which waits for every container to reach restartCount >= 5. The number is the kubelet’s backoff schedule read off: 10s, 20s, 40s, 80s, 160s puts the fifth restart about 150 seconds in, and from there the pod spends 160 seconds of every 160 in waiting: CrashLoopBackOff. Before then the backoff windows are shorter than the gaps between them, which is the real reason polling for the waiting reason earlier was a coin flip — the state is not yet where the pod lives.

That threshold is where a crash loop becomes continuously observable by anything. It is not tuned to any subject’s thresholds, and it should not be: a gate calibrated against one detector’s rules would score that detector on Simian’s timing rather than on its judgement.

The cost is real and is paid on purpose. A crash-loop scenario cannot settle in under two and a half minutes, its lease has to cover that twice over, and the parity pack’s crash-loop and cascade scenarios grew to 10m and 12m. A gate that passes early is not faster; it is wrong sooner.

Landed is not the same as steady

Even then the harness asked its question about a second after the restart counter ticked, which is the one moment a loop looks least like one. Three consecutive live runs of the same scenario against the same cluster scored severity 0.67, 1.00, 1.00 — full recall every time, and a namespace verdict that turned on which side of a restart the scan happened to land. Two runs of one scenario that disagree are a harness bug by definition: with a deterministic subject there is nothing else left to vary. That is the property k8s-lookout is in the suite to provide, and no agent subject could have shown it.

So both kinds end on a third gate, simian-crash-loop-visible, which is state.waiting.reason contains CrashLoopBackOff — the criterion the first two were written to avoid. It is safe here and nowhere earlier, and the difference is the gate before it: below five restarts the backoff window is 10s to 80s and sampling inside one that short caught the state one poll in six, while from the fifth restart the window is 160s and the pod both enters CrashLoopBackOff and stays there.

It is not instant. Measured across three consecutive runs, the gate passed in 65.6s, 78.4s and 82.4s — 32 to 40 polls, each well inside one 160s window and nowhere near its start. Its timeout is set above the slowest of those with room to spare rather than trimmed to them.

It is not there to prove the fault landed; the two gates before it did that. It is there to hand the subject a steady state instead of a transient. A test in pkg/catalog enforces the ordering for every kind: nothing may read state.waiting.reason unless a restart-count gate has already run, with ImageUnresolvable the one exception — its container never starts, so there is no restart cycle to race.

ImageUnresolvable is the one kind that may read state.waiting.reason outright: its container never starts, so there is no restart cycle to race against and the pod stays in ImagePullBackOff. Unschedulable reads the PodScheduled condition’s reason rather than phase == Pending, which would also pass while an image is still pulling.

When the evidence is an absence, something else has to prove it is not vacuous

SelectorDrift breaks a Service by pointing it past its own pods, so the state that proves the fault landed is no endpoint addresses — and an empty read is also what a namespace where nothing has been created yet produces. Gated on that alone, the probe would pass in the moment before the workload existed and report a fault that never landed as verified. That is the exact failure the gates exist to prevent, arriving through the gate itself.

So the kinds whose evidence is an absence get two probes, and Settle probes run in order, stopping at the first that does not pass. simian-workload-ready runs first and only passes once the pods report Ready; simian-no-endpoints runs second, against a namespace the first probe has just proved is populated. A test in pkg/catalog refuses any gate that puts an expect_empty probe first.

The window between the two is narrow enough to measure: on GKE 1.36 a Service whose selector does match has its addresses published by the time kubectl wait --for=condition=Ready returns, and the second probe polls ~100ms after the first passes.

BackendCrashLoop is the same table read from the other end, and the pair is the reason both kinds exist. Its Service selects its pods correctly and its pods are crash-looping, so the EndpointSlice lists their addresses with conditions.ready: false — where SelectorDrift’s lists no addresses at all. Neither gate can pass against the other’s fault: an emptiness assertion fails against a populated slice, and a read of conditions.ready renders nothing when there are no endpoints to have conditions. Measured side by side on GKE 1.36 in one namespace:

SLICE                       ADDRS                           READY
orders-api-ymjvzs12-p5gxn   [10.13.128.55],[10.13.128.58]   false,false   # BackendCrashLoop
storefront-sr92vf56-qtm4t   <none>                          <none>        # SelectorDrift

Its first probe is the crash loop rather than a readiness assertion, because here the pods being broken is the fault and not the thing that keeps the second probe honest. The order still matters, for a different reason: this is the kind a scoring run uses to ask whether a subject found the root cause or stopped at the symptom, so the gate proves them in that order too.

One thing this kind needs that no gate could give it: the crash-looping container carries a readiness probe that can never succeed. Without it the kubelet calls the container Ready for as long as it is Running, which on a container that exits in 200ms is 200ms out of every restart — measured at 2 ready reads in the first 45 polls on GKE. The gate would ride that out, since it polls until it sees what it wants. A subject triaging the namespace in those first ninety seconds would not, and a symptom that is intermittently absent is one a correct diagnosis can be graded wrong against.

UnboundClaim is paired for a different reason. The pod it blocks reports Unschedulable, which the scheduler also writes for taints, node selectors and genuine resource shortage — so the gate reads the claim’s own Pending phase first. The cause before the symptom.

DependencyStall gets three probes, and its evidence is not an absence at all — it is the opposite problem. A gate that only grepped the log would pass just as happily against a crash-looping workload that happened to print the line once on its way down, which is the wrong fault entirely. simian-workload-ready, simian-workload-rolled-out and simian-endpoints-ready run first, and what they buy is the word only: with both green, the log gate means “and only the log is wrong”, which is the whole claim this kind makes. They are also the cheapest possible check that the fault is the one advertised, since a stall that accidentally broke readiness would fail its own gate rather than land as a workload that merely looks healthy.

PDBGridlock is gated on a jsonpath filter rather than on a rendered value, and the difference matters. The obvious spelling — render status.disruptionsAllowed and expect "0" — is a substring match against a decimal number, so it passes just as happily against 10 and 20. The gate instead selects budgets whose disruptionsAllowed is exactly zero and renders their name, so the value either matches exactly or the gate sees nothing at all. (This is also why the gate reads the dynamic client’s int64: client-go’s jsonpath refuses to compare a float64 against an integer literal.)

RolloutStuck is the one kind gated on a reason rather than on a condition’s presence, because Progressing is a condition every healthy Deployment also carries — a gate that matched on the type would pass against a rollout that completed perfectly. Its second probe asserts the old revision is still fully available, which is the half that makes the fault what it claims to be: a deploy that broke while the previous pods kept serving. A wedged rollout in front of nothing is just an outage.

CertExpiry’s gate is honestly weaker than its fault, and says so. No probe type here can parse a certificate, so the gate proves the Secret landed and a pod mounted it — the base64 of the PEM header, checked as a prefix. That the certificate actually expires when the spec says is proved in the driver’s unit tests against the generated DER. Gating on “a certificate is present” and claiming to have verified the expiry would be exactly the vacuous pass this document exists to refuse; gating on the arithmetic elsewhere and saying which half is which is the honest version.

NoOp, the control, is gated on its workload being healthy. A control needs a gate as much as a fault does: without one it would “inject” successfully against a cluster too broken to run anything, and the subject’s correct report of nothing wrong would be scored as a correct answer rather than as the vacuous pass it is.

When the fault is an age, the gate has to hold

Unschedulable is the one kind in this table whose fault is a duration rather than a state. Every other kind has a moment where the thing became true — the image failed to pull, the container was OOM-killed. “No node can place this pod” is true two seconds after Apply, and two seconds of Pending is also exactly what a scheduler working through a queue looks like, or an autoscaler about to add a node and heal the fault mid-experiment.

So this gate takes a dwell: the condition has to keep holding, poll after poll, before the probe agrees. The clock starts when the condition first holds and restarts if it stops — a flicker is not a hold, and a read that failed is not evidence the state was still there. The prober rejects a dwell that fills its whole timeout, since the hold cannot start before the condition does.

The default is 90 seconds, which is this repo’s own line: a pod Pending for two seconds is a slow scheduler and a pod Pending for ninety is a fault. It is deliberately not chosen to clear any particular observer’s grace period, and it does not clear every one of them — k8s-lookout’s --pending-age defaults to five minutes. A gate tuned to one subject’s threshold would score that subject on Simian’s clock rather than on its judgement, which is the same argument that sets the crash-loop restart count.

A scenario that needs to outlast a longer grace says so itself, with spec.pending_dwell, and the gate’s budget follows it up. The lookout parity pack’s pending scenario is the one place that is the right call: a pack whose purpose is to reproduce another project’s own examples has to be visible to that project’s detector, or it measures the grace period instead of the diagnosis. The whole hold comes out of the fault’s lease, so raising it means raising the duration too.

Healthy is two clocks, and the second one trails

Every kind whose workload is supposed to be healthy — NoOp, DependencyStall, PDBGridlock, CertExpiry — waits twice: simian-workload-ready on the pod’s Ready condition, then simian-workload-rolled-out on the Deployment’s status.readyReplicas. Those two fields are written by different controllers. The kubelet flips the pod condition; the deployment controller then observes the pod and updates the workload’s status. In between, the pods are Ready and the Deployment says zero of one is.

The window is invisible on a fast machine, which is how it survived. It showed up on a two-core GitHub runner, where lookout-healthy — the control, the one scenario whose entire score is that there is nothing to find — dropped from severity 1.00 to 0.33 because the subject was asked inside the gap and reported Deployment/… RolloutIncomplete. Locally the same scenario scored 1.00 twice in a row. A control that flakes with machine speed makes every hallucinated_fault number in the pack untrustworthy.

The gate reads readyReplicas and not updatedReplicas, because these workloads are synthesized at a single revision and nothing can be ready without being updated. The count comes from the manifest, computed the same way the driver computes it before Apply creates anything — the field is omitempty, so a Deployment whose status has not been written yet renders nothing, and expect_at_least over an empty render fails rather than passes.

It always runs after the pod gate, never instead of it. Alone it is the weaker assertion: readyReplicas counts pods some other controller has already decided are Ready, so it says nothing the pod condition did not say first, later.

Defaults are on unless the operator turns them off:

simian serve --default-efficacy-probes=false

Size a delay against the workload, not against the drama

The delay gate is a 4× signal-to-noise requirement in disguise. SOT demands the target answer in under latency/4; Settle demands at least latency/2. That ratio is what stops “the app was always slow” from passing as “the fault landed”, and it means the injected number has to be chosen relative to the target’s own baseline.

Online Boutique’s frontend answers in 40–240ms depending on what its downstreams are doing, so a 250ms delay puts the SOT threshold at 62.5ms — inside the noise, and the precheck passes or fails on which sample it happens to take. Injecting 2s moves the threshold to 500ms, clear of it. A precheck that keeps failing on a fault you believe in usually means the delay is too small for the workload, not that the gate is broken.

The envoy gate reads the value back

envoy-fault’s Apply is a POST to the sidecar’s /runtime_modify. A 200 from that endpoint says the request was accepted; it does not say the filter is live. The gate does GET /runtime on the admin port and asserts on the value Envoy reports it is actually running with:

jsonpath: {.entries['fault\.http\.delay\.fixed_delay_percent'].final_value}
expect_equals: "100"

which is the difference between “we asked” and “it happened”. After clear_fault, the same key reads "0".

What is deliberately left ungated

A gate that fires on a working fault is worse than no gate: it teaches the operator to disable gates. Each of these has no default probe, on purpose.

Ungated is not silently ungated. The executor.validated event lists the probes Simian attached under default_probes; no such key means the fault ran unverified, which is a fact about the data point rather than a footnote.

What happens on failure

Apply returns a typed *simian.ExecutorError with stage probe and reason probe-failed, naming the probe and quoting what it last saw:

executor[probe:probe-failed]: probe "payments crash-loops" (k8s) never passed
in 1m30s (45 polls): wanted "CrashLoopBackOff" in output, last saw "Running"

That is deliberately distinguishable from driver-failed. A driver failure means the cluster rejected the fault; a probe failure means the cluster accepted it and nothing happened — a different bug with a different fix.

The fault is then backed out: an unverified fault is not a valid experiment, and leaving it running would contaminate the next one while the caller, holding an error and no UID, has no way to clear it. If the rollback itself fails the lease is deliberately left in place so the reaper collects it at the deadline, and the audit record says so with left_to_reaper: true.

A failing SOT probe is the cheaper case, and reads differently:

executor[precheck:precheck-failed]: probe "simian-reachable-before" (http)
never passed in 33.05s (7 polls)

Stage precheck, reason precheck-failed, and nothing to roll back — the driver was never called, no object was created, no lease was taken. The fault is simply refused.

A manifest carrying probes submitted to a controller with no prober wired in is rejected with probe-not-configured rather than applied unverified. Skipping a gate that cannot run would mark unverified faults verified, which is the whole failure being prevented.

The audit trail

One event per probe, pass or fail, carrying the observed value: fault.precheck for SOT, fault.efficacy for Settle.

{"event":"fault.efficacy","fault_uid":"f-01M1E9...","payload":{
  "probe":"payments crash-loops","type":"k8s","mode":"Settle","passed":true,
  "observed":"CrashLoopBackOff","expected":"\"CrashLoopBackOff\" in output",
  "attempts":7,"elapsed_ms":13840}}

The observed value, not just the boolean, is the point — a pass/fail flag cannot be debugged once the arena is gone. For a dataplane gate it is the whole record of the experiment:

fault.precheck  simian-reachable-before  passed  "payments-0 (http://10.244.1.7:8080/): 200 in 2ms"
fault.efficacy  simian-partitioned       passed  "payments-0 (http://10.244.1.7:8080/): unreachable after 3s: context deadline exceeded"

An eval result whose fault carries no passing fault.efficacy record is not a data point. It is a harness bug, and downstream consumers should report it as such rather than average it in.

Timing

The lease deadline is not extended to cover the settle wait. A fault that takes 30s to manifest spends 30s of its own duration doing so.

This keeps Simian’s lease and the engine’s own server-side spec.duration in agreement — Chaos Mesh starts its clock at apply time regardless of what Simian is waiting for, and letting the two drift apart would leave the fault outliving its lease. The cost is visible as elapsed_ms on the efficacy event rather than hidden as a discrepancy.