Eval substrate plan

Status: draft, 2026-08-18. Not implemented. Re-sequences roadmap.md M5/M6 and supersedes the M5 simian evaluate stub. Companion to design.md.

1. Purpose

Simian exists to break clusters intelligently so that the go-steer SRE agents can be tested, evaluated, and demonstrated.

That sentence has consequences, and most of them cut against how the code is currently shaped:

Simian is the adversary. It is not the harness, it is not the detector, and it is not the judge of whether the cluster is healthy. It is the only component that knows, by construction, exactly what is wrong — and that knowledge is the ground truth everything downstream is scored against.

1.1 The distinction that governs the whole design

Simian must verify efficacy and must not verify outcome.

• Efficacy — did the fault actually land? Is the pod genuinely in CrashLoopBackOff, is the netem qdisc genuinely installed, is the partition genuinely dropping packets? Simian must know this with certainty.
• Outcome — did the fault matter? what broke downstream? what is the blast radius in user-visible terms? This is precisely what the agent under test is being scored on. If Simian computes it, the experiment is contaminated.

This is not a theoretical hazard. NetworkChaos on GKE Dataplane V2 is silently bypassed (M1 verification notes, dpv2-chaos-engines.md), and NetworkPolicy was long a no-op under kindnet. A fault that silently does nothing produces an eval result that reads “the agent missed a network partition” when there was no network partition. That is worse than no measurement, because it is a confident wrong number.

core-sre-agent already learned this the hard way. Its live harness bails with:

%d/%d fixtures never manifested — the harness is broken, not the agent

Efficacy confirmation is therefore Phase 1, ahead of everything else.

1.2 Corollary: Simian stays independent of the harness

The earlier suggestion to rebuild Simian’s loop on top of mast inverts here. mast is (or hosts) the system under test. An adversary that shares a prompt template, a model client, a tool-calling loop, or a K8s client library version with the subject can fail in a correlated way and produce an eval that passes for the wrong reason. Simian keeps its own loop.

This does not mean Simian’s loop stays as it is — see §7.

2. The customer already exists

core-sre-agent contains a hand-rolled miniature of what Simian should be:

• internal/faults/ — 1,540 LOC, 11 fixtures, each declaring injection YAML, settle conditions, and machine-checkable expected findings.
• cmd/sre-eval-live/ — creates a fresh kind cluster, injects fixtures, runs the agent, scores four evaluators, emits a transcript.

It was written because nothing else existed. Its design decisions are correct and are adopted wholesale below:

• The prompt names the task, never the fault. checkNamespace(ns) is the entire prompt. “A prompt that leaks the diagnosis turns tier 2 back into tier 1.”
• Settle conditions gate the agent. “A CrashLoopBackOff takes two restarts and a backoff to exist at all — an agent that looks too early correctly reports a healthy cluster and is then scored wrong.”
• Ground truth is the machine-stable triple Kind + Name + Reason, matched leniently: reason token sets (ImagePullBackOff and ErrImagePull are the same fault seconds apart), name-as-prefix (generated pod suffixes), and AlsoAcceptKinds (a finding about the Deployment stands in for one about the Pod). Prose is deliberately not graded.
• Root-cause is scored separately from recall. On fault-sessions, an agent reporting only the CrashLoop root scores the same recall as one reporting only the empty-Service symptom, and those are not equally good answers.
• A healthy control fixture exists (fault-none), because an agent that reports problems everywhere is not a good agent.
• Inject failure is recorded separately from agent failure.

2.1 Hard constraint: that code is not modified

core-sre-agent/internal/faults, cmd/sre-eval-live, and evals/ are frozen for the purposes of this work. They keep working exactly as they do today, on their own fixtures, with their own kind cluster lifecycle. The existing baselines stay re-runnable with the existing binary.

Simian therefore does not import them and is not imported by them. Parity is established by transcription plus a checked-in equivalence matrix (§4), not by a cross-repo Go dependency in either direction. Keeping the dependency graph empty is the same independence argument as §1.2 — the adversary must not share code with the subject’s repo.

3. What Simian can reproduce today: none of it

Simian’s four engines (chaos-mesh, network-policy, envoy-fault, and the unimplemented litmus) all perturb a running dataplane. Every one of the eleven fixtures is a declarative-state fault — an object that is wrong, or born wrong, in the API server.

Zero of eleven when this was written, and the single most useful fact in the document. It was not an indictment of the engines — Chaos Mesh is excellent at what it does — it was a statement that Simian had been building one half of the fault space and the eval rig needed the other half first. Deliverable A below is the answer, and eleven of eleven fixtures are reproducible today.

The last one to close is the one worth naming. cascade is the only fixture whose ground truth distinguishes a root from a symptom, so it is the only one that can ask whether a subject stopped at what a user noticed. Reproducing it took a thirteenth kind rather than a composition of two existing ones, because the point is precisely that the two findings are not independent.

Conversely, internal/faults structurally cannot produce anything Simian’s existing engines are good at. Applying YAML cannot create latency, packet loss, DNS blackholes, clock skew, IO stalls, or L7 aborts. Those are the failures where the symptom is not where the fault is, which is the class that separates a real diagnostic agent from a kubectl get pods wrapper — and there is currently no way to put one in front of the agents at all.

So the two halves are complementary, and Simian should own both.

4. Deliverable A — the kube-state engine

Shipped for synthesize mode and all thirteen kinds (#56, #57, #58, #61). mutate mode is #59 and the node-level kinds are the second half of #58; the driver rejects mode: mutate with an explanatory error rather than ignoring the field.

A fifth driver, EngineKubeState Engine = "kube-state", that produces declarative-state faults. It slots in behind the existing ChaosDriver interface and the existing executor chokepoint — no new privileged path, same validation, same lease, same reaper.

It operates in two modes:

Mode synthesize — apply a self-contained bundle that is born broken, into a namespace Simian created. Byte-for-byte the shape internal/faults uses. This is the parity mode: deterministic, no dependency on what is already running, and directly comparable to existing baselines.

Mode mutate — patch an existing healthy workload in the arena so that it becomes broken, recording the original for revert. This is the strategic mode: it is what lets Simian break Online Boutique, or a customer’s real namespace, rather than a synthetic busybox stand-in — and it is the mode topology-driven generation (§7) needs.

Both modes go through Apply/Clear and the lease registry, so mutate reverts on TTL expiry the same way a Chaos Mesh CR is deleted.

4.1 Fault kinds

NoOp is not a curiosity. It is how the eval measures false positives, and it must flow through the identical code path so that nothing about the run distinguishes it from a real fault.

DependencyStall is the hardest and the most valuable, and it shipped in synthesize mode with the logs probe type that gates it. Getting it right in mutate mode against a real SUT is the difference between a rig that grades kubectl transcription and one that grades diagnosis.

4.2 Reuse

pkg/sut/manager.go:330 (applyOne) already does dynamic-client server-side apply, and pkg/sut/envoy/inject.go already does deployment mutation with revert. The driver is assembly, not invention.

5. Deliverable B — efficacy confirmation

Shipped. Mode: "Settle", the k8s probe type, the Apply gate and the fault.efficacy audit event are implemented — see Efficacy probes for the user-facing reference. The http and logs types have since shipped too; cmd is deliberately not implemented and prometheus remains future work.

FaultManifest.Probes []ProbeSpec becomes the settle/efficacy mechanism.

type ProbeSpec struct {
    Name string         // "pods report an image pull failure"
    Type string         // k8s | cmd | http | prometheus
    Mode string         // Settle | SOT | EOT | Edge | Continuous | OnChaos
    Spec map[string]any // jsonpath, expect-contains / expect-empty, timeout
}

Adding Mode: "Settle" gives us internal/faults’ Condition with a superset of its expressiveness — the k8s type covers every existing fixture’s kubectl get -o jsonpath poll, and the cmd/http types cover dataplane faults where the proof is tc -s qdisc output or an observed latency percentile rather than an object field.

Behaviour:

• Apply returns only after every Settle probe passes, or fails with a new typed error when one times out.
• A new audit event fault.efficacy records pass/fail per probe with the observed value. This is the recording that keeps the dataset honest — an eval result whose fault has no passing efficacy record is not a data point, it is a harness bug, and it must be reported as such rather than averaged in.
• Every dataplane fault kind gets a probe. NetworkChaos on DPv2 then fails loudly at inject time instead of silently producing a false negative months later.

6. Deliverable C — scenarios, ground truth, and the runner

6.1 The Scenario type

A fixture is not one fault — multipleFaults is three and cascade is one fault with two graded findings. So ground truth attaches to a scenario, not a manifest:

type Scenario struct {
    ID        string            // stamped into every audit event; the join key
    Name      string            // "cascade", "latency-not-saturation"
    Prompt    string            // names the task, never the fault
    Substrate string            // a registered SUT to stand up first; usually empty
    Faults    []FaultManifest   // each carrying Settle probes
    Expect    []ExpectedFinding
    AlsoTrue  []string          // reason tokens the fault really produces, not required
    Severity  string            // scenario-level expected severity
    Source    string            // "pack:parity" | "generated:topology"
}

type ExpectedFinding struct {
    Kind            string
    Name            string   // matched as a prefix
    Reasons         []string // any one counts; empty means ungraded
    AlsoAcceptKinds []string
    MinSeverity     string
    Root            bool     // root cause vs downstream symptom
}

Substrate is the odd one out, and it is worth saying why it exists. Every scenario up to the dataplane pack synthesized its own subject matter: a kube-state fault is the Deployment it creates, so there is nothing to deploy first. That works for any symptom a Kubernetes object records — a pod that will not pull, a rollout that will not finish. It stops working the moment the symptom is a property of traffic, because nothing about “the p99 doubled” is stored in the API server and a fault cannot slow down a caller that does not exist.

The obvious workaround does not work, and the reason is worth recording: kube-state appends a suffix derived from the fault UID to every workload it creates, so a second fault in the same scenario cannot name, label-select or otherwise predict what the first one made. Substrate is the deterministic-name half — the same SUT registry, manifests and readiness wait that simian sut deploy uses — that the faults then attack. The harness stands it up after the arena and before the first fault, and takes it down before the arena goes.

ExpectedFinding is deliberately field-for-field faults.Want. Same matching semantics, same tolerances, so numbers from the two rigs are comparable without a translation argument.

simian.AuditEvent.ScenarioID was defined and stamped by the audit sink long before anything populated it. Scenario.ID populates it. That one field joins Simian’s audit log, k8s-lookout’s findings, and the agent’s transcript into a single correlatable record, which is the mechanical prerequisite for “see what we did and did not detect, did and did not fix.”

It is carried in the context, not passed as an argument. Simian emits audit events from around thirty call sites across the executor, the autonomous loop and the lease reaper; threading a parameter through all of them is churn that the next new call site silently forgets, and a missing join key is invisible until someone tries to score a run and finds a hole in it. audit.WithScenarioID puts the ID on the context and the sink stamps every event that does not already carry one. An event that sets the ID explicitly wins, which is how the lease reaper — outliving the context that applied the fault — still attributes a late expiry to the right scenario.

6.2 The packs

pkg/scenario/packs/parity/ — the eleven scenarios from §3, transcribed one file per scenario, embedded rather than read from a path so a scored run cannot be affected by the working directory. Both binaries take either kind of reference on --pack: a built-in name resolves to the embedded pack, anything with a path separator in it is read from disk, and ./parity is how you say you meant the directory.

Drift is caught in two halves, because there are two different things that can drift and only one of them is visible to CI:

• The pack moved and the record did not. testdata/upstream-fixtures.yaml records what the upstream fixtures said at transcription time — every expected kind, name, accepted reason, AlsoAcceptKinds, MinSeverity and root marker. TestParityPackMatchesTheUpstreamRecord compares the loaded pack against it on every run. This is where a comparability claim quietly stops being true: an extra accepted reason changes what the number means without changing anything looser than a deep comparison would notice.
• Upstream moved and the record did not. TestUpstreamRecordStillMatchesTheSource answers that one and is opt-in, via SIMIAN_UPSTREAM_FIXTURES=<path to fixtures.go>. It skips otherwise, because CI has no access to that repository and never will — a build dependency would be the exact coupling §2.1 exists to avoid. It reads the source as text, which is weaker than compiling against it and is the strongest check available with the dependency graph empty in both directions.

Four namespaces are renamed. fault-imagepull, fault-crashloop, fault-oomkill and fault-unschedulable name their own fault, the prompt quotes the namespace, and LintPrompt refuses a prompt that leaks its diagnosis. That makes those four scenarios harder here than upstream, which is a divergence and is recorded per fixture rather than inferred — a test asserts that every rename was one the linter would actually have refused, so a difference nobody had to make cannot survive as a comment. Upstream applies the same rule from cascade onward; the pack applies it to the earlier six too.

One deviation changes what is measured, and it is silent-failure. Upstream produces the failure — the container runs wget against a port nothing serves — precisely so the diagnosis is not readable out of the pod spec. DependencyStall writes a configured line that reaches the container through its environment, so a subject that reads the Deployment and never reads a log can still answer. Recall stays comparable; the read-path claim does not.

pkg/scenario/packs/lookout/ — k8s-lookout’s examples/scenarios/ is a third fixture corpus, ten scenarios each with inject/verify/revert. Eight map onto a kube-state kind:

The last three are worth having independently of lookout: pdb-gridlock and bad-rollout in particular are failure modes an operator meets constantly and that neither of the other two corpora contains.

Two do not map, and an earlier draft of this document got one of them wrong by calling failed-mount an approximate unboundVolume. It is not: a claim that will never bind and a volume referencing a ConfigMap that was never created are different diagnoses with different fixes, and treating them as the same would credit a subject for the wrong answer.

The lookout pack also carries a control the original set does not have. A pack without one cannot detect a subject that reports every failure mode everywhere — recall would be perfect and precision unmeasured — and the risk is higher for a watcher than for a triager, whose whole job is deciding that most of what it sees is fine. A test asserts every shipped pack has at least one.

Then pkg/scenario/packs/dataplane/ — the scenarios internal/faults cannot express. Five, chosen because each has a symptom that appears somewhere other than the fault:

stress-real and latency-not-saturation as a matched pair is the point: an agent that says “it’s slow, scale it up” scores well on one and badly on the other, and no fixture set that only contains one of them can tell.

Two things about the pair turned out to be load-bearing and neither was in the plan. The first is that the discrimination lives in the scoring vocabulary, not only in the fixtures: network-degradation and cpu-saturation are separate failure families, so a claim in one is charged as an invention in the other’s scenario, while HighLatency is in neither family and is therefore creditable as an observation and never chargeable as a diagnosis. A subject that reports only slowness scores the shared symptom in both halves and the cause in neither. pkg/eval/matched_pair_test.go asserts that rather than describing it.

The second is that neither of the two scenarios this table assigned to Envoy uses it. Envoy injection is off for this substrate because the sidecar breaks the gRPC kubelet probes, and both replacements turned out to be better fits anyway: netem degrades the path without touching the callee at all, and an HTTPChaos response replacement leaves the callee’s own health probes returning 200 on a different path, which is what makes the 5xx scenario’s misattribution trap work.

All five critical scenarios ended up producing the same object status — caller 0/2 Ready, callee 2/2 — which was not planned and is the pack’s best property. They are five causes behind one symptom, and the API server distinguishes none of them.

dns-blackhole-partial was the one that needed a substrate change rather than just a scenario file, and the reason is worth recording because it is a claim about live clusters and not only about this fixture. Chaos Mesh injects DNS chaos by rewriting a running pod’s /etc/resolv.conf; nginx reads that file once, at config load. So the first attempt applied cleanly, reported AllInjected=true, made nslookup fail inside the caller — and the caller went on serving 200 throughout. The substrate now proxies through a variable and reloads on a resolv.conf change, which is what a real nginx deployment does for the same reason. Any DNS finding an agent reports about a long-lived process is worth less than it looks for exactly this reason.

With it, the three network-shaped scenarios form a second matched set alongside the pair: a slow link, a severed link and an unresolvable name are one symptom and three families, and each of the three charges the other two’s diagnosis. The measurement that separates the last two is the cheapest in the pack — resolve the name, then dial the address.

6.3 cmd/simian-eval

A second binary, alongside cmd/simian. Rationale: cluster-lifecycle management, subject adapters, and scoring have no business linking into the operator binary that runs in-cluster with chaos RBAC, and mirroring cmd/sre-eval-live’s shape keeps the two rigs recognisable to the same reader.

simian-eval \
  --pack parity,lookout \
  --subject exec:./bin/sre-agent \
  --cluster kind \
  --out runs/2026-08-18/

Flow, per scenario:

1. Provision the arena (and SUT, for mutate-mode scenarios).
2. Inject via the normal executor path — same validation, same audit, same leases. The eval must not have a privileged back door, or it stops measuring the product.
3. Gate on efficacy probes. Failure here is InjectError, reported separately and never scored as an agent miss.
4. Hand the subject the prompt.
5. Collect the report; score.
6. Watch for external remediation (§6.5); revert; verify reverted.

What “fresh cluster” means in practice. Fresh arena per scenario, not a fresh cluster: a kind cluster takes minutes to stand up and a pack has dozens of scenarios. --cluster kind stands one throwaway cluster up for the whole run and deletes it afterwards, including on Ctrl-C; --cluster current (the default) uses the kubeconfig’s cluster and leaves it standing. The isolation scenarios actually need is namespace isolation, and that is enforced rather than assumed: a scenario holds every namespace it touches for its whole lifetime, so two scenarios never share one, and a control — which names no namespace — takes the whole cluster to itself. A control running beside a live fault would see real breakage, report it correctly, and be scored as having hallucinated it.

It destroys only what it created. A scenario naming a namespace that already exists gets that namespace annotated as an arena and left standing at the end, with a log line saying so. A rig that deletes namespaces it merely found is one bad scenario file away from deleting something that mattered.

Both artifacts, written as it goes. audit.log is opened before the cluster is touched, and the scorecard printed at the end is produced by reading the two files back through the same pkg/eval code simian evaluate uses (§6.6) — not from the runs still in memory. If the artifacts could not be scored tomorrow, the run finds out now, while the cluster is still there to look at. Every scenario the harness attempts emits an eval.scenario_started line, so a scenario that failed before any fault event existed is still in the log the join runs against: the offline read reports a harness failure rather than a corrupt pair of files.

Refusals that happen before a cluster is touched. An --only ID that is not in the pack (a typo that silently grades nothing is how a suite comes back green having measured nothing); a fault shorter than --subject-timeout, because the lease expires mid-investigation, the reaper clears it, and the harness records that disappearance as the subject having remediated a fault it never touched — --allow-short-faults accepts the measurement out loud.

6.4 The subject seam

Simian must not import mast, core-agent, or core-sre-agent.

type Subject interface {
    Name() string
    Investigate(ctx context.Context, prompt string) (Report, error)
}

Adapters:

• exec: ✅ — run a binary, read a JSON report on stdout. Covers core-sre-agent, mast workload bundles, claude -p, gemini-cli, and a shell script. Built first; it covers everything that matters.

• lookout: ✅ — run k8s-lookout’s health scan and translate its finding stream into a report. Not an exec: subject, because the detector already emits everything Simian grades and should not grow a Simian-shaped output mode to say so; the shape translation lives on this side of the process boundary. It reads the namespace out of the prompt, the same place an agent reads it from — a side channel would give the deterministic subject something no agent subject gets, and the comparison between those two rows is the point of running it.

• sre-agent: ✅ — run core-sre-agent’s one-shot assessment and read the report out of the transcript it writes. Not an exec: subject for a mechanical reason rather than a conceptual one: the agent’s schema.HealthReport is already the graded triple field-for-field, including the severity vocabulary, so no translation is needed at all — but it prints prose on stdout and puts the machine-readable answer in the file named by -out, which exec: would never find. The adapter names that file, runs the agent, and reads the one assessment back. It also refuses a spec that sets -namespace, -out or -repeat, each of which breaks the measurement quietly rather than loudly.

  Alone among the subjects, one investigation spends real model tokens against a real cluster, so the agent’s transcript — its tool trajectory, its delegations, its token usage — is kept in --out as transcript-<namespace>.json. The scorecard records what a subject answered and never why; for the detector that is the whole story, because the answer is a function of the cluster, and for an agent it is not. The interesting question about a 0.00 is which tools it called and what they returned, and that evidence exists exactly once.

• noop: ✅ — the null subject: reports nothing, ever. The zero-score floor a scorecard is read against, and the cheapest way to find out whether a pack actually manifests before an agent is pointed at it.

• http: — REST + SSE. Covers mast-web and, notably, ChaosBlade’s Blade AI, which turns a competitor into a benchmarkable subject.

• mcp: — for subjects that expose themselves as tools.

Report mirrors the machine-stable triple and nothing else. The exec adapter translates core-sre-agent’s schema.HealthReport into it; that translation is ~30 lines and lives on Simian’s side of the fence.

The exec: adapter hands the prompt over three ways at once — on stdin, in $SIMIAN_PROMPT, and substituted for a {prompt} placeholder in the argv if one is there — so a subject can be a Go binary, a shell one-liner or an agent CLI without a wrapper script in between. It reads the last JSON object on stdout, which is what lets a subject narrate: agents print reasoning as they go, and requiring clean stdout would mean grading whichever tool happened to be quiet. A subject that exits non-zero, prints nothing parseable, or runs past --subject-timeout is a SubjectError — scored as a hard zero, never skipped, because a subject must not be able to improve its mean by crashing.

6.4.1 The prompt is not the only channel

LintPrompt keeps the diagnosis out of the question. The cluster is the other way it can get out, and that one has no linter.

Every object a kube-state fault synthesizes is stamped by the driver so the reaper can find it if the process dies holding a lease. One of those stamps used to be simian.chaos/kind, whose value is the fault kind — a Deployment wearing simian.chaos/kind=ContainerExitLoop while a subject is asked what is wrong with it. It was there for an operator with kubectl get deploy -L, no code ever read it back, and the first LLM subject pointed at the rig read it and reported it as a finding. It is gone; the fault UID joins to the audit log instead.

What remains — simian.chaos/managed, simian.chaos/bundle, simian.chaos/fault-uid, and the simian.chaos/expires-at annotation — says that Simian is here and never what it did. That much is irreducible. The reaper finds orphans by selecting on managed=true, and a mark a controller can select on is a mark a subject can read; the alternative is a rig that leaks faults into a cluster it cannot clean up, which is a worse trade than a subject knowing it is in an experiment. Two consequences worth stating plainly:

• A subject can tell it is inside a chaos experiment, and an agent that says so is not hallucinating — it is reporting a true property of the cluster it was pointed at. Whether that biases its diagnosis toward the injected fault is not something the current scorecard can measure.
• The line the rig does hold is that no label, annotation or object name may name the fault. TestApplyNeverWritesTheDiagnosisOntoTheObjects asserts it for every kind, so the next convenience label cannot quietly cross it.

Pod templates are cleaner still: none of these labels reach them, because pods are what a subject inspects first.

6.5 Scoring

Deliberately the same four measures core-sre-agent/evals uses, so the numbers are comparable, plus three the adversary is uniquely positioned to provide:

The fourth measure was drafted as general precision — “findings outside ground truth” — and shipped as something narrower, because general precision is the wrong metric here. A scenario’s manifests are minimal: no liveness probes, no PDBs, no resource limits on the deliberately-broken workloads. A subject that notes those is correct, and precision would mark it down for thoroughness, which pushes us to prompt subjects to say less. So exactly one class of finding is charged: claiming one of the concrete failure modes the fault kinds know how to inject, in a scenario that did not inject it. Calling a Pending pod a CrashLoopBackOff is a misdiagnosis; noting that it also has no resource limits is not. That is what makes a healthy control cost something, and it matches what core-sre-agent actually scores.

The vocabulary of concrete failure modes is ported from the agent rig rather than re-derived, exclusions included. Most of those exclusions were paid for with a live run that scored an honest report as an invention: Unschedulable and FailedScheduling say scheduling failed and not why, NotReady is written about pods, containers and nodes alike, and DeadlineExceeded is a substring of what the Deployment controller writes on a stalled rollout. Matching is on the exact normalized token in consequence — a bare family member must not annex every longer token containing it.

The set of injected failure modes is read out of each scenario’s own ground truth rather than a table in the scorer, so a new scenario cannot forget to register itself. That read was originally over the expectations alone, and it conflated two different things: what a correct report must contain, and what it may contain without being wrong. A stuck rollout is diagnosed at Deployment altitude, and that is the only finding worth demanding — but the reason the new revision is not ready is that its container exits non-zero, so there is also a pod in CrashLoopBackOff. bad-rollout scored hallucinated_fault 0.50 against a detector that was right twice.

Promoting the pod to an expectation would have fixed precision by breaking recall: the subject that reports only the Deployment gives the better answer and would have scored 0.50 for it. So a scenario carries a second, weaker list, also_true — reason tokens the fault mechanically produces, which suppress the hallucination charge and contribute nothing to recall. It is reason tokens rather than objects, because the claim being licensed is “this namespace contains a crash loop”; which pod is the subject’s business.

Every entry is a claim the subject can no longer be charged for, so the list has to stay short, and two rules keep it honest. A control may not have one at all — a scenario that injects nothing has no consequences, and an exemption on the one scenario that exists to measure precision would disarm the measure. And an entry that names no failure family is rejected rather than ignored, because the failure mode is otherwise invisible: the author wrote it to license a claim, the lookup misses, and the claim is charged anyway.

The exemption is unconditional, which means it is a judgement about the scenario. oom declines the identical one deliberately: a container killed for exceeding its limit is also technically backing off restarts, but there the crash-loop token is offered instead of the diagnosis, and a report that says CrashLoopBackOff has named the wrong fix. In bad-rollout it is offered alongside one. A subject that reports only the crash loop there is charged by recall and not by precision, which is the correct number of times to charge one mistake.

Time-to-remediate falls out for free and is worth saying plainly: the lease reaper, built to stop Simian leaking faults, becomes a measuring instrument the moment the subject is allowed to write.

Scoring is pure. Nothing in pkg/eval touches a cluster, a clock or a network: a Run carries the expectations, the report, the timestamps and whether the fault landed, so the same inputs always produce the same scores. That is what lets simian evaluate (§6.6) reproduce a live run’s numbers offline, hours later, from artifacts alone.

6.6 simian evaluate

The M5 stub already describes itself as “Drive an external evaluation harness against scenario records” — the intent was right, it just predates knowing what the harness and the records were. It becomes the offline scorer: read an audit log plus a subject report, emit the scorecard. No cluster lifecycle, no subject execution. simian-eval orchestrates and calls the same pkg/eval code. This is what makes the scorecard usable in kube-agent-demo-e2e, where the clusters are long-lived and nobody is going to run a kind harness.

simian evaluate --pack parity --audit run.log --report agent.json

Two artifacts, split along the line of who observed what. The audit log is Simian’s record of breaking things: which faults applied, whether their efficacy gates passed, and when. None of it can be taken from the subject — the whole point of the gate is that the harness does not take the subject’s word for the cluster’s state. The report is the subject’s side: what it found, when the report came back, and on a write-enabled run, when the fault was observed gone. They join on the ScenarioID that pkg/audit stamps onto every event, which is the reason the reconstruction is possible at all.

Absence of evidence is not manifestation. A scenario counts as manifested only when every one of its faults has a passing efficacy record and no failing one. An applied fault with no efficacy record at all is not a fault that landed — it is a fault nobody checked, and every score built on it would be a confident number about a cluster whose state is unknown. Those scenarios render as NOT SCORED — <why> rather than as a row of zeros, and are excluded from every mean. The strictness extends to partial cascades: half an incident is not the incident the expectations describe, so grading against it would score the subject on ground truth that was never true.

A healthy control leaves exactly the same trace as a scenario nobody injected, and only the pack can tell them apart — which is why the join takes the pack. A control that reached the subject at all has done its job, and its measures must be scored: measuring invention is the only reason controls are in the pack.

The harness reports before the subject does. The scorecard puts the efficacy rate above any measure, and below --min-efficacy (0.8 by default) it prints the numbers, says they are unmeasured rather than poor, and exits non-zero. Reported first and refused second, because the rows that failed to inject are the ones that explain the refusal. A rig with a known-flaky gate can lower the bar, but has to say so on the command line rather than getting the numbers by default.

6.7 End-to-end with k8s-lookout — the rig’s own control

k8s-lookout should be subject number one, before any agent. Not as a courtesy to a sibling repo — because a deterministic subject is the only way to calibrate the instrument.

An LLM agent’s score moves for three reasons: the fault, the agent, and sampling noise. With lookout there is no third term. Run the same scenario twice and get two different scores and the harness is broken; that is a test you cannot run with any agent subject. core-sre-agent already reaches for this — its -bounded flag “swaps the producer: the same fixtures and the same four evaluators, scoring internal/bounded instead of the agent.” Same idea, one repo over.

It is also nearly free. lookout health --format=json already emits the §4.2 finding stream, and emit.Finding already carries the fields ground truth is keyed on:

The exec: adapter for lookout is a subprocess call and a field rename. Against an agent we must match reason tokens leniently because an agent writes prose; against lookout we can assert on Fingerprint and get an exact, unarguable answer.

What the e2e measures, in both directions:

• Simian is wrong — lookout does not see a fault Simian believes it injected. Either the fault never landed (an efficacy bug: §5 should have caught it, and did not) or the ground truth is mislabelled. Both are Simian bugs and both must be found before any agent number is trustworthy.
• Lookout is wrong — Simian injected a fault, efficacy probes confirm it landed, and lookout reports the namespace clean. That is a genuine detector coverage gap, filed against lookout, discovered by a rig built for something else. silentFailure and the dataplane pack are where this will happen: lookout’s graphfeed watches pods, nodes and ReplicaSets, and netprobe probes from the operator’s vantage rather than from inside the mesh.

And it sets the floor for every agent score. Lookout is the baseline row of the scorecard. An agent that has lookout as a tool and scores below lookout on a scenario is not failing to diagnose — it is failing to use its tools, which is a completely different bug with a completely different fix. Without the lookout row you cannot tell those apart. This is the single most actionable number the rig can produce, and it costs one subprocess adapter.

The e2e runs in CI as simian-eval --pack lookout --subject lookout:<binary>, on kind, on a schedule — deliberately mirroring lookout’s own .github/workflows/e2e-kind.yml (push-to-main smoke, weekly full) so the two repos’ cluster jobs stay recognisably the same shape. Shipped in e2e-kind’s Lookout eval step, driven by dev/tools/eval-lookout (make eval-lookout).

The scheme is lookout: rather than exec: because the detector emits the finding stream Simian grades — object kind, name, canonical reason, severity, one JSON record per line — but not a report envelope with a findings key, and it should not grow one. A detector with a Simian-shaped output mode is the coupling this whole section refuses. The shape translation lives on our side of the process boundary, in pkg/harness/subject/lookout.go.

What CI gates on is the efficacy rate, not the score. Every fault must land; recall is a measurement of the detector, and a detector that gets worse is a coverage gap to file against lookout rather than a reason to fail Simian’s build. The whole pack takes about six minutes on kind, of which four are lookout-crash-loop waiting for the loop to become continuously observable.

7. Deliverable D — the intelligence

Everything above is a better fixture corpus. This is the part that is genuinely Simian’s and that no static corpus can do.

pkg/topology already builds an informer-backed dependency graph. Today it is serialised into a prompt string at pkg/planner/generate.go:258 and the model is asked to be creative. That is prompt filler, not reasoning over a graph.

Make it load-bearing:

• Structural target selection. Compute the interesting properties from the graph rather than hoping the model infers them from a blob: chokepoints (high fan-in, no alternate path), single-replica services on a critical path, workloads with no PDB, Services whose selector matches exactly one ReplicaSet, cross-namespace edges. Then choose faults whose symptom lands somewhere other than the target.
• Generate whole Scenarios, not faults. The generator’s output must include the settle probe and the expected findings, or it cannot be scored — and this is tractable precisely because the generator chose the fault. It knows the ground truth by construction. It must also produce the task-shaped prompt without leaking the diagnosis, which is checkNamespace’s discipline stated as a generation constraint.
• Adversarial curriculum. Track which fault classes the stack misses and generate more of those. This is the payoff and the thing a fixed corpus structurally cannot do: a corpus tells you your score, a curriculum finds your blind spots.

This is also where the loop finally needs to be tool-using rather than single-shot. CompletionRequest.Tools and CompletionResponse.ToolCalls already exist in the interface and are already marshalled by the Gemini provider (pkg/llm/gemini/gemini.go:151-158, :186-195); no caller sets them. Graph queries are the tools worth exposing first — “what depends on X”, “what is unreplicated”, “what has no PDB” — because they are read-only, cheap, and exactly the questions the generator needs answered.

Note the sequencing argument: making the loop tool-using before there are tools worth calling produces a worse planner than the single-shot one, because its only tools would return replica counts and {"configured":false}. The graph tools are the first ones with real answers.

8. Phasing

Phases 0–4 are the rig. Phase 5 is the product. Phase 6 is the demo.

8.1 Prerequisite: Simian had no local cluster story

Resolved by #53. The diagnosis is kept because the decision it forced is recorded below.

At plan time: grep -rl 'kind create cluster' across this repo returned nothing. There was no kind config, no cluster script, no e2e workflow — CI was test, lint, tidy, govulncheck and nothing else. Every verification claim in the roadmap had been made by hand against a live GKE cluster.

An eval rig cannot be built on a cluster someone has to remember to create. This is the true first task, and it is copy-work rather than design work: k8s-lookout/examples/kind/{cluster.yaml,up,down} and core-sre-agent/internal/kindcluster are both known-good and both solve exactly this. kindcluster is the closer fit — it is Go, it is already shaped as a library for a test harness, and it does fresh-per-run with a context.WithoutCancel teardown.

Note the constraint discovered earlier: how much of NetworkPolicy the CNI enforces varies, and GKE Dataplane V2 bypasses Chaos Mesh NetworkChaos. No single environment runs all of Simian’s engines. The kind config must therefore either pin a CNI that enforces NetworkPolicy (Calico) or the dataplane pack must declare which environments it is valid in — and §5’s efficacy probes are what make that failure loud instead of silent.

Decision: kind + Calico is the reference environment (#53, shipped)

Calico, pinned. Older kindnet accepted NetworkPolicy objects without enforcing them, and Simian’s network-policy engine works by creating exactly those objects: a partition fault would apply cleanly, report success, and block nothing.

For a chaos tool that is a bug. For an eval rig it is disqualifying — the subject under test gets scored on an incident that never happened, and there is no error anywhere to notice, just a fault that did not land. That is the precise failure mode §5 exists to prevent, so the substrate must not be the thing introducing it.

Recent kindnet closes that particular gap — measured on kind v0.31.0 / Kubernetes v1.35.0, a workload is reachable before a deny-all-ingress policy and refused after. The rig still pins Calico, for reasons unrelated to the old caveat: Calico is what the eval targets look like, and a pinned CNI does not change behaviour underneath the rig when the node image moves.

kind + Calico is the reference because it runs all three implemented engines on a version Simian pins. GKE Dataplane V2 stays a second target where chaos-mesh NetworkChaos scenarios are invalid; declaring that per-scenario lands with the dataplane pack (#67).

The decision is enforced rather than documented: TestCNIEnforcesNetworkPolicy in test/e2e proves connectivity, applies a deny-all, and fails the build if traffic still flows. A comment in a YAML file would not have survived the first CNI bump.

What shipped with it: internal/kindcluster (create/delete/kubeconfig, Calico, Chaos Mesh, verification), make cluster / make cluster-down / make e2e, and an e2e-kind workflow on push to main. Credentials land in .kube/e2e.yaml inside the work tree and never in ~/.kube/config, so the file naming the throwaway cluster physically cannot name a real one.

8.2 Vertical slice first

The roadmap’s own M1 note says the right thing — “sequenced as a vertical slice first, then breadth and depth” — and it applies with more force here, because the expensive mistake available is building thirty fixtures against a scoring model that turns out to be wrong.

So: four fault kinds → one pack → lookout as subject → a scored run in CI, end to end, before any breadth. If that chain works, everything after it is filling in a table. If it does not, we find out having written four fixtures instead of thirty.

The slice is #53 → #54 → #56 → #60 → #62 → #63 → #64, with #48 alongside it because a live namespace escape should not sit. That chain is the only thing in the ledger that must be strictly serial.

8.3 Issue ledger

Tracked by #74–#79, one tracking issue per phase. Sizes are relative: S ≈ a focused sitting, M ≈ a day’s work, L ≈ several. Everything within a group is independent of its siblings.

Critical path: #54 → #56 → #60 → #62 → #63 → #64. Everything in Phase 0 is off the path and can land in any order by anyone. #53 is off the path but blocks #63, so it wants doing early.

First three merges, in order: #48 (the namespace escape is a live correctness bug and does not want to sit), #53 (nothing can be tested until there is a cluster), #54 (efficacy is the foundation everything else stands on).

8.4 What “done” looks like

A single command, in CI, producing a table:

scenario                 lookout   sre-agent   mast+lookout
fault-crashloop            1.00        1.00          1.00
fault-invoicing            0.00        0.50          1.00
latency-not-saturation     0.00        0.00          0.50
stress-real                1.00        1.00          1.00
pdb-gridlock               1.00        0.00          1.00      ← agent below its own tool

That table is the deliverable. Everything in this document exists to make it trustworthy: §5 so a row is not silently measuring nothing, §6.1 so the columns can be joined, §6.7 so the first column is a floor rather than another opinion.

It renders in the browser as a view of the existing web UI (#45), not a second application — see web-ui-design.md, “One site, two data sources”. That decision constrains #63: the simian-eval --out run artifact must be self-describing JSON that renders with no backend at all. Which is a gift rather than a cost — the same property makes a run attachable to a CI job, diffable between two commits, and openable by someone who has never installed Simian.

Two rendering rules matter enough to state here as well as there: a scenario whose efficacy probe failed renders as not measured, never as a zero; and the deterministic-detector column is a floor, so a subject scoring below it is flagged as failing to use its tools rather than failing to diagnose.

9. Phase 0 in detail

These were found as safety bugs. Under this plan they are also dataset-integrity bugs — an eval is only valid if the fault was exactly what the label says it was, and a fault that escapes its namespace has mislabelled every other scenario running beside it.

1. Namespace escape. pkg/driver/chaosmesh/driver.go:95-105 copies spec verbatim and injects only spec.duration; eligibility validation (pkg/executor/executor.go:241-265) checks only m.Targets[].Namespace, never spec.selector.namespaces. A manifest can therefore target any namespace in the cluster.
2. executor.permittedTiers is inert. deploy/helm/simian/values.yaml:46-48 is read by nothing — no flag, never passed by deployment.yaml. Node-tier chaos cannot be disabled by configuration.
3. Per-arena Role is missing networking.k8s.io. deploy/manifests/00-rbac.yaml:66-82 and the chart’s serviceaccount.yaml:55-74 omit NetworkPolicy create/delete, so the DPv2-recommended engine is Forbidden in-cluster. It works from simian serve locally because that path uses the operator’s own kubeconfig, which is why this was not caught.
4. arena.activeFaultCount omits NetworkPolicies (pkg/arena/arena.go:367), so the pre-destroy safety check under-reports exactly the fault class that leaks permanently on crash — the networkpolicy driver has no TTL of its own (pkg/driver/networkpolicy/driver.go:75-176).

Also in scope, lower severity: TOCTOU on the concurrency and cooldown checks (executor.go:276-293), the advisory-only severity cap (pkg/loop/loop.go:253 compares the LLM’s self-declared tier), tierOrdinal defaulting unknown tiers to least severe (loop.go:313-324), the pkg/topology informer goroutine leak (discoverer.go:75-90), and err.(*simian.ExecutorError) where errors.As is wanted (executor.go:309).

10. Non-goals

• Outcome verification. §1.1. Simian never reports what broke downstream.
• Detection logic. That is k8s-lookout. Simian never grows checks.
• An agent harness. That is mast / core-agent. Simian’s loop stays its own and stays small.
• Competing with ChaosBlade / Blade AI. Different category — a conversational chaos operator for human drill-running. Under this plan it is a subject the http: adapter can benchmark, which is more useful than competing with it.
• Replacing internal/faults. §2.1. It keeps working, unmodified, on its own fixtures.

11. Open decisions

1. Engine name. kube-state is proposed over workload because the driver also mutates Services, PVCs, and Jobs. Not load-bearing; easy to change before Phase 2.
2. Where the dataplane packs’ SUT comes from. Settled: a purpose-built topology, pkg/sut/edgeupstream, named by a scenario’s substrate: field. Online Boutique was too heavy, but the deciding argument was not weight — a graph we designed is a graph we can write assertions about, and the callee has to be expensive to serve or CPU saturation produces no latency and the matched pair stops discriminating. A borrowed demo app gives no control over that.
3. Whether mutate-mode reverts are trustworthy enough to run scenarios sequentially on one cluster, or whether fresh-per-scenario (as sre-eval-live does) stays mandatory. Fresh is correct and slow; this is a throughput decision to make after Phase 2, with data.
4. Litmus. pkg/driver/litmus/ is an empty directory whose constant, tier rule, and RBAC all ship, so an apply fails with no driver registered. Either implement it or remove the surface — under this plan there is no urgency for a second dataplane engine, so removal is the cheaper honest answer.