Seeing what Simian is doing

Simian records everything it does as audit events. Every view on this page reads those same events, so they cannot disagree with each other: a dashboard, the audit export and simian watch all tell the same story.

The commands assume the CLI can reach the controller — see step 4 of Getting started (SIMIAN_MCP_URL and a port-forward).

In a browser: the web UI

The controller serves a page at /ui/ on its own port, 8082 (chart values ui.*; v0.3 and later). Reached by port-forward it is read-only:

kubectl -n simian-system port-forward svc/simian-controller 18082:8082 &
# then open http://localhost:18082/ui/

Pick an arena at the top. The page shows the faults running now with their countdowns; what autonomous mode decided — the latest hypothesis and plan, with each step’s rationale and the reason for its duration, then each recent cycle’s outcome or skip reason; recent faults with how they ended, who asked for them, whether the engine confirmed them, whether their effect was seen and whether the workload recovered; the arena’s workloads against their desired replicas; and a live feed of audit events. It updates as events arrive.

Injecting and clearing faults from the page

Behind Identity-Aware Proxy (ui.auth=iap), the page also has an Inject a fault form — the kind and spec exactly, or a sentence for the LLM to translate — and a Clear button on each running fault, for the people listed in ui.iap.writers:

helm upgrade simian deploy/helm/simian -n simian-system --reuse-values \
    --set ui.auth=iap --set ui.iap.projectNumber=<project-number> \
    --set 'ui.iap.writers={you@example.com,domain:sre.example.com}'

• IAP decides who may see the page; ui.iap.writers who may also inject and clear. An empty list keeps it view-only for everyone.
• Every request must carry IAP’s signed assertion. The controller checks its signature against IAP’s keys, its issuer, and that its audience is a backend service in your project (ui.iap.audience pins the exact one), so nothing that reaches the pod without passing through IAP is let in. Only /healthz answers without it, for the load balancer.
• Faults from the page go through the executor like any other — arenas only, no excluded workloads, within the duration and concurrency limits — and the audit trail records who applied and who cleared each one (requested_by and cleared_by in the JSON audit export).
• Only the UI’s port is exposed. The MCP endpoint stays unauthenticated and inside the cluster.

The web UI behind Identity-Aware Proxy sets it up on GKE, from a hostname to the first fault injected from a browser.

Configuration and taking over

A Configuration panel shows everyone what the controller runs with: the safety limits (faults at once, longest fault, cooldown, blast radius), each arena and its excluded workloads, and autonomous mode’s settings — each named by the chart value that sets it, and, when an admin changed it, by whom and when.

The people in ui.iap.admins can also, from that panel:

• turn autonomous mode on or off and configure it — which arenas, how often, how many faults per cycle, the severity cap, a hypothesis hint — within the safety limits, which only the install changes;
• pause and resume it in one arena or everywhere (a paused arena’s cycles are recorded as skipped, paused);
• clear every running fault at once;
• halt: pause autonomous mode everywhere and then clear every running fault, in one request, so the loop cannot start a fault in between. It is one autonomous.configured event with action halt, plus a lease.cleared per fault, each with who did it; it takes effect even if it cannot be kept across a restart (the response says so). Resume is a separate, deliberate step.

A change takes effect at the next cycle; one already running finishes. It is kept in the controller’s simian-runtime-config ConfigMap, so it survives a restart and a helm upgrade; Revert returns to the chart’s autonomous.* values. Each change is an autonomous.configured audit event with who made it and the settings before and after. If the install’s limits later narrow below a kept setting, the controller starts on the install’s settings and says so in its log.

Without IAP (ui.auth=none, the default) the form is replaced by a note and the write endpoints refuse: with no one identified, anyone who could reach the port could inject faults.

Several Simians in one page

The page can watch several Simians — one controller per cluster, say — and drive whichever is selected. The header names the one shown, with a switcher once there is more than one; a strip along the bottom shows every Simian: whether it answers (live, needs sign-in, not allowed / sign in, not reachable), its running faults, autonomous mode on, off or paused, HALTED, and your role there. It glows amber for a running fault and rose for a halted Simian or one that does not answer; click one to show it. The panes, the live events, the Configuration panel and HALT all act on the selected Simian only. With one Simian the strip is hidden and the page looks as it always has.

The list is this page’s controller, the others its deployment names (ui.controllers, or simian web --controllers), and any you add by URL under ⧉ (or by clicking the name in the header), kept in your browser. A listed one can be hidden there and shown again.

The browser calls each controller directly, with your own sign-in to it: nothing is proxied, each controller checks who you are, and what you do is in that controller’s audit trail under your name. So each controller must list the page’s origin in ui.allowedOrigins, and you sign in to each one once; a Simian you have not signed in to offers Sign in to …, which opens its own page in a new tab. Behind IAP, IAP must also let CORS preflights through — Several Simians in one UI sets it all up.

The standalone UI. The same image serves the page with no controller behind it — no Kubernetes, no LLM — for a page that fronts several:

simian web --addr :8080 \
    --controllers simian-1=https://simian.example.com \
    --controllers simian-2=https://simian-2.example.com
# then open http://localhost:8080/ui/ (or put it behind IAP like any page)

It serves /ui/, /healthz and GET /api/controllers; every other /api/ path is 404, which is how the page knows no Simian serves it and asks which to watch. With --scorecards DIR it also serves the scorecard documents in DIR, read-only, at /scorecards/.

Scorecard mode

The page has a second mode, for a recorded evaluation rather than a live controller: Scorecard in the top bar (and Live to go back). It reads the JSON document simian-eval report writes, from a dropped or picked file, a ?scorecard=<url> link, or the list simian web --scorecards serves, and needs nothing behind it. With no controller to watch, the standalone page opens in it. See Reading an evaluation: the scorecard.

The page reads a small JSON API under /api/ (info, me, config, arenas, catalog, active, faults, cycles, topology, and controllers, the deployment’s other Simians), proposes a fault from a sentence with POST /api/translate (nothing is applied), writes with POST /api/faults (answered when the executor is done, or after 15 seconds with 202 and the fault’s UID, the rest following on the event stream) and POST /api/faults/{uid}/clear, administers with POST /api/admin/autonomous, /api/admin/autonomous/reset, /api/admin/pause, /api/admin/clear-all and /api/admin/halt, and streams events from /api/events; other tools can use it too.

Live: simian watch

bin/simian watch --namespace boutique

Redraws every few seconds (Ctrl-C to leave):

simian watch — boutique   (updated 18:29:49)
────────────────────────────────────────────────────────────────────────
ACTIVE FAULTS (1)
  f-01M497MC2X  chaos-mesh       PodChaos         → adservice             [1m1s remaining]

AUTONOMOUS CYCLES (4)
  latest plan (18:28:10):
    If the adservice is unavailable, the frontend will handle the
    error gracefully and continue to serve user requests for other
    product features without significant latency or errors.
    1. PodChaos → adservice for 1m30s
       why that long: The pod-kill action is instantaneous. The 90-second duratio…
  18:28:10  completed PodChaos→adservice (1 applied)
  18:26:10  completed NetworkChaos→paymentservice (1 applied)
  18:24:10  skipped   budget-full
  18:22:10  completed NetworkChaos→currencyservice (1 applied)

RECENT FAULTS (4)
  18:29:20  f-01M497MC2X chaos-mesh       PodChaos         → adservice             applied
  18:28:40  f-01M497FE8N chaos-mesh       NetworkChaos     → paymentservice        expired (deadline-reached)
  18:26:10  f-01M497AJ44 chaos-mesh       NetworkChaos     → currencyservice       expired (deadline-reached)
  18:23:10  f-01M49770TJ chaos-mesh       PodChaos         → productcatalogservice  expired (deadline-reached)

The cycles section is what autonomous mode is thinking: the latest hypothesis and plan in full, then one line per recent cycle with what it applied or why it was skipped. It needs a v0.1.14 or newer controller.

What autonomous mode decided

kubectl -n simian-system exec deploy/simian-controller -- \
    simian audit export --cycles /var/lib/simian/audit.jsonl

One row per cycle and arena: when it started, its outcome (completed, skipped, unfinished if the controller restarted mid-cycle), the skip reason, the plan’s steps, how many steps were applied and refused, and the hypothesis — or, for a skipped cycle, what the health gate, the LLM or the plan validator said. --format json adds each step’s rationale and the reason for its duration.

What was injected

kubectl -n simian-system exec deploy/simian-controller -- \
    simian audit export /var/lib/simian/audit.jsonl

One row per fault: source (autonomous or directed), kind, target, how it ended (expired deadline-reached is the normal end; refused with the safety check’s reason; cleared early), whether Chaos Mesh confirmed the injection, whether Simian’s probes saw the effect, and whether the workload was Ready again afterwards (RECOVERED). --since 24h limits it; --format json adds the full spec and, for a failed recovery, which pods stayed down and why.

The audit file lives on the controller’s volume and survives restarts. Both exports also read the controller’s log: kubectl -n simian-system logs deploy/simian-controller | bin/simian audit export --cycles.

Metrics and alerts

The controller serves Prometheus metrics on port 9090 at /metrics (metrics.enabled in the chart, on by default). The pod carries the usual prometheus.io/scrape annotations; on GKE, set --set metrics.podMonitoring.enabled=true and Managed Service for Prometheus collects them, queryable in Cloud Monitoring. With prometheus-operator, use metrics.serviceMonitor.enabled=true.

The arena is labelled arena, not namespace: Managed Service for Prometheus and most Prometheus scrape configs set namespace to the namespace of the pod they scraped, which is simian-system for every series. The controller also creates each arena’s series at 0 — for every fault kind in the catalog, every cycle outcome and every refusal reason — as soon as it sees the arena, so increase() counts the first fault rather than starting from it. A refusal in a namespace that is not an arena still counts, but its series starts at 1.

Dashboards

Two ready-made dashboards ship in deploy/dashboards/, with the same panels: faults applied by kind and by arena, active faults, how faults ended, recovery failures (in red — the panel that needs a person), autonomous cycles by outcome and skip reason, refusals by reason, and scorecards for the running version, faults held now and broken workloads in the last day.

• GKE (Cloud Monitoring): with metrics.podMonitoring.enabled=true,

  gcloud monitoring dashboards create --project "$GOOGLE_CLOUD_PROJECT" \
      --config-from-file=deploy/dashboards/cloud-monitoring.json
  

  It appears as Simian under Monitoring → Dashboards. It shows every Simian install in the project; the cluster selector in the toolbar narrows it to one. The label is the cluster the controller runs in, which is also the cluster it tests.

• Grafana: Dashboards → New → Import, upload deploy/dashboards/grafana.json, and pick your Prometheus data source. A Cluster and an Arena selector filter every panel. Without a cluster label on your series — a single Prometheus that does not add one — leave Cluster on All. To provision it instead, point a file provider at the directory holding the JSON.

Useful queries:

# faults applied per hour, by kind
sum by (kind) (increase(simian_faults_applied_total[1h]))

# share of cycles skipped, by reason, over the last hour
sum by (reason) (increase(simian_cycles_total{outcome="skipped"}[1h]))
  / ignoring(reason) group_left sum(increase(simian_cycles_total[1h]))

# refusals, by reason — the safety checks at work
sum by (reason) (increase(simian_faults_refused_total[1d]))

Alerts worth having:

- alert: SimianFaultLeftWorkloadBroken      # a fault ended and its pods did not come back
  expr: increase(simian_fault_recovery_checks_total{passed="false"}[15m]) > 0
- alert: SimianAutonomousModeStalled         # on, but nothing applied for an hour
  expr: sum(increase(simian_cycles_total[1h])) > 0 and sum(increase(simian_faults_applied_total[1h])) == 0

A broken workload after a fault is the one that needs a person: the JSON audit export names the pods, and deleting them usually restores them.

From an agent: MCP

The controller’s MCP endpoint (port 8081) answers the same questions as tools any MCP client can call: list_active_faults, get_recent_faults, get_recent_cycles (the cycle view above), get_topology, get_baseline. simian watch is built on them.

The raw events

kubectl -n simian-system logs deploy/simian-controller | grep '"msg":"audit"'

Every event is one JSON line: cycle.started, plan.generated, executor.received, executor.validated, driver.applied, fault.injected, fault.efficacy, lease.expired, fault.recovered, cycle.skipped, and the rest. On GKE the controller’s stdout reaches Cloud Logging; in Logs Explorer:

resource.labels.namespace_name="simian-system"
jsonPayload.event="plan.generated"

Chaos Mesh’s dashboard

Chaos Mesh installs its own dashboard (chaos-dashboard), which shows its experiments as they run:

kubectl -n chaos-mesh port-forward svc/chaos-dashboard 2333:2333

It sees only Chaos Mesh’s side — not Simian’s network-policy or kube-state faults, and nothing of why a fault was chosen — and it shows the experiments' objects, which is exactly what a triaging agent must not read (see Agents that triage the cluster). Keep it for operators.

Not yet

The eval scorecard view from the web UI’s design is not built yet. Simian does not write Kubernetes Events: events in an arena would be readable by any agent with the view role, and would hand it the answer.