Skip to content

Interop surfaces

An unattended agent is only useful if something can reach it. mast exposes several surfaces rather than one, because “an alertmanager webhook”, “an operator watching a live incident”, “another team’s agent”, and “a chat UI” are genuinely different callers with different auth models — and collapsing them into one endpoint means the weakest one sets the bar.

Every surface is off by default except inject, and each carries its own token. That is deliberate: they are separate trust boundaries, not four views of one.

A model’s thinking is not part of any answer mast gives. Frontier models return their reasoning as a distinct kind of content — Anthropic calls it a thinking block, Gemini a thought part — and it comes back in the same field as the answer, marked rather than separated. mast reads the mark. Reasoning is replayed to the provider on the next turn, because a signed thinking block has to come back intact for the model to continue, and it goes nowhere else: not into the A2A result artifact, not into an AG-UI TextMessage frame or RunFinished.result, not into the /parks projection, not into a specialist result the planner reads, and not into the daemon’s own operator log.

This is a floor, not a policy, and one surface has already asked: an AG-UI workload that sets agui.emit_reasoning streams the thinking as its own REASONING_* frames. That does not weaken the floor — it is a deliberate read of the thought parts, by name, rather than the removal of a filter, so the answer stream is unchanged whether the opt-in is on or off and “reasoning reached a browser” stays a grep for one symbol. Off by default, and off publishes nothing at all rather than an empty placeholder. What is ruled out is publishing reasoning by forgetting to check.

The one thing no opt-in reaches is the thought signature — the opaque blob a provider wants back to continue. It is a replay credential, not a thought, so it is replayed to the provider and modeled nowhere on any wire.

The daemon’s own HTTP endpoint (--listen, default :7777): /inject, /resume, /abort, /parks, /metrics, /healthz. This is how an incident gets in — an Alertmanager webhook, a Cloud Scheduler job, a CI step, a curl in a runbook. Bearer auth via MAST_INJECT_TOKEN; unset means unauthenticated, which is a dev-only posture and is warned about at startup.

If you are wiring monitoring to mast, this is the surface. The others are additions to it.

A Cloud Scheduler job or a Kubernetes CronJob posting here still works, and it remains the right answer when you already run one — your schedules live in one place and mast is just another target. What changed in v0.4 is that you no longer need one to run something periodically: see no caller at all below.

Two routes answer that, and they answer different questions.

routeaskscan it go red?
GET /is this process still accepting connections?no — it consults nothing and is a static 200
GET /healthzcan it still do the job it accepts work for?yes — 503 when a dependency is down

GET /healthz runs the daemon’s registered checks and answers 200 {"ok":true,"checks":{"session_db":"ready"}} or 503 with that check failed. Today the one check is the session database, registered when --session-db is set: a real read against the table ADK writes events to, not a pool ping, so it catches a dropped table, a lock that outlasts the busy timeout, a closed pool, a Postgres that went away, and a volume that was unmounted out from under a running pod. It does not catch a filesystem that went read-only — a read still succeeds there.

A daemon with no session store reports 200 {"ok":true,"checks":{}}. The empty object is the point: “the process is up and has nothing durable to vouch for” is a different statement from “the database is fine”, and an in-memory daemon that answered 503 would never join its Service.

The route is unauthenticated, because a kubelet has no bearer token and readinessProbe has nowhere to put one. So the body carries only ready or failed per check — never the error, which routinely names a filesystem path or a DSN host. The error goes to the daemon’s log, once per transition rather than once per probe.

Point readinessProbe at /healthz and leave livenessProbe on /. That split is deliberate. The action behind a failed liveness probe is a restart, and a restart does not fix a deleted volume or an unreachable database — it turns one unready pod into a crash loop that also kills every in-flight turn. Readiness is the probe that should be able to go red: it takes the pod out of the Service, which stops it being handed injects it cannot durably record. The chart’s daemon StatefulSet (charts/mast/templates/statefulset-daemon.yaml) ships wired that way.

POST /resume has been able to answer a parked approval since v0.2. What no out-of-process caller could do until v0.9 was find out what it was answering — so a chat bot or a web UI wiring up an Approve button could only offer “approve the thing, whatever it is”. That is uninformed consent with an audit trail, which is worse than the CLI it replaces, because it looks like the opposite.

Two read-only routes close that:

routereturns
GET /parksevery parked session, each in full, oldest question first
GET /parks/{session}the named session whether it is parked or not — 200 with an empty parks list when it is not, 404 only when no such session exists

Both need --session-db; a daemon without a session store answers 404 rather than an empty list, because “nothing to approve” and “I cannot see parks” are different answers and only one of them means it is safe to stop looking. Auth is the same rule as /resume: the shared token, or a user from the token table. A caller who may answer a park may read it, and a caller who may not read it has no business answering one.

The body is always an object with a sessions array — for the single-session route too, and when nothing is parked. Each session carries its state, an awaiting (approval or input, an approval outranking a question, the same ranking turn_state uses), the open parks, an operator hold if one is active, and the applied calls the gate captured a revert for.

An approval park carries a change: the tool, the arguments, the rendered key, the gate policy that parked it, the proposing specialist, the verdict_format the gate will accept, a stale line when a grant the operator already gave no longer covers the call — and the change_set the call belongs to, when it belongs to one, because an operator cannot sensibly trade scope: change_set for a set they have not been shown. A question park carries none of that: a question is not a change.

What it does not carry is as much of the contract as what it does. No model output — no reasoning, no narration, none of the assistant text around the call. No tool results, including the result of the read that produced a capture: only that read’s name, its digest, and the declared field names travel, because the values are cluster state and can be anything. And nothing the write gate did not park on. The projection is written out field by field in internal/cli/parks.go, so a field added to the transcript cannot reach this wire without someone deciding it should.

A hold is reported beside the parks and never as one. Nothing resolves by answering a hold, so a client that rendered it as a question would put a prompt in front of somebody with nothing to say back.

--attach-listen serves the mast-native attach protocol (HTTP + SSE) that mast-web speaks: live-tail a running turn, read a session’s transcript, inject an operator message, answer a parked approval. It needs a durable session store, since the live tail pumps from the event log — so it implies one: with no --session-db of your own, the daemon opens ~/.mast/sessions.db and says so at startup. Name a path to put it anywhere else.

Prose is not a contract. The JSON these routes return is pinned as fixtures in internal/attach/testdata/conformance/ — rest-*-v1.json for the plain HTTP bodies, the SSE event fixtures beside them for the stream. Write a client against those, not against a mock you wrote yourself: mast-web’s bundled mock invented snake_case names for the sessions list, its client was written against the mock, every one of its tests passed, and the list rendered empty against every real daemon (mast-web#41). A fixture is the thing a mock can be checked against.

Three shapes a client gets wrong by reading the examples rather than the fixtures:

  • last_touched_at and next_wake_at are always on the wire, as 0001-01-01T00:00:00Z when they do not apply. They are tagged omitempty, which does nothing to a time.Time. Read them for the zero value; presence does not mean the field applies.
  • GET /sessions with nothing to show is {"sessions":[]}, never null. So are viewers and contributors on an ACL. Everything else optional is genuinely absent.
  • On GET .../usage, a cache write counts as uncached input — it bills at a premium over fresh input, not a discount — and cost_usd_uncached_reference is a counterfactual, not a floor. A turn that warmed the cache and has not reused it yet reports a reference below its cost. That is correct, not a rounding error to hide.

The fixtures’ own README lists which routes are pinned and, more usefully, which are not.

POST /sessions opens an empty session and answers 201 with its id (attach-<16 hex>) and URL. The client then injects into it. Each one runs the daemon’s workload, exactly like a session an inject created. That is the shape of a client that starts its own conversations, not one that reacts to events: k8s-lookout’s sinks open one per incident, and switchboard opens one per chat thread. Before #524 every mast daemon answered 501 here: the route existed, but mast serve never configured the factory behind it.

The daemon is still single-user. A session created here has no owner and no ACL, the same trust model as an inject-created one. Give each user or conversation its own daemon (or sandbox) when they must not see each other’s sessions.

Every status-update frame carries a turn_state, and a client that renders “is this session busy, or does it need me?” reads that and nothing else. It has four values:

turn_statemeans
streaminga turn is running right now
awaiting_permissionthe write gate parked a mutating call; somebody has to approve or reject it
awaiting_elicitthe session asked a question — request_operator_input — and is waiting for the answer
idlenone of the above

The distinction matters because a parked session finishes its turn. The gate does not block inside the call; it records the question durably and returns, so from the daemon’s point of view nothing is running. Before v0.9 that is exactly what the wire said: a session waiting on a human reported idle, the same string a session that finished its work reports, and a gateway rendering an approval card had no frame to hang it on.

Two things follow from where the answer comes from. It is read from the transcript, not from in-memory turn state, so it survives a daemon restart — a session parked yesterday still reports awaiting_permission today. And it is in the boot snapshot, the status-update every client receives on connect, so a client that attaches an hour after the park sees it without having been present for the frame.

Two cases are deliberately not awaiting_*. A session paused by pause_session reports idle: that is a hold an operator placed, not a question anyone is waiting on an answer to. And a session resuming from a park reports streaming for the whole resume turn, even though the interrupt it is answering stays open on the transcript until mid-turn — a live turn outranks the park it is resolving, because reporting otherwise would freeze the session on an operator’s screen while it works.

One ordering guarantee comes with this. turn-complete and turn-error are now held until the turn’s own event-log frames have reached every subscriber. They previously raced: the terminal frame went straight to subscribers the moment the turn returned, while the answer travelled through the event log and the pump, so a client that finalized its render on turn-complete could drop the last message. If the log cannot be queried, or does not catch up within two seconds, the terminal frame is released anyway and the delay is logged — a late frame is a correctness bug, a missing one is worse.

awaiting_permission tells a client there is a question; two routes let it carry the answer back without leaving the attach protocol:

routedoes
GET /sessions/{app}/{id}/perms/streamSSE; one prompt event per open approval park
POST /sessions/{app}/{id}/perms/respond{"id": "<interrupt id>", "decision": "allow-once"}
GET /sessions/{app}/{id}/permsthe rules this session runs under, and what it has already decided

This is a second door onto POST /resume, not a second approval model. A press resolves the same durable park, is attributed to the same authenticated caller, and is subject to the same rules — including the ones that make this surface narrower than the protocol it inherited:

  • decision takes deny and allow-once. The four wider values the protocol defines are refused 400 by name rather than narrowed to allow-once, the same refusal scope: session gets on /resume. A button that means less than it says is a bad button; one that means more is worse.
  • Answering a park that is already resolved is a 404. Durability cuts both ways: a question that outlives the process can also be answered by someone else first.
  • A 200 is {"acknowledged": true, "approver": "<identity>"}. When approver is absent, mast recorded nobody — do not render a name.
  • perms_stream in the capability report is true exactly when these routes will answer. A daemon that cannot serve them answers 501.

Before v0.9 the routes existed and every mast daemon answered 501: they came with a ported package built for a synchronous prompt at a keyboard, and mast has no keyboard. They now sit on the mechanism mast does have. A park carries kind: control_plane_write, which is the field a client should switch its button set on.

The read beside them, GET /perms, has its own capability flag — perms, not perms_stream — because they are separate wirings. A daemon can say what its rules are and what it has adjudicated without ever streaming a live question, and mast is that daemon. The read answers mode (the permissions gate, absent when there is none), on_mutation (the bundle’s write-gate policy, on its own axis), and the session’s durable approval log. Through v0.8 it answered 200 with the zero value on every mast daemon, which is a description of something that gates nothing; it now refuses instead, and the write-gate reference spells out the fields.

GET /sessions/{id}/tools lists the tools the daemon actually holds, each with a source, the MCP server it came from if it has one, and a gate_state — what the write gate would do to a call of it: allowed, prompted (it parks for approval), or denied (on_mutation: dry_run, so it will never run).

Two sources appear. builtin is the daemon’s own control plane — under planner dispatch, invoke_specialist, run_shape_llm_router, run_shape_fan_out_fan_in, request_operator_input, and pause_session. mcp is everything a server declared. Builtins sort ahead of the servers and are always listed: unlike a server, they cannot fail to answer or vanish between polls. The MCP half is read from the live servers rather than from the bundle’s declaration, and refreshed at most every 30 seconds; a server that fails to answer is left out and logged rather than blanking the rest — including the builtins, which stay listed when every server is down.

What is still missing is the handful of tools ADK installs itself: finish_task, and a coordinator’s per-specialist transfer tools. mast never wires those, so it cannot enumerate them without keeping a hand-written list — and a catalog naming tools that do not exist is worse than one omitting tools that do.

GET /sessions/{id}/subagents lists the roster the daemon loaded — what this thing can do. (/agents lists what has been spawned, which for mast is always empty: every dispatch shape resolves its specialists inside the turn.) Each entry carries the specialist’s description, its model: override if it declared one, its declared capability (read_only or change_executor), its agent_mode (Task or SingleTurn), and an invocation — how the composed root actually reaches it:

invocationmeans
parent_toolthe planner calls it via invoke_specialist
transfercoordinator dispatch hands the turn to it
graph_nodea node in a graph, or fan-out’s synthesis merger
fanout_branchone of fan-out’s concurrent analysts
(empty)nothing in this shape reaches it

That last row is the useful one. A roster can carry a member the composed shape never routes to — a _fallback under fan-out dispatch, a second SingleTurn spec under graph dispatch — and an empty invocation is how you see it without reading the composition code.

Each entry also carries a tools grant, which is the other half of what can this thing do: capability says whether the specialist is allowed to change anything, tools says what it can reach.

"tools": {
"mcp_grant": "listed",
"mcp": [
{"server": "gke", "tools": ["get_pod", "get_events"], "whole_server": false},
{"server": "slack", "whole_server": true}
],
"builtin_declared": ["apply_manifest"]
}

mcp_grant is the field to read first, because the underlying declaration means opposite things one character apart. A spec with no mcp: key inherits every MCP toolset the workload has ("all"); a spec that writes mcp: [] is denied all of them ("none"); a non-empty list is a whitelist ("listed", and only then is mcp present). Both of the first two spellings are an empty list on the wire, so a catalog that just transcribed the declaration would show the deny-all specialist and the reach-the-whole-cluster one identically. whole_server draws the same distinction per entry: a listed server with no tools: of its own passes whole.

builtin_declared is named for what it is: nothing in the declaration installs a tool, so the list grants and narrows nothing. What reads it is the capability-split check, which refuses a read_only specialist that lists a mutating name, the fan-out branch check, which does the same for an analyst, and the startup log that reports a workload’s declared write surface. Read it as a claim the spec makes about itself and is held to, not as a set of tools the specialist holds.

The one built-in mast does install on a Task specialist is retrieve_raw, and it deliberately does not appear here: it is a property of the daemon’s --mcp-digest setting rather than of the roster, and it returns bytes the specialist has already received, so it widens no grant this catalog describes.

GET /sessions/{id}/guardrails answers the question an operator actually has when a session stops responding: what stopped it, and what do I do? It reports the budget ceilings in force and the usage against them across all three dimensions plus each specialist’s own, and the watchdog’s posture (advisory: true under --watchdog=warn and under the default --watchdog=feedback, which corrects and can end a looping turn but never halts the session; false under --watchdog=enforce, whether or not it has fired yet).

The watchdog raises three signals by default, and each names a different way an unattended run goes wrong:

signalseverityfires when
repeated-tool-callCriticalthe same call, five times running — path spellings (./main.go, /workspace/main.go) count as the same file
alternating-tool-cycleCriticala short loop repeating, e.g. list_agents → check_agent three times over — the shape a consecutive-repeat check is structurally blind to
tool-failure-streakWarnthree calls in a row all returned errors, so nothing the agent ran has verified the state of anything

The last one is the one to read carefully in a report. It does not mean the agent was stuck; it means the agent’s conclusions rest on tools that all failed. It stays Warn deliberately: three denials into a legitimate RBAC probe is not a runaway, and halting there would make the backstop the outage.

A fourth signal, dominant-tool-call, ships and is not wired by default. It covers the shape between the first two — one call repeated with occasional others wedged in (a a a b a a a c a a a), which resets the repeat detector’s run and shows the cycle detector no repeating block. Density does not care where the interleaves fall, so it reaches the verdict inside one twelve-call window instead of waiting for the interleaves to stop; when they are the whole delay, that is most of a loop’s cost. It is off by default because a third Critical detector changes what every unattended workload is told about itself under feedback, and a polling workload with any variation in it is exactly a dominant call with interleaves. Nothing turns it on today: the watchdog is internal since #301, and until then only an embedder composing its own runner could hand it a signal list. When it runs, it stands down wherever one of the other two already owns the shape, so one loop still produces one alert.

Severity is a property of the pattern, not of the posture. What changes between postures is the reaction, and the three of them are a ladder — each rung includes the one below it:

  • --watchdog=warn logs every alert and lets the turn run.
  • --watchdog=feedback (default) also tells the model. Each alert carries a model-facing sentence alongside the operator-facing one, and on the session’s next turn those are prepended to the prompt as a [watchdog] block: an automated observation about your own previous turn — this is not a message from the user. Every posture below this one routes the observation to a reader who may not be there; the model about to repeat the call is the party that can decide not to. Inside a turn, a call repeated with the same arguments and the same result is told on its result and the turn ends if the model keeps going — see a loop inside one turn. Across turns it is a correction, not a backstop: nothing halts a session whose model reads the block and loops anyway, which is why a workload with a bounded tool loop still wants enforce.
  • --watchdog=enforce also cancels the turn in flight on a Critical alert and refuses the session’s every subsequent turn — auto-resume, a scheduled fire, and an attach inject all included — until an operator resets. The cancel happens during the turn, not at its boundary: the loop the watchdog catches usually lives inside a single turn, and a reaction that waits for the turn to end waits for the thing it is supposed to stop.

A turn cancelled that way — and a turn an operator stops with POST /sessions/{id}/interrupt — comes back on the stream as a turn-error of kind canceled, with retryable: false. That is the flag a client keys its “run it again” affordance off, and both of these are somebody deciding the turn should stop; offering to re-drive the loop the watchdog just halted would be the wrong end of the decision. A turn that ran out of time is still transient_network and still retryable, because nobody asked for that one.

The halt itself — the refusal the tripping turn returns, and the one every turn after it gets until a reset — comes back as kind watchdog_halt, also retryable: false, with a hint naming the reset endpoint. It is a separate kind from canceled because it describes a standing state rather than one stopped turn: retrying a canceled turn is arguably the operator’s call, while retrying a watchdog_halt fails identically until somebody clears it. Both are separate from cost_ceiling, whose remedy is a bigger budget rather than a fixed loop.

refusal_loop is the fourth of these, and the distinction it turns on is the one a client acts on. It ends a turn in which the model kept re-proposing a call an operator had already refused — see a refusal the model cannot talk its way past. Also retryable: false, and deliberately not a watchdog_halt: nothing latched, no guardrail tripped, and the next turn starts clean. A client that rendered the halt’s remedy here would send an operator off to clear a guardrail that never tripped. Its hint says so, and says what to do instead — if the change should happen after all, send a new turn and approve the call when it parks.

loop_stop is the fifth. It ends a turn in which the model kept repeating one call after being told, inside the turn, that it was looping — see a loop inside one turn. Also retryable: false, since re-driving the turn hands the same model the context that looped, and like refusal_loop it latches nothing. Its hint points at what an operator can change: the tool’s output or the model.

enforce including feedback is deliberate. An enforce halt is cleared by an operator reset, and a reset resumes a model whose context still ends in the loop it was halted for; without the injected observation the very next turn re-issues the same call, and the reset is a treadmill. For the same reason, a reset clears the halt but keeps the queued observation.

The [watchdog] block is steering, not a trust boundary — nothing downstream grants authority based on it, and a user prompt may contain the literal string.

Three sources, in order: --watchdog beats the bundle’s safety.watchdog beats mast’s default. The daemon logs which one won at startup (watchdog posture resolved mode=… source=…), because a posture nobody can see is a posture nobody audits — and enforce, the one an operator most needs to know is armed, otherwise announces itself only by refusing a turn.

workload.yaml
safety:
watchdog: enforce

The bundle is where a workload ships its own backstop: mast’s deployment unit is the bundle, and without this every invocation and every deploy manifest had to carry the flag by hand. The flag sits above it so an operator debugging a halted workload can drop the posture for one run without editing — and later forgetting to revert — the deployed manifest.

The default is feedback. Not warn: every mast run is unattended, so warn routes the alert to a log nobody is tailing, which is indistinguishable from off. Not enforce either: alternating-tool-cycle has a workload-shaped false positive — a scheduler-driven daemon watching a rollout settle calls the same tool with the same arguments on purpose — and on an unattended deployment a false halt is an outage that waits for the morning. A false feedback costs one paragraph the model is free to disregard. Recoverable beats unrecoverable when nobody is watching. Set safety.watchdog: enforce on a workload whose tool loop is bounded by construction — a triage run that reads, concludes, and stops.

The library-embedded surface (mast.RunWorkload) reads the same safety.watchdog field and taps the same signals, with the rungs bounded by what that surface holds: enforce abandons the runaway turn, but there is no cross-call session state for the “refuse every later turn” half, and no next turn for feedback to inject into — though its in-turn loop guard works there as it does everywhere.

The [watchdog] block arrives on the session’s next turn, and the loop the watchdog exists for usually lives inside a single one. Measured: a self-hosted Gemma 4 called one tool with identical arguments 441 times in one turn until the server refused the context (#514). The repeat signal fired at five, the observation was queued for a turn that never came, and nothing reached the model.

So under feedback and enforce, a loop guard also acts inside the turn, on the tool side:

  • From the fifth consecutive call to one tool with the same arguments and the same result, the result carries a watchdog field: an automated observation that the result will not change, what to do instead, and how many more repeats end the turn.
  • The third repeat after that is not made, and the turn ends as loop_stop — a mast_turns_total outcome and a turn-error kind of its own. Nothing latches: the next turn starts clean, with the usual [watchdog] block in front of it.

The result has to match as well as the arguments, which is stricter than the signal. The signal can count calls alone because under feedback its false positive costs a paragraph; this one ends a turn, and the false positive named below — a daemon polling a rollout with the same call on purpose — is exactly a run of identical calls whose results change. A changed result starts the count again.

Under enforce the session halts at the fifth call as before, so the guard’s stop is never reached; under warn the repeated-call guard is off.

A run of identical calls is one shape of runaway. For the rest — a cycle through two or three calls, a repeat with other calls wedged in, exploration that never converges — the same guard enforces budget.max_model_calls_per_turn, default 100, under every posture: the call over the cap is not made and the turn ends as loop_stop (#519). It is a budget rather than a detector, so it needs no price and recognizes nothing; it just stops counting at a number no healthy turn on the judged corpus came near. It runs on the daemon, a one-shot, and mast.RunWorkload alike, and in the eval harness’s judged tier, so a looping model is scored the way a daemon would run it. It does not yet reach inside a planner dispatch, whose sub-runner carries no plugins (#235).

Under planner dispatch, invoke_specialist runs each specialist on a runner of its own, and a private runner is a private event stream. Through v0.4 that meant a specialist spinning inside one dispatch emitted its calls where the watchdog could not see them — --watchdog=enforce could not halt a loop it could not observe, which is the exact shape it exists to catch. Since v0.5 the dispatch feeds those events to the session’s watchdog, the same one the outer stream feeds.

The session’s, deliberately, and not one minted per dispatch: the signals count repetition, so a per-dispatch watchdog would reset the count every time a dispatch ended and a specialist making three identical calls in each of ten dispatches would never reach a threshold of five. The cost is worth knowing — the planner’s own calls and its specialists’ now land in one signal set, so an invoke_specialist between two dispatches breaks a consecutive run the repeat detector was building. That is the interleaved shape alternating-tool-cycle and the opt-in dominant-tool signal exist for, and it is the same trade-off the outer stream already makes between a coordinator and its sub-agents. The loop this catches — a specialist spinning inside one dispatch — has no interleaver at all.

A trip here halts the session, not just the dispatch. This is the deliberate difference from a crossed budget cap, which stops only the dispatch and hands the planner the reason (budgets). A ceiling is cumulative, so stopping the sub-run is the whole remedy. A watchdog trip is a latch meaning this session is behaving pathologically and an operator must reset it, and every other door to it refuses the whole session; a halt that stopped only the dispatch would let the planner re-dispatch the same specialist immediately, which is the treadmill enforce exists to break. So both fire: the dispatch stops with a labelled partial, and the turn is cancelled. Under warn and feedback nothing is cancelled — the alert is retained and reportable, and the model-facing paragraph goes to the planner, since the specialist’s sub-session is already gone and the planner is the one that would otherwise dispatch again.

Library embeds that compose the planner themselves get this by wiring compose.RootConfig.SubRunObserver, the same seam the meter uses; mast.RunWorkload and the daemon already do.

A halt outlives the process that observed it

Section titled “A halt outlives the process that observed it”

An enforce halt is written to the session database, and a daemon that restarts adopts it on the halted session’s next turn — before any model call, whichever surface drives that turn. A halt a restart cleared would not be a halt: mast’s restarts are automatic and unattended, so the loop → halt → crash → restart cycle enforce exists to break would simply resume, each restart handing the loop a clean backstop. The reset is durable in the same place, so clearing a halt clears it for good, and the row records who cleared it and what runway they added.

Two things to know about it:

  • It needs --attach-listen (which implies --session-db, so the store the halt is written to exists exactly when the reset does). The reset endpoint is attach-only, so persisting a halt on a daemon with no attach surface would leave an operator no way to clear it. Starting with --watchdog=enforce and no attach listener logs a warning saying exactly that.
  • The posture still wins over the history. A deployment dialed back from enforce does not inherit a halt it would no longer produce.

Restore fails open: an unreadable guardrail table logs a warning and the turn runs. A storage fault must not halt every session in the deployment with no trip behind it, and the per-turn backstops are all still armed.

Budget spend is durable in the same place and fails open the same way — a ledger of priced calls, folded back before the first turn after a restart — but not on the same terms: it needs --session-db and not --attach-listen. A ledger is not a latch, so the reset-endpoint argument above does not reach it, and there is nothing an operator has to be able to clear. Through v0.6.0 it inherited that argument anyway by riding the same database handle, which denied a durable ceiling to unattended daemons — the deployment least likely to bind an operator socket and most likely to crash-loop (issue #274). See spend survives a restart.

POST /sessions/{id}/guardrails/reset is the way out: a budget trip is otherwise permanent, since enforcement is re-derived from usage against the ceiling on every priced event. See getting unstuck after a trip for what a reset does and the three things it deliberately refuses to do.

GET /sessions/{id}/usage answers the question /guardrails does not: not am I close to the ceiling but what am I spending it on. It reports the session’s token split three ways — overall, per model, and per call — with each call’s timestamp, the model its price resolved against, and what it cost.

"overall": {
"input_tokens": 58864, "input_tokens_cached": 28804,
"input_tokens_uncached": 30060, "output_tokens": 504,
"turns": 3, "cost_usd": 0.0826,
"cost_usd_uncached_reference": 0.1201
},
"per_turn": [
{"turn": 2, "ts": "2026-09-19T14:30:09Z", "model": "claude-sonnet-5",
"input_tokens": 28850, "input_tokens_cached": 28804,
"input_tokens_uncached": 46, "output_tokens": 200,
"total_tokens": 29050, "cost_usd": 0.0079}
]

Four things about those numbers are worth knowing before you act on them:

  • A cache write is not a cache hit. Writing an entry bills at a premium over fresh input (1.25x on Anthropic’s 5-minute TTL), so a warming turn costs more than not caching at all. Written tokens land in input_tokens_uncached, because that is what they were billed as.
  • cost_usd_uncached_reference is the counterfactual, not a floor. It is what the same calls would have cost on a backend with no prompt cache at all, and it is deliberately allowed to fall below cost_usd: a workload that warms caches it never reuses is paying the write premium for nothing, and that is a thing you want to be able to see. Where the rendered /usage block would otherwise print cache saved $X, a session in that state prints cache cost $X, not yet repaid instead.
  • thoughts_tokens is a subset of output_tokens, never an addition. Thinking bills at the output rate and is already inside it. It reads zero on Anthropic models, which report no split — a genuine “did not say” rather than a measured zero.
  • turns and cost_usd cover the session; the token buckets cover this process. The durable spend ledger stores what a call cost and not what it was made of, so a session resumed after a restart reports its whole spend — the same figure the ceiling is enforced against, so /usage and /guardrails cannot disagree — with a breakdown that starts at the resume. per_turn is capped at the last 500 calls, and turn numbers are absolute, so a first row numbered higher than 1 is how you know rows were dropped.

The model server’s KV cache, for a self-hosted model

Section titled “The model server’s KV cache, for a self-hosted model”

When the session’s model comes from a provider profile whose server publishes KV-cache metrics (vLLM, SGLang or llama.cpp: a metrics_url, and metrics: vllm | sglang | llamacpp, which a profile extending the built-in vllm or sglang inherits), /usage also carries a kv_cache block. One poller per daemon reads the server’s Prometheus endpoint every five seconds, and each turn brackets itself with a reading at its start, taken before its first model call, and one at its end. mast reports the last turn’s window and the session’s:

"kv_cache": {
"fleet": true,
"last_turn": {"queries_tokens": 3843, "hits_tokens": 2112, "hit_rate": 0.55,
"peak_kv_usage": 0.0055, "readings": 3, "start": "…", "end": "…"},
"session": {"queries_tokens": 3843, "hits_tokens": 2112, "hit_rate": 0.55,
"peak_kv_usage": 0.0055, "readings": 3, "windows": 1, "start": "…", "end": "…"}
}
  • It is fleet-level, and says so. "fleet": true is always set: these are the server’s prefix-cache counters across all of its traffic while this session’s turns ran, not this session’s share. On a server only this session uses they match its input tokens; on a shared one they don’t, and nothing here pretends otherwise. It never enters cost, the token totals or a budget.
  • Counts are tokens, as the server reports them. A field the server doesn’t expose is left out, never shown as zero. reset means an engine restarted inside the window, and end_failed that the closing reading failed, so the window ends at the last good one. session folds every sampled turn (windows is how many) and carries no instantaneous gauge; last_turn may carry the server’s hit_rate_gauge.
  • A session’s windows don’t overlap. The end reading is taken before the turn returns, under the session’s turn lock, so the next turn starts where the last one ended and no token is counted twice.
  • Sampling costs a turn at most a fraction of a second. An edge uses the poller’s latest reading when it is under a second old, and otherwise takes one scrape capped at 300 ms. After a failed scrape the edges stop scraping for 30 seconds and use the latest reading instead, marking the window approximate (an edge up to one poll interval off the turn’s). A dead endpoint is logged once, and leaves the block out.
  • The profile’s credential is sent only to its own origin: same scheme and host as base_url, and a Google token only over https.

The judged eval tier reports the same window around each incident’s run, and for the whole corpus, when the model under test is such a profile.

The breakdown is folded from the same reading of the provider’s counters that priced the call, rather than derived a second time from the event stream, so the split you read is the split the money came from.

Attach can read transcripts and drive turns, so it gets its own token (MAST_ATTACH_TOKEN) and a hard rule: a non-loopback bind without auth is refused, not warned about. It also stays up through a shutdown drain, so an operator watching a finishing turn sees its last events rather than a dropped connection.

A session created over attach is owned by the caller who created it, and that ownership is what the per-session authorization matrix is built on: the owner reads and writes, viewers read, contributors read and write, and everyone else gets a 404 — not a 403, so an unauthorized caller cannot enumerate which sessions exist by the shape of the refusal.

GET /sessions/{id}/acl reads that list back and PUT replaces it:

{"owner": "alice@example.com",
"viewers": ["carol@example.com"],
"contributors": ["bob@example.com"],
"enforced": true, "persisted": true}

The PUT is a whole-document replace, not a patch. A list you omit is a list you cleared — which is the only sane reading, because the alternative (omission means “leave it alone”) would leave no way to remove the last viewer, and revocation is half of why the endpoint exists. Identities are trimmed and de-duplicated; a blank one is a 400 rather than a stored grant that could never match a caller.

GET sits at the read bar, not the admin bar. Anyone the ACL already admits can read the whole transcript, so the membership list is not the sensitive part — and a viewer who wants write access needs to know whom to ask. PUT is admin: the owner, or a daemon admin.

Two things the response tells you that the ACL itself cannot:

  • enforced is false when the daemon is not running multi-session, in which case the ACL governs nothing and every authenticated caller gets in. A PUT against such a daemon is refused with 501 rather than accepted: an amendment nothing consults would report success for an access restriction that does not exist.
  • persisted is false when the amendment lives only in this process — no ACL store wired, or a session registered without an owner (the daemon’s own bootstrap session is one). The grant works until the daemon restarts, and saying so is the difference between a durable decision and one that quietly evaporates overnight.

When persisted is true the grant outlives the session’s stay in memory. Idle sessions are evicted and rebuilt from disk on the next request, and a restart evicts all of them; either way the ACL that comes back is the one on disk, not one reconstructed by whatever rebuilt the session. That matters most for the grant you just made: a viewer added at 4pm is still a viewer after the overnight restart, and an owner whose session was swept out is not locked out of it by the act of asking for it again.

Ownership transfer — sending a different owner — is daemon-admin only. It is there for the case where the owner left, and it fails with a 403 for anyone else rather than being silently dropped, because an ignored field in an accepted request reads as a completed transfer. An owner cannot be cleared at all: a session with no owner is reachable by admins alone, which is a lockout rather than an edit.

--a2a-listen publishes an A2A agent card and a JSON-RPC endpoint (message/send, tasks/get, tasks/cancel, message/stream) for workloads that opt in with a2a.expose: true.

The opt-in is per workload and never automatic, because publishing an agent card is an external contract: you are telling other systems this skill exists and is callable. MAST_A2A_TOKEN enables per-skill scope enforcement; a non-loopback bind without a token is refused, since tasks/cancel is destructive.

Use it when another team’s agent — or another mast deployment — should be able to hand you work.

--agui-listen serves an AG-UI run endpoint per opted-in workload, plus a discovery descriptor, for CopilotKit apps and chat-platform bots. Also opt-in (agui.expose: true), also its own token (MAST_AGUI_TOKEN), with rate limiting, since a run drives a budgeted turn.

The one concept worth deciding up front is agui.session_model: per_thread (default) maps one continuing mast session to each AG-UI thread, which is what chat UX expects; per_run gives every run a fresh session, which is right for stateless one-shots. Either way the daemon derives and namespaces the session id — a client never supplies a raw one.

A threadId is the client’s own correlation string, not a secret, so on an authenticated endpoint the derivation folds in the caller: two principals naming the same threadId get two different sessions. Holding the endpoint’s scopes answers “may this caller run this workload”, which is a different question from “is this conversation yours” — without the caller in the id, any authorized principal could continue, and read back, another one’s thread. The separation is structural rather than a verdict: a second caller is not refused, they simply address a thread of their own and never reach the first. An endpoint with no validator has no subject to own anything, so its session ids are unchanged.

Read that at the grain it actually has. MAST_AGUI_TOKEN is a single token resolving to a single principal, so on a daemon holding one the caller folded into the id is a constant: the separation is between deployments, not between the people using one. Two browsers sharing the token share the thread, and the per-workload scopes published on the descriptor state what the workload requires rather than anything that distinguishes one token from another — there is only one. Supply your own token validator and the same machinery separates users, with no change to the server.

Two publication surfaces sit behind their own bundle keys, both empty-or-off by default and both silent rather than redacted when off: agui.state_projection names the session-state keys a run may publish as StateDelta patches, and agui.emit_reasoning lets the workload stream the model’s thinking as REASONING_* frames. Neither is a display preference — the client on the other end is a browser — which is why the defaults publish nothing and the daemon logs what a running workload has enabled.

Because both are per-bundle, a client cannot infer either from the protocol version, and inferring them from a finished run is hopeless: a stream with no reasoning in it looks the same whether the workload publishes none or the model simply didn’t think. So the public discovery descriptor states them up front, as a capabilities object per workload. It is read from the same value the run’s emitter is built from rather than recomputed from the bundle — a capability claim that restates a config instead of reading what it describes is how /tools and /perms each spent releases advertising something untrue.

A third frame family, the StepStarted/StepFinished bracket, has no key and no capability bit, and the contrast is the useful part. A step is named after the agent that authored that stretch of the run, so it tells a client where a coordinator handed off to a specialist — and that is something a caller permitted to run already sees, because the handoff also arrives as a transfer_to_agent tool call naming the same agent. State and reasoning are gated because they disclose something new; steps describe what is already on the wire, so gating them would be a switch with nothing behind it.

AG-UI has delta frames — TEXT_MESSAGE_CONTENT carries a delta, so does TOOL_CALL_ARGS — and mast sends each of them exactly once per message, carrying the whole thing. An answer arrives as one TEXT_MESSAGE_START/CONTENT/END triad holding the entire text; a tool call arrives as one TOOL_CALL_START/ARGS/END triple holding the entire argument object. A client written to append deltas is correct and will simply receive one. Nothing arrives twice, and no TOOL_CALL_END ever describes a call the model had not finished writing.

That is a promise about completeness, not a claim to stream. Turning the answer into per-token frames, so a browser can render it as it is produced, is #407 and is deliberately not built yet: mast runs every model non-streaming today, so there is nothing to forward, and the contract above is what a client can build against until there is.

If the SSE connection drops — a closed laptop, a proxy timeout, a reload — the run keeps going. It finishes, and everything it did lands in the session transcript. Earlier releases cancelled the turn along with the request, which reads like the polite thing to do and is not: a drop that lands while a tool is mid-flight leaves the side effect done and no record of it, so the retry the client naturally makes applies the change a second time. A turn is work, and a browser going away is not a decision to abandon work.

The consequence runs the other way too, and it is the part to plan for. Closing the tab stops nothing. A run with no reader left is bounded by the workload’s budget.max_wallclock_seconds, by the watchdog, and by an explicit mast sessions pause <id> --cancel-turn — the same three that bound a run the daemon started on a schedule with nobody watching it. There is no AG-UI verb for “stop” yet; if a client needs to mean that, it means pause --cancel-turn.

Rejoining the stream is a different feature and is not built: reconnecting with the same threadId starts a new run rather than resuming the view of the old one. What a disconnect costs today is the tail of the event stream, not the work.

AG-UI lets a client declare tools of its own in RunAgentInput.tools — things the browser will execute and report back on, like highlighting a range in an editor or filling a form the user is looking at. mast parses that field and ignores it. No workload can call a client-declared tool today.

That is deliberate rather than unfinished. Everywhere else in mast the operator decides what an agent may call and the caller decides only what to ask for; a tool declared per run inverts that, and the tool’s name and description go into the model’s context, so an unguarded version is an injection surface before it is a feature. The gate is designed and lands in a later release: a workload opts in through its bundle, the opt-in bounds the class rather than listing names — an operator cannot know what a frontend will ship next week — and client tools count as mutating by default, so a workload with human approval configured parks for one before the browser is asked to do anything. They will be callable by the workload’s root agent only, not by its specialists.

A thread runs one turn at a time. Send a second run while the first is still going and it waits — a mast session is a single-writer store, so every turn-driving surface serializes on the same per-session lock.

The wait is bounded. A thread admits agui.run_queue.depth waiting runs on top of the executing one — three by default, so four in all — and the next one is refused with 409 and a Retry-After before the stream opens. That is a different answer from the 429 the rate limiter gives: 429 is your arrival rate and means slow down everywhere, 409 means this conversation is busy and you can retry it or use another thread. Set depth: 0 if you want no concurrency on a thread at all.

So a client that stacks runs behind a slow turn gets told so. It is still better manners to wait for RunFinished before sending the next run on the same thread: a queued run is a turn you have committed to, and until the 409 there is nothing distinguishing a deep queue from a fast one.

The surfaces above are inbound. invoke_remote_agent is the outbound counterpart: a specialist calling another agent over A2A.

It classifies as mutating, unconditionally. Effects on the far side of a federation call are invisible from here, and a call whose consequences you cannot see is not one to fire unattended — so it goes through the write gate like any other change.

No caller at all — the workload’s own clock

Section titled “No caller at all — the workload’s own clock”

Every surface above answers “what reached the daemon”. A workload that declares edge_trigger.scheduled answers nothing: it wakes on its own interval, and the run has no caller to take an identity from. That is why it runs as mast:scheduler — a namespaced identity no human login can produce — and why a mutating call inside a scheduled run still parks for a real approver. The trigger is a bundle field rather than a surface: nothing is listening, so there is no token and no trust boundary to configure.

Which to use is a question about where your schedules should live, not about capability. An external scheduler posting to /inject gives you one place to see every job your org runs and one place to change them; the bundle’s own cadence gives you a workload that is still periodic after it is copied to a cluster where that scheduler does not exist.

The caller is…Surface
Alertmanager, CI, a runbook curlinject
A scheduler you already run — Cloud Scheduler, a CronJobinject (it is just another caller)
Nothing — the workload should run periodically on its ownedge_trigger.scheduled, no surface
A human operator watching or steering an incidentattach (mast-web)
Another agent, in or out of your orgA2A
A chat UI or CopilotKit app with end users in itAG-UI
mast calling out to another agentinvoke_remote_agent

Wire shapes, flags, and auth details for all of them: CLI reference. Opt-in fields and per-skill policy: workload bundle.