Interop surfaces
An unattended agent is only useful if something can reach it. mast exposes several surfaces rather than one, because “an alertmanager webhook”, “an operator watching a live incident”, “another team’s agent”, and “a chat UI” are genuinely different callers with different auth models — and collapsing them into one endpoint means the weakest one sets the bar.
Every surface is off by default except inject, and each carries its own token. That is deliberate: they are separate trust boundaries, not four views of one.
What none of them publish
Section titled “What none of them publish”A model’s thinking is not part of any answer mast gives. Frontier models
return their reasoning as a distinct kind of content — Anthropic calls it a
thinking block, Gemini a thought part — and it comes back in the same field
as the answer, marked rather than separated. mast reads the mark. Reasoning
is replayed to the provider on the next turn, because a signed thinking block
has to come back intact for the model to continue, and it goes nowhere else:
not into the A2A result artifact, not into an AG-UI TextMessage frame or
RunFinished.result, not into the /parks projection, not into a specialist
result the planner reads, and not into the daemon’s own operator log.
This is a floor, not a policy, and one surface has already asked: an AG-UI
workload that sets agui.emit_reasoning
streams the thinking as its own REASONING_* frames. That does not weaken
the floor — it is a deliberate read of the thought parts, by name, rather
than the removal of a filter, so the answer stream is unchanged whether the
opt-in is on or off and “reasoning reached a browser” stays a grep for one
symbol. Off by default, and off publishes nothing at all rather than an
empty placeholder. What is ruled out is publishing reasoning by forgetting
to check.
The one thing no opt-in reaches is the thought signature — the opaque blob a provider wants back to continue. It is a replay credential, not a thought, so it is replayed to the provider and modeled nowhere on any wire.
Inject — the machine trigger
Section titled “Inject — the machine trigger”The daemon’s own HTTP endpoint (--listen, default :7777): /inject,
/resume, /abort, /parks, /metrics, /healthz. This is how an incident gets in — an
Alertmanager webhook, a Cloud Scheduler job, a CI step, a curl in a
runbook. Bearer auth via MAST_INJECT_TOKEN; unset means unauthenticated,
which is a dev-only posture and is warned about at startup.
If you are wiring monitoring to mast, this is the surface. The others are additions to it.
A Cloud Scheduler job or a Kubernetes CronJob posting here still works, and it remains the right answer when you already run one — your schedules live in one place and mast is just another target. What changed in v0.4 is that you no longer need one to run something periodically: see no caller at all below.
Asking whether it is ready
Section titled “Asking whether it is ready”Two routes answer that, and they answer different questions.
| route | asks | can it go red? |
|---|---|---|
GET / | is this process still accepting connections? | no — it consults nothing and is a static 200 |
GET /healthz | can it still do the job it accepts work for? | yes — 503 when a dependency is down |
GET /healthz runs the daemon’s registered checks and answers 200 {"ok":true,"checks":{"session_db":"ready"}} or 503 with that check
failed. Today the one check is the session database, registered when
--session-db is set: a real read against the table ADK writes events to,
not a pool ping, so it catches a dropped table, a lock that outlasts the
busy timeout, a closed pool, a Postgres that went away, and a volume
that was unmounted out from under a running pod. It does not catch a
filesystem that went read-only — a read still succeeds there.
A daemon with no session store reports 200 {"ok":true,"checks":{}}. The
empty object is the point: “the process is up and has nothing durable to
vouch for” is a different statement from “the database is fine”, and an
in-memory daemon that answered 503 would never join its Service.
The route is unauthenticated, because a kubelet has no bearer token and
readinessProbe has nowhere to put one. So the body carries only ready
or failed per check — never the error, which routinely names a
filesystem path or a DSN host. The error goes to the daemon’s log, once
per transition rather than once per probe.
Point readinessProbe at /healthz and leave livenessProbe on /.
That split is deliberate. The action behind a failed liveness probe is a
restart, and a restart does not fix a deleted volume or an unreachable
database — it turns one unready pod into a crash loop that also kills
every in-flight turn. Readiness is the probe that should be able to go
red: it takes the pod out of the Service, which stops it being handed
injects it cannot durably record. The chart’s daemon StatefulSet
(charts/mast/templates/statefulset-daemon.yaml) ships wired that way.
Reading what is parked
Section titled “Reading what is parked”POST /resume has been able to answer a parked approval since v0.2. What
no out-of-process caller could do until v0.9 was find out what it was
answering — so a chat bot or a web UI wiring up an Approve button could
only offer “approve the thing, whatever it is”. That is uninformed consent
with an audit trail, which is worse than the CLI it replaces, because it
looks like the opposite.
Two read-only routes close that:
| route | returns |
|---|---|
GET /parks | every parked session, each in full, oldest question first |
GET /parks/{session} | the named session whether it is parked or not — 200 with an empty parks list when it is not, 404 only when no such session exists |
Both need --session-db; a daemon without a session store answers 404
rather than an empty list, because “nothing to approve” and “I cannot see
parks” are different answers and only one of them means it is safe to stop
looking. Auth is the same rule as /resume: the shared token, or a user
from the token table. A caller who may answer a park may read it, and a
caller who may not read it has no business answering one.
The body is always an object with a sessions array — for the
single-session route too, and when nothing is parked. Each session carries
its state, an awaiting (approval or input, an approval outranking a
question, the same ranking turn_state uses), the
open parks, an operator hold if one is active, and the applied calls
the gate captured a revert for.
An approval park carries a change: the tool, the arguments, the rendered
key, the gate policy that parked it, the proposing specialist, the
verdict_format the gate will accept, a stale line when a grant the
operator already gave no longer covers the call — and the change_set the
call belongs to, when it belongs to one, because an operator cannot
sensibly trade scope: change_set for a set they have not been shown. A
question park carries none of that: a question is not a change.
What it does not carry is as much of the contract as what it does. No
model output — no reasoning, no narration, none of the assistant text
around the call. No tool results, including the result of the read that
produced a capture: only that read’s name, its digest, and the declared
field names travel, because the values are cluster state and can be
anything. And nothing the write gate did not park on. The projection is
written out field by field in internal/cli/parks.go, so a field added to the
transcript cannot reach this wire without someone deciding it should.
A hold is reported beside the parks and never as one. Nothing resolves by
answering a hold, so a client that rendered it as a question would put a
prompt in front of somebody with nothing to say back.
Attach — the operator’s live view
Section titled “Attach — the operator’s live view”--attach-listen serves the mast-native attach protocol (HTTP + SSE) that
mast-web speaks: live-tail a
running turn, read a session’s transcript, inject an operator message,
answer a parked approval. It needs a durable session store, since the live
tail pumps from the event log — so it implies one: with no
--session-db of your own, the daemon opens ~/.mast/sessions.db and says
so at startup. Name a path to put it anywhere else.
The normative response shapes
Section titled “The normative response shapes”Prose is not a contract. The JSON these routes return is pinned as
fixtures in
internal/attach/testdata/conformance/ —
rest-*-v1.json for the plain HTTP bodies, the SSE event fixtures beside
them for the stream. Write a client against those, not against a mock you
wrote yourself: mast-web’s bundled mock invented snake_case names for the
sessions list, its client was written against the mock, every one of its
tests passed, and the list rendered empty against every real daemon
(mast-web#41). A fixture is the thing a mock can be checked against.
Three shapes a client gets wrong by reading the examples rather than the fixtures:
last_touched_atandnext_wake_atare always on the wire, as0001-01-01T00:00:00Zwhen they do not apply. They are taggedomitempty, which does nothing to atime.Time. Read them for the zero value; presence does not mean the field applies.GET /sessionswith nothing to show is{"sessions":[]}, nevernull. So areviewersandcontributorson an ACL. Everything else optional is genuinely absent.- On
GET .../usage, a cache write counts as uncached input — it bills at a premium over fresh input, not a discount — andcost_usd_uncached_referenceis a counterfactual, not a floor. A turn that warmed the cache and has not reused it yet reports a reference below its cost. That is correct, not a rounding error to hide.
The fixtures’ own README lists which routes are pinned and, more usefully, which are not.
Opening a session
Section titled “Opening a session”POST /sessions opens an empty session and answers 201 with its id
(attach-<16 hex>) and URL. The client then injects into it. Each one runs
the daemon’s workload, exactly like a session an inject created. That is the
shape of a client that starts its own conversations, not one that reacts to
events: k8s-lookout’s sinks open one per incident, and switchboard opens one
per chat thread. Before #524 every mast daemon answered 501 here: the
route existed, but mast serve never configured the factory behind it.
The daemon is still single-user. A session created here has no owner and no ACL, the same trust model as an inject-created one. Give each user or conversation its own daemon (or sandbox) when they must not see each other’s sessions.
What turn_state says
Section titled “What turn_state says”Every status-update frame carries a turn_state, and a client that
renders “is this session busy, or does it need me?” reads that and nothing
else. It has four values:
turn_state | means |
|---|---|
streaming | a turn is running right now |
awaiting_permission | the write gate parked a mutating call; somebody has to approve or reject it |
awaiting_elicit | the session asked a question — request_operator_input — and is waiting for the answer |
idle | none of the above |
The distinction matters because a parked session finishes its turn.
The gate does not block inside the call; it records the question durably
and returns, so from the daemon’s point of view nothing is running. Before
v0.9 that is exactly what the wire said: a session waiting on a human
reported idle, the same string a session that finished its work reports,
and a gateway rendering an approval card had no frame to hang it on.
Two things follow from where the answer comes from. It is read from the
transcript, not from in-memory turn state, so it survives a daemon
restart — a session parked yesterday still reports awaiting_permission
today. And it is in the boot snapshot, the status-update every client
receives on connect, so a client that attaches an hour after the park sees
it without having been present for the frame.
Two cases are deliberately not awaiting_*. A session paused by
pause_session reports idle: that is a hold an operator placed, not a
question anyone is waiting on an answer to. And a session resuming from a
park reports streaming for the whole resume turn, even though the
interrupt it is answering stays open on the transcript until mid-turn —
a live turn outranks the park it is resolving, because reporting otherwise
would freeze the session on an operator’s screen while it works.
One ordering guarantee comes with this. turn-complete and turn-error
are now held until the turn’s own event-log frames have reached every
subscriber. They previously raced: the terminal frame went straight to
subscribers the moment the turn returned, while the answer travelled
through the event log and the pump, so a client that finalized its render
on turn-complete could drop the last message. If the log cannot be
queried, or does not catch up within two seconds, the terminal frame is
released anyway and the delay is logged — a late frame is a correctness
bug, a missing one is worse.
Answering a park over attach
Section titled “Answering a park over attach”awaiting_permission tells a client there is a question; two routes let it
carry the answer back without leaving the attach protocol:
| route | does |
|---|---|
GET /sessions/{app}/{id}/perms/stream | SSE; one prompt event per open approval park |
POST /sessions/{app}/{id}/perms/respond | {"id": "<interrupt id>", "decision": "allow-once"} |
GET /sessions/{app}/{id}/perms | the rules this session runs under, and what it has already decided |
This is a second door onto POST /resume, not a second approval model.
A press resolves the same durable park, is attributed to the same
authenticated caller, and is subject to the same rules — including the ones
that make this surface narrower than the protocol it inherited:
decisiontakesdenyandallow-once. The four wider values the protocol defines are refused 400 by name rather than narrowed to allow-once, the same refusalscope: sessiongets on/resume. A button that means less than it says is a bad button; one that means more is worse.- Answering a park that is already resolved is a 404. Durability cuts both ways: a question that outlives the process can also be answered by someone else first.
- A
200is{"acknowledged": true, "approver": "<identity>"}. Whenapproveris absent, mast recorded nobody — do not render a name. perms_streamin the capability report is true exactly when these routes will answer. A daemon that cannot serve them answers501.
Before v0.9 the routes existed and every mast daemon answered 501: they
came with a ported package built for a synchronous prompt at a keyboard,
and mast has no keyboard. They now sit on the mechanism mast does have.
A park carries kind: control_plane_write, which is the field a client
should switch its button set on.
The read beside them, GET /perms, has its own capability flag —
perms, not perms_stream — because they are separate wirings. A daemon
can say what its rules are and what it has adjudicated without ever
streaming a live question, and mast is that daemon. The read answers
mode (the permissions gate, absent when there is none), on_mutation
(the bundle’s write-gate policy, on its own axis), and the session’s
durable approval log. Through v0.8 it answered 200 with the zero value
on every mast daemon, which is a description of something that gates
nothing; it now refuses instead, and
the write-gate reference
spells out the fields.
GET /sessions/{id}/tools lists the tools the daemon actually holds, each
with a source, the MCP server it came from if it has one, and a
gate_state — what the write gate would do to a
call of it: allowed, prompted (it parks for approval), or denied
(on_mutation: dry_run, so it will never run).
Two sources appear. builtin is the daemon’s own control plane — under
planner dispatch, invoke_specialist, run_shape_llm_router,
run_shape_fan_out_fan_in, request_operator_input, and
pause_session. mcp is everything a server
declared. Builtins sort ahead of the servers and are always listed: unlike
a server, they cannot fail to answer or vanish between polls. The MCP half
is read from the live servers rather than from the bundle’s declaration,
and refreshed at most every 30 seconds; a server that fails to answer is
left out and logged rather than blanking the rest — including the
builtins, which stay listed when every server is down.
What is still missing is the handful of tools ADK installs itself:
finish_task, and a coordinator’s per-specialist transfer tools. mast
never wires those, so it cannot enumerate them without keeping a
hand-written list — and a catalog naming tools that do not exist is worse
than one omitting tools that do.
GET /sessions/{id}/subagents lists the roster the daemon loaded —
what this thing can do. (/agents lists what has been spawned, which for
mast is always empty: every dispatch shape resolves its specialists inside
the turn.) Each entry carries the specialist’s description, its model:
override if it declared one, its declared capability (read_only or
change_executor), its agent_mode (Task or SingleTurn), and an
invocation — how the composed root actually reaches it:
invocation | means |
|---|---|
parent_tool | the planner calls it via invoke_specialist |
transfer | coordinator dispatch hands the turn to it |
graph_node | a node in a graph, or fan-out’s synthesis merger |
fanout_branch | one of fan-out’s concurrent analysts |
| (empty) | nothing in this shape reaches it |
That last row is the useful one. A roster can carry a member the composed
shape never routes to — a _fallback under fan-out dispatch, a second
SingleTurn spec under graph dispatch — and an empty invocation is how
you see it without reading the composition code.
Each entry also carries a tools grant, which is the other half of what
can this thing do: capability says whether the specialist is allowed to
change anything, tools says what it can reach.
"tools": { "mcp_grant": "listed", "mcp": [ {"server": "gke", "tools": ["get_pod", "get_events"], "whole_server": false}, {"server": "slack", "whole_server": true} ], "builtin_declared": ["apply_manifest"]}mcp_grant is the field to read first, because the underlying declaration
means opposite things one character apart. A spec with no mcp: key
inherits every MCP toolset the workload has ("all"); a spec that writes
mcp: [] is denied all of them ("none"); a non-empty list is a
whitelist ("listed", and only then is mcp present). Both of the first
two spellings are an empty list on the wire, so a catalog that just
transcribed the declaration would show the deny-all specialist and the
reach-the-whole-cluster one identically. whole_server draws the same
distinction per entry: a listed server with no tools: of its own passes
whole.
builtin_declared is named for what it is: nothing in the declaration
installs a tool, so the list grants and narrows nothing. What reads it is
the capability-split check, which refuses a read_only specialist that
lists a mutating name, the fan-out branch check, which does the same for an
analyst, and the startup log that reports a workload’s declared write
surface. Read it as a claim the spec makes about itself and is held to, not
as a set of tools the specialist holds.
The one built-in mast does install on a Task specialist is
retrieve_raw,
and it deliberately does not appear here: it is a property of the daemon’s
--mcp-digest setting rather than of the roster, and it returns bytes the
specialist has already received, so it widens no grant this catalog
describes.
GET /sessions/{id}/guardrails answers the question an operator actually
has when a session stops responding: what stopped it, and what do I do?
It reports the budget ceilings in force and the usage against them across
all three dimensions plus each specialist’s own, and the watchdog’s posture
(advisory: true under --watchdog=warn and under the default
--watchdog=feedback, which corrects and can end a looping turn but never
halts the session; false under --watchdog=enforce, whether or not it
has fired yet).
The signals
Section titled “The signals”The watchdog raises three signals by default, and each names a different way an unattended run goes wrong:
| signal | severity | fires when |
|---|---|---|
repeated-tool-call | Critical | the same call, five times running — path spellings (./main.go, /workspace/main.go) count as the same file |
alternating-tool-cycle | Critical | a short loop repeating, e.g. list_agents → check_agent three times over — the shape a consecutive-repeat check is structurally blind to |
tool-failure-streak | Warn | three calls in a row all returned errors, so nothing the agent ran has verified the state of anything |
The last one is the one to read carefully in a report. It does not mean the agent was stuck; it means the agent’s conclusions rest on tools that all failed. It stays Warn deliberately: three denials into a legitimate RBAC probe is not a runaway, and halting there would make the backstop the outage.
A fourth signal, dominant-tool-call, ships and is not wired by
default. It covers the shape between the first two — one call repeated
with occasional others wedged in (a a a b a a a c a a a), which resets
the repeat detector’s run and shows the cycle detector no repeating
block. Density does not care where the interleaves fall, so it reaches
the verdict inside one twelve-call window instead of waiting for the
interleaves to stop; when they are the whole delay, that is most of a
loop’s cost. It is off by default because a third Critical detector
changes what every unattended workload is told about itself under
feedback, and a polling workload with any variation in it is exactly a
dominant call with interleaves. Nothing turns it on today: the watchdog
is internal since #301,
and until then only an embedder composing its own runner could hand it
a signal list. When it runs, it stands down wherever one of the other two
already owns the shape, so one loop still produces one alert.
Severity is a property of the pattern, not of the posture. What changes between postures is the reaction, and the three of them are a ladder — each rung includes the one below it:
--watchdog=warnlogs every alert and lets the turn run.--watchdog=feedback(default) also tells the model. Each alert carries a model-facing sentence alongside the operator-facing one, and on the session’s next turn those are prepended to the prompt as a[watchdog]block: an automated observation about your own previous turn — this is not a message from the user. Every posture below this one routes the observation to a reader who may not be there; the model about to repeat the call is the party that can decide not to. Inside a turn, a call repeated with the same arguments and the same result is told on its result and the turn ends if the model keeps going — see a loop inside one turn. Across turns it is a correction, not a backstop: nothing halts a session whose model reads the block and loops anyway, which is why a workload with a bounded tool loop still wantsenforce.--watchdog=enforcealso cancels the turn in flight on a Critical alert and refuses the session’s every subsequent turn — auto-resume, a scheduled fire, and an attach inject all included — until an operator resets. The cancel happens during the turn, not at its boundary: the loop the watchdog catches usually lives inside a single turn, and a reaction that waits for the turn to end waits for the thing it is supposed to stop.
A turn cancelled that way — and a turn an operator stops with
POST /sessions/{id}/interrupt — comes back on the stream as a turn-error
of kind canceled, with retryable: false. That is the flag a client keys
its “run it again” affordance off, and both of these are somebody deciding
the turn should stop; offering to re-drive the loop the watchdog just halted
would be the wrong end of the decision. A turn that ran out of time is
still transient_network and still retryable, because nobody asked for that
one.
The halt itself — the refusal the tripping turn returns, and the one every
turn after it gets until a reset — comes back as kind watchdog_halt,
also retryable: false, with a hint naming the reset endpoint. It is a
separate kind from canceled because it describes a standing state rather
than one stopped turn: retrying a canceled turn is arguably the operator’s
call, while retrying a watchdog_halt fails identically until somebody
clears it. Both are separate from cost_ceiling, whose remedy is a bigger
budget rather than a fixed loop.
refusal_loop is the fourth of these, and the distinction it turns on
is the one a client acts on. It ends a turn in which the model kept
re-proposing a call an operator had already refused — see a refusal the
model cannot talk its way
past.
Also retryable: false, and deliberately not a watchdog_halt: nothing
latched, no guardrail tripped, and the next turn starts clean. A client
that rendered the halt’s remedy here would send an operator off to clear
a guardrail that never tripped. Its hint says so, and says what to do
instead — if the change should happen after all, send a new turn and
approve the call when it parks.
loop_stop is the fifth. It ends a turn in which the model kept
repeating one call after being told, inside the turn, that it was looping
— see a loop inside one turn. Also
retryable: false, since re-driving the turn hands the same model the
context that looped, and like refusal_loop it latches nothing. Its hint
points at what an operator can change: the tool’s output or the model.
enforce including feedback is deliberate. An enforce halt is cleared
by an operator reset, and a reset resumes a model whose context still ends
in the loop it was halted for; without the injected observation the very
next turn re-issues the same call, and the reset is a treadmill. For the
same reason, a reset clears the halt but keeps the queued observation.
The [watchdog] block is steering, not a trust boundary — nothing
downstream grants authority based on it, and a user prompt may contain the
literal string.
Where the posture comes from
Section titled “Where the posture comes from”Three sources, in order: --watchdog beats the bundle’s
safety.watchdog beats mast’s default. The daemon logs which one won at
startup (watchdog posture resolved mode=… source=…), because a posture
nobody can see is a posture nobody audits — and enforce, the one an
operator most needs to know is armed, otherwise announces itself only by
refusing a turn.
safety: watchdog: enforceThe bundle is where a workload ships its own backstop: mast’s deployment unit is the bundle, and without this every invocation and every deploy manifest had to carry the flag by hand. The flag sits above it so an operator debugging a halted workload can drop the posture for one run without editing — and later forgetting to revert — the deployed manifest.
The default is feedback. Not warn: every mast run is unattended,
so warn routes the alert to a log nobody is tailing, which is
indistinguishable from off. Not enforce either: alternating-tool-cycle
has a workload-shaped false positive — a scheduler-driven daemon watching
a rollout settle calls the same tool with the same arguments on purpose —
and on an unattended deployment a false halt is an outage that waits for
the morning. A false feedback costs one paragraph the model is free to
disregard. Recoverable beats unrecoverable when nobody is watching. Set
safety.watchdog: enforce on a workload whose tool loop is bounded by
construction — a triage run that reads, concludes, and stops.
The library-embedded surface (mast.RunWorkload) reads the same
safety.watchdog field and taps the same signals, with the rungs bounded
by what that surface holds: enforce abandons the runaway turn, but there
is no cross-call session state for the “refuse every later turn” half, and
no next turn for feedback to inject into — though its in-turn loop guard
works there as it does everywhere.
A loop inside one turn
Section titled “A loop inside one turn”The [watchdog] block arrives on the session’s next turn, and the loop
the watchdog exists for usually lives inside a single one. Measured: a
self-hosted Gemma 4 called one tool with identical arguments 441 times in
one turn until the server refused the context
(#514). The repeat signal
fired at five, the observation was queued for a turn that never came, and
nothing reached the model.
So under feedback and enforce, a loop guard also acts inside the turn,
on the tool side:
- From the fifth consecutive call to one tool with the same arguments
and the same result, the result carries a
watchdogfield: an automated observation that the result will not change, what to do instead, and how many more repeats end the turn. - The third repeat after that is not made, and the turn ends as
loop_stop— amast_turns_totaloutcome and a turn-error kind of its own. Nothing latches: the next turn starts clean, with the usual[watchdog]block in front of it.
The result has to match as well as the arguments, which is stricter than
the signal. The signal can count calls alone because under feedback its
false positive costs a paragraph; this one ends a turn, and the false
positive named below — a daemon polling a rollout with the same call on
purpose — is exactly a run of identical calls whose results change. A
changed result starts the count again.
Under enforce the session halts at the fifth call as before, so the
guard’s stop is never reached; under warn the repeated-call guard is off.
A run of identical calls is one shape of runaway. For the rest — a cycle
through two or three calls, a repeat with other calls wedged in,
exploration that never converges — the same guard enforces
budget.max_model_calls_per_turn, default 100, under every posture:
the call over the cap is not made and the turn ends as loop_stop
(#519). It is a budget
rather than a detector, so it needs no price and recognizes nothing; it
just stops counting at a number no healthy turn on the judged corpus came
near. It runs on
the daemon, a one-shot, and mast.RunWorkload alike, and in the eval
harness’s judged tier, so a looping model is scored the way a daemon
would run it. It does not yet reach inside a planner dispatch, whose
sub-runner carries no plugins
(#235).
It watches inside a planner dispatch too
Section titled “It watches inside a planner dispatch too”Under planner dispatch, invoke_specialist runs each specialist on a
runner of its own, and a private runner is a private event stream. Through
v0.4 that meant a specialist spinning inside one dispatch emitted its calls
where the watchdog could not see them — --watchdog=enforce could not halt
a loop it could not observe, which is the exact shape it exists to catch.
Since v0.5 the dispatch feeds those events to the session’s watchdog,
the same one the outer stream feeds.
The session’s, deliberately, and not one minted per dispatch: the signals
count repetition, so a per-dispatch watchdog would reset the count every
time a dispatch ended and a specialist making three identical calls in each
of ten dispatches would never reach a threshold of five. The cost is worth
knowing — the planner’s own calls and its specialists’ now land in one
signal set, so an invoke_specialist between two dispatches breaks a
consecutive run the repeat detector was building. That is the interleaved
shape alternating-tool-cycle and the opt-in dominant-tool signal exist
for, and it is the same trade-off the outer stream already makes between a
coordinator and its sub-agents. The loop this catches — a specialist
spinning inside one dispatch — has no interleaver at all.
A trip here halts the session, not just the dispatch. This is the
deliberate difference from a crossed budget cap, which stops only the
dispatch and hands the planner the reason
(budgets).
A ceiling is
cumulative, so stopping the sub-run is the whole remedy. A watchdog trip is
a latch meaning this session is behaving pathologically and an operator
must reset it, and every other door to it refuses the whole session; a
halt that stopped only the dispatch would let the planner re-dispatch the
same specialist immediately, which is the treadmill enforce exists to
break. So both fire: the dispatch stops with a labelled partial, and the
turn is cancelled. Under warn and feedback nothing is cancelled — the
alert is retained and reportable, and the model-facing paragraph goes to
the planner, since the specialist’s sub-session is already gone and the
planner is the one that would otherwise dispatch again.
Library embeds that compose the planner themselves get this by wiring
compose.RootConfig.SubRunObserver, the same seam the meter uses;
mast.RunWorkload and the daemon already do.
A halt outlives the process that observed it
Section titled “A halt outlives the process that observed it”An enforce halt is written to the session database, and a daemon that
restarts adopts it on the halted session’s next turn — before any model
call, whichever surface drives that turn. A halt a restart cleared would
not be a halt: mast’s restarts are automatic and unattended, so the loop →
halt → crash → restart cycle enforce exists to break would simply
resume, each restart handing the loop a clean backstop. The reset is
durable in the same place, so clearing a halt clears it for good, and the
row records who cleared it and what runway they added.
Two things to know about it:
- It needs
--attach-listen(which implies--session-db, so the store the halt is written to exists exactly when the reset does). The reset endpoint is attach-only, so persisting a halt on a daemon with no attach surface would leave an operator no way to clear it. Starting with--watchdog=enforceand no attach listener logs a warning saying exactly that. - The posture still wins over the history. A deployment dialed back
from
enforcedoes not inherit a halt it would no longer produce.
Restore fails open: an unreadable guardrail table logs a warning and the turn runs. A storage fault must not halt every session in the deployment with no trip behind it, and the per-turn backstops are all still armed.
Budget spend is durable in the same place and fails open the same way —
a ledger of priced calls, folded back before the first turn after a
restart — but not on the same terms: it needs --session-db and not
--attach-listen. A ledger is not a latch, so the reset-endpoint argument
above does not reach it, and there is nothing an operator has to be able
to clear. Through v0.6.0 it inherited that argument anyway by riding the
same database handle, which denied a durable ceiling to unattended daemons
— the deployment least likely to bind an operator socket and most likely
to crash-loop (issue #274). See
spend survives a restart.
POST /sessions/{id}/guardrails/reset is the way out: a budget trip is
otherwise permanent, since enforcement is re-derived from usage against the
ceiling on every priced event. See
getting unstuck after a trip
for what a reset does and the three things it deliberately refuses to do.
Where the tokens went
Section titled “Where the tokens went”GET /sessions/{id}/usage answers the question /guardrails does not:
not am I close to the ceiling but what am I spending it on. It reports
the session’s token split three ways — overall, per model, and per call —
with each call’s timestamp, the model its price resolved against, and what
it cost.
"overall": { "input_tokens": 58864, "input_tokens_cached": 28804, "input_tokens_uncached": 30060, "output_tokens": 504, "turns": 3, "cost_usd": 0.0826, "cost_usd_uncached_reference": 0.1201},"per_turn": [ {"turn": 2, "ts": "2026-09-19T14:30:09Z", "model": "claude-sonnet-5", "input_tokens": 28850, "input_tokens_cached": 28804, "input_tokens_uncached": 46, "output_tokens": 200, "total_tokens": 29050, "cost_usd": 0.0079}]Four things about those numbers are worth knowing before you act on them:
- A cache write is not a cache hit. Writing an entry bills at a
premium over fresh input (1.25x on Anthropic’s 5-minute TTL), so a
warming turn costs more than not caching at all. Written tokens land
in
input_tokens_uncached, because that is what they were billed as. cost_usd_uncached_referenceis the counterfactual, not a floor. It is what the same calls would have cost on a backend with no prompt cache at all, and it is deliberately allowed to fall belowcost_usd: a workload that warms caches it never reuses is paying the write premium for nothing, and that is a thing you want to be able to see. Where the rendered/usageblock would otherwise print cache saved $X, a session in that state prints cache cost $X, not yet repaid instead.thoughts_tokensis a subset ofoutput_tokens, never an addition. Thinking bills at the output rate and is already inside it. It reads zero on Anthropic models, which report no split — a genuine “did not say” rather than a measured zero.turnsandcost_usdcover the session; the token buckets cover this process. The durable spend ledger stores what a call cost and not what it was made of, so a session resumed after a restart reports its whole spend — the same figure the ceiling is enforced against, so/usageand/guardrailscannot disagree — with a breakdown that starts at the resume.per_turnis capped at the last 500 calls, and turn numbers are absolute, so a first row numbered higher than 1 is how you know rows were dropped.
The model server’s KV cache, for a self-hosted model
Section titled “The model server’s KV cache, for a self-hosted model”When the session’s model comes from a provider
profile whose server publishes
KV-cache metrics (vLLM, SGLang or llama.cpp: a metrics_url, and
metrics: vllm | sglang | llamacpp, which a profile extending the built-in
vllm or sglang inherits), /usage also carries a kv_cache block. One
poller per daemon reads the server’s Prometheus endpoint every five seconds,
and each turn brackets itself with a reading at its start, taken before its
first model call, and one at its end. mast reports the last turn’s window
and the session’s:
"kv_cache": { "fleet": true, "last_turn": {"queries_tokens": 3843, "hits_tokens": 2112, "hit_rate": 0.55, "peak_kv_usage": 0.0055, "readings": 3, "start": "…", "end": "…"}, "session": {"queries_tokens": 3843, "hits_tokens": 2112, "hit_rate": 0.55, "peak_kv_usage": 0.0055, "readings": 3, "windows": 1, "start": "…", "end": "…"}}- It is fleet-level, and says so.
"fleet": trueis always set: these are the server’s prefix-cache counters across all of its traffic while this session’s turns ran, not this session’s share. On a server only this session uses they match its input tokens; on a shared one they don’t, and nothing here pretends otherwise. It never enters cost, the token totals or a budget. - Counts are tokens, as the server reports them. A field the server
doesn’t expose is left out, never shown as zero.
resetmeans an engine restarted inside the window, andend_failedthat the closing reading failed, so the window ends at the last good one.sessionfolds every sampled turn (windowsis how many) and carries no instantaneous gauge;last_turnmay carry the server’shit_rate_gauge. - A session’s windows don’t overlap. The end reading is taken before the turn returns, under the session’s turn lock, so the next turn starts where the last one ended and no token is counted twice.
- Sampling costs a turn at most a fraction of a second. An edge uses
the poller’s latest reading when it is under a second old, and
otherwise takes one scrape capped at 300 ms. After a failed scrape the
edges stop scraping for 30 seconds and use the latest reading instead,
marking the window
approximate(an edge up to one poll interval off the turn’s). A dead endpoint is logged once, and leaves the block out. - The profile’s credential is sent only to its own origin: same scheme
and host as
base_url, and a Google token only over https.
The judged eval tier reports the same window around each incident’s run, and for the whole corpus, when the model under test is such a profile.
The breakdown is folded from the same reading of the provider’s counters that priced the call, rather than derived a second time from the event stream, so the split you read is the split the money came from.
Attach can read transcripts and drive turns, so it gets its own token
(MAST_ATTACH_TOKEN) and a hard rule: a non-loopback bind without auth is
refused, not warned about. It also stays up through a shutdown drain,
so an operator watching a finishing turn sees its last events rather than a
dropped connection.
Who can see a session
Section titled “Who can see a session”A session created over attach is owned by the caller who created it, and that ownership is what the per-session authorization matrix is built on: the owner reads and writes, viewers read, contributors read and write, and everyone else gets a 404 — not a 403, so an unauthorized caller cannot enumerate which sessions exist by the shape of the refusal.
GET /sessions/{id}/acl reads that list back and PUT replaces it:
{"owner": "alice@example.com", "viewers": ["carol@example.com"], "contributors": ["bob@example.com"], "enforced": true, "persisted": true}The PUT is a whole-document replace, not a patch. A list you omit is a
list you cleared — which is the only sane reading, because the alternative
(omission means “leave it alone”) would leave no way to remove the last
viewer, and revocation is half of why the endpoint exists. Identities are
trimmed and de-duplicated; a blank one is a 400 rather than a stored grant
that could never match a caller.
GET sits at the read bar, not the admin bar. Anyone the ACL already
admits can read the whole transcript, so the membership list is not the
sensitive part — and a viewer who wants write access needs to know whom to
ask. PUT is admin: the owner, or a daemon admin.
Two things the response tells you that the ACL itself cannot:
enforcedis false when the daemon is not running multi-session, in which case the ACL governs nothing and every authenticated caller gets in. APUTagainst such a daemon is refused with 501 rather than accepted: an amendment nothing consults would report success for an access restriction that does not exist.persistedis false when the amendment lives only in this process — no ACL store wired, or a session registered without an owner (the daemon’s own bootstrap session is one). The grant works until the daemon restarts, and saying so is the difference between a durable decision and one that quietly evaporates overnight.
When persisted is true the grant outlives the session’s stay in memory.
Idle sessions are evicted and rebuilt from disk on the next request, and a
restart evicts all of them; either way the ACL that comes back is the one
on disk, not one reconstructed by whatever rebuilt the session. That
matters most for the grant you just made: a viewer added at 4pm is still a
viewer after the overnight restart, and an owner whose session was swept
out is not locked out of it by the act of asking for it again.
Ownership transfer — sending a different owner — is daemon-admin
only. It is there for the case where the owner left, and it fails with a
403 for anyone else rather than being silently dropped, because an ignored
field in an accepted request reads as a completed transfer. An owner
cannot be cleared at all: a session with no owner is reachable by admins
alone, which is a lockout rather than an edit.
A2A — other agents
Section titled “A2A — other agents”--a2a-listen publishes an A2A agent card and
a JSON-RPC endpoint (message/send, tasks/get, tasks/cancel,
message/stream) for workloads that opt in with a2a.expose: true.
The opt-in is per workload and never automatic, because publishing an agent
card is an external contract: you are telling other systems this skill
exists and is callable. MAST_A2A_TOKEN enables per-skill scope
enforcement; a non-loopback bind without a token is refused, since
tasks/cancel is destructive.
Use it when another team’s agent — or another mast deployment — should be able to hand you work.
AG-UI — user-facing clients
Section titled “AG-UI — user-facing clients”--agui-listen serves an AG-UI
run endpoint per opted-in workload, plus a discovery descriptor, for
CopilotKit apps and chat-platform bots. Also opt-in
(agui.expose: true), also its own token (MAST_AGUI_TOKEN), with rate
limiting, since a run drives a budgeted turn.
The one concept worth deciding up front is agui.session_model:
per_thread (default) maps one continuing mast session to each AG-UI
thread, which is what chat UX expects; per_run gives every run a fresh
session, which is right for stateless one-shots. Either way the daemon
derives and namespaces the session id — a client never supplies a raw one.
A threadId is the client’s own correlation string, not a secret, so on an
authenticated endpoint the derivation folds in the caller: two principals
naming the same threadId get two different sessions. Holding the
endpoint’s scopes answers “may this caller run this workload”, which is a
different question from “is this conversation yours” — without the caller
in the id, any authorized principal could continue, and read back, another
one’s thread. The separation is structural rather than a verdict: a second
caller is not refused, they simply address a thread of their own and never
reach the first. An endpoint with no validator has no subject to own
anything, so its session ids are unchanged.
Read that at the grain it actually has. MAST_AGUI_TOKEN is a single token
resolving to a single principal, so on a daemon holding one the caller folded
into the id is a constant: the separation is between deployments, not
between the people using one. Two browsers sharing the token share the
thread, and the per-workload scopes published on the descriptor state what
the workload requires rather than anything that distinguishes one token from
another — there is only one. Supply your own
token validator and the same
machinery separates users, with no change to the server.
Two publication surfaces sit behind their own bundle keys, both empty-or-off
by default and both silent rather than redacted when off:
agui.state_projection names the session-state keys a run may publish as
StateDelta patches, and agui.emit_reasoning lets the workload stream the
model’s thinking as REASONING_* frames. Neither is a display preference —
the client on the other end is a browser — which is why the defaults publish
nothing and the daemon logs what a running workload has enabled.
Because both are per-bundle, a client cannot infer either from the protocol
version, and inferring them from a finished run is hopeless: a stream with no
reasoning in it looks the same whether the workload publishes none or the
model simply didn’t think. So the public discovery descriptor states them up
front, as a
capabilities object per workload. It is
read from the same value the run’s emitter is built from rather than
recomputed from the bundle — a capability claim that restates a config
instead of reading what it describes is how /tools and /perms each spent
releases advertising something untrue.
A third frame family, the StepStarted/StepFinished bracket, has no
key and no capability bit, and the contrast is the useful part. A step is
named after the agent that authored that stretch of the run, so it tells a
client where a coordinator handed off to a specialist — and that is
something a caller permitted to run already sees, because the handoff also
arrives as a transfer_to_agent tool call naming the same agent. State and
reasoning are gated because they disclose something new; steps describe
what is already on the wire, so gating them would be a switch with nothing
behind it.
Every frame is whole
Section titled “Every frame is whole”AG-UI has delta frames — TEXT_MESSAGE_CONTENT carries a delta, so does
TOOL_CALL_ARGS — and mast sends each of them exactly once per message,
carrying the whole thing. An answer arrives as one
TEXT_MESSAGE_START/CONTENT/END triad holding the entire text; a tool
call arrives as one TOOL_CALL_START/ARGS/END triple holding the entire
argument object. A client written to append deltas is correct and will
simply receive one. Nothing arrives twice, and no TOOL_CALL_END ever
describes a call the model had not finished writing.
That is a promise about completeness, not a claim to stream. Turning the answer into per-token frames, so a browser can render it as it is produced, is #407 and is deliberately not built yet: mast runs every model non-streaming today, so there is nothing to forward, and the contract above is what a client can build against until there is.
Hanging up is not an interrupt
Section titled “Hanging up is not an interrupt”If the SSE connection drops — a closed laptop, a proxy timeout, a reload — the run keeps going. It finishes, and everything it did lands in the session transcript. Earlier releases cancelled the turn along with the request, which reads like the polite thing to do and is not: a drop that lands while a tool is mid-flight leaves the side effect done and no record of it, so the retry the client naturally makes applies the change a second time. A turn is work, and a browser going away is not a decision to abandon work.
The consequence runs the other way too, and it is the part to plan for.
Closing the tab stops nothing. A run with no reader left is bounded by the
workload’s budget.max_wallclock_seconds, by the
watchdog, and by an explicit
mast sessions pause <id> --cancel-turn — the same three that bound a run the
daemon started on a schedule with nobody watching it. There is no AG-UI
verb for “stop” yet; if a client needs to mean that, it means
pause --cancel-turn.
Rejoining the stream is a different feature and is not built: reconnecting
with the same threadId starts a new run rather than resuming the view of
the old one. What a disconnect costs today is the tail of the event stream,
not the work.
Tools the client brings
Section titled “Tools the client brings”AG-UI lets a client declare tools of its own in RunAgentInput.tools —
things the browser will execute and report back on, like highlighting a
range in an editor or filling a form the user is looking at. mast parses
that field and ignores it. No workload can call a client-declared tool
today.
That is deliberate rather than unfinished. Everywhere else in mast the operator decides what an agent may call and the caller decides only what to ask for; a tool declared per run inverts that, and the tool’s name and description go into the model’s context, so an unguarded version is an injection surface before it is a feature. The gate is designed and lands in a later release: a workload opts in through its bundle, the opt-in bounds the class rather than listing names — an operator cannot know what a frontend will ship next week — and client tools count as mutating by default, so a workload with human approval configured parks for one before the browser is asked to do anything. They will be callable by the workload’s root agent only, not by its specialists.
One run at a time, per thread
Section titled “One run at a time, per thread”A thread runs one turn at a time. Send a second run while the first is still going and it waits — a mast session is a single-writer store, so every turn-driving surface serializes on the same per-session lock.
The wait is bounded. A thread admits
agui.run_queue.depth
waiting runs on top of the executing one — three by default, so four in all —
and the next one is refused with 409 and a Retry-After before the
stream opens. That is a different answer from the 429 the rate limiter
gives: 429 is your arrival rate and means slow down everywhere, 409 means
this conversation is busy and you can retry it or use another thread. Set
depth: 0 if you want no concurrency on a thread at all.
So a client that stacks runs behind a slow turn gets told so. It is still
better manners to wait for RunFinished before sending the next run on the
same thread: a queued run is a turn you have committed to, and until the
409 there is nothing distinguishing a deep queue from a fast one.
Federation — calling out
Section titled “Federation — calling out”The surfaces above are inbound. invoke_remote_agent is the outbound
counterpart: a specialist calling another agent over A2A.
It classifies as mutating, unconditionally. Effects on the far side of a federation call are invisible from here, and a call whose consequences you cannot see is not one to fire unattended — so it goes through the write gate like any other change.
No caller at all — the workload’s own clock
Section titled “No caller at all — the workload’s own clock”Every surface above answers “what reached the daemon”. A workload that
declares
edge_trigger.scheduled
answers nothing: it wakes on its own interval, and the run has no caller to
take an identity from. That is why it runs as mast:scheduler — a
namespaced identity no human login can produce — and why a mutating call
inside a scheduled run still parks for a real approver. The trigger is a
bundle field rather than a surface: nothing is listening, so there is no
token and no trust boundary to configure.
Which to use is a question about where your schedules should live, not
about capability. An external scheduler posting to /inject gives you one
place to see every job your org runs and one place to change them; the
bundle’s own cadence gives you a workload that is still periodic after it
is copied to a cluster where that scheduler does not exist.
Picking one
Section titled “Picking one”| The caller is… | Surface |
|---|---|
Alertmanager, CI, a runbook curl | inject |
| A scheduler you already run — Cloud Scheduler, a CronJob | inject (it is just another caller) |
| Nothing — the workload should run periodically on its own | edge_trigger.scheduled, no surface |
| A human operator watching or steering an incident | attach (mast-web) |
| Another agent, in or out of your org | A2A |
| A chat UI or CopilotKit app with end users in it | AG-UI |
| mast calling out to another agent | invoke_remote_agent |
Wire shapes, flags, and auth details for all of them: CLI reference. Opt-in fields and per-skill policy: workload bundle.