Approvals and the write gate
The question an SRE team asks about an autonomous agent is not “how good is its diagnosis”. It is “what can it do to my cluster, and who said it could?”
mast answers that with four boundaries, deliberately layered so that no single one has to be perfect. Three are enforced by the runtime; the fourth is your cluster’s, and mast is built to compose with it rather than replace it.
1. Capability — declared per specialist, checked at startup
Section titled “1. Capability — declared per specialist, checked at startup”Every specialist declares what class of thing it is:
capability: read_only # the defaultcapability: change_executor # may reach mutating toolsAt construction — before the daemon serves a single request — mast walks
the roster and refuses to start if a read_only specialist can reach a
mutating tool. Three shapes are refused, each with the offending file and
tool named in the error:
- it names a mutating tool in its allowlist,
- it grants a whole MCP server (under default-deny-unknown, that is a grant of every mutating tool on it),
- it declares no
tools.mcpat all while a tool catalog exists — an unrestricted specialist is not a read-only one.
SingleTurn specialists are exempt: the mode carries no tools, so there is
nothing to reach.
The point of this layer is that the read/write split is structural, not
prompt-held. A prompt saying “you are a read-only diagnoser” is a
suggestion to a language model. A construction-time refusal is not. Startup
also logs every change_executor in the roster, so the answer to “what can
change my cluster” is a log line, not an intersection of three files.
2. Mutation classification — declared, never guessed
Section titled “2. Mutation classification — declared, never guessed”The gate can only stop calls it knows are mutating, so how a tool gets classified is load-bearing. Two parties can declare it, and unclassified means mutating if neither does.
The workload’s tool_catalog is read first and wins: a tool it names is
classified the way it says, in either direction, and every un-gating is
audit-logged at startup. Below that, mast reads the readOnlyHint
annotation the MCP server publishes about its own tools, so a server that
says a tool only reads is believed and that tool does not park. A tool
neither party has classified is mutating — as is one the server annotates as
not read-only, which on the wire is the same thing.
What mast will not do is guess from the verb in a tool’s name. The failure
mode it chooses instead is the safe one — a tool nobody classified parks for
an approval that may not have been needed, rather than a mutating tool
firing unseen — and where a server’s word is not something you want to take,
the tool_catalog override above is how you say so. Details, the full
precedence order, and the reason a name-based heuristic was rejected: tools
and MCP.
3. The write gate — one call, one operator, one answer
Section titled “3. The write gate — one call, one operator, one answer”With the classification in hand, the gate is simple to state: a call that would change something stops before it fires.
hitl: on_mutation: require_approval # default. also: apply | dry_runThe session parks with a durable interrupt carrying the specialist, the tool, and the exact arguments. The agent is not spinning; the process can die and the pause survives it (durability). An operator answers with one of three verdicts:
| Verdict | Effect |
|---|---|
approve | the call fires, exactly as shown |
reject | the call does not fire, and the agent stops rather than looking for another way |
edit | the operator supplies corrected arguments, and those run |
Three properties are worth calling out, because each closes a hole that looks small on paper:
- Scope is
once, unless the operator is looking at the whole set. A verdict approves the single call in front of them. The one exception ischange_set, below: it authorizes the specific other calls quoted in the same question, and nothing else. Anything wider — “approve all scale operations for this session” — is refused and audited asapproval_scope_refused, because that is the blanket consent the gate exists to prevent. - The approver is the authenticated caller, not the payload. Whatever name arrives in the request body is overwritten by the identity that actually authenticated. An audit trail an agent could write into is not an audit trail. The same rule governs who a resumed pause names — see who a resume names.
- An edit is recorded durably. The substrate re-fires the original call on resume, so the edited arguments are persisted as their own record and re-applied — the thing that ran and the thing the operator authorized are the same thing, and the log shows both.
Every outcome lands in the audit log under a fixed vocabulary:
awaiting_approval, denied_by_policy, denied_by_operator,
approval_scope_refused, edit_unattributed, edit_refused,
edit_applied, apply, dry_run, already_refused, refusal_loop, and
the change-set outcomes change_set_approved, approved_by_change_set,
change_set_scope_refused, change_set_refused.
Telling somebody the park happened
Section titled “Telling somebody the park happened”Everything above assumes an operator finds out. Four things surface a park —
GET /parks, the attach stream, mast sessions show, and the audit log —
and all four are pulls. For an unattended workload that is a real gap: a
fifteen-minute cadence can park at 03:00 and sit there until somebody thinks
to look, and the whole point of a durable park is that nobody has to be
watching when it opens.
--park-notify <conversation> closes it
by pushing, through the same chat ingress --notify-url already configures.
One message when the park opens, naming the session and the call and ending
with the command that prints the resume line. Not repeated, budgeted at three
then one per five minutes, and counted in
mast_park_notifications_total.
It announces every park rather than only the ones nobody is watching. mast could check whether an operator is attached at the moment the park opens, but that answer has a short shelf life — the park outlives the process, and somebody who detaches thirty seconds later would get silence all night. The cost of announcing anyway is a redundant message to somebody already looking, which is the cheaper of the two mistakes.
Note what the message contains before choosing a channel: the park hint renders the call with its argument values, elided at 120 characters each. The full arguments stay in the confirmation record.
Whatever the chat ingress does — throttled, failed, misconfigured — the park itself is untouched. A failed announcement is an error on the daemon’s books and never on the turn’s.
A refusal the model cannot talk its way past
Section titled “A refusal the model cannot talk its way past”“The agent stops rather than looking for another way” is what the refusal tells the model. The gate does not depend on it complying.
For the rest of the turn, the call an operator refused is suppressed: the
model can propose it again, and the operator is not asked again. It comes
back as already_refused. A different call still parks normally, so the
suppression narrows nothing the operator has not already answered.
Three suppressed proposals and the turn ends, counted as refusal_loop.
Nothing latches — no guardrail trips, there is nothing to reset, and the
next turn starts clean, refusal and all. The scope is one turn on purpose:
a refusal that outlived its turn would silently block a call the operator
would have approved.
The behavioral watchdog would eventually catch the same loop, but it halts the session and makes an operator clear a guardrail to get the daemon working again. One “no” should not cost that, so the gate cuts first, at a lower threshold, and tells the watchdog to forget the calls it turned away — they never ran. The watchdog’s own halt is never cleared by this; an arm that could un-halt a session would let a looping agent overrule its operator.
Details and the exact wording the model gets: rejection stops the agent.
The other two modes exist for real situations, not as escape hatches:
dry_run runs the whole loop and reports what would have happened, which
is how you evaluate a roster before trusting it; apply is for
already-bounded automation where the RBAC layer below is the real
constraint. Both are opt-in, in the bundle, reviewable in a diff.
Field-level detail, request and response shapes: write gate reference.
4. RBAC — the boundary mast does not own
Section titled “4. RBAC — the boundary mast does not own”The first three layers are mast’s, which means they are only as good as mast’s own correctness. The fourth is not: the shipped Kubernetes base mirrors the capability split into two service accounts, so a compromised or buggy read-only path has no credential to change anything with, whatever it believes about itself.
That is the layer to lean on hardest, because it holds when the others are wrong. See cluster permissions for the manifests and the current caveat on narrowing the GKE IAM binding.
The change set — approving the call, not the prose
Section titled “The change set — approving the call, not the prose”The three layers above stop a call. This one decides which call the operator is being shown.
A finding that says “scale the api Deployment back to 2 replicas” is prose, and prose cannot be approved, only agreed with. If the change executor composes the actual tool call afterwards, from that sentence, on its own turn, then the thing that reached your cluster is not the thing anybody read.
So a report can carry the call. A workload’s report schema declares
proposed_change, and a finding fills it with entries naming a tool from
the workload’s own catalog:
"proposed_change": [ {"tool": "apply_change", "arguments": "{\"namespace\":\"prod\",\"replicas\":2}"}]Two checks run the moment the report is returned, before it becomes the
specialist’s answer: the tool has to be one this workload declares, and the
arguments have to fit that tool’s own input schema. A report that fails
either comes back to the specialist as an error it can correct, rather than
travelling on to an operator who has no way to tell. It is the same code
that validates an operator’s edit at the gate, so the two cannot drift
apart on what “valid arguments” means.
An empty list is a right answer. “Raise the memory limit, but I don’t know to what” is an honest diagnosis; a specialist that cannot name an exact call returns nothing and escalates, instead of inventing a plausible number to fill a field.
Under graph dispatch this closes the loop the first three layers left
open. A finding with a non-empty change set that the operator approved is
routed to the roster’s change executor, which is handed those calls
verbatim — the edge is a structural rule about the finding, not an
instruction in a prompt. Everything else about the gate is unchanged: each
call still parks, still names its arguments, still needs one authenticated
answer. A roster that declares no proposed_change, or a finding that
proposes nothing, behaves exactly as it did before; and a roster carrying
two change executors logs an error and leaves the edge unwired rather than
guessing which was meant.
Field reference: the report contract.
Two gates, and how many questions they add up to
Section titled “Two gates, and how many questions they add up to”Under graph dispatch with hitl.require_approval: true, a remediation
meets two gates, and they are not one gate asked twice:
- the finding gate asks whether a diagnoser’s report should be acted on at all. Answering it is what routes the change set to the executor.
- the write gate asks about each call, with its arguments, as the executor makes it. Answering it is what changes the cluster.
So a single-call remediation costs an operator two answers, and that is the
design rather than a redundancy: approving a finding is not approving a
call, and hitl.change_set_ttl does not collapse them — a grant is minted
at the write gate for calls 2..N of a set, so a set of one has nothing to
grant.
What the executor does not add is a third question about its own report. By the time it has one, its calls have already been made, each past the write gate; approving the account afterwards would decide nothing.
The two gates also arrive on separate turns, and stay answered. A specialist parked at the write gate has not produced a finding yet, so its node does not raise the finding gate over a result that does not exist; and an answer given at either gate is recorded, so re-entering the graph on a later turn does not ask for it again. Earlier versions did both — the two parks landed on one turn, and answering either stranded the other (#269).
One answer for a set of calls — and what makes it go stale
Section titled “One answer for a set of calls — and what makes it go stale”A remediation is often several calls: scale two Deployments, or patch a ConfigMap and restart what reads it. Parking each one separately means the operator who has already read the plan answers the same question three times, and the third answer is not a decision, it is a reflex.
So when a parked call belongs to a change set, the question says so and
quotes the rest of the set. Answering with scope: change_set mints a
grant for each remaining call:
Message: Approve mutating call scale_deployment(deployment=api, replicas=2)? It is one of 2 calls in the change set ScaleUp proposed; answer with scope=change_set to authorize all 2. Change set: 2 call(s) proposed by ScaleUp 1. scale_deployment(deployment=api, replicas=2) 2. scale_deployment(deployment=worker, replicas=2) freshness of scale_deployment: re-checked against get_deployment before the call fires Approve all with: --response='{"verdict":"approve","scope":"change_set"}' (valid for 600s)A grant is deliberately narrow:
- it authorizes one exact
(tool, arguments)pair — not the tool, not the verb, not “scaling” — so a call the model composes afterwards, however similar, still parks; - it is single-use and durable, so a restart between the answer and the call changes nothing, and a second identical call needs its own approval;
- the deny policy still runs first, and spending a grant is still recorded as an allow-once decision in the audit log. A grant removes the question, never the accounting;
- if any call in the set cannot be granted — an argument too large to render in the question, a precondition mast cannot evaluate — the whole verdict is refused with a reason rather than covering the part that fits. A partial “approve all of this” is the one outcome nobody chose.
What makes an approval stale is the cluster, not the clock. A set approved twenty seconds ago against a Deployment somebody has since scaled by hand is stale. One approved an hour ago against an untouched object is not. mast cannot work out which read tells it that — it does not know which tool reads the object a write is about, or which of the write’s arguments names it — so the bundle declares it, per tool, and mast runs it twice: once when the operator answers, once just before each granted call fires. If a watched field moved in between, the grant is voided and the call goes back to the operator with a question naming the field and both values.
Both of those reads run the tool itself, underneath the digest wrap. Digesting exists to shrink what a model reads; this read is mast’s own, and a digest of it would drop the very field the comparison is about.
hitl.change_set_ttl (default 10 minutes) is the backstop for everything a
precondition cannot see. A tool that declares no precondition gets the TTL
and nothing else — and its question says so, in those words, rather than
implying a check mast is not making.
How to declare one, and the mistake to avoid on your first try (a
precondition watching the field the set itself rewrites invalidates its own
set): precondition: in the workload
bundle.
What was there before
Section titled “What was there before”Everything above is about whether a change happens. This is about what is true after it has.
An operator who approves a change is making a bet, and a fair bet needs a way to fold. Without one the position is the worst of the three available: you can approve a change, watch the incident get worse, and have no path back except reconstructing the old state by hand while the pager is going.
So when the workload declares it, the gate reads the target’s prior state
before the call fires and records it — the old values, and the call that
restores them. mast sessions show prints both.
The undo is a proposal, not a button. mast will not fire it. Running it sends it back through this same gate with a person answering, because a rollback mast decided on by itself is a mutating call nobody approved.
Two limits are worth knowing up front, because both are decisions rather
than gaps. mast cannot work out the read or the inverse on its own —
which tool reads the object a write is about is domain knowledge, and
scale_deployment inverts by re-scaling while delete_pod does not invert
at all — so the workload declares them, and a workload that declares no
inverse gets a record of the old value with the undo marked undeclared.
That is honest and still useful. And a capture proves a record exists,
not that the revert would work; nothing executes one to check.
How to declare it:
capture:.
What it looks like in the transcript: what the change
overwrote.
What the operator decided, after the incident is over
Section titled “What the operator decided, after the incident is over”A verdict is expensive. Somebody stopped what they were doing, read a call, and made a judgement about a production system. Spending that once and keeping only a log line wastes the most informative thing a gated fleet produces: a case where the model’s proposal and a human’s answer are known to differ.
So every adjudication is a durable record beside the call it decided, and
mast sessions export-decisions writes them out as JSON Lines — one row per
decision, with what the model proposed, what actually executed, which of
approve / reject / edit the operator chose, and whether the call ended up
authorized, refused by the operator, or refused by mast. Rejections are in
there for the same reason edits are: an export of nothing but approvals is a
dataset that never shows the model being wrong.
Two things to know before pointing it at a production store. Approver
identities are digested by default — a stable hash, so you can still group
by approver or count how many people approved a class of change, without
naming anyone; --include-approver opts out and the file records which mode
produced it. Tool arguments are exported verbatim, because the
proposed→executed diff is the whole label — which makes an export as
sensitive as the arguments your tools take.
Mast captures and exports; it does not score, retrain, or read the file back. What it is for is the evaluation you run yourself. Field detail and the full record shape: exporting decisions.
Composing the four
Section titled “Composing the four”| Layer | Stops | Does not stop |
|---|---|---|
| Capability split | a read-only specialist ever reaching a mutating tool | a change_executor doing something wrong |
| Mutation predicate | an unclassified tool sliding through as safe | a tool wrongly declared mutating: false |
| Write gate | any classified mutation firing unseen | an operator approving a bad call |
| RBAC | any credential-less change, whatever mast believes | a change inside the granted scope |
Nothing here removes the operator from the loop. That is the design: mast’s job is to make sure the operator is asked, with the actual call in front of them, and that the record of what they said survives the process that asked.
The fifth column of that table — what the four layers add up to against a deliberate attacker, and the places they are known not to reach — is the threat model.