The watchboard
Leading indicators must not each open a page-priority session. In
per-incident mode, severity routing sends each signal class its own way:
critical opens a per-incident session with enrichment, warning
batches into the shared watchboard session as a rolling digest, and
info is stored only (with --store; counted and dropped without).
In --mode=shared the watchboard is disabled — everything routes to
--target-session.
Digest cadence
Section titled “Digest cadence”Warnings buffer until either --watchboard-batch of them accumulate
(default 5) or the oldest buffered warning reaches --watchboard-flush
in age (default 60s) — whichever comes first. The board session is
created lazily on the first flush. Captured live in a validation drill:
00:58:45 watchboard: buffered objectstate.endpoints_empty weblab/web (severity=warning, buffered=1/5)00:59:47 watchboard: digest 1 entry(ies) → sid=stub-sess-0009 (generation=1, injects=1/200, mode=per-incident)And the digest inject itself — one schema-stable record per flush, each entry carrying its own fingerprint and object coordinates:
{"kind":"watchboard.digest","cluster":"kl-m2","board_generation":1,"sequence":1, "window_start":"2026-07-25T00:58:45.68219189Z","window_end":"2026-07-25T00:59:47.098792925Z", "entries":[{"kind":"objectstate.endpoints_empty","fingerprint":"sha256:638f0f4d…", "reason":"endpoints_empty","namespace":"weblab","kind_of_object":"Service", "name":"web","uid":"052e28a1-…","count":1, …}]}Rotation lifecycle
Section titled “Rotation lifecycle”A shared session would otherwise grow without bound, so the board
rotates by size, not by calendar: after --watchboard-rotate digest
injects (default 200), the next flush opens a successor session. The cap
bounds the agent’s cost of consuming the board no matter how noisy the
cluster is, while a quiet cluster keeps one session for months. The
mechanics:
POST /sessionscreates the successor (same--ownerasserted-caller as every sentinel session).- The final inject into the old session is the lineage record —
kind=watchboard.rotatedwithsuccessor_session_id,injects_count,rotated_at,cluster, andboard_generation— so anyone reading the old session can follow the pointer forward. - The pending digest flushes into the successor; counters restart at
board_generation+1,sequence=1.
Failure posture: if creating the successor fails, rotation is deferred — the digest flushes into the over-threshold session and rotation retries on the next flush. Warnings are never dropped to enforce a size cap.
Lineage and finding the board
Section titled “Lineage and finding the board”The daemon’s session API has no name parameter, so watchboard sessions
identify themselves in-band: every inject into one is kind=watchboard.*,
and each digest carries board_generation (1-based count of board
sessions this sentinel has opened) and sequence (digest ordinal within
the session). watchboard.rotated links predecessor → successor, so the
chain is walkable from either end.
What rotation does — and does not — touch
Section titled “What rotation does — and does not — touch”- Incident bindings stay put. Each flushed warning’s dedup entry is
bound to the session its digest landed in. After rotation, followups
and
kind=resolved/resolved.revertedoutcomes for an incident bound to the old board keep routing there; only new warnings flow to the successor. Old boards drain naturally and never orphan an open fix-verify loop. Bindings persist through--dedup-persistlike all bindings. - Storms bypass the board. A storm can be warning-class (severity is the max of its members), but it always opens its own session — an aggregate incident an agent works is not a digest entry.
- Triage-status overrides feed it. An incident an agent downgraded
via
triage status--severity-overrideroutes its next dedup cycle to the watchboard instead of re-paging — a live drill captured the digest entry carrying the downgraded incident’s exact fingerprint.
Per-kind routing is tunable with --severity kind=level (repeatable) —
one recorded drill demoted a default-critical kind to warning
(--severity=objectstate.endpoints_empty=warning) and another
promoted rollout.stall to critical for a staging cluster where bad
deploys are the hunt. The full decision record is
docs/watchboard-rotation-design.md;
the wire shapes are pinned byte-exact by tests.