Restarts and shutdown

What happens when a core-agent daemon stops — a K8s rolling upgrade, a config-change restart, a crash — and what survives it. Short version: with a durable session DB, restarts are boring. That’s the design goal.

What SIGTERM does

The daemon catches SIGTERM (and only SIGTERM — SIGINT belongs to the REPL’s double-Ctrl+C flow) and cancels the process context. From there:

In-flight turns are interrupted immediately. There is no drain phase and no drain knob, deliberately: agent turns can run unboundedly long, so no timeout is “long enough”, and a drain longer than the supervisor’s kill timeout just invites SIGKILL mid-cleanup. Interrupted work is recoverable instead (see below).
Teardown runs with bounded steps: peer-hub deregistration (2s cap), attach listener drain (SSE streams hung up, then graceful HTTP shutdown — default 5s, tunable via attach.shutdown_timeout), background-subagent drain (5s, stragglers abandoned and logged), MCP stdio children (SIGTERM → 3s grace → SIGKILL, concurrently), then telemetry flush and Vertex context-cache cleanup (3s each).
Message intake refuses instead of lying. Once SIGTERM fires, POST /inject and POST /wake return 503 with Retry-After — a message accepted in that window would sit in an in-memory inbox and die with the process after the client got a success response. Clients redeliver after the restart; committed history is unaffected.
The process exits 0. Restart-on-exit is the supervisor’s job (K8s restartPolicy), not an exit-code contract.

Worst case with defaults, the whole sequence takes ≈ 24 seconds — inside Kubernetes’ default terminationGracePeriodSeconds: 30 with headroom. If you raise attach.shutdown_timeout, raise the grace period to keep that inequality true.

No preStop hook is needed. The container image runs the Go binary as PID 1 via an exec-form ENTRYPOINT (distroless, no shell, no tini/s6), so the kubelet’s SIGTERM arrives unwrapped.

What survives: the durability contract

Persistence is per-event, during the turn — not at turn end and not at shutdown. Every user message, model response, tool call, and tool result is committed to the session DB as it is produced. Nothing about durability depends on shutdown code getting a chance to run, which is why the contract holds equally for SIGTERM, SIGKILL, and OOMKill:

At most the in-flight model response or tool execution is lost. Everything already committed survives.

Two preconditions:

--session-db must be on. The default session store is in-memory; without the flag, a restart loses everything. See Sessions and event log.
The DB must be on storage that survives the pod — a PVC in K8s. Sessions, the event log, ACL rows, run locks, and digest state all live in that one SQLite file (or Postgres/MySQL for multi-writer deployments).

If a restart catches a turn between a persisted tool call and its result, the history is repaired automatically on the next turn — no operator action.

One narrow caveat: the event log closes slightly before the background-subagent drain finishes, so a subagent that exits during the 5s drain window may fail to persist its final events. Committed state is unaffected.

What resumes, and how

Interactive / attach sessions resume lazily: the first request that touches a session after restart reconstructs its agent from the persisted ACL row and event log — same session ID, same history, same ACL. Nothing is scanned or re-run at boot. See Multi-session → Session resume.
Autonomous runs resume explicitly: per-turn checkpoints (goal, continuation prompt, budgets, next wake time) are persisted as events, and the orchestrator that owns the process (K8s CronJob, supervisord, AX) calls autonomous.Resume on the next start. A stale run lock from a crashed process is stolen automatically after 30s. See Autonomous → Crash-resume.
Interrupted turns are finished automatically (on by default for daemons; #559): via agent.auto_continue, a session whose turn was cut off by the restart gets a synthesized “continue the task” turn — on first touch for attached sessions, and via a bounded boot-time scan for channel sessions nobody re-touches. The same detection also finishes a turn that stopped short because it hit the model’s output-token cap mid-answer (a MAX_TOKENS truncation), picked up on the session’s next touch or boot scan (#582). Guarded by a freshness window, a per-boot cap, a per-session attempt cap with in-lifetime self-heal (retries a transiently-failed continuation on a timer instead of waiting for a reboot; #575), and a crash-loop breaker (agent_boot_log); deliberately-interrupted turns (POST /interrupt) are never resurrected, an operator message already queued when the restart is detected (e.g. a stop) drives the next turn on its own rather than competing with a continuation note, and an interrupted read-only introspection call is not blindly re-issued (#624). Without it, a session interrupted mid-question resumes with intact history but waits for the next message.

On by default where it can apply — a multi-session daemon or a single-user headless daemon (--no-repl, the examples/gke-deploy shape), both with --session-db. Rolling upgrades are exactly what it exists for, so these deployments get it for free (pair it with a per-turn cost ceiling); the daemon logs a one-line notice when it turns on by default. It stays off — silently — for interactive REPL/TUI runs and in-process library use, which the precondition excludes: a human is present to re-ask. Set { "agent": { "auto_continue": { "enabled": false } } } to opt out on a daemon. (This flipped from off-by-default to on once the guard stack had production soak — #559.)

Kubernetes checklist

spec:
  terminationGracePeriodSeconds: 30   # ≥ teardown budget (~24s with defaults)
  containers:
    - name: core-agent
      # exec-form ENTRYPOINT, binary is PID 1 — no preStop needed
      volumeMounts:
        - name: state
          mountPath: /data
      args: ["--session-db-path", "/data/sessions.db", ...]
  volumes:
    - name: state
      persistentVolumeClaim:
        claimName: core-agent-state

PVC for the session DB — without it every other guarantee on this page is moot.
strategy: Recreate (or a leader lock) with RWO volumes — two daemons must not share one SQLite file.
Raise terminationGracePeriodSeconds in lockstep if you raise attach.shutdown_timeout.