Skip to content

lookout triage delta

Every abnormal object in one scan — the first call for “anything wrong in this cluster?”: broken/pending pods, stalled rollouts, workloads blocked from creating pods at all, node pressure/NPD/preemption, gridlocked PDBs, degraded kube-system add-ons, quotas at their limits.

MCP tool: k8s_triage_delta (MCP profile: triage)

Terminal window
lookout triage delta [flags]
FlagTypeDefaultMeaning
--onlystringpods,nodes,pdb,system,quotacomma-separated finding classes to scan: any subset of pods,nodes,pdb,system,quota
--restartsint5flag containers restarted at least this many times
--pending-ageduration5mflag Pending pods older than this; also the grace before a not-ready container in a Running pod is flagged
--quota-warnint90warn when a ResourceQuota resource reaches this percent of its hard limit (the hard limit itself is always critical)
--cron-graceduration5mhow late a CronJob activation may be before it counts as missed; absorbs normal controller scheduling latency
FlagTypeDefaultMeaning
--namespacestring—limit the scan to one namespace
-Abool—scan all namespaces
--workloadstring—target one workload as <Kind>/<namespace>/<name>, e.g. Deployment/prod/api
--sinceduration—how far back to look (0 = command default)
--formatstringlogfmtoutput format: logfmt|json (one record per line either way)
--timeoutduration10sabort the invocation after this long (exit 1)
--kubeconfigstring—path to a kubeconfig file, instead of $KUBECONFIG / ~/.kube/config
--contextstring—kubeconfig context to read, instead of its current-context. Selects a cluster for THIS invocation only — nothing is written back — so concurrent invocations can target different clusters. Reported as context=<name> in the summary line
--exemptionsstring—path to a git-reviewed exemption file (YAML); covered findings are ANNOTATED with their reason and expiry and counted as exempt=<n> in the summary, never dropped

Every kind= this command can emit, and the severities it carries them at. Nothing else appears in its output; a kind absent from a run means the check looked and found nothing. See the finding-kind glossary for the whole vocabulary.

KindSeverityClaim
pod.crashloopcriticala container is crash looping
pod.imagepullcriticala container cannot pull its image
pod.waitingwarninga container is stuck in an error waiting state (CreateContainerConfigError, InvalidImageName, …)
pod.oomkilledwarninga container’s last termination was an OOM kill
pod.restartswarninga container has restarted at least —restarts times
pod.notreadywarninga container in a Running pod has been not-ready past the —pending-age grace
pod.failedwarningthe pod reached phase Failed
pod.pendingcritical, warningthe pod has been Pending longer than —pending-age with no container-level diagnosis; critical when the scheduler has declared it Unschedulable, which is a capacity or constraint problem rather than latency
workload.replicafailurecriticalthe controller cannot create pods at all (quota, PodSecurity, admission) — no pod exists to diagnose
workload.stalledcriticala Deployment’s Progressing condition is False: the rollout has given up
workload.rolloutcritical, warningreplicas are short of desired; critical when nothing is serving at all
job.failedwarninga Job’s Failed condition is set
cron.missedcritical, warningan unsuspended CronJob’s schedule said to run more than —cron-grace ago and status says it did not; critical once several activations in a row are gone
cron.unparseablewarninga CronJob’s spec.schedule could not be parsed, so its activations cannot be judged at all
node.notreadycriticalthe node’s Ready condition is not True
node.pressurecriticalthe node reports Memory/Disk/PID pressure
node.conditioncritical, warninga non-standard node condition is True — NPD and its cousins publish problems that way
node.cordonedwarningthe node is unschedulable but still holds pods: a stuck drain or a forgotten maintenance step
node.preemptcritical, warning, infoa reclaim taint marks the node for termination; severity tracks how imminent
pdb.gridlockedcritical, warningthe budget permits no disruptions; critical when healthy pods are already below the required minimum
addon.degradedcritical, warninga kube-system add-on (dns, proxy, cni, csi, metrics, connectivity) is short of replicas; critical when none are available
quota.nearwarninga ResourceQuota resource is at or past —quota-warn percent of its hard limit
quota.exhaustedcriticala ResourceQuota resource is at its hard limit: the next create is rejected

Beyond the shared envelope fields (kind, severity, namespace, kind_of_object, name, reason, message, fingerprint, exempt_reason, exempt_expires):

FieldMeaning
containercontainer the finding is about (init containers prefixed init:)
imageimage reference that failed to pull
restartscontainer restart count
exit_codeexit code of the container’s last termination
last_statereason of the container’s last termination (e.g. OOMKilled)
agehow long the abnormal state has persisted
desireddesired replica/scheduled count from spec
readyready count from status
updatedupdated-to-current-revision count from status
availableavailable count from status
failedfailed pod count of a Job
schedulea CronJob’s spec.schedule
expectedthe activation a CronJob should have run and did not
missed_runsactivations missed since the anchor; ≥N when the walk was capped
anchorwhat the missed count was measured from: last_schedule or creation
time_zonea CronJob’s spec.timeZone, when set
last_schedulea CronJob’s status.lastScheduleTime, or never
active_jobsJobs a CronJob still has running
conditionnode condition type that is abnormal
tainttaint key indicating reclaim/drain
podspods affected (behind a cordoned node or a PDB)
healthycurrently healthy pods behind a PDB
requiredpods the PDB requires healthy
addonsystem add-on role: dns, proxy, cni, csi, metrics, connectivity
resourceResourceQuota resource name at or near its limit
usedquota usage from status
hardquota hard limit from status
pctquota usage as percent of the hard limit

Output: one finding per line (logfmt; —format=json for one JSON object per line), keys in fixed order; healthy resources emit nothing. The final line is always the summary: scanned= findings= elapsed= — findings=0 with a summary present means “scanned and healthy”; a stream without a summary line is void. Exit 0 data, 1 runtime error (diagnostics on stderr only), 2 usage.

Terminal window
lookout triage delta
lookout triage delta --namespace=prod --only=pods,quota
lookout triage delta --only=nodes --format=json