Skip to content

lookout state volumes

When pods hang in ContainerCreating with Multi-Attach or FailedAttachVolume events — join VolumeAttachment + PV/PVC + pods to name the exact conflict: RWO claims wanted on two nodes, attachments stuck in error, cross-zone PV locks, orphaned attachments.

MCP tool: k8s_volume_conflicts

Terminal window
lookout state volumes [flags]
FlagTypeDefaultMeaning
--namespacestring—limit the scan to one namespace
-Abool—scan all namespaces
--workloadstring—target one workload as <Kind>/<namespace>/<name>, e.g. Deployment/prod/api
--sinceduration—how far back to look (0 = command default)
--formatstringlogfmtoutput format: logfmt|json (one record per line either way)
--timeoutduration10sabort the invocation after this long (exit 1)
--kubeconfigstring—path to a kubeconfig file, instead of $KUBECONFIG / ~/.kube/config
--contextstring—kubeconfig context to read, instead of its current-context. Selects a cluster for THIS invocation only — nothing is written back — so concurrent invocations can target different clusters. Reported as context=<name> in the summary line
--exemptionsstring—path to a git-reviewed exemption file (YAML); covered findings are ANNOTATED with their reason and expiry and counted as exempt=<n> in the summary, never dropped

Every kind= this command can emit, and the severities it carries them at. Nothing else appears in its output; a kind absent from a run means the check looked and found nothing. See the finding-kind glossary for the whole vocabulary.

KindSeverityClaim
volume.multi_attachcriticalan RWO claim is wanted by pods on more than one node — the second pod never starts
volume.zone_conflictcriticalthe PV is locked to a zone the pod’s node is not in
volume.attach_errorcritical, warningthe attach or detach is failing; critical once it has been failing long enough to be stuck rather than slow
volume.orphaned_attachmentinfoa VolumeAttachment survives its PV or its node

Beyond the shared envelope fields (kind, severity, namespace, kind_of_object, name, reason, message, fingerprint, exempt_reason, exempt_expires):

FieldMeaning
podsscheduled pods referencing the conflicted claim, sorted (list capped, then +K more)
nodesdistinct nodes those pods are scheduled on, sorted
access_modesthe claim’s declared access modes
pvPersistentVolume behind the claim or attachment
pvcPersistentVolumeClaim the pod mounts (same namespace as the pod)
nodenode the attachment targets or the pod is scheduled on
attacherCSI driver responsible for the attachment (spec.attacher)
agehow long the attach/detach error has persisted, truncated to seconds
errorthe attach/detach error message, truncated to 200 chars
attachedthe attachment’s status.attached at scan time
pv_zoneszones the PV’s node affinity allows, sorted
node_zonezone label of the node the pod is scheduled on
orphanwhich referenced side is gone: “pv missing”, “node missing”, or both

Output: one finding per line (logfmt; —format=json for one JSON object per line), keys in fixed order; healthy resources emit nothing. The final line is always the summary: scanned= findings= elapsed= — findings=0 with a summary present means “scanned and healthy”; a stream without a summary line is void. Exit 0 data, 1 runtime error (diagnostics on stderr only), 2 usage.

Terminal window
lookout state volumes
lookout state volumes --namespace=prod
lookout state volumes --format=json