Skip to content

lookout perf probe

Control-plane and startup performance via metrics query packs: —pack=apiserver (p99 latency by verb/resource), apf (queue saturation + 429 rejects), etcd (WAL fsync p99 + DB size), startup (pod-first-ready p95 trend); apf/etcd need GKE control-plane metrics enabled — absence degrades to an explicit pack_unavailable finding.

MCP tool: k8s_perf_probe

Terminal window
lookout perf probe [flags]
FlagTypeDefaultMeaning
--packstring—which query pack to run (required): apiserver|apf|etcd|startup
FlagTypeDefaultMeaning
--namespacestring—limit the scan to one namespace
-Abool—scan all namespaces
--workloadstring—target one workload as <Kind>/<namespace>/<name>, e.g. Deployment/prod/api
--sinceduration—how far back to look (0 = command default)
--formatstringlogfmtoutput format: logfmt|json (one record per line either way)
--timeoutduration10sabort the invocation after this long (exit 1)
--kubeconfigstring—path to a kubeconfig file, instead of $KUBECONFIG / ~/.kube/config
--contextstring—kubeconfig context to read, instead of its current-context. Selects a cluster for THIS invocation only — nothing is written back — so concurrent invocations can target different clusters. Reported as context=<name> in the summary line
--exemptionsstring—path to a git-reviewed exemption file (YAML); covered findings are ANNOTATED with their reason and expiry and counted as exempt=<n> in the summary, never dropped

Every kind= this command can emit, and the severities it carries them at. Nothing else appears in its output; a kind absent from a run means the check looked and found nothing. See the finding-kind glossary for the whole vocabulary.

KindSeverityClaim
perf.apiserver_p99critical, warningapiserver request latency p99 crossed the pack threshold for a verb/resource — warning from 1s, critical from 4s
perf.apf_saturationcritical, warningan API Priority and Fairness level is holding a sustained queue — warning from 10 queued, critical from 100
perf.apf_rejectscritical, warningAPF is shedding load: the apiserver is returning 429s at a priority level
perf.etcd_fsynccritical, warningetcd WAL fsync p99 crossed the pack threshold — warning from 10ms, critical from 100ms
perf.etcd_db_sizecritical, warningthe etcd database is approaching its quota — warning from 4 GiB, critical from 5.5 GiB
perf.startup_p95critical, warningpod first-ready p95 crossed the pack threshold — warning from 60s, critical from 300s
perf.pack_unavailablewarninga metric the requested pack needs is not in the metrics workspace, so part of the pack could not run; the rest still did (no coverage lies)
cloud.unavailableinfothe cloud capability this check needs is unavailable, so nothing was examined — an explicit degradation record, never silence

Beyond the shared envelope fields (kind, severity, namespace, kind_of_object, name, reason, message, fingerprint, exempt_reason, exempt_expires):

FieldMeaning
packthe pack this finding belongs to; also the summary-line note naming the pack that ran
metricthe backend-neutral metric the query measured (pack_unavailable: the absent metric)
verbapiserver request verb for this series (apiserver pack)
resourceapiserver request resource for this series (apiserver pack)
priority_levelAPF priority level for this series (apf pack)
codethe HTTP status code the query matched (apf pack: 429)
observedthe worst (maximum) aligned value in the window, in the query’s unit — the breach basis
latestthe newest aligned value in the window
thresholdthe crossed threshold: the critical one when severity=critical, else the warning one
windowthe lookback the series cover (—since, or the pack default); also a summary-line note
trendstartup pack: second-half vs first-half mean delta of the window, e.g. “+34%” — the p95 trend direction
capabilitycloud.unavailable: the provider capability this command needed (metrics)
providercloud.unavailable: the provider that was asked
unavailablesummary-line note: why the metrics backend could not be served

Output: one finding per line (logfmt; —format=json for one JSON object per line), keys in fixed order; healthy resources emit nothing. The final line is always the summary: scanned= findings= elapsed= — findings=0 with a summary present means “scanned and healthy”; a stream without a summary line is void. Exit 0 data, 1 runtime error (diagnostics on stderr only), 2 usage.

Terminal window
lookout perf probe --pack=apiserver
lookout perf probe --pack=apf
lookout perf probe --pack=etcd --since=6h
lookout perf probe --pack=startup
lookout perf probe --pack=apiserver --format=json