Prometheus metrics

lookout watch --metrics-addr=host:port serves Prometheus metrics on /metrics (plus /healthz and /readyz). Every metric carries the lookout_ prefix.

Generation note (the documented choice): metric names and help strings are derived from the live collectors (internal/watch.MetricsInventory); the type and label columns are stamped per collector in that inventory because the Prometheus client does not expose them before first observation. A presence-check test (TestMetricsInventoryComplete) fails when a collector is added without an inventory row.

The lookout_leeway_* block is the exception: those instruments are declared on the OpenTelemetry metric API and bridged into the same registry, so they have no collector to describe. Their rows are written out in pkg/sources/topologydrift.MetricDocs and pinned there against the real exporter, names, types and labels included.

Rows marked opt-in are absent from a default scrape until a flag turns them on; see the lookout watch flag table.

Metric	Type	Labels	Meaning
lookout_events_seen_total	counter	reason, namespace	Total k8s events observed by the informer, before filter.
lookout_events_injected_total	counter	reason, namespace	Total events that survived filter + dedup and were POSTed to the daemon.
lookout_events_deduped_total	counter	reason, namespace	Total events suppressed by the rolling-window dedup cache.
lookout_events_filtered_total	counter	gate	Total signals rejected by the engine filter before dedup, by the rule that rejected them (reason_not_allowed|namespace_excluded|namespace_not_allowed|unhealthy_debounce|crashloop_debounce|imagepull_transient_debounce). The leading-edge debounces deliberately swallow events; without this counter a gate tuned too tight is indistinguishable from a broken watcher.
lookout_inject_errors_total	counter	reason, http_code	Total payload deliveries (or incident opens) against the configured sink that returned a non-2xx response or transport error. Counts sink operations regardless of —sink.
lookout_inject_shrinks_total	counter	shed	Total payloads shrunk to fit —inject-max-bytes before delivery (issue #198), by what was shed (enrichment|message|member_fingerprints|watchboard_entries). Identity is never dropped; a counted incident still routed. A rising enrichment count means —enrich-cap is set too high for the sink’s inject ceiling; member_fingerprints means a storm outgrew the ceiling and its member list was cut to the earliest arrivals (#336); watchboard_entries means —watchboard-batch is too large for it and the oldest warnings in the digest were dropped (#337).
lookout_session_creates_total	counter	outcome	Total incident-open attempts against the configured sink (core-agent: POST /sessions; webhook: POST /incidents), labeled by outcome.
lookout_active_incidents	gauge	—	Current number of incidents in the sidecar’s dedup cache.
lookout_recoveries_observed_total	counter	resolution	Total kind=resolved outcome records emitted, by resolution (recovered|object_deleted).
lookout_recoveries_reverted_total	counter	—	Total kind=resolved.reverted records emitted: symptom recurred within the revert window after a resolve.
lookout_recovery_tracking	gauge	—	Current number of bound incidents the recovery tracker is watching for clearance.
lookout_recovery_drops_total	counter	cause	Total resolved signals dropped instead of injected, by cause (unknown_session: binding lost, e.g. restart without —dedup-persist).
lookout_storms_formed_total	counter	—	Total kind=storm incidents opened by blast-radius correlation.
lookout_storms_resolved_total	counter	—	Total storms resolved because every member incident cleared.
lookout_storms_active	gauge	—	Currently open (unresolved) storms.
lookout_storm_members_total	counter	kind	Total incidents folded into storms, by how they joined (suppressed: per-incident session never opened; superseded: pre-storm session pointed at the storm; attached: late arrival).
lookout_storm_updates_total	counter	—	Total kind=storm.update size refreshes injected into storm sessions (membership grew past a reporting threshold: doubling or +10, max one per minute).
lookout_watchboard_entries_total	counter	kind	Total warning-class signals buffered onto the shared watchboard digest, by signal kind.
lookout_watchboard_digests_total	counter	—	Total kind=watchboard.digest injects flushed to the watchboard session.
lookout_watchboard_rotations_total	counter	—	Total size-based watchboard session rotations: a fresh session opened after —watchboard-rotate digest injects.
lookout_watchboard_buffered	gauge	—	Warning-class signals currently buffered awaiting the next watchboard digest flush.
lookout_watchboard_reattached_total	counter	kind	Total buffered warnings delivered as a kind=family.member followup into an existing per-incident session sharing their blast-radius ancestor, instead of a digest entry (issue #220), by signal kind.
lookout_info_dropped_total	counter	kind	Total info-severity signals routed to the stored-only class (no inject anywhere), by signal kind. With —store set they are persisted; without it they are dropped after counting.
lookout_findings_total	counter	kind, severity	Total distinct findings the sentinel detected, by signal kind and severity — counted once per fresh dedup window, before severity routing, so stored-only and watchboard-batched findings count alongside injected ones. Always carries the cluster label. Namespace is deliberately absent (cardinality); rate() over this is the cluster’s health trend.
lookout_store_records_total	counter	route	Total occurrences committed to the store, by routing outcome (injected|suppressed|storm|storm-member|watchboard|info-stored|resolved).
lookout_store_write_drops_total	counter	cause	Total occurrence records LOST by the store’s write path, by cause (buffer_full: the non-blocking writer buffer overflowed; write_error: a batch insert failed). The store is telemetry, not a system of record — drops are loud, never blocking.
lookout_store_pruned_rows_total	counter	cause	Total occurrence rows deleted by the prune loop, by cause (ttl: older than —store-ttl; size: oldest-first eviction after —store-max-mb was exceeded).
lookout_enrichments_total	counter	outcome	Total enrichment runs, by outcome (ok: every stage succeeded; partial: some stage failed, the rest attached; failed: no section computed — the inject still fires, carrying enrichment_error trailers; skipped: nothing to build, the incident object is not a workload and names none, so the inject fires with no bundle at all rather than one describing the resolver).
lookout_enrichment_bytes	histogram	—	Size of the attached enrichment bundle in bytes, after the —enrich-cap prefix cut (the telemetry that will inform the fixed-vs-model-aware cap revisit).
lookout_enrichment_truncated_total	counter	—	Total enrichment bundles the —enrich-cap byte budget truncated at a section boundary (dropped sections become overflow trailers naming the follow-up command).
lookout_enrichment_failures_total	counter	stage	Total enrichment stage failures, by stage (resolve|spec|delta|edges|radius|logs). Failures never block the inject; they surface as enrichment_error trailers in the attached bundle.
lookout_memory_facts_total	counter	class	Total distilled facts written (upserts included) by the scheduled distiller pass, by fact class.
lookout_distill_errors_total	counter	—	Total failed distiller passes. A failed pass loses freshness only — the next pass re-derives every fact from the occurrence window.
lookout_triage_overrides_total	counter	action	Total severity-routing decisions refined by an open triage-status record, by action (downgraded: agent’s severity_override lowered the class; upgraded: it raised it; escalated: status=escalated pinned critical).
lookout_triage_resolved_flips_total	counter	—	Total triage-status records flipped to resolved by recovery injects (the automatic lifecycle — resolved records join the corpus).
lookout_triage_regressed_total	counter	—	Total kind=triage.regressed evidence followups: a downgraded incident’s dedup-window count reached —triage-regress-factor times its count at downgrade time. Evidence only, never a re-page.
lookout_cross_source_followups_total	counter	source	Total dedup-window duplicates injected as followups because their source family differs from the incident’s opening source (leading/reactive joins made session-visible), by joining source family.
lookout_sink_info	gauge	sink	The configured agent sink (—sink), value fixed at 1 on the active label (core-agent|webhook). ADDITIVE metric: the sink is process-level config, so it rides this info gauge instead of a new label on the operation counters — existing scrapes keep their exact series identities.
lookout_runner_up	gauge	—	1 while this cluster’s watch loop is running, 0 otherwise. Always carries the cluster label; in a multi-cluster process (issue #208) one series per watched cluster reports that runner’s liveness independently.
lookout_runner_restarts_total	counter	—	Total in-process restarts of this cluster’s runner by the supervisor after it exited while the process stayed up (multi-cluster fate isolation, issue #208). Stays zero in the single-cluster default, where a runner exit ends the process and the kubelet owns restart.
lookout_runner_terminal	gauge	reason	1 when the supervisor has GIVEN UP on this cluster: the runner exited for a reason no retry can fix (access_denied — the authorizer refused a required permission), so it is no longer being watched and no longer being restarted (issue #383). ADDITIVE metric rather than a label on lookout_runner_up, which keeps its exact series identity. The alert to write: a series at 1 means a cluster in the fleet is dark until someone changes a grant. Stays absent in the single-cluster default, where such an exit ends the process instead.
lookout_source_denied	gauge	source, resource, required	1 when a permission this source held at STARTUP is denied now, confirmed over consecutive SelfSubjectAccessReview sweeps (—access-recheck, issue #385); back to 0 when the grant returns. required=true means the source cannot run at all and this cluster’s runner is stopping for it; required=false is one degraded dimension on a source that keeps going. The alert to write: any series at 1 means the sentinel has lost coverage it used to have — the silence from that source no longer means the cluster is healthy.
lookout_cluster_resolve_errors_total	counter	cluster, cause	Total clusters this process was told to watch and did not, by cluster and cause (issues #388, #410). credentials: the cluster could not be resolved into a client. duplicate_name: two clusters in the fleet share this name, which is the only handle the sentinel has on a cluster, so neither is watched. The cluster is SKIPPED, not fatal, so the rest of the fleet still runs — which means a non-zero value is a coverage gap: nothing is watching that cluster and its silence means nothing. Counted at startup, so it moves on process restart and on nothing else.
lookout_leeway_subjects_tracked	gauge	subject_kind	Subjects with a tracked distribution, by kind.
lookout_leeway_node_groups_discovered	gauge	—	Distinct node groups resolved from the node-group label precedence list, before the tracking bound is applied. Read it against lookout_leeway_subjects_tracked with subject_kind=NodeGroup: the two agree on a healthy cluster, and a large number here with none tracked is leeway refusing a precedence list that resolved to something per-node. Raise —topology-max-node-groups only once you believe the count.
lookout_leeway_domains_unavailable	gauge	topology_key	Topology domains with no usable node, by axis (leeway). Reported per configured axis and zero when nothing is out, so the series exists before the first outage. This is the live reading; lookout_leeway_alert_state with subject_kind=Domain is the same fact after its dwell, and that is what leeway.domain_unavailable is emitted from.
lookout_leeway_domain_ready_nodes	gauge	topology_key, domain	Usable nodes per topology domain.
lookout_leeway_domain_objects	gauge	namespace, subject, subject_kind, topology_key, domain, state	Opt-in. Objects counted per subject, topology domain and scheduling state. By default state is collapsed to two values: active for an object holding the domain’s capacity (running, or terminating and not yet gone) and waiting for one that is not (pending or unschedulable). Pass —topology-per-domain-collapse-states=false for the four raw states, which share no label value with the two, so a query written for one mode returns nothing at all under the other rather than an undercount.
lookout_leeway_domain_series_withheld	gauge	reason	Subjects whose per-domain breakdown is absent or incomplete, by the cardinality control responsible. Read this before concluding a subject has no objects in a domain: the absence of a domain_objects row means either that or that leeway declined to export it, and this is which. gate is below the drift floor and not alerting, which is the intended standing state and is normally most of the estate; namespace is the allow or deny list; key_cap counts subjects that kept some axes and lost the rest, and is the one to watch, because it is the reading that says a dashboard is missing an axis rather than a subject. Counted in subjects for all three, so they are summable.
lookout_leeway_domain_expected	gauge	namespace, subject, subject_kind, topology_key, domain	Opt-in. Objects apportioned to each topology domain, the expectation domain_objects is scored against.
lookout_leeway_observed_skew	gauge	namespace, subject, subject_kind, topology_key	S, the difference between the fullest and emptiest eligible domain (leeway).
lookout_leeway_excess_skew	gauge	namespace, subject, subject_kind, topology_key	E, observed skew beyond what the arithmetic and the declared bound allow (leeway). Zero is the normal reading: a subject that cannot be spread any more evenly than it already is scores zero here however lopsided S looks, which is the whole reason drift is not alerted on S.
lookout_leeway_drift	gauge	namespace, subject, subject_kind, topology_key	ρ, the fraction of a subject’s objects that would have to move to meet its expectation (leeway).
lookout_leeway_max_domain_share	gauge	namespace, subject, subject_kind, topology_key	The share of a subject’s objects held by its fullest domain (leeway).
lookout_leeway_relocation_distance	gauge	namespace, subject, subject_kind, topology_key	R, the number of objects that would have to move to meet the expectation (leeway).
lookout_leeway_intent_info	gauge	namespace, subject, subject_kind, topology_key, mode, source, confidence, weighting, max_skew	Placement intent inferred for a subject on one topology axis, as labels on a constant 1 (leeway). Only subjects that expressed an intent are present: a workload with no spread constraint, anti-affinity or affinity has no row here, which is what makes the series count a property of the estate’s declarations rather than of its size. A learned baseline is deliberately absent for the same reason, even though it scores and routes like any other intent: it is not something anybody declared, and it is eventually present for every subject. See lookout_leeway_baselines for those. source is what the intent was read from and confidence how much that source is worth — assumed means k8s-lookout guessed a cluster default it could not read, and every finding derived from it rests on that guess.
lookout_leeway_alert_state	gauge	namespace, subject, subject_kind, topology_key, tier, phase	Where one subject-axis sits in the dwell machine: 1 pending, 2 firing. Only subjects with an open episode are present — a subject that is not drifting has no row rather than a zero, which keeps the series bounded by how much trouble a cluster is in rather than by how large it is. A resolving subject (clear, but inside the resolve dwell) still reads 2, because its finding is still outstanding.
lookout_leeway_transient_subjects	gauge	topology_key, transient	Subject-axes whose judgement suppressed or relaxed, by the transient state responsible. This is the series to look at before believing a quiet estate: a fleet-wide domain-outage row is leeway declining to page four hundred workloads about one dead zone, and a cluster-warmup row that never clears is a sentinel that never synced. Only the axes under a transient are present, so zero rows is the healthy reading.
lookout_leeway_last_event_timestamp_seconds	gauge	resource	Unix time of the last informer event leeway processed, per resource.
lookout_leeway_evaluation_duration_seconds	histogram	subject_kind	Time spent evaluating one coalesced subject.
lookout_leeway_counter_mismatch_total	counter	subject_kind	Subjects whose incremental distribution disagreed with a rebuild from the pod cache and were repaired in place, by kind (leeway). The alert to write: threshold zero. The two numbers are two computations of the same thing, so any non-zero rate is a BUG IN K8S-LOOKOUT and not a cluster condition — every finding derived from the drifted counters until it is fixed is wrong in the same direction. The repair keeps the next hour’s numbers usable; it is not a fix.
lookout_leeway_baselines	gauge	state	Subject-axes with a learned baseline, by state: learning is still inside a maturity gate, mature is old enough and sampled enough to be scored against, and frozen is being held still because its subject is firing or under a transient. The three overlap — a frozen baseline is also learning or mature — so they do not sum to the total. This is the series to read before turning Tier C baseline signals on: mature is how many subjects would start being judged against what they normally do rather than against an even split, and a mature that never climbs means something is resetting the baselines — check the reset outcome on baseline_samples_total.
lookout_leeway_baseline_samples_total	counter	outcome	Baseline samples, by what the estimator did with each (leeway). A healthy estate is almost all applied. A sustained reset rate is the failure mode worth alerting on: the domain set is the invalidation fingerprint, so something churning it — a zone label appearing and disappearing on nodes — restarts every affected baseline’s history and keeps it permanently immature, silently. empty is subjects with nothing to learn from (scaled to zero, or every pod Pending) and held is a frozen set, and neither is a problem.
lookout_leeway_preference_pod_time_seconds_total	counter	provider, axis, spec_hash, rank	Pod-seconds accumulated at each preference rank of a compute class (leeway). This is the series the whole source exists for, and it is time-weighted on purpose: a ninety-second burst of rank-3 pods during a scale-up and three weeks parked on a spot fallback look identical to a gauge. rank is the derived preference TIER, not the raw ccc_priority_index — on a class that sets priorityScore those are different numbers and can even run in opposite directions. Three ranks are not numbers at all: unknown is a node whose rank is not yet resolved, unsatisfiable one GKE could not fit to any rule, and off-axis one provisioned outside the priority list entirely. spec_hash changes when the class is edited, which starts a new series rather than averaging two different definitions of rank 1 together.
lookout_leeway_preference_rank_weighted_time_seconds_total	counter	provider, axis, spec_hash	Sum of rank times pod-seconds, over tier ranks only (leeway). Divide by the pod_time summed over the same ranks to get mean achieved rank: 0.0 is an estate getting its first choice, and a number that climbs is capacity quietly draining out from under it. The unknown, unsatisfiable and off-axis buckets are excluded from both halves, because a mean over a bucket whose rank is unsatisfiable is not a mean of anything.
lookout_leeway_preference_pods	gauge	provider, axis, rank	Pods currently occupying each preference rank. The supporting gauge to pod_time, not a substitute for it.
lookout_leeway_preference_nodes	gauge	provider, axis, rank, rule_index, rule, source	Nodes currently resolved to each preference rank, by where the rank came from. source=annotation is GKE’s own ccc_priority_index, inferred is k8s-lookout matching the node’s attributes against the class’s priority rules, and none is neither answering. rule_index is the raw list position and rule renders it for humans — both identity, never ordering.
lookout_leeway_preference_transitions_total	counter	provider, axis, from_rank, to_rank, lateral	Nodes moving between placements on one axis (leeway). Node-level, not pod-level: spike S1 established that a compute-class fallback provisions a NEW node and the ReplicaSet creates a new pod on it, so no pod ever changes rank. lateral=true is a move between two rules sharing a tier — capacity churn within a preference level, counted because it shows where the estate is thrashing, and excluded from every degradation signal because an equal-score alternative is not a demotion. A from_rank of unknown is the node’s annotation arriving, which happens once in the first minute of every node’s life and is not a fallback.
lookout_leeway_preference_unmatched	gauge	provider, axis	Nodes no priority rule matched. The alert to write: threshold zero. Every one of these is a node whose rank rests on GKE’s undocumented annotation with nothing checking it, and a sustained non-zero reading means the matcher has fallen behind the rules people are actually writing. Counted even when the annotation answered, because the coverage gap that matters is the one the working primary path is hiding.
lookout_leeway_preference_ambiguous	gauge	provider, axis	Nodes more than one priority rule matched. Two rules admitting the same node is legal and usually harmless — GKE takes the first — but it means the inferred rank is a guess between them, so the cross-check on these nodes is worth less than it looks.
lookout_leeway_preference_disagreement	gauge	provider, axis	Nodes where the annotation and the inferred rank disagree. The other threshold-zero alert, and the more serious of the two: the annotation wins, so a non-zero reading is not a wrong rank — it is evidence that the matcher’s model of the rules is wrong, and therefore that unmatched and ambiguous cannot be trusted either.
lookout_leeway_preference_out_of_range	gauge	provider, axis	Nodes whose annotation names a priority index the class no longer has. This is a class that was edited under a running node: GKE stamped index 3 and somebody has since deleted a rule. The node keeps running; its rank is simply unknowable, and it reads as rank unknown until it is replaced.
lookout_leeway_preference_axis_invalid	gauge	provider, axis	Nodes on a class whose priorities are partially scored. A class where some rules set priorityScore and others do not has no well-defined order — list position and score would rank it differently — so k8s-lookout declines to score it at all rather than pick one. Fix the class: score every rule or none.
lookout_leeway_preference_no_rule_matching	gauge	provider, axis	Nodes GKE itself could not fit to any priority rule (ccc_no_rule_matching). GKE’s own verdict, not ours, and a different statement from unmatched: this is the provisioner saying the node it made satisfies nothing the class asked for. Reads as rank unsatisfiable.
lookout_leeway_preference_off_axis	gauge	provider, axis	Nodes GKE provisioned outside the priority list (ccc_scale_up_anyway). The class’s whenUnsatisfiable let the autoscaler ignore the priorities rather than leave pods Pending, so these nodes are not at any rank — they are off the axis. Reads as rank off-axis, and their time is still reported under pod_time, because a week spent off-axis is itself the finding.
lookout_leeway_preference_rank_pending	gauge	provider, axis	Nodes carrying a compute class with no rank resolved yet. Normal and brief: GKE writes ccc_priority_index 33 to 44 seconds after a node registers, and inference has nothing to say about a node whose labels have not all landed either. A reading that does not decay to zero is the one to look at.
lookout_leeway_preference_unsupported_rules	gauge	provider, axis, field	Priority rules the matcher could not evaluate, by the field responsible. The matcher fails closed: a rule naming a field pkg/leeway does not model is excluded from inference rather than matched on the fields it does understand, because a matcher that ignores what it cannot read attributes nodes to the wrong rule and corrupts the cross-check into agreement with nothing. A new field appearing here is a feature request with the field name already filled in.
lookout_leeway_preference_unreadable_class_nodes	gauge	provider, class	Nodes labelled with a compute class this source has no decoded spec for. Either the class object has not synced, which is brief, or it failed to decode, which is not — see decode_errors_total. These nodes are ranked against nothing and their pods accrue no pod-seconds, so a sustained reading means the rank shares are being computed over less than the whole estate.
lookout_leeway_preference_decode_errors_total	counter	provider, class	ComputeClass objects this source refused to read, by name. A refusal is deliberate: a spec whose priorities are not a list, or whose rules are not objects, would produce a plausible-looking axis that is wrong. The row survives the object’s deletion, because an error counter that vanishes with the broken object hides the edit that broke it.
lookout_leeway_preference_tracker_underflows_total	counter	—	Unbalanced pod departures in the rank accounting. The alert to write: threshold zero. This counts a pod leaving a rank that had nobody at it, which is a BUG IN K8S-LOOKOUT and not a cluster condition — every rank share is skewed by an unknown amount until it is fixed. The count is clamped rather than allowed to run negative, so the pod-second counters stay monotonic; that keeps them readable, it does not make them right.
lookout_leeway_preference_axis_info	gauge	provider, axis, spec_hash, ordering, rules, tiers, scale_up, active_migration	One preference axis, as labels on a constant 1. ordering is how the tiers were derived — list-position is bare list order, priority-score is the field of that name (higher is more preferred, the OPPOSITE direction to list position), and invalid is a partially-scored class. rules is how many priorities the class declares and tiers how many distinct preference levels they collapse to; tiers of 1 means every node on the class is rank 0 by construction and nothing here can degrade. scale_up is whenUnsatisfiable as DECLARED — unset is not the same evidence as GKE’s documented default, and gates a finding on the difference.
lookout_leeway_preference_alert_state	gauge	provider, axis, rank_rule, tier, phase	Where one rule on one axis sits in the dwell machine: 1 pending, 2 firing. Episodes are per RULE, not per axis: a class can be running almost entirely on its last rank and also have a tier nobody has touched in a month, and resolving the first must not close the second. Only rules with an open episode are present — a healthy class has no rows rather than zeroes. A resolving episode still reads 2, because its finding is still outstanding.
lookout_leeway_preference_wedged_pods	gauge	provider, class	Unscheduled pods that named a compute class by nodeSelector and did not get one. The only symptom a wedged class has: on a DoNotScaleUp class no priority can be satisfied and the autoscaler will not provision outside the list, so the pods stay Pending and nothing in the rank distribution moves at all. Non-zero on a scale-up-anyway class is a different story — those pods are waiting on something else, which is why the finding is gated on whenUnsatisfiable being DECLARED and this gauge is not.
lookout_otlp_exports_total	counter	outcome	Opt-in. Total OTLP metric export attempts, by outcome (ok: the collector accepted the batch; failed: it did not, and the batch was dropped). A rising failed rate means the OTLP backend is stale; the scrape endpoint is unaffected.
lookout_otlp_points_exported_total	counter	—	Opt-in. Total metric data points the OTLP exporter delivered to the collector.
lookout_otlp_points_dropped_total	counter	—	Opt-in. Total metric data points discarded because their OTLP export failed or ran out of time. There is no retry queue by design, so a dead collector costs samples rather than memory; temporality is cumulative, so what is lost is the sample and not the counter value.
lookout_otlp_export_last_success_timestamp_seconds	gauge	—	Opt-in. Unix timestamp of the last OTLP metric export the collector accepted; zero until the first one. Alert on its age rather than on the failure counter alone, which stays flat when the export path stops running at all.
lookout_otlp_export_inflight	gauge	—	Opt-in. 1 while an OTLP metric export is in flight, 0 otherwise. A reading stuck at 1 across scrapes is an export wedged against its deadline.