# Normal Is a Moving Target > Every detection claim in infrastructure rests on a definition of normal that almost nobody writes down. - URL: https://blog.asama.ai/blog/normal-is-a-moving-target/ - Published: 2026-07-04 - Tags: Articles ## Executive summary Anomaly detection, drift analysis and health scoring are all downstream of one claim: what this system should look like when nothing is wrong. If that claim is wrong, everything built on it is wrong too, and almost no monitoring system states it explicitly. Normal is not the absence of problems. It is a statistical characterisation of expected behaviour under expected conditions, and it is a property of a metric, in a context, at a point in time. Remove any of those qualifiers and the concept is undefined. Three forces keep it moving: workloads that change what the hardware is doing, fleets that change what the hardware is, and deliberate operating choices that change what counts as acceptable. None are data quality problems. They are structural properties of the systems being observed. The useful reframe is to stop asking along which axes normal varies and start asking what conditions must hold for two observations to be validly comparable. That question resolves into four: are we measuring the same thing, is the generative process unchanged, is the comparison group genuinely exchangeable, and how much of the relevant signal was visible at all. They are not four parallel models. They are a pipeline in which each stage gates the next. The output is not a score. It is a score with an envelope describing how well each condition held. --- Is this node degraded? Has this configuration drifted? Is this metric anomalous? All three are comparisons, and none can be answered without a reference point. In most estates that reference point has never been written down. It is implicit in a threshold set four years ago, or in a baseline function quietly averaging the last fourteen days, and it carries more weight than anything else in the detection stack. > "Every detection is a comparison. The interesting question is never the measurement. It is what the measurement is being compared against, and whether that thing is still true." For a single static server running a fixed workload, this is tractable. Observe it long enough, fit a distribution, flag deviations. Fleets are not static. They are heterogeneous, continuously evolving systems in which the conditions that define normal are themselves in motion. --- ## 01 — Why It Moves Three forces drive that motion, and none of them is a modelling failure. **Workload evolution.** Infrastructure is purpose-built to be repurposed. The same cluster may run large-scale training one week and low-latency inference the next. Those are not variations, they are different operational regimes. Power draw, thermal profile, memory access pattern and PCIe utilisation are determined more by what the hardware is doing than by what it is. A baseline built during training is actively misleading during inference. **Fleet and system evolution.** Firmware updates change power management behaviour. New nodes arrive with slightly different silicon. Topology changes alter traffic patterns. OS upgrades shift how software touches hardware. Each can move the distribution of normal for part or all of the fleet without any degradation occurring at all. **Operational and economic context.** Operators deliberately redefine acceptable behaviour. Cost optimisation may mean running with less thermal headroom than a conservative baseline allows. SLA tiering may mean some nodes are intentionally operated differently. From a naive baseline this looks like drift. It is intent, and a system that cannot tell the difference generates noise that erodes trust in everything else it says. The answer to "what is normal for this system" therefore cannot be a static number, a static distribution, or even a slowly-updated model. --- ## 02 — The Question Worth Asking The obvious framing is to ask along which axes normal varies, and then to model each axis. That framing produces a long list and no structure. The better question is narrower: **what conditions must hold for two observations to be validly comparable?** That turns a modelling problem into a set of preconditions. An anomaly score is a comparison between an observation and a baseline. If the two are not comparable, the score is not weak, it is meaningless, and no amount of statistical sophistication downstream repairs it. Four conditions have to hold. | Condition | The question | If it fails | | --- | --- | --- | | **Signal identity** | Are these observations drawn from the same operational regime? | The baseline fits the data and the scoring is systematically wrong. Degradation in one regime is masked by healthy behaviour in another | | **Distributional stationarity** | Is the generative process behind the metric unchanged? | Real anomalies get absorbed into the baseline, or legitimate evolution gets flagged forever | | **Peer validity** | Is this comparison group a genuine equivalence class? | Fleet-wide problems become invisible, because every node looks normal against neighbours that share the problem | | **Observer bias** | How much of the relevant signal is actually being seen? | High-confidence scores are produced in exactly the cases where confidence is least warranted | These are not four models running in parallel. They form a pipeline, and the order matters. Signal identity gates stationarity: you cannot assess whether a distribution has changed until you know which regime's distribution you are looking at. Stationarity gates peer validity: peer comparison is only meaningful once the baselines being compared are current. Observer bias gates nothing and qualifies everything, because a score is only as good as the fraction of relevant signal that was visible when it was computed. --- ## 03 — What Each Condition Requires **Signal identity** makes workload regime a first-class dimension of the data model rather than a metadata tag applied afterwards. Regimes can be discovered from telemetry alone by clustering multivariate signals such as utilisation, memory bandwidth, PCIe throughput, power and thermals, then classified in near real time. Every reading carries a regime label and baselines are partitioned by regime. The hard part is transitions. Workload changes are not instantaneous, and mid-move a node's metrics belong cleanly to neither regime. Scoring has to be suspended through the transition, and the detector has to be conservative, requiring sustained evidence before committing to a new label. A jumpy classifier produces anomalies that are artefacts of its own indecision. **Distributional stationarity** means separating three kinds of change that look identical on a chart. Gradual drift, from ageing, thermal environment or incremental load, should be absorbed into the baseline. A structural break, from a firmware update or hardware replacement, should trigger recalibration rather than an alert. A transient deviation should be flagged as an anomaly and kept out of the baseline entirely. Change-point detection that returns a posterior probability rather than a binary flag makes this principled instead of heuristic. On detection the right move is not immediate recalibration but a provisional window: collect evidence on whether the new distribution is stable. A stable one triggers baseline versioning. A continuing shift means a transient or an ongoing degradation event. Distinguishing a legitimate break from degradation needs a side channel. That is a **confound register**: a log of discrete fleet events such as firmware upgrades, scheduled maintenance, hardware replacements and topology changes. A change-point coinciding with a known confound is tagged as likely-legitimate. One that does not is escalated. **Peer validity** is the strongest condition of the four, because exchangeability is a strong claim. Two nodes are exchangeable when, conditional on group membership, the variation between them is noise and individual difference rather than systematic difference in what they are doing or how they are configured. Two nodes with the same SKU fail that test if they run different workloads, sit at different firmware versions, occupy different thermal neighbourhoods, or differ substantially in age. Peer groups therefore have to be built on several axes at once, and membership has to be fluid. As a node's regime shifts, it should move groups. Below a minimum group size the system should fall back to broader priors rather than produce an unreliable peer-relative score. **Observer bias** is the least intuitive and arguably the most important. A baseline never represents the true state of the infrastructure. It represents the infrastructure as captured by a specific set of sensors, at a specific sampling rate, with a specific set of failure modes in mind. A node with full thermal telemetry and no memory error counters has a known gap. A metric sampled once a minute cannot see a fault that resolves in thirty seconds. These gaps are encoded silently into the baseline and shape normal without ever being stated. The response is a **sensor coverage map**: a matrix indexed by node and failure mode class, recording what fraction of the relevant signal is observed at sufficient resolution. Every score is annotated with a coverage confidence derived from it. --- ## 04 — The Two Cases That Break the Method Two failures sit outside what the pipeline can resolve on its own, and both need an anchor from outside. **Collective degradation.** If every node in a peer group degrades together, from a fleet-wide firmware issue, a systemic thermal problem or a shared environmental factor, peer comparison finds nothing. Each node looks normal relative to its peers precisely because they are all affected. The group's centroid has moved, and the centroid is the reference point. Detecting it requires tracking how each peer group's centre of mass moves over time, with an anchor that sits outside the group. In practice that means one of three things: a pinned historical snapshot, a cross-site reference cohort running comparable work, or a physics-based prior derived from what the hardware is specified to do. Without an external anchor, a fleet quietly normalises its own decline. **Systematic blind spots.** Individual instrumentation gaps show up in the coverage map. Gaps shared across the whole fleet do not, because they reflect design choices in the monitoring architecture rather than per-node failures. A fleet where every node is missing the same sensor class develops a baseline that is coherent, stable, and wrong in a way that is invisible from inside the data. These cannot be found from the fleet's own telemetry. The mechanism is periodic cross-fleet audit: comparing this fleet's coverage map and baseline structure against fleets with richer instrumentation, to find signal classes that are absent here and informative elsewhere. That is an audit function, not a real-time model, and it belongs on a calendar rather than in a pipeline. --- ## 05 — The Output Is an Envelope Because each stage qualifies the next, the honest output of the pipeline is not a single number. Two nodes flagged at identical severity can be in completely different epistemic positions. One sits in a large, stable peer group, in a confidently classified regime, with comprehensive coverage and no recent confounds. The other is a rare SKU with six peers, mid-transition between workloads, on a chassis missing a sensor class relevant to the failure mode in question. Both may be correct. Only one should be acted on without a second look, and the number alone does not say which. So a score should arrive with all four conditions attached. Not "this node is anomalous" but "this node is anomalous, the peer group is valid and large, the regime is confidently assigned, coverage is comprehensive." Or, equally usefully: "this looks anomalous, but the peer group is thin, the regime classifier is uncertain, and coverage of this failure mode class is partial." The goal is not confident scores. It is accurate ones, with honesty about where accuracy is limited. An operator who trusts a system's confidence estimates will act on them. One who has learned they are unreliable will ignore all of them, including the correct ones. --- ## From Asama > ### What this changes in the product > **Baselines are versioned, not continuous.** Every baseline carries a version, a timestamp, the event that triggered it, and the peer group composition in force at the time. Every score is computed against an identifiable baseline version, so any past detection can be audited after the fact. > > **Peer groups are explicit and auditable.** The composition of the group used to score an observation is visible, and it is possible to see why two nodes are or are not treated as peers. Where domain knowledge exceeds what the model captures, membership can be pinned. > > **Confidence is a first-class output.** Low-confidence scores are surfaced distinctly rather than blended in. Scores derived from partial sensor coverage carry that annotation explicitly. > > **The system models its own blind spots.** It can report not only that a node looks healthy, but that it looks healthy given the sensors available, and that a named failure mode class has limited coverage on that node. --- ## Food for thought > When your monitoring says a node is fine, what is it comparing against, and when did anyone last check that the comparison is still valid? --- *Asama is an AI infrastructure engineer built by operators who lived inside the problem for two decades. Our focus is MTTR. Detection is the prerequisite. Remediation is the mission.*