# Determinism, Probability, and the Limits of Observability: what the residual says about a fault

> Every fault model is some version of y = mx + c + ε. What the residual's shape says about a fault, where chaos caps prediction, and why ε never reaches zero.

- URL: https://blog.asama.ai/blog/determinism-probability-and-the-limits-of-observability/
- Author: Manish Gautam
- Published: 2026-05-05
- Tags: Frameworks

## Executive summary

Strip a fault model down to its simplest form and it reads y = mx + c + ε, where the residual ε is everything the model can't explain, and the limits of observability live in that one term. The common belief is that enough telemetry drives ε to zero. It doesn't. The world stays causal; what runs out is engineering. Some instruments don't exist, some interactions are too numerous to fit, and in chaotic regimes errors grow exponentially, so a tenfold gain in measurement precision buys only a fixed extra slice of forecast time.

Deterministic faults leave a small residual: establish the cause, then automate the fix. Probabilistic faults leave a large one with structure in it, so you rank the likely causes and state your confidence. A few show sensitive dependence on initial conditions; stop predicting those and design the system to absorb them. The shape of the residual tells you which zone you're in, and on a server fleet a large, structured residual is often state that nobody recorded.

## The equation every monitor is fitting

The simplest useful model in statistics is a straight line, y = mx + c. Thresholds, baselines and anomaly detectors all fit some version of it, and a neural network fits the general form, y = f(x) + ε. The full straight-line equation has one more term:

`y = mx + c + ε`

- **y** = observed outcome
- **x** = causal variable
- **m** = relationship strength
- **c** = baseline constant
- **ε** = the residual the model has not yet explained

Heat obeys it. A component's steady-state temperature follows

`T = T_inlet + R × P`

where P is the power it dissipates, R is the thermal resistance from die to air in °C per watt, and T_inlet is the temperature of the air at the server's front panel. It's Ohm's law for heat, and it's y = mx + c with x = P, m = R and c = T_inlet. Fit it across a few hundred identical nodes, plot temperature against power, and you get a straight line with a cloud of residuals around it, some nodes above it and some below.

Textbook regression assumes the errors behind that cloud are dull: zero mean, constant variance, each independent of the last and of x, and normally distributed if you want exact tests. In an engineered system ε misbehaves in four ways. Size breaks none of those assumptions, but it buries the signal; the other three break them outright, and each points at a different piece of missing physics.

## What the residual is made of

**Large.** The residual swamps mx: low signal-to-noise, R² near zero. The cause is often mundane, a sensor that reports whole degrees when the effect you're chasing is a fraction of one, or a sensor so far from the die that it reads the chassis. A large residual says your x is wrong or your instrument is too coarse. It doesn't yet say which.

**Structured.** Sort the residuals by rack position and a pattern appears: nodes near the top run warm, because hot exhaust curls over the top of the rack and back into the intakes. Most BMCs report inlet temperature, but the fit treated T_inlet as one number, so its variation went into ε. If a missing variable also moves with x, the estimate of m comes out wrong too, which statisticians call omitted-variable bias.

**Heteroskedastic.** The spread widens as power rises. Under load the fans climb in pitch, and fan speed changes R itself: more air, lower thermal resistance. Two nodes on different fan profiles agree at idle. At full load they part company. A residual whose variance moves with x points at a moderating variable, something that changes the slope; here it's a fan curve, which is a setting.

**Autocorrelated.** Each residual predicts the next. Fit the steady-state equation to a node whose load keeps changing and the residuals arrive in runs: the heatsink's thermal mass makes temperature lag power, so after each step up in load the node reads below the line for minutes, and after each step down it reads above. A daily cycle in inlet temperature does the same over hours. Any slow process outside the model leaves this signature, and the Durbin-Watson and Ljung-Box tests exist to catch it.

When I first wrote this framework down, in May 2025, I described an autocorrelated residual as one that grows over time. I had it wrong. Growth is drift, from dust settling on the fins or a thermal interface drying out, and drift is only one cause: the thermal lag and the daily cycle produce autocorrelation with no trend at all. Autocorrelation is correlation with the residual's own past.

All four point at something missing, in the model or in its instruments, and each says where to dig.

The universe is still causal. Your model just hasn't caught up with it yet.

Behaviour is a function of state, workload and environment, and monitoring records the workload and some of the environment, so on a server fleet the variable hiding in ε is often [state that nobody recorded](https://blog.asama.ai/blog/issue-10-asama-the-firmware-layer-nobody-owns): a fan profile, a firmware version.

## Where the engineering stops

In 1814 Pierre-Simon Laplace imagined an intelligence that knew every force in nature and the position of every particle. For it, he wrote, nothing would be uncertain. (He never called it a demon; that word was added later, by other people.) He drew the conclusion this framework rests on: ["Probability is relative, in part to this ignorance, in part to our knowledge."](https://en.wikisource.org/wiki/A_Philosophical_Essay_on_Probabilities/Chapter_2)

Quantum mechanics later put genuine randomness at the bottom of physics, and an alpha particle from decaying trace uranium or thorium, flipping a bit in a memory chip, is one place it reaches a server. Almost everything else in ε is ignorance, and ignorance decomposes into more x's. With infinite instruments, infinite compute and infinite patience, y = mx + c + ε could be completed down to that quantum floor, which lies far below the residuals in a fleet model.

Engineering is not conducted under infinite conditions.

At some depth the project stops paying. The sensor that would close the gap doesn't exist, the model can't be fitted, or the answer costs more than the failure it explains. Combinatorics gets there quickly: 200 candidate variables have 19,900 pairwise interactions and 1,313,400 three-way ones, each needing data from conditions you may never see.

Reliability engineering has names for the two sides of that line. Uncertainty you could reduce with more data or a better model is epistemic; uncertainty you treat as irreducible randomness is aleatory. In 2009 Armen Der Kiureghian and Ove Ditlevsen made the point that matters here: [the label is a property of the model](https://doi.org/10.1016/j.strusafe.2008.06.020). An uncertainty is aleatory when "the modeler does not foresee the possibility of reducing" it.

The engineering boundary sits wherever the modeller stops. It also moves. Hardware performance counters, eBPF and fleet-wide baselines keep pulling variables out of ε that were out of reach a few years earlier.

> "The epsilon does not go to zero not because the universe is non-deterministic, but because the engineering project of making it go to zero is unbounded in cost."

## Three zones

Inside today's boundary, faults fall into three zones. They shade into each other, and you won't always know which one you're in until you've investigated.

### Deterministic: close ε, then automate

The residual is small, stable and cheap to close. The causal chain is observable with the instruments you have, and the relationship holds across nodes, loads and weeks. [A PCIe replay counter climbing under GPU load](https://blog.asama.ai/blog/issue-07-asama-the-correctable-error-trap) alongside a link-width downgrade is one example; [a BIOS power profile](https://blog.asama.ai/blog/issue-10-asama-the-firmware-layer-nobody-owns) that caps CPU frequency is another, because the setting explains the clock speed almost completely.

Find the cause first. Detection gives you the signal. Root-cause analysis gives you the right to act on it. Once the chain holds and a wrong action is cheap to undo, automated remediation under human approval is the right call.

### Probabilistic: rank the causes, state the confidence

The residual is large, for engineering reasons. The secondary variables exist but today's instruments can't reach them, or the interactions run too deep to model at a sane cost. [SMART read retries and error-correction counts](https://blog.asama.ai/blog/issue-06-asama-what-smart-data-isnt-telling-you) carry real information about a drive's remaining life without a clean path from today's value to tomorrow's failure; corrected ECC errors do the same for memory.

Pretending to certainty here is how automation does damage. The precursor to action is probabilistic root-cause analysis: the likely causes, ranked, each with a confidence and the evidence behind it. That confidence, not the anomaly, decides how much you automate.

This zone also hides what systems engineers call common-cause failure. Der Kiureghian and Ditlevsen note that epistemic uncertainties "may introduce dependence among random events", and redundancy maths assumes dependence away. Give each of two redundant units a 1 in 50 chance of failing in a year, with no repairs, and if the failures are independent the chance of losing both is 1 in 2,500. Now let 5% of each unit's failure probability come from a cause the pair shares, a batch defect or a firmware bug; 5% is the lower of the two screening values that the [US Nuclear Regulatory Commission's common-cause guidance](https://nrcoe.inl.gov/publicdocs/CCF/NUREGCR-5485_Guidelines%20on%20Modeling%20Common-Cause%20Failures%20in%20PRA.pdf) gives for a redundant pair. The chance of losing both rises to about 1 in 735, 3.4 times higher. Neither unit changed. The shared variable was in ε all along.

### Irreducible: budget for resilience

The third zone is rare, and its physics is old. In 1889 Henri Poincaré won King Oscar II's prize for a memoir on the three-body problem, then found an error in it: two asymptotic surfaces he had assumed to join smoothly cross instead. [He paid for the scrapped printing himself](https://www.mittag-leffler.se/about-us/history/prize-competition/), over 3,500 kronor against a 2,500-kronor prize, and the correction was the first glimpse of what is now called chaos. By 1908 he could put the result in one line: "small differences in the initial conditions produce very great ones in the final phenomena."

In the winter of 1961 Edward Lorenz [restarted a weather simulation halfway through](https://www.aps.org/apsnews/2003/01/lorenz-butterfly-effect-1961), typing a value from a printout, .506, in place of the .506127 the machine held in memory. He walked down the hall for a coffee. When he came back an hour later the new run had wandered far from the old one, and his 1963 paper, "Deterministic Nonperiodic Flow", worked out why.

Lorenz's equations contain no random term. Two nearby states still separate, roughly as `δ(t) ≈ δ0 · e^(λt)`, where λ is the largest Lyapunov exponent, so the forecast horizon is

`T ≈ (1/λ) · ln(Δ / δ0)`

with δ0 your measurement error and Δ the error you can tolerate. Precision sits inside a logarithm. For Lorenz's system [λ is about 0.9056](https://sprott.physics.wisc.edu/chaos/lespec.htm) per unit of model time, so a sensor ten times better buys 2.54 more units of forecast and one a thousand times better buys 7.63. Every extra decimal place of precision buys the same slice of time.

Charles Perrow, a sociologist who studied accidents in engineered systems, reached a neighbouring conclusion without chaos theory. His Normal Accidents (1984), written after Three Mile Island, argued that complex, tightly coupled systems produce interacting failures however carefully you design them, and [a synchronous training job](https://blog.asama.ai/blog/issue-12-asama-gpu-fleets-are-the-new-mainframes), where one failed node stalls every GPU, is tightly coupled in his sense.

Use the label late. Most failures that look irreducible are under-instrumented probabilistic ones, and better telemetry has already tamed plenty. The two mistakes don't cost the same. Call a preventable failure irreducible and you pay for resilience against it on every future occurrence; keep investigating a chaotic one too long and you pay once, having built the resilience anyway, because probabilistic faults need it too.

Whether any fleet failure is chaotic in Lorenz's strict sense, I can't tell you. Showing it means estimating a Lyapunov exponent from measured data, which takes long, clean, densely sampled series, and fleet telemetry is mostly short, noisy and coarse.

## Read the zone off the residual

You don't get a label that says deterministic or probabilistic. You infer the zone from the residual, and the inference can be wrong, so treat every zone as a working hypothesis.

| What the residual looks like | Probable zone | What to do |
| :--- | :--- | :--- |
| Small and flat: the same across nodes, loads and weeks | Deterministic | Close it. Confirm the causal chain, then automate under approval |
| Large but patterned: it tracks another signal, widens with load or runs in streaks | Probabilistic | Hunt the missing variable, and meanwhile act on ranked causes with stated confidence |
| Large and patternless after the signals run out, with near-identical states diverging | Possibly irreducible | Conclude it last. Spend on isolation, buffer capacity and fast recovery |

The checks are cheap. [NIST's engineering statistics handbook](https://www.itl.nist.gov/div898/handbook/pri/section2/pri24.htm) recommends plotting residuals in time order to show runs and against fitted values to show a funnel; plot them against the variables you left out too, and structure shows up there. [Comparing each node with its peers](https://blog.asama.ai/blog/issue-11-asama-peer-comparison-is-the-baseline) does the same job across a fleet.

None of this is for the first five minutes of a novel incident. Until you know the zone, the safe default is the resilience playbook: isolate, cut the blast radius, restore service, and classify in parallel.

## Knight on risk and uncertainty

In 1921 Frank Knight's [Risk, Uncertainty and Profit](https://www.econlib.org/library/Knight/knRUP.html), a book that grew out of his Cornell dissertation, split the unknown in two. Risk is an unknown outcome from a known distribution, and Knight thought it so different from the unmeasurable kind that it was "not in effect an uncertainty at all." You can't say which drive in a batch will fail next week, but an annualised failure rate measured across enough drives of the same model lets you price the batch in spares, redundancy and insurance. Uncertainty is a failure mode nobody has seen, in a configuration that has never existed, with no history for data to come from. Optimisation doesn't touch it.

Resilience does.

Observability, at bottom, is the project of turning uncertainty into risk. It's worth doing, and it's accelerating, but it can't be finished. The epsilon term is not a problem to be solved; it is a feature of the system to be understood, and designed for.

---

## From Asama

> ### Automation earned zone by zone
>
> Asama is designed around this framework:
>
> - Deterministic faults get the full causal chain first, then a remediation plan that runs under human approval
> - Probabilistic faults get ranked causes, each with a confidence and the evidence behind it, and the confidence decides where a person steps in
> - For the irreducible zone, we recommend a calculated buffer of capacity, sized for the failures you can't predict, because that is the right design for dynamics no finite engineering project can predict far enough ahead
>
> It is designed to tell a residual worth chasing from one worth bounding.

---

## Food for thought

> Take one model you already trust, such as component temperature against power, and plot a month of its residuals in time order, then against load. Flat? If not, which variable is your model missing?

**Update, October 2026.** Asama's fleet visibility is generally available. Detection has been validated on Dell, HPE and Lenovo hardware, and root-cause analysis and remediation are expanding. About 70% of Asama's root-cause work is deterministic correlation and statistical inference across time, topology, cohort, and state and change; agentic reasoning then takes those results and the raw evidence and writes the causal chain. When Asama can't read something, it says so. To see which of your fleet's residuals are hiding state, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.