# Configuration drift is the silent killer: the settings nobody tracks (Problem Scope 2/5)

> Configuration drift hides in the BIOS, firmware and kernel settings no one told config management to check. Why no alert fires, and how to see it on every node.

- URL: https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer/
- Author: Manish Gautam
- Published: 2025-10-20
- Tags: Fleet problems

Configuration drift is the gap between what your fleet is set to do and what you meant it to do. It builds up in the settings no configuration management system was told to manage, such as BIOS profiles, firmware defaults and kernel parameters, and it rarely trips an alert.

Google's SRE team found that roughly 70% of outages are due to changes in a live system ([Site Reliability Engineering, chapter 1](https://sre.google/sre-book/introduction/)), and a configuration change is a change like any other.

The reason drift slips through is structural and counterintuitive. Configuration management systems were built to enforce consistency at scale, but only across the relatively small set of parameters teams explicitly choose to manage. Firmware defaults, BIOS profiles, driver behaviours, kernel settings, power states: much of the infrastructure still operates outside that control surface. Most of those undeclared settings live in the hardware layer, which is why [the firefighting loop so often starts there](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts).

When configuration systems deliver the right state, they are extraordinarily powerful. When they deliver the wrong state, impact propagates everywhere simultaneously because consistency was the entire objective. Reality, however, is usually more subtle: slow drift across untracked configuration layers that [quietly degrade performance over time](https://blog.asama.ai/blog/issue-01-asama-uptime-is-not-reliability). Some are eventually discovered and corrected. Most continue running unnoticed.

> A software defect might take out one region. A configuration mistake hits everywhere at once, because consistency was the whole point.

## The drift nobody notices

The more expensive failure mode is gradual drift: the accumulated delta between what a system is currently configured to do and what it was originally designed to do, built up through a series of small, individually reasonable changes over months or years.

| How drift accumulates | Why it's hard to catch |
|---|---|
| [Firmware update resets a BIOS performance setting](https://blog.asama.ai/blog/article-01-asama-your-team-is-trapped-in-a-loop) | No alert fires: all values within thresholds |
| Kernel parameter changed on subset of nodes | Each change is individually reasonable |
| Driver version diverges across procurement batches | Asset databases updated manually, fall behind |
| Security patch applied to production but not staging | Monitoring watches metrics, not config state |
| BMC firmware changes a power management default | Root cause requires manual node-by-node archaeology |

## The question most teams cannot answer

Ask any infrastructure team: does what is currently running on your fleet match what you intended to deploy? Not in theory: in practice, right now, across every node.

Most teams cannot answer with confidence. Asset databases fall behind. Configuration management systems re-check only the settings they were told to manage, and nothing else. Monitoring tools watch metrics, not configuration state.

Closing this gap requires treating configuration state (across firmware, OS, drivers, kernel parameters, BIOS settings) as a first-class observable. Not something you check when something breaks. Something you monitor continuously and alert on when it deviates.

## Food for thought

When was the last time you verified that your fleet's actual configuration matched your intended baseline: not a sample, but every node?

**Update, October 2026.** Asama's fleet visibility, down to BIOS settings, firmware levels and PCIe links on every node, is generally available, and detection built on it has been validated; investigation and remediation are expanding. To see where your fleet has drifted, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.