# The firmware layer nobody owns: infrastructure state your monitoring can't see (Problem Specifics 5/5) > Infrastructure state is what a server is, not what it does. Why a BIOS setting or firmware change surfaces only as a symptom, days later, and how to read it. - URL: https://blog.asama.ai/blog/issue-10-asama-the-firmware-layer-nobody-owns/ - Author: Jain Johny - Published: 2026-03-01 - Tags: Fleet problems Infrastructure state is the record of what a server is, as opposed to what it is doing: its firmware and microcode versions, its BIOS and BMC settings, its kernel parameters, driver versions and tunings, what each of those should be, and what last changed it. The firmware layer, everything between the operating system and the silicon, is the part of that state nobody owns. A BIOS setting reset to its vendor default emits no metric and fires no alert. A metric moves later, sometimes days later, and your monitoring sees the symptom without the setting that caused it. If you ask an infrastructure team who owns the application layer, they will name a team immediately. The network layer has an owner. The OS layer has an owner. Storage has an owner. Ask who owns the firmware layer (BIOS settings, BMC firmware, microcode versions) and the driver baselines that go with it, and the answer is usually a pause followed by something like *"it depends."* The firmware layer is owned in the same way a shared hallway is owned: notionally by everyone, practically by no one. > "The firmware layer is not invisible because the signals aren't there. It is invisible because your monitoring records what your servers do, not what they are." --- ## State is not telemetry Monitoring collects behaviour: metrics, logs and events. It tells you what a box is doing. Infrastructure state tells you what the box is, what it should be and what changed it. | | Question it answers | Firmware-layer example | Where it usually lives | | :--- | :--- | :--- | :--- | | **Behaviour** | What is the box doing? | CPU clocks run lower under load than on identical nodes | Monitoring | | **State** | What is the box? | Server profile set to Performance per Watt; BIOS on a new version | Nowhere, consistently | | **Intent** | What should it be? | This cohort's golden config: the Performance profile | A wiki page, a golden image or [one engineer's memory](https://blog.asama.ai/blog/issue-03-asama-the-tribal-knowledge-tax) | | **Change** | What moved it, and when? | Last week's BIOS update | A change ticket, if anyone raised one | Monitoring holds the first row. Infrastructure state is the other three, kept current for every node. Behaviour is a function of state, workload and environment. Monitoring watches the output of that function and has to guess at the inputs, and many different states produce the same slow job or the same hot CPU. When state changes, behaviour changes with it, but the metric that moves carries no record of why. --- ## The stack nobody manages The firmware layer is the part of your infrastructure state with the fewest owners. | Component | How it changes | Why nobody notices | | :--- | :--- | :--- | | **BIOS and UEFI settings** | Set at deployment; a BIOS update, a CMOS clear or a replacement system board can put them back to defaults | Rarely read again after deployment | | **BIOS and BMC firmware** | Updated ad hoc, node by node, with each vendor's own tools | No single view across vendors | | **CPU microcode** | Shipped in BIOS updates and in OS packages, and loaded at every boot | An OS package update changes it at the next reboot, often untracked | | **NIC, storage controller and drive firmware** | Arrives with the hardware, then changes in vendor update campaigns and RMAs | Revisions [diverge across procurement batches](https://blog.asama.ai/blog/issue-08-asama-bring-up-is-where-schedules-go-to-die) | | **GPU firmware (VBIOS, and GSP firmware on recent NVIDIA GPUs)** | VBIOS changes only when someone flashes a vendor firmware package; GSP firmware ships with the NVIDIA driver and changes with every driver update | Checked at install, rarely after | | **Power management policies** | Vendor defaults, unless someone sets a fleet baseline | Defaults vary by vendor and BIOS revision | Drivers live in the OS, not the firmware, though some firmware ships in OS packages (CPU microcode, NVIDIA's GSP firmware), and most device firmware is supported only with a range of driver versions, so the two belong in the same baseline. --- ## How firmware drift happens [Firmware drift accumulates through normal operations](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer). A BIOS update during a maintenance cycle resets a performance-critical setting to a default that does not match the fleet's golden baseline. A BMC firmware upgrade introduces a new default for a power management parameter that was previously customised. A CPU microcode update changes instruction-set behaviour in a way that interacts with a workload characteristic nobody anticipated. Intel's 2023 microcode fix for the Gather Data Sampling vulnerability, for example, [slows the AVX2 and AVX-512 gather instructions](https://access.redhat.com/solutions/7027704) that some workloads lean on. A replacement system board can arrive with factory defaults, and a replacement drive on a different firmware revision from its neighbours. None of these changes are malicious, and none of them are metrics. Each is a change in state, so nothing fires when it happens. What fires later, if anything does, is the degradation or instability they cause, and those symptoms [present as application or workload problems](https://blog.asama.ai/blog/issue-01-asama-uptime-is-not-reliability). --- ## Why monitoring sees the symptom, not the change A state change and the problem it causes are two events, at two layers, often days apart. Monitoring alerts on the second. The first, if it is collected at all, usually sits in a label nothing compares. | What changed in state | What your monitoring sees | Where it points you | | :--- | :--- | :--- | | A BIOS update resets the server profile to the vendor default | Lower clocks and slower responses under load | The application | | A BMC firmware upgrade brings a new fan or power policy default | Higher CPU temperatures and throttling under sustained load | Cooling, or the room | | A microcode update ships a security fix | Slower runs of the workloads that use the affected instructions | The code, or the compiler | | A reboot drops tunings someone applied by hand and never persisted, such as NIC ring sizes or IRQ affinity | Packet drops and softirq load on those nodes | The network | That is how [the ten-server BIOS incident](https://blog.asama.ai/blog/article-03-asama-what-happens-when-the-loop-ends) went: a BIOS update reset the server profile from Performance to Performance per Watt, CPU utilisation looked normal, the BMC logged no faults, and it took 3 to 4 days, and three affected customers, to find one changed setting. Three things keep the change out of sight: - **Most of it is never collected.** Monitoring polls the OS for metrics and the BMC for sensors and events. The BIOS attribute registry, BMC settings, IRQ affinity and most kernel and driver tunings are rarely read at all. - **What is collected has no reference.** A version string can ride along as a label on an info metric, and [node_exporter's pcidevice collector turns PCIe link width, speed and power state into metrics](https://blog.asama.ai/blog/pci-device-metrics-in-prometheus-node-exporter). But a metrics store keeps a value against a timestamp, not what the value should be. The expected value lives in alert rules someone writes by hand, one setting and one hardware model at a time, and a tool that diffs a value against its last reading can tell you it changed, not that it is wrong. - **Cause and symptom are days and layers apart.** The state changes in a maintenance window. The symptom appears under load, days later, at another layer. A time-window correlation rarely reaches back that far, and by then the change ticket is closed. Some state never shows up as behaviour before the failure itself. In 2019, [HPE warned](https://www.hpe.com/us/en/services/sas-ssd-advisory.html) that certain SAS SSDs on firmware older than HPD8 would fail at 32,768 power-on hours, with data loss. Drives put into service together reach that hour together. Until then, a drive on the old firmware showed nothing its peers didn't, and its peers were due to fail with it, so no threshold and no peer comparison could flag it. Only its firmware revision, checked against the advisory, gave it away. Infrastructure faults come in two kinds: problems of behaviour, and problems of state, such as drift from the golden config, version skew and known defects. A tool that holds only behaviour sees a state problem only after it has become a behaviour problem. A tighter threshold won't fix any of this. A threshold watches behaviour, and the cause is in state. --- ## Reading the firmware layer ```bash sudo dmidecode -s bios-version # BIOS version grep -m1 microcode /proc/cpuinfo # microcode revision the CPU is running sudo ipmitool mc info | grep 'Firmware Revision' # BMC firmware, major.minor only ethtool -i eth0 | grep firmware-version # NIC firmware sudo smartctl -i /dev/sda | grep -E 'Firmware Version|Revision' # drive firmware (SAS drives print Revision) nvidia-smi --query-gpu=gpu_bus_id,vbios_version --format=csv # GPU VBIOS ``` Behind a RAID controller, `/dev/sda` is the virtual disk, so address each drive with `-d`, for example `sudo smartctl -i -d megaraid,0 /dev/sda`. Out-of-band, the BMC's Redfish service gives its own full version string as `FirmwareVersion` on its `Manager` resource, lists component firmware, as far as each BMC inventories it, under `/redfish/v1/UpdateService/FirmwareInventory`, and holds the BIOS settings under each system's `Bios` resource, though [the attribute names differ by vendor](https://blog.asama.ai/blog/issue-09-asama-the-heterogenous-problem-nobody-admits). Read all of this on every node, on a schedule, not once at deployment. Keep each value with the time it changed, and compare each node with the baseline for its cohort and with vendor advisories, since a firmware revision that matches its peers can still be one an advisory flags. --- ## What holding state takes Reading a version once is inventory. Holding it as state takes five things. | | What it takes | What breaks without it | | :--- | :--- | :--- | | **Breadth** | Every layer the state lives in, from BIOS, BMC, microcode and device firmware up through kernel, drivers and tunings | The cause sits in a layer nobody read | | **Depth** | Settings and versions held as values you can compare, against a baseline for each cohort | You see a value, not whether it is wrong | | **Normalisation** | One name for one setting, whichever vendor built the box | Comparing two vendors' nodes takes someone who knows both | | **Topology** | What connects to what (switch ports, PCIe trees, power feeds) and which nodes share a setting, a build or a cohort | The blast radius is worked out by hand | | **History** | Every change, fault and RMA, with when it happened and, where known, what triggered it | A metric shift can't be lined up with the change behind it | --- ## From Asama > ### Firmware held as state, not inventory > > Asama is designed to read the firmware layer on every node and hold it as state: > > - BIOS settings and versions > - BMC firmware > - CPU microcode > - NIC, storage controller, drive and GPU firmware > - Power management policies > > A snapshot taken at onboarding would bake in whatever drift already exists, so each node is designed to be compared against the golden config for its cohort, against vendor advisories and against its peers, with each change matched, where it can be, to a known fleet event such as a firmware rollout or a part swap. > > When a BIOS update silently resets a performance-critical setting, Asama is designed to flag the state change on the affected nodes before their metrics move, and plan the fix before the first customer complaint arrives. --- ## Food for thought > Do you know the current BIOS configuration of every node in your fleet, not what it was when deployed, but what it is right now, and what it should be? **Update, October 2026.** Asama now holds infrastructure state as one live model of the fleet, covering topology, baselines, configuration and history, updated as the fleet changes, and the firmware layer is part of it. Fleet visibility is generally available: it reads BIOS/UEFI, BMC, CPLD, NIC, HBA, drive and GPU firmware, CPU microcode and BIOS attributes on every node, and reads each BMC out-of-band over Redfish and IPMI, so a node can still be checked when its OS can't answer. Detection built on it, which compares each node against its cohort's golden config, vendor advisories, its peers and its own baseline, has been validated; investigation and remediation are expanding. To see where your fleet's firmware has drifted, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.