# Where the loop always starts: the hardware signals your monitoring misses (2/3) > PCIe replays, BIOS drift and disks that SMART calls healthy: the hardware signals behind silent degradation, and what 35,000 servers at Flipkart taught us. - URL: https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts/ - Author: Sumeet Ghosh - Published: 2025-07-28 - Tags: Fleet problems The loop most often starts in the hardware layer: PCIe links that train at half width, BIOS settings that drift after maintenance, disks that degrade while SMART still reports them healthy. None of these trips an alert, so each one surfaces days or weeks later as an application problem. ## The Hardware Layer In part 1, [I described the loop](https://blog.asama.ai/blog/article-01-asama-your-team-is-trapped-in-a-loop): the detect, firefight, manual-fix, repeat cycle that consumes infrastructure teams and compounds over time. I want to go deeper into where that loop most often starts: not in application code, not in scheduling logic, but in the hardware layer that every tool above it assumes is healthy. It usually isn't. Not in a dramatic way. Not in the way that trips alerts or causes outages. In the quiet, compounding way that infrastructure degrades: PCIe links negotiating at half capacity, BIOS settings silently drifting from their golden baseline, disks whose SMART data says "healthy" while their read error counts have been climbing for weeks. The hardware layer doesn't fail loudly. It fails in ways that the tools above it have no language for. Understanding this (really understanding it, at a component level) is the prerequisite for everything else. It's what I spent a decade learning at Flipkart, and it's the foundation Asama is built on. ## The Bathtub Curve: A Predictable Science Most Teams Ignore Many hardware components follow what reliability engineers call the bathtub curve: a failure rate that, plotted over time, looks exactly like its name. | Phase | Description | |---|---| | *Infant Mortality* | High failure rates from manufacturing defects, design flaws, component incompatibilities | | *Normal Life* | Stable, low failure rate. Random failures only, no pattern to predict | | *Wear-Out* | Failure rates climb as components approach end-of-life and degrade | Field data is less tidy. In a study of about 100,000 disks, replacement rates rose steadily with age instead of holding flat ([Schroeder and Gibson, FAST '07](https://www.usenix.org/conference/fast-07/disk-failures-real-world-what-does-mttf-1000000-hours-mean-you)), so for disks, at least, the quiet middle is shorter than the textbook suggests. The infant mortality phase is where the loop most reliably gets triggered. New hardware enters production, passes basic health checks, and is allocated to workloads. Then, under real load (not lab conditions but actual production pressure), the marginal components surface. By then, your engineers are already in the loop. > "The question is never whether hardware will fail. The question is whether you are engineering the failure to happen on your terms, in your lab, under your conditions, before it happens in production under your customers' load." At Flipkart, we learned to compress the infant mortality phase deliberately. Burn-in testing under extreme heat and load. Accelerated ageing via high-temperature and humidity cycles. Workload stress tests that reproduced peak production load before a single server touched a customer workload. The goal was simple: find the marginal hardware in the lab, not the data centre. Break it before it breaks you. ## Why We Built Our Own Servers, and What It Taught Us When Flipkart's fleet reached a certain scale, we stopped buying off-the-shelf servers. Not for the reason most people assume: cost was a byproduct, not the goal. We did it because no pre-built server is a precise fit for a specific workload, and the gap between "close enough" and "optimised" compounds across tens of thousands of nodes. We went directly into the supply chain, working with Intel, AMD, NVIDIA and Ampere to optimise CPU and GPU performance for our actual workload profiles, not the median workload these components were designed for. The servers we built from this process ran for 7+ years in production. That longevity was the result of understanding the hardware deeply enough to know what to watch, what to tune, and what to replace before it caused an incident. But here's what that experience really taught me: **The signals that predict hardware failure are not the signals that existing monitoring tools collect.** PCIe replay counters. Link width negotiation state. SMART attributes beyond the vendor thresholds that get reported as "healthy." ECC error correction rates trending upward over weeks. BIOS parameter drift from a golden baseline. These signals exist. They are present in the infrastructure right now. Most tools don't collect them. The ones that do don't correlate them. And no human team can reason across all of them simultaneously at fleet scale. ## What the Hardware Layer Actually Produces Let me make this concrete with the failure patterns we see most often. --- > ### PCIe link signal instability > > GPU utilisation looks normal. Throughput is inconsistent. Engineers chase application code, driver versions, thermal issues, for weeks. > > **Actual cause:** PCIe replay counters climbing steadily, link width downgraded from x16 to x8, riser quality variation across procurement batches. > > The errors behind the replays are logged as correctable (AER), and an x8 link is not an error at all, so no alert fires. Application retries mask the degradation. --- > ### BIOS configuration drift > > Five servers return from maintenance. A BIOS update silently changes cores per CCD from All to 4. > > **Impact:** On eight-core CCDs, half the cores on all five nodes are switched off. The OS sees only the cores that are left, the kubelet reports the smaller capacity, and the scheduler quietly places fewer pods. With the kubelet's default CPU manager policy, nothing fails, so nothing alerts, and utilisation looks normal. > > Detection time: 3 to 4 days. Multiple engineers debugging independently. No one realising they're all chasing the same root cause. --- > ### Disk degradation before SMART thresholds > > P99 latency slowly increases. SMART data still reports all disks as healthy. In Google's study of more than 100,000 drives, over 36% of failed drives had zero counts on every SMART parameter studied except temperature ([Pinheiro, Weber and Barroso, FAST '07](https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf)). > > **Actual signal:** Read retry counts and error correction behaviour trending upward over weeks, visible only when compared against a historical baseline. --- > ### Kernel module misconfiguration > > Intermittent network failures under load. Retries usually succeed. CPU, memory, disk all normal. > > **Root cause:** A kernel module loaded with a non-default configuration after an OS update. > > Hundreds of kernel parameters to compare. Engineers check only what they already suspect. ## The Maintenance Trap: Where Accumulated Debt Gets Paid Maintenance is not a cost centre. It is where accumulated hardware and configuration debt gets paid. At Flipkart, we built a quarterly maintenance system over several years that tried to solve this structurally. The principles we arrived at: | Maintenance principle | Operational intent | | :--- | :--- | | **Pre-cycle risk assessment** | Identify priorities from monitoring and health systems before maintenance begins | | **Independent maintenance domains** | No single maintenance event should affect the whole fleet | | **Centralised approvals and execution tracking** | One operational system for approvals, deferrals, execution history, and coordination | | **Real-condition validation** | Maintenance workflows tested against actual fleet conditions, not assumed compatibility | | **Rollback-ready post-maintenance verification** | The cycle is complete only after the fleet is confirmed stable, with a rollback ready wherever the firmware allows one | The insight that changed everything: **Build a single source of truth.** When maintenance systems become systems of engagement, not just systems of record, tribal knowledge stops living only in engineers' heads. > "Maintenance is where the loop either tightens or loosens. Get the system right and accumulated debt gets paid on a predictable schedule." ## Why Asama Starts at the Hardware Layer Everything I've described in this post (the bathtub curve, the signals that existing tools miss, the maintenance debt that compounds cycle over cycle) is why we are building Asama the way we are. We ingest at the hardware and OS layers as the foundation, not as an afterthought. BMC telemetry. PCIe error counters. SMART attributes. CPU topology and frequency deltas. Kernel module states. BIOS parameter baselines. We call it the 'Infrastructure State' - it is a live graph of your topology (BIOS, BMC, Kernel - tunings, settings, & attributes; horizontal and downstream dependencies; baselines, drifts and telemetry). [more on this later] We compare every node against its own historical baseline and against peer cohorts running similar workloads. And then, critically, we don't plan to stop at detection. In our design, when Asama identifies PCIe link instability across a subset of nodes, it generates a scoped remediation plan: which nodes, which actions, in what sequence, respecting your maintenance policies and SLA constraints. The engineer reviews and approves. Asama executes, tracks, and validates. The hardware layer is where the loop starts. It's also where we chose to break it. **Update, October 2026.** Asama's visibility at the hardware and OS layers, down to BIOS settings, firmware levels and PCIe links, is now generally available, and detection built on it has been validated. Investigation and remediation are expanding. To see what it finds on your own hardware, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.