# Your team is trapped in a loop: why infra teams keep firefighting (1/3) > Hardware telemetry split across tools, runbooks that age with every firmware change, servers stuck in maintenance: the loop that keeps infra teams firefighting. - URL: https://blog.asama.ai/blog/article-01-asama-your-team-is-trapped-in-a-loop/ - Author: Sumeet Ghosh - Published: 2025-07-12 - Tags: Fleet problems Infra teams firefight the same kinds of incident again and again for three reasons: faults below the application layer go unseen for days, every fix is done by hand, and the automation meant to help ages with every firmware change. The loop costs you idle capacity and your senior engineers' best hours. ## The Loop No One Talks About I want to tell you about a loop. Not a software loop. A human one, the kind that plays out in infrastructure teams across the world, every single day, with such regularity that most engineers have stopped noticing it. It goes like this. Something degrades. The dashboards stay green, because the dashboards are built to catch what you already know to look for, not what you don't. Engineers spend days, sometimes weeks, manually correlating signals from fragmented tools: iDRAC here, SMART data there, kernel logs somewhere else. Eventually a senior engineer, always a senior engineer, gets pulled in. Then comes remediation. Someone writes a runbook. Or finds an old one. Or tries to adapt an Ansible playbook that was built for a slightly different firmware version and now breaks on three of the fifteen affected nodes. And then it happens again. On a different node. With a different root cause. And the loop begins once more. > "Everything soon becomes an infra engineer's job: reliability, fault detection, vendor coordination, stakeholder management, and issue resolution. Already stretched teams are stretched thinner. Delivery is bound to drop." --- ## What the Loop Actually Looks Like Ten servers go through maintenance: BIOS update, CPU microcode update, BMC firmware upgrade. Basic health checks pass. They're reintroduced into production and allocated to customers. Two customers flag performance degradation within days. The third customer is also degraded, but silent. Your team investigates each complaint independently. CPU utilisation is normal. No hardware faults in BMC. Applications observe unexplained latency, but nothing is broken. Working assumption: increased workload demand. Multiple engineers now debugging in parallel, each unaware the others are chasing the same root cause on different nodes. Days pass. A senior engineer finally runs a manual comparison across affected versus healthy nodes and finds it: the BIOS update silently reset the server profile from *Performance* to *Performance per Watt* across all ten servers. Physical cores intact. CPU usage normal. But Performance per Watt lets the platform clock cores down under light load and turns on C-states and C1E, so idle cores drop into sleep states and take time to wake. [Dell's own tuning guide](https://dl.dell.com/manuals/all-products/esuprt_solutions_int/esuprt_solutions_int_solutions_resources/high-computing-solution-resources_white-papers6_en-us.pdf) calls it a poor fit for latency-sensitive work. It is often the factory default, so a BIOS update that resets settings lands right on it. Monitoring did not catch it. Kubernetes and the dashboards read utilisation from the OS, and utilisation measures how busy a core is, not how fast it gets work done. The cause showed only in the CPU's frequency and sleep-state counters, and nobody alerts on those. Detection time: 3 to 4 days. Two customer escalations. One customer degraded silently the entire time. This is not an unusual incident. This is Tuesday. > ### 7 to 15% | **Of the fleet sits in the maintenance queue at any given time.** > Deployed capital sitting idle: powered, cooled, depreciating, contributing nothing. > > ### 40 to 60% | **Of senior engineering bandwidth goes to debugging and issue resolution.** > Senior engineers trapped in firefighting instead of platform and reliability work. > > Ranges from Asama fleet observations. --- ## Why the Loop Compounds Over Time Detection gets harder because your data fragments further with every new hardware generation, [every new vendor](https://blog.asama.ai/blog/unpopular-opinion---do-not-build-what-customers-want), every new telemetry format. iDRAC, iLO, Redfish, IPMI, DCGM, AMD SMI, EDAC, SMART, BMC SEL: correlating signals across these is already a full-time job, and the fleet keeps growing. Every runbook your team writes is a snapshot of the world at the moment it was written. The moment firmware versions change, new SKUs arrive, or kernel versions bump, that runbook begins to age. Ansible playbooks that worked perfectly six months ago now fail on three nodes because the state they assumed no longer exists. Scripts built for one hardware generation break silently on the next procurement batch. New automation needs to be continuously written, tested, and maintained, consuming the same senior engineering bandwidth that detection already drains. > "Traditional automation tools execute pre-defined scripts. The moment the environment changes (new firmware, new SKU, new kernel), the script breaks. Your automation degrades over time. The loop gets slower." The result is an infrastructure team that spends an ever-increasing proportion of its best engineering talent maintaining the maintenance system itself. --- ## The Hidden Cost Nobody Puts on the Spreadsheet The deeper number is the **7 to 15%** of your server fleet that sits in the maintenance queue at any point in the year. That's deployed capital (hardware you've paid for, allocated powered & cooling for) contributing nothing. Then there's the buffer. Because the loop is slow and unreliable, infra teams over-provision, building in redundancies and capacity buffers to absorb the failures they can't prevent. That over-provisioning compounds at every planning cycle, because you're indexing on past waste plus projected demand. And finally there's the invisible cost that never appears on any spreadsheet: the best infrastructure engineers in your team spending their peak cognitive hours on BIOS comparisons and ad-hoc scripting instead of architecture and platform work. --- ## What Breaking the Loop Actually Requires Breaking it requires more than a better dashboard. What actually breaks the loop is collapsing the distance between detection and remediation: not as two separate phases requiring human handoff at every step, but as a single continuous workflow driven by intelligence that understands your fleet and the specific context of each incident. | Capability | Operational shift | | :--- | :--- | | *Detection below the application layer* | Hardware, firmware, OS, and low-level signals become first-class operational inputs | | *Topology-aware root cause reasoning* | Workload -> GPU -> PCIe -> motherboard relationships instead of isolated logs and alerts | | *Context-aware remediation plans* | Generated for the specific incident, nodes, firmware versions, and fleet state at execution time | | *Engineer-in-the-loop execution* | Engineers approve actions without spending hours on repetitive investigation and coordination | | *Adaptive automation* | Workflows improve with each incident instead of degrading as the fleet evolves | --- ## Why We Are Building Asama We are building Asama as **an AI infrastructure engineer** that works alongside your team: ingesting data from every layer of the stack, correlating signals at fleet scale, generating remediation plans, and executing them with engineers in the loop. Our focus is not observability. Our focus is MTTR. Detection is the prerequisite. Remediation is the mission. The goal is not a faster loop. It is a loop that stops repeating. This is the first in a three-part series on infrastructure reliability: the theory, the operational experience, and the frontier of what AI makes possible. **Update, October 2026.** Asama's fleet visibility is now generally available, and detection built on it has been validated; investigation and remediation are expanding. To see it on your own hardware, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai and tell us what your loop looks like.