# What happens when the loop stops: why remediation is the mission (3/3)

> Finding faults faster does not end the firefighting loop. One BIOS incident, told two ways, shows why the fix itself has to be planned, approved and verified.

- URL: https://blog.asama.ai/blog/article-03-asama-what-happens-when-the-loop-ends/
- Author: Sumeet Ghosh
- Published: 2025-08-08
- Tags: Fleet problems

Detection alone does not end the firefighting loop. It ends when the fix is part of the same workflow: every affected node found, a plan built for its exact hardware and firmware, an engineer's approval, then execution and verification across the fleet.

## The Resolution

In the first two parts of this series, [we described the loop](https://blog.asama.ai/blog/article-01-asama-your-team-is-trapped-in-a-loop): the detect, firefight, manual-fix, repeat cycle that compounds over time and consumes the best engineering talent infrastructure teams have. And [we went into the hardware layer](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) where that loop most reliably starts: the signals below the threshold of existing tools, the firmware drift, the PCIe degradation, the BIOS changes that halve compute capacity while dashboards stay green.

## The Same Incident, Two Ways

Ten servers go through maintenance. BIOS update, CPU microcode update, BMC firmware upgrade. Basic health checks pass. They re-enter production.

In part 1, we described what happens next without the right system in place: days of fragmented debugging, two customer escalations, a third customer degraded silently, multiple engineers chasing the same root cause independently, 3 to 4 days to a diagnosis that then requires its own coordination overhead to fix. Total cost: detection time, engineer time, customer trust, and the maintenance window to push a BIOS correction across all ten nodes under SLA constraints.

Here is what the same incident looks like with Asama.

| Phase | Without Asama | With Asama (as designed) |
|---|---|---|
| **Detection** | Customers report degradation. Each complaint investigated independently. CPU utilisation normal, no BMC faults. Multiple engineers debugging in parallel, unaware they share a root cause. | The moment the serviced servers re-enter the fleet, Asama detects configuration drift: all ten now run a BIOS profile that differs from the fleet baseline, and, once they take load, their CPU frequency and sleep-state counters diverge from peers on the same workload. Recent maintenance event correlated as the trigger. |
| **Diagnosis** | Days pass. Senior engineer manually compares node configs. Finds the BIOS update reset the server profile from Performance to Performance per Watt across all ten servers. | Root cause surfaced automatically: server profile reset from Performance to Performance per Watt across the entire batch. Impacted server list, affected customers, and expected vs actual BIOS settings provided. |
| **Remediation** | Identifying all affected servers, coordinating BIOS changes across multiple clusters, sequencing reboots under SLA constraints: all manual, all serial. | Asama generates a targeted remediation plan: which nodes, which BIOS changes, reboot sequencing aligned with SLA policies. Engineer reviews and approves. Asama executes, tracks, validates. |
| **Outcome** | **3 to 4 days to diagnosis. Three customers impacted.** | **Flagged when the servers return. Fixed before customers notice.** |

The left column is the week part 1 described. The right column is the same week as we are designing Asama to run it. The difference is not that Asama is smarter than your engineers. It isn't. The difference is that Asama is built to watch every signal it collects, on every node, continuously: comparing each against its historical baseline and its peer cohort, and correlating changes in the fleet with recent maintenance events. No human team does that continuously at fleet scale. Asama is designed to do it as its baseline operating state.

> "Detection is the prerequisite. Remediation is the mission. The loop doesn't slow when you find the problem faster. It stops when you fix it faster: safely, with the engineer in control."

## Why Remediation Has to Be the Mission

Most infrastructure tools are built to help you see the problem. Better dashboards. Better alerting. Maybe a little bit of correlation. They assume that once you can see it clearly, your team will handle the rest.

The rest is the hard part. And it's where the loop lives.

Consider what remediation actually requires once a root cause is confirmed: understanding the full blast radius across the fleet, identifying every affected node, knowing the correct fix for this specific hardware model and firmware version, sequencing the execution to respect maintenance policies and SLA constraints, validating the outcome, and maintaining an audit trail. For a ten-node BIOS issue, that's a half-day coordination exercise. For a fleet-wide kernel module misconfiguration across hundreds of nodes, it's a week.

The reason existing automation tools don't solve this is structural. Ansible playbooks execute pre-defined scripts. The moment the environment changes (new firmware version, new SKU, new kernel), the script breaks or produces undefined behaviour on nodes where state has drifted. Over time, the automation estate grows, ages, and requires maintenance of its own.

We are designing Asama's remediation to work differently. Its agentic workflows don't execute pre-defined scripts: they generate a remediation plan that is custom-built for the infrastructure state of the affected nodes - the specific incident, the affected stack, and the specific fleet context at the moment of execution. The plan accounts for your organisational policies, your maintenance constraints, your SLA commitments. And it is built to improve with every incident, learning from what worked, what needed adjustment, what the fleet's specific behaviour requires. Your Ansible playbooks degrade as your fleet evolves. Asama is built to get better.

The engineer reviews the plan and approves it. Execution is tracked end-to-end. Failures trigger replanning with human-in-the-loop approval. Post-fix validation is automatic. The audit trail is complete.

## What the Maintenance Queue Really Costs

At any point in the year, 7 to 15% of a fleet sits in the maintenance queue, and 40 to 60% of senior engineering bandwidth goes to debugging and issue resolution. Both ranges are from Asama fleet observations.

A server in the queue is deployed capital with a constant cost (power, cooling, depreciation) contributing nothing. At Flipkart, roughly 40% of the machines in the queue had problems a digital fix could resolve: firmware, configuration or kernel parameters, with no one needed on the data centre floor. That share is where remediation pays. Fix it rapidly and predictably, and the queue shrinks, the capital becomes productive again, and future capacity planning can be indexed on actual demand rather than on the waste numbers of the past.

## Why This Can Only Be Built From the Inside

There is a concept in Indian epistemology called *pramana*: a valid means of knowledge. Most schools accept more than one, nearly always perception (*pratyaksha*) and inference (*anumana*), often testimony (*shabda*) as well.

For building infrastructure tools, we accept only one: *pratyaksha*, direct operational experience. Not market research. Not user interviews. Not technical specifications. You can only build the right tool from deep inside the problem: from the specific memory of the incident, the specific failure mode, the specific coordination overhead that no documentation captures.

Asama's technical founders carry those memories from a decade at Flipkart: the BIOS resets, the PCIe links degrading, the Ansible playbook that worked perfectly until the firmware version changed, the senior engineer pulled from three other incidents to diagnose a SMART attribute drift that should have been caught weeks earlier.

Those memories are the specification for Asama. But they are also the reason this problem is solvable. The loop that infrastructure teams live in is not a law of nature. It is a consequence of tools that were built to observe rather than to resolve: tools that stop at the boundary between detection and action and hand the hardest part back to the humans.

## What Changes When the Loop Stops

When detection and remediation collapse into a single continuous workflow, something fundamental shifts for infrastructure teams. The senior engineers who spent their peak cognitive hours on BIOS comparisons and ad-hoc scripting get those hours back. The maintenance queue that consumed 7 to 15% of deployed capital shrinks. The runbooks that aged out with every firmware bump are replaced by remediation intelligence that improves with every incident.

Infrastructure stops being a reactive discipline and becomes a predictable one. Not because failures stop happening (they won't) but because the distance between a failure and its resolution shrinks to the point where the loop, as most teams know it, simply ceases to exist.

That is what Asama is built to do. And it is the only thing we are building.

**Update, October 2026.** Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. Remediation plans are made in Asama's SaaS, which never touches your infrastructure, and run on your premises by an executor that accepts only signed, in-scope manifests. BMC credentials stay in your own Vault. On day one, your engineers approve every plan. To see it on your own hardware, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.