# The Automation Boundary: when remediation should run on its own, and when it should ask > When should automated remediation run without a person? Suggest, orchestrate and autonomous modes, and three tests: reversibility, determinism, blast radius. - URL: https://blog.asama.ai/blog/the-automation-boundary/ - Author: Sumeet Ghosh - Published: 2026-05-28 - Tags: Frameworks The automation boundary is the line between the fixes a remediation system may run on its own and the ones it must hand to a person. Three tests set it: reversibility, whether the action can be undone; determinism, whether its cause was measured or inferred; and blast radius, how many nodes one wrong call would touch. Most of the recoverable hours sit one step short of autonomy, in orchestration, where an engineer approves and the system does the rest. An engineer asks why an API is running slowly on one host. Ninety seconds later the answer comes back. The software RAID array on that node is rebuilding. A filesystem discard job is running at the same time. Neither is a problem alone. Together they saturated the array, application I/O queued behind storage maintenance, and the API slowed down. Confidence: high. CPU, memory and load pressure were checked and ruled out. Three actions follow, with the exact commands: ``` sysctl -w dev.raid.speed_limit_max=10000 pkill -x fstrim systemctl stop fstrim.timer ``` Detection is the easy part of that story. Correlation and root cause are harder, and they are where the engineering hours go. But there is a question after all of them, and it is about authority rather than capability. The cause is established, the commands are known, the node is named. Should the system run them? > "Detection tells you what happened. Root cause tells you why. Authority decides what happens next. Most products are precise about the first and vague about the third." Look at those three commands again, because they are not equally safe. The first is a runtime sysctl. Set it, observe, set it back. It lowers the ceiling on rebuild speed to 10,000 KiB/s per device, a twentieth of the kernel's default of 200,000, which extends the window in which the array runs degraded: a trade-off rather than a free win. The second kills a discard job that is safe to rerun, and its undo is the timer's next weekly run. The third stops that timer. Stopped is not disabled: the timer stays off until someone starts it again or the server reboots, which can be months away, so the second command's undo now depends on the third's. The risk is not that the third fails. The risk is that it succeeds, nobody remembers, and discard debt builds up on those SSDs in the meantime. Three commands, one diagnosis, three risk profiles. Any system with authority to run them needs a way to tell those apart. --- ## Three modes: suggest, orchestrate, autonomous Remediation operates in three modes. They are not a maturity ladder where each supersedes the last. | Mode | What the system does | Who decides | Who executes | | --- | --- | --- | --- | | **Suggest** | Establishes root cause and proposes exact actions with evidence | Human | Human | | **Orchestrate** | Generates the plan against this node's state, stages it for approval, carries it out and verifies | Human | System | | **Autonomous** | Determines the fix and executes it. Humans are notified afterwards | System | System | The distinction that matters most is between the first two. Suggest still leaves real work with your engineers. Even with the commands written out, someone has to check them against the maintenance window, decide the order, run them on the right hosts, and confirm the API recovered. Skilled time, spent after the interesting thinking is done. Orchestrate removes that burden without removing the decision. The industry spends its attention on the jump to autonomous. The jump from suggest to orchestrate is where the recovered hours actually live. --- ## Where Asama draws the line Asama is designed to work in suggest and orchestrate, and not to execute remediation without human approval. That is a design position, not a gap in the roadmap. Onboarding starts read only. Execution privileges are designed to be granted per remediation class, under credentials you control and can revoke. Each plan is designed to be generated against the node's exact state, not replayed from [a runbook gone stale](https://blog.asama.ai/blog/issue-04-asama-your-automation-has-an-expiry-date) on a box that has since changed, then staged for an engineer to read before anything runs, with a rollback path computed in advance. Once a plan is approved, execution is designed to be the system's job: sequencing, maintenance windows, canary rollout, verification and the change record, with no further human effort. If a step fails, Asama is designed to replan and hand back to the engineer. The human owns the decision. The system owns the work. Three conditions decide whether any system, ours included, should be trusted to go further than that. --- ## Reversibility: can the node go back to where it was? If this action is wrong, can the node be returned to where it was? Not whether a rollback exists in principle, but whether the prior state was captured and the path back is a procedure rather than an incident. | Action | Reversibility | | --- | --- | | `sysctl -w dev.raid.speed_limit_max=10000` | High. Prior value captured, runtime only, restore in one command | | `pkill -x fstrim` | High while the timer runs: the job is safe to rerun and returns on its next weekly run, though `pkill` leaves fstrim.service marked failed until then. With the timer stopped, it returns only when the timer does | | `systemctl stop fstrim.timer` | Medium. `systemctl start fstrim.timer` reverses it, best once the rebuild ends, because a weekly run missed in the meantime fires soon after the restart. Easy to forget: the timer stays stopped until the next reboot. Consequence accrues silently | | `ethtool -G rx 4096` | Medium. Reinitialises the ring and briefly interrupts traffic. Reverts on reboot unless persisted | | Firmware rollback | Low. Requires a reboot, the prior image must still be available, and [some releases cannot be downgraded at all](https://blog.asama.ai/blog/issue-13-asama-maintenance-as-compunding-capital) | | Hardware replacement trigger | Very low. Starts a chain that does not unwind | The two middle rows are the interesting ones, because both look safe and neither quite is. One is reversible but forgettable. The other takes the interface down for a moment while it applies, which on a live NIC is a decision rather than a detail. Reversibility is the first filter because the cost of being wrong is asymmetric. A wrong outcome with a clean undo path costs a change window. Without one it costs considerably more, and no confidence score compensates for that. --- ## Determinism: was the cause measured or inferred? We wrote earlier about [three zones of degradation](https://blog.asama.ai/blog/determinism-probability-and-the-limits-of-observability): deterministic, probabilistic and irreducible. The framework applies to remediation as directly as to detection, and it is why every diagnosis should carry its confidence in the open. The storage case returned at high confidence. Both maintenance jobs were observed running concurrently, the queueing on the array was measured, and the alternatives were excluded. The mechanism was not inferred. Compare a second case from the same fleet. Receive-buffer drops on a NIC peaked in the thousands, kernel receive-processing pressure peaked in the same minute, and a large inbound burst occurred during the hour. The recommendation was to raise the ring depth from 1,024 towards 4,096. Confidence: medium, and stated as medium, because the peak timestamp of that burst was not retained in the available history and the alignment could not be confirmed. A system that reports both cases with equal certainty has taken away your ability to calibrate. A diagnosis that hides what it could not establish is worse than one that admits it. Irreducible cases are not automated at any level of sophistication. Executing an action whose rationale is not understood is not operational intelligence. It is operational risk with better tooling. --- ## Blast radius: what one wrong call does to 500 nodes Reversibility and determinism describe the risk of being wrong once. Blast radius describes the risk of being wrong at scale. An action at 99% confidence on one node carries a 1% chance of creating one problem. Across 500 nodes, what that 1% stands for decides the outcome. If each node's chance of harm is independent of the others, expect about five damaged nodes, with a 99.3% chance of at least one. If the 1% is the chance that the diagnosis itself is wrong, every node shares it: one bad call damages all 500 at once, with correlated symptoms, on a fleet whose baseline just moved. Both cases average five damaged nodes. The second is far worse, because correlated failures are much harder to diagnose: the [peer comparison](https://blog.asama.ai/blog/issue-11-asama-peer-comparison-is-the-baseline) that would normally surface the anomaly is now comparing damaged nodes against other damaged nodes. | Situation | Mode | | --- | --- | | Deterministic, reversible, single node | Orchestrate | | Deterministic, reversible, cohort or fleet | Orchestrate, canaried, validated before the rest follows | | Deterministic, irreversible, any scope | Orchestrate, explicit approval, no exceptions | | Probabilistic or irreducible, any scope | Suggest, with the confidence score shown | Read the table for what is missing from it. Nothing in our operating model produces autonomous as an output today. These are the gates we use to decide what is safe to orchestrate, and they are the same gates any future expansion of authority would have to clear first. --- ## Rollback first, authority later Rollback is usually reached for when an action fails. In remediation it belongs earlier. Before a plan is approved: the current state captured rather than assumed, the path back defined rather than improvised, and the criteria that trigger it set in advance. Authority accumulates the same way. It is earned per class of action and it can be withdrawn. This is why pilots run read only at the start, not as a sales concession but because the first weeks are when the system knows least about the fleet and the fleet knows least about the system. --- ## From Asama > ### Designed to ask before it acts > > Asama is designed to work this way: > > - Read only at onboarding, then execution privileges per remediation class, under credentials you control and can revoke > - Every diagnosis carries its confidence and names what it could not establish > - Plans generated against the node's exact hardware, firmware and OS combination, staged for approval, with a rollback path computed in advance > - Execution sequenced against maintenance windows, SLA policies and freeze periods, and verification against the pre-incident baseline, not whether an alert cleared > - Every plan, approval, execution and verification recorded as evidence --- There is a class of deterministic, reversible, single-node action where autonomous execution will eventually be right. The gates above are how we would decide when that line moves, and the reason has to be a demonstrated record on a specific class of action on a specific fleet. Not a general improvement in model capability, and not a competitive response to someone else's marketing. The gap between suggest and orchestrate is where most of an infrastructure team's time goes today. That gap can be closed without asking anyone to give up the decision. --- ## Food for thought > Where does your automation draw the line between executing and asking? Is that line written down, or has it simply never been tested? **Update, October 2026.** Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. Each remediation plan is composed for the node's exact hardware, firmware, OS and configuration, with preconditions and a rollback worked out before anything runs; execution then goes dry run, canary, then waves, and on day one your engineers approve every plan. The [security architecture](https://blog.asama.ai/blog/asama-ais-compliance-journey-building-trust-through-architecture) separates planning from execution: the SaaS plans and never touches your infrastructure, an executor on your premises accepts only signed, in-scope manifests, and BMC credentials stay in your own Vault. To see where the boundary would sit on your fleet, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.