# How Asama Investigates: parallel hypothesis trajectories for AI root cause analysis > How Asama, an AI infrastructure engineer, investigates: one agent per hypothesis, all running in parallel, verified into an evidenced root cause. - URL: https://blog.asama.ai/blog/how-asama-investigates/ - Author: Manish Gautam - Published: 2026-10-28 - Tags: Product updates ## Executive summary An application team tells you their VM is slow. No alert has fired. Today you work that ticket in series: form a hypothesis for the most likely cause, gather evidence, accept or reject it, then move to the next. When the cause sits in another team's system, you pull that team in and wait. Asama's agents work the same ticket in parallel. Asama builds every plausible hypothesis trajectory across the systems behind the VM and gives each one its own agent. The agents run at the same time, each following its own path through a live topology of your fleet and writing its evidence onto one shared graph. Asama then correlates and verifies what they find into one causal chain, and every hypothesis, accepted or rejected, traces back to its evidence. | | Today | With Asama | |---|---|---| | Hypotheses | one at a time, most likely first | all at once, one agent each | | Correlation | you, across tools, in your head | Asama, on one graph | | A cause in another team's system | a handoff and a wait | already in the topology | | The ticket in this post | nine steps, about three hours hands-on | about three minutes | On that ticket the cause sat on a storage server two hops from the VM, where a RAID rebuild and a trim job were colliding. ## Watch it run The video with this post shows the investigation below, recorded with real product footage, trimmed and sped up where marked. ## How it is done today The ticket names a VM and a symptom. It does not name a hypervisor, a storage server, an array or a drive. No alert is attached, because nothing crossed a threshold: CPU is fine, memory is fine, the host is not saturated. Behind it, three routine things are happening on systems that belong to different people. The VM's second virtual disk is a file on an NFS mount on its hypervisor. The export behind that mount sits on a four-member NVMe RAID1 array on a storage server. A member of that array was swapped, so the array is degraded and rebuilding at a throttled rate, and a trim job is running on the same array. Whoever swapped the drive had no reason to know about the trim job, and the application team had no reason to know about either. None of it is visible from the VM. So you work the ticket the way every infra team does, one hypothesis at a time. Here is the path, with the hands-on time for an engineer who knows where to look and never waits on another team: | Step | What you do | Hands-on time | |---|---|---| | 01 | Confirm the VM is actually slow | ~10 min | | 02 | Check guest CPU, memory and disk | ~15 min | | 03 | Find the hypervisor the VM runs on | ~10 min | | 04 | Follow the virtual disk to the NFS mount | ~10 min | | 05 | Find the NFS server behind the mount | ~15 min | | 06 | Check the RAID array and the rebuild | ~15 min | | 07 | Check the NVMe members | ~15 min | | 08 | Find the background I/O | ~45 min | | 09 | Line up the timing, rule out the rest | ~45 min | | | **Total, hands-on** | **~3 h** | Steps 01 to 07 are navigation: working out what is connected to what, in a different tool at every hop. From step 05 the work sits with the storage team, so in most organisations the ticket changes hands there, and the wait never shows in the table. Steps 08 and 09 are correlation, and they take half of the three hours: finding a workload nobody mentioned, then proving its timing lines up with a symptom on a different host. That correlation is the work your tools leave to you. Dashboards make each layer queryable; they do not connect the layers, so you [carry the topology yourself](https://blog.asama.ai/blog/issue-03-asama-the-tribal-knowledge-tax). In [The Four Levels of Detection Maturity](https://blog.asama.ai/blog/four-levels-of-detection-maturity), most of the distance from L2 and L3 tooling to L4 is exactly this correlation. A tool that reasons over the alert stream does not close it either: with no alert fired, it has nothing to start from, which is [where L3 root cause analysis stalls](https://blog.asama.ai/blog/why_l3_stalls). ## How Asama investigates You ask Asama the application team's question, in their words: why is the VM slow? Five stages follow. ### 1. Start from infrastructure state An investigation is only as fast as the infrastructure state it starts with. Before you ask anything, Asama already holds: - **A live topology** of physical, virtual and architectural dependencies, kept current as hardware changes, not loaded from a static CMDB. - **Telemetry across those layers**, normalised into one model whatever the vendor or collector. - **Baselines** for how each component normally behaves, so deviation is measured [against the component's own history and its peers](https://blog.asama.ai/blog/normal-is-a-moving-target), not against a threshold someone picked years ago. A question that names only a VM is therefore enough. Asama resolves it to the path from the VM through its virtual disk, the hypervisor, the NFS mount and export, to the array and its four NVMe members, and knows its scope before it runs a single command. ### 2. Map the subject, then form hypotheses The main agent maps what the question names: the VM's virtual disks, its NIC and its vCPUs, each with its current state on the live graph. What is already flagged there sharpens the candidates. Here, the VM's NFS-backed disk is flagged severe while CPU and memory look normal, and the main agent forms three hypotheses: the storage path behind the disk is degraded, the VM is starved of CPU on its host, or something else is driving I/O against the disk's backing storage. ### 3. Investigate every hypothesis in parallel Each hypothesis gets its own agent, and the agents start together. Each one follows its own trajectory, a route across hosts and layers that the evidence either supports or closes, and writes what it finds onto the shared graph as it goes. Nothing waits for another hypothesis to be rejected, and no trajectory stops at a team boundary, because every system behind the VM is already in the topology. Within seconds of your question, three agents are running: 1. **NFS storage agent**: follows the disk to the NFS mount, the export on the storage server and the RAID array behind it. 2. **Hypervisor CPU agent**: checks where the vCPUs are pinned, how loaded the host is and how long the guest waits to be scheduled. 3. **Guest disk agent**: traces the virtual disk to its backing store and looks for the processes driving I/O against it. You can watch the shared graph grow: the VM on one server, the mount and export crossing to the second server, then the array opening out into its four NVMe members. At 88 seconds the guest disk agent reports the hidden workload, a trim service on the storage server issuing sustained discards to the array behind the VM's disk. The storage agent finds the array degraded and rebuilding and drills into the member in recovery. The CPU agent finds a narrow vCPU pin on the host, a real constraint. Three verdicts, each with its own evidence, and none of them the answer on its own. Most of this work is not agentic. About 70% of an Asama root cause analysis is [deterministic correlation and statistical inference](https://blog.asama.ai/blog/determinism-probability-and-the-limits-of-observability): topology traversal, baseline deviation, timing alignment. The other 30% is agentic reasoning, used for one job: taking those data points and statistical findings to a causal explanation. The agents do not guess at telemetry. They reason over evidence the deterministic layer has already assembled. ### 4. Correlate and verify Parallel agents produce findings, and findings are not yet a root cause. Asama correlates them four ways. By time: the rebuild, the discards and the disk pressure overlap in the same window. By topology: all three sit on one path to the VM's disk. By change: the array is degraded because a member was swapped, a recent hardware change. By cohort: the other disks and hosts in scope show no such pressure. Verification then picks one explanation. Asama checks causal order (the array contention precedes and spans the disk stalls), consistency (every layer between the array and the VM shows the expected effect) and the alternatives. Guest CPU steal was present but low, and the hypervisor reported vCPU wait as zero, so the CPU trajectory closes. The vCPU pin is recorded, not blamed. ### 5. Write the root cause with its evidence The root cause comes as a causal chain, across two hosts: > trim service + RAID rebuild on the storage server → contention on the array → NFS export delay → NFS mount on the hypervisor → I/O stalls on the VM's disk Neither the rebuild nor the trim job is wrong on its own; the interaction between them, on shared storage, is the root cause. Every claim in the chain links to the data behind it, so you can [check it before you act on it](https://blog.asama.ai/blog/trust-is-the-real-interface): the disk's busy time and write latency, discard activity on the array, the rebuild state, the trim process and how it was launched, and CPU steal next to vCPU wait, each opening its graph, process snapshot or the `/proc/mdstat` capture showing four members, three in sync, one rebuilding. Confidence is stated, and so are its limits. Asama reported high confidence in the trim actor and the storage mechanism, and said plainly what it could not establish: the trim job's exact share of the VM's I/O against the guest's own workload. The recommended next step follows from the evidence: pause the trim service until the rebuild completes, [once you approve it](https://blog.asama.ai/blog/the-automation-boundary). ## What changes for you About three minutes from question to root cause, against about three hours of hands-on work: 177 minutes back on one ticket, 98% of it, before counting the handoff to the storage team. The bigger change is your role. In the serial loop you are the correlation engine: you hold the topology, run the hypotheses one by one and stitch the evidence together across teams. With Asama you review a finished diagnosis. You open the evidence links that matter to you, and you decide what to do. That shift from investigator to approver is what defines L4, and it is the core of [Why Asama Is Different](https://blog.asama.ai/blog/why-asama-is-different). It also changes what gets investigated at all. A three-hour investigation for "the VM feels slow" often never happens: the ticket is closed as transient, and the same interaction comes back during the next rebuild. A three-minute investigation is cheap enough to run every time. Asama runs live in your environment in under two hours. To watch it investigate a ticket from your own fleet, book a live run against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.