Blog
35 posts
Why Asama Is Different: Infrastructure State and the Detect, Investigate, Remediate Loop
What it takes to run the detect, investigate, remediate loop on a large fleet, why stale and stitched data breaks it, and why Asama is built on infrastructure state.
Asama vs LogicMonitor + Edwin AI: A component-by-component comparison
LogicMonitor + Edwin AI compared with Asama AI & scored on Infrastructure State, Detection and Remediation Capability.
Why L3 Stalls
L3 is where every infrastructure platform has landed, and none has left. The gap to L4 is not reasoning. It is whether anything underneath knows how your fleet is wired together.
Asama vs Zabbix + Claude: A component-by-component comparison
A combination of Zabbix + Claude compared with Asama AI & scored on Infrastructure State, Detection and Remediation Capability.
Asama vs Zabbix : A component-by-component comparison
Zabbix and Asama scored on Infrastructure State, Detection and Remediation Capability.
Normal Is a Moving Target
Every detection claim in infrastructure rests on a definition of normal that almost nobody writes down.
The Four Levels of Detection Maturity
At every level the work is the same. What changes is how much of it the platform does, and how much is left to your engineers.
Trust Is the Real Interface: what makes an AI root cause analysis checkable
An AI root cause analysis earns trust when you can check it: findings tied to evidence, confidence stated with its gaps, and overrides that change the answer.
The Automation Boundary: when remediation should run on its own, and when it should ask
When should automated remediation run without a person? Suggest, orchestrate and autonomous modes, and three tests: reversibility, determinism, blast radius.
Determinism, Probability, and the Limits of Observability: what the residual says about a fault
Every fault model is some version of y = mx + c + ε. What the residual's shape says about a fault, where chaos caps prediction, and why ε never reaches zero.
The hardware intelligence gap: BMC, SMART and PCIe data that nothing joins (Solution Landscape 4/4)
BMCs, SMART and PCIe counters already report on every server. Why the readings stay in separate tools and vendor dialects, and what joining them takes.
Maintenance as compounding capital: paying down hardware and configuration debt (Solution Landscape 3/4)
Why maintenance is where firmware and configuration debt gets paid, a playbook for cycles that leave the fleet more current, and what gets better with age.
GPU fleets are the new mainframes: why one failed node stalls a training job (Solution Landscape 2/4)
In synchronous training, one failed GPU node stops the whole job. What Meta's Llama 3 run shows about failure rates, and how to run GPU nodes as non-fungible.
Peer comparison is the baseline: what cohorts catch, and the faults they miss (Solution Landscape 1/4)
Why comparing each node with its peers catches degradation that thresholds miss, the cohort-wide faults it cannot see, and the four comparisons that cover them.
The firmware layer nobody owns: infrastructure state your monitoring can't see (Problem Specifics 5/5)
Infrastructure state is what a server is, not what it does. Why a BIOS setting or firmware change surfaces only as a symptom, days later, and how to read it.
The heterogeneous fleet problem nobody admits: what Redfish doesn't normalise (Problem Specifics 4/5)
Why Redfish gives Dell, HPE and other BMCs a common API but not common BIOS settings or event messages, and what a normalisation layer has to do across vendors.
Bring-up is where schedules go to die: what new servers need before production (Problem Specifics 3/5)
Bring-up is the work between racked and first workload accepted. What a slip costs while purchased capacity waits, and a five-step gate for every new server.
The correctable error trap: when PCIe correctable errors start to matter (Problem Specifics 2/5)
Why a climbing PCIe correctable error rate can signal a failing link, how to read AER counters in sysfs, and why peer comparison beats a fixed replay threshold.
What SMART data isn't telling you: which attributes predict drive failure (Problem Specifics 1/5)
Why a drive's SMART status can say PASSED right up to failure, which attributes predict it (5, 187, 188, 197 and 198), and how to trend them against peers.
The maintenance queue is a CapEx problem: what idle servers cost while they wait (Problem Scope 5/5)
Servers in the maintenance queue keep depreciating while they wait. Worked arithmetic for a 1,000-server fleet, and the share of the queue a digital fix clears.
Your automation has an expiry date: how playbooks go stale as the fleet changes (Problem Scope 4/5)
Why automation written for last year's hardware skips or misapplies fixes on today's fleet, and how to build each remediation plan from the node's actual state.
GPU health checks on AMD: amd-smi telemetry and RAS, plus RDC and RVS diagnostics (Part 2)
How amd-smi reads health, telemetry and RAS errors from AMD Instinct GPUs, which commands to run, and where RDC and RVS take DCGM's role for diagnostics.
Asama AI's SOC 2 journey: building trust into the architecture
How Asama AI prepared for a SOC 2 Type I audit in five weeks, how the Trust Services Criteria shaped its design, and what a CISO checks before an AI agent acts.
The tribal knowledge tax: what walks out when your senior engineers leave (Problem Scope 3/5)
Why a pinned BIOS setting or held-back firmware version often lives in one engineer's head, and what that costs in investigation time and deferred maintenance.
Clustering for infrastructure observability: finding outlier servers with HDBSCAN
How HDBSCAN finds servers that behave unlike their peers, how to judge the clusters, and how to compare each outlier's infrastructure state with its peers'.
Configuration drift is the silent killer: the settings nobody tracks (Problem Scope 2/5)
Configuration drift hides in the BIOS, firmware and kernel settings no one told config management to check. Why no alert fires, and how to see it on every node.
GPU health checks with NVIDIA DCGM: health watches and dcgmi diag run levels (Part 1)
How NVIDIA DCGM checks GPU health: health watches for PCIe, memory, thermal, power and NVLink faults, and what each dcgmi diag level tests and how long it runs.
Uptime is not reliability: the gray failures green dashboards miss (Problem Scope 1/5)
A server can pass every health check and still run below spec. Why uptime misses gray failure, and the three properties that make infrastructure reliable.
node_exporter pcidevice collector: NUMA, SR-IOV and PCIe power state metrics
PCIe link width, NUMA node, SR-IOV virtual functions, power state and vendor names as Prometheus metrics: what each one reports and how to turn them on.
What happens when the loop stops: why remediation is the mission (3/3)
Finding faults faster does not end the firefighting loop. One BIOS incident, told two ways, shows why the fix itself has to be planned, approved and verified.
Where the loop always starts: the hardware signals your monitoring misses (2/3)
PCIe replays, BIOS drift and disks that SMART calls healthy: the hardware signals behind silent degradation, and what 35,000 servers at Flipkart taught us.
Your team is trapped in a loop: why infra teams keep firefighting (1/3)
Hardware telemetry split across tools, runbooks that age with every firmware change, servers stuck in maintenance: the loop that keeps infra teams firefighting.
Unpopular opinion: build the foundation first, not what customers ask for
Most fleets run servers from two or more vendors, and each vendor's tools stop at its own hardware. Why we started with one model of every server, before any feature.
How we named Asama: Sanskrit for "without an equal"
Asama (असम) is Sanskrit for "that which does not have an equal". Why we chose it over Orca and Raven, and what it says about the product we build.
Hello World
The first post on the Asama blog.
No posts match your search. Try different keywords.