Jain Johny
Co-founder & CTO, Asama AI
Co-founder and CTO of Asama, the AI infrastructure engineer for teams that run their own servers. He was Lead Architect for cloud orchestration and platform services at Flipkart.
14 posts
-
The hardware intelligence gap: BMC, SMART and PCIe data that nothing joins (Solution Landscape 4/4)
BMCs, SMART and PCIe counters already report on every server. Why the readings stay in separate tools and vendor dialects, and what joining them takes.
-
Maintenance as compounding capital: paying down hardware and configuration debt (Solution Landscape 3/4)
Why maintenance is where firmware and configuration debt gets paid, a playbook for cycles that leave the fleet more current, and what gets better with age.
-
GPU fleets are the new mainframes: why one failed node stalls a training job (Solution Landscape 2/4)
In synchronous training, one failed GPU node stops the whole job. What Meta's Llama 3 run shows about failure rates, and how to run GPU nodes as non-fungible.
-
Peer comparison is the baseline: what cohorts catch, and the faults they miss (Solution Landscape 1/4)
Why comparing each node with its peers catches degradation that thresholds miss, the cohort-wide faults it cannot see, and the four comparisons that cover them.
-
The firmware layer nobody owns: infrastructure state your monitoring can't see (Problem Specifics 5/5)
Infrastructure state is what a server is, not what it does. Why a BIOS setting or firmware change surfaces only as a symptom, days later, and how to read it.
-
The heterogeneous fleet problem nobody admits: what Redfish doesn't normalise (Problem Specifics 4/5)
Why Redfish gives Dell, HPE and other BMCs a common API but not common BIOS settings or event messages, and what a normalisation layer has to do across vendors.
-
Bring-up is where schedules go to die: what new servers need before production (Problem Specifics 3/5)
Bring-up is the work between racked and first workload accepted. What a slip costs while purchased capacity waits, and a five-step gate for every new server.
-
The correctable error trap: when PCIe correctable errors start to matter (Problem Specifics 2/5)
Why a climbing PCIe correctable error rate can signal a failing link, how to read AER counters in sysfs, and why peer comparison beats a fixed replay threshold.
-
What SMART data isn't telling you: which attributes predict drive failure (Problem Specifics 1/5)
Why a drive's SMART status can say PASSED right up to failure, which attributes predict it (5, 187, 188, 197 and 198), and how to trend them against peers.
-
GPU health checks on AMD: amd-smi telemetry and RAS, plus RDC and RVS diagnostics (Part 2)
How amd-smi reads health, telemetry and RAS errors from AMD Instinct GPUs, which commands to run, and where RDC and RVS take DCGM's role for diagnostics.
-
Asama AI's SOC 2 journey: building trust into the architecture
How Asama prepared for SOC 2 in five weeks, how the Trust Services Criteria shaped its design, and what a CISO checks before an AI infrastructure engineer acts.
-
Clustering for infrastructure observability: finding outlier servers with HDBSCAN
How HDBSCAN finds servers that behave unlike their peers, how to judge the clusters, and how to compare each outlier's infrastructure state with its peers'.
-
GPU health checks with NVIDIA DCGM: health watches and dcgmi diag run levels (Part 1)
How NVIDIA DCGM checks GPU health: health watches for PCIe, memory, thermal, power and NVLink faults, and what each dcgmi diag level tests and how long it runs.
-
node_exporter pcidevice collector: NUMA, SR-IOV and PCIe power state metrics
PCIe link width, NUMA node, SR-IOV virtual functions, power state and vendor names as Prometheus metrics: what each one reports and how to turn them on.