# GPU health checks on AMD: amd-smi telemetry and RAS, plus RDC and RVS diagnostics (Part 2) > How amd-smi reads health, telemetry and RAS errors from AMD Instinct GPUs, which commands to run, and where RDC and RVS take DCGM's role for diagnostics. - URL: https://blog.asama.ai/blog/gpu-health--diagnostics-part-2-amd-gpus-with-amd-smi/ - Author: Jain Johny - Published: 2025-11-18 - Tags: Field notes AMD SMI (`amd-smi`) is AMD's counterpart to nvidia-smi: a command-line tool and C library, with Python bindings, that reads health, telemetry and RAS error data from AMD GPUs and sets power and clock policy. For the DCGM-style layer, with GPU groups, job statistics and diagnostics, AMD has the ROCm Data Center Tool (RDC), and the deeper stress tests live in the ROCm Validation Suite (RVS). [Part 1 of this series](https://blog.asama.ai/blog/gpu-health--diagnostics----why-they-matter) covered NVIDIA's DCGM toolkit: real-time GPU health monitoring, diagnostics and alerts for data-centre environments. ## What Is AMD SMI **AMD SMI (System Management Interface)** is the command-line and library toolset that gives you visibility into the operational health of AMD GPUs. It’s part of AMD's **ROCm** software stack, succeeds the older rocm-smi tool, and provides low-level access to telemetry, configuration, and RAS data. It maps to nvidia-smi and NVML rather than to DCGM: | NVIDIA | AMD | What it does | | --- | --- | --- | | nvidia-smi, NVML | `amd-smi`, the [AMD SMI library](https://rocm.docs.amd.com/projects/amdsmi/en/latest/) | Per-GPU telemetry, RAS error counts and settings | | DCGM (`dcgmi`) | [ROCm Data Center Tool](https://rocm.docs.amd.com/projects/rdc/en/latest/) (`rdci`) | GPU groups, field monitoring and job statistics | | `dcgmi diag` | `rdci diag` and the [ROCm Validation Suite](https://github.com/ROCm/ROCmValidationSuite) (`rvs`) | Active diagnostics and stress tests | | DCGM exporter | [AMD Device Metrics Exporter](https://github.com/ROCm/device-metrics-exporter) | Prometheus metrics | With AMD SMI you can: - Monitor GPU health and utilisation - Track power, temperature, and memory stats - Read RAS error counts and reset GPUs - Manage performance states (clocks, power caps) - Collect hardware telemetry for observability systems ## Basic GPU Information The first step in understanding any GPU node is gathering hardware identity and topology details. AMD SMI enables you to query model names, firmware versions, and interconnect topology, allowing you to know exactly what’s running in your system. It provides data such as: - **Product name and SKU**: confirms whether the node runs MI300X, MI250, or older Instinct series. - **Firmware and VBIOS versions**: critical for validating driver compatibility across mixed clusters. - **[PCIe configuration and NUMA locality](https://blog.asama.ai/blog/pci-device-metrics-in-prometheus-node-exporter)**: shows how each GPU is wired to the CPU or switch fabric, helping diagnose I/O bottlenecks and cross-NUMA latency. - **Topology information**: maps multi-GPU interconnects (xGMI or Infinity Fabric links) for debugging peer-to-peer bandwidth issues. Having this baseline inventory makes it easier to detect [firmware drift](https://blog.asama.ai/blog/issue-02-asama-config-drift-is-the-silent-killer), inconsistent driver stacks, or nodes provisioned with mismatched GPUs. ## Health & Telemetry Health telemetry is at the core of day-to-day GPU monitoring. AMD SMI exposes real-time operational metrics that reflect the GPU’s thermal, electrical, and workload conditions. It exposes data such as: - Temperature sensors: reports edge, junction (hotspot), and memory temperatures. Rising averages often signal airflow problems or degrading thermal interface material. - Power draw and voltage rails: track instantaneous and average consumption versus TDP limits; useful for identifying throttling or PSU saturation. - Fan speed: reported on actively cooled cards. Instinct accelerators are passively cooled, so watch the chassis fans and inlet temperature instead. - Performance Levels: shows current clock domains and frequency scaling behaviour; helpful for detecting down-clocking under load. - HBM/VRAM utilisation: monitors buffer pressure and helps correlate performance dips with paging or over-subscription. Collecting these metrics periodically builds the baseline for **drift detection, capacity planning, and thermal optimisation** across large GPU clusters. ## RAS Errors and Events AMD GPUs incorporate **RAS (Reliability, Availability, and Serviceability)** features to detect and report hardware errors before they impact workloads. AMD SMI provides direct access to this layer. These outputs include: - **ECC error counts:** both correctable and uncorrectable events for VRAM or cache. Spikes often indicate memory degradation or cooling issues. - **RAS feature status**: confirms whether ECC, page retirement, and poison handling are active. - **Retired and pending pages**: memory pages taken out of service after errors, and those waiting to be. - **Events**: GPU resets, thermal throttling and VM page faults, plus process, queue and memory-migration events, reported as they happen. - **Error blocks**: per-block counts cover memory (UMC), PCIe, the xGMI fabric, GFX and more, which helps automate root-cause analysis. Regularly parsing this data helps you detect failing GPUs early, isolate unstable hosts, and correlate hardware faults with workload or environmental changes. It’s the backbone for proactive maintenance and warranty tracking. ## A First Health Check with amd-smi ```bash amd-smi static # model, VBIOS, PCIe link, NUMA node and limits amd-smi firmware # firmware versions, for drift checks amd-smi topology # xGMI and PCIe links between GPUs amd-smi metric --temperature --power --clock # temperatures, power draw and clocks amd-smi metric --ecc --ecc-blocks # ECC counts, in total and per block amd-smi bad-pages # pending and retired memory pages amd-smi event # resets, throttling and VM faults as they happen amd-smi monitor -w 5 # one line per GPU, every 5 seconds ``` Flags vary between ROCm releases; `amd-smi --help` shows what your version supports. The `set` and `reset` commands need root, and act on every GPU unless you pass `-g`. ## Running Diagnostics: RDC and RVS AMD SMI reads state; it does not stress the GPU. The active tests that match `dcgmi diag` live in two other tools: - **RDC**: `rdci diag -g ` runs a quick health check on a GPU group: no compute processes running, node topology, GPU parameters, a compute queue test and a system memory test. - **RVS**: `rvs -c ` runs deeper modules, including GST for compute stress, IET for sustained power, MEM for memory test patterns, BABEL for memory bandwidth, and PEBB and PBQT for PCIe and peer-to-peer bandwidth. AMD ships per-GPU configurations, such as for MI300X. As with DCGM's longer levels, run the RVS stress modules on a drained node, and keep their output with the node's records before a reset or an RMA. ## Control & Management Beyond observation, AMD SMI allows operators to control power, clock, and performance policies directly from the CLI. This is especially valuable for consistent benchmarking, workload tuning, or enforcing cluster-wide limits. Capabilities include: - **Power cap management**: define GPU-specific wattage ceilings (`amd-smi set -g --power-cap `) to balance performance and power budgets across dense racks. - **Clock domain tuning**: lock core and memory frequencies by setting their minimum and maximum (`amd-smi set -g --clk-limit sclk max `) for reproducible benchmarks or stress tests. - **Fan control**: available on actively cooled cards; on Instinct accelerators, cooling belongs to the chassis. - **State resets**: return clocks and power profiles to their defaults and the power cap to its maximum after tests (`amd-smi reset -g --clocks`, then `--profile` and `--power-cap`, one flag per call), or reset the GPU itself (`amd-smi reset -g --gpureset`). In production, these controls are typically automated through orchestration agents or monitoring frameworks that apply safe limits, ensure uniform settings, and trigger remediation when thresholds are exceeded. ## Integrating with Monitoring & Automation The AMD SMI library (`libamd_smi.so`), with its Python bindings (the `amdsmi` package), gives programmatic access to the same telemetry, enabling: - Prometheus exporters, such as AMD's Device Metrics Exporter - Custom health checks (temperature drift, RAS error thresholds) - Periodic baseline verification across GPU nodes For example, a monitoring agent can periodically call the SMI API to record temperature and ECC counts, and raise an alert when a GPU drifts away from [the rest of its cluster](https://blog.asama.ai/blog/clustering-for-infrastructure-observability). ## Conclusion AMD SMI gives you continuous health, telemetry and RAS data from AMD GPUs, and RDC and RVS add the diagnostics. And yes, we’re always looking for developers who care about **infrastructure, observability, and open source**. **Update, October 2026.** Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. To see what it finds on your own fleet, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.