# GPU health checks with NVIDIA DCGM: health watches and dcgmi diag run levels (Part 1)

> How NVIDIA DCGM checks GPU health: health watches for PCIe, memory, thermal, power and NVLink faults, and what each dcgmi diag level tests and how long it runs.

- URL: https://blog.asama.ai/blog/gpu-health--diagnostics----why-they-matter/
- Author: Jain Johny
- Published: 2025-10-07
- Tags: Field notes

NVIDIA's Data Center GPU Manager (DCGM) checks GPU health in two ways. Background health watches run alongside your jobs and flag PCIe, memory, thermal, power and NVLink faults as they appear. Active diagnostics, `dcgmi diag`, range from a software check that takes seconds to stress tests that load the GPU for hours, depending on the level you choose.

When operating GPU infrastructure, whether for AI, ML, HPC, rendering, or simulations, your uptime and performance depend on knowing whether your GPUs are healthy before they fail catastrophically. Faulty memory, ECC errors, power instability, or thermal issues can [degrade performance](https://blog.asama.ai/blog/issue-01-asama-uptime-is-not-reliability) or cause silent errors.

DCGM is NVIDIA's toolkit for real-time health checks, diagnostics, and alerts across NVIDIA GPU fleets. The health and diagnostic features help you:

- Detect latent hardware or configuration issues early
- Automate routine validation and alerting
- Correlate hardware-level failures with workload anomalies

## What DCGM Offers

### Continuous Health Monitoring
For infrastructure engineers, health monitoring is baseline hygiene.
DCGM's background health watches cover six subsystems:

- **PCIe**: [replay events](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) (more than 8 a minute) and XIDs for bus or fabric errors
- **Memory**: uncorrectable ECC errors, faulty memory and retired pages
- **InfoROM**: InfoROM corruption (XID 93)
- **Thermal and power**: thermal and power violations
- **NVLink**: link errors, and error rates above a threshold
- **Driver**: faults that need a GPU reset, a drain or a reboot to recover

Turn them on with `dcgmi health -s`, one letter per watch (p, m, i, t, n and d, or a for all), and read the result with `dcgmi health -c`.

These continuous checks are non-invasive, low-overhead, and essential for keeping a GPU cluster in steady state.

### Diagnostics
Monitoring tells you something’s wrong.
Diagnostics tell you what and why.

From level 2 up, DCGM diagnostics are **invasive**: they put their own load on the GPU and stress it for minutes to hours, so drain the node first. Level 1 runs software checks in seconds, which makes it a readiness check before a job.
Ideal for:
- Maintenance windows
- Burn-in testing
- Root-cause analysis

They uncover:
- Deployment and driver issues
- Integration or container runtime conflicts
- Stress-induced thermal/power anomalies
- Hardware faults ([PCIe links](https://blog.asama.ai/blog/pci-device-metrics-in-prometheus-node-exporter), GPU memory, power delivery)

## dcgmi diag Run Levels: What Each Tests and How Long It Takes

### Diagnostic Levels
Each level runs everything in the levels below it, plus the tests listed:

| Level | Name | Adds | Runtime, 4 GPUs / 8 GPUs | When to run |
| --- | --- | --- | --- | --- |
| `-r 1` | Short | Software checks | Under 2.5 seconds | Before each job, as a readiness check |
| `-r 2` | Medium | PCIe and NVLink, GPU memory, memory bandwidth | Under 2.5 / 10.5 minutes | After maintenance, on a drained node |
| `-r 3` | Long | Diagnostic, targeted stress, targeted power, NVBandwidth, NCCL tests | Under 10 / 35 minutes | Maintenance windows, suspected hardware faults |
| `-r 4` | Extra long | Memtest, pulse test | Under 45 minutes / 2.25 hours | Burn-in and acceptance testing |

Test lists and run times are from [NVIDIA's DCGM diagnostics documentation](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/dcgm-diagnostics.html); they vary with the DCGM version and the number of GPUs.

You choose a level depending on how deep you want to go and how long you can afford the test to run.

**Running Diagnostics via DCGMI**
DCGMI is the CLI front end for DCGM. Example commands:

```bash
dcgmi diag -r 1     # run level 1 diagnostic: seconds
dcgmi diag -r 4     # run the deepest diagnostic: up to hours, on a drained node
dcgmi health -s a   # start health monitoring on all GPUs
dcgmi health -c     # query current health status
```

You can also tailor diagnostics by adjusting parameters (e.g., memory thresholds, enabling or disabling specific tests).

## What to Do When a Check Fails

A failed health watch is a reason to drain the node and run a longer diagnostic, not to keep scheduling jobs on it. When a diagnostic fails on PCIe, memory or power, save its output, and an `nvidia-bug-report.sh` log, before you reset the GPU or open an RMA, so the failure is on record when you talk to your hardware vendor.

## Conclusion

NVIDIA’s DCGM is a toolkit that helps with continuous health monitoring and diagnostics of NVIDIA’s GPUs.

And yes, we’re always looking for developers who care about infrastructure, observability, and open source.

**Update, October 2026.** The health watches and run levels above follow NVIDIA's DCGM documentation as of October 2026, which also lists a ConnectX watch (x) for ConnectX network adapters that report an unhealthy status or uncorrectable PCIe AER errors. Asama's fleet visibility is generally available, and detection built on it has been validated; investigation and remediation are expanding. To see what it finds on your own fleet, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.