# The correctable error trap: when PCIe correctable errors start to matter (Problem Specifics 2/5) > Why a climbing PCIe correctable error rate can signal a failing link, how to read AER counters in sysfs, and why peer comparison beats a fixed replay threshold. - URL: https://blog.asama.ai/blog/issue-07-asama-the-correctable-error-trap/ - Author: Jain Johny - Published: 2026-01-18 - Tags: Fleet problems, Field notes A PCIe correctable error is an error the hardware recovers from on its own, most often by retransmitting a packet that was damaged on the link. Linux [counts them per device](https://github.com/torvalds/linux/blob/v6.18/Documentation/ABI/testing/sysfs-bus-pci-devices-aer) in `/sys/bus/pci/devices//aer_dev_correctable` and logs them only as warnings. Prometheus's node_exporter does not collect them, so in many stacks they never reach a dashboard. The retransmissions are not free: each one costs bandwidth and latency, so a link whose correctable error rate is climbing can already be losing throughput before anything fails. PCIe correctable errors occupy an uncomfortable middle ground in infrastructure monitoring. They self-heal: the bus-level retransmission protocol catches them, corrects them, and the transaction completes. No data is lost. No service is interrupted. By every operational definition, [nothing went wrong](https://blog.asama.ai/blog/issue-01-asama-uptime-is-not-reliability). Most monitoring systems classify them accordingly: non-critical noise. Alerts are suppressed. Dashboards ignore them. And quietly, the correctable error rate on a subset of nodes climbs, until the link degrades, the errors become uncorrectable, and an incident begins that will take days to investigate. > "A correctable error rate that is low and stable is ignorable. A correctable error rate that is climbing under specific load conditions is a failure in progress until proven otherwise." --- ## What PCIe degradation actually looks like > ### Early: errors and replays climb together > > **Signal** > Correctable AER counts climb on specific nodes under GPU load: receiver errors, bad TLPs and DLLPs, replay timer timeouts and replay number rollovers, which tend to rise together when a link is marginal. > > **What monitoring sees** > Noise. Alerts suppressed. No incident opened. > > **Reality** > The link is already retransmitting, and every replay costs bandwidth. The degradation process has already started. --- > ### Middle: the link slows > > **Signal** > The link [retrains at a lower speed or width](https://blog.asama.ai/blog/article-02-asama-where-the-loop-always-starts) than it supports, or replays run constantly. > > **What monitoring sees** > Workload still functional. No outage condition triggered. > > **Reality** > Throughput is constrained and hardware instability is accelerating. --- > ### Late: the incident > > **Signal** > Errors become uncorrectable. GPU falls off the bus ([Xid 79](https://docs.nvidia.com/deploy/xid-errors/contents.html) on NVIDIA GPUs) or becomes unstable. > > **What monitoring sees** > GPU job failures, crashes, application instability. > > **Reality** > Engineers begin a multi-day investigation of a problem that was visible long before. --- ## Reading the counters ```bash cat /sys/bus/pci/devices/0000:41:00.0/aer_dev_correctable # counts since boot by type (RxErr, BadTLP, BadDLLP, Rollover, Timeout, ...), then TOTAL_ERR_COR cat "$(dirname "$(readlink -f /sys/bus/pci/devices/0000:41:00.0)")/aer_dev_correctable" # the same counters on the port above sudo lspci -vv -s 41:00.0 | grep -E 'LnkCap|LnkSta' # what this end supports, and what the link trained at; read under load sudo journalctl -k -b | grep -E 'PCIe Bus Error|aer_status|Hardware Error' # native AER lines, and the error records firmware passes on sudo journalctl -k -b | grep -i '_osc' # no AER in "OS now controls [...]" means firmware owns it ``` The counters run from boot, so track their rate, not their value. They are counted where the error was seen, which may be the other end of the link: if a GPU's counters are all zero, read the port above it. If every counter on the path stays at zero, check who owns AER. Linux handles it [only when firmware grants control through ACPI _OSC](https://github.com/torvalds/linux/blob/v6.18/Documentation/PCI/pcieaer-howto.rst); where firmware keeps it, corrected errors reach the OS only if the BIOS passes them on, and the BMC's event log may be the only record. Since kernel 6.16, the log is rate-limited by default to 10 correctable-error reports per device every 5 seconds, which is one more reason to read the counters rather than the log. Read LnkSta under load: many GPUs drop the link speed at idle to save power, so a slow link at idle is usually normal. A link narrower than both ends support (LnkCap on the device and on the port above it), or narrower than the same slot on its peers, points at lanes that failed to train. For Prometheus, [node_exporter's pcidevice collector](https://blog.asama.ai/blog/pci-device-metrics-in-prometheus-node-exporter), which is off by default, exports each link's current and maximum speed and width, so a link that trained narrow shows up as a metric. It does not read the AER counters, so collect `aer_dev_correctable` with a textfile collector script. On NVIDIA GPUs, [DCGM's PCIe health watch](https://blog.asama.ai/blog/gpu-health--diagnostics----why-they-matter) counts replays from the GPU's end. --- ## The signal you're suppressing The right way to treat correctable PCIe errors is not to ignore them and not to alert on every instance. It is to track the rate over time, compare it against a baseline for the specific hardware under the specific workload, and flag nodes where the rate is deviating from [cohort behaviour](https://blog.asama.ai/blog/clustering-for-infrastructure-observability). A node with a low, stable correctable error rate is likely fine. A node whose rate has doubled over the past two weeks under the same GPU workload, on hardware from a specific procurement batch, while its peers held steady, is a failure in progress until proven otherwise, even though every error has self-healed and no alert has fired. This distinction requires rate-of-change tracking and peer comparison. A fixed threshold can't express it on its own: DCGM's PCIe watch fires at more than 8 replays a minute, but the node to look at first is the one that went from a few replays a day to six a minute while its peers stayed near zero. And none of it can be seen by a system that has decided to suppress correctable error data entirely. --- ## Food for thought > Does your monitoring track PCIe correctable error rates over time, or does it only fire when errors become uncorrectable and the damage is already done? **Update, October 2026.** Asama's fleet visibility, which reads PCIe correctable-error and replay counters on every node, is generally available. Detection built on it, which compares each node against its own baseline and against its peers, has been validated; investigation and remediation are expanding. To see which of your links are climbing ahead of their peers, book a live comparison against your stack at [asama.ai](https://www.asama.ai), or write to contactus@asama.ai.