Diagnosing an InfiniBand Fabric: Counters, Tools, and a Workflow

An InfiniBand fabric fails differently from an Ethernet network. There is no routing protocol to converge, no ARP to go stale, and no spanning tree to blame. When something is wrong, it is usually one of: a link that trained badly, a port accumulating errors, a subnet manager that has not done what you think it has, or a topology that does not match the one in your head.

ibdiagnet finds all four. This is how to read it.

Run it first, read it second#

ibdiagnet

With no arguments it discovers the entire fabric, walks every node and port, checks link state and speed consistency, reads error counters, validates the routing tables, and writes a set of files to /var/tmp/ibdiagnet2/.

The terminal summary is a triage tool, not the output. Warnings and errors there tell you whether to keep going; the files tell you where to look.

To reset counters after reading them, so the next run measures a fresh interval rather than everything since boot:

ibdiagnet -pc

This is the habit that makes the tool useful. Counters accumulated since the last reboot tell you a port has had errors at some point over eight months. Counters from a clean ten-minute window under load tell you which port is failing right now. Always clear, reproduce, then read.

The counters that matter#

ibdiagnet2.pm is long. Most of it is noise for a given investigation. These are the ones worth knowing by name:

SymbolErrorCounter — the receiver could not decode a symbol. A few over a long uptime are unremarkable. A number that climbs while you watch means a physical problem on that link: connector, fibre, transceiver, or a cable running beyond its budget. This is the counter that sends you to mlxlink and mlxcables.

LinkDownedCounter and LinkErrorRecoveryCounter — how often the link has failed and had to retrain. Any non-zero value on a port that has not been physically touched is worth explaining. A port that retrains repeatedly is usually marginal rather than broken, which is exactly the kind of fault that survives casual inspection.

PortXmitDiscards — packets the switch dropped on egress rather than transmitting, typically because of congestion or a credit problem downstream. Distinguishes congestion from corruption: symbol errors say the physical layer is damaged, transmit discards say the fabric is backed up.

PortRcvErrors — packets received with errors of any kind. Read alongside symbol errors.

VL15Dropped — dropped subnet management packets. VL15 carries SM traffic, and it is the one virtual lane that is not flow controlled. Losses here mean the SM is struggling to manage the fabric, which is a different and more serious class of problem than a single bad link.

PortXmitWait — cycles spent with data to send and no credit to send it. This is the congestion signal. High and rising means the far end is not returning credits fast enough, which points at a bottleneck downstream rather than at this port.

The pattern to internalise: symbol and receive errors mean the physical layer; discards and transmit-wait mean congestion; VL15 drops mean the control plane. Those three get investigated in completely different directions, and reading the wrong one sends you a long way down the wrong path.

Checking the fabric is the one you think it is#

Two questions ibdiagnet answers that people forget to ask.

Is the topology the one you designed? ibdiagnet2.net_dump and ibdiagnet2.lst are the discovered reality. Diffing them against a known-good capture catches a cable moved during maintenance and never moved back — a fault that produces no errors at all and quietly ruins the routing.

Is every link at the speed and width you are paying for? One port negotiated down to a narrower width will never appear as an error, and will act as a permanent bottleneck for every path routed through it. ibdiagnet flags inconsistencies, and ibdiagnet2.lst lets you check by hand.

And which subnet manager is actually master? ibdiagnet2.sm will tell you. On a fabric with redundant SMs, the one you think is running and the one that is running are not always the same, and the routing engine in effect is the master’s.

Measuring instead of guessing#

When the complaint is performance rather than errors, measure the path before investigating it.

ib_write_bw -d mlx5_0 --report_gbits <server>
ib_read_bw  -d mlx5_0 --report_gbits <server>
ib_write_lat -d mlx5_0 <server>
ib_read_lat  -d mlx5_0 <server>

Bandwidth and latency, read and write, between a specific pair of hosts. Two things to do with them:

Bisect. If a job is slow, test a pair of nodes within one leaf, then a pair across leaves, then across the spine. Where the number falls off tells you which layer to examine. This turns “the fabric is slow” into “traffic crossing spine 2 is slow”, which is a question you can actually answer.

Baseline. Run the same pair on a healthy fabric and keep the result. Without it you are comparing today’s measurement against an expectation, which is not a comparison.

Capturing fabric traffic#

For problems that survive counters and benchmarks, capture. Wireshark decodes InfiniBand, including management datagrams, and reading actual SM traffic is the fastest way to resolve questions about subnet manager behaviour, partitioning, or what happened during a sweep.

This is a specialist step and needs a capture path — a port mirror on a switch that supports it, or capture on the host side. Not a first resort, but the only honest answer when the counters are clean and the fabric is still misbehaving.

The workflow#

1. ibdiagnet -pc to clear counters.
2. Reproduce the problem, or run under load for a known interval.
3. ibdiagnet and read the summary for errors and warnings.
4. ibdiagnet2.pm — classify: physical errors, congestion, or control plane?
5. Physical → mlxlink and mlxcables on the implicated port, per-lane BER and optical power.
6. Congestion → PortXmitWait along the path, and check the routing: is traffic distributed the way the routing engine intended?
7. Control plane → ibdiagnet2.sm, SM logs, VL15 counters.
8. Verify the topology against a known-good net_dump before concluding anything.
9. Re-measure with perftest to confirm the fix, rather than declaring victory because the counters stopped.

What I would tell someone starting#

Take a known-good ibdiagnet capture now, while the fabric is healthy, and keep it. Almost every diagnosis is a diff, and you cannot diff against something you never recorded.

Always clear counters before investigating. Lifetime counters are nearly useless for a live problem and actively misleading — they make a port that had a bad day in March look like today’s culprit.

Classify before you investigate. The three counter families point in three different directions. Deciding which one you are looking at, before you start pulling cables, is most of the work.

References

• rdma-core — OpenSM and the InfiniBand diagnostic utilities
• perftest — ib_write_bw, ib_read_bw, ib_write_lat, ib_read_lat
• NVIDIA networking documentation — ibdiagnet reference
• Wireshark — InfiniBand and MAD dissection

Counter names and ibdiagnet output file layout vary somewhat across versions; check against your own installation. Thresholds for what constitutes an unacceptable error rate depend on link speed and FEC configuration.