Wirewalk Advisory

An MPI troubleshooting playbook: symptom to cause

2026-09-09T10:00:00-04:00

This is the reference half of the series. The earlier articles argue for testing in a particular order; this one is the lookup table for when something is already broken and you need the shortest path to a cause.

The single most useful habit underneath all of it: capture the node list, the environment and the module versions with every run. Most of the diagnoses below are trivial with that information and genuinely difficult without it.

The index

Symptom	Most likely causes	First command
Hangs at `MPI_Init`	PMIx mismatch, launcher, hostfile	`mpirun --mca plm_base_verbose 10`
Hangs at first collective	QP setup, memlock, one unreachable node	`srun -N all bash -c 'ulimit -l'`
Runs, but ~10x slow	Silent TCP fallback	`UCX_TLS=rc_x,sm,self`
Fails only above N ranks	RC queue-pair exhaustion, registration memory	`UCX_TLS=dc_x,sm,self`
One rank always late	Degraded link, throttled node, stray daemon	Pairwize `ib_write_bw` sweep
Timings vary run to run	Allocation varies, shared ISLs, jitter	Record and compare node lists
Bandwidth exactly half	Width negotiation, PCIe width, single rail	`iblinkinfo`, `lspci -vv`
`Retry exceeded` errors	Fabric drops, RoCE PFC/ECN, dead peer	Port counters, `ethtool -S`
Worse after a maintenance window	Firmware skew, routing engine fell back	`ibv_devinfo` fleet-wide, SM log

Everything below expands one row.

Hangs at startup

A job that never reaches the application is a launcher problem, not a fabric problem, and the fabric tooling will waste your time here.

mpirun --mca plm_base_verbose 10 --mca rmaps_base_verbose 5 -np 4 hostname

That prints the launch sequence as it happens, and the point at which it stops is the diagnosis. Common stopping points:

Cannot reach a node. The launcher uses SSH or the scheduler’s own launch mechanism. If it is SSH, key-based access between compute nodes must work non-interactively, and on a cluster with pam_slurm_adopt it must work from within an allocation, which is not the same test as from the login node.

PMIx version mismatch. This is the common one on schedulers, and it produces either a hang or an error naming PMIX. Slurm and Open MPI are each built against a PMIx, and if the two disagree, srun-launched MPI jobs fail while mpirun-launched ones work, or vice versa. Establish what each side has:

srun --mpi=list
ompi_info | grep -iE 'pmix|prrte'
pmix_info --version

Then be explicit rather than relying on the default:

srun --mpi=pmix -N 2 -n 4 ./a.out

If srun --mpi=list does not offer a PMIx entry, Slurm was not built with PMIx support and mpirun under an allocation is your path.

“There are not enough slots available in the system.” The hostfile or the scheduler allocation says fewer slots than -np asked for. Under a scheduler, this usually means mpirun did not pick up the allocation. --oversubscribe silences it and does not fix it; check $SLURM_JOB_NODELIST and $SLURM_NTASKS instead.

Hangs at the first collective

Startup succeeded, ranks exist, and the job stops the first time they all have to talk. This is connection establishment.

Locked memory limit. RDMA needs pinned pages. If the limit is not unlimited inside the job, registration fails. The limit that matters is the one inside the launched process, which is not necessarily the one in /etc/security/limits.conf:

srun -N 8 --ntasks-per-node=1 bash -c 'echo $(hostname) $(ulimit -l)'

Anything other than unlimited on any node is the answer.

A node that is up but has no working adapter. The job launches everywhere and then waits for the one rank that cannot connect. Check the whole allocation rather than a sample:

srun -N $SLURM_NNODES --ntasks-per-node=1 bash -c \
  'echo "$(hostname) $(ibstat mlx5_0 1 | awk "/State:/{print \$2}" | head -1)"'

Firewall on the fabric interface. Rare on InfiniBand, common on RoCE, where the traffic is UDP and a host firewall will happily block port 4791.

To see where it is stuck rather than guessing, attach to a hung rank:

pstack $(pgrep -n a.out)          # or: gdb -p <pid> -batch -ex bt

A stack sitting in a UCX progress or connection-establishment function is a transport problem. A stack in application code is not.

Silent fall back to TCP

The most expensive quiet failure in MPI, because nothing is broken. The job completes, the numbers are bad, and the conclusion recorded is that the application does not scale.

The signature is quantitative: bandwidth roughly an order of magnitude below the link rate and small-message latency roughly an order of magnitude above what the fabric should give. If a two-node osu_bw on a 200 Gb/s fabric reports a couple of gigabytes per second, stop tuning and check the transport.

The conclusive test removes the fallback:

mpirun -np 2 -x UCX_TLS=rc_x,sm,self -x UCX_NET_DEVICES=mlx5_0:1 ./osu_bw

No TCP transport is listed. If the job runs, RDMA was working and the problem is elsewhere. If it fails to establish connections, TCP was carrying the traffic.

Corroborate by watching the device counters move — or not — during a run:

before=$(cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data)
mpirun -np 2 -map-by ppr:1:node ./osu_bw -m 8388608:8388608 >/dev/null
after=$(cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_xmit_data)
echo "delta: $(( after - before ))"

Causes, in the order they are usually found: ulimit -l not unlimited; a device name in UCX_NET_DEVICES that does not exist on some nodes; a node with a different OFED; a port that is down on one node; and, on GPU nodes, a CUDA-aware build mismatch that causes UCX to reject the RDMA path for device buffers.

Put the UCX_TLS=rc_x,sm,self run in your acceptance suite as a pass/fail gate. It costs seconds and it converts the single most damaging silent degradation into a loud, obvious failure. Almost nobody does this, and it is the highest-value five lines in an MPI test harness.

Fails only above a certain scale

A job that works at 8 nodes and fails at 64 is usually a resource that scales with the square of the rank count.

Reliable Connection queue pairs. With RC, each rank maintains a QP to each peer it talks to. Full connectivity across n ranks is n² QPs, and each consumes adapter and host resources. On a node running many ranks, this becomes the constraint well before the fabric does.

Dynamically Connected transport solves it by multiplexing:

mpirun -x UCX_TLS=dc_x,sm,self -x UCX_NET_DEVICES=mlx5_0:1 ./a.out

If a job fails at high ranks-per-node with connection or resource errors and succeeds with dc_x, that was it. Note that DC availability depends on the adapter generation, and that the failure mode when DC resources themselves are exhausted is different and less obvious — usually a hang rather than an error.

Memory registration limits. Large registered regions need translation table entries. The relevant firmware parameters differ by adapter and driver generation; the symptom is a registration failure at a size or rank count that worked before.

Aggregate memory per node. Each connection carries buffers. At high ranks-per-node, MPI’s own buffer footprint becomes significant, and the job dies of OOM in a way that looks like an application memory bug. dmesg on the node that died settles it in one line.

One rank is always late

A collective takes as long as its slowest participant, so one degraded node degrades everything. Finding it is mechanical.

Bisection is fastest: split the allocation, run both halves, keep the slow half, repeat. Six rounds for 64 nodes.

The systematic version tests every node against one reference, at the verbs layer so MPI is not in the picture:

REF=node001
for n in $(scontrol show hostnames $SLURM_JOB_NODELIST); do
  [ "$n" = "$REF" ] && continue
  bw=$(ssh $n "ib_write_bw -d mlx5_0 -F -s 1048576 -D 5 --report_gbits $REF" \
        | awk '/1048576/ {print $4}')
  echo "$n $bw"
done | sort -k2 -n | head -5

Once you have the node, the causes are a short list:

Narrow or degraded link — iblinkinfo, mlxlink -d mlx5_0 -p 1 --show_fec.
Downtrained PCIe slot — lspci -vv | grep LnkSta.
CPU frequency capping or thermal throttling — turbostat, cpupower frequency-info, and dmesg | grep -i thermal.
A daemon on a core that a rank is pinned to.
Memory running at a lower speed, or a DIMM that failed into a degraded configuration — dmidecode -t memory, and the platform’s own health log.

On the last point about cores: aggregate CPU idle proves nothing on a many-core node. One rank spinning uselessly on a 192-core machine is half a percent of system-wide CPU. top shows a healthy node. Look per-core:

mpstat -P ALL 1 5
pidstat -t -p $(pgrep -n a.out) 1 5

Timings vary wildly between runs

Before treating variance as a signal, establish that it is not the experiment.

The allocation changed. Different nodes, different leaves, different number of spine crossings. This is the most common cause by a wide margin. Record scontrol show hostnames $SLURM_JOB_NODELIST with every result and compare.

Other jobs share the fabric. Exclusive node allocation does not give you exclusive links. Another job’s alltoall on the same spine uplinks changes your result, and you have no visibility into it from inside your job. If you are producing numbers that matter, get an exclusive fabric window, or at minimum record what else was running.

Cached or warm state. The first iteration of anything includes connection setup and memory registration. Every benchmark here has a warmup flag; use it. Where filesystem I/O is in the loop, a repeat run served from page cache is not a measurement of anything.

OS jitter. Unpinned ranks, an unsynchronized housekeeping daemon, a monitoring agent that wakes every thirty seconds. Collectives amplify jitter because every rank waits for the one that was interrupted. --bind-to core and --report-bindings first; if variance persists with clean binding, look at what runs on the compute nodes on a timer.

Power and frequency policy. A node that boosts for the first thirty seconds and then settles produces a benchmark whose result depends on its duration. Fix the governor for measurement runs and record what it was.

A run-to-run spread you cannot get under a few percent is not a slow cluster. It is an uncontrolled experiment, and no amount of tuning against it will converge.

Bandwidth that is a clean fraction of expectation

Half, a quarter, a tenth. Clean fractions have mechanical causes and are worth a minute of arithmetic before any investigation.

Fraction	Look at
Exactly half	Link width 2X not 4X; one rail of two in use; PCIe x8 not x16
Exactly a quarter	Link width 1X; PCIe x4; two independent halvings
About a tenth	TCP fallback, or IPoIB instead of RDMA
About 60 percent	Often real — protocol overhead plus an unsaturated single rank

ibstatus mlx5_0 | grep rate
iblinkinfo -l | grep -v 4X
lspci -vv -s <bdf> | grep -E 'LnkCap|LnkSta'

MTU

On InfiniBand, path MTU is negotiated and normally 4096. On RoCE it is derived from the Ethernet MTU, and a single hop at 1500 anywhere in the path drops the RoCE MTU to 1024 with a real bandwidth cost.

The ib_write_bw header prints what was actually negotiated, which is the number to trust over any configuration file:

 Mtu             : 4096[B]

If that reads 1024 on a fabric you configured for jumbo frames, walk the path. The offender is usually a recently added switch port or a host whose interface configuration did not include the MTU.

IPoIB has its own version of this — datagram mode caps at 2044 bytes while connected mode allows much larger — which matters if any part of your workload, including the scheduler or the filesystem, is running over IPoIB.

After a maintenance window

Performance that changed after a window, with no application change, has a short suspect list.

Firmware skew. Some nodes were updated and some were not, or a node was replaced with one from a different batch. Group by PSID and compare within groups:

pdsh -w node[001-256] 'ibv_devinfo | grep -E "fw_ver|board_id"' | dshbak -c

The routing engine fell back. A switch that was down during the sweep, or a node cabled differently on the way back in, can make ftree decline the topology and fall back to updn for the entire fabric. The subnet manager logs it and nobody reads it:

grep -iE 'routing engine|fallback|ftree' /var/log/opensm.log | tail -40

A standby subnet manager took over. If the standby holds an older configuration, the whole fabric is now routed by that configuration. sminfo tells you which SM is master.

A driver or kernel change altered defaults. UCX and OFED defaults do move between releases. Diff the effective configuration, not the configuration files:

ucx_info -f -c > /var/tmp/ucx-$(date +%F).conf
diff /var/tmp/ucx-<previous>.conf /var/tmp/ucx-$(date +%F).conf

Error messages worth recognizing

IBV_WC_RETRY_EXC_ERR / “Retry exceeded”. The sender gave up after retransmitting. On InfiniBand, this usually means the peer died or a link went down mid-transfer. On RoCE, it much more often means the fabric is dropping — PFC not enabled on the RoCE priority, or a mismatched trust mode. Go to the per-priority discard counters.

RNR retry exceeded. The receiver had no buffer posted and kept saying so until the sender gave up. Usually an application or middleware issue rather than a fabric one, though severe congestion can produce it.

The fork() warning. Registered memory and fork() interact badly. If your application or a library it calls forks after registering memory, you get either a warning or corruption. ibv_fork_init and the corresponding MPI settings exist for this; a job that calls out to shell commands from inside a rank is the usual trigger.

UCX ERROR ... Connection reset by remote peer. A peer rank died. Find out why that rank died; the UCX error is a consequence, not a cause. Check dmesg on its node for OOM.

The habit that shortens all of this

Keep a baseline file, in version control, per cluster:

OFED, firmware and PSID per node group
PCIe LnkSta per adapter
ib_write_bw and ib_read_bw plateaus for a named intra-leaf and cross-spine pair
osu_latency at 8 bytes and osu_bw at 8 MB
Allreduce and alltoall curves at the job sizes you run, with node lists
The width-and-rate histogram for the whole fabric

Regenerate it after every maintenance window. Almost every diagnosis in this article becomes a diff against that file, and a diff takes minutes where an investigation takes days.

EDR to XDR in one fabric: rate reporting, width traps, and the node that gates the job

2026-09-09T09:00:00-04:00

Clusters accumulate generations. An EDR island from 2017 that still runs production, an HDR expansion, an NDR refresh, and now XDR arriving for the GPU tier — frequently all reachable from one another, sometimes in one subnet. Mixed fabrics work. What does not work is reasoning about them from a mental model built on a single generation, because the reporting is inconsistent and the failure modes are quiet.

The rate table, and which number each tool prints

The confusion is entirely avoidable and it costs people days.

Generation	Lanes	Per-lane signalling	4X link rate	Encoding
FDR	4	14.0625 Gb/s	56 Gb/s	64b/66b
EDR	4	25.78125 Gb/s	100 Gb/s	64b/66b
HDR	4	53.125 Gb/s	200 Gb/s	PAM4, 64b/66b + RS-FEC
NDR	4	106.25 Gb/s	400 Gb/s	PAM4, RS-FEC
XDR	4	212.5 Gb/s	800 Gb/s	PAM4, RS-FEC

Now the part that matters. ibstat prints the aggregate link rate:

	Rate: 400

ibstatus prints the aggregate rate and helpfully names the generation:

	rate:            400 Gb/sec (4X NDR)

iblinkinfo prints the width and the per-lane rate as separate fields:

  37   12[  ] ==( 4X 106.25 Gbps Active/  LinkUp)==>  91   1[  ] "node17 mlx5_0" ( )

That is a 400 Gb/s link. Read the 106.25 as the link rate and you conclude your NDR fabric is running at roughly 100 Gb/s, which is a conclusion people reach, escalate, and occasionally publish. The 4X prefix is doing the work and it is easy to skim past.

On a mixed fabric this is worse, because 4X 106.25 Gbps (NDR400) and 4X 25.78125 Gbps (EDR100) look superficially similar in a long scan and the distinguishing digits are in the middle of the field.

The rule that keeps this straight: ibstat and ibstatus report the link, and iblinkinfo reports a lane. Whenever you quote a number from iblinkinfo, quote the width with it. A capacity claim sourced from that field without the 4X attached is wrong by a factor of four, and it is wrong in the direction that makes people buy hardware.

The width trap

Width is the field that silently costs you three quarters of a link.

An InfiniBand link can negotiate 1X, 2X or 4X. A link that comes up at 1X is Active, LinkUp, routable, pingable, and passes every functional test. It runs at a quarter rate. Nothing logs an error, because from the fabric’s point of view nothing is wrong — the two ends negotiated the best width they could agree on, and that is what negotiation is for.

Causes: a damaged or partially seated cable, a transceiver with a failed lane, a splitter cable in a port that is not configured to split, or a port configured to split that has a straight cable in it.

The scan to run across the whole fabric, on a schedule:

iblinkinfo -l | awk '$0 !~ /4X/ && /Active/ {print}'

And the more useful version, which catches both narrow links and links that negotiated a lower generation than they should have:

iblinkinfo -l | grep -oE '[0-9]+X +[0-9.]+ Gbps' | tr -s ' ' | sort | uniq -c | sort -rn

That gives you a histogram of every distinct width-and-rate combination in the fabric. On a homogeneous fabric it should have one row. On a mixed fabric it should have exactly as many rows as you have generations, and every count should match what you cabled. Any row with a small count is the anomaly, and small counts are the whole point of the histogram.

HDR100 and the two ways to get it

HDR100 is the case where the same nominal rate arrives by two different physical arrangements, and the difference matters when you are diagnosing.

An HDR switch port carries four HDR lanes. Split, it presents as two ports of two lanes each — HDR100, reported as 2X 53.125 Gbps. Separately, there are HDR100 adapters, which are physically capable of two lanes only and report 2X even into an unsplit port.

So 2X 53.125 Gbps is normal and expected on a split fabric with HDR100 adapters, and is a fault on a fabric where you believed everything was HDR200. The histogram above is what tells you which situation you are in; the individual port reading does not.

Three practical consequences:

A single HDR200 adapter plugged into a split HDR100 port runs at 100. Half of the adapter is unused and nothing reports it.
A single HDR100 adapter plugged into an unsplit HDR200 port also runs at 100, and wastes half a switch port.
Cable compatibility is not guaranteed across these arrangements. Active optical cables in particular have power class requirements that some port and adapter combinations do not satisfy, and the symptom is a port that will not come out of Polling with no error message that explains why. Check the transceiver explicitly before assuming a bad cable:

mlxlink -d mlx5_0 -p 1 -m

That reports the module’s vendor, part number, type and supported rates, which is the information the failure message does not give you.

When one node gates the job

This is the expensive lesson, and it is the reason the width scan belongs on a schedule rather than in a runbook.

A synchronized collective — allreduce, alltoall, a barrier before a timing region — completes when its slowest participant completes. The whole job proceeds at the rate of its worst link. One node at 2X in a 32-node NDR allocation does not cost you one thirty-second of the throughput. Depending on the communication pattern, it can cost a large fraction of it, because every other rank waits.

What makes it expensive is not the degradation, it is the presentation. The symptom is:

Results that vary between runs by a wide margin.
The variation correlating with nothing you changed.
Point-to-point tests that are clean, because the odds of any given pair including the bad node are low.
Fabric-wide diagnostics that report no errors, because a narrow link is not an error.

It presents as a fabric-wide capacity problem. It gets investigated as one. The routing engine gets blamed, ISL capacity gets blamed, a purchase order for more spine ports gets discussed — and the actual finding is one transceiver.

The tell is the correlation with allocation. If you keep the node list with every result, the correlation is visible in an afternoon. If you do not, it is not visible at all, which is why the earlier articles in this series keep insisting on recording the node list.

The other tell is arithmetic. If a synchronized benchmark comes in at close to a clean fraction of expectation — half, a quarter — suspect a width negotiation before suspecting anything subtle. Fabrics rarely degrade by exactly 50 percent for interesting reasons.

Mixing generations in one subnet

Several things happen that are worth anticipating.

Links negotiate down to the lower common generation. An NDR adapter into an HDR switch port gives HDR, with the appropriate cable. That is correct behavior and it is only a problem when nobody recorded the intent, and a year later somebody is trying to work out why a node is at 200 when the inventory says 400.

Path MTU is set by the minimum along the path. OpenSM computes per-path MTU, so a subnet containing something that supports only 2048 will have paths through it at 2048 while other paths run at 4096. Two node pairs then produce different ib_write_bw headers on the same fabric, which is confusing if you have not seen it before. Check the SM’s configured maximum and the per-device capability:

grep -i max_mtu /etc/opensm/opensm.conf
ibv_devinfo -v | grep -E 'max_mtu|active_mtu'

Service levels and virtual lanes may differ. The number of data VLs a device supports varies across generations. If you use QoS and service-level mapping, the mapping has to be valid on the least capable device in the path, and an invalid mapping does not fail loudly — traffic lands on VL0 with everything else.

The routing engine is constrained by the least regular part of the fabric. ftree requires a regular fat tree; an older island bolted on with a different uplink ratio is exactly the sort of irregularity that makes it decline and fall back to updn, fabric-wide. So adding a generation can change the routing of the parts you did not touch.

Forwarding table capacity is set by the oldest switch. Every switch needs an entry for every LID it must reach. The smallest table in the fabric governs, and older switches have smaller tables. The article on LID budgets in this series covers the arithmetic.

Firmware skew across a mixed fleet

Mixed generations mean multiple firmware trains, and mixed firmware within a single model is where the odd behavior lives.

pdsh -w node[001-256] 'ibv_devinfo | grep -E "hca_id|fw_ver|board_id"' | dshbak -c
mlxfwmanager --query
flint -d /dev/mst/mt4123_pciconf0 q

Group by board_id — the PSID — not by model name. Within a PSID group, the firmware version should be identical. Across PSID groups it will not be and should not be expected to be. The check is uniformity within a group, and the detail of doing the update is covered in the ConnectX firmware article in this series.

Skew within a group is worth taking seriously because the symptoms are generation-specific and unhelpful: a link that trains at a lower width with one firmware and correctly with another, a counter that reads differently, a congestion-control feature present on some cards and not others.

Physical quality, which is different from link state

A link can be at full width and full rate and still be marginal. Forward error correction hides a great deal, and it hides it right up until the offered load is high enough that it cannot.

mlxlink -d mlx5_0 -p 1 -c --show_fec --show_eye --show_counters

The fields to compare across the fleet are raw BER, effective BER, and the FEC histogram if the tool provides one for your device. A port whose raw BER is orders of magnitude worse than its peers while its effective BER is clean is a link that FEC is currently rescuing. It will pass every functional test. It is also the one that will produce intermittent, load-correlated performance problems that no configuration change fixes.

Comparing across the fleet is the essential part. A raw BER figure in isolation is hard to judge; the same figure alongside sixty-three peers is obvious.

Practical policy for a mixed estate

Keep generations in separate scheduler partitions and separate topology groups. A job that straddles an NDR island and an EDR island runs at EDR, and the user will not know why. Make the boundary explicit in the scheduler rather than implicit in the fabric.

Run the width and rate histogram on a schedule and alert on any change to the row counts. It is one command, it is cheap, and it converts the most expensive class of quiet failure into a notification.

Record intent alongside the fabric. For every link that is deliberately at a lower rate than the adapter supports, write down why. Otherwise every future audit rediscovers it as an anomaly and someone eventually “fixes” it.

Baseline mlxlink output per port at install and compare periodically. It is the only way to see a link degrading rather than a link failed.

MPI on InfiniBand and on Spectrum-X: what transfers and what does not

2026-09-08T14:00:00-04:00

Running MPI over an Ethernet fabric with RoCEv2 and running it over InfiniBand look the same from the application. Same verbs API, same UCX, same Open MPI, same OSU benchmarks, same ib_write_bw. The adapter presents an /sys/class/infiniband/ device either way, which is why so much tooling appears to be portable.

What is not portable is everything that makes the fabric lossless, and that is where the operational differences live. On InfiniBand, link-level flow control is built into the architecture and is on by default. On Ethernet it is a configuration you have to get right on every port of every switch and every NIC in the path, and getting it wrong on one hop degrades the whole fabric.

The concept mapping

InfiniBand	Ethernet / RoCEv2
Subnet manager assigns LIDs and computes routes	No SM. L3 routing, BGP/ECMP, ARP and neighbour discovery
Credit-based link flow control, always on	PFC (802.1Qbb) per priority, configured per port
Congestion control notification (CCT/CCTI), rarely enabled	ECN marking plus CNP, and DCQCN on the NIC — normally mandatory
`ibstat`, `iblinkinfo`, `ibdiagnet`, `perfquery`	`ethtool -S`, `mlnx_qos`, `lldpctl`, switch telemetry
Path MTU 256–4096, negotiated	RoCE MTU derived from the L2 MTU; jumbo frames needed for 4096
Static routing by default; adaptive routing a switch feature	ECMP by flow hash; Spectrum-X adds per-packet adaptive routing
Fabric is a single administrative object	Fabric is an IP network, with everything that implies

Everything in the left column that you know how to use has an equivalent in the right column that behaves differently enough to catch you out.

What Spectrum-X changes about the Ethernet side

Plain RoCEv2 on general-purpose Ethernet is workable and fragile. The fragility comes from two places: PFC is a blunt instrument that spreads congestion backwards through the fabric, and ECMP hashes flows onto links, so a small number of large elephant flows — which is exactly what a collective produces — can collide on one link while others sit idle.

Spectrum-X addresses both. The switch performs adaptive routing at packet granularity rather than flow granularity, spreading a single flow’s packets across available paths, and the NIC handles the resulting out-of-order arrival and places data correctly without the sender having to care. Congestion control is done in coordination between switch telemetry and the NIC rather than by the generic DCQCN loop alone.

The practical consequence for MPI is that the ECMP collision problem — the one that makes plain RoCE alltoall performance erratic — is substantially addressed by the platform rather than by your hashing configuration. The PFC and ECN configuration discipline does not go away.

Treat the specific feature names and defaults for your switch OS release as authoritative over anything written here. What is stable is the shape of the problem, not the syntax.

PFC, and why one hop matters

Priority Flow Control sends a PAUSE frame for a specific traffic class when a port’s ingress buffer for that class fills. It is hop-by-hop: the upstream device stops sending that class until the pause expires or is canceled.

Two properties make it treacherous.

It is per-priority, and the priority has to match end to end. The NIC marks RoCE traffic with a DSCP value or a PCP value; the switch maps that to a traffic class; PFC is enabled on that class. Break the chain at any point — one switch that trusts PCP where the NIC is marking DSCP, one port where PFC was never enabled on priority 3, one uplink that was added later from a different template — and RoCE traffic on that hop lands in a lossy class. It then drops under congestion.

A drop is far more expensive than it looks. RoCE’s reliable connection transport recovers from loss, but recovery costs a round trip and, on older implementations, a retransmission of everything after the lost packet. A drop rate that would be invisible to TCP is enough to flatten collective performance.

And because a collective completes when its slowest participant completes, a single misconfigured hop that affects a handful of flows degrades the whole job. This is the mechanism behind the most confusing class of RoCE incident: every link is up, no interface shows errors, throughput between any two nodes you test is fine, and the alltoall is half what it should be.

Checking the chain

On the NIC:

mlnx_qos -i enp1s0f0

DCBX mode: OS controlled
Priority trust state: dscp
Cable len: 7
PFC configuration:
	priority    0   1   2   3   4   5   6   7
	enabled     0   0   0   1   0   0   0   0

Two lines matter. Priority trust state must agree with what the switch is trusting. PFC configuration must have the RoCE priority enabled and must agree with the switch’s per-port configuration.

Set it explicitly rather than inheriting:

mlnx_qos -i enp1s0f0 --trust dscp
mlnx_qos -i enp1s0f0 --pfc 0,0,0,1,0,0,0,0

Confirm what the RoCE traffic is actually marked with:

cat /sys/class/net/enp1s0f0/ecn/roce_np/cnp_dscp
cma_roce_tos -d mlx5_0

Then read the pause counters after a run. These are the evidence that PFC is being exercised, and whether it is being exercised on the priority you intended:

ethtool -S enp1s0f0 | grep -E 'prio[0-7]_(pause|buf_discard)'

rx_prio3_pause: 148213
rx_prio3_pause_duration: 3320119
tx_prio3_pause: 96204
tx_prio3_pause_duration: 1904772
rx_prio0_buf_discard: 0
rx_prio3_buf_discard: 0

Read it as follows. Pause counts on priority 3 that are non-zero and growing mean PFC is active and congestion is real — that is the mechanism working, not a fault, though sustained heavy pausing means the fabric is oversubscribed for the offered load. A non-zero buf_discard on the RoCE priority means packets were dropped in a class that was supposed to be lossless, which is a configuration failure. And pause counters on priority 0 while RoCE was running means your traffic is not in the class you think it is.

The counter that most often reveals the problem is rx_prio0_pause being non-zero on a fabric that was configured for RoCE on priority 3. It means the marking or the trust mode is wrong somewhere, and the traffic is being pause- controlled — or not — in the default class. Every link is up, nothing logs an error, and the collectives are poor.

ECN and DCQCN

PFC stops congestion at the cost of pushing it upstream. ECN is the mechanism that instead tells the sender to slow down, and it is what you want doing most of the work, with PFC as the last resort that prevents loss.

The loop: the switch marks packets ECN-CE when its queue exceeds a threshold; the receiving NIC sees the mark and returns a Congestion Notification Packet; the sending NIC reduces its rate for that queue pair and then recovers according to the DCQCN parameters.

The switch side is a WRED-style configuration with a minimum threshold, a maximum threshold and a marking probability, per queue. Those numbers are the main tuning lever and they interact with buffer size, link rate and round-trip time, so they are genuinely site-specific — a setting tuned for a two-tier 100 GbE fabric is not right for a 400 GbE one.

The NIC side lives in sysfs:

ls /sys/class/net/enp1s0f0/ecn/roce_np/    # notification point (receiver)
ls /sys/class/net/enp1s0f0/ecn/roce_rp/    # reaction point (sender)

cat /sys/class/net/enp1s0f0/ecn/roce_np/enable/3
cat /sys/class/net/enp1s0f0/ecn/roce_rp/enable/3

Both must be enabled on the RoCE priority, on every node. A node with the reaction point disabled does not slow down when told to, which means it wins against its better-behaved neighbours and drives the fabric into PFC. One such node in a large allocation is enough to make the whole job’s timing erratic, and it is a difficult thing to suspect if you are not looking for it.

The counters that show the loop working:

ethtool -S enp1s0f0 | grep -E 'cnp|ecn'

np_ecn_marked_roce_packets: 41229
np_cnp_sent: 40871
rp_cnp_handled: 39655
rp_cnp_ignored: 0

np_ecn_marked_roce_packets rising means the switch is marking, so ECN is configured on the switch and the path. np_cnp_sent should track it closely. rp_cnp_handled rising on the senders means they are reacting. rp_cnp_ignored being non-zero means CNPs are arriving for queue pairs that no longer exist or in a state where they cannot be acted on, and a large value is worth investigating.

The failure signature to recognize: marking is happening, CNPs are being sent, and rp_cnp_handled is zero or near it. The sending side is not reacting — the reaction point is disabled, or the CNPs are being lost or misclassified on the return path. CNPs themselves are typically marked with a different DSCP and should be in a high-priority, non-paused class; if they are queued behind the congestion they are meant to relieve, the control loop does not close.

There are also hardware counters on the RDMA device side, which are per-device rather than per-netdev and sometimes easier to correlate with a specific job:

grep . /sys/class/infiniband/mlx5_0/ports/1/hw_counters/* | \
  grep -E 'out_of_buffer|out_of_sequence|packet_seq_err|local_ack_timeout_err|rnr_nak|np_cnp|rp_cnp|roce_adp_retrans'

packet_seq_err and out_of_sequence are the ones to watch. On a fabric that is genuinely lossless they stay near zero. On Spectrum-X with packet-level adaptive routing, out-of-order arrival is expected and handled, so interpret these against the platform’s own guidance rather than against an InfiniBand intuition.

MTU

This is a small thing that costs a lot of bandwidth.

RoCE selects a path MTU from the standard set — 256, 512, 1024, 2048 or 4096 bytes — and it cannot exceed what the underlying Ethernet MTU can carry once headers are accounted for. If any hop in the path is at the default 1500, the RoCE MTU drops to 1024, and you lose a meaningful fraction of achievable bandwidth to per-packet overhead and processing.

The requirement is jumbo frames configured consistently on every NIC and every switch port in the path, typically 9000. One port left at 1500 — a newly added uplink, a replaced switch, a host that did not get the configuration — is enough.

Confirm what was negotiated rather than what was configured:

ib_write_bw -d mlx5_0 -x 3 -F -a --report_gbits <server>   # header prints Mtu
ip link show enp1s0f0 | grep mtu

The -x 3 selects a GID index, which on RoCE selects the RoCEv2 IPv4 or IPv6 entry. Get it from:

show_gids

Picking the wrong GID index is how people accidentally benchmark RoCEv1, or run over link-local IPv6 when they meant to use the routed IPv4 address.

What transfers between the two worlds

Transfers cleanly: the MPI layer entirely — Open MPI, MPICH, UCX, PMIx, the OSU benchmarks, your application. perftest works on both with the addition of a GID index. NCCL works on both, and refers to RoCE devices through the same NCCL_IB_HCA variable, which confuses people into thinking they have InfiniBand. The layered testing order from the first article in this series applies without modification.

Does not transfer: ibstat, iblinkinfo, ibdiagnet, ibnetdiscover, perfquery, opensm and everything built on LIDs and the subnet manager. There is no routing engine to choose and no LID budget to compute. In their place you have per-priority counters, switch telemetry, LLDP for topology discovery, and whatever your switch vendor’s fabric management provides.

Transfers, but means something different: latency figures. RoCE adds UDP/IP encapsulation and the congestion-control loop, and end-to-end small-message latency is generally higher than InfiniBand at an equivalent rate. How much depends on the NIC generation and the switch, so measure it with ib_send_lat on your own hardware rather than accepting a figure. The gap has narrowed considerably across recent generations; it has not closed.

Transfers, but with a different failure mode: congestion. On InfiniBand, credit-based flow control makes the fabric lossless without configuration, and a congested fabric slows down. On Ethernet, an incorrectly configured fabric drops, and RoCE recovery from drops is expensive enough that the performance cliff is steep rather than gradual. That difference — gradual degradation versus a cliff — is the single most important thing to internalize when moving MPI workloads from one to the other.

A minimum acceptance check on the Ethernet side

Before believing any MPI number on a RoCE fabric:

mlnx_qos -i <dev> on every node — trust mode and PFC priority identical everywhere.
ecn/roce_np/enable/<prio> and ecn/roce_rp/enable/<prio> set on every node.
MTU 9000 confirmed on every host interface and every switch port in the path, and RoCE MTU 4096 confirmed in the ib_write_bw header.
show_gids — the GID index in use is the RoCEv2 entry you intended.
Clear counters, run a full-scale alltoall, then check every node for buf_discard on the RoCE priority and rp_cnp_handled moving. Any discard on the lossless class is a stop-and-fix.

That takes half an hour and it is the difference between a fabric that performs predictably and one that performs well on Tuesday.

Multi-rail MPI: making the second adapter carry its share

2026-09-08T09:00:00-04:00

Dual-rail hosts are now the default on anything built for AI and common on general HPC nodes. Two adapters, two cables, two ports on the leaf, and a reasonable expectation that a bandwidth-bound job goes twice as fast.

It rarely does, and the gap between what was bought and what arrives is attributed to fabric overhead far more often than it deserves. In practice, the second rail underdelivers for one of a short list of reasons, most of which are inside the node and all of which are measurable.

First: is the second rail carrying anything at all?

Before tuning anything, establish whether the second adapter is being used. This is not a rhetorical question — a substantial fraction of dual-rail nodes in production are running every byte over one rail, and nothing reports it.

The direct method is the port counters. Clear, run, read:

for d in mlx5_0 mlx5_1; do
  echo -n "$d before: "
  cat /sys/class/infiniband/$d/ports/1/counters/port_xmit_data
done

mpirun -np 2 -map-by ppr:1:node ./osu_bw -m 8388608:8388608

for d in mlx5_0 mlx5_1; do
  echo -n "$d after: "
  cat /sys/class/infiniband/$d/ports/1/counters/port_xmit_data
done

If mlx5_1 did not move, you have a single-rail cluster with a spare adapter in it. That is a two-minute check and it settles the question that the rest of the work depends on.

Do this check before any tuning conversation. A node whose second adapter has never transmitted a byte is not a multi-rail tuning problem, and the discussion about striping thresholds and rendezvous rails that usually follows is entirely beside the point. The counter delta settles it in two minutes.

The second method is to ask UCX what it selected:

mpirun -np 2 -map-by ppr:1:node \
  -x UCX_LOG_LEVEL=info \
  ./osu_bw 2>&1 | grep -iE 'selected|device|transport|rndv'

Recent UCX releases also expose a protocol selection table that is considerably more readable than the log:

mpirun -np 2 -x UCX_PROTO_INFO=y ./osu_bw

It prints, per message-size range, which protocol and which devices UCX will use — including whether a large rendezvous transfer is being striped across multiple devices or sent over one. That table answers the multi-rail question directly rather than by inference.

Telling MPI which devices to use

UCX enumerates devices itself and its default selection is often not what you want, particularly on a node that also has a management NIC, a storage NIC, or an adapter whose port is administratively down.

ucx_info -d | grep -E '^#.*Transport|Device:'
ucx_info -v

The device list to hand to MPI is explicit and includes the port number:

mpirun -np 16 -map-by ppr:8:node --bind-to core \
  --mca pml ucx --mca osc ucx \
  -x UCX_NET_DEVICES=mlx5_0:1,mlx5_1:1 \
  -x UCX_TLS=rc_x,sm,self \
  ./osu_bw

Three parameters do most of the work:

UCX_NET_DEVICES — the allowlist. Naming devices explicitly is better practice than relying on discovery, because discovery changes when a port goes down or a card is added.
UCX_MAX_RNDV_RAILS — how many devices a single large (rendezvous) transfer may be striped across. The default has historically been 2. If you have four rails and want them all used by one transfer, this is the setting that is quietly capping you.
UCX_MAX_EAGER_RAILS — the same idea for the eager protocol used by smaller messages.

Confirm the values in effect rather than assuming the defaults:

ucx_info -f -c | grep -E 'MAX_RNDV_RAILS|MAX_EAGER_RAILS|NET_DEVICES|TLS'

Why the second rail underdelivers

Work through these in order. They are roughly ordered by how often they are the answer.

The transfer is not big enough to stripe. Rail striping applies to the rendezvous protocol. Messages below the rendezvous threshold go over one device. An application whose messages are all 64 KB will not benefit from a second rail no matter how it is configured, and that is a correct outcome rather than a misconfiguration. Check where your application’s message sizes sit before concluding anything.

Both adapters are on the same NUMA node. Two cards on one socket share that socket’s PCIe root complex, its memory bandwidth, and its inter-socket link for any traffic sourced from the other socket. Check:

for d in /sys/class/infiniband/mlx5_*; do
  echo "$(basename $d): numa_node=$(cat $d/device/numa_node)"
done
lstopo --output-format txt
nvidia-smi topo -m     # on GPU nodes: also shows GPU-to-HCA affinity

nvidia-smi topo -m is the one to read on a GPU node. PIX means the GPU and the adapter sit under the same PCIe switch, which is the case GPUDirect RDMA is designed for. NODE means same NUMA node but traversing the root complex. SYS means the traffic crosses the inter-socket link, and on that pairing the second rail is a good deal less than a second rail.

The ranks are not bound to the rail that is local to them. This is the common and fixable case. If every rank uses both rails, half of every node’s traffic crosses the inter-socket link. The better arrangement on a two-socket, two-rail node is that ranks on socket 0 use mlx5_0 and ranks on socket 1 use mlx5_1, with no striping at all. That is not multi-rail in the striping sense; it is rail affinity, and it usually outperforms naive striping.

A small wrapper does it, driven by the local rank the launcher exports:

#!/bin/bash
# rail-bind.sh - one rail per socket, selected by node-local rank
local_rank=${OMPI_COMM_WORLD_LOCAL_RANK:-${SLURM_LOCALID:-0}}
half=$(( OMPI_COMM_WORLD_LOCAL_SIZE / 2 ))
if [ "$local_rank" -lt "$half" ]; then
  export UCX_NET_DEVICES=mlx5_0:1
else
  export UCX_NET_DEVICES=mlx5_1:1
fi
exec "$@"

mpirun -np 32 -map-by ppr:8:numa --bind-to core ./rail-bind.sh ./osu_mbw_mr

Measure both arrangements. Which wins depends on the message-size distribution and on whether the application is bandwidth- or latency-bound, and it is not predictable from the topology alone.

The PCIe slots cannot carry it. Two NDR400 adapters is 100 GB/s of offered load in each direction. Confirm both slots trained to their rated speed and width, and confirm the platform is not sharing lanes between them:

lspci -vv | grep -A1 Mellanox | grep -E 'LnkCap|LnkSta'

Both rails land on the same leaf switch and the same uplinks. This is the fabric-side case, and it is why some sites deliberately build the second rail as a separate subnet with its own switches and its own subnet manager. If both rails leave the node, arrive at the same leaf, and take the same uplinks to the spine, the second rail doubles the host’s egress capacity and does nothing for the fabric’s ability to carry it. Check where each rail’s port actually lands:

ibnetdiscover | grep -A2 "$(ibstat mlx5_0 | awk '/Port GUID/{print $3}')"
iblinkinfo -l | grep -iE 'node01'

The application is not bandwidth-bound. A latency-bound code gains nothing from a second rail and may lose a little to the extra selection logic. This is the outcome nobody wants to hear and it is frequently the correct one. It is also the argument for measuring with osu_mbw_mr at your application’s real message sizes rather than with a large-message osu_bw that flatters the configuration.

Proving RDMA is engaged, not TCP over IPoIB

Silent fall back to TCP over IPoIB is the single most consequential quiet failure in this area. Everything works. The job completes. The bandwidth is roughly a tenth of what it should be and the latency is roughly ten times what it should be, and both of those are within the range of “the code doesn’t scale well,” which is what it gets recorded as.

There are three ways to establish the truth, in increasing order of how conclusive they are.

Read the counters. As above: if port_xmit_data on the InfiniBand device does not move during the run, the traffic is not on the InfiniBand transport. Corroborate on the IPoIB side, where it will have gone instead:

cat /sys/class/net/ib0/statistics/tx_bytes

Read the UCX transport selection. UCX_PROTO_INFO=y or UCX_LOG_LEVEL=info will name the transport. Seeing tcp where you expected rc_x or dc_x is the finding.

Remove the fallback and see if it still runs. This is the conclusive test, and it takes one flag:

mpirun -np 2 -x UCX_TLS=rc_x,sm,self ./osu_bw

That list contains no TCP transport. If the job runs, RDMA was working. If it fails to establish connections, TCP was carrying the traffic and now you know. Run this as a gate in your acceptance suite; it converts a silent degradation into a loud failure, which is the whole objective.

The usual root causes, once you have established fallback is happening:

Cause	Check
Locked memory limit not raised	`ulimit -l` inside the job, not on the login node
Device down or not present on some nodes	`ibstat` across the whole allocation
`UCX_NET_DEVICES` naming a device that does not exist there	Node-by-node device enumeration
Node with a different OFED or missing `rdma-core`	`ofed_info -s` fleet-wide
Firewall or SELinux policy on a subset of nodes	Compare a working and a failing node

ulimit -l deserves emphasis. Memory registration for RDMA requires locked pages, and if the limit is not unlimited inside the job’s environment, UCX cannot register buffers and falls back. The limit set in /etc/security/limits.d applies to login sessions; what matters is the limit inside the scheduler’s launched process, which can differ. Check it from inside a job:

srun -N 2 bash -c 'echo $(hostname) $(ulimit -l)'

Rail-optimized topologies

On AI clusters the multi-rail arrangement is often not “two rails to the same fabric” but a rail-optimized design: each GPU has an adapter, and adapter k on every node connects to switch k. Same-index GPUs across nodes then communicate within a single switch, which is what makes large all-reduces efficient.

Two consequences for MPI. First, the affinity between rank, GPU and adapter is not a tuning preference, it is the design, and getting it wrong sends traffic across the fabric that was supposed to stay in one switch. Second, if the rails are separate subnets, each has its own subnet manager and its own LID space, and tooling that assumes one fabric will report on one of them and silently ignore the rest.

Verify the mapping is what the cabling intended before benchmarking anything:

nvidia-smi topo -m
for d in /sys/class/infiniband/mlx5_*; do
  n=$(basename $d)
  printf '%s numa=%s lid=%s sm_lid=%s\n' "$n" \
    "$(cat $d/device/numa_node)" \
    "$(ibstat $n 1 | awk '/Base lid/{print $3}')" \
    "$(ibstat $n 1 | awk '/SM lid/{print $3}')"
done

What to measure, and in what order

ib_write_bw -d mlx5_0 and -d mlx5_1 separately. Each rail alone should reach the same number. If one is lower, stop — that is a link or slot problem and it is not a multi-rail question.
Both rails simultaneously, as two concurrent ib_write_bw processes pinned to the correct sockets. This is the host’s true aggregate ceiling and it is the number multi-rail MPI is trying to reach.
osu_bw with one rail, then with both. Compare against step 2.
osu_mbw_mr at your application’s message sizes, one rail versus both, with and without rail affinity.

Step 2 is the one that gets skipped, and it is the one that tells you whether the shortfall is in the host or in MPI. If two concurrent ib_write_bw processes cannot reach twice a single rail, no MPI configuration will, and you have a PCIe, NUMA or memory-bandwidth investigation rather than a UCX one.

Fan-out tests: what one-to-many finds that point-to-point cannot

2026-09-05T14:00:00-04:00

A point-to-point bandwidth test between two nodes exercises one path. Under static routing — which is what InfiniBand does by default — that path is fixed by the subnet manager’s forwarding tables, and one flow on one path is indifferent to how well or badly those tables distribute traffic overall. You can have a routing engine that has collapsed eight parallel inter-switch links onto three, and osu_bw between a well-chosen pair will report a perfect number, every time, reproducibly.

That is the gap fan-out tests exist to close. The moment you put many simultaneous flows on the fabric, path selection stops being a detail and becomes the dominant term.

The three shapes

One-to-many. One node sends to N others concurrently. This finds the sender’s own ceiling — its adapter, its PCIe slot, its memory bandwidth — and it finds the first hop out of that node’s leaf switch. It is the right test for a storage node, a parameter server, or any rank-zero-heavy pattern.

Many-to-one. N nodes send to one. This is the incast pattern, and it is a different test, not a mirror image. It is where switch buffering, flow control and congestion control get exercised, and it is where a fabric that looks fine under one-to-many falls apart.

Many-to-many. Every node to every node. This is what an alltoall does, and it is the test that actually measures the fabric rather than any endpoint.

Run all three. A cluster that passes one-to-many and fails many-to-many has a fabric problem. A cluster that fails all three at the same node has a node problem. That distinction is available in about twenty minutes and is otherwise the subject of a long argument.

osu_mbw_mr, and why message rate is a separate ceiling

osu_mbw_mr is the multiple-bandwidth / message-rate test. It takes 2N ranks, makes the first N senders and the second N receivers, pairs them up, and runs all pairs concurrently. It reports aggregate bandwidth and aggregate message rate for each message size.

# 16 ranks per node, one node sending, one node receiving
mpirun -np 32 -map-by ppr:16:node --bind-to core \
  -x UCX_NET_DEVICES=mlx5_0:1 \
  ./osu_mbw_mr

Two things make it worth running that a pair test is not.

First, it is the only cheap way to find a per-node ceiling that is below the per-link rate. A single pair of ranks frequently cannot saturate a modern adapter — one rank, one queue pair, one core is not enough to fill NDR400. If you conclude from a single-pair osu_bw that the link is at half rate, you may simply be measuring one core. Increase the concurrency and the number moves. If it does not move, the ceiling is real and it is in the node.

Second, the message rate column is a ceiling that bandwidth never reveals. Adapters have a maximum packets-per-second they can process, and applications with many small messages hit that ceiling while the bandwidth graph looks comfortable. If the message rate flattens as you add pairs while bandwidth is nowhere near the link rate, the constraint is packet processing, not bytes, and no amount of fabric work changes it. The fixes live elsewhere: message aggregation, larger buffers, or a different communication pattern.

Sweep the concurrency deliberately rather than accepting the default:

for ppn in 1 2 4 8 16 32; do
  echo "== ppn=$ppn"
  mpirun -np $((ppn*2)) -map-by ppr:${ppn}:node --bind-to core ./osu_mbw_mr
done

The shape of that sweep is the useful output. Bandwidth rises and plateaus; where it plateaus is your real per-node number. Message rate rises and plateaus somewhere else entirely.

Where fan-out exposes the fabric

Here is the mechanism, because it is the part that is worth understanding rather than memorizing.

Under static routing, the subnet manager assigns each destination LID an output port on every switch. Every flow to a given destination therefore leaves a given switch through the same port, regardless of how busy that port is. On a leaf-and-spine fabric with, say, eight uplinks from a leaf to the spine tier, the routing engine distributes destination LIDs across those eight uplinks — and how evenly it does that is the whole question.

With one flow, you use one uplink and it is fine. With sixty-four concurrent flows, if the distribution assigned twenty of them to one uplink and two to another, that first uplink is oversubscribed by a factor of ten and the other is idle. The aggregate number you measure is set by the busiest uplink, not by the total capacity.

This is not hypothetical and it is not rare. It is the normal consequence of a routing engine that declined the topology and fell back to something simpler, of a updn configuration with no root GUID list, or of a fabric whose leaf uplink counts are not uniform. The article on OpenSM routing engines in this series covers how to confirm which engine is actually running.

Measuring the distribution directly

Do not infer it from bandwidth. Read the switch counters.

# baseline
ibclearerrors
for lid in $SPINE_LIDS; do
  perfquery -x $lid   # extended (64-bit) counters, all ports
done > /var/tmp/counters-before.txt

mpirun -np 512 -map-by ppr:8:node ./osu_alltoall -m 1048576:1048576

for lid in $SPINE_LIDS; do perfquery -x $lid; done > /var/tmp/counters-after.txt

PortXmitData is in 32-bit words on most implementations, so multiply by four for bytes; what you care about is the ratio between uplinks, not the absolute value. Take the delta per port and look at the spread. An engine distributing well produces uplink deltas within a few percent of each other. An engine that is not produces a spread you can see without a calculator, and that spread is your missing bandwidth.

PortXmitWait on the same ports is the corroborating evidence. It counts ticks during which the port had data queued and no credit to transmit. Large and uneven PortXmitWait across parallel uplinks is congestion concentrated on a subset of links, which is exactly the signature of a routing imbalance.

Use the 64-bit extended counters (perfquery -x, or perfquery -x -a). The legacy 32-bit PortXmitData counter wraps in seconds at modern link rates, and a wrapped counter produces a delta that is not merely wrong but arbitrarily wrong — sometimes negative, sometimes plausible. This has cost people real conclusions.

Reading the knee

Fan-out results are curves, and the useful information is in where the curve bends rather than in any single point.

Run the same collective at a fixed message size across a doubling sequence of node counts, holding ranks-per-node constant:

for n in 2 4 8 16 32 64; do
  srun -N $n --ntasks-per-node=8 ./osu_allreduce -m 4194304:4194304
done

For allreduce, the expected shape depends on the algorithm. NCCL-style and MPI ring implementations move 2(n-1)/n of the buffer per rank, which approaches a constant as n grows, so time at a large fixed message size should flatten out rather than grow. Tree and recursive-doubling implementations trade that for a latency term that grows as log(n), which dominates at small message sizes. So: at large messages you expect a plateau, at small messages you expect a gentle logarithmic rise. Anything that rises linearly with node count is not the algorithm — it is the fabric or a straggler.

For alltoall, every rank sends to every other rank, so the total bytes crossing the fabric grows as n² while the bisection capacity of a non-blocking fat tree grows as n. On a genuinely non-blocking fabric, per-rank alltoall time at a fixed per-peer message size should stay roughly flat. The node count where it stops being flat is a measurement of where your fabric stops being non-blocking, and that is a genuinely useful number to own.

Three distinct knee shapes and what each means:

Shape	Reading
Flat, then a step at a specific node count	You crossed a topology boundary — filled a leaf, started using spine uplinks, or crossed into a second island
Flat, then a smooth linear rise	Oversubscription. The fabric’s aggregate capacity is now the limit
Noisy from the start, no clean curve	Not a scaling result at all. Something is varying between runs — go find it before drawing any curve

That third row is the common one and it is worth taking seriously. If you cannot reproduce a point to within a few percent across three runs, you do not yet have a measurement, and fitting a curve through it produces a confident wrong conclusion.

Controlling for allocation

A scaling curve is only comparable if the node set is comparable. Two runs at 32 nodes that land on different parts of the fabric are two different experiments.

Pin the placement and record it:

srun -N 32 --ntasks-per-node=8 --nodelist=$(cat nodes.32) ./osu_alltoall
scontrol show hostnames $SLURM_JOB_NODELIST > run-$(date +%s).nodes

If your scheduler has topology awareness configured, use it — --switches=1 asks Slurm for an allocation within a single switch, and the difference between a within-leaf run and a cross-spine run at the same node count is itself a measurement worth taking.

And record the node list with every result. The single most common reason a benchmark “regressed” is that it ran somewhere else.

The one-slow-node problem

A collective completes when its slowest participant completes. That makes fan-out tests exquisitely sensitive to a single degraded node — which is useful, and also dangerous, because the symptom presents as a fabric-wide problem.

The signature: results that vary run to run in a way that correlates with the allocation rather than with anything you changed. Some 32-node runs are fine and some are 30 percent down, and which is which tracks whether a particular node was included.

The bisection method finds it quickly. Split the node set in half, run both halves, keep the slow half, repeat. Six rounds covers 64 nodes. It is crude and it is faster than reasoning about it.

The systematic alternative is a pairwize sweep — every node against a fixed reference node, using ib_write_bw rather than MPI so you are testing one layer at a time:

for n in $(scontrol show hostnames $SLURM_JOB_NODELIST); do
  ssh $n "ib_write_bw -d mlx5_0 -F -s 1048576 -D 5 --report_gbits $REF" \
    | awk -v n=$n '/1048576/ {print n, $4}'
done | sort -k2 -n | head

The bottom of that sorted list is your answer, and it is usually one node with a narrow link, a downtrained slot, or a transceiver on its way out. It is worth running this on a schedule rather than during an incident: a node that degrades quietly poisons every synchronized job it touches, and the cost of finding out late is measured in withdrawn results, not in wasted cycles.

What to keep

Per fabric, keep: the osu_mbw_mr bandwidth and message-rate plateaus with the concurrency at which each was reached; the alltoall and allreduce curves across your real node-count range with the node lists attached; and the per-uplink counter deltas from one representative alltoall.

The counter deltas are the item people omit and the one that ages best. Bandwidth numbers move when hardware changes. A routing distribution that was even in January and is lopsided in June is a change in the fabric, and having the January baseline is the difference between knowing that and arguing about it.

A layered order for MPI testing, and what each layer eliminates

2026-09-05T09:00:00-04:00

An MPI job that runs slower than expected is a question with about nine plausible answers, spread across four layers of stack that were installed by different people at different times. The reason these investigations run long is almost never that the answer is subtle. It is that the search was unordered.

The discipline that fixes it is simple to state. Each layer has a performance ceiling determined by the layer below it. Measure the bottom first, establish what the ceiling is, and only then ask whether the layer above is achieving it. A number at layer four means nothing until you know what layers one through three can deliver.

The layers

Layer	Tool	What a clean result eliminates
0. Inventory	`ibv_devinfo`, `ofed_info`, `flint`, `lspci`	Version skew, downtrained slots, wrong adapter
1. Link	`ibstat`, `iblinkinfo`, `mlxlink`, `perfquery`	Width, rate, cabling, physical errors
2. Verbs	`ib_write_bw`, `ib_read_bw`, `ib_send_lat`	The entire MPI and UCX stack
3. MPI point-to-point	`osu_latency`, `osu_bw`, `osu_bibw`, `osu_mbw_mr`	Transport selection, protocol thresholds, pinning
4. Collectives	`osu_allreduce`, `osu_alltoall`, `osu_barrier`	Routing distribution, jitter, one slow rank
5. Application	The application	Nothing — it is what you were asked about

Nothing here is novel. The value is entirely in refusing to skip.

Layer 0: inventory before measurement

Half the MPI investigations that get escalated are version skew, and skew is free to check before anything is measured. You want, across every node in the allocation: the same OFED or DOCA-OFED release, the same adapter firmware, the same MPI build, and PCIe slots that trained to their rated speed and width.

pdsh -w node[01-64] 'ofed_info -s; ibv_devinfo -d mlx5_0 | grep -E "fw_ver|board_id"' \
  | dshbak -c

board_id is the PSID, and it identifies the adapter model and OEM variant. Two adapters with the same marketing name and different PSIDs are different parts, take different firmware images, and do not necessarily behave alike.

Then the slot. This is the check people skip, and it is the one that most often turns a two-week investigation into a two-minute one:

lspci -s $(cat /sys/class/infiniband/mlx5_0/device/uevent | grep PCI_SLOT_NAME | cut -d= -f2) -vv \
  | grep -E 'LnkCap:|LnkSta:'

LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM not supported
LnkSta:	Speed 16GT/s, Width x8, TrErr- Train- SlotClk+ DLActive+

That adapter is capable of PCIe Gen5 x16 and is running at Gen4 x8. Nothing will report an error. The link will come up at its full InfiniBand rate. The host bus will cap you at roughly a quarter of what the card can do, and every measurement above this layer will be consistent, reproducible and wrong.

The arithmetic is worth internalizing because it recurs across generations. A PCIe Gen3 x16 slot delivers usefully under 13 GB/s after encoding and protocol overhead, which is approximately one HDR100 link. Put an HDR200 adapter in one and you have bought an HDR100 adapter. Gen4 x16 is roughly the right size for HDR200 or NDR200. NDR400 wants Gen5 x16. XDR at 800 Gb/s is 100 GB/s in each direction, which is more than a Gen5 x16 slot can carry, so on XDR hardware the slot generation is not a footnote — check it explicitly and check what the platform actually populated, not what the chassis datasheet says is possible.

Layer 1: the link

Two questions only: is every port at the width and rate you paid for, and is any port accumulating errors.

ibstat mlx5_0 1

Port 1:
	State:            Active
	Physical state:   LinkUp
	Rate:             200
	Base lid:         91
	LMC:              0
	SM lid:           1
	Link layer:       InfiniBand

Rate: 200 here is the aggregate link rate in Gb/s. Note that for the same link, iblinkinfo reports the per-lane rate and the width separately, and conflating the two is a genuine and expensive misread:

  91    1[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  12   9[  ] "leaf01" ( )

That is four lanes at 53.125 Gb/s each — an HDR200 link, not a 53 Gb/s link. On NDR the same field reads 106.25 Gbps and the link is NDR400. Someone reading that column as the link rate concludes their NDR fabric is running at 100 Gb/s and opens a support case.

The scan that matters is not any single port but the whole allocation at once, looking for anything that is not at full width:

iblinkinfo -l | grep -vE '4X *(25\.78125|53\.125|106\.25|212\.5)' 

A port that negotiated 1X or 2X is Active, passes traffic, answers ping, and runs your job at a fraction of the speed. It will not appear in any error log.

For physical link quality on modern adapters, mlxlink is more informative than the counters:

mlxlink -d mlx5_0 -p 1 -m -c --show_fec --show_eye

It reports the negotiated speed, the FEC mode, raw and effective BER, and the transceiver’s vendor and part number. A link that is up but marginal shows as a raw BER several orders of magnitude worse than its neighbours while the effective BER stays clean, because FEC is absorbing it. That link works until the day it is under load, and then it is a fabric-wide mystery.

Finally, clear the counters, run something, and read them back. Counters that have been accumulating since the last power cycle tell you nothing about today:

ibclearerrors
# ... run a representative workload ...
ibqueryerrors -s PortXmitWait,PortRcvErrors,SymbolErrorCounter,LinkDownedCounter

PortXmitWait is not an error. It counts ticks where a port had data to send and no credit to send it, which is congestion, and it belongs to layer four rather than here. Note it and move on.

Layer 2: verbs, without MPI in the picture

This is the layer people skip, and skipping it is why so many fabric problems get diagnosed as MPI problems. perftest talks to the adapter directly. If ib_write_bw is slow, no MPI tuning parameter is going to help you, and you have just eliminated UCX, PMIx, Open MPI, process binding and the application in one command.

# server
ib_write_bw -d mlx5_0 -i 1 -F -a --report_gbits
# client
ib_write_bw -d mlx5_0 -i 1 -F -a --report_gbits node02

-F suppresses the CPU-frequency warning on machines with a scaling governor, -a sweeps all message sizes, -i 1 selects the physical port. Add -x <n> to select a GID index when you are on RoCE rather than InfiniBand.

Read the header before the numbers. It tells you the MTU that was negotiated, the transport, and the connection type:

 Number of qps   : 1        Transport type : IB
 Connection type : RC       Using SRQ      : OFF
 Mtu             : 4096[B]
 Link type       : IB

An MTU of 1024 where you expected 4096 is a finding in itself, and on RoCE it usually means the network MTU is not jumbo end to end.

Three runs, in this order, on the same pair:

ib_write_bw — one-directional RDMA write. The cleanest bandwidth number the hardware can produce.
ib_read_bw — RDMA read. Reads are round-trip and depend on how many outstanding reads the adapter allows, so this is normally lower than write and a good deal more sensitive to the fabric.
ib_send_lat -a — send/receive latency across sizes. Sub-microsecond at small sizes on a single-hop modern fabric; each additional switch hop adds a well-defined increment you can measure directly by comparing an intra-leaf pair against a cross-spine pair.

What “good” is here is a property of your hardware, not something to assert from an article. The check you can do without a reference number is arithmetic: take the signalling rate from layer one, subtract the line encoding overhead, and see whether ib_write_bw at large message sizes lands close to it. If it lands at half, look at the width and the PCIe slot before looking anywhere else.

Run layer two between several different node pairs, chosen deliberately: two nodes on the same leaf, two nodes on different leaves, and the two nodes furthest apart in the topology. One pair tells you almost nothing. Three pairs chosen by topology tell you whether the problem is a node, a link, or the core of the fabric.

Layer 3: MPI point-to-point

Now, and only now, introduce MPI. If layer two was clean and layer three is not, the problem is in the MPI stack, and that is a much smaller search space than “somewhere in the cluster.”

mpirun -np 2 -map-by ppr:1:node --bind-to core \
  -x UCX_NET_DEVICES=mlx5_0:1 \
  ./osu_latency
mpirun -np 2 -map-by ppr:1:node --bind-to core \
  -x UCX_NET_DEVICES=mlx5_0:1 \
  ./osu_bw

Compare osu_bw against the ib_write_bw number from the same pair. A gap of a few percent is the MPI protocol overhead and is expected. A gap of a factor of several means MPI is not using the transport you think it is, and the single most common reason is a silent fall back to TCP over IPoIB.

osu_bibw runs traffic in both directions at once and is the test that finds half-duplex-like behavior and PCIe bottlenecks, because it is the first test that asks the host bus for full bandwidth in both directions simultaneously. A machine that does well on osu_bw and poorly on osu_bibw is usually telling you about its slot, not its fabric.

osu_mbw_mr is the one that belongs at the end of this layer. It runs multiple concurrent pairs and reports both aggregate bandwidth and message rate. It is the bridge into layer four, and it is where a per-node ceiling that no single-pair test can see becomes visible.

Pinning is not optional at this layer. An unpinned rank measures the scheduler. Confirm the adapter’s NUMA affinity and place accordingly:

cat /sys/class/infiniband/mlx5_0/device/numa_node
lstopo --output-format txt
mpirun --report-bindings -np 2 ...

--report-bindings prints where each rank actually landed. Read it. Do not assume the binding you asked for is the binding you got, particularly under a scheduler that has its own cgroup opinions.

Layer 4: collectives

Collectives are where fabric problems that point-to-point cannot see finally appear, because they are the first thing that puts many flows on the fabric at once and the first thing that is gated by its slowest participant.

mpirun -np 512 -map-by ppr:8:node --bind-to core ./osu_alltoall
mpirun -np 512 -map-by ppr:8:node --bind-to core ./osu_allreduce
mpirun -np 512 -map-by ppr:8:node --bind-to core ./osu_barrier

A collective that degrades while layers one through three stayed clean is telling you about routing distribution, congestion, or jitter — never about the link. That distinction is the whole reason to have run the lower layers first. Fan-out behavior and how to read the knee in the scaling curve is a large enough topic that it has its own article in this series.

Layer 5: the application

By the time you get here you know the ceiling at every layer beneath, which means an application number can finally be interpreted. If the application is at sixty percent of what layer four delivers, that is an application question: message sizes, communication/computation overlap, decomposition. If it is at five percent, something below is still wrong and the earlier layers were not tested carefully enough.

What testing out of order actually costs

The failure mode is not that you fail to find the problem. It is that you find a problem, and it is real, and it is not the one that is hurting you.

Start at the application, and its message sizes will look suboptimal, because application message sizes usually are. You will spend a week on the decomposition. The 1X link stays there the whole time. Start at the collectives, and you will conclude the routing engine is badly distributed, which it may well be, and you will schedule a maintenance window to change it while the downtrained PCIe slot continues to cap the node.

Both of those are true findings. Neither is the constraint. The ordering exists because the only way to know whether a finding is the constraint is to know the ceiling underneath it.

Recording it

The output of this exercise is not a diagnosis, it is a baseline. Write down, in version control, for a named node pair and a named node set: the OFED and firmware versions, the PCIe link status, the ib_write_bw and ib_read_bw plateaus, the osu_latency small-message number, and the collective curves at the job sizes you actually run.

That file is what converts the next incident from an investigation into a diff. It takes half a day to produce on a machine that is known good, and there is no other moment when producing it is as cheap.

Mounting GPFS from a separate client cluster, end to end

2026-09-04T10:00:00-04:00

A GPFS filesystem does not have to be mounted by the cluster that owns it. The owning cluster exports it, a separate client cluster mounts it, and the two are administered independently. This is the multicluster or remote mount model, and for most sites with distinct storage and compute hardware it should be the default.

The mechanics are four commands. What follows is the reasoning for the model, the full sequence, and the failures that appear later — including one that will kill every running job on a node without logging anything that points at the cause.

Why separate the clusters at all

Blast radius. A client cluster can be restarted, upgraded, reinstalled or broken without touching the storage cluster. Where compute and storage share a cluster, every compute-side action is a storage-side risk.

Version independence. The two clusters run their own minimum release levels. You can upgrade compute nodes without moving the storage cluster, which matters because the storage cluster is the one you least want to touch.

Quorum separation. Compute nodes come and go. Storage nodes must not. Sharing a cluster means the quorum design has to tolerate the churn of the noisiest half.

Administrative boundary. The storage team owns the storage cluster’s configuration. A compute-side administrator adding a node cannot alter the storage cluster’s quorum, licensing or tunables by accident.

The cost is one more cluster to manage and an authentication relationship to maintain. It is worth it.

Create the client cluster

Install the same GPFS packages, build the portability layer, then create a cluster containing only the clients:

mmcrcluster -N clientnodes -r /usr/bin/ssh -R /usr/bin/scp -C compute-cluster
mmchlicense client --accept -N all

Note the license designation. Nodes that only mount a filesystem take a client license; nodes that serve NSDs take a server license. Getting this wrong is a compliance problem rather than a technical one, but it is easier to set correctly than to correct in an audit.

Give the client cluster its own quorum — three nodes, chosen for stability rather than convenience. Login nodes and infrastructure nodes make better quorum members than batch compute nodes that are rebooted between jobs.

Exchange keys

Each cluster generates a key and receives the other’s public half.

On the storage cluster:

mmauth genkey new
mmauth update . -l AUTHONLY

On the client cluster:

mmauth genkey new
mmauth update . -l AUTHONLY

Then copy each cluster’s public key to the other. On the storage cluster, register the client and grant access to the filesystem:

mmauth add compute-cluster.example -k /path/to/client_id_rsa.pub
mmauth grant compute-cluster.example -f fs1 -a rw

-a rw can be ro for a read-only export, which is worth using where it fits — an analysis cluster that only reads a curated dataset does not need write access to it.

Define the remote cluster and filesystem on the client

mmremotecluster add storage-cluster.example \
    -n storage01,storage02,storage03 \
    -k /path/to/storage_id_rsa.pub

mmremotefs add gpfs0 -f fs1 -C storage-cluster.example \
    -T /gpfs/gpfs0 -o rw,atime,mtime -A yes

The -n list is the contact nodes: the storage-cluster nodes the client will approach to find the filesystem. Give it several. A single contact node is a single point of failure for mounting, though not for I/O once mounted.

-A yes sets automount, so the filesystem mounts at daemon start.

Understand the local name and the remote name — they are not the same thing

This is the detail that causes the most confusion later, and it is also the feature that makes filesystem rebuilds survivable.

In mmremotefs add gpfs0 -f fs1, gpfs0 is the local device name on the client cluster and fs1 is the remote name on the storage cluster. They are independent. mmremotefs show displays the mapping:

Local Name  Remote Name  Cluster name              Mount Point
gpfs0       fs1          storage-cluster.example   /gpfs/gpfs0

The practical consequence: the storage cluster can rebuild and rename its filesystem without any client-side path changing. Build the replacement filesystem under a new name, migrate, then repoint the remote name. Clients keep their local device name and their mount point, so module paths, job scripts, provisioning references and automation all continue to resolve.

The trap that comes with it: anything that consumes the filesystem name rather than the mount point sees the new name. GPFS callbacks receive %fsName, and a callback guard written against the old name silently stops matching. Cron entries that name the device — snapshot scripts are the usual example — keep pointing at a device that no longer exists. Before any rename, grep your automation for the device name; after it, verify the callbacks still fire.

Configure the client network

Clients need verbsPorts set for their own adapters, and it is not inherited from anywhere useful:

mmchconfig verbsPorts="mlx5_0/1" -N clientnodes
mmchconfig verbsRdmaSend=yes

Then verify RDMA is actually carrying data, rather than assuming:

mmdiag --network | grep -E "by (TCP|RDMA) connection"
mmfsadm test verbs conn

A client silently falling back to TCP over IPoIB will work. It will be several times slower than it should be, it will not log an error, and the problem will be attributed to the storage cluster. Check this on new nodes as a matter of routine — particularly on any node added with mmaddnode, which does not inherit per-node overrides.

Size the client pagepool deliberately

The default is small — often 1 GiB — and on a compute node with hundreds of gigabytes of RAM that is usually wrong in one direction or the other.

Too small and you lose read caching and prefetch effectiveness on exactly the workloads that benefit most. Too large and you have taken memory away from jobs; on a node where the scheduler enforces memory limits, the pagepool is memory the scheduler does not know it has already lost.

A few GiB is a reasonable starting point for a general compute node. Match it to what the node is for, set it per node class rather than globally, and remember it only takes effect on daemon restart — a configured value with no restart is a value that is not running.

Bind mounts over GPFS: the hazard that kills jobs silently

Sites commonly bind-mount a directory inside GPFS onto a conventional path — a scratch area onto /scratch, a shared application tree onto a module path, a home directory cache onto /home. It is convenient and it introduces a specific, serious failure mode.

A process whose current working directory is under a bind mount does not follow that mount when it is re-established. Tear the bind down and put it back, and the process’s CWD is now invalid. getwd() starts returning NULL.

The consequences are not immediate and they do not point at the cause:

The job does not die at the moment of the remount. It dies at its next call that needs the working directory, which may be minutes later.
The error surfaces deep inside application code. In R, for instance, a library that saves and restores the working directory around a compile step fails with Error in setwd(cur) : character argument expected, because cur is NULL. The message names the application’s function. It reads unmistakably like a user bug.
Slurm marks the job FAILED rather than requeuing it, because from the scheduler’s point of view the job exited non-zero of its own accord.

A rolling restart of a bind-mount unit across a compute fleet, with jobs running, kills every job that has a working directory under that path. In one measured case a sequential loop across eighteen nodes produced a 100% mortality rate among the jobs it touched — every job spanning its node’s restart failed, none survived — while the naive correlation test (“did jobs fail within sixty seconds of the restart?”) found almost nothing and would have exonerated the change entirely. The correct test is whether the restart fell inside each job’s start-to-end window.

The rules that follow from this:

Drain the node before touching a bind mount. Treat it exactly as you would a reboot. scontrol update NodeName=<n> State=DRAIN, wait for the node to empty, make the change, verify with findmnt, resume.
Never loop across a fleet without a per-node gate. Restarts spaced a few seconds apart are the signature of a script with no check in it.
A bind restart is in one respect worse than a reboot. A reboot kills jobs visibly and the scheduler requeues them. A bind restart corrupts the state of surviving processes, which then fail later with an error that blames the user.
If a fleet-wide change genuinely cannot wait for an empty cluster, the honest options are a maintenance reservation or announcing the job losses. A silent rolling loop is neither.

Validate the client side

mmmount all -a
mmlsmount all -L                       # who has it mounted
mmdf gpfs0                             # capacity as the client sees it
mmdiag --network                       # RDMA in use, not TCP
findmnt -no SOURCE /gpfs/gpfs0         # the mount, not the unit state

Then test as a user, not as root. Many sites export storage through NFS with root-squash somewhere in the path, and a root test that succeeds proves less than a user test that fails.

Confirm on every class of node, not a sample. Head and login nodes frequently differ from compute nodes in ways nobody documented — a local directory shadowing a shared mount, a different module tree, an export that was never extended to the control plane. A configuration checked only on the head node is a configuration checked on the one machine that is least representative of where jobs run.

Day-2 failures worth recognizing

A node that mounts but performs badly. Check verbsPorts and RDMA state first; silent TCP fallback is the most common cause and the least visible.

A node that will not mount after rejoining. Per-node configuration — verbsPorts, pagepool, license designation — is not inherited by mmaddnode. Compare a working node against the new one line by line.

Automount not mounting. -A yes mounts at daemon start. If the daemon started before the network was ready, the mount fails and does not retry on its own. Ordering the GPFS unit after network-online.target avoids it.

Contact nodes all down. Clients cannot mount if none of the -n contact nodes answer, even when the rest of the storage cluster is healthy. Keep the list current: a contact node that was decommissioned two years ago is a mount failure waiting for a coincidence.

Stale mounts after a storage-side event. If the storage cluster force-unmounts, clients can be left with a mount point that exists but does not work. The GPFS mount callback described in the companion ECE guide is the reliable repair; systemd will not notice on its own.

The pattern underneath all of these

Every failure in this guide has the same shape. The command returned success. The configuration file says the right thing. The systemd unit reports active. And the running state is different.

The habit that prevents most of it is cheap: after any change, check the thing itself rather than the thing that describes it. findmnt rather than systemctl is-active. mmfsadm dump config rather than mmlsconfig. A job run as a user rather than a command run as root. mmdiag --network rather than the verbsPorts setting you just made.

LID budgets: what happens to addressing when the endpoint count changes

2026-09-04T09:00:00-04:00

A Local Identifier is the address a subnet manager assigns to a port so switches can forward to it. It is sixteen bits. The unicast range runs from 0x0001 to 0xBFFF, which is a little over forty-eight thousand addresses, with the remainder reserved for multicast.

Forty-eight thousand sounds like plenty, and on most fabrics it is. The number that actually constrains you is usually not the LID space but the forwarding table capacity of the switches, and both move when the endpoint count changes.

What consumes a LID

Each channel adapter port gets at least one. Each switch gets one for its management port. So a first approximation is: adapter ports, plus switches.

The multiplier is LMC.

LMC, and why it multiplies

LID Mask Control assigns each adapter port a contiguous block of 2^LMC addresses instead of a single one. The point is multipathing: with several LIDs pointing at the same port, the routing engine can compute different paths to each, and traffic between a pair of nodes can be spread across them.

LMC 0    1 LID per port
LMC 1    2 LIDs per port
LMC 2    4 LIDs per port
LMC 3    8 LIDs per port

Set it in opensm.conf:

lmc 0

The cost is linear in the multiplier and it applies to every port. Going from LMC 0 to LMC 2 does not add a few addresses; it quadruples consumption across the entire fabric, and it quadruples the number of entries every switch has to hold to reach those ports.

Leave LMC at 0 unless you have established that you need multipathing and that your routing engine will use it. Raising it is a fabric-wide cost paid to solve a problem many fabrics do not have.

The constraint that actually bites

Switch forwarding tables are the real limit. A linear forwarding table holds one entry per reachable LID, and its capacity is a property of the switch silicon, not of your configuration. When the number of LIDs in the subnet exceeds what a switch can hold, that switch cannot route to all of them.

This is why the LID budget is not simply “am I under 48,000.” It is “is every switch able to hold an entry for every LID in the subnet,” and the answer depends on the smallest table in the fabric. A mixed-generation fabric is governed by its oldest switch.

Check the capacity your switches actually report rather than assuming from the model number:

ibdiagnet -o /var/tmp/ibdiag
smpquery SI <lid>          # switch info, including linear FDB capacity
saquery SI

Counting before you change anything

The arithmetic is straightforward and worth doing on paper.

LIDs  =  (adapter ports × 2^LMC)  +  switches  +  headroom

Headroom matters because fabrics grow between maintenance windows, and because a fabric at ninety-nine percent of its forwarding capacity behaves fine until the day someone plugs in one more node.

Two changes push this number hard, and they often arrive together:

Splitting switch ports. Splitting does not itself consume LIDs — switches still take one each — but it exists to let you attach more endpoints, and those endpoints do. A split that doubles attachable ports is a plan to double the adapter port count.

Adding dual-port adapters, or turning on the second port. Each active port is an endpoint. A fleet where the second port was cabled but never brought up will double its LID consumption the day someone brings it up, and that is rarely recorded as a change to addressing.

Multiply the two together and a fabric that was comfortable becomes a fabric that is not, without anyone having made a decision about addressing.

What exhaustion looks like

It does not announce itself as an addressing problem. The symptoms are:

Nodes that enumerate in ibnetdiscover but are unreachable from some parts of the fabric and reachable from others, because the switches that ran out are the ones on those paths.
Subnet manager logs reporting failures to set forwarding table entries.
Errors during table distribution on a sweep, often on the largest switches first.
Intermittent, position-dependent reachability that looks like a cabling fault and survives recabling.

The tell is that it is topology-dependent rather than node-dependent. A broken adapter fails from everywhere. An exhausted table fails from behind that switch.

grep -iE 'lid|forwarding|fdb|table' /var/log/opensm.log | tail -60

Interaction with the routing engine

Routing engines use LMC differently, which matters when you are deciding whether to raise it.

ftree can make use of multiple LIDs per port to distribute traffic across parallel paths in a fat tree, and on a fabric that qualifies it is the case where raising LMC has a defensible payoff. updn derives less benefit, because the up/down constraint already limits which paths are legal. minhop does the least with it.

So the sequence is: confirm which engine is genuinely running, then decide about LMC. Raising LMC to enable multipathing under an engine that will not use it buys you four times the forwarding table pressure and nothing else.

Practical discipline

Record the LID count as a monitored number, not as something you look up during an incident. It is cheap to collect and it turns a class of confusing failures into a threshold you crossed.

Recompute the budget as part of planning any change to endpoint count — new nodes, split ports, second ports brought up, a new storage tier. The calculation takes a few minutes and the alternative is discovering the limit during a maintenance window with the change half applied.

And keep LMC deliberate. It is one line in a configuration file and it is the single largest multiplier on everything above.

Deploying a GPFS ECE cluster, end to end

2026-09-04T09:00:00-04:00

Erasure Code Edition (ECE) runs IBM Storage Scale’s declustered RAID in software, across the internal drives of ordinary servers. There is no controller, no external array, and no appliance. You get the ESS data protection model on hardware you specify yourself.

The installation itself is mechanical and IBM documents it well. What follows concentrates on the decisions that are cheap to make correctly on day one and painful to revisit later, and on the failures that are hard to diagnose because the system reports success while doing the wrong thing.

Decide whether ECE is the right answer at all

ECE earns its place when you want parallel-filesystem performance from servers you control, and you have a fast private network to spend on it. It is the wrong answer in three cases.

Too few nodes. The documented minimum is four servers per recovery group. Four is a number you can build, not a number you should run — see the fault tolerance arithmetic below. Six is a sensible floor and eight or more is where the model starts behaving the way the marketing describes.

Mixed hardware inside a recovery group. Every server in a recovery group should be the same model with the same drive count, drive model, CPU and memory. ECE distributes strips on the assumption that servers are interchangeable. They do not have to match across recovery groups, but within one they effectively do.

No dedicated fast network. ECE reads amplify. Delivering one byte to a client moves roughly 1.875 bytes on the storage fabric with an 8+2p code, because reconstructing a block gathers strips from every other server. If that traffic shares a congested network with everything else, you have built an expensive way to be slow.

Get the fault tolerance arithmetic right before you buy anything

This is the single decision most likely to be regretted, and it is made before a single package is installed.

Fault tolerance is not the p in the code name. A code writes a fixed number of strips, and those strips land on distinct servers. If the strip count exceeds the server count, some server necessarily holds two strips of the same stripe, and losing that one server costs you two strips.

Code	Strips	Servers	Strips per server	Node fault tolerance
8+2p	10	8	some hold 2	1
8+2p	10	11+	1	2
4+2p	6	8	1	2
4+3p	7	8	1	3

An 8+2p array on eight servers gives you node fault tolerance of 1. That is not a theoretical concern:

One server down and you are one failure from data loss.
More immediately, mmvdisk refuses to suspend a server while any vdisk in the recovery group sits at fault tolerance 1. Every maintenance action that needs a server offline — firmware, kernel updates, a BIOS setting, a physical move — is blocked. Not warned about: blocked.

The cost of fault tolerance 2 is capacity. A 4+2p code writes six strips for four of data, so usable capacity is raw divided by 1.5. An 8+2p code divides by 1.25. On the same hardware, moving from 8+2p to 4+2p costs you a fifth of your usable space and buys you the ability to service the cluster.

Decide this deliberately. Converting later means building a second filesystem alongside the first and copying everything across, which needs enough free capacity to hold both.

Validate the hardware before you install anything

IBM publishes two readiness tools in the SpectrumScaleTools repository: ece_network_readiness and ece_storage_readiness. Run both before installing. They check the things that will otherwise be discovered as a performance mystery three weeks later.

The thresholds worth knowing, because you can check them yourself:

Measure	Threshold
ICMP latency between storage nodes, average	≤ 1 ms
ICMP latency, maximum	≤ 2 ms
ICMP standard deviation	≤ 0.333 ms
NVMe random 128 KiB IOPS, per drive	> 15,000
Single-client read throughput	> 2,000 MB/s

A cluster that misses the latency numbers will work and will never perform. The standard deviation matters more than the average — jitter on the storage network shows up as unexplained tail latency in every job on the cluster.

Prepare the operating system

Pin the kernel. ECE builds a portability layer against the running kernel. An unplanned kernel update on one node produces a node that will not start mmfsd until you rebuild. Exclude kernel packages from routine updates and treat kernel changes as a scheduled activity with a rebuild step.

Name devices by path, not by enumeration. NVMe controllers are probed asynchronously. The device that is /dev/nvme4n1 today may be /dev/nvme1n1 after a reboot, and the mapping is not stable across identical servers. Any configuration that names /dev/nvmeXnY — a disk setup definition, a script, a monitoring check — is a latent failure that surfaces at the worst time, which is during a reboot you performed for an unrelated reason.

Use /dev/disk/by-path/ or the WWN. ECE itself identifies drives by their unique identifier and is unbothered; it is the tooling around ECE that breaks.

A related trap: with NVMe native multipath, block devices sit under a virtual subsystem and have no PCI parent. Reading a PCI address from the block device returns nothing. The address lives on the controller, at /sys/class/nvme/nvmeN/address. A validation script that reads the wrong path returns empty and, if written carelessly, reports success.

AMD EPYC platforms need two specific fixes. On family 23 and 25 processors, mmaddnode and mmstartup can hang for twenty to thirty minutes at 99% CPU inside the GSKit cryptographic library. The fix is documented by IBM as APAR IJ47407 and amounts to setting ICC_SHIFT=3 in the ICCSIG configuration. Bake it into the image; discovering it per node is a slow way to build a cluster.

Separately, some Genoa-based platforms panic on late microcode load. If you see kernel panics on nodes that otherwise boot, add dis_ucode_ldr to the kernel parameters and load microcode from the firmware instead.

Create the cluster

Install gpfs.base, gpfs.gpl, gpfs.ece and the license packages, then build the portability layer with mmbuildgpl. Do this on every node before creating the cluster, not as part of it.

mmcrcluster -N nodefile -r /usr/bin/ssh -R /usr/bin/scp -C ece-cluster
mmchlicense server --accept -N all

Quorum design is worth a moment. Use an odd number — three for most clusters, five for large ones. Quorum nodes should be spread across failure domains: racks, power feeds, and switches. A three-node quorum with all three in one rack is a single-rack outage away from a filesystem that will not mount.

Designate manager nodes explicitly rather than letting every server carry the role. On a storage cluster the servers are busy; giving the manager role to all of them means a manager failover happens under exactly the conditions where you least want extra work.

Configure the network, and verify it is actually being used

This is where clusters silently underperform.

verbsPorts tells GPFS which RDMA devices to use. Set it, then confirm it took:

mmlsconfig verbsPorts                 # what is stored
mmfsadm dump config | grep -i verbs   # what is running
mmdiag --network | grep -E "by (TCP|RDMA) connection"
mmfsadm test verbs conn               # per-peer RDMA counters

Three failure modes, all of which present as “it works, but slowly”:

Per-node overrides are not inherited. A node added later with mmaddnode does not pick up a per-node verbsPorts setting. It falls back to the cluster default, and if that default names a device the new node does not have, GPFS quietly uses TCP over IPoIB instead. Throughput drops by a factor you will notice; nothing logs an error. Check verbsPorts, pagepool and the license designation after every node addition or rejoin.

Pointing at the wrong port type. A dual-port adapter may present one InfiniBand port and one Ethernet/RoCE port. A global default of mlx5_1/1 aims GPFS at whichever port happens to be second, which on some platforms is the RoCE port. Check link_layer in sysfs before trusting a device name.

The negotiated rate is not what you ordered. iblinkinfo reports the per-lane rate, not the link rate. 4X 106.25 Gbps is NDR400; 4X 53.125 Gbps is HDR200; 2X 53.125 Gbps is a half-width HDR100 link that will gate every synchronized operation across the cluster. The authoritative per-host view is /sys/class/infiniband/<dev>/ports/1/rate. Check every node, not a sample — one degraded link is enough to make an entire benchmark meaningless, and it will be blamed on the filesystem.

Build the recovery group

mmvdisk nodeclass create --node-class NC_ECE -N storage01,storage02,...,storage08
mmvdisk server configure --node-class NC_ECE --recycle one
mmvdisk recoverygroup create --recovery-group RG1 --node-class NC_ECE

mmvdisk server configure sets the server-side parameters, including pagepool and the RAID buffer pool percentage, and --recycle one restarts daemons one at a time to apply them.

Confirm the recycle actually completed. pagepool takes effect only on daemon restart. A configured value with no completed restart leaves the cluster running the old value indefinitely, and mmlsconfig will happily report the new one. The running value is in the daemon’s own log at every start, and in mmdiag --config. A cluster running months on a fraction of its intended pagepool is not a hypothetical failure; the symptom is mediocre write performance that no tuning fixes.

Define vdisk sets, and read the size units carefully

mmvdisk vdiskset define --vdisk-set VS1 --recovery-group RG1 \
        --code 4+2p --block-size 2M --set-size 100T
mmvdisk vdiskset create --vdisk-set VS1
mmvdisk filesystem create --file-system fs1 --vdisk-set VS1

--set-size is usable capacity, not raw. Requesting 150 TiB at 4+2p consumes 225 TiB of raw space, and the command fails with “not enough space” on an array that appears to have plenty. Multiply by the code’s expansion factor before you ask.

Two more things that are difficult to change afterwards:

Block size changes the subblock count, not the subblock size. Moving from 4 MiB to 2 MiB blocks takes you from 512 subblocks to 256, both of 8 KiB. Small-file efficiency is unchanged. If you are rebuilding a filesystem in the belief that a smaller block size will help small files, check mmlsfs -f first and confirm the subblock size you expect.

mmvdisk vdiskset list prints per-vdisk size, not set totals. A set of sixteen vdisks showing “81 TiB” is 1.27 PiB. Misreading that column is a straightforward way to conclude a filesystem is nearly full when it is under ten percent used. Allocation is not utilization: a declustered array can be 99% allocated to vdisk sets while the filesystem on top is nearly empty.

Tune, and verify each change actually applied

mmchconfig returning success means the value is stored. It does not mean the value is in effect, and for several parameters it does not mean the value will ever be in effect.

mmlsconfig <param>                        # stored
mmfsadm dump config | grep -i <param>     # running   <- the one that matters

In mmfsadm dump config, ! marks a value differing from default, and * marks one changed at runtime. A value that appears with ! but never with * may have been accepted and discarded.

Parameters worth setting, with measured effect on an NVMe ECE cluster:

Parameter	Effect
`pagepool` (servers)	Large. Shared-file writes improve substantially; file-per-process much less, because it was never buffer-starved
`maxMBpS`	~6% on writes, nothing on reads
`workerThreads`, `prefetchThreads`	~3% on writes
`nsdRAIDBufferPoolSizePct`	Sets the RAID buffer; leave at the configured default unless you have a reason

Parameters that are accepted and do nothing on current versions. These are documented here so nobody spends a rolling restart proving it again:

Parameter	Why it does nothing
`verbsRdmasPerNode`	Supported in 4.2.x only. Runtime value stays 0
`verbsRdmasPerConnection`	Obsolete since 5.0. Stored, discarded at runtime
`nsdRAIDThreadsPerQueue`	No measurable effect
`verbsNumSendChannel`, `verbsNumRecvChannel`	No measurable effect
`verbsRdmaMaxSendBytes`	No measurable effect
`verbsRdmaSend`	Genuinely moves RPCs from TCP to RDMA — and changes throughput not at all. It governs daemon-to-daemon traffic, not the data path

Make the cluster repair its own mounts

Anything bind-mounted over a GPFS path — a scratch directory, a module tree, a shared application area — is torn down when the filesystem goes away and is not re-established when it returns. A systemd unit with BindsTo=gpfs.service covers a service restart but not an mmshutdown/mmstartup cycle, because the filesystem can disappear and return without the service unit changing state.

GPFS knows when the filesystem mounts even when systemd does not. Register a callback:

mmaddcallback ScratchBind \
  --command /usr/local/sbin/gpfs-post-mount.sh \
  --event mount --parms "%fsName"

Three details decide whether this works:

Match every device name the filesystem is known by. If the filesystem is ever renamed, %fsName changes and a guard written against the old name exits silently. Accept both, and log any name you do not recognize, so the next rename leaves evidence instead of a silent outage.
Make the script idempotent and fast. A slow callback delays the mount for every node.
RemainAfterExit=yes is load-bearing on any Type=oneshot bind unit. Without it, systemd deactivates the unit the instant ExecStart returns and runs ExecStop as part of that normal deactivation — the unit binds the path and unbinds it a second later.

Be aware of the corollary: systemctl is-active on such a unit reports active whether or not the bind exists. Check the mount, with findmnt -no SOURCE <path>, never the unit state.

Validate with numbers that survive scrutiny

Size the dataset from the cache, not from convenience. Client-side O_DIRECT bypasses the client pagepool but not the servers’ RAID buffer pool. That buffer is:

pagepool × nsdRAIDBufferPoolSizePct × number_of_servers

Eight servers with a 128 GiB pagepool at 80% is roughly 800 GiB of read cache. A read benchmark smaller than that measures memory. The error is large — a test that reports 155 GiB/s cached can be 82 GiB/s cold. Size reads at several times the buffer and confirm by re-running at double: if the number drops, you were measuring cache.

Do not mistake under-driving for a ceiling. Too few outstanding I/Os looks exactly like a hard limit. Sweep concurrency; a genuinely saturated system is flat across 64, 96, 128 and 192 jobs, while an under-driven one climbs. Sweep node count too, since a per-node limit and an aggregate limit are different problems with different fixes.

Check the fabric before tuning the filesystem. If clients and servers sit on different switches, measure the inter-switch links under load before changing a single GPFS parameter. Static InfiniBand routing pins each destination to one output port, and it can collapse many links into a few in one direction while the other direction spreads perfectly. The signature is a large read/write asymmetry with nothing apparently saturated, and no amount of GPFS tuning touches it.

nsdperf exercises the NSD protocol with no disk layer at all, which separates network from storage in one test. If nsdperf and your filesystem report similar numbers, the bottleneck is at or below the network and no tunable will help.

Restart the cluster without an outage

Many parameters need a daemon restart, which on a storage cluster means restarting servers underneath a live filesystem. Done with gates it is routine.

Refuse to start unless all of these hold, and re-check them between every node:

every server active
recovery group needs service: no
all pdisks ok
no rebuild or rebalance running
fault tolerance still at its design value — this is the safety budget
the filesystem still mounted on a client

Order matters: non-quorum nodes first to prove the procedure, then quorum nodes one at a time, and the recovery group master last so you do not trigger a pointless master failover at the start.

Run it detached — setsid nohup ... &. An administrator’s dropped SSH session should not be able to leave a storage node down with no orchestration.

The failures worth recognizing on sight

“Node appears to already belong to a cluster.” Usually a stale mmsdrserv process on the node from a previous attempt. Kill it and retry; reinstalling the node is not required.

A node that answers SSH but has no /var/mmfs. It is sitting in a provisioning installer environment with a read-only NFS root, not the installed OS. Probe for a real local root before running anything against a node you have just rebooted.

Reading the filesystem mounts it. mmlsfs and mmrepquota trigger an internal mount. Doing a record-keeping step immediately before an unmount gate makes that gate fail, and the internal mounts are not visible as user processes — fuser shows only kernel threads. Record first, force-unmount second, gate third. mmchfs -T requires zero mounts including internal ones, and a plain mmumount -a does not guarantee that.

Fileset inode limits do not carry across. A fileset created on a new filesystem defaults to a little over 100,000 inodes regardless of what the source held. A migration that copies millions of files into it dies partway with no obvious cause. Set --inode-limit at creation.

Snapshots cannot be migrated. They live with the filesystem. If a rebuild is planned and any rollback history matters, extract it first; the new filesystem starts with none.

What to watch once it is running

mmhealth covers the basics. Beyond it, the signals that catch real problems early:

long waiters — mmdiag --waiters; anything over 30 seconds repeatedly is a stuck subsystem, not a busy one
server queue depth — mmfsadm dump nsd | grep -E "pending|active". Queues reporting pending 0 under load mean the servers are starved and the bottleneck is upstream. This one measurement short-circuits most performance investigations and is worth taking first, not last
RDMA actually in use — mmdiag --network, per the earlier warning
pdisk state and fault tolerance — a background rebuild is normal; a fault tolerance that has dropped and stayed down is not
config versus running state — periodically diff mmlsconfig against mmfsadm dump config. Divergence is silent and accumulates

The last one deserves emphasis. Nearly every long-running problem described here shares a shape: the system reported success, the configuration said one thing, and the running state said another. Build the habit of checking the second one.

Changing the OpenSM routing engine: updn, ftree, and the silent fallback

2026-09-04T08:00:00-04:00

The subnet manager decides which path every packet takes through the fabric, and the routing engine is the algorithm it uses to decide. Changing it is a one-line edit to opensm.conf. Confirming that the change actually took is the part worth learning, because one of the common engines refuses fabrics it does not like and falls back to another without failing.

The engines worth knowing

minhop routes by shortest path. It is simple, fast to compute, and offers no protection against credit loops. On a topology where cycles are possible, that is a deadlock waiting for the right traffic pattern. It is a reasonable default on small, regular fabrics and a poor one on anything irregular.

updn applies an up/down turn rule: traffic may go up the tree and then down, but never up again after descending. That constraint makes credit loops impossible on an arbitrary topology, which is why up/down is the safe answer when you do not know what shape the fabric is. The cost is path quality — some routes are longer than they need to be, and the distribution across links is uneven.

Up/down depends entirely on which nodes it treats as roots. Left to itself, OpenSM guesses, and the guess is often poor on a fabric where spine and leaf switches are not obviously distinguishable. Give it the answer explicitly:

routing_engine updn
root_guid_file /etc/opensm/root_guids.txt

That file holds one spine switch GUID per line. Getting it right is the difference between up/down performing acceptably and performing badly, and it is the most common thing left undone.

ftree is optimized for fat trees, and on a fabric that genuinely is one it produces materially better link utilization than up/down. It also has strict requirements about what counts as a fat tree: the topology must be regular, constant bisectional bandwidth, with the compute nodes at the leaves and a consistent number of links between tiers.

dnup inverts the up/down rule, and is occasionally the better fit where the traffic pattern is dominated by leaf-to-leaf rather than leaf-to-core.

There are others — lash, dor, torus-2QoS, dfsssp — that matter on specific topologies. If you are running a torus or a dragonfly, the engine is not a matter of taste and the vendor guidance for that topology governs.

The silent fallback

routing_engine accepts an ordered, comma-separated list, and that is not a preference list in the way people assume. It is a fallback chain:

routing_engine ftree,updn,minhop

OpenSM tries ftree. If the fabric does not satisfy its requirements, it logs the reason and moves to updn. The subnet comes up. Traffic flows. Nothing appears to be wrong. You are simply not running the engine you configured, and the performance you were expecting from the change does not arrive.

This happens more often than it should, for mundane reasons: one node cabled to a spine instead of a leaf, a leaf with a different uplink count from its siblings, a switch that was down during the sweep, a section of the fabric added later with a different ratio.

After changing the engine, read the log and confirm which one is actually running. Do not infer it from the configuration file. The configuration file records what you asked for.

Confirming what is running

The subnet manager log is the primary source:

grep -iE 'routing engine|ftree|fallback|topology' /var/log/opensm.log | tail -40

You are looking for a line naming the engine that was used, and — if ftree declined — a line explaining why. That explanation is usually specific enough to fix: it will name the node or the tier that broke the assumption.

For a fuller picture, have OpenSM write out what it computed:

opensm --dump_files_dir /var/tmp/osm-dump -o

-o runs a single sweep and exits, which is what you want for inspection rather than for taking over the subnet. The dump directory gets the LID matrices and the per-switch forwarding tables, which is the ground truth about where traffic will actually go.

Then confirm from the fabric side:

sminfo
ibdiagnet -o /var/tmp/ibdiag --routing

Applying a change

Routing is recomputed on a sweep, and the engine is read at startup. A change means restarting the subnet manager:

systemctl restart opensm

Two cautions. On a large fabric the recompute and redistribution of forwarding tables is not instant, and traffic during that window can take paths that are in the process of changing. Do this in a maintenance window on anything busy.

And if you run a standby subnet manager, its configuration has to change too. A failover onto a standby still holding the old engine will silently reroute the entire fabric back, and the resulting “the fabric got slow again for no reason” is genuinely hard to diagnose if you have forgotten the standby exists.

Measuring whether it helped

The point of changing engines is link utilization, so measure that rather than a single pair’s bandwidth.

Point-to-point between one pair of nodes is usually indifferent to the routing engine, because a single flow takes a single path either way. What changes is what happens when many flows share the fabric. Use an all-to-all or an all-reduce across the real job size:

mpirun -np <ranks> ./osu_alltoall
mpirun -np <ranks> ./build/all_reduce_perf -b 1G -e 1G -n 20

Take a baseline before the change, on the same node set, and compare. Also look at per-port counters across the spine after a representative run: an engine that is distributing well produces even traffic across parallel uplinks, and one that is not produces a visible imbalance. That imbalance is often easier to see, and more convincing, than a bandwidth number.

Adaptive routing is a different layer

Modern switches can make per-packet forwarding decisions to route around congestion. That is a switch feature, configured on the switch and in some cases coordinated by the fabric manager, and it operates on top of whatever path set the subnet manager computed. It is not a substitute for choosing the right routing engine, and enabling it does not repair a fabric whose static routing is badly distributed. Get the static routing right first, then decide whether adaptive routing adds anything.

The short version

Set the engine, restart, then read the log to find out which engine you are actually running. Give updn an explicit root list. Expect ftree to decline if anything about the topology is irregular, and treat its explanation as a cabling bug report rather than a configuration problem. Measure with collectives, not with a single pair.

Splitting ports on Quantum switches: what it costs you

2026-09-04T07:00:00-04:00

Port splitting turns one high-rate port into two or four lower-rate ports. On a Quantum-class HDR switch that is 40 ports at HDR200 becoming 80 at HDR100. On a Quantum-2 NDR switch it is 64 logical NDR400 ports, presented through 32 physical cages, becoming 128 at NDR200.

The reason to do it is almost always economics. If your endpoints have HDR100 or NDR200 adapters, an unsplit switch wastes half of every port’s capability, and you buy twice the switches you need. The reason to think carefully first is that splitting changes four things at once, and only the first is obvious.

What a cage actually carries

The physical connector is not the port. On Quantum-2, one OSFP cage carries eight lanes, presented by default as two NDR400 ports of four lanes each. Split, those same eight lanes present as four NDR200 ports of two lanes each. Nothing about the switch got faster or slower; the lanes were regrouped.

This is why the aggregate switch bandwidth does not change when you split, and why “we doubled the ports” and “we doubled the capacity” are different claims. You have the same total, divided more finely.

The renumbering

Split ports get a new naming scheme, and every piece of tooling that referenced the old names breaks quietly.

unsplit      1/1        1/2        1/3
split        1/1/1 1/1/2   1/2/1 1/2/2   1/3/1 1/3/2

Anything holding port identifiers is now wrong: monitoring definitions, cabling records, port-to-node maps, error-counter baselines, and any script that walks a port list. None of that fails loudly. It just reports on ports that no longer exist, or misses ports that now do.

Regenerate the topology from the fabric rather than editing your records by hand:

ibnetdiscover > topology-$(date +%F).txt
iblinkinfo
ibdiagnet -o /var/tmp/ibdiag-$(date +%F)

Do that before the change as well as after. The diff is what you hand to whoever maintains the monitoring.

Enabling it

The exact syntax depends on the switch OS release, and this is one of the places where an old runbook will mislead you. The general shape on MLNX-OS is a system profile that permits splitting, then a per-port module type, then a reboot:

switch (config) # system profile ib split-ready
switch (config) # interface ib 1/1 module-type qsfp-split-2
switch (config) # write memory
switch (config) # reload

Three things to check against your own release notes rather than against this: whether the profile change is required at all on your platform, whether the split factor is expressed as split-2 or split-4, and whether the reboot is required or merely recommended. Treat the vendor documentation for your exact firmware as authoritative.

The reboot is the real constraint. Enabling split-ready mode is a switch restart, which is a fabric event. On a single-switch fabric that is an outage. On a leaf-and-spine fabric it is a capacity reduction and a routing re-convergence, and if it is a spine, it is a large one. Plan it as maintenance, not as a configuration change.

Cabling

A split port needs a splitter cable, and the breakout ratio has to match the configured split factor. A two-way split needs a two-way breakout; a four-way split needs a four-way. Mixing them produces ports that never link, with no error message that says why.

Two practical points. Splitter cables are physically bulkier at the switch end and have real bend-radius limits, which matters in a dense cabinet more than the datasheet suggests. And a partially split switch — some cages split, some not — is entirely legal and genuinely useful, but it makes the cabling records the only source of truth about which is which. Label at both ends.

What it does to the subnet

This is the part that gets underestimated.

Splitting increases the number of endpoints the subnet manager has to enumerate, assign and route. Every added endpoint consumes at least one LID, occupies entries in the forwarding tables of every switch on any path to it, and adds work to every sweep.

Three consequences:

Routing table pressure. Switch forwarding tables have a hard capacity. Doubling endpoint count moves you toward that limit, and the failure mode when you reach it is not graceful.
Longer sweeps. More ports means more discovery and more table distribution. On a large fabric the difference between a sweep that finishes quickly and one that does not is operationally significant, because sweeps happen on every topology change.
Changed oversubscription. If you split leaf ports facing compute and leave the spine uplinks alone, you have just changed the ratio between edge and core capacity. That number should be a decision, not a side effect.

Work out the LID budget and the forwarding table headroom before you split, not after. Both are countable in advance, and both are much cheaper to discover on paper than during a maintenance window.

Verifying it took

After the reboot, confirm the switch presents what you expect, then confirm the subnet manager agrees:

ibswitches
iblinkinfo | grep -c Active
ibdiagnet -o /var/tmp/ibdiag-after
sminfo

Compare the port count and the link rates against what you intended. A cage that split but whose cable did not match will show ports in a down or polling state rather than an error, so count active links rather than trusting the absence of errors.

Then check the error counters are clean from a fresh baseline, because the old baseline referred to ports that no longer exist:

ibclearerrors
# run representative traffic
ibqueryerrors

When not to split

If your endpoints have full-rate adapters, splitting throws away half of each one’s capability for no gain. If you are already close to the forwarding table limit, splitting is how you find the limit. And if the fabric is a single switch with no maintenance window available, the reboot is the whole problem and no amount of planning removes it.

Splitting is a good answer to a specific question: you have more endpoints than ports, and those endpoints do not need full rate. Outside that question it is usually the wrong tool.

ConnectX firmware management: PSID, burn, reset, verify

2026-09-04T06:00:00-04:00

Firmware on a ConnectX adapter is not one thing. There is the image in flash, the device configuration stored alongside it, and the running state, and all three can disagree after a partial update. A card that reports the new version but behaves like the old one is common, and the reason is almost always that nothing reset it.

Start the driver, find the device

Nothing in the MFT toolset works until the MST kernel module is loaded and the device nodes exist.

sudo mst start
sudo mst status -v

That prints paths under /dev/mst/ of the form mt4123_pciconf0 for a ConnectX-6 or mt4125_pciconf0 for a ConnectX-7, alongside the PCI address, the RDMA device name and the network interface. Everything downstream takes one of those paths as -d.

The MFT version matters here in a way that is easy to miss. A toolset older than the silicon will either not enumerate the device or will enumerate it and then refuse the image. If mst status shows nothing for a card you can see in lspci, update MFT before debugging anything else.

Query before touching anything

sudo mlxfwmanager --query -d /dev/mst/mt4125_pciconf0

Three fields matter, and only one of them is the version.

PSID is the parameter set identifier, and it binds an image to a specific hardware variant. Two cards with the same model number and the same firmware version can carry different PSIDs, because one is an OEM part with different defaults, a different port configuration or a different flash layout. Firmware is published per PSID. This is the field to match on.

FW version appears in two forms: what is in flash, and what is running. When those differ, an image has been burned and not activated. That is the state a reset clears, and it is the most common reason a machine comes back from an update behaving exactly as it did before.

Status reports whether an update is available, which only means anything when you are querying online.

Where the host has outbound access:

sudo mlxfwmanager --online -u -d /dev/mst/mt4125_pciconf0

On an isolated fabric, which is the normal case, fetch the image matching the PSID and burn it explicitly:

sudo mlxfwmanager -u -d /dev/mst/mt4125_pciconf0 -i fw-ConnectX7-rel.bin

The PSID trap

flint will burn an image whose PSID does not match the card, if you insist:

sudo flint -d /dev/mst/mt4125_pciconf0 -i image.bin burn                       # refuses
sudo flint -d /dev/mst/mt4125_pciconf0 -i image.bin --allow_psid_change burn   # does not

The refusal is a feature. --allow_psid_change exists for genuine cases — recovering a card whose flash was corrupted, converting a part with vendor approval — and it is not the flag to reach for because a burn failed. A card running an image built for a different variant can come up with the wrong port configuration, the wrong link type, or not come up at all, and the way back involves the same tool in a much less comfortable mode.

Match on PSID, not on model. Two adapters that look identical in lspci can need different images, and any fleet bought in two tranches is likely to hold both.

Device configuration is separate from firmware

mlxconfig reads and writes non-volatile settings that survive firmware updates and are not part of the image:

sudo mlxconfig -d /dev/mst/mt4125_pciconf0 query
sudo mlxconfig -d /dev/mst/mt4125_pciconf0 set LINK_TYPE_P1=1

On a VPI-capable part, LINK_TYPE_P1 and LINK_TYPE_P2 decide whether a port comes up as InfiniBand or Ethernet. Getting that wrong gives you a port that never links and no error that explains why. Other settings worth knowing: SR-IOV enablement and virtual function count, ATS, and the PCIe and performance knobs that appear in vendor guidance from time to time.

Two things to hold onto. A firmware update can introduce new configuration keys carrying defaults you did not choose, so query after an update as well as before. And mlxconfig changes also need a reset to take effect, which brings us to the step people skip.

Reset, and when a reset is not enough

sudo mlxfwreset -d /dev/mst/mt4125_pciconf0 query
sudo mlxfwreset -d /dev/mst/mt4125_pciconf0 -y reset

The query subcommand reports which reset levels the card and the platform will actually support in the current state, which is more useful than it sounds: a warm reset the platform cannot perform is reported rather than attempted.

Some combinations of card, host and BIOS cannot do a live reset at all. In that case the update is not applied until a full power cycle, and on some platforms specifically an AC power cycle rather than a warm reboot. If you have burned an image, run a reset, and the query still shows the old running version, stop reasoning about firmware and go find out whether the machine actually power-cycled.

Verify against running state, from three directions

sudo mlxfwmanager --query -d /dev/mst/mt4125_pciconf0 | grep -E 'PSID|FW'
ethtool -i <netdev>
ibv_devinfo -d <rdma-dev> | grep -i fw_ver

Three sources because they read from different places. mlxfwmanager reads flash, ethtool reports what the driver bound, ibv_devinfo reports what the verbs layer sees. Agreement across all three is what “the update worked” means. Disagreement between the first and the other two is the unreset-card signature.

Doing this across a fleet

Inventory by PSID, not by hostname. Group the estate by PSID and firmware version before planning anything. The distribution is reliably less uniform than the purchase orders suggest.

Stagger by failure domain. Adapters in the same rack, on the same leaf, or serving the same storage class should not reset in the same window. A reset drops the link and the subnet manager re-sweeps; doing that everywhere at once turns a routine update into a fabric event.

Record before and after. Query output for every card, stored and dated. The value shows up months later, when something is behaving oddly and the only question that matters is whether it changed.

Pair firmware with driver. Vendor release notes pair firmware versions with specific OFED or in-box driver versions. Updating one and not the other is supported far less often than it is done, and the symptoms get blamed on the application.

What this does not cover

Switch firmware is a different toolchain and a different risk profile, and deserves its own written procedure. So does recovery from an interrupted burn — flint in its more surgical modes, and the cases where a card has to come up on a recovery image. That one is worth writing down once, by whoever has just done it, rather than improvised under pressure.

Command syntax drifts between MFT releases. What is above describes the shape of the tools rather than one version, and --help on the release you are running is authoritative over anything written down elsewhere, this included.

nccl-tests: bus bandwidth, and the one column worth reading

2026-09-04T00:00:00-04:00

Distributed training spends a large share of its wall clock inside collective operations. In data-parallel training the dominant one is an all-reduce over the gradients, once per step, sized by the model. If that collective is slower than the hardware allows, every step is slower, and no amount of framework tuning recovers it.

nccl-tests from NVIDIA is how you measure that path on its own, with no model and no framework in the way.

Building it

Build against MPI if you intend to run across more than one node, which is the only interesting case:

git clone https://github.com/NVIDIA/nccl-tests
cd nccl-tests
make MPI=1 MPI_HOME=/opt/openmpi CUDA_HOME=/usr/local/cuda

That produces one binary per collective: all_reduce_perf, all_gather_perf, reduce_scatter_perf, broadcast_perf, alltoall_perf, sendrecv_perf and a few others. Start with all-reduce, because it is what training actually does.

mpirun -np 16 -map-by ppr:8:node \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 -w 5 -n 20

One rank per GPU, -g 1. The sweep runs from 8 bytes to 8 GB doubling each step, with five warmup iterations discarded and twenty timed. Warmup is not optional: the first call pays for buffer registration and connection setup, and including it drags the small-message numbers into nonsense.

The two bandwidth columns

The output has, for both out-of-place and in-place variants, a time, an algbw, a busbw and a correctness count.

algbw is algorithm bandwidth. It is simply the message size divided by the elapsed time. It answers “how fast did my data get processed,” and it is the number that looks intuitive.

busbw is bus bandwidth. It is algbw scaled by a factor that accounts for how much traffic the collective genuinely has to move across the interconnect, given the number of ranks. For a ring all-reduce, each byte has to traverse the ring twice, once in the reduce-scatter phase and once in the all-gather phase, and each rank ends up sending (n-1)/n of the buffer in each phase:

all-reduce         busbw = algbw × 2(n-1)/n
all-gather         busbw = algbw × (n-1)/n
reduce-scatter     busbw = algbw × (n-1)/n
all-to-all         busbw = algbw × (n-1)/n
broadcast, reduce  busbw = algbw

The consequence is the reason busbw exists. As you add ranks, algbw for a fixed message size falls, because there is more work to do per byte. busbw stays roughly flat, because it is measuring what the wire is doing. That makes busbw comparable against your hardware’s per-link capability and comparable between a two-node run and a hundred-node run.

Track busbw. Someone reporting that all-reduce “got slower when we scaled up” is almost always reading algbw, where a decline is arithmetic rather than a fault. Compare busbw at the same message size across job sizes; if that holds, the fabric is behaving.

Reading the sweep

The curve has two regimes and the transition tells you where your model sits.

At small sizes the result is latency-bound. Time is nearly flat as size increases, so bandwidth rises roughly linearly. What you are measuring is the fixed cost of a collective: kernel launch, synchronization, and the depth of the communication pattern. NCCL uses tree algorithms here specifically because tree latency grows logarithmically with rank count while ring latency grows linearly.

At large sizes the result is bandwidth-bound and busbw flattens into a plateau. That plateau is the number to record and to compare against what the links should deliver.

Work out where your gradient all-reduce lands. For a bucketed data-parallel setup, the bucket size is the message size, and it is frequently in the region where neither regime dominates cleanly. Moving the bucket size is a real tuning knob, and this sweep tells you which direction to move it.

When the plateau is too low

The plateau being well below what the hardware should do almost always comes down to one of a short list.

NCCL chose the wrong interface. On a node with a management NIC and several fabric adapters, NCCL can select the wrong one. Verify rather than assume:

NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH \
  mpirun -np 16 -map-by ppr:8:node ./build/all_reduce_perf -b 1G -e 1G -n 20

The output names each adapter it selected and prints the rings it constructed. If it is using a socket transport where you expected InfiniBand, that is your answer. NCCL_IB_HCA and NCCL_SOCKET_IFNAME constrain the choice.

GPUDirect RDMA is not engaged. Without it, every message stages through a host bounce buffer. The INIT log says whether GDR is in use per device. NCCL_NET_GDR_LEVEL controls how aggressively it is applied relative to PCIe topology distance.

PCIe ACS is enabled. Access Control Services forces peer-to-peer transfers up through the root complex instead of across the switch, which quietly destroys intra-node bandwidth. It is a BIOS or setpci change, and it is worth checking on every new node type.

The topology is not what you think. NCCL_TOPO_DUMP_FILE writes out the topology NCCL detected. Compare it against nvidia-smi topo -m and against how the machine is actually cabled. Ring construction is only as good as the topology discovery underneath it.

Correctness is a column too

The #wrong column exists for a reason. Run with -c 1 at least occasionally. A collective that returns wrong values under load is a hardware or driver fault, and it is a much more serious finding than a disappointing bandwidth plateau. It is also the kind of thing that shows up in a training run as a loss curve that diverges for no apparent reason, days later, after a great deal of compute has been spent.

What to do with the result

Record the busbw plateau, the message size where the curve leaves the latency-bound regime, the rank count, and the driver, NCCL and firmware versions. Re-run it after every driver update and every firmware change.

Collectives degrade quietly. Nothing fails, nothing logs an error, and step time drifts up by a percentage nobody notices until the quarter’s throughput is compared against the previous one. A five-minute benchmark on a known baseline catches it the same day.

IOR and mdtest: why your storage benchmark says the filesystem is fine

2026-09-03T00:00:00-04:00

Two tools cover most of what you need from a parallel filesystem. IOR measures bandwidth by moving large blocks. mdtest measures metadata by creating, stating and removing enormous numbers of files. Both come from the same repository and build together.

The common failure is running only IOR, getting a large number, and concluding the storage is healthy. For a training workload reading millions of small files, that number describes almost nothing you care about.

Defeating the cache, which is most of the work

The single most important thing about IOR is that a naive invocation measures page cache. The client caches the write, reports success, and IOR reports a bandwidth figure that reflects memory speed rather than storage.

Three flags matter:

mpirun -np 128 ./ior \
  -a POSIX -w -r \
  -b 16g -t 4m -s 1 \
  -F -e -C -i 3 \
  -o /gpfs/scratch/iortest/file

-e calls fsync before the write phase timer stops, so the write is on stable storage rather than in the client’s cache.
-C reorders tasks between the write and read phases, so no rank reads back the file it just wrote and finds it sitting in local page cache.
-b sets the block size per task. Make the aggregate substantially larger than the total client memory, or the read phase is served from RAM.

Without those, your read bandwidth will look wonderful and mean nothing. If a result looks impossibly good, this is the reason, every time.

-F selects file-per-process. The alternative, a single shared file, tests a different thing: lock contention and the filesystem’s ability to handle concurrent writers to one inode. Both are worth measuring, because applications do both, and on some filesystems they perform very differently.

-t is the transfer size, the size of each individual read or write call. Sweeping it is how you find where the client stops being able to fill the pipe.

The metadata problem

Now the part that usually explains the complaint you were called about.

mpirun -np 128 ./mdtest \
  -n 10000 -u -z 2 -b 8 \
  -i 3 -d /gpfs/scratch/mdtest

mdtest reports operations per second for file creation, stat, read, and removal, plus the directory equivalents. -u gives each task its own working directory, -z and -b control the depth and branching of the directory tree, and -n is files per task.

Run it with -u and without. The difference is often dramatic, and it is one of the more actionable results you can produce, because it tells you directly whether the application’s directory layout is the problem. A single directory holding a very large number of entries serializes on the metadata path in a way that the same files spread across a tree do not.

For AI training, metadata rate and small-file read latency usually predict throughput far better than streaming bandwidth. A dataset of millions of small files spends its time in open and stat, not in moving bytes. If you measure one thing, measure that.

Interpreting the gap between the two

The relationship between the IOR and mdtest results is more informative than either alone.

High bandwidth, high metadata rate. The filesystem is healthy. If an application is still slow, the problem is in the application’s access pattern or in the client, not the storage.
High bandwidth, poor metadata rate. Classic parallel filesystem shape. The data path is fine and the metadata path is the bottleneck. Look at metadata server placement, whether metadata sits on the same media as data, and how the workload lays out directories.
Poor bandwidth, high metadata rate. Usually the network or client configuration rather than the filesystem: check RDMA is engaged, check the mount options, check that clients are not falling back to a slower transport.
Both poor. Look at the client before the server. A single-client run that is also slow points at the node; a slow aggregate with fast single-client results points at the server or the fabric between.

Running it so the numbers survive scrutiny

Use -i to repeat and report the spread, not a single lucky run. Record the client count, the ranks per client, the transfer and block sizes, the filesystem version and the mount options alongside the result. Run against a directory with the same striping or placement policy the real workload will use, because a default-policy test directory can behave nothing like a production dataset.

And run the same configuration on a schedule. A storage benchmark taken once during acceptance is an anecdote. The same benchmark taken monthly is a baseline, and a baseline is the only thing that turns “it feels slower lately” into a number you can act on.

HPL-MxP: reading a mixed-precision result without fooling yourself

2026-09-01T00:00:00-04:00

HPL-MxP, previously published as HPL-AI, solves the same problem as HPL and returns a much larger number. The machine did not get faster. Understanding the gap is the whole value of the benchmark, and misunderstanding it is how the number ends up in a slide deck meaning nothing.

What it does differently

Classic HPL performs the LU factorization entirely in double precision. HPL-MxP performs the factorization in a lower precision, then repairs the answer.

The sequence is:

Factorize the matrix in reduced precision, which is where the hardware’s tensor or matrix units are fastest.
Use that low-precision factorization as a preconditioner.
Run an iterative refinement loop, in practice a Krylov solver such as GMRES, until the residual meets the same accuracy standard classic HPL must meet.

The final answer satisfies the same correctness criterion. The path there spent most of its time in arithmetic the hardware executes at several times the rate of double precision.

This is not a trick invented for benchmarking. Mixed-precision iterative refinement is a legitimate, long-standing technique in numerical linear algebra, and it is used in production solvers. The benchmark exists to reward hardware that implements it well.

Where the big number comes from

Here is the part that matters, and the part most often skipped.

HPL-MxP does not report the number of low-precision operations it performed. It reports the operation count of the equivalent double-precision HPL algorithm, (2/3)N³ + 2N², divided by the time the mixed-precision run actually took.

So the metric answers a specific question: if you needed an HPL-accurate solution to this system, how fast did you get one? That is a fair and useful question. It is not the same question as “how many double-precision floating point operations per second can this machine sustain,” and the answer must never be substituted for Rmax.

A machine’s HPL-MxP rate and its HPL Rmax are not comparable quantities, and the ratio between them is not an efficiency. It is a measure of how much the low-precision units outrun the double-precision units on that hardware, filtered through how well the refinement converged.

What actually determines the result

Three things move the number, and only one of them is raw hardware.

The precision used for the factorization. FP16, BF16 and TF32 have different dynamic range and different mantissa widths, and they are not interchangeable. BF16 keeps FP32’s exponent range with a short mantissa, which makes it robust against overflow but coarse. FP16 has a finer mantissa and a much narrower range, so it needs scaling to stay numerically safe. Which one performs better is a hardware question; which one converges better is a numerical one.

How many refinement iterations were needed. Every iteration costs time and none of it counts toward the reported operation count, because the count is fixed at the classic HPL figure. A run that converges in a handful of iterations posts a strong number. A run that struggles posts a weak one even on identical hardware. Convergence depends on the conditioning of the generated matrix and on the precision choice, which is why the benchmark specifies how the matrix is produced.

The usual HPL parameters. Problem size, block size and process grid still matter, and the tuning intuition carries over, though the optimum block size is generally larger because the point is to keep wide matrix units saturated.

When it is worth running

Run it if you are buying or operating hardware where the low-precision units are a large fraction of what you are paying for. On a GPU cluster intended for training workloads, the double-precision rate may be close to irrelevant to everything the machine will actually do, and HPL alone will undersell it badly.

Run it as a companion to HPL rather than a replacement. The pair tells you two different things about the same machine: HPL gives you the double-precision ceiling and a hardware health check, HPL-MxP gives you an indication of how well the reduced-precision path performs on a problem with a verifiable answer.

Do not run it expecting it to predict training throughput. It does not model attention, it does not model collectives at scale, it does not model the data pipeline, and it does not model memory capacity pressure. For that question, measure collectives directly with nccl-tests and then measure your actual model.

Reporting it responsibly

If you publish an HPL-MxP figure, publish the precision used for the factorization, the iteration count, the problem size, and the classic HPL result from the same machine alongside it. Without those four pieces, the number is unfalsifiable, and an unfalsifiable performance number is worth what it costs to produce.

HPCG: the benchmark that tells you what the memory system can do

2026-08-26T00:00:00-04:00

High Performance Conjugate Gradients was written by the HPL authors as a deliberate counterweight to their own benchmark. Where HPL is dense, regular and compute-bound, HPCG is sparse, irregular, and limited by memory bandwidth and communication latency. It exists because procurement decisions were being made on a number that predicted almost nothing about real workloads.

The headline result is startling the first time you see it. A machine that hits a large fraction of peak on HPL will typically return a low single-digit percentage of peak on HPCG. Nothing is wrong. That gap is the honest distance between what the floating point units can do and what the memory system can feed them.

What it actually runs

HPCG solves a sparse linear system with a preconditioned conjugate gradient method. The preconditioner is a geometric multigrid with a symmetric Gauss-Seidel smoother, and that choice is what makes the benchmark hard in an interesting way.

Symmetric Gauss-Seidel is inherently sequential in its data dependencies. You cannot simply vectorize your way out of it the way you can with a dense matrix multiply. Implementations have to reorder, color or otherwise restructure the sweep to expose parallelism, and doing that changes the convergence behavior, which the benchmark checks. The rules constrain how much you are allowed to change, which is what stops HPCG from degenerating into the same optimization contest HPL became.

The result is a benchmark dominated by sparse matrix-vector products, which means streaming large arrays with indirect indexing. Arithmetic intensity is low. Every cycle spent waiting on a cache miss is a cycle the floating point units sit idle.

Configuration

The input is small. hpcg.dat takes a local problem size per rank and a run time:

HPCG benchmark input file
Sandia National Laboratories; University of Tennessee, Knoxville
104 104 104
1800

The three numbers are the local subdomain dimensions nx ny nz, and they must each be a multiple of eight. The fourth is the target run time in seconds.

Sizing the local domain. The subdomain has to be large enough that it does not fit comfortably in cache, or you are benchmarking cache and the result is meaningless. A reasonable approach is to size it so the working set is a substantial fraction of the memory you want to represent per rank, then confirm that increasing it further does not change the reported rate much. When the number stops moving, you are measuring DRAM rather than last-level cache.

Run time. For an official submission the timed run must be at least 1800 seconds. For internal use you can run shorter, but be aware that short runs overweight the setup phase and flatter the machine slightly.

HPCG will happily produce a large number if you give it a small local domain. This is the single most common way sites accidentally publish a wrong HPCG result. Sweep the local size upward and take the value from the plateau, not from the peak.

What the output tells you

HPCG writes a YAML file with a full breakdown. The headline GFLOP/s rating is what gets quoted, but the per-phase timings underneath are more useful operationally, because they separate the sparse matrix-vector product, the symmetric Gauss-Seidel sweeps, the dot products and the halo exchange.

Those phases fail differently:

The matrix-vector and smoother phases track memory bandwidth. If they regress, look at DIMM population, memory clock, NUMA balance and whether something changed in the BIOS power profile.
The dot products are global reductions, so they track collective latency and, more importantly, jitter. A single noisy node shows up here first.
The halo exchange tracks neighbor communication, which is point-to-point and usually fine unless the topology mapping is poor.

That decomposition is why HPCG is worth running even when nobody is asking for a TOP500-style number. It is a repeatable, structured way of asking whether the memory subsystem and the collective path are behaving the way they did last quarter.

Where it fits against HPL

Run both. They fail for different reasons and finding the difference is diagnostic.

	HPL	HPCG
Limited by	Floating point throughput	Memory bandwidth, latency
Fraction of peak	High	Low
Sensitive to	Clocks, BLAS, thermals	DIMM config, NUMA, jitter
Representative of	Dense linear algebra	Most sparse scientific codes

If HPL holds steady and HPCG drops, suspect memory configuration or a change in NUMA behavior rather than the processors. If both drop together, suspect clocks, power or cooling. If HPCG holds and HPL drops, suspect the math library or a thermal limit that only sustained dense work reaches.

Neither benchmark predicts your applications. Together they bracket the machine between what it can do when everything is favorable and what it can do when almost nothing is, and most real codes live somewhere in between.

HPL: what Linpack actually measures, and what it is good for

2026-08-19T00:00:00-04:00

High Performance Linpack solves a dense system of linear equations by LU factorization with partial pivoting, in double precision, distributed across a process grid. It is the benchmark behind the TOP500 list, which is why it carries more political weight than technical weight.

The useful framing is this: HPL is a poor predictor of how your applications will perform, and an excellent way to find out whether a machine is broken. Both things are true at once, and most arguments about HPL come from people holding one half of that and assuming the other half is wrong.

Why it flatters hardware

The work HPL does is (2/3)N³ + 2N² floating point operations, and almost all of it lands in large dense matrix multiplies. That is the single most favorable operation any processor has: high arithmetic intensity, perfectly predictable access, and vendor-tuned kernels behind it.

So HPL reaches a high fraction of theoretical peak, and a well-tuned run on sensible hardware lands in the region of sixty to ninety percent of Rpeak. Very little real scientific code behaves this way. Most applications are limited by memory bandwidth, sparse or irregular access, latency, or synchronization, none of which HPL exercises meaningfully.

That is precisely why HPCG was created as a counterweight, and it is why quoting Rmax as a procurement figure of merit for a mixed workload is close to meaningless.

The parameters that matter

Everything lives in HPL.dat. Most of its knobs are noise; four are not.

Parameter	What it controls	Practical guidance
`N`	Problem size	Fill most of aggregate memory, leaving headroom
`NB`	Block size	Match the BLAS kernel’s preference
`P` × `Q`	Process grid	As close to square as possible, with `P` ≤ `Q`
`BCAST`	Broadcast algorithm	Try a few; the ring variants usually win at scale

Sizing N. The matrix is N × N in eight-byte doubles, so memory is 8N² bytes plus workspace. Solve for the memory you want to use:

N ≈ sqrt( total_bytes × utilization / 8 )

Utilization around 0.8 is a reasonable starting point. Push it too high and the run swaps or the OOM killer intervenes several hours in; too low and you spend proportionally more time in the parts of the algorithm that do not scale, and the result understates the machine.

Choosing NB. This is the blocking factor the factorization uses, and the right value is whatever the underlying BLAS wants. CPU runs with a tuned library typically want something in the low hundreds. GPU runs, particularly through a vendor container, want considerably larger blocks, because the point is to keep the device busy with big multiplies. Do not carry a CPU value across to a GPU run and expect it to hold.

The grid. P × Q must equal the number of MPI ranks. The panel factorization runs down columns of the grid and the update broadcasts across rows, so a grid that is far from square puts a disproportionate amount of the serial-ish work in one dimension. Square, or slightly wider than tall, is the rule.

Sizing N to fill memory means a full run takes hours. Do your parameter sweeps at a small N where a run takes minutes, find the shape of the answer, then do one or two large runs to get the real number. People routinely burn a week tuning at full size.

Where it earns its keep

Run HPL at close to full memory across every node, and you have built a sustained, uniform, several-hour stress test that touches nearly all of DRAM, nearly all of the floating point units, and the interconnect between every pair of ranks. It is the best acceptance test most sites have, and it finds things:

Nodes that thermally throttle only once the whole rack is hot, which no single-node test will reproduce.
Correctable memory errors that appear only when a large fraction of DRAM is touched repeatedly.
One node whose BIOS profile differs from its siblings, which shows up as the whole job running at the speed of that node.
Links that negotiated at a lower width or rate than expected.
Power delivery and cooling limits that only appear at full draw.

HPL prints a residual check at the end. Treat a failed residual as a hardware fault until proven otherwise, because that is usually what it is. A machine that returns wrong arithmetic under load is a much more urgent finding than a disappointing Rmax.

Reading the result honestly

Rmax is the best rate achieved. Rpeak is the theoretical maximum from clock rate, core count and per-cycle floating point width. Efficiency is the ratio, and it is the number worth tracking, because it is comparable across time on the same machine.

A drop in efficiency between two runs of the same configuration is a real signal. An absolute efficiency number compared against a different site’s machine is mostly a comparison of two vendors’ BLAS libraries and two people’s patience with parameter sweeps.

Use HPL to prove the machine is healthy and to catch regressions after firmware and driver changes. Use HPCG, and your own applications, to decide what the machine is worth.

DNS and mail authentication: a TXT answer is not a valid policy

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A DNS query returning text is not proof that the requested security record exists. Wildcards, stale records, and malformed content can make a superficial check look successful. Mail authentication needs both DNS inspection and an authorized message test.

Inspect the exact names

For a domain you administer, replace example.com and the known DKIM selector:

dig +short TXT example.com
dig +short TXT _dmarc.example.com
dig +short TXT selector1._domainkey.example.com
dig +short TXT deliberately-unused-label.example.com

The last query helps identify a wildcard response. An answer identical to the apex TXT content at an unrelated name deserves investigation. Confirm against authoritative nameservers and parse the actual policy rather than accepting any nonempty answer.

Separate the mechanisms

SPF authorizes sending infrastructure for the envelope identity. DKIM associates a cryptographic signature with a signing domain. DMARC evaluates alignment with the visible From domain and publishes handling/reporting policy. The primary specifications are SPF, RFC 7208, DKIM, RFC 6376, and DMARC, RFC 7489.

Consult current provider guidance and subsequent standards updates when implementing. This article’s inspection method does not depend on copying a universal TXT record.

Inventory legitimate senders

List the primary mail provider, billing system, CRM, website forms, and any other service sending as the domain. Record the actual envelope and signing domains from message headers. A vendor’s generic setup page cannot establish which identity your account is currently using.

Remove stale authorizations through a reviewed change after confirming they are no longer needed. Do not tighten policy blindly: an overlooked legitimate sender can be rejected along with unwanted mail.

Test with real headers

Send an approved test message from each legitimate route to a controlled mailbox. Inspect Authentication-Results, the visible From address, envelope identity, and DKIM signing domain. Confirm alignment and final delivery.

A website form displaying a thank-you message does not prove it sent mail. Verify that the receiving mailbox actually contains the message and that the sender identity is what the design intended.

Roll policy out with evidence

Begin with an inventory and reporting workflow, review legitimate failures, and adopt stricter handling according to the organization’s risk decision. Retain an explicit rollback and propagation plan. DNS caching means different receivers can observe different states during a change.

Keep checks semantic

The automated check should validate policy syntax, uniqueness where required, record type, and the relationship between DNS and observed message authentication. It should report lookup errors as incomplete evidence, not absence or success.

Repeat after adding a SaaS sender, changing mail providers, or replacing the website’s form handler. Domain authentication is a maintained dependency of business communication, not a one-time DNS task.

OSU Micro-Benchmarks: measuring the fabric before you blame the application

2026-08-12T00:00:00-04:00

When a distributed job runs slower than expected, the fabric is the first thing accused and the last thing measured. The OSU Micro-Benchmarks exist to close that gap. They are small MPI programs from the MVAPICH group at Ohio State that measure one communication pattern at a time, with nothing else running.

The value is not the numbers themselves. It is that you take them once when the machine is known good, write them down, and compare against them every time something feels slow afterwards.

The two that matter most

osu_latency sends a message between two ranks and reports half the round trip time. osu_bw streams a window of messages in one direction and reports sustained bandwidth. Everything else in the suite is a variation on those two ideas.

# two ranks, one per node, pinned
mpirun -np 2 -map-by ppr:1:node --bind-to core \
  ./osu_latency
mpirun -np 2 -map-by ppr:1:node --bind-to core \
  ./osu_bw

Both sweep message sizes and print a table. Read them as two different regimes. Small messages measure the fixed cost of getting a message onto the wire and back off it: software stack, doorbell, interrupt or polling behavior, and the switch hop count. Large messages measure how much of the link you can actually fill. The interesting part is the middle, where the transport switches protocol.

The eager to rendezvous transition

Below a threshold, MPI implementations send small messages eagerly: the sender pushes the payload without waiting to hear that the receiver has somewhere to put it. Above the threshold they switch to rendezvous, where the sender first exchanges control messages and then the data moves by RDMA directly into the destination buffer.

You can see the switch in the osu_bw table as a visible discontinuity in the bandwidth curve. Knowing where it sits on your system is genuinely useful, because an application whose typical message size lands just above the threshold will behave very differently from one that lands just below. Both MPICH-derived and Open MPI stacks let you move that threshold; whether you should is a question to answer with measurements from your own application, not from a micro-benchmark.

Pinning changes the answer

An unpinned two-rank latency test measures the scheduler as much as the network. If the process migrates to a core on the socket that does not own the HCA, every message crosses the inter-socket link on the way to the wire.

Always pin, always place one rank per node for the point-to-point tests, and check which NUMA domain your adapter is attached to:

cat /sys/class/infiniband/mlx5_0/device/numa_node
lstopo --output-format txt | head -40

If that returns a socket your ranks are not running on, fix the placement before recording anything. This single mistake accounts for a large share of “the network is slow” reports that turn out not to involve the network.

Collectives, and why they are a different question

osu_allreduce, osu_alltoall and osu_barrier scale to the full job size and are much closer to what real applications do. They are also much more sensitive to noise. One slow node, one core running a stray daemon, one rank waiting on a filesystem, and the whole collective inherits the delay, because a collective finishes when its slowest participant finishes.

That sensitivity is the point. A collective benchmark that degrades while point-to-point stays clean is telling you the problem is not the fabric. It is jitter, and jitter comes from the operating system, the scheduler, thermal behavior or a filesystem mount, not from the switch.

Run the point-to-point tests between several different node pairs, not just the first two the scheduler hands you. A single bad cable or a port that negotiated a lower width will only show up when that specific link is on the path, and it will look like an application problem for weeks.

GPU buffers

Recent versions build CUDA-aware variants that take -d cuda, which places the send and receive buffers in device memory instead of host memory:

mpirun -np 2 -map-by ppr:1:node ./osu_bw -d cuda D D

This is worth measuring separately, because the path is different. With GPUDirect RDMA working, the adapter reads and writes device memory directly. Without it, every message takes a detour through a host bounce buffer, and you will see it immediately in the bandwidth number. Comparing -d cuda against the host-memory run is the fastest way to confirm GPUDirect is actually engaged rather than merely installed.

What to record

Keep a file in version control with the date, the firmware and driver versions, the MPI build, the node pair, and the resulting curves. It takes ten minutes and it converts every future performance argument from opinion into a comparison.

The next articles in this series work up the stack from here: dense compute with HPL, memory and network behavior with HPCG, mixed precision with HPL-MxP, and collectives on GPU clusters with nccl-tests. All of them are easier to interpret once you know what a single link on your machine can do.

Nine ways an environment module makes working software look broken

2026-07-29T10:00:00-04:00

A user reports a tool is broken. You load the module, run the tool, and it works. The user runs the same three lines and it does not. Nothing in either transcript looks wrong.

That shape of report is almost always the module system, not the software, and it costs so much time because environment modules fail silently by construction. A modulefile sets variables. Setting one to a directory that does not exist is not an error. Forgetting the matching PATH entry is not an error. Loading a module whose files were deleted six months ago is not an error. The module system reports success in all three cases, and the failure surfaces elsewhere — as command not found, a missing Python package, or a linker error naming an unfamiliar symbol version.

What follows is nine mechanisms, each with the command that tells it apart, then a harness that catches most of them before a user does.

What `module` actually is

Everything here follows from one fact. module is not a program. It is a shell function that runs a program, captures the shell code it prints, and evaluates that in your current shell. Lmod defines it around $LMOD_CMD, Tcl Environment Modules around modulecmd. Watch it work:

$ $LMOD_CMD bash load gcc/12.3.0 2>/dev/null
PATH=/opt/apps/gcc/12.3.0/bin:/usr/local/bin:/usr/bin:/bin; export PATH;
LD_LIBRARY_PATH=/opt/apps/gcc/12.3.0/lib64; export LD_LIBRARY_PATH;
LOADEDMODULES=gcc/12.3.0; export LOADEDMODULES;

That is the shape of it, abridged: a real Lmod run also emits _LMFILES_, reference counts and its serialized module table, which is noise for this purpose but worth seeing once.

Two consequences follow. First, the effect of module load is confined to the shell that ran the eval; put it anywhere the shell forks and it lands in a child that exits immediately. Second, the shell code has to have stdout to itself, so diagnostics go somewhere else — and where they go depends on which implementation you have. Lmod writes the output of module list, module avail and friends to stderr by default; LMOD_REDIRECT=yes (or --redirect) moves it to stdout, and the Lmod documentation notes that only works for bash and zsh, not csh or tcsh. Tcl Environment Modules went the other way: current releases document a redirect_output configuration option that defaults to on, so on sh, bash, ksh, zsh and fish the messages arrive on stdout unless someone passes --no-redirect. Check yours with module config rather than assuming either default.

The practical consequence is that module list | grep works on one of your clusters and silently returns nothing on the other, and the reader concludes the module is not loaded. Redirect explicitly and the question does not arise:

$ module list | grep -c gcc      # 0 under Lmod's default, 1 under Modules' default
$ module list 2>&1 | grep -c gcc # 1 under both

For anything you intend to parse rather than read, module -t list (terse) is a better input than the decorated human output.

1. A load inside a pipe or a command substitution goes nowhere

This is the one that costs hours, because every line of output looks correct.

$ cat wanted.txt | while read -r m; do module load "$m"; done
$ module list 2>&1
No modules loaded
$ samtools --version
bash: samtools: command not found

No error was printed and the loads succeeded. In bash each stage of a pipeline runs in a subshell, so the while loop — and the eval inside module — executed in a forked child. The child exported the variables and exited. The parent shell was never touched. (Bash has one escape hatch here, shopt -s lastpipe, which runs the final stage in the current shell — but only when job control is off, so it does nothing in an interactive session, which is exactly where people test it.)

The command substitution version is worse, because the inner answer is right:

$ VER=$(module load python/3.11.6 && python3 --version)
$ echo "$VER"
Python 3.11.6
$ python3 --version
Python 3.9.18

The load did work — inside the subshell, where python3 really was 3.11.6, which is why the captured string looks like proof that the module is loaded. The parent shell has nothing loaded, and the next line in the script is the one that fails.

Two more places the fork is invisible: every recipe line in a makefile runs in its own shell, so module load on one line and $(CC) on the next are unrelated environments, and xargs execs a fresh process per batch. Over ssh without a login shell the function is often not defined at all.

The fixes are mechanical — a process-substitution loop keeps the body in the current shell, make wants && or .ONESHELL:, and ssh wants the init script sourced explicitly.

$ while read -r m; do module load "$m"; done < <(grep -v '^#' wanted.txt)
$ ssh node 'source /usr/share/lmod/lmod/init/bash && module load gcc/12.3.0 && gcc --version'

The diagnostic that settles it in one line: after the load, print LOADEDMODULES in the shell you care about. If it is empty there, the load happened somewhere else.

2. `which` says one thing and a different binary runs

This is the section where the obvious reading is wrong twice — once about the symptom, and once about the fix nearly everyone prescribes for it.

$ module load python/3.11.6
$ which python3
/opt/apps/python/3.11.6/bin/python3
$ python3 --version
Python 3.9.18

The standard answer is that bash has cached the old location and you need hash -r. Test that before believing it, because in bash it is not what happened:

$ mkdir -p /tmp/ht/a /tmp/ht/b
$ printf '#!/bin/sh\necho A\n' > /tmp/ht/a/foo
$ printf '#!/bin/sh\necho B\n' > /tmp/ht/b/foo
$ chmod +x /tmp/ht/a/foo /tmp/ht/b/foo
$ PATH=/tmp/ht/a:$PATH
$ foo                                        # runs, and is now hashed
A
$ eval 'PATH=/tmp/ht/b:$PATH; export PATH;'  # exactly what module load evaluates
$ foo
B

Assigning to PATH makes bash discard every remembered location — plainly, with export, from inside a function, or inside the eval that the module function runs. All four behave the same way on the bash used for the transcripts here (3.2.57); the test above takes two minutes on whatever bash your cluster actually runs, and is worth doing before you accept either account. zsh behaves the same, and tcsh rebuilds its hash when path is set. A module load is a PATH assignment, so it cannot leave a stale entry behind, and hash -r after module load is a ritual aimed at a failure bash does not have. Prescribing it costs you the one thing you were short of: a reason to look somewhere else.

Two things which genuinely cannot see, and one that it can.

A function or an alias beats PATH outright, and which is an external program that knows only PATH. Sites wrap interpreters this way, users copy the wrapper into .bashrc, and module itself is a shell function, so nothing here is exotic:

$ type python3
python3 is a function
python3 () 
{ 
    /usr/bin/python3 "$@"
}

That is what the transcript at the top of this section really was: PATH changed exactly as asked, and a function defined in a site profile went on answering to the name.

command python3 --version bypasses it for one call, unset -f python3 for the session. Note that command -v python3 prints just python3 for a function — correct, and useless as evidence. type is the builtin to reach for. Aliases produce the same symptom interactively and then disappear in batch, because bash does not expand aliases in non-interactive shells unless expand_aliases is set: that is the “works in my shell, fails in my job script” report.

The hash cache is real, but its trigger is the opposite of the folklore — it bites when PATH does not change and a same-named binary appears earlier on it. A pip install --user into ~/.local/bin, a wrapper dropped into /usr/local/bin, an install finishing while the user’s shell is open:

$ PATH=/tmp/ht/a:/tmp/ht/b:$PATH        # a is earlier and has no bar yet
$ cp /tmp/ht/b/foo /tmp/ht/b/bar; bar
B
$ cp /tmp/ht/a/foo /tmp/ht/a/bar        # an earlier bar now exists; PATH unchanged
$ which bar
/tmp/ht/a/bar
$ type bar
bar is hashed (/tmp/ht/b/bar)
$ bar
B
$ hash -r; bar
A

One detail worth knowing before you rely on the diagnostic: on the bash tested here, type -a bar re-searched PATH and listed both directories without mentioning the cache at all, while plain type bar named it. Use type without -a, or the hash builtin, which prints the table with hit counts. In tcsh the equivalent situation needs rehash, and there which is a builtin that consults the same cache, so it agrees with the wrong answer rather than contradicting it.

3. The module that sets a variable but no interpreter

A common pattern for Python tooling is a modulefile pointing at a virtual environment:

setenv    TOOLKIT_VENV  /shared/apps/toolkit/1.4/venv
setenv    PYTHONPATH    /shared/apps/toolkit/1.4/venv/lib/python3.11/site-packages
prepend-path PATH       /shared/apps/toolkit/1.4/bin

The user does the obvious check and gets a genuinely misleading answer:

$ module load toolkit/1.4
$ python3 -c "import toolkit"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'toolkit'

The package is installed, under an interpreter that is not on PATH. python3 here is still the system 3.9, and PYTHONPATH has pointed it at a site-packages built for 3.11 — so on top of the version mismatch, any compiled extension there is named _core.cpython-311-x86_64-linux-gnu.so and 3.9 will not look at it.

Ask the interpreter where it is, not whether the import worked:

$ python3 -c "import sys; print(sys.executable); print(sys.version_info[:2])"
/usr/bin/python3
(3, 9)
$ "$TOOLKIT_VENV/bin/python" -c "import toolkit; print(toolkit.__version__)"
1.4.0

That is the diagnosis. The fix is to put the venv’s bin on PATH and stop setting PYTHONPATH — a venv does not need it, and setting it turns a clean “wrong interpreter” failure into a confusing “half the packages import” one.

prepend-path PATH  /shared/apps/toolkit/1.4/venv/bin
unsetenv     PYTHONPATH

PYTHONPATH deserves a blanket rule: a shared modulefile should never set it. Its entries are searched ahead of the interpreter’s own standard library and site-packages — everything except the script’s own directory — and they leak into every unrelated Python process started from that session, including ones belonging to other modules.

4. The module that does put an interpreter on the path, and shadows everything

The opposite failure is quieter and lasts longer. A module that prepends a Python to PATH has changed the meaning of python3 for everything else in that shell. Usually that is wanted. The parts that are not:

a script with #!/usr/bin/env python3 now runs under the module’s interpreter regardless of what it was tested against; pip install --user writes into ~/.local/lib/python3.11/, which then becomes visible to every other 3.11 on the system; python3 -m venv creates an environment whose pyvenv.cfg records the module’s interpreter, and that venv keeps working right up until the module directory is deleted.

Load order decides the winner, and module list will not tell you who won. Ask for the resolution:

$ module load toolA toolB
$ type -a python3 | head -3
python3 is /opt/apps/toolB/2.1/bin/python3
python3 is /opt/apps/toolA/1.9/bin/python3
python3 is /usr/bin/python3

The structural fix is family. Lmod’s family("python") makes modules in the same family mutually exclusive, so the second load swaps the first rather than shadowing it:

family("python")
prepend_path("PATH", "/opt/apps/python/3.11.6/bin")

$ module load python/3.11.6 python/3.12.2
Lmod is automatically replacing "python/3.11.6" with "python/3.12.2".

That message is the point. The load still succeeds, but the user now knows.

For application modules the better pattern is not to expose an interpreter at all: ship wrappers in bin/ with an absolute shebang pointing at the venv’s Python, and put only that bin on PATH. The tool works; python3 keeps meaning what it meant.

5. Several modules in one command

A recurring ticket: loading modules one per line works, loading them in a single command fails, and the user is told to stop doing the thing that failed without anyone explaining why.

$ module purge
$ module load compiler/gcc-12.3.0 mpi/openmpi-5.0.3 app/solver-4.2
$ solver --version
solver: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory
$ module purge
$ module load compiler/gcc-12.3.0
$ module load mpi/openmpi-5.0.3
$ module load app/solver-4.2
$ solver --version
solver 4.2.1

Two mechanisms are in play, and neither is as simple as it is usually told — including in the direction most people assume.

The first is that what a child process launched from a modulefile sees is an implementation detail, not the environment the user will end up with, and the two implementations differ. Tcl Environment Modules documents that setenv “will also change the process’ environment” and that it “is useful for changing the environment prior to the exec or system command” — so there, a value set by an earlier sibling in the same command is visible to an exec in a later one. Lmod assembles the new environment internally and emits it at the end, and its two ways of running a command differ on exactly this point: the documentation for capture() says it uses the LD_PRELOAD and LD_LIBRARY_PATH values that were current when Lmod was configured, and directs you to subprocess() if you want the values in force now. So a modulefile doing this:

set mpiroot $env(MPI_ROOT)
set mpiver  [exec $mpiroot/bin/mpirun --version]

can behave one way on the cluster running Modules, another on the cluster running Lmod, and a third way when the module that sets MPI_ROOT is loaded in a separate command rather than as a sibling — reading the modulefile tells you nothing about which. The same applies to any [file exists $env(SOMETHING)/...] test fed by a sibling.

The second: whether prerequisite checks and hierarchical MODULEPATH additions are honored within a single command varies across Tcl Environment Modules 3.2, 4.x and 5.x and across Lmod 6 through 9. I am not going to give a version table I cannot stand behind. You can settle it for your own stack in about thirty seconds:

$ module purge
$ module load compiler/gcc-12.3.0 mpi/openmpi-5.0.3; echo "exit=$?"
$ module list 2>&1

If module list shows fewer modules than you asked for while exit=0, your implementation loads partially and does not report it. Know that before building automation on top of it.

Two rules fall out regardless of version. In scripts, load one module per command and check the result by inspecting LOADEDMODULES rather than trusting the exit status. And never write a modulefile that runs an external command at load time; compute the value at install time and hard-code it.

6. Class file version 61.0, and what the numbers mean

A Java tool that was built elsewhere produces this:

$ module load gatk/4.5.0.0
$ gatk --version
Error: LinkageError occurred while loading main class org.broadinstitute.hellbender.Main
	java.lang.UnsupportedClassVersionError: org/broadinstitute/hellbender/Main has been
	compiled by a more recent version of the Java Runtime (class file version 61.0),
	this version of the Java Runtime only recognizes class file versions up to 52.0

The message names two numbers and explains neither. They are class file format major versions, not Java versions, and for Java 5 onward major version = Java release + 44. So 61 is Java 17 and 52 is Java 8: the tool needs 17 and the node’s default is 8.

Class file major	Java release	Class file major	Java release
49	5	60	16
50	6	61	17
51	7	62	18
52	8	63	19
53	9	64	20
54	10	65	21
55	11	66	22
56	12	67	23
57	13	68	24
58	14	69	25
59	15

Below that, 45 is Java 1.1 through 48 for 1.4. The .0 after the major is the minor version; it is zero except for preview features, where it is 65535.

You do not need the table if you read the bytes. A class file is CAFEBABE, a two-byte minor at offset 4, then a two-byte major at offset 6, big-endian:

$ unzip -p gatk.jar org/broadinstitute/hellbender/Main.class \
    | od -An -tu2 --endian=big -j6 -N2
  61

(--endian is a GNU coreutils option; on a box without it, od -An -tx1 -j6 -N2 and read the two bytes by hand.)

Or, if a JDK is to hand:

$ javap -verbose -cp gatk.jar org.broadinstitute.hellbender.Main | grep -E 'major|minor'
  minor version: 0
  major version: 61

The manifest often records the build JDK (Build-Jdk-Spec: 17), which is a useful cross-check but is not authoritative — it reflects the compiler, not the --release target.

One wrinkle before you conclude anything from a single class: a multi-release jar carries extra copies under META-INF/versions/N/ that are deliberately compiled for higher majors than the jar’s baseline. If Multi-Release: true is in the manifest, check a class from the root of the jar instead.

The fix is a prereq in the modulefile, so the tool cannot load without its runtime — not a wiki line telling people to load Java first:

prereq java/17
setenv JAVA_HOME /opt/apps/java/17.0.9

7. The modulefile that loads cleanly and sets a variable to nothing

Modulefiles get copied between clusters. The copy succeeds, the load succeeds, the software is not there.

$ module load fsl/6.0.7
$ echo "FSLDIR=$FSLDIR"
FSLDIR=/opt/apps/fsl/6.0.7
$ ls "$FSLDIR"
ls: cannot access '/opt/apps/fsl/6.0.7': No such file or directory
$ bet --help
bash: bet: command not found

In their default configurations neither Environment Modules nor Lmod checks that a path exists before putting it in a variable, and neither treats a prepend-path onto a missing directory as an error. There is a defensible reason — automounted directories may not be materialized at load time — but the result is a module that reports success and delivers nothing.

Harder to spot is the variant where the variable ends up empty rather than wrong. $FSLDIR/bin/bet then expands to /bin/bet, and where that path happens to exist you get a different program and no error at all.

Make the modulefile assert its preconditions. In Tcl, break ends the evaluation there; Modules documents that the module is then not listed as loaded and other modules being loaded concurrently are unaffected, with everything up to that point still performed:

set root /opt/apps/fsl/6.0.7

if { [module-info mode load] && ![file isdirectory $root] } {
    puts stderr "ERROR: fsl/6.0.7 is not installed on [info hostname] ($root missing)"
    break
}

setenv       FSLDIR $root
prepend-path PATH   $root/bin

The module-info mode load guard matters: without it the check also runs during module display, module avail and unload, which is noisy and can break module purge on a node where the directory genuinely is absent. Note also that Modules documents break on unload as advisory — a module unload --force proceeds anyway — so do not use it to make a module unremovable.

In Lmod the equivalent is if (mode() == "load" and not isDir(root)) then LmodError(...) end.

8. The deprecated module that loads instead of failing

A version is retired and its install directory removed, but the modulefile stays because deleting it would “break people’s scripts”. It breaks them anyway, later and less legibly: the load succeeds, PATH gains a directory that is not there, and the user gets command not found from a module they can see in module list.

Lmod’s admin file is right for a module that still works but should not be used. It is a list of keys — a module name, or a full path to a modulefile when the key begins with / — each followed by a message that runs to the next blank line:

$ cat /opt/apps/lmod/etc/admin.list
samtools/1%.9:
    Retired 2026-09-30. Use samtools/1.19.

Its location is wherever LMOD_ADMIN_FILE was configured to point, which module --config will tell you.

Those keys are Lua patterns rather than literal strings, which is the detail that catches people. . matches any character and - is a quantifier, so a version number wants %. and a module name containing a dash wants %-. Written as plain samtools/1.9 the entry still matches samtools/1.9, and also samtools/119.

Loading a module that matches prints a block in this shape and then loads it:

$ module load samtools/1.9

-----------------------------------------------------------------
There are messages associated with the following module(s) :
-----------------------------------------------------------------
samtools/1.9:
    Retired 2026-09-30. Use samtools/1.19.
-----------------------------------------------------------------

Note what that does: it prints and loads. The Lmod documentation is explicit that the admin file “in no way controls user access to the module”. That is correct for a deprecation window and wrong for a module whose files are gone. For that, the modulefile must refuse:

LmodError([[
samtools/1.9 was removed on 2026-09-30. The installation no longer exists.
Use: module load samtools/1.19
]])

That refuses the load, and Lmod documents that a failed load returns a non-zero exit code you can trap. Confirm that on your own version before a job script leans on it, and confirm it separately under Tcl Environment Modules, where the exit status of a failed load has not always been reliable.

State the principle plainly, because it is the one people argue about: a module that cannot do its job must fail at load time, loudly, with a non-zero exit. A job that dies in the first second with a clear message is a five-minute fix. A job that runs nine hours and fails at the output stage because one of six tools was missing costs a day, and nobody will connect it to a module that loaded successfully that morning.

9. Build-host paths baked into a shared installation

This one is not the module system’s fault, but the module system is where it surfaces: software that works perfectly on the node it was compiled on and nowhere else.

$ module load rstudio-deps/2026.05
$ R -e 'library(sf)'
Error: package or namespace load failed for 'sf':
 unable to load shared object '/shared/apps/R/4.4.1/library/sf/libs/sf.so':
  libproj.so.25: cannot open shared object file: No such file or directory

It works on the build node because the library was in that node’s /usr/lib64, or because the build shell had an LD_LIBRARY_PATH no other shell has. Four things get baked in:

# 1. RPATH/RUNPATH pointing at a build-only directory
$ readelf -d /shared/apps/tool/bin/tool | grep RUNPATH
 0x000000000000001d (RUNPATH)   Library runpath: [/scratch/build/deps/lib]

# 2. libraries resolving outside the install tree, or not at all
$ ldd /shared/apps/tool/bin/tool | grep -v '=> /shared'
	libproj.so.25 => not found

# 3. build paths in generated config, which propagate to everything users compile later
$ grep -n '/scratch/build' /shared/apps/R/4.4.1/lib64/R/etc/Makeconf
128:LDFLAGS = -L/scratch/build/deps/lib

# 4. pkg-config and libtool files carrying an absolute prefix
$ grep -rl '/scratch/build' /shared/apps/tool/lib/pkgconfig/*.pc /shared/apps/tool/lib/*.la

A fifth catches conda-based installs and looks like a filesystem problem: the shebang length limit. The kernel reads a fixed-size buffer from the start of an executable, and a #! line longer than that buffer does not survive it. On a kernel with the older 128-byte buffer the line is truncated and the truncated path is executed, so the error names an interpreter that does not exist while ls shows the real one plainly does:

$ head -1 /gpfs/shared/apps/genomics/pipelines/annotation/1.2.0/bin/annotate
#!/gpfs/shared/apps/genomics/pipelines/annotation/1.2.0/conda/envs/annotation-tools-py311-bioconda-pinned-2026-05-rebuild/bin/python3.11
$ head -1 /gpfs/shared/apps/genomics/pipelines/annotation/1.2.0/bin/annotate | wc -c
137
$ ./annotate
bash: ./annotate: /gpfs/shared/apps/genomics/pipelines/annotation/1.2.0/conda/envs/annotation-tools-py311-bioconda-pinned-2026-05-rebuild/bin/py: bad interpreter: No such file or directory

Read the truncation point rather than the message: the path in the error stops at 126 characters, which is the 128-byte buffer less the two bytes taken by #!. That is the signature.

The number is a kernel compile-time constant, BINPRM_BUF_SIZE in include/uapi/linux/binfmts.h, and it is 256 in current mainline. Current kernels also handle the overflow differently: fs/binfmt_script.c returns -ENOEXEC when the interpreter path runs to the end of the buffer with no terminator after it, rather than executing a path it had to cut. That changes the symptom instead of removing it — a shell that gets ENOEXEC falls back to running the file itself as a shell script, so a Python entry point fails with shell syntax errors from its own source instead of bad interpreter. Check the constant on the kernel in front of you rather than quoting either number. The fix is the same in both cases: a shorter install prefix, or a small POSIX sh wrapper that execs the real interpreter.

The rule that prevents all five: never validate a build on the node that produced it. Build anywhere; test on a node that has never had the toolchain loaded.

The principle, and a harness

A modulefile is a contract: load me and this tool will run. A contract that fails silently is worse than no contract, because the user stops checking. If a module cannot keep its promise it must say so at load time, in the shell the user is sitting in, with a non-zero exit.

Hold it to that by testing the promise directly — load it in a shell with nothing inherited and execute the binary. Not ls the directory, not check the variable. Run the thing.

#!/bin/bash
# modcheck.sh — load each module in a clean shell and actually run its binary.
# Manifest is tab-separated:  <module>  <binary>  <smoke arguments>
set -u
MANIFEST=${1:?usage: modcheck.sh manifest.tsv}
INIT=${MODULE_INIT:-/usr/share/lmod/lmod/init/bash}
MP=${MODULEPATH:?MODULEPATH must be set in the calling shell}
fail=0

while IFS=$'\t' read -r mod bin args || [[ -n ${mod:-} ]]; do
    [[ -z ${mod:-} || ${mod:0:1} == "#" ]] && continue

    out=$(env -i \
            HOME="$HOME" USER="$USER" LOGNAME="$USER" TERM=dumb \
            PATH=/usr/bin:/bin MODULEPATH="$MP" \
          bash --noprofile --norc -c '
            source "$1" >/dev/null 2>&1 || exit 91
            set -u
            module load "$2" >/dev/null || exit 92
            case ":${LOADEDMODULES:-}:" in
              *":$2:"*|*":$2/"*) ;;
              *) exit 92 ;;
            esac
            command -v "$3" >/dev/null  || exit 93
            exec "$3" $4
          ' _ "$INIT" "$mod" "$bin" "${args:-}" 2>&1)
    rc=$?

    [[ $rc -ne 0 ]] && fail=1
    case $rc in
      0)  printf 'ok    %-28s %s\n' "$mod" "$bin" ;;
      91) printf 'INIT  %-28s init script missing: %s\n' "$mod" "$INIT" ;;
      92) printf 'LOAD  %-28s module did not load\n' "$mod" ;;
      93) printf 'PATH  %-28s loaded, but %s is not on PATH\n' "$mod" "$bin" ;;
      *)  printf 'RUN   %-28s %s exited %d: %s\n' "$mod" "$bin" "$rc" \
                 "$(printf '%s' "$out" | head -1)" ;;
    esac
done < "$MANIFEST"

exit $fail

Run against a manifest of four modules — one good, one that sets variables but puts no binary on PATH, one whose binary runs and fails, and one whose modulefile is gone:

$ ./modcheck.sh manifest.tsv
ok    samtools/1.19                samtools
PATH  toolkit/1.4                  loaded, but toolkit is not on PATH
RUN   gatk/4.5.0.0                 gatk exited 1: Error: LinkageError occurred while loading main class org.broadinstitute.hellbender.Main
LOAD  samtools/1.9                 module did not load
$ echo $?
1

Several things there are deliberate. env -i stops your interactive session leaking in, which is the only way to catch a module that works for you because of your .bashrc. --noprofile --norc stops the site profile quietly fixing the problem. set -u is applied after the init script is sourced, because module init scripts are not written to survive it. The load is checked twice — exit status and then LOADEDMODULES — which is the rule from section 5 applied to the harness itself; a partial load reported as success is exactly what this is meant to catch. The sentinel exit codes are 91 to 93 rather than 2 to 4 so that a tool exiting 3 on its own account is not misfiled as a load failure. And the exec runs the real binary rather than testing for its presence — $4 is deliberately unquoted so the smoke arguments word-split.

Run it on more than one node — that is what catches the build-host class:

$ for n in $NODES; do echo "== $n"; srun -N1 -w "$n" --time=10 ./modcheck.sh manifest.tsv; done

Run it from a batch job as well as a login node. Those environments differ, and a module that only works interactively fails for every job.

Symptom against cause

Symptom	Likely cause	Command that confirms it
`command not found` right after a successful load	load happened in a subshell (pipe, `$( )`, `xargs`, make recipe)	`echo "$LOADEDMODULES"` in the shell that matters
`module list` shows nothing but the load printed no error	same, or the output went to stderr (Lmod’s default) and was filtered by a pipe	`module list 2>&1`, or `module -t list`
`which` shows the new binary, old version still runs	a shell function or alias of the same name, which `which` cannot see	`type <cmd>`; `command <cmd>` to bypass once
Same, and `type` says “hashed”	stale command hash — only possible when `PATH` itself did not change	`type <cmd>`, `hash`; clear with `hash -r` (`rehash` in tcsh)
`ModuleNotFoundError` for a package that is installed	module set a venv variable but no interpreter on `PATH`	`python3 -c "import sys; print(sys.executable)"`
Unrelated Python scripts break after loading a tool	module prepended an interpreter and shadowed the system one	`type -a python3`; fix with `family()`
Two modules loaded, second one’s tool wins silently	no `family`/`conflict` declared	`module list 2>&1` then `type -a` on the contested binary
Works loaded one per line, fails in a single command	modulefile shelling out, or a partial load reported as success	load in one command, then `module list 2>&1` and compare against what you asked for
`UnsupportedClassVersionError: class file version N`	tool built for a newer JDK than the default	Java release = N − 44; add `prereq java/<n>`
Variable set, directory absent, no error	modulefile copied between clusters without an existence check	`ls "$THAT_VAR"`; add a `break` / `LmodError` guard
`$VAR/bin/tool` resolves to `/bin/tool`	variable expanded to empty string	`echo "[${VAR:-UNSET}]"`
Module in `module list`, binary missing from disk	retired install, modulefile left behind	`module show <mod> 2>&1`, then `ls` each path it prints
`libX.so.N: cannot open shared object file`	build-host library not present on this node	`ldd <binary>`; `readelf -d <binary> \\| grep RUNPATH`
`GLIBCXX_3.4.NN not found`	built against a newer libstdc++ than the runtime provides	`strings <libstdc++ in use> \\| grep GLIBCXX \\| sort -V \\| tail -1`
`bad interpreter: No such file or directory`, file exists	shebang longer than the kernel’s buffer (`BINPRM_BUF_SIZE`), path truncated	`head -1 <script> \\| wc -c` and compare with the length in the error text
Shell syntax errors from a Python or Perl entry point	same overflow on a newer kernel: `ENOEXEC`, and the shell then runs the file itself	`head -1 <script> \\| wc -c` against `BINPRM_BUF_SIZE` on that node
Works on a login node, fails in a batch job	different profile scripts or `MODULEPATH`	run the same check under `srun` and diff the environments
Works for you, fails for a user	something in your `.bashrc`	re-test under `env -i ... bash --noprofile --norc`

None of this is exotic. It is one failure repeated: an environment change that did not happen, or happened where it could not be observed, reported as success. It is worth writing down because the evidence points at the software, and the software is fine.

Restore benchmarks: measure the time until the business can work

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A restore speed of several gigabytes per second can sound reassuring while saying little about the time needed to recover a real service. The benchmark must include the dataset shape and the work that happens before and after copying bytes.

Define the finish line

Choose a service and define the recovery-time objective with its owner. Separate data availability from application readiness and business acceptance. Record the recovery point and acceptable data-loss window independently.

Use a representative dataset: large files, small files, directory depth, permissions, and application metadata. A single large sequential file is an inadequate model for a namespace containing millions of small objects.

Establish a lower bound

A simple transfer-time calculation is useful as a sanity check:

100 TiB / 2 GiB per second = 51,200 seconds = about 14.2 hours

This is arithmetic, not a measured product result. It excludes initialization, metadata work, protocol overhead, retries, and application checks. Use consistent units and actual sustained throughput from the intended recovery route.

If the business requires recovery in four hours, a fourteen-hour byte-transfer lower bound already demonstrates a design mismatch. No dashboard wording can close that gap.

Time the complete sequence

Milestone	Record
Recovery authorized	Start timestamp
Credentials and keys available	Access preparation delay
Destination ready	Provisioning delay
Namespace available	Initial mount or restore completion
Working set read	Cold-access completion
Application accepted	Business validation timestamp

WEKA’s Snap-To-Object documentation notes on-demand recovery considerations. Similar distinctions should be investigated for any product that exposes data before all of it is local. Do not equate a quick mount with a fully recovered working set.

Avoid cache-driven optimism

Run the first representative read separately from repeat reads. Record whether source or destination caches were warm and whether object-store retrieval was involved. A second run can measure a different system path.

Measure file-count and metadata progress as well as bytes. Preserve failure and retry counts. A benchmark that silently skipped unreadable files can report excellent throughput while failing its purpose.

Validate data and authority

Compare a pre-recorded manifest, checksums for the test corpus, ownership, and expected access. Then execute an application transaction. Include a denied-access test for an unrelated user so the restored service does not trade availability for excessive exposure.

Report the actual result

Publish the dataset, tool versions, topology, elapsed milestones, throughput, errors, and manual interventions. Label estimates as estimates and lab results as lab results. Do not extrapolate a small trial linearly without explaining the limits.

The useful deliverable is a recovery time the business can assess, together with the bottleneck that must change if it is too slow.

Incident containment: preserve the route you need to investigate and recover

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Disconnecting a compromised system may limit damage, but an unplanned isolation step can also remove the only management route, interrupt evidence collection, or strand a critical dependency. A containment playbook should make those tradeoffs explicit before an emergency.

Establish authority and scope

Identify who can authorize host isolation, account revocation, storage protection changes, and service shutdown. Keep an incident communication route available outside the systems under investigation.

Record the known facts separately from hypotheses. A suspicious alert may justify precautionary isolation, but the incident record should not claim confirmed compromise solely because a detection fired.

CISA’s ransomware guide provides incident-response and recovery guidance. The worksheet below is an operational method for applying those objectives to a specific environment.

Build a containment decision sheet

Action	Intended effect	Dependency to preserve
EDR network isolation	Restrict host communication	Supported responder/management channel
Account revocation	Stop further identity use	Independent recovery authority
Firewall restriction	Limit affected traffic	DNS, logging, or recovery flows as justified
Suspend replication workflow	Avoid propagating harmful state	Existing protected recovery points

These are possible actions, not automatic instructions. The incident lead selects them based on scope, platform behavior, and business impact.

Rehearse the endpoint control

For an EDR product such as Microsoft Defender for Endpoint or another deployed platform, use its current vendor procedure on a lab endpoint. Establish which communication remains available during isolation and how authorized responders reverse it.

Do not assume all operating systems or agent versions behave identically. Observe the lab endpoint from a second host, verify the responder path, and time restoration. Record what happens if the agent is unhealthy or the endpoint is already offline.

Preserve evidence deliberately

Capture relevant timestamps, alerts, host identity, and action records. Decide with the response team which volatile evidence must be collected before disruptive actions. Avoid publishing sensitive logs in a general collaboration channel.

Containment should not trigger uncontrolled cleanup. Deleting files or rebuilding immediately can remove evidence needed to establish scope and root cause. Preserve recovery options while the incident team makes that decision.

Separate isolation from recovery acceptance

A host can be contained and still be unsafe to reconnect. Establish the criteria for rebuilding or restoring it, rotating exposed credentials, and accepting the recovered application. Confirm the selected recovery point and its dependencies rather than restoring the most recent backup automatically.

Use a tabletop scenario that includes loss of the normal directory and backup console. It reveals whether the procedure depends on the same authority that may be compromised.

Record outcomes, not just button clicks

Keep the requested action, observed network effect, remaining access, evidence preserved, and restoration result. The product console’s “isolated” label should be corroborated by the behavior the team intended.

Continue with recovery after administrator compromise.

Ninety-seven percent of an annotation pipeline was annotating nothing

2026-06-24T10:00:00-04:00

A variant annotation pipeline takes per-sample genomic VCF files in and produces annotated variants out. It is slow. The obvious response is to make the annotator faster: more forks, faster storage for the cache, a different annotation tool, a bigger node. All of those are real levers and all of them are the wrong place to start.

The right place to start is a stopwatch on each stage, because the shape of this problem recurs: the annotator is not slow, it is being asked to do roughly thirty-eight times more work than the question requires. The records it spends its time on carry no alternate allele at all. There is nothing to annotate, so it annotates nothing, carefully, twenty-five million times per sample.

What follows is the pattern — how to measure it, the pipeline that removes the waste, the correctness trap that eats sites without telling you, the two tuning parameters that only showed up in a sweep, and the caveat that limits the whole result to one file format.

Measure the stages before touching anything

The first number you need is the split of wall time across stages. For a shell pipeline this does not need a profiler. It needs timestamps that survive into the log:

stamp() { printf '%s\t%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$1" >> "$TIMING"; }

stamp "start"
bcftools index -t "$GVCF"                                   ; stamp "index"
run_annotator "$GVCF" "$OUT"                                ; stamp "annotate"
postprocess "$OUT"                                          ; stamp "postprocess"

If the work runs under Slurm, take the per-step figures from the accounting database rather than from the job script, and take them from the job steps, not the allocation:

$ sacct -j 4182377 --units=G \
    --format=JobID,JobName%18,Elapsed,TotalCPU,MaxRSS,State
JobID          JobName      Elapsed   TotalCPU     MaxRSS      State
4182377        annot-batch  05:02:11  19:55:38                COMPLETED
4182377.batch  batch        05:02:11  00:01:46      0.31G     COMPLETED
4182377.0      vep          04:53:02  19:41:08         9.8G   COMPLETED
4182377.1      post         00:08:57  00:12:44      1.20G     COMPLETED

Read the TotalCPU column carefully: it is [DD-]HH:MM:SS, so a leading day field changes the value by a factor of twenty-four and is easy to skim past. Here 19:41:08 of CPU against 04:53:02 of wall on the annotation step is 4.03x, which is the four forks the step was given and a useful sanity check that the parallelism you asked for is the parallelism you got.

Two further warnings about that output, both of which produce confidently wrong answers. MaxRSS is blank on the allocation line because the allocation itself has no measured process tree, so any memory figure derived from a query with -X comes back as zero for every job. And if you derive CPU efficiency from CPUTimeRAW rather than TotalCPU you get 1.0 for every job ever run, because CPUTimeRAW is allocated core-seconds and not consumed ones. Use TotalCPU over NCPUS * Elapsed, and query without -X.

The split from that run, for one representative sample on one node:

annotation: 293.0 min
everything else — indexing, splitting, post-processing, compression, QC summary: 9.0 min
total: 302.0 min

Annotation is 97.0% of the wall time. Nothing else on the list can matter. If you halved the cost of every other stage you would save four and a half minutes out of five hours.

What is actually in the file

A per-sample genomic VCF is not a list of variants. It is a description of the whole callable genome, in which the overwhelming majority of records say “this stretch matches the reference and here is how confident I am”. Those records carry a symbolic placeholder allele instead of a real one: <NON_REF> for GATK and DRAGEN output, <*> for htslib-style gVCF. Reference blocks also carry an END tag giving the last position of the block.

The important structural detail is that in a GATK-style gVCF the placeholder is present on every record, including real variant records, where it sits last in the ALT list. So a reference block looks like ALT=<NON_REF> and a variant looks like ALT=A,<NON_REF>. The test for “this record has something to annotate” is therefore not “does the ALT column mention the placeholder” — it is “does the ALT column contain anything besides the placeholder”.

That distinction has teeth in bcftools expression syntax, because a comma inside a string is read as a value separator and multiple values are combined with OR. ALT="<NON_REF>" is therefore true for ALT=A,<NON_REF> — it matches the second value and stops caring about the first. Written alone as an exclude expression it would discard every variant record in the file. The N_ALT=1 conjunction is what narrows it to records where the placeholder is the only allele, and it is the whole reason the filter is two clauses rather than one.

$ bcftools view -H sample.g.vcf.gz | head -4 | cut -f1-5,10
chr1  10001  .  T  <NON_REF>    0/0:...
chr1  10002  .  A  <NON_REF>    0/0:...
chr1  10108  .  C  CAACCCT,<NON_REF>  0/1:...
chr1  10109  .  A  <NON_REF>    0/0:...

Count both classes:

$ bcftools view -H sample.g.vcf.gz | wc -l
26584113

$ bcftools view -H -e 'N_ALT=1 && ALT="<NON_REF>"' sample.g.vcf.gz | wc -l
690412

690,412 out of 26,584,113 is 2.60%. Removing the rest is a 38.5-fold reduction in records entering the stage that is 97% of the cost.

Two honest caveats on those figures. They are one file, and absolute record counts in gVCFs are not portable: reference-block banding differs sharply between callers and settings, and a file emitted at base-pair resolution has a record per callable base, which is two orders of magnitude more. The ratio is the durable number, and across the per-sample whole-genome gVCFs I have counted it sits in the low single digits of a percent. If your own files give a ratio above about ten percent, either they are not per-sample gVCFs or the caller banding is unusual, and you should check before assuming this result transfers.

Cross-check the classification against the END tag rather than trusting a single expression:

$ bcftools query -f '%INFO/END\n' sample.g.vcf.gz | grep -cv '^\.$'
25893701

25,893,701 records carry an END, and 26,584,113 − 690,412 = 25,893,701. Two tests reading two different fields — ALT and INFO/END — agree exactly, which is the point of running both. They are not fully independent, since a caller that got one wrong could plausibly get the other wrong the same way, but they do catch the failure that actually happens, which is an expression that means something other than what you thought it meant.

The pipeline

Order matters, for reasons covered in the next section. Filter first, split multi-allelic records second, drop the placeholder third.

set -euo pipefail

REF=/ref/GRCh38.fa
IN=sample.g.vcf.gz
SITES=sample.sites.vcf.gz

bcftools view --threads 4 -e 'N_ALT=1 && ALT="<NON_REF>"' -Ou "$IN" \
  | bcftools norm -m -any -f "$REF" -Ou \
  | bcftools view --threads 4 -e 'ALT="<NON_REF>" || ALT="*"' -Oz -o "$SITES"

bcftools index -t "$SITES"

--threads appears only on the first and last commands on purpose. It is a BGZF compression and decompression pool, so it buys something where a bgzipped file is being read and where one is being written, and nothing at all on the middle stage, which takes uncompressed BCF on a pipe and emits the same. Putting it everywhere is harmless but it is cargo cult, and it makes the stage look more parallel than it is.

After norm -m -any, a record with ALT=A,<NON_REF> becomes two records, one with ALT=A and one with ALT=<NON_REF>, so the final view removes the now-standalone placeholders. The same pass drops spanning-deletion * alleles. A * is not a new allele: it marks a position overlapped by a deletion called at an earlier position, so there is no consequence to compute for it. Drop them deliberately rather than letting the annotator decide.

Then annotate the small file:

vep --offline --cache --dir_cache /ref/vep --assembly GRCh38 \
    --fasta "$REF" \
    --format vcf --vcf --compress_output bgzip \
    --allele_number \
    --no_stats \
    --fork 4 --buffer_size 50000 \
    -i "$SITES" -o sample.vep.vcf.gz

--allele_number is the one flag here whose purpose is routinely misread, including by me when I first wrote this pipeline. The documentation says it will “identify allele number from VCF input, where 1 = first ALT allele, 2 = second ALT allele etc.” The obvious reading is that it preserves the pre-split allele ordering so you can get back to the original multi-allelic record. It does not. The index it records is the index within the file VEP was handed, and that file has already been split, so ALLELE_NUM is 1 on every record. Recovering the unsplit record is a join on position plus REF and ALT, and no VEP flag does it for you.

The flag still earns its place, for a different reason. The Allele field inside CSQ is a minimal representation — VEP trims the sequence common to the reference and alternate alleles off both ends before computing consequences — so for an insertion written as REF=C ALT=CAACCCT the CSQ allele is AACCCT, which matches nothing in the ALT column. String-matching CSQ alleles against ALT works for SNVs and quietly fails for indels. ALLELE_NUM being unconditionally 1 is what makes that matching unnecessary: one input allele per record, so every consequence block on that record — and there is one per overlapping transcript — belongs to that allele, tied by index rather than by string.

--no_stats is worth a mention, with the caveat that the VEP documentation itself describes the saving as marginal. I did not measure it in isolation, and on a 690,412-record input I would not expect it to be visible against the other numbers here. The reason to set it is not speed, it is that the summary is an HTML file nobody opens in a batch run and one more artefact per sample to clean up.

The correctness trap

The first version of the filter was one command instead of three, and it used --trim-alt-alleles:

# WRONG - do not do this
bcftools view -a -e 'N_ALT=1 && ALT="<NON_REF>"' "$IN" -Oz -o "$SITES"

-a removes alternate alleles that are not present in any genotype. On a gVCF this looks like exactly what you want: it strips the placeholder, it strips alleles the caller considered and rejected, and it leaves a tidy file. The output was slightly smaller on disk than the three-command version — not fewer records, fewer bytes — which read as the filter working harder.

It was not working harder. It was throwing away 2,157 sites per file, and the record count did not move, which is exactly why it took a two-way comparison to find.

The sites it lost all had the same shape: a real alternate allele in ALT, and a genotype that did not use it. Hom-ref and no-call genotypes on records that nonetheless carry a called alternate allele are ordinary in a gVCF — the caller saw allele evidence, emitted the allele, and genotyped the sample as reference or as uncallable. -a sees an allele absent from the genotype and removes it — and it removes the placeholder on the same grounds, because <NON_REF> is not in any genotype either. The documented behavior when nothing survives is the part that bites: “if no alternate allele remains after trimming, the record itself is not removed but ALT is set to ‘.’”. So the record stays. It passes the exclude expression, because N_ALT is now zero and the expression asks about N_ALT=1. It is counted in the filter output. It simply has no allele left to annotate, and it falls out inside the annotator instead of at the filter, which is why the record count after the filter stage still looks right. -a also trims Number=A, G and R tags to match, so the per-allele fields go with it.

The near-neighbour makes this worse. bcftools 1.19 added -A, --trim-unseen-allele, which removes <*> or <NON_REF> specifically — at variant sites with -A, at all sites with -AA — without ever consulting a genotype. That is the operation the first version was reaching for. One character of difference between the flag that is safe here and the flag that silently drops called sites, and no error from either.

Here is what finding it looked like. Compare the fast output against the baseline, in both directions:

$ bcftools isec -C -w1 baseline.norm.vcf.gz fast.vep.vcf.gz | grep -vc '^#'
2157
$ bcftools isec -C -w1 fast.vep.vcf.gz baseline.norm.vcf.gz | grep -vc '^#'
0

Asymmetric loss, which immediately rules out a representation difference — those show up in both directions. Then look at what was lost:

$ bcftools isec -C -w1 baseline.norm.vcf.gz fast.vep.vcf.gz \
    | bcftools query -f '[%GT]\n' - | sort | uniq -c | sort -rn
   1188 ./.
    902 0/0
     67 0|0

Every single lost site has a genotype that does not use the allele. That is the fingerprint, and it points straight at the trimming flag.

Whether this matters depends on what the annotated output is for. If the only consumer is a per-sample report of the variants this sample carries, the lost records were arguably noise. If the output feeds a review queue, a re-genotyping step, a family or cohort comparison where “the caller saw this allele and called hom-ref” is different from “the caller never saw this allele”, or any regulated workflow where the site list must be reproducible, then losing 2,157 sites per sample without a log line is unacceptable. Removing work that is genuinely unnecessary is optimization. Removing work you did not know you were doing is data loss, and the two are indistinguishable until you compare outputs.

The fix is the ordering above: filter on the placeholder alone, split, then drop what the split exposed. No stage ever consults a genotype, so no stage can drop a record for having the wrong one. -A would do the third step too, and on a recent enough bcftools it is the more direct spelling; the explicit view -e is what I left in because it also handles the * alleles and because it says out loud what is being removed.

What it cost and what it saved

One sample, one node, single file at a time:

Stage	Baseline	Filter only	Filter + tuned annotator
Filter, split, index	—	5.5 min	5.5 min
Annotation	293.0 min	7.6 min	5.4 min
Post-processing, compress, QC	9.0 min	9.0 min	9.0 min
Total wall	302.0 min	22.1 min	19.9 min
Speed-up vs baseline	1.0x	13.7x	15.2x
Records entering annotation	26,584,113	690,412	690,412
Mean cost per record	0.661 ms	0.661 ms	0.469 ms
Peak RSS, annotation	9.8 GB	4.1 GB	11.6 GB

The row that changed how I think about this is the second-to-last one. Before measuring, the expectation was that reference blocks would be much cheaper per record than variant records — a block has no consequence to compute, and a variant has to be intersected with every overlapping transcript. If that were true, removing 97% of the records would remove far less than 97% of the time, and the whole exercise would disappoint.

The measurement says the opposite. Cost per record was flat to three significant figures across a 38-fold change in what was in the file. The fixed per-record overhead — parse, buffer, region lookup, serialize — dominates the consequence calculation so completely that the consequence calculation barely registers. That is why the saving tracked record count almost exactly, and it is also a useful general warning: the intuition that “the interesting records are the expensive ones” is worth checking rather than assuming, and here it was simply false.

The arithmetic is then worth doing honestly, because the headline number is smaller than the naive prediction. Amdahl on the profile says: 9.0 minutes of untouchable work plus 293.0/38.5 = 7.6 minutes of annotation, which is 16.6 minutes, an 18.2x speed-up. The measured filter-only figure is 13.7x. The gap is the filtering pass itself, which still has to decompress and read all 26,584,113 records — you cannot skip records without looking at them — and which costs 5.5 minutes that Amdahl on the original profile does not know about. Tuning the annotator then recovered 1.41x on the annotation stage alone (7.6 minutes down to 5.4), which is worth only 1.11x on the total, because by that point the total is dominated by the 14.5 minutes of filtering and post-processing that tuning does not touch. 13.7x to 15.2x. Anyone quoting 38x from the record ratio alone has forgotten both the residual 3% and the cost of the skip.

Two parameters that only turned up in a sweep

Neither of these would have been found by reasoning. Both came out of running the same 690,412-record file across a grid.

Fork count, at a buffer of 50,000, on a node with 48 physical cores otherwise idle:

`--fork`	Wall	Speed-up vs 1	Parallel efficiency
1	19.0 min	1.00x	100%
2	10.2 min	1.86x	93%
4	5.4 min	3.52x	88%
8	6.1 min	3.11x	39%
12	7.9 min	2.41x	20%

There is a knee at four and a genuine regression beyond it, not a plateau. Eight forks is slower in wall-clock terms than four while consuming twice the cores. My reading of the mechanism is that forked workers hand results back to a single parent that has to collect them and re-emit records in input order, so past a certain width the parent becomes the constraint and the workers wait on it — but that is inference from the shape of the curve, not something I instrumented, and the knee may well sit somewhere else for a different cache, a different record mix or a different VEP release. What does transfer is the practical consequence: measure before widening, because on this stage “give it more cores” made things worse, which is the opposite of the instinct that sends you to a bigger node.

Buffer size, at four forks:

5,000 (the default): 7.6 min, 4.1 GB peak RSS
25,000: 5.9 min, 7.2 GB
50,000: 5.4 min, 11.6 GB
100,000: 5.4 min, 22.4 GB

1.41x from the default to 50,000, and nothing after that except memory. The buffer is how many variants are held and processed as a batch, and a batch is resolved against the cache by region, so a larger buffer means the same cache region is fetched and parsed fewer times. Once the buffer is comfortably larger than the typical run of variants within one cache region, there is nothing left to amortize, which is consistent with the flat line between 50,000 and 100,000. I am confident in the measurement; the explanation is my reading of the behavior and not something I have confirmed against the implementation.

Memory is the real limit here, and it interacts with the next section: 11.6 GB per file times six concurrent files is 70 GB, which fits on a 192 GB node; 22.4 GB times six does not leave room for page cache, and buys nothing.

Concurrency, measured rather than assumed

The stage is now short enough that per-file parallelism is not the interesting axis. Running several files at once is. With four forks each, six concurrent files put 24 forked workers on a 48-core node, plus six parent processes that are collecting and serializing rather than computing.

Measured by running the identical file six times simultaneously and taking the wall time of each:

one file alone: 19.9 min
six files concurrently: mean 20.9 min, slowest 21.0 min

Per-file slowdown is 5%, so aggregate throughput is 95% of six times the single-file rate. Eight concurrent files dropped to 82% efficiency, and the drop was in the annotation stage, not the filter stage. Two explanations fit and I did not separate them: eight files is 32 workers plus eight parents, which is 40 runnable processes on 48 cores and getting close enough to contend, and it is also more concurrent readers against the same cache files, so page-cache pressure is equally plausible. I would not present either as the cause.

Scaled to a batch of roughly 210 samples on four nodes, at six files in flight per node — 24 concurrent, so nine waves with the last one part-empty — the baseline pipeline is about 45 hours of wall time and the revised one is a little over three. Both figures are arithmetic over the per-sample measurements above, not a full batch re-run at both settings, and the two sides are not measured equally: the revised figure uses the 20.9-minute six-concurrent number, while the baseline figure uses the 302-minute single-file number, because I never ran six baseline files at once. If the baseline carries the same concurrency penalty, it is nearer 47 hours and the ratio is slightly better than stated. Scale to your own sample count before quoting any of it — the wave arithmetic is lumpy, and a batch that does not divide evenly into 24 lands worse than the ratio suggests.

Proving the output is identical

A 15x speed-up you cannot verify is a 15x speed-up you should not ship. Two properties need proving: the same set of sites, and the same consequence calls at each site.

The trap in the comparison is that the fast path normalizes and splits multi-allelic records and the baseline does not, so a direct diff shows thousands of differences that are pure representation. Normalize both sides identically before comparing, using the same reference and the same options:

bcftools norm -m -any -f "$REF" baseline.vep.vcf.gz -Ou \
  | bcftools view -e 'ALT="<NON_REF>" || ALT="*"' -Oz -o baseline.norm.vcf.gz
bcftools index -t baseline.norm.vcf.gz

Site set, both directions:

$ bcftools isec -C -w1 baseline.norm.vcf.gz fast.vep.vcf.gz | grep -vc '^#'
0
$ bcftools isec -C -w1 fast.vep.vcf.gz baseline.norm.vcf.gz | grep -vc '^#'
0

Zero in both directions is the claim worth making. One direction alone proves nothing: a fast path that emits a superset passes a one-way check.

What isec counts as “the same site” is worth knowing before you trust the zero. It defaults to --collapse none, under which only records with identical REF and ALT are compatible, so this comparison is on alleles and not merely on positions. That is the strict reading and the one you want. Had it defaulted to all, two records at the same coordinate with completely different alleles would have matched and both directions would have read zero while the alleles disagreed.

Consequence calls, by digest of the annotation payload rather than by eye:

$ for f in baseline.norm.vcf.gz fast.vep.vcf.gz; do
    bcftools query -f '%CHROM:%POS:%REF:%ALT\t%INFO/CSQ\n' "$f" \
      | sort | sha256sum | awk -v f="$f" '{print f, $1}'
  done
baseline.norm.vcf.gz  9f3c1b...e4a7
fast.vep.vcf.gz       9f3c1b...e4a7

Compare the body, never the whole file: the headers differ legitimately because they record the command line and the run time. And the digest comparison is only meaningful if both runs used the same annotator version, the same cache version and the same options — a cache upgrade between the two runs changes consequence calls for real reasons and will produce a mismatch that has nothing to do with your filter. Pin both, and record both in the run log.

A useful third check, cheap to run and independent of the first two:

$ for f in baseline.norm.vcf.gz fast.vep.vcf.gz; do bcftools +counts "$f"; done
Number of samples: 1
Number of SNPs:    598204
Number of INDELs:  92208
Number of MNPs:    0
Number of others:  0
Number of sites:   690412

Identical on both. 598,204 + 92,208 = 690,412, which also ties back to the pre-annotation count and confirms nothing was added or lost inside the annotator.

That sum only works because the records are biallelic. The plugin increments a counter for every variant type a record contains and increments the site counter once, so a multi-allelic record carrying a SNV and an indel adds one to each of three counters. Run +counts on an unsplit file and the categories over-sum against the site count, which looks like corruption and is not. The cross-check is valid here because norm -m -any has already guaranteed one allele per record.

The caveat that limits all of this

The waste removed here is a property of one file format. A per-sample genomic VCF describes every callable position, so it is mostly reference blocks, so most of it is not annotatable. That is why the filter works.

A joint-called multi-sample VCF has no reference blocks at all. It stores sites — positions where at least one sample in the cohort carries something — and every record in it has a real alternate allele. There is no 97% to remove. Applying this filter to a joint callset gives you a full decompression-and-parse pass over the file that removes nothing, which is a pure regression, and the “records entering annotation” row of that table would read the same before and after. It will not cost the 5.5 minutes quoted above — a joint callset has far fewer records than a per-sample gVCF, so the pass is proportionally cheaper — but cheaper than useless is still useless.

The equivalent saving on a joint callset exists but is on a completely different axis. The redundancy there is across samples, not across positions: if you annotate N per-sample files independently, you annotate the same common variants N times. The fix is to build one union site list, annotate it once, and join the annotations back onto the per-sample files by position and allele:

bcftools merge -m none -Ou sample*.sites.vcf.gz \
  | bcftools norm -m -any -f "$REF" -Oz -o cohort.sites.vcf.gz
bcftools index -t cohort.sites.vcf.gz

vep --offline --cache --dir_cache /ref/vep --assembly GRCh38 --fasta "$REF" \
    --fork 4 --buffer_size 50000 --no_stats --vcf --allele_number \
    -i cohort.sites.vcf.gz -o cohort.csq.vcf.gz --compress_output bgzip
bcftools index -t cohort.csq.vcf.gz

bcftools annotate -a cohort.csq.vcf.gz -c INFO/CSQ \
    -o sample.annotated.vcf.gz -Oz sample.sites.vcf.gz

The join in that last command is the place it goes wrong. When the annotation file is a VCF, bcftools annotate matches on CHROM, POS, REF and ALT, and a record that does not match is left unannotated in silence — there is no warning and no non-zero exit. So both sides have to have been normalized with norm -m -any against the same reference FASTA, or a left-alignment difference at a repeat-adjacent indel produces two records that a human would call the same variant and bcftools will not. Check the join by counting records with a missing CSQ afterwards rather than assuming it landed. The annotation file also has to be bgzip-compressed and indexed, which is what the index -t above is for.

The scaling of that is set by the overlap between samples, not by the reference-block fraction, and the overlap depends on cohort size and ancestry composition. I have not measured it carefully enough to quote a factor, so I will not quote one.

One last thing that is easy to get wrong at the end of a project like this. The filtered file is an annotation input, not a replacement for the gVCF. The reference blocks you removed are what distinguishes “this sample has no variant here” from “this position was never callable in this sample”, and that distinction is the entire reason the gVCF format exists. Keep the original. The optimization is that the annotator never sees it.

What to take from this

Profile first, in stages, with timestamps that end up in a log someone else can read. If one stage is 97% of the cost, nothing you do to the other stages matters, and the only question worth asking about that stage is whether the work it is doing needs doing at all.

Then verify in both directions before you believe your own result, because the fastest version of any pipeline is the one that quietly does less than it should, and it will pass every check that only looks at how long it took.

Kubernetes security: test admission, network policy, and storage access separately

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A workload placed in its own namespace is easier to organize, but that fact alone does not prove isolation. Kubernetes admission controls, network policy, identities, and storage permissions each enforce a different part of the boundary.

Define the workload’s permitted behavior

Record whether the application needs privilege, host access, outbound network destinations, secrets, and persistent volumes. Start with actual requirements rather than cloning the permissions of an older deployment.

Kubernetes documents the Privileged, Baseline, and Restricted Pod Security Standards. Choose and enforce an appropriate policy for the target release, with reviewed exceptions for infrastructure components that need additional privileges.

Inspect the deployed objects

Using an authorized read-only Kubernetes context:

kubectl config current-context
kubectl get namespace example-app --show-labels
kubectl get networkpolicy -n example-app
kubectl get serviceaccount -n example-app
kubectl get rolebinding -n example-app

example-app is a placeholder namespace. Confirm the context before every administrative operation. These commands reveal configuration; they do not prove enforcement or effective authorization across all bindings.

Validate network policy behavior

The Kubernetes NetworkPolicy documentation explains that enforcement depends on a supporting network implementation. The existence of a policy object is insufficient evidence.

In a lab namespace, run a permitted client and an unrelated client. Test the intended application port, required DNS resolution, and an outbound destination that should be denied. Include both ingress and egress expectations. Keep the tests harmless and restricted to the lab.

Do not assume policies are an ordered firewall rule list. Review their combination and selection semantics for the deployed implementation. A broad additional allow can change the intended boundary.

Include the storage path

A pod with an authorized mount may still receive excessive filesystem permissions or access a shared dataset intended for another workload. For WEKA, PowerScale, VAST, or other CSI-backed storage, examine the storage-side identity and export/share policy as well as the Kubernetes objects.

Test a synthetic file through the application’s actual container identity. Confirm both required access and denied access outside its scope. An admission policy cannot repair an overly broad storage authorization model by itself.

Exercise rejected workloads

Use a disposable manifest that requests a prohibited capability and confirm admission rejects it. Then deploy a compliant workload and run its business transaction. Both results are necessary: a policy that rejects all work is not an acceptable application platform.

Keep the evidence tied to the release

Record Kubernetes and network-plugin versions, policy objects, test identities, and expected-versus-observed results. Rerun after cluster upgrades and CNI or CSI changes. The accepted boundary belongs to the tested system, not to a YAML file considered in isolation.

Reading Slurm accounting without fooling yourself about efficiency

2026-05-27T10:00:00-04:00

Someone asks how well the cluster is being used. It is a reasonable question and there is a database full of the answer, so you write a query. Twenty minutes later you have a number, and the number is wrong in a way that will not announce itself. Both of the common mistakes produce output that looks like a measurement: one gives you a beautiful figure, one gives you an alarming figure, and neither has anything to do with how busy the cores were.

This is about getting a defensible number out of sacct, knowing which fields lie to you, and knowing what the number means once you have it — because the usual interpretation of a low efficiency figure is also wrong.

What CPU efficiency actually is

CPU efficiency is consumed CPU time divided by allocated CPU time:

efficiency = TotalCPU / (NCPUS * Elapsed)

The denominator is what the scheduler took off the table. If a job holds eight cores for four hours, nobody else can have those eight cores for those four hours, whether the job uses them or not. The numerator is what the job’s processes actually burned, user time plus system time, as measured by the accounting plugin.

That is exactly what seff computes. From the shipped Perl (contribs/seff/seff.pl):

my $corewalltime = $walltime * $ncpus;
if ($corewalltime != 0) {
    $cpu_eff = $cput / $corewalltime * 100;
}

So for a single job you can just run seff and stop reading:

$ seff 918442
Job ID: 918442
Cluster: (redacted)
User/Group: user01/user01
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 04:14:07
CPU Efficiency: 12.55% of 1-09:44:40 core-walltime
Job Wall-clock time: 04:13:05
Memory Utilized: 5.47 GB
Memory Efficiency: 8.55% of 64.00 GB

seff does not scale. It is one job per invocation and it goes through the Slurm Perl API each time. For a fleet report over a month of jobs you need sacct, and that is where the trouble starts.

Trap one: CPUTimeRAW is allocation, not consumption

sacct offers a field called CPUTimeRAW. The name reads like “the raw CPU time this job used”. It is not. The manual is unambiguous:

CPUTimeRAW — Time used (Elapsed time * CPU count) by a job or step in cpu-seconds.

That is the denominator. CPUTime is the same quantity in HH:MM:SS. It is allocated core-time, computed arithmetically from the allocation, and it has no idea whether the cores were doing anything. You can verify this in one line — the identity holds exactly, not approximately:

$ sacct -j 918442 -X -o JobID,NCPUS,ElapsedRaw,CPUTimeRAW --parsable2 --noheader
918442|8|15185|121480

$ python3 -c "print(8*15185)"
121480

The failure mode is that someone builds the ratio out of the field whose name sounded right:

# WRONG — this is the definition of 1.0
sacct -a -S 2026-04-01 -E 2026-05-01 -X --parsable2 --noheader \
      -o CPUTimeRAW,NCPUS,ElapsedRaw \
  | awk -F'|' '{u+=$1; a+=$2*$3} END {printf "efficiency %.1f%%\n", 100*u/a}'

efficiency 100.0%

It is not 100.0% because the cluster is perfectly used. It is 100.0% because you divided a quantity by itself. Every job contributes exactly its own allocation to both sides, so the answer is identically one for any job mix, any time window, any partition. The number is not even slightly sensitive to reality: drain half the nodes, let jobs idle for days, and it still reads 100.0%.

The reason this survives review is that 100% is not obviously absurd to a non-specialist. On a busy cluster with a deep queue, “we are at 100%” reads as a statement about the queue rather than about the cores. It gets into a slide. The give-away is that it is exactly 100.0% and stays exactly 100.0% when you change the window — a real measurement never does that.

Trap two: the allocation-only flag zeroes the numerator

The other field you need is TotalCPU:

TotalCPU — The sum of the SystemCPU and UserCPU time used by the job or job step.

That is the right numerator. But CPU time is gathered per step, not per allocation. A batch job produces several accounting records: the job allocation record (918442), the batch step (918442.batch), the external step (918442.extern) that holds anything adopted into the job’s cgroup, and one record per srun (918442.0, 918442.1 …). The -X / --allocations flag says:

Only show statistics relevant to the job allocation itself, not taking steps into consideration.

Which means the step records are not fetched at all, and the step-derived accounting fields are not populated from them. Everyone knows this about MaxRSS, because a blank column is visible. Rather fewer people notice it about TotalCPU, because it comes back as a plausible-looking time string rather than a blank:

$ sacct -j 918442 -X -o JobID,NCPUS,ElapsedRaw,TotalCPU
JobID          NCPUS ElapsedRaw   TotalCPU
------------ ------- ---------- ----------
918442             8      15185   00:00:00

$ sacct -j 918442 -o JobID,NCPUS,ElapsedRaw,TotalCPU
JobID          NCPUS ElapsedRaw   TotalCPU
------------ ------- ---------- ----------
918442             8      15185  04:14:07
918442.batch       8      15185  04:14:07
918442.extern      8      15185   00:00.001

Same job, same database, two different answers, and the difference is one flag. The job row’s 04:14:07 in the second listing is assembled from the step rows that the same query returned; with -X there are no step rows to assemble it from. Build a fleet report on the first form and every ratio is 0 / something, so the cluster reads 0.0% efficient.

That ought to be caught — and usually is, if the whole report is built that way. It is not caught when the query includes steps for some jobs and not others, or when someone filters the zeroes out as “bad records” before averaging. Drop the zero rows and you are left with exactly the jobs that ran many srun steps, which are the well-parallelized ones, and your cluster suddenly reports 70%.

Whether the job-level record carries a rolled-up TotalCPU depends on your Slurm version, on whether the query pulled the steps, and on how the job was launched, so do not take my word for it. Check on your own installation before you trust any script (this quick check splits on : only, so run it on a job under a day of CPU time; the tosec() function further down handles the DD- form):

sacct -j 918442 --parsable2 --noheader -o JobID,TotalCPU \
  | awk -F'|' '{ n=split($2,p,":"); s=(n==3)?p[1]*3600+p[2]*60+p[3]:p[1]*60+p[2];
                 if ($1 ~ /\./) step+=s; else job=s }
               END {printf "job row: %.1fs   sum of steps: %.1fs\n", job, step}'

job row: 15247.0s   sum of steps: 15247.0s

If those two agree on your cluster, either source is fine. If the job row is zero, you must sum the steps. Write the script so it works either way and you never have to care.

Trap three: the one that produces a believable number

The first two traps produce 100.0% and 0.0%. Those are at least suspicious. The trap that actually ships is the averaging.

There are two things you might mean by “cluster efficiency”, and they are not close to each other:

Job-mean: compute efficiency per job, then take the arithmetic mean over jobs.
Core-weighted: sum consumed CPU-seconds over the whole fleet, sum allocated CPU-seconds over the whole fleet, divide once.

Take a constructed month with two populations, chosen to make the arithmetic visible:

population	count	cores	elapsed	per-job eff	allocated core-h	consumed core-h
array tasks	12,000	1	5 min	95%	1,000	950
wide jobs	180	128	12 h	18%	276,480	49,766

Job-mean: (12000 × 0.95 + 180 × 0.18) / 12180 = 93.9%.

Core-weighted: (950 + 49,766) / (1,000 + 276,480) = 18.3%.

Same data. 93.9% or 18.3%, depending on a choice most people do not realize they are making. The job-mean is dominated by twelve thousand five-minute array tasks that together account for 0.4% of the machine. The core-weighted figure is the one that answers “how much of the cluster’s capacity turned into computation”, which is the question that was asked.

Use core-weighted for any capacity or spend conversation. Job-mean is useful for exactly one thing: finding which users have a habit of over-requesting, where you want each job to count once regardless of size. Label whichever you publish.

The working query

Query without -X, take the denominator from the job row and the numerator from the step rows, and parse the time formats properly. TotalCPU is printed as [DD-[HH:]]MM:SS[.mmm], so you will see 00:00.001 and 2-03:14:22 in the same column; a naive HH:MM:SS split gets both wrong.

sacct -a -S 2026-04-01T00:00:00 -E 2026-04-30T23:59:59 \
      --parsable2 --noheader --delimiter='|' \
      --format=JobID,User,Account,Partition,State,NCPUS,ElapsedRaw,CPUTimeRAW,TotalCPU,ReqMem,MaxRSS,NNodes \
  > /tmp/acct-apr.psv
wc -l /tmp/acct-apr.psv

2841577 /tmp/acct-apr.psv

Two and a half million rows is a few hundred megabytes of text, so put it somewhere with room — filling the root filesystem of a login or head node is an unpleasant way to learn that a month of accounting is bigger than it looks.

Then aggregate:

#!/usr/bin/awk -f
# slurm-eff.awk — core-weighted CPU efficiency by partition
BEGIN { FS = "|" }

function tosec(t,   d, p, n, s) {
    d = 0
    if (t ~ /-/) { split(t, p, "-"); d = p[1] + 0; t = p[2] }
    n = split(t, p, ":")
    if      (n == 3) s = p[1]*3600 + p[2]*60 + p[3]
    else if (n == 2) s = p[1]*60 + p[2]
    else             s = p[1] + 0
    return d*86400 + s
}

$1 ~ /\./ {                       # step record: the numerator lives here
    split($1, a, ".")
    consumed[a[1]] += tosec($9)
    next
}
{                                 # job record: the denominator lives here
    if ($5 !~ /^(COMPLETED|FAILED|TIMEOUT|OUT_OF_MEMORY|NODE_FAIL)/) next
    if ($7 + 0 < 120) next        # sub-2-minute jobs are startup, not work
    alloc[$1] = $8 + 0            # CPUTimeRAW == NCPUS * ElapsedRaw
    part[$1]  = $4
}
END {
    for (j in alloc) {
        A[part[j]] += alloc[j]
        C[part[j]] += consumed[j]
        TA += alloc[j]; TC += consumed[j]
    }
    printf "%-12s %14s %14s %8s\n", "partition", "alloc_core_h", "used_core_h", "eff"
    for (p in A)
        printf "%-12s %14.1f %14.1f %7.1f%%\n", p, A[p]/3600, C[p]/3600, 100*C[p]/A[p]
    printf "%-12s %14.1f %14.1f %7.1f%%\n", "TOTAL", TA/3600, TC/3600, 100*TC/TA
}

$ awk -f slurm-eff.awk /tmp/acct-apr.psv

Illustrative output — these figures are constructed to show the shape of the result, not a measurement of any particular machine. for (p in A) iterates in an unspecified order, so the partition block comes out shuffled; sort it yourself if the order matters, but do not pipe the whole output through sort or the header and the TOTAL line get sorted along with the data:

partition      alloc_core_h    used_core_h      eff
normal             276480.4        50716.3    18.3%
long               188411.2        61533.9    32.7%
gpu                 41203.6         9877.1    24.0%
short                1000.2          950.1    95.0%
TOTAL              507095.4       123077.4    24.3%

Three details in that script are load-bearing.

The state filter excludes RUNNING and PENDING. A running job has a growing denominator and a numerator that is only as fresh as the last accounting poll, so it drags the average down for no reason. CANCELLED is excluded too, though that one is a judgment call: a job canceled at 90% of its walltime really did consume the allocation, and if your users cancel a lot you should count it. Say which you did.

The 120-second floor removes the long tail of jobs that failed at startup. On a cluster with heavy array use these can be the majority of records by count and a rounding error by core-hours, so the floor barely moves the core-weighted number — but it removes thousands of 0% rows that would wreck a job-mean.

The step loop sums all step records including .extern. That is deliberate, and it is the next trap.

Where the obvious reading is wrong: extern and SSH-launched ranks

The tidy instinct is to drop .extern from the numerator. It is the container step for the job’s cgroup, it normally shows a millisecond or two of CPU, and it feels like noise.

On a site running pam_slurm_adopt, it is not always noise. The thing to understand is that “mpirun instead of srun” is not by itself the problem — the question is whether the launcher bootstraps through Slurm or through SSH. Open MPI’s mpirun, inside an allocation, uses srun to start its daemons, so a step record does exist. Intel MPI exported with I_MPI_HYDRA_BOOTSTRAP=ssh, Open MPI forced onto the rsh/ssh launcher or built without Slurm support, and hand-rolled ssh-in-a-loop launchers all start the remote processes over SSH, and none of those produce a 918442.0 on the remote nodes.

What there is on those nodes, if adoption is configured, is the external step — pam_slurm_adopt puts the incoming SSH session into the job’s extern cgroup, and the adopted processes’ CPU time is then accounted against .extern rather than against any launch step.

Drop .extern and every one of those jobs reports roughly 1/N efficiency, where N is the node count, because you counted only the rank on the batch node. A 16-node job that is running perfectly reads 6%. That is a very convincing-looking finding. It is also completely false, and it points the investigation at the users who are doing the most demanding work.

The check takes one run:

$ awk -f slurm-eff.awk /tmp/acct-apr.psv | tail -1
TOTAL              507095.4       123077.4    24.3%

$ grep -v '\.extern|' /tmp/acct-apr.psv | awk -f slurm-eff.awk | tail -1
TOTAL              507095.4        94112.8    18.6%

A gap that size between the two runs tells you a meaningful fraction of your CPU time is being recorded against the external step, which tells you that work is arriving on compute nodes by some route other than a Slurm launch step. That is worth knowing on its own. If the two runs agree to within a percent, .extern genuinely is noise on your cluster and either choice is fine.

There is a worse version of this. If adoption is not configured, the remote ranks belong to no step and no cgroup, and their CPU time is not recorded anywhere at all. No amount of careful querying recovers it. The symptom is an MPI-heavy partition that reports implausibly low efficiency while the nodes are visibly hot; confirm by watching ps on a compute node during one of those jobs, or by comparing sacct against a node-level metric such as a CPU-utilization exporter. If that is your situation, the accounting database cannot answer the efficiency question for those jobs and you should say so rather than publish the number.

The memory half

CPU efficiency alone will mislead you about waste. On most general-purpose clusters memory is the binding constraint more often than cores, and a job that holds a node’s entire RAM while using 6 GB of it is wasting the machine even at 100% CPU.

The fields:

$ sacct -j 918442 -o JobID,ReqMem,MaxRSS,MaxRSSNode,MaxRSSTask,TRESUsageInTot%40 --units=G
JobID          ReqMem     MaxRSS MaxRSSNode MaxRSSTask                        TRESUsageInTot
------------ -------- ---------- ---------- ---------- -------------------------------------
918442            64G
918442.batch            5.47G      nodeA             0   cpu=04:14:07,energy=0,fs/disk=12.1G,
                                                          mem=5.47G,pages=0,vmem=6.02G
918442.extern              0       nodeA             0   cpu=00:00:00,mem=0,pages=0,vmem=0

(The TRESUsageInTot column is wrapped here to fit the page; sacct truncates it to the width you asked for instead. Exact unit rendering — trailing zeroes, which fields --units reaches — moves about between versions, so match your own output rather than this one.)

Two things to fix in your head.

ReqMem changed meaning in 21.08. The release notes are explicit: it now “shows the requested memory of the whole job with a letter appended indicating units”, and it is only displayed for the job record, not the steps. Before that it carried a Mn / Mc suffix meaning per-node or per-core, and you had to multiply by node count or core count yourself to get the job total. A script written against the old format and run against a new cluster — or the reverse — is wrong by a factor equal to the core count, which on a 128-core node is not subtle. Check your version before you write the parser:

$ sacct -V
slurm 24.05.7

MaxRSS is a maximum over tasks, not a sum. The existence of MaxRSSTask and MaxRSSNode tells you this directly: they name which task and which node hit the peak. For a single-task job, MaxRSS is the job’s memory. For a 128-rank MPI job where every rank uses 3 GB, MaxRSS reports 3 GB while the job actually held 384 GB. Divide that by ReqMem and you will conclude the job over-requested by a factor of 128 and go and tell the user so.

The field to reach for instead is mem= inside TRESUsageInTot, which the manual defines as “Tres total usage in by all tasks in job” — the per-task peaks added up. Note what that is and is not: because the peaks need not have happened at the same moment, the sum is an upper bound on what the job held simultaneously, not a measurement of it. It is still much closer to the truth than MaxRSS for a multi-task job.

Do not assume seff settles this for you. seff does not read tres_usage_in_tot at all: it takes the per-task maximum from tres_usage_in_max, keeps the largest one across the steps, and multiplies by that step’s task count. On a job whose ranks are all doing the same thing that is a fair estimate; on one where a single rank holds most of the memory it can be far out in either direction. seff is honest about this — for a multi-task step it labels the line “(estimated maximum)”. So seff’s “Memory Utilized”, a MaxRSS query and a TRESUsageInTot query can produce three different numbers for the same job, and none of them is a true simultaneous job-wide peak, because Slurm does not record one.

One more limit worth stating plainly: memory accounting is, historically, sampled. JobAcctGatherFrequency defaults to 30 seconds for the task datatype, and a job that allocates 200 GB for four seconds and frees it can poll clean and show a MaxRSS of 8 GB.

The cgroup/v2 plugin improves on that by taking MaxRSS from the kernel’s memory.peak, which is a high-water mark rather than a sample. Two things then follow, and they pull in opposite directions. First, filesystem-backed memory counts: the page cache a job generates by reading and writing files is included in the reported RSS unless you set JobAcctGatherParams=no_file_cache, so an I/O-heavy job can look memory-hungry when it is not. Second, the manual is explicit that no_file_cache “disables the use of the memory.peak interface, which can result in MaxRSS failing to record short memory spikes” — the cleaner number costs you the high-water mark. Slurm 24.11 also lists a fix for jobs that run shorter than two gather intervals, which previously could report nothing at all.

So there is no single answer to “is my MaxRSS trustworthy”: it depends on your JobAcctGatherType, your cgroup version, your JobAcctGatherParams and your Slurm version. Check which path your cluster is on before you tell users to cut their memory requests, and leave headroom either way. A job killed by the OOM handler costs more than the memory it was holding.

Array jobs

Arrays need care for three separate reasons.

$ sacct -j 918500 -X -o JobID,JobIDRaw,State,NCPUS,ElapsedRaw
JobID            JobIDRaw      State   NCPUS ElapsedRaw
--------------- ---------- ---------- ------- ----------
918500_[4-500]      918500    PENDING       1          0
918500_1            918501  COMPLETED       1        363
918500_2            918502  COMPLETED       1        358
918500_3            918503  COMPLETED       1        371

First, the pending master record 918500_[4-500] is a real row with a real JobID. It has zero elapsed time so it contributes nothing to a core-weighted sum, but it is one row out of many in a job-mean, and if you are counting jobs it inflates your count by one per array rather than one per task. Filter on state.

Second, JobID and JobIDRaw differ for array tasks. Step records are keyed on the JobID form (918500_2.batch), so join on JobID, not JobIDRaw. Mixing the two silently drops the numerator for every array task. The raw ids are handed out as tasks are scheduled, so do not assume they are contiguous, ordered, or related to the task index in any useful way.

Third, and most important: arrays are how one user generates a hundred thousand accounting rows. Any per-job average is now a report on that one user’s array. This is the single biggest reason to publish the core-weighted figure.

The traps, and the answer each one produces

Trap	Wrong answer it produces	How to spot it
`CPUTimeRAW` used as consumed CPU	Exactly 100.0%, always	Figure does not move when you change the window
`-X` with `TotalCPU`, where the job row is not rolled up	Exactly 0.0%	`TotalCPU` column reads `00:00:00` for every job
Zero rows filtered out as “bad data” after using `-X`	60–80%, entirely fictional	Row count after filter is a small fraction of jobs
Summing `TotalCPU` over job and step rows	Roughly double; some jobs >100%	Any per-job efficiency above 100%
Step `NCPUS` used in the denominator	Inflated; varies by launch style	Denominator smaller than `CPUTimeRAW` on the job row
Mean of per-job ratios	90%+ on an array-heavy cluster	Job count in the thousands, core-hours dominated by tens of jobs
`.extern` dropped where `pam_slurm_adopt` adopts SSH-launched ranks	MPI partitions read at `1/N`	Re-run including `.extern` and compare
No adoption configured, ranks launched over SSH	Unrecoverable under-count	`sacct` disagrees with node-level CPU metrics
`RUNNING` jobs included	Understated, drifts by time of day	Efficiency changes when you re-run the same window
Jobs straddling the window counted whole	Monthly totals exceed the machine’s capacity	Sum of months ≠ the year
`-T` used to fix that	Some jobs above 100%	Truncates elapsed, does not prorate `TotalCPU`
`MaxRSS` used as job memory	Understated by the task count	Multi-node jobs all read ~1% memory efficiency
`seff` memory quoted as measured	An estimate: per-task max × task count	`seff` prints “(estimated maximum)” on multi-task steps
`ReqMem` parsed without checking version	Off by core count or node count	Memory efficiency above 100% or absurdly below
SMT threads counted as cores	Hard ceiling at 50%	No job in the fleet exceeds 50%
`sacct` efficiency compared to `sreport` “Used”	The two never agree	They are different quantities — see below

On that last row: sreport’s usage columns are built from what jobs were charged, which is allocation-derived — the same family of quantity as the usage that feeds fairshare, though fairshare additionally applies TRESBillingWeights and a decay half-life, so the two are not interchangeable either. What matters here is that none of it is TotalCPU. So sreport and a correct sacct efficiency query are not meant to agree, and the ratio between them is roughly your efficiency figure. You can confirm the relationship on your own cluster by checking that sreport totals track your summed CPUTimeRAW rather than your summed TotalCPU — if they do, the model above is right for your version.

Field reference

Field	What it is	Which record carries it	Gotcha
`NCPUS` / `AllocCPUS`	Allocated CPU count	Job and step	Step value can be smaller than the job’s
`ElapsedRaw`	Wall seconds	Job and step	Step elapsed ≠ job elapsed
`CPUTimeRAW`	`NCPUS × ElapsedRaw`	Job and step	Allocation, never consumption
`TotalCPU`	`UserCPU + SystemCPU`	Step; job row varies by version	Format is `[DD-[HH:]]MM:SS[.mmm]`
`UserCPU` / `SystemCPU`	The two halves	Step	High system time can mask an idle job
`ReqMem`	Requested memory	Job only, since 21.08	Per-node/per-core suffix before 21.08
`MaxRSS`	Peak RSS of the largest single task	Step	Not a sum across tasks
`MaxRSSTask` / `MaxRSSNode`	Which task and node peaked	Step	Their existence proves `MaxRSS` is a max
`TRESUsageInTot`	Per-TRES totals, `mem=` summed over tasks	Step	Sum of per-task peaks: an upper bound, not a simultaneous peak
`TRESUsageInMax`	Per-TRES per-task maxima	Step	The `MaxRSS` number, in TRES form; what `seff` reads
`AllocTRES`	Full allocation including GRES	Job	Where GPU counts live
`State`	Final state	Job and step	Filter before aggregating
`NNodes` / `NTasks`	Shape of the job	Job / step	`NTasks` blank on the job record
`JobID` / `JobIDRaw`	Array-form and underlying id	Both	Join on `JobID`

Worked example

Back to job 918442. Eight cores, 15,185 seconds elapsed, so 121,480 allocated core-seconds. The steps sum to 15,247 seconds of CPU.

efficiency = 15247 / 121480 = 12.55%
cores used = 15247 / 15185 = 1.004
allocated  = 8
memory     = 5.47 GiB of 64 GiB requested = 8.55%
waste      = (121480 - 15247) / 3600 = 29.5 core-hours parked

The percentage is the least useful line there. “1.004 cores out of 8” tells you what happened: this is a single-threaded program that asked for eight cores. It is not a program that is 12.55% efficient at using eight cores — it is a program that is 100% efficient at using one core and was given eight. Nothing the user can do to their code will move the 12.55%. Changing --cpus-per-task from 8 to 1 moves it to roughly 100%, instantly, and on this job — one thread, 5.47 GiB of a 64 GiB request — with no change to runtime. Check the memory before you send that advice, though: where the site sets DefMemPerCPU, cutting the core count cuts the job’s memory with it, and the user who was quietly using cores as a memory allocation will get an OOM kill for following your guidance. That is the second of the three rational reasons below, and it is a site configuration problem, not a user problem.

That reframing is the whole point. Report average cores used against cores allocated, not a percentage, and the diagnosis falls out of the number:

Cores used vs allocated	Reading	What to do
~1.0 of N, N > 1	Single-threaded job over-requesting	Fix the submission script
~N of N	Working as intended	Nothing
~0.5N of N	One thread per SMT core, or half the ranks idle	Check `ThreadsPerCore` before blaming the user
≪1.0 of N	I/O-bound, or waiting on a license or a lock	Look at `SystemCPU` and at the filesystem
> N	Oversubscribed, or no cgroup pinning	Check `TaskPlugin` and the thread-library env
~N early, ~1 late	Serial post-processing tail	Split the job; the average hides it

The SMT row is worth dwelling on. Depending on SelectTypeParameters and the node’s ThreadsPerCore, allocating one core can charge you two CPUs, in which case a perfectly saturating single-threaded task caps out at 50% and no job on the machine will ever read higher. Before concluding your users waste half the cluster, check:

$ scontrol show node nodeA | grep -E 'CPUAlloc|CPUTot|ThreadsPerCore|Boards'
   CPUAlloc=128 CPUEfctv=128 CPUTot=128
   Boards=1 SocketsPerBoard=2 CoresPerSocket=64 ThreadsPerCore=1

ThreadsPerCore=1 there, so it is not the explanation on that node. If it reads 2, every efficiency figure you have is measured against threads.

What a realistic fleet number looks like

I am not aware of a systematic published survey of core-weighted CPU efficiency across academic clusters, so treat any single quoted figure — including mine — as a prior rather than a benchmark. The working range I would expect on a general-purpose, mixed-workload cluster is roughly 35% to 65%. A number in the high thirties on a machine that serves a lot of bioinformatics and statistics is unremarkable. Below about 25% something structural is usually going on: a partition configured exclusive so every job charges a whole node, a default --cpus-per-task that does not match the default thread count of the dominant application, or GPU jobs whose CPU allocation is incidental.

Two caveats on the number itself.

GPU partitions should not be assessed on CPU efficiency at all. A job holding four GPUs and eight cores to feed them is doing exactly the right thing and will read 10%. You need GPU utilization from a device-level exporter; the accounting database records that GPUs were allocated, not that they were busy.

And the honest reading of a low figure is almost never “users are wasteful”. Sort the parked core-hours by cause and, on the mixed-workload clusters I have looked at, most of them come from single-threaded jobs asking for several cores. Users do this for three rational reasons: they were told to by a tutorial written for a different scheduler, they are using cores as a proxy for memory because the site’s memory-per-core default is low, or the application is genuinely threaded but only for one phase of its run. The first is a documentation fix, the second is a scheduler configuration fix, and only the third is really the user’s problem.

So publish two columns, not one. Core-hours parked, and the count of jobs where cores-used is below 1.5 while cores-allocated is above 2. The first is the size of the prize. The second is the list of submission scripts to go and fix, and it is usually short — a handful of templates that have been copied across a department.

This one needs the same tosec() as before. awk accepts -f more than once, so keep that function in a file of its own and prepend it rather than pasting it twice — an inline program that calls tosec() without it is a fatal “calling undefined function”, not a silent zero.

# parked.awk — worst over-requesting jobs by parked core-hours
BEGIN { FS = "|" }
$1 ~ /\./ { split($1, a, "."); c[a[1]] += tosec($9); next }
$5 == "COMPLETED" && $7 + 0 > 600 { n[$1] = $6; e[$1] = $7; u[$1] = $2 }
END {
    for (j in n)
        if (n[j] > 2 && c[j]/e[j] < 1.5)
            printf "%-12s %-10s %3d cores  %.2f used  %8.1f core-h parked\n",
                   j, u[j], n[j], c[j]/e[j], (n[j]*e[j] - c[j])/3600
}

$ awk -f tosec.awk -f parked.awk /tmp/acct-apr.psv | sort -k7 -gr | head -20

Run that, and instead of a percentage you have twenty lines, each naming a submission script and the number of core-hours it costs per month. That is a conversation you can actually have.

OpenSCAP reports: preserve the exceptions behind the score

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A hardening score can improve because controls were implemented, because rules were excluded, or because the wrong profile was scanned. An assessment needs enough context to distinguish those outcomes.

Pin the benchmark inputs

Record the operating-system release, OpenSCAP version, content package version, selected profile, tailoring file, and target identity. Retain the original machine-readable result along with any HTML report.

OpenSCAP documents configuration evaluation with XCCDF and OVAL and the use of profiles in its tool overview. A tool’s ability to evaluate a benchmark does not certify the organization or establish that the benchmark fits the workload.

Inspect before evaluating

On a lab host with OpenSCAP and the relevant content installed:

oscap -V
oscap info /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml

The content path is an example for an installed RHEL 9 SCAP Security Guide package. Use the actual available data stream and read its listed profiles. Do not paste a profile identifier from a different release and assume it selected the intended controls.

Run a non-remediating evaluation first. Preserve both the process status and the report’s semantic results. Evaluation errors, not-applicable rules, and failed rules mean different things.

Review exceptions individually

For each excluded or accepted finding, record the rule, affected systems, operational reason, compensating control, owner, and review date. Avoid a blanket exception such as “HPC nodes need performance.” A specific setting may be necessary for one workload and unnecessary elsewhere.

Finding state	Required follow-up
Failed and applicable	Remediate or approve a bounded exception
Not applicable	Verify why it does not apply
Evaluation error	Repair the assessment path
Excluded by tailoring	Review the tailoring decision

A score without these states hides the information an auditor or operator needs most.

Test remediation as a service change

Do not enable automatic remediation across an estate as the first experiment. Review proposed changes, apply a small subset to a pilot, and test application behavior and recovery access.

Some controls affect authentication, mounts, kernel behavior, or cryptography. Their operational consequences may not appear until a reboot or a new session. Include those lifecycle events in the pilot acceptance test.

Compare like with like

When a later scan differs, compare content and profile versions before attributing the change to configuration drift. A revised benchmark can produce a different result on an unchanged host.

Keep a small evidence bundle for each assessment: input hashes, target inventory, results, exceptions, remediation record, and post-change checks. That supports a repeatable engineering review. It should never be shortened to an unsupported claim that the whole estate is “compliant.”

The green check that hides a broken result

2026-04-22T10:00:00-04:00

A crash is a gift. It has a timestamp, a stack, a non-zero status and somebody’s attention. Crashes get fixed because they are impossible to ignore.

The failures that survive in a well-run estate are the other kind. The job exits zero. The log ends with “completed”. The output file is there, with the right name, the right owner and a plausible size. Nothing in the monitoring turns amber. And the contents are wrong — empty where they should be populated, defaulted where you chose something else, stale where you expected fresh.

This class has no single cause. It has about seven, and they are mechanical enough to be listed. What follows is the taxonomy, then the checking discipline that actually catches them, which is narrower and more annoying than the one most of us write by default.

The organizing claim: a check that asserts on status rather than content is not a check. It is a record that something ran.

The taxonomy

Class	What the operator sees	Why the probe passes	The assertion that catches it
Error printed, status zero	An error line in a log nobody reads	Probe tests `$?` only	Compare stdout to an expected value
Plugin missing, schema intact	Correct headers, blank column	Header check passes	Assert a populated value for a case that must produce one
List parsed as one string	Silent fallback to a default	Config parses, service starts	Read the effective value back from the running process
Zero-byte index fetched	“Nothing to update”	Transfer tool exits zero	Assert a minimum size and a known marker in the content
Symlink to a path on another host	Module loads, variable empty	`module load` exits zero	Resolve the path and test the binary runs
Environment change inside a pipe	A working tool reported broken	Probe ran in a subshell	Set environment, then test in the same shell
Suite that has never failed	An unbroken run of green	Nothing has been injected	Prove each check fails on a deliberate fault

Each row below gets its own section, because the mechanism matters more than the symptom.

Class one: the error that exits zero

Exit status is a convention, not a guarantee. Plenty of widely used tools print a diagnostic and return success, either by design or by oversight.

jq is the honest case. A missing key is not an error in its data model — it is null:

$ echo '{"a":1}' | jq -r '.b'
null
$ echo $?
0

A probe written as “run it, check the status, check that stdout is non-empty” scores that as a pass. The string null is four bytes of non-empty output. Downstream, a config generator writes endpoint = null into a file, the service parses it, and you get whatever the service does with a nonsense endpoint.

grep -c is the inverse and catches people the other way:

$ printf '' | grep -c .
0
$ echo $?
1

It prints something (0) and returns failure. A probe keying on status calls that a failure when the count is the answer you wanted; a probe keying on “is there output” calls it a pass. Neither is reading the number.

The most expensive member of this family is HTTP. curl treats an HTTP error response as a successful transfer, because from its point of view it was one:

$ curl -s -o index.json -w 'http=%{http_code}\n' https://example.com/api/index.json
http=404
$ echo $?
0
$ ls -l index.json
-rw-r--r-- 1 svc svc 559 Apr 22 09:14 index.json

The file exists. It is 559 bytes. It is an HTML error page. Every “the file was created and is non-empty” check passes. curl --fail suppresses the body on a 4xx or 5xx and exits non-zero, which is what you want in a script; --fail-with-body (curl 7.76 and later) does the same but keeps the body if you need it for the log. Neither is the default, and the default is what ends up in automation written in a hurry.

One caveat, because it bites checks that are too specific. The documented exit code for --fail is 22, and over HTTP/1.1 that is what you get. Over HTTP/2 the same 404 can surface as 56 instead, because curl tears the stream down and reports a receive error rather than an HTTP one:

$ curl -s --fail -o /dev/null https://example.com/api/index.json ; echo $?
56
$ curl -s --http1.1 --fail -o /dev/null https://example.com/api/index.json ; echo $?
22

That pair is from curl 8.7.1, and I would treat the specific code as build- and version-dependent rather than a rule — which is exactly the point. Test --fail for non-zero, not for 22. A check written as [ $? -eq 22 ] is itself a member of this article’s taxonomy: it goes green on the day the negotiated protocol, or the packaged curl, changes under it.

The general fix is not to distrust exit codes. It is to stop treating them as the only evidence. Ask for the value, state what you expect, compare.

got=$(jq -r '.endpoint // empty' config.json)
[ "$got" = "https://collector.internal:8443" ] || {
  echo "endpoint mismatch: got [$got]" >&2; exit 1; }

That check can fail for a reason other than the process crashing, which is the only property that makes it a check.

Class two: the plugin that does not load

This is the one that has cost me the most time, because the output looks completely normal.

Many tools declare their output schema separately from the code that populates it. The formatter knows the column is called MaxRSS. Whether anything ever writes a number into MaxRSS depends on a collector, a plugin or an accounting module that may not be configured at all. When it is not, you do not get an error. You get the column, with nothing in it.

Slurm’s accounting is the canonical example. Ask for per-job memory:

$ sacct -j 1847221 --format=JobID,JobName,Elapsed,MaxRSS,AveCPU
JobID           JobName    Elapsed     MaxRSS     AveCPU
------------ ---------- ---------- ---------- ----------
1847221         run.sh     02:14:07
1847221.bat+      batch    02:14:07     8412K   02:11:53
1847221.0        python    02:13:44  41235280K  02:11:40

Now the same query with -X, which restricts output to allocation records:

$ sacct -X -j 1847221 --format=JobID,JobName,Elapsed,MaxRSS,AveCPU
JobID           JobName    Elapsed     MaxRSS     AveCPU
------------ ---------- ---------- ---------- ----------
1847221         run.sh     02:14:07

MaxRSS and AveCPU are gathered per step and stored on step records. The allocation record has the columns and no values. A reporting script written around sacct -X — a reasonable choice, since it gives one row per job — will produce a memory report for the whole fleet in which every single job used no memory. The header row is correct. The CSV parses. The chart renders. It is entirely empty of information and looks like a finding.

The same shape occurs with a metrics exporter whose collector throws. The exporter keeps serving /metrics; the failed collector’s series are simply absent. Any alert of the form rate(thing_total[5m]) > 0 on a series that no longer exists evaluates against an empty vector and never fires. The alert is not firing and the thing it watches is unmonitored, and those two states are indistinguishable on the dashboard. That is what absent() and absent_over_time() are for, and why a rule that watches a metric should usually be paired with a rule that watches for the metric’s disappearance.

The assertion that catches the whole family is the same: pick a case that must produce a populated value, and assert on the value. Not on the header, not on the row count, not on the file size.

# a job you know allocated real memory; --noconvert keeps every value in K,
# so you are not comparing 8412K against 39.33G as strings
rss=$(sacct -n -j "$JOB" --format=MaxRSS -P --noconvert \
      | tr -d 'K ' | grep -E '^[0-9]+$' | sort -n | tail -1)
[ -n "$rss" ] && [ "$rss" -gt 1000 ] || {
  echo "accounting returned no MaxRSS for a job that used memory" >&2; exit 1; }

Two details in that snippet are load-bearing, and both are easy to get wrong in a way that leaves the check passing. --noconvert matters because newer Slurm renders large values in human units — a sort -n over a mixture of 8412K and 39.33G puts the small number last. And the grep -E '^[0-9]+$' matters because step records without a value emit a blank line: strip newlines along with the K (tr -d 'K \n', which is the form that comes naturally) and every row concatenates into one integer.

$ printf '8412K\n41235280K\n\n' | tr -d 'K \n' | sort -n | tail -1
841241235280

That number is larger than any threshold you would set, so the check passes forever, on any input, including no input at all.

The interpretation that is wrong

A worked example, because this is where the trap closes.

A CPU-efficiency report built on sacct produced a fleet average of exactly 1.00. Every user, every partition, every month. The obvious reading is a perfectly tuned estate. The correct reading is that the two quantities being divided are the same quantity.

CPUTimeRAW is allocated CPU time: elapsed seconds multiplied by allocated cores. It is not consumed CPU time. Divide CPUTimeRAW by itself, or by any other allocation-derived figure, and you get 1.0 by construction, for every job that has ever run. The consumed figure is TotalCPU (system plus user), and it lives on step records — so the “one row per job” convenience of -X gives you TotalCPU of zero, and an efficiency of 0.00 for everything. Two different wrong answers, each uniform enough to look like a real property of the fleet.

# wrong: always 1.00 — CPUTime is CPUTimeRAW in a different notation
sacct -X -a -S 2026-03-01 --format=JobID,CPUTimeRAW,CPUTime -P

# workable: consumed over allocated, steps included
sacct -a -S 2026-03-01 --format=JobID,TotalCPU,CPUTimeRAW -P

Mind the units when you divide those two. CPUTimeRAW is an integer count of CPU-seconds; TotalCPU is a formatted duration, [DD-]HH:MM:SS[.mmm]. Divide the string by the integer without parsing it and you will get a number — a small, uniform, entirely fictitious one — which is the same failure this section is about, arrived at one layer further down.

Uniformity is the signal. A real efficiency distribution on a shared cluster is wide and messy, and the mean lands somewhere below half — I would treat roughly 30 to 60 percent as the plausible band for a general-purpose fleet, and I am giving that as a band rather than a figure because it depends heavily on the workload mix and I have no basis for a tighter claim. What I am confident about is the mechanism: if a derived metric is the same for every row, suspect the arithmetic before you believe the estate.

Class three: the list that is one string

Configuration parsers disagree about lists. Some split on whitespace, some on commas, some take the whole line, and some accept either and quietly prefer the wrong one. When a list is read as a single literal, it usually matches nothing, and “matches nothing” is frequently indistinguishable from “not configured” — which means a fallback to a default you did not choose.

The shell version is the one everyone has written:

$ L="mlx5_0 mlx5_1"
$ for p in "$L"; do echo "iter: [$p]"; done
iter: [mlx5_0 mlx5_1]

One iteration, one token containing a space, and any lookup keyed on it fails. Unquoted it splits into two; quoted it does not. Both are correct for some purpose, and the difference is invisible in the log because the loop body ran and exited zero.

The config-file version is more insidious because the file passes validation. OpenSSH’s server config, for most keywords, uses the first value obtained and ignores later ones. sshd -t reports the file is fine. systemctl reload sshd succeeds. Your carefully placed directive at the bottom of the file does nothing at all, because something 200 lines above set it first — often a vendor-shipped include you did not write. The only reliable check is to ask the running daemon what it actually believes:

# run as root; sshd -T reads the host keys
# sshd 9.8 and later ship as a split binary, but -T still lives on sshd itself
$ sudo sshd -T | grep -i '^ciphers'
ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr

sshd -T prints the effective configuration after all parsing and includes. The file is what you wrote; -T is what is true.

One qualification, because it is the reading people get wrong: bare sshd -T prints the configuration for a connection that matches no Match block. If the directive you care about lives inside a Match Group or Match Address, plain -T will not show it and you will conclude, wrongly, that it never took. Supply the connection you are asking about:

$ sudo sshd -T -C user=svc,host=jump.example.com,addr=203.0.113.10 | grep -i '^permitrootlogin'

The output is the effective value for that connection. There is no single effective value for a config with Match blocks in it, and a check that assumes there is will be right for most users and quietly wrong for the ones the block was written for.

The general form: where a service distinguishes between the file it was given and the configuration it is running, the effective-value output is the only thing worth asserting on, and you must ask it about the specific case you care about. Where a service offers no such output, read the value back through whatever interface it does expose — a status command, an admin socket, an API — and compare it to the value you intended, in writing, in the check.

Class four: the zero-byte index

A package index, a manifest, a rule feed and a firmware catalog all share a property: when they arrive empty, the consuming tool reports that there is nothing to do.

$ curl -s -o /var/cache/feed/index.json -w 'http=%{http_code} size=%{size_download}\n' \
    https://feeds.example.com/v2/index.json
http=200 size=0
$ echo $?
0
$ ls -l /var/cache/feed/index.json
-rw-r--r-- 1 root root 0 Apr 22 03:00 /var/cache/feed/index.json
$ ./apply-feed --index /var/cache/feed/index.json
0 entries loaded, 0 applied, 0 errors

Three zeros and a clean exit. Read it out loud: nothing to apply, so nothing was applied, so no errors. That is a correct description of a broken pipeline.

Note what that transcript is and is not, because the failure modes here do not rank the way intuition ranks them. It is a 200 with an empty body — a half-deployed origin, a proxy serving a truncated cache entry, a signing step that produced nothing. --fail does not save you, because there is no HTTP error to fail on: the transfer genuinely succeeded and what succeeded was nothing.

The counter-intuitive part is which failure is worse for the file on disk. A transport failure is the gentle one — curl exits non-zero and leaves an existing destination untouched, so the last good copy survives:

$ echo "PREVIOUS GOOD CONTENT" > index.json
$ curl -s -o index.json https://feeds.example.invalid/v2/index.json ; echo $?
6
$ cat index.json
PREVIOUS GOOD CONTENT

An HTTP error without --fail is the destructive one. The transfer succeeds, so curl writes, and the error page lands on top of the only good copy you had:

$ curl -s -o index.json https://example.com/api/index.json ; echo $?
0
$ ls -l index.json
-rw-r--r-- 1 root root 559 Apr 22 03:00 index.json

Adding --fail restores the gentle behavior — non-zero status, destination left alone. Which reframes --fail from a tidiness flag into a data-protection one: without it, every scheduled fetch is one origin misconfiguration away from overwriting a good cache with an error page, at three in the morning, exiting zero.

The reason this survives is that “no changes” is also the normal steady state. On most days the honest answer genuinely is “0 applied”. A check that only alarms on errors will never distinguish “we are current” from “we are blind”, and the second state can persist for months.

Two assertions, both cheap:

# 1. minimum plausible size, not merely non-empty
# (GNU coreutils; on BSD/macOS the equivalent is stat -f %z)
sz=$(stat -c %s /var/cache/feed/index.json)
[ "$sz" -ge 4096 ] || { echo "index implausibly small: ${sz}B" >&2; exit 1; }

# 2. a structural marker, not just bytes
jq -e '.entries | length > 0' /var/cache/feed/index.json >/dev/null || {
  echo "index parsed but contains no entries" >&2; exit 1; }

And one more that is worth the trouble on anything time-sensitive: assert on the age of the content, not the age of the file. Any transfer that opens the destination before it knows the result — a 200 that dies half way through the body, an error page written over a good cache — leaves you a fresh mtime over bytes that are stale, partial or wrong. find -mtime is then measuring when something was written, not when it was current. If the payload carries a generation timestamp, compare that instead, and treat the file’s own mtime as evidence of nothing but write activity.

Class five: the symlink that resolves nowhere

Shared software trees accumulate links. A module file or a wrapper script points at /opt/apps/tool/current, current points at a release directory, and at some point a release directory is moved, or the link is created on a build host against a path that only exists there.

The link itself is not broken in any way the filesystem will tell you about. ln -s will happily create a link to a path that does not exist. ls shows it. And the module that consumes it does this:

prepend-path PATH    $prefix/bin
prepend-path LD_LIBRARY_PATH $prefix/lib64

where $prefix was computed by resolving the link. If the resolution produced an empty string, PATH gains the entry /bin and LD_LIBRARY_PATH gains /lib64. Both are real directories. Neither errors. module load exits zero, module list shows the module loaded, and the binary you wanted is not on the path — so you get the system’s version, silently, at a different version than the one the module claims.

The distinguishing commands:

$ ls -l /opt/apps/tool/current
lrwxrwxrwx 1 root root 33 Feb 11 16:02 /opt/apps/tool/current -> /build/stage/opt/apps/tool/3.11.2

$ readlink -e /opt/apps/tool/current
$ echo $?
1

readlink -e requires every component of the resolved path to exist and prints nothing when it does not. For a check, that is the flag you want.

readlink -f is the one usually reached for, and the difference is worth spelling out because the obvious summary of it is wrong. -f is not “the same but it does not check”. It requires every component but the last to exist. So its behavior splits by which part of the target went missing:

# release directory pruned; the parent is still there → -f is happy, -e is not
$ readlink -f /opt/apps/tool/releases/3.11.2
/opt/apps/tool/releases/3.11.2
$ echo $?
0
$ readlink -e /opt/apps/tool/releases/3.11.2 ; echo $?
1

# link written on a build host; /build/stage does not exist here at all
$ readlink -f /opt/apps/tool/current ; echo $?
1

Which means a check built on -f catches the build-host case and misses the pruned-release case — and the pruned release is the one that happens on a schedule. It fails in the direction that looks fine.

Both flags are GNU coreutils. BSD and macOS readlink has no -e at all, and its -f does not behave like the GNU one, so a check that runs on a mixed estate should use test -e "$(readlink -f …)" or simply [ -e /opt/apps/tool/current ], which follows the link and answers the question directly.

The assertion is not “does the module load”. It is “after loading, does the thing run and say the right version”:

module purge
module load tool/3.11.2
command -v tool | grep -q '^/opt/apps/' || { echo "tool not from /opt/apps" >&2; exit 1; }
tool --version | grep -q '3\.11\.2'     || { echo "wrong tool version" >&2; exit 1; }

Two lines, and they fail for the right reasons: wrong location, wrong version.

Class six: the environment change inside a pipe

This one is the mirror image of the rest. Here the estate is fine and the check is wrong, and the natural conclusion — “the module is broken” — sends you off to rebuild something that was never damaged.

In bash, every element of a pipeline runs in a subshell by default. Anything it does to the environment dies with it:

$ x=1
$ echo hi | { x=2; }
hi
$ echo "x=$x"
x=1

“By default” is doing real work in that sentence, and this is the one place in the article where the portable-looking rule is the unportable one. Bash exempts the last stage if lastpipe is set and job control is off — the usual state of a non-interactive script. zsh and ksh93 run the last stage in the current shell unconditionally:

$ bash -c 'echo hi | read y; echo "bash y=[$y]"'
bash y=[]
$ zsh  -c 'echo hi | read y; echo "zsh y=[$y]"'
zsh y=[hi]

So the same validation script, interpreted by a different shell, produces a different verdict about whether the estate is broken. Neither shell is wrong. Pin the interpreter in the shebang and do not rely on which side of that line you are on.

Now the operational version. A validation script wants to load a module and capture the result:

module load fftw/3.3.10 | tee -a validate.log
echo "FFTW_DIR=[$FFTW_DIR]"

module is a shell function. Inside the pipe it runs in a subshell, sets its variables there, and the subshell exits. FFTW_DIR is empty in the parent. The script concludes the module sets nothing and reports a broken module. A second engineer loads it by hand, it works perfectly, and the disagreement gets attributed to “something about the environment”.

The same mechanism hides real failures too:

$ bash -c 'false | true; echo "rc=$?"'
rc=0
$ bash -c 'set -o pipefail; false | true; echo "rc=$?"'
rc=1

Without pipefail, a pipeline reports the status of its last command only. Pipe a failing producer into tee and the failure is gone. This is why so much automation that pipes its output to a log has no idea when it failed.

set -e has its own blind spot in the same family:

$ bash -c 'set -eu; f(){ local v=$(false); echo "inside rc=$? v=[$v]"; }; f; echo after'
inside rc=0 v=[]
after

local is itself a command, and its exit status is the status of local, not of the substitution inside it. export, declare and readonly swallow a failure the same way, for the same reason:

$ bash -c 'set -eu; export v=$(false); echo "reached"'
reached
$ bash -c 'set -eu; declare v=$(false); echo "reached"'
reached

Split the declaration from the assignment and the same script stops:

$ bash -c 'set -eu; f(){ local v; v=$(false); echo "never reached"; }; f'
$ echo $?
1

One more in the same category, worth knowing because it appears in cleanup and maintenance scripts — and worth stating carefully, because the usual version of this folklore is half true. find -exec cmd {} \; returns zero even when every invocation of cmd fails. find -exec cmd {} + does not: in GNU findutils a non-zero invocation makes find itself exit non-zero.

$ find /tmp/fecheck -type f -exec false {} \; ; echo "semicolon rc=$?"
semicolon rc=0
$ find /tmp/fecheck -type f -exec false {} +  ; echo "plus rc=$?"
plus rc=1

The terminator, not the tool, decides whether failures propagate. xargs also reports them — GNU xargs exits 123 if any invocation exits 1 to 125 — so if a maintenance script’s success depends on the command being run, use + or xargs, and do not read anything into the status of a \; form.

The rule that covers all of it, for any harness you write:

set -Eeuo pipefail

plus: do not put environment-modifying commands in pipelines, and do not hide a command substitution inside a local, declare or export.

Class seven: the suite that has never failed

The last class is not a bug. It is the reason the other six survive.

A validation suite that has returned green on every run since it was written has demonstrated exactly one thing: that it returns green. It has never been shown to be capable of returning anything else. If half its assertions were deleted tonight, tomorrow’s run would be identical, and you would never know.

This is the same argument as the one for restore drills and for fire alarms with a test button. A detector that has never been triggered under controlled conditions is decoration until proven otherwise.

The fix is negative controls: for every check, construct the fault it claims to detect, run the check, and require that it fails. Keep those injections in the repository next to the suite, and run them on a schedule.

#!/usr/bin/env bash
# negctl.sh — prove each check can fail
set -Eeuo pipefail

fixtures=/var/lib/validate/fixtures
fail=0

expect_fail() {           # expect_fail <label> <cmd...>
  local label=$1; shift
  if "$@" >/dev/null 2>&1; then
    echo "NEGATIVE CONTROL DID NOT FIRE: $label"
    fail=1
  else
    echo "ok (failed as required): $label"
  fi
}

expect_fail "empty index rejected"      check_index "$fixtures/index.empty.json"
expect_fail "html error page rejected"  check_index "$fixtures/index.404.html"
expect_fail "truncated index rejected"  check_index "$fixtures/index.truncated.json"
expect_fail "blank MaxRSS rejected"     check_accounting "$fixtures/sacct.blank.csv"
expect_fail "dangling symlink rejected" check_prefix   "$fixtures/dangling-link"
expect_fail "wrong version rejected"    check_version  "$fixtures/tool-3.10.0"

exit "$fail"

The fixtures matter as much as the script. index.404.html should be a real error page you captured, not one you typed; sacct.blank.csv should be real output from a query that really did return blank columns. Saving the broken artefact at the moment you find it is the cheapest thing in this article and the one most often skipped.

There is a second-order benefit. Running the negative controls tells you when a check has quietly stopped being able to fail — because someone loosened a threshold, or because the tool’s output format changed and a grep that used to match now matches nothing and the surrounding || true swallows it.

The discipline, condensed

Four rules, in the order they pay off.

Assert on content, not on status. Exit zero means the process reached its own idea of the end. It says nothing about the bytes it produced. Every check should name a value it expects and compare.

Write the expected value down in the check. Not “non-empty”, not “more than zero rows” — the actual string, the actual minimum, the actual version. A check that does not encode an expectation cannot detect a wrong answer, only a missing one.

Pick a case that must produce a populated result. The whole plugin-shaped failure class hides behind aggregate checks. One known job, one known file, one known value, asserted exactly, is worth more than a thousand rows counted.

Prove every check can fail. If you cannot show the injected fault that turns it red, it is not a check. Delete it or fix it, because leaving it there is worse than having nothing — it produces the feeling of coverage without the coverage.

And one habit underneath all four: when a number is uniform across every row, or a report is empty on a day when it should not be, or a pipeline says “0 applied” for the ninth week running, treat that as a finding about the measurement rather than a finding about the estate. In my experience the uniform answer is wrong far more often than it is right.

What this does not cover

Nothing here detects a computation that is wrong in a plausible way — output that is populated, well-formed and numerically incorrect. That needs a reference result, a known-answer test or an independent implementation, and it is a different and harder problem.

Nor does any of it help if the check and the thing being checked share a dependency. A probe that reads the same broken index, or resolves the same dangling link, or inherits the same empty variable, will agree with the system perfectly. When you build a negative control, inject the fault as far upstream as you can reach, and confirm the alarm travels all the way to the place a human would actually see it. A check nobody reads is the same as a check that cannot fail, arrived at by a different route.

Ansible patching: verify the running service after the package transaction

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A package update can complete while the vulnerable process remains running or the application becomes unhealthy. Patching is a controlled transition of a service, not merely a package-manager transaction.

Define the change boundary

Identify the affected inventory, application owners, package source, reboot requirements, and rollback route. Separate hosts with different availability roles. A database replica, login node, and stateless worker should not share an unexamined reboot policy.

Start with read-only inventory of installed packages, running kernels, and service health. Use vendor advisories for applicability rather than assuming that an upstream version string maps directly to a distribution’s security state.

Control the batch size

Ansible documents serial for batched execution in its strategy guide. A skeleton for a reviewed playbook is:

- name: Validate a small Linux maintenance batch
  hosts: maintenance_canary
  serial: 1
  any_errors_fatal: true
  tasks:
    - name: Record the running kernel
      ansible.builtin.command: uname -r
      changed_when: false

This example only records a kernel version; it deliberately does not install packages or reboot. Add application-specific prechecks, the approved update operation, restart handling, and postchecks through the normal change process.

Treat check mode as a planning aid

Ansible’s check-mode documentation describes module-dependent simulation. A clean check-mode run does not demonstrate a successful reboot, kernel-module compatibility, or a healthy application afterward.

Run the full workflow on a representative canary. Include external client checks, not only service-manager status. A process can be active while its database connection or authentication path is broken.

Verify the running state

After maintenance, record the installed package version and the process or kernel actually in use. Confirm mounts, dependent services, application transactions, and monitoring. For hosts with specialized NIC, storage, or GPU modules, include compatibility checks appropriate to the deployed stack.

If the canary fails, stop the rollout. Do not mark a host recovered merely because a second package command returned success. Use the established rollback or repair path and repeat the original service checks.

Account for incomplete hosts

Report unreachable hosts, skipped tasks, deferred reboots, and accepted exceptions separately. An inventory of 200 targets with 190 successful transactions is not a fully patched fleet. Preserve the original target list so disappearing machines cannot improve the success percentage.

The release record should show what was targeted, what changed, which running services were verified, and what remains outstanding. Repeat the same acceptance checks after the next image rebuild so the patched state survives replacement as well as reboot.

SELinux denials: repair the application boundary before generating policy

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

An application failing under SELinux enforcement is evidence to investigate, not proof that enforcement must be disabled. The failure may be a mislabeled path, a supported configuration option, or behavior that the service should not be performing.

Capture the actual denial

On a RHEL-family host with SELinux and audit tools installed, begin with read-only inspection:

getenforce
sestatus
sudo ausearch -m AVC,USER_AVC -ts recent
ls -Zd /srv /srv/example-app

The paths are examples. Use the real application path, and preserve timestamps so the denial can be connected to the failed transaction. Absence of an AVC does not prove SELinux is unrelated; inspect the logging state and relevant troubleshooting guidance.

Use the distribution’s supported policy and tools. The SELinux project maintains the userspace tools, while the kernel documentation points to the subsystem and policy resources.

Identify the boundary being crossed

Read the source context, target context, object class, and denied permission. Connect those to the application operation. Is a web process trying to read its content, write an upload directory, contact a database, or access something unrelated?

Check file labeling against the intended path policy. For an approved path, a nonmodifying preview can help:

sudo restorecon -nRv /srv/example-app

The -n preview reports prospective relabeling rather than applying it. A correct custom path may need a persistent file-context definition before any relabel operation; repeatedly applying a temporary label is not a durable deployment method.

Prefer a narrow explanation

If the distribution provides a documented boolean for the intended behavior, review its scope before enabling it. If the service was configured to write into a read-only content path, correcting the application layout may be the better fix.

Do not feed a large collection of historical denials into an automatic policy generator and install the result without review. That can authorize unrelated activity and preserve a misconfiguration as policy.

Validate the change in a pilot

Apply the approved label, configuration, or minimal policy change to a lab or pilot system. Run the previously failing transaction in enforcing mode. Then run an operation that should remain denied. Successful application behavior alone does not prove the boundary stayed narrow.

Reboot or redeploy the pilot where appropriate to confirm the fix survives normal lifecycle operations. A label repair that disappears at the next image rollout is incomplete.

Retain the reason

Keep the denial, diagnosis, exact change, positive test, and negative test. Document any exception with its owner and review date. This turns an operational inconvenience into a repeatable hardening improvement instead of a permanent loss of enforcement.

Four ways a storage benchmark lies and how to catch each one

2026-03-18T10:00:00-04:00

Storage benchmarks are unusually good at producing wrong answers that look right. The number comes out, it is in the units you expected, it is in the neighbourhood you hoped for, and nothing in the output says “this measurement did not test what you think it tested”. Then someone writes it in a capacity plan, or opens a support case about it, or spends a maintenance window tuning a subsystem that was never the constraint.

Four mechanisms account for most of it. Each has a specific arithmetic signature, and each has a check that costs a few minutes and settles it.

The worked figures throughout are from one eight-server erasure-coded tier on HDR200, and they are illustrative rather than a specification for anything: page pool sizes, code widths and link rates all move the numbers. The arithmetic is the part that transfers. Run it against your own configuration before quoting any figure here.

1. The read you measured was served from memory

Most benchmarking guides tell you to use O_DIRECT. fio --direct=1, IOR’s --posix.odirect, dd iflag=direct for a read or oflag=direct for a write. This is correct advice and it is routinely misunderstood, because O_DIRECT is a flag on a file descriptor on one machine. It bypasses the page cache on the client that opened the file. It has no authority whatsoever over what the storage servers do with their own memory.

On a parallel filesystem the servers hold a large read buffer. For IBM Storage Scale in its erasure-coded form, that buffer is a percentage of the daemon’s page pool, set by nsdRAIDBufferPoolSizePct. The documented default has been 50 percent historically and 80 percent in more recent releases, so read it rather than assume it. The upper bound on the cache your O_DIRECT read is still landing in is:

server-side buffer pool = pagepool × nsdRAIDBufferPoolSizePct × number of servers

That is an upper bound rather than a read cache exactly: the same pool holds write and track buffers. It is still the right number to size a dataset against, because it is the largest thing your read can hide in.

Check yours rather than assuming. Note that mmlsconfig tells you what is stored and mmfsadm dump config tells you what is actually running, and these are not always the same value. mmfsadm is a diagnostic tool that IBM documents as being for use under service direction; dump config is read-only, but treat the rest of the command surface as off-limits on a production daemon:

mmlsconfig pagepool
mmlsconfig nsdRAIDBufferPoolSizePct
# and on one storage server, what is genuinely live:
mmfsadm dump config | grep -iE 'pagepool|nsdRAIDBufferPoolSizePct'

Take a tier of eight storage servers with a 37.7 GiB page pool each and the buffer pool at 80 percent. That is 30.2 GiB per server and 241 GiB across the tier. Any read test whose working set is comfortably below 241 GiB is measuring DDR, not NVMe, no matter how many O_DIRECT flags are in the command line.

Here is what that looks like. Twelve clients, sixteen ranks each, 1 GiB per rank — 192 GiB of data, which is 80 percent of the buffer pool:

mpirun -np 192 -N 16 ./ior \
  -a POSIX --posix.odirect -w -r \
  -b 1g -t 1m -F -e -C -i 1 \
  -o /gpfs/scratch/bench/ior.dat

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)   wr/rd(s)  close(s)  total(s)
------    ---------  ----       ----------  ---------- ---------  --------  --------  --------  --------
write     68812      68812      0.002790    1048576    1024.00    0.021443  2.857     0.003120  2.882
read      159539     159539     0.001203    1048576    1024.00    0.004472  1.232     0.001196  1.238

159,539 MiB/s is 155.8 GiB/s. It is also nonsense. The IOPS column equals the bandwidth figure here only because the transfer size is exactly 1 MiB — IOR reports completed transfers per second, not GiB/s, and at -t 4m the two columns would differ by four. Now the same test with -b 16g — 3 TiB, nearly thirteen times the buffer pool:

access    bw(MiB/s)  IOPS       Latency(s)  block(KiB) xfer(KiB)  open(s)   wr/rd(s)  close(s)  total(s)
------    ---------  ----       ----------  ---------- ---------  --------  --------  --------  --------
write     68240      68240      0.002814    16777216   1024.00    0.019987  46.10     0.004011  46.13
read      83942      83942      0.002287    16777216   1024.00    0.003991  37.47     0.001330  37.48

Read falls from 155.8 to 82.0 GiB/s. The first figure was 1.9 times the cold number. Write barely moves — 68,812 to 68,240 MiB/s, or 67.2 to 66.6 GiB/s — which is the part that tells you the mechanism rather than just the magnitude: writes were never being served out of a read buffer, so they had nothing to lose.

Two tells you can spot without doing the arithmetic:

The read phase finished in a second. Any storage read phase that completes in under roughly ten seconds has measured startup transients and cache. IOR prints total(s) for exactly this reason. A 1.2 second read phase is not a measurement, it is a rounding error with units.

Read is more than twice write on hardware where it should not be. Some asymmetry is normal. A factor of 2.3 on a tier whose read and write paths use the same NICs deserves an explanation before it gets published.

Here the obvious interpretation is wrong in a way worth stating plainly. When the number falls after you enlarge the dataset, the natural conclusion is “so it was cache, and 82 is the truth”. That conclusion is probably right, but two points do not yet support it. Enlarging the dataset also lengthens the runtime, changes which regions of the address space you touch, and may cross a storage pool or fileset boundary with different placement. Any of those could depress the number on its own.

The discriminator is more points along the same axis. Run at 2×, 4× and 8× the buffer pool as well as the 12.7× above. A cache artefact produces a sharp fall and then a flat floor — for instance 155.8 at 0.8×, then 82.4, 81.6, 82.3, 82.0 as the multiple climbs. A layout or locality effect produces a continuing slide. If it keeps falling, you have found a second problem and you still do not know the cold number.

There is no supported command that flushes the server-side pool on demand, so sizing the dataset past it is the practical answer. On the clients, echo 3 > /proc/sys/vm/drop_caches clears the Linux page cache but does not touch the GPFS page pool, which is separate memory the daemon manages itself; unmounting the filesystem on that client is what invalidates its cached buffers.

2. An under-driven benchmark is indistinguishable from a ceiling

This is the most expensive of the four, because it does not merely give you a wrong number. It gives you a wrong number with a confident shape. You run at one concurrency level, get 44 GiB/s, run again and get 44 GiB/s, add a node and get 44 GiB/s, and conclude you have found a hard wall. Everything about the result says saturation. What you have actually found is that you did not ask for enough work at once.

The arithmetic is Little’s law. The number of I/Os that must be outstanding to sustain a given bandwidth is:

outstanding = bandwidth ÷ I/O size × latency

At 82 GiB/s with 1 MiB transfers, that is 88.0 GB/s ÷ 1,048,576 B = about 84,000 IOPS. If per-I/O latency at the knee is 18 ms, you need roughly 1,510 requests in flight across the whole client set to get there. If your benchmark has 384 outstanding, then at that latency you are arithmetically incapable of reaching the number, and no amount of repeating the run will reveal that.

So sweep concurrency. Twelve clients, iodepth=32, varying numjobs:

for j in 1 2 4 8 16; do
  pdsh -w node[01-12] "fio --name=r --filename=/gpfs/scratch/bench/\$(hostname)/f \
    --rw=read --bs=1m --direct=1 --ioengine=libaio \
    --iodepth=32 --numjobs=$j --size=256g --runtime=60 --time_based \
    --group_reporting --output-format=json \
    --output=/gpfs/scratch/bench/\$(hostname)/sweep-$j.json"
done

Write each node’s JSON to its own file rather than redirecting pdsh to one. pdsh interleaves and line-prefixes the output of all twelve nodes, so a single redirect gives you twelve mangled JSON documents in one file and jq will refuse it. --direct=1 is not optional here either: fio documents libaio as only supporting queued behavior with non-buffered I/O, so on a buffered job it degrades toward serialized submission and reintroduces exactly the under-driving this sweep exists to detect.

--size=256g is chosen against problem 1, not at random. With --time_based the 60-second run reads about 408 GiB from a 256 GiB file, so each node does re-read its own data — but twelve nodes hold 3 TiB between them against a 241 GiB server pool, which keeps the hit rate low enough not to distort the shape of the sweep. Shrink the file to something a single server could hold and the whole curve lifts and flattens, and you will read the artefact as a ceiling.

`numjobs` per node	Outstanding I/Os, all clients	Aggregate read (GiB/s)	Δ from previous
1	384	44.0	—
2	768	68.1	+55%
4	1,536	81.5	+20%
8	3,072	82.0	+0.6%
16	6,144	81.7	−0.4%

The shape is the result, not any single row. It climbs steeply, the climb decays, and then three consecutive points agree within one percent while concurrency quadruples. That flat tail is what licenses the sentence “this tier saturates at 82 GiB/s”. Without it you have a lower bound and nothing more.

Cross-check the knee against Little’s law. Here is the per-node fio output at numjobs=4:

read: IOPS=6955, BW=6955MiB/s (7292MB/s)(408GiB/60001msec)
    slat (usec): min=12, max=1843, avg=41.28, stdev=22.17
    clat (msec): min=2, max=118, avg=17.94, stdev=9.61
     lat (msec): min=2, max=118, avg=17.98, stdev=9.61

128 outstanding per node divided by 17.98 ms is 7,120 IOPS. fio reports 6,955, and twelve nodes at that rate is 81.5 GiB/s, the table’s numjobs=4 row. The two agree to within about two and a half percent, which means the queue was genuinely full and the latency figure is describing the same system the bandwidth figure is. When those two disagree badly — Little’s law predicting double what fio reports, say — the queue was not full and one of the figures is measuring something else.

There is a second sweep people skip, and it answers a different question. Sweep node count at fixed per-node concurrency:

If one node and twelve nodes each get the same rate, you found a per-node limit — a NIC, a PCIe link, a single-threaded path.
If one node gets the whole budget and twelve nodes divide it, you found an aggregate limit — the servers, the disks, or the fabric between them.

These are different problems with different fixes, and a single-point measurement cannot tell them apart.

The fio trap that silently caps you at one

Setting iodepth=64 with a synchronous I/O engine does nothing. sync, psync and pvsync submit one request and wait for it; fio documents that iodepth beyond 1 has no effect for them. A job file with ioengine=psync and iodepth=64 produces exactly the flat, confident, badly under-driven number described above, and the command line looks like it asked for depth.

grep -E '^(ioengine|iodepth|numjobs)' bench.fio
# ioengine=psync      <-- iodepth below is inert
# iodepth=64
# numjobs=8

With a synchronous engine, numjobs is your only source of concurrency. With libaio or io_uring, total outstanding is numjobs × iodepth per node.

3. A counter is not a rate

This is among the most common measurement errors in the field and it is entirely mechanical. Almost every hardware and driver statistic is cumulative since boot, since daemon start, or since the last reset. Reading it once and dividing by the duration of your benchmark produces a number with the right units and no meaning.

The tell is distinctive: the reported rate falls as you run the test longer, because the numerator is fixed history and the denominator is your runtime. If changing only --runtime changes your answer, you divided a counter by elapsed time.

The fix is to sample twice and difference. InfiniBand port counters, using the extended 64-bit set so they do not wrap mid-run:

lid=37; port=12; interval=30
a=$(perfquery -x $lid $port | awk -F: '/^PortXmitData/{gsub(/[^0-9]/,"",$2); print $2}')
sleep $interval
b=$(perfquery -x $lid $port | awk -F: '/^PortXmitData/{gsub(/[^0-9]/,"",$2); print $2}')
awk -v a="$a" -v b="$b" -v t="$interval" \
  'BEGIN{printf "%.2f GiB/s\n", (b-a)*4/t/1073741824}'

23.14 GiB/s

Two details in that snippet are easy to get wrong. PortXmitData is defined in the InfiniBand architecture specification in units of four octets, so the delta is multiplied by 4 to get bytes — the source of a great many capacity claims that are exactly four times too low. And -x requests the extended (PortCountersExtended) set; the basic counters are 32-bit, which at four octets per unit is 17.2 GB of traffic before they wrap. On an HDR200 port carrying 25 GB/s that is a wrap in well under a second, producing a negative delta and, if your script does not check the sign, a negative or nonsensical bandwidth. The snippet above does not check it either — add the guard before you trust it unattended.

Resist the urge to reach for perfquery’s reset flags (-r, -R) first. On a shared fabric you have just zeroed the inputs to somebody else’s monitoring, and the difference method does not need it.

The same discipline applies everywhere. NVMe wear counters, which carry a unit almost nobody gets right on the first attempt:

nvme smart-log /dev/nvme0n1 | grep -iE 'data.units'

data_units_read                     : 4,283,915,042
data_units_written                  : 1,118,730,455

The grep pattern is deliberately loose because the field label changed: older nvme-cli prints data_units_read, newer versions print Data Units Read, and a pattern pinned to the underscore form silently returns nothing on half the fleet.

A data unit here is a thousand 512-byte sectors — 512,000 bytes, not 512 and not 1 MiB. Assume 512 bytes and you are low by a factor of a thousand; assume 1 MiB and you are high by a factor of 2.05. Sample twice, subtract, multiply by 512,000, divide by elapsed.

And the ordinary network counters in /proc/net/dev and ethtool -S are cumulative too. sar -n DEV 5 and iostat -x 5 do the differencing for you, which is precisely why the first sample they print — the one covering time since boot — should always be discarded:

iostat -x 5 2 nvme0n1     # use the SECOND report, not the first

Another place the obvious reading is wrong. iostat reports %util, and on a spinning disk 100 percent %util meant the device was saturated. On NVMe it means nothing of the kind. %util is the fraction of wall time during which at least one request was in flight. An NVMe drive with 64 hardware queues can be at 100 percent %util while servicing one request at a time and running at three percent of its capability.

The fields that actually carry information are aqu-sz, the average queue depth, and r_await, the average service latency. aqu-sz is the sysstat 12 name; on sysstat 11 and earlier the same field is avgqu-sz, so a parser that greps for one of them will come back empty on the other. A device with %util=100, aqu-sz=1.2 and r_await=0.3 ms is nearly idle. The same device at aqu-sz=180 with r_await climbing run over run is the constraint. Judge NVMe by queue depth and latency, never by %util.

4. Aggregate CPU idle proves nothing on a many-core machine

On a dual-socket server with 96 physical cores and SMT enabled — 192 logical CPUs — one thread pinned at 100 percent contributes 100 ÷ 192 = 0.52 percent to the aggregate. top reports 99.5 percent idle. Every dashboard is green. The job is bottlenecked on a single saturated core and the summary statistic physically cannot show it.

mpstat has the answer, provided you ask for per-CPU output and an interval. Run without an interval and it prints averages since boot, which returns you to problem 3.

LC_ALL=C mpstat -P ALL 5 1

03:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %idle
03:16     all    0.41    0.00    0.22    0.09    0.00    0.51    0.00   98.77
03:16       0    0.20    0.00    0.20    0.00    0.00    0.00    0.00   99.60
03:16       1    0.40    0.00    0.20    0.00    0.00    0.00    0.00   99.40
03:16      37    0.20    0.00    2.61    0.00    0.00   96.99    0.00    0.20

all says 98.8 percent idle. CPU 37 says 99.8 percent busy, essentially all of it in softirq — and note that CPU 37’s 96.99 percent softirq shows up in the all row as 96.99 ÷ 192 = 0.51, which is the whole of the aggregate %soft column. The evidence is present in the summary; it is just divided by 192. Sort for the busiest core rather than reading the summary:

LC_ALL=C mpstat -P ALL 5 1 | \
  awk '$2 ~ /^[0-9]+$/ {printf "cpu%-4s busy %5.1f%%\n", $2, $3+$4+$5+$7+$8}' | \
  sort -k3 -nr | head -5

cpu37   busy  99.8%
cpu112  busy  41.2%
cpu38   busy   3.1%
cpu4    busy   1.1%
cpu0    busy   0.4%

LC_ALL=C is load-bearing rather than decorative. In several locales sysstat prints a 12-hour timestamp with a trailing AM/PM, which puts an extra field at the front of every row and shifts $3 onto %usr, $4 onto %nice and so on — the awk above then sums the wrong columns and reports a plausible, wrong number. S_TIME_FORMAT=ISO achieves the same fixed layout if you would rather keep the locale.

Summing %usr + %nice + %sys + %irq + %soft rather than 100 − %idle keeps %iowait out of the total. %iowait is not CPU work; it is idle time that happened to coincide with a blocked task, and on a 192-core box it is diluted by exactly the same factor that hides the busy core.

A single core at 96 percent softirq looks like receive-side packet processing landing on one CPU. Confirm before acting:

grep -E 'mlx5|nvme' /proc/interrupts | awk '{print $1, $(NF)}' | head
awk 'NR==1 || /NET_RX/' /proc/softirqs | cut -c1-120

Do not stop at “the interrupts are unbalanced, run irqbalance”. Read the column, not just the total. On an RDMA path the payload does not traverse the kernel network stack, so %soft on a verbs benchmark is rarely NET_RX — it is more often completion-queue work landing on the single CPU the queue pair’s vector was bound to, which irqbalance will not move because RDMA drivers pin their own vectors. And if the hot column is %sys or %usr rather than %soft, the busy core is not interrupt handling at all: it is a single-threaded submission path, in the benchmark or in a filesystem daemon. Establish which before you rebind anything:

top -H -p "$(pgrep -d, -f 'fio|mmfsd')" -b -n1 | head -20

If the hot thread belongs to your benchmark rather than to an interrupt, the fix is more jobs, not IRQ affinity — and you are back at problem 2. Rebinding interrupts to fix a problem that was never interrupts is how a tuning exercise consumes a maintenance window and changes nothing.

State the expected value before you measure

The four mechanisms above share one property: in every case the wrong number looked plausible. That is what made it survive. The defense is to decide what the number should be before you run anything, write the arithmetic down, and treat agreement as weak evidence and disagreement as a finding.

Build the budget layer by layer, cheapest measurement first. For a tier of eight storage servers on HDR200 serving twelve clients:

Layer	Per unit	Count	Aggregate	How it was obtained
Server DDR read bandwidth	~190 GiB/s	8	~1.5 TiB/s	8-channel DDR4-3200 per socket, arithmetic
NVMe random 128k read, per server	62 GiB/s	8	496 GiB/s	`fio` against raw devices, measured
HCA PCIe link (Gen4 ×16)	29.3 GiB/s	8	234 GiB/s	`current_link_speed` × `_width`, 128b/130b, arithmetic
Storage server NIC (HDR200, 200 Gb/s)	23.3 GiB/s	8	186 GiB/s	line rate, arithmetic
Inter-switch links, under load	23.1 GiB/s	8	185 GiB/s	`perfquery` delta, measured
Client NIC (HDR200)	23.3 GiB/s	12	280 GiB/s	line rate, arithmetic
Single client alone, all 12 idle	22.9 GiB/s	1	—	`fio` on one node, measured
Erasure-code read amplification	×1.88	—	99 GiB/s delivered	modeled from 8+2p, checked against `perfquery`

The PCIe row is worth a caution, because it is the row most often quoted as a measurement when it is nothing of the kind. current_link_speed and current_link_width under /sys/bus/pci/devices/ report what the link negotiated, not what it carries: Gen4 ×16 at 16 GT/s with 128b/130b framing is 31.5 GB/s, or 29.3 GiB/s, and no adapter delivers that after TLP headers and DMA overhead. Read it as a ceiling that is comfortably above the NIC line rate, which is the only conclusion the row has to support here.

The binding constraint for a cold read is the storage-side NIC aggregate, reduced by the read amplification of the erasure code. With k data strips distributed one per node, delivering one byte to a client moves roughly 1 + (k−1)/k bytes on the storage fabric: the node fronting the client holds 1/k of the track locally, pulls the other (k−1)/k across the fabric from its peers, and then sends the whole byte on to the client. For k = 8 that is 1.875. 186 ÷ 1.88 is 99 GiB/s of deliverable read bandwidth. The measured 82 GiB/s is 83 percent of that, which is a believable efficiency for a real filesystem and is therefore not interesting.

This is a model, not a measurement, and it is the weakest row in the table. It assumes full-track reads with no coalescing and no parity-strip traffic on a healthy array, and it will be wrong under rebuild, wrong for partial-track reads, and wrong for any layout that places more than one strip per node. Treat it as the right order of magnitude for the correction rather than a constant, and confirm it by comparing fabric bytes from perfquery against client bytes from IOR on the same run.

The 155.8 GiB/s figure is 157 percent of that budget. It is not physically impossible — it sits at 84 percent of the raw 186 GiB/s NIC aggregate — but it can only be reached by removing a layer from the path, and the excess tells you which layer. A buffer-pool hit serves the reconstructed logical track from the server’s own memory, so it skips both the NVMe read and the inter-node strip gather, and the ×1.88 amplification disappears with them. The number went above the budget because the budget’s most expensive term stopped applying. Reading it as “the storage is faster than we modeled” inverts the finding exactly.

That inversion is the whole discipline. A number that exceeds the budget is the loudest alarm your benchmark can raise, and it is the one people are least likely to investigate, because it is the number they wanted.

Two unit traps will move a figure enough to matter. One GB is 10⁹ bytes and one GiB is 2³⁰; the gap is 7.37 percent, and one step down the scale the MiB-to-MB gap is 4.86 percent. fio prints both — BW=6955MiB/s (7292MB/s) — and IOR prints MiB/s only. Take fio’s binary figure, relabel it with the decimal unit, and you understate by 4.9 percent at the MiB level or 7.4 percent at the GiB level: roughly the size of the improvements people schedule maintenance windows for. And link rates are decimal: HDR200 is 200 × 10⁹ bit/s, which is 25.0 GB/s and 23.3 GiB/s, not 25.

Symptom against true cause

Symptom	Tempting conclusion	Also consistent with	Check that distinguishes them
Read ≫ write on symmetric hardware	Read path is better optimized	Read served from the server buffer pool	Re-run at 2× and 4× the pool size; cache gives a sharp fall then a floor
Read phase finished in under 10 s	Fast storage	Working set fits in cache	`total(s)` in IOR; raise `-b` until the phase runs 60 s
Throughput flat across repeated runs	Saturation	Under-driven at a fixed, insufficient depth	Sweep `numjobs`; a real ceiling is flat while concurrency quadruples
Throughput flat as nodes are added	Aggregate ceiling	Per-node NIC or single-thread limit	Compare 1-node and N-node per-node rates
`iodepth` raised, nothing changed	Queue already full	Synchronous ioengine ignoring `iodepth`	`grep ioengine` in the job file
`%util` at 100 percent	Device saturated	One request in flight on a 64-queue device	`aqu-sz` and `r_await`, not `%util`
Rate falls as runtime rises	Thermal or cache-fill effect	Cumulative counter divided by elapsed	Sample the counter twice and difference
Bandwidth exactly 4× too low	Link running at 1X	`PortXmitData` read without the ×4	Multiply the delta by 4; confirm width with `ibstatus`
CPU 98 percent idle under load	Not CPU-bound	One core at 100 percent out of 192	`mpstat -P ALL`, sort by busiest core
Total rises as clients are added	Storage scales	You were client-limited the whole time	Per-node rate at 1 node versus N nodes
Second run much faster than the first	Warm-up effect	Second run read what the first run wrote	IOR `-C`, and drop client caches between phases
Bandwidth and latency both healthy, job still slow	Storage is fine	Metadata-bound, not bandwidth-bound	`mdtest` create/stat/remove rates
Array tool reports twice what `fio` reports	`fio` is misconfigured	Fabric counters include the strip gather	Compare the ratio against `1 + (k−1)/k`
Result matches the vendor datasheet	Configured correctly	Measured a cache, or copied the expectation	Compute the budget independently first

What to keep

Three habits close most of the gap.

Write the expected number down before the run, with the arithmetic that produced it, in the same file as the result. This costs five minutes and converts every benchmark from a measurement into a test with a pass condition.

Never report a single point. Report a sweep, and report the shape. “82 GiB/s” is a claim; “82 GiB/s, flat within one percent from 1,536 to 6,144 outstanding I/Os, at 3 TiB against a 241 GiB server buffer pool” is a result somebody else can check.

And when the number comes out better than you expected, stop and find out why before telling anyone. In the shape of problem described here, that is nearly always the moment the measurement broke.

Management networks: isolate the systems that can rebuild everything

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A management interface often has authority below the operating system’s security controls. A BMC can affect power and console access; a provisioning service can replace a host image. These paths deserve their own network and identity review.

Inventory privileged control planes

Include Dell iDRAC, HPE iLO, hypervisor managers, storage administration, switch management, backup consoles, and image repositories. Record the supported firmware or software release and the intended management route.

Do not discover authority solely by scanning ports. Some capabilities exist behind a shared API or reverse proxy, while an open port says little about who can use it. Pair network observations with configuration and role review.

Specify allowed flows

Use an explicit matrix rather than “management VLAN accessible to IT”:

Source	Destination	Purpose
Approved admin workstation/bastion	Selected management interface	Interactive administration
Monitoring service	Read-only management endpoint	Health collection
Provisioning service	Defined hosts and repositories	Controlled image lifecycle
Ordinary user network	Management interface	Denied unless specifically justified

This is a design example. Actual protocols and ports come from the installed product documentation. Some products need additional flows for discovery, virtual media, or telemetry.

Validate identity and transport

Redfish is a standardized management API, but implementations, authentication, and available resources differ. Use DMTF’s Redfish specifications and the hardware vendor’s release documentation. Do not assume one manufacturer’s endpoint layout or role model applies to another.

Use named administrative identities where supported, restrict automation to required operations, and validate certificates. An emergency script that disables TLS verification permanently removes an important check from a highly privileged path.

Test isolation without changing host power

From an approved administrator path, verify an authorized read-only health operation. From a representative ordinary client network, confirm the management service is unreachable or access is denied as designed. Test IPv4 and IPv6 where both are deployed.

Then verify emergency console access to a pilot host under a documented procedure. Do not issue resets or power operations simply to test API authentication.

Keep management available during an outage

Avoid making the only recovery route depend on the application network being repaired. Document access during firewall failure, directory outage, and loss of the primary bastion. Protect recovery credentials separately and monitor their use.

A segmented network with one shared administrator password still has a concentrated credential risk. Conversely, excellent identity controls do not justify exposing management services unnecessarily.

Acceptance record

Keep the inventory, allowed-flow matrix, read-only access tests, denied-path tests, firmware review, and emergency-access procedure. Repeat after network changes and hardware additions. The management plane is a moving estate, not a one-time VLAN configuration.

When a storage read ceiling is really an InfiniBand routing problem

2026-02-11T10:00:00-05:00

There is a symptom shape that sends people looking in the wrong layer for a week. A parallel filesystem writes close to the number the design predicted and reads at roughly half that. Nothing is saturated. Client adapters sit well below half their line rate, the storage servers are not CPU-bound, the disks are not busy, and fabric monitoring shows aggregate utilization in the twenties. Every component reports headroom and the aggregate refuses to move. On the storage servers, during the read phase that is supposedly the problem:

$ iostat -x 5 | grep -E 'Device|nvme0n1'   # write-side columns trimmed
Device   r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz  aqu-sz  %util
nvme0n1  812.4  830617.6    0.0   0.00    0.41   1022.6    0.33  31.20

Thirty-one percent device utilization and sub-millisecond read latency, with a queue that never gets deeper than a third of one outstanding request. The disks are not the constraint.

The reflex is to call it a storage problem, because the number you are unhappy with came out of a storage benchmark. That reflex is usually wrong when the asymmetry is this clean. Read and write paths share almost everything. The short list of genuine differences is readahead, the disk-side read/write mix, and the direction the bytes travel. Rule out the first two and the third is not an exotic explanation, it is the only one left.

Why the fabric is a plausible suspect at all

InfiniBand unicast routing, in the configuration almost everyone runs, is static and destination-based. Each switch holds a linear forwarding table mapping a destination LID to exactly one output port. Not a set of ports, not a hash over flows — one port. Every packet addressed to that LID leaves through that cable for as long as the table stands.

Where two switches are joined by several inter-switch links, the subnet manager decides which link each destination uses. OpenSM’s default engine is Min Hop, and its balancing rule is in its own documentation: among ports offering the same hop count, pick the one with fewer LIDs already assigned. That is round-robin over destinations.

Read that rule again, because the whole article is in it. The engine balances the number of destination addresses per output port. It has no idea which of those addresses carry traffic, how much, or when. Two ports holding eight LIDs each are equally loaded as far as the routing engine is concerned, even if all the traffic in the cluster is addressed to three LIDs that happen to share one port.

Apply that to a filesystem and the two directions stop being symmetric:

Writes are addressed to the storage servers. Eight of them over eight inter-switch links, and round-robin over eight consecutive GUIDs puts one storage LID on each. The spread is perfect. That is not good engineering, it is small-n luck — reliable luck, which is why nobody notices.
Reads are addressed to the clients, drawn from a much larger pool: every compute node in the subnet, not just the ones in this job. Where a given client lands in the global round-robin has nothing to do with whether it is running your job. If the spacing between your job’s nodes in the routing order aliases against the number of links, groups of them share one cable and other cables carry nothing.

The aliasing is not hypothetical. OpenSM ships an option whose purpose is to break it — scatter_ports, documented as randomizing port selection “rather than using a round-robin algorithm (which is the default)”. Options do not get written for problems nobody has.

I would not try to predict from first principles which links a given fabric collapses onto. The alignment depends on GUID ordering, discovery order, how many LIDs sit behind the same links, and whether the tables were computed before or after the last cable was plugged in. Measure it; the measurement is cheap and the prediction is not.

Finding the inter-switch links

Before any counters, establish which physical ports are ISLs and confirm they run at the width and rate you think. One link that came up at 1X instead of 4X produces a similar-looking aggregate shortfall from a completely different cause, and it is embarrassing to find in week two.

$ ibswitches
Switch  : 0x043f720300xxxxxx ports 40 "SwitchA" enhanced port 0 lid 1 lmc 0
Switch  : 0x043f720300yyyyyy ports 40 "SwitchB" enhanced port 0 lid 2 lmc 0

$ iblinkinfo -S 0x043f720300yyyyyy | grep -i switch
       2   25[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   13[  ] "SwitchA"
       2   26[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   14[  ] "SwitchA"
       2   27[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   15[  ] "SwitchA"
       2   28[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   16[  ] "SwitchA"
       2   29[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   17[  ] "SwitchA"
       2   30[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   18[  ] "SwitchA"
       2   31[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   19[  ] "SwitchA"
       2   32[  ] ==( 4X  53.125 Gbps Active/  LinkUp)==>  1   20[  ] "SwitchA"

Eight links, all 4X, all at the same per-lane rate. iblinkinfo prints the per-lane figure, not the link figure: 53.125 Gbps across four lanes is HDR, a 200 Gb/s link. Read that column as the link rate and you will invent a capacity problem that does not exist.

The second trap is to take the encoding overhead off twice. HDR uses 64b/66b line coding, but the 200 Gb/s that everyone quotes for 4X HDR is the IBTA data rate, already net of that coding — the per-lane signalling rate is the 53.125 Gbps in the column above. The same relationship holds one generation down, where EDR’s 25.78125 Gbps per lane × 64/66 is exactly 25, and 4X EDR is exactly 100 Gb/s. Dividing 200 by 66/64 a second time gets you 193.9 Gb/s and a ceiling that is 3% too low for no reason.

So: 200 Gb/s is 25 GB/s, or 23.3 GiB/s, and that is the theoretical figure. Subtract IB transport headers, ICRC and flow control and a healthy 4X HDR link delivers around 22.6 GiB/s of payload in practice, which is roughly what ib_write_bw returns on an idle one. That 22.6 is the per-link ceiling used throughout this article, and eight links give about 180.8 GiB/s each way. It is a measured number rather than a derived one, so measure it on your own hardware before you quote a percentage against it.

Reading per-ISL counters

perfquery reads the performance management agent on a switch port. Query the ISL ports on one switch and you get both directions from one place: PortXmitData leaves that port toward the far switch, PortRcvData arrives from it. On a switch at the storage side, reads are PortXmitData and writes are PortRcvData. Use the extended counters; at these rates it is not optional:

$ perfquery -x 2 25
# Port extended counters: Lid 2 port 25
PortSelect:......................25
CounterSelect:...................0x0000
PortXmitData:....................347892350976
PortRcvData:.....................183609851904
PortXmitPkts:....................339741882
PortRcvPkts:.....................179309714
PortUnicastXmitPkts:.............339740115
PortUnicastRcvPkts:..............179307946
PortMulticastXmitPkts:...........1767
PortMulticastRcvPkts:............1768

Two things about those numbers before you divide anything by anything.

The data counters are in units of four octets. The IBA specification defines PortXmitData and PortRcvData as the number of data octets divided by four, summed over all virtual lanes. Multiply by 4 to get bytes. Forget this and you get a number four times too small — comfortingly close to a quarter of link rate, and exactly the kind of wrong answer that survives a sanity check.

The basic 32-bit counters wrap in well under a second. A 32-bit counter in units of 4 octets tops out at 2^32 × 4 = 17.18 GB, or 16 GiB. On an HDR link carrying 21.6 GiB/s that is 0.74 seconds; on NDR, half that. Sample the basic counters over a 60-second window and you are measuring the remainder after eighty wraps. The -x flag, and only that flag, gives the 64-bit versions. The extended counter attribute is optional in the specification — a device advertises it in the performance class CapabilityMask — but present on anything you are likely to be buying.

One counter does not come along for the ride. PortXmitWait, which matters later, lives in the basic PortCounters attribute and not in PortCountersExtended, so perfquery -x will not show it. Read it with plain perfquery, and remember it is 32 bits wide like everything else in that attribute.

Sampling twice and differencing

Counters are cumulative. The measurement is a difference over a known interval while a known workload runs. Do not reset them if anything else on the site reads them; you will corrupt a monitoring baseline, and the delta method does not need a reset.

#!/bin/bash
# per-ISL throughput, both directions, over a fixed window
SW_LID=2
PORTS="25 26 27 28 29 30 31 32"
WINDOW=60

sample() {
  for p in $PORTS; do
    perfquery -x "$SW_LID" "$p" \
      | awk -v p="$p" '
          /^PortXmitData:/ { gsub(/[^0-9]/,"",$0); x=$0 }
          /^PortRcvData:/  { gsub(/[^0-9]/,"",$0); r=$0 }
          END { print p, x, r }'
  done
}

sample > /tmp/isl.t0
sleep "$WINDOW"
sample > /tmp/isl.t1

join /tmp/isl.t0 /tmp/isl.t1 | awk -v w="$WINDOW" '
  { dx=($4-$2)*4; dr=($5-$3)*4;
    printf "port %-3s  xmit %7.2f GiB/s   rcv %7.2f GiB/s\n",
           $1, dx/w/1073741824, dr/w/1073741824 }'

The gsub is there because perfquery pads with dots rather than whitespace, so the value is not in a predictable field. Parsing it as $2 works on some releases and silently returns zero on others.

Run it twice: once under a write-only workload, once under a read-only workload over a dataset too large for client or server page cache. If the read set fits in cache you measure memory, get a spectacular number and learn nothing.

The table that ends the argument

A twelve-client run against eight storage servers, sampled over 60 seconds on the storage-side switch. The left columns are measured; the right two come from the forwarding table, which comes next.

ISL port	Write run, GiB/s	Read run, GiB/s	Read-run link use	Active client LIDs routed here	Total LFT entries via this port
25	11.2	21.6	96%	6	9
26	11.6	21.3	94%	5	8
27	11.4	7.4	33%	1	9
28	11.3	0.0	0%	0	8
29	11.5	0.0	0%	0	9
30	11.4	0.0	0%	0	8
31	11.2	0.0	0%	0	9
32	11.8	0.0	0%	0	8
Total	91.4	50.3	28% mean	12	68

Read the write column first. Eight links carrying 11.2 to 11.8 GiB/s, every one at about half its 22.6 GiB/s ceiling. That is a healthy fabric, and it is why nobody suspected the fabric: anyone who looked at ISL utilization during a write test saw 50% and moved on.

Now the read column. Five of eight cables carry nothing. Two carry 21.6 and 21.3 GiB/s, which is 96% and 94% of what an HDR link delivers as payload. Those two are full. The mean across all eight is 28%, and 28% is what an aggregate fabric graph shows, which is why an aggregate fabric graph cannot find this. The unit of scarcity is a cable, and averaging over cables destroys the information you need.

The row that makes it certain

Port 27 is the most useful row in the table. One client is routed over it, and it reads at 7.4 GiB/s. In the write run twelve clients moved 91.4 GiB/s, or 7.62 GiB/s each. A client with an uncontended path therefore reads within 3% of what it writes. There is nothing wrong with readahead, prefetch, the disks, the servers or the client: a client that gets its own cable performs as designed.

That row turns “the fabric is suspicious” into “the fabric is the constraint”, and it lets you predict the aggregate rather than merely describe it. Build the prediction only from quantities that did not come out of the read run itself, or you are adding up the answer and calling it a forecast. Two of them qualify: the per-link payload ceiling, established on idle hardware, and the per-client rate from the write run.

two ISLs at payload ceiling    2 × 22.6 = 45.2 GiB/s
one uncontended client (write-run rate) =  7.6 GiB/s
                              predicted   52.8 GiB/s
                               measured   50.3 GiB/s   (-4.8%)

Within 5%, from two numbers neither of which was measured during the read test, with the shortfall in the direction you would expect — a shared link running a little under its idle ceiling. That is a different class of evidence from a plausible story. Adding the measured read column to 50.3 is arithmetic, not a prediction, and it is worth keeping the two apart when someone challenges the result.

The obvious interpretation, and why it is wrong

Everyone seeing that table for the first time says the same thing: five cables are dead, so five cables are broken, check the transceivers.

They are not broken. Look one column left. In the write run those same five cables carried 11.2 to 11.8 GiB/s each, indistinguishable from the two that saturate on reads. The link is up, at 4X, at the right rate, with no errors:

$ ibqueryerrors -c
## Summary: 22 nodes checked, 56 ports checked, 0 ports have errors beyond threshold

ibqueryerrors with no target sweeps the fabric and prints only the ports that have something wrong, which is the behavior you want here. Two of its flags read backwards from the obvious guess, and both will cost you: -s takes a list of counters to suppress, not to select, so naming the counters you care about is the one way to guarantee you do not see them. And -k / -K clear error and data counters as they read — run those on a production fabric and you have silently reset the baseline of whatever else monitors it, which is the same mistake the counter-differencing section warns against. -c, used above, suppresses the common side-effect counters and clears nothing.

A cable carries nothing on reads because no destination LID with read traffic is mapped to it, and a full share on writes because storage LIDs are. Same cable, same hour, two different pictures depending on which way the bytes go.

The second wrong interpretation is subtler and costs more. “No link is saturated, therefore the fabric is not the bottleneck.” Sound when links are used uniformly; useless here, because the reported statistic uses a denominator that includes five cables the traffic cannot reach. The correct denominator is the capacity the routing tables permit: 3 × 22.6 = 67.8 GiB/s, of which the run achieves 50.3. Seventy-four percent of a hard ceiling with two of three links at 95% is a saturated network.

Corroboration without a disk in the path

You now have a fabric-shaped explanation built entirely from fabric counters. Before acting on it, prove the ceiling exists with the filesystem out of the picture. If a pure RDMA test between the same endpoints, in the same direction, with no block device in the path, lands near the filesystem’s number, the bottleneck is at or below the network and no storage tuning will move it.

Run RDMA writes from the storage nodes to the clients, so bytes travel in the read direction. Start all pairs together; one pair at a time tells you nothing about aggregate behavior.

# on each client (receiver), one server process per pair
ib_write_bw -d mlx5_0 -F -q 4 -s 1048576 -D 30 --report_gbits -p 18515 &

# on each storage node (sender), one process per assigned client, with -p
# matching that client's listener. Twelve pairs over eight senders means some
# nodes run two; give each pair its own port or the second will fail to bind.
ib_write_bw -d mlx5_0 -F -q 4 -s 1048576 -D 30 --report_gbits -p 18515 <client-ip>

 #bytes  #iterations   BW peak[Gb/sec]  BW average[Gb/sec]  MsgRate[Mpps]
 1048576    112367        32.10              31.42             0.003746
 1048576    112116        31.88              31.35             0.003737
 1048576    133324        38.04              37.28             0.004444
 ...
 1048576    227273        64.02              63.55             0.007576

Twelve pairs, summed: 51.0 GiB/s. The filesystem read run gave 50.3 GiB/s. The gap is 1.5%, and there is no disk, no metadata server, no page cache and no filesystem client in the RDMA path.

The per-pair spread corroborates the routing story more sharply than the total does. The pairs fall into three groups rather than scattering: six at about 31.4 Gb/s, five at about 37.3 Gb/s, and one at 63.6 Gb/s. Those are the ISL ceilings divided by the number of pairs sharing them — 21.6 GiB/s split six ways, 21.3 GiB/s split five ways, and port 27’s single client taking the cable to itself at about 7.4 GiB/s. The grouping matches the ibroute counts exactly, in a tool that knows nothing about the filesystem and was never told which clients share a cable.

The MsgRate column is a free consistency check on the whole block, and worth using. At 31.42 Gb/s with 1 MiB messages the rate is about 3,750 messages per second, which perftest reports in Mpps as 0.00375. A message rate of several whole Mpps next to a megabyte message size is arithmetically impossible — it would be terabits — and seeing one means the run used a different -s than you think, or the figure was pasted from somewhere without being read.

This test is only corroboration if the pairing is the same. If you let the test pick a different set of nodes, or a different number of them, you have changed which LIDs are destinations and therefore which cables carry the traffic. A network-only test on a different node list can easily come out fast and send you back to the storage layer for another week.

The one command that would have short-circuited all of it

Everything above reconstructs something the switch will simply tell you. ibroute dumps a switch’s linear forwarding table: destination LID on the left, output port on the right.

$ ibroute 2 | head -10
Unicast lids [0x0-0x2f] of switch Lid 2 guid 0x043f720300yyyyyy (SwitchB):
  Lid  Out   Destination
       Port     Info
0x0001 025 : (Switch portguid 0x043f720300xxxxxx: 'SwitchA')
0x0003 025 : (Channel Adapter portguid 0x...: 'client-a HCA-1')
0x0004 026 : (Channel Adapter portguid 0x...: 'client-b HCA-1')
0x0005 025 : (Channel Adapter portguid 0x...: 'client-c HCA-1')
0x0006 026 : (Channel Adapter portguid 0x...: 'client-d HCA-1')
0x0007 025 : (Channel Adapter portguid 0x...: 'client-e HCA-1')
0x0008 026 : (Channel Adapter portguid 0x...: 'client-f HCA-1')

Count how many of your job’s LIDs land on each output port:

# $JOB_LIDS = path to a file holding the LIDs of the clients in the run,
# one per line, in the same 0x%04x form ibroute prints (0x0003, not 3).
$ ibroute -n 2 \
    | awk 'NR==FNR {want[$1]; next}
           FNR>3 && ($1 in want) {c[$2]++}
           END {for (p in c) printf "port %s: %d\n", p, c[p]}' "$JOB_LIDS" - \
    | sort
port 025: 6
port 026: 5
port 027: 1

Match the LIDs as whole fields, not as substrings. An earlier version of this used grep -Ff against the LID list with ^ prefixed to each line, which does nothing useful: -F makes the ^ a literal character to search for, so the pattern matches no line at all and the pipeline reports an empty distribution — a result indistinguishable, at a glance, from a fabric with no job traffic on it. The awk form above compares field 1 for equality and has no such failure mode.

Twelve clients, three of eight available cables, two of them holding eleven of the twelve. That is the answer. It takes about four seconds and needs no benchmark, no maintenance window and no counter arithmetic.

The reason it is not the first thing anyone tries is structural. Parallel filesystem tuning documentation is thorough about page pool sizing, block size, prefetch depth and queue depths, and says essentially nothing about inter-switch links or routing engines, because those belong to another vendor’s product and another team’s runbook. A filesystem-shaped investigation follows a filesystem-shaped checklist, and that checklist has no entry for “count how many of your nodes are routed over the same cable”. The elimination order below exists to add one.

Telling this apart from the things it resembles

Symptom shape	Likely cause	Command that distinguishes	What you see if it is this
Both directions slow, ISLs even and well below ceiling	Not the fabric. Storage or client concurrency.	`iostat -x 5` on servers; rerun with 2× the clients	Device utilization high, or throughput scales with client count
One direction slow, several ISLs at exactly zero	Static routing collapse	`ibroute <sw-lid>`	Active destination LIDs concentrated on a few output ports
One direction slow, all ISLs loaded, all near ceiling	Genuinely out of ISL bandwidth	the perfquery table	Aggregate ≈ number of ISLs × per-link payload ceiling
Slow and erratic, ISLs uneven but none at zero	A degraded link	`iblinkinfo \\| grep -v "4X.*53.125"`	One link at 1X, or negotiated to a lower rate
Slow, `PortXmitWait` climbing, no link near ceiling	Credit starvation from a slow receiver downstream	plain `perfquery <lid> <port>` deltas on `PortXmitWait` (not in the `-x` set)	XmitWait rising on the ports that feed one endpoint
Slow with rising error counters	Physical: cable, connector, transceiver	`ibqueryerrors -c`, twice, differenced	Non-zero deltas isolated to specific cables
Slow for some jobs and not others	Allocation-dependent LID mapping	rerun on a different node list	The ISL distribution changes with the node list

The last row causes the most confusion on a shared cluster. Once the mechanism is clear it follows that the throughput a user gets depends on which nodes the scheduler handed them, and that two runs of an identical job on identical hardware can differ by a factor of two with nothing wrong anywhere. A benchmark result quoted without its node list is not reproducible, and a regression that comes and goes between runs may be the scheduler rather than the storage.

PortXmitWait deserves a note. It counts ticks during which a port had data to send and no credits to send it with, which makes it the best early indicator of downstream congestion and also famously noisy. Difference it like the data counters — but with plain perfquery, since it is a basic 32-bit counter and the -x attribute does not carry it:

$ perfquery 2 25 | grep PortXmitWait
PortXmitWait:....................18446231

Non-zero XmitWait is normal on a busy fabric. Only the rate of change, on specific ports, correlated with a specific workload, means anything. Do not raise a ticket on an absolute value. The tick is a device-specific unit rather than a fixed interval, so treat the counter as an ordering between ports on the same switch and not as a duration you can convert to seconds.

The elimination order

#	Step	Command	Cost	Rules out
1	Confirm every ISL is at full width and rate	`iblinkinfo -S <switch-guid>`	seconds	A degraded cable masquerading as a capacity shortfall
2	Count active job LIDs per ISL output port	`ibroute -n <sw-lid>`	seconds	Static routing collapse — or confirms it outright
3	Difference per-ISL counters under a write load	`perfquery -x`, two samples	2 minutes	Establishes the healthy-direction baseline
4	Repeat under an uncached read load	same	2 minutes	Produces the asymmetry table
5	Reproduce the ceiling with RDMA only, same pairing	`ib_write_bw` fan	10 minutes	Everything above the network: filesystem, cache, disk
6	Check error and wait counters over the same window	`ibqueryerrors`, `PortXmitWait` deltas	minutes	Physical faults and downstream credit starvation
7	Only now, touch a filesystem tunable	—	hours to days	—

Steps 1 and 2 take under a minute and answer the question on most fabrics. Everything after them builds a case you can hand to someone else, which matters, because the remedy usually needs a change to a subnet manager another team owns.

What you can actually do about it

Adaptive routing. The real fix is to stop choosing the output port once per destination and choose it per packet based on port load. On NVIDIA switches this comes from the subnet manager, by selecting an AR-capable routing engine — the ar_* family, such as ar_updn, alongside further ar_ options that control the mode and which service levels participate:

# opensm.conf — illustrative; take the exact directives from your own man page
routing_engine ar_updn

Three caveats, and the third is the one that bites. These engines are NVIDIA extensions shipped with their subnet manager rather than upstream OpenSM, so whether you have them at all depends on which SM package you installed. Adaptive routing can also deliver packets out of order, which reliable-connected queue pairs do not enjoy, so the SM generally restricts it to adapters advertising tolerance for it. And the option names, defaults and permitted values in this family have moved between MLNX_OFED and DOCA-OFED releases — read them out of the opensm man page and sample opensm.conf shipped with the package you actually have, not out of an article, this one included.

Then verify it took rather than assuming. NVIDIA’s smparquery reads the vendor-specific adaptive-routing MADs from a switch, and ibdiagnet reports AR state in its fabric summary; check invocation syntax against your installed version, as it differs between tool generations.

More links to spread over. Adding ISLs raises the ceiling without fixing the mechanism. The distribution improves with the number of cables, but only up to the number of active destinations, and only if the aliasing does not follow you. The collapse can reappear at the next SM sweep with a different node allocation.

Break the round-robin. scatter_ports takes a seed and randomizes selection among equally loaded ports instead of cycling. One line, no hardware implication, attacks the aliasing directly — and it is a lottery, not a guarantee: you swap a systematically bad distribution for a randomly chosen one, usually much better and occasionally not.

Route the important nodes first. guid_routing_order_file sets the order in which port GUIDs are routed for Min Hop and Up/Down. Put the genuinely hot GUIDs at the top and they take the clean round-robin before the rest of the subnet consumes the balance. Targeted, deterministic and underused.

Change the engine. On a genuine fat tree with all channel adapters at the leaves, ftree routes with the topology in mind rather than counting LIDs. It falls back rather than running on a topology it does not recognize, which is a feature. Note that ftree does not support LMC greater than zero, and several of the other engines carry restrictions of their own on LMC and on topology; check the constraints for the specific engine and OpenSM version before planning around one.

Raise LMC, carefully. Giving each port 2^LMC LIDs lets the engine compute different paths to the same port. It helps only if the upper-layer protocol issues path records for the alternate LIDs, which many do not, and the cost multiplies across every port in the subnet and every forwarding table entry needed to reach them. Establish that your client uses multiple paths first.

Check who runs the subnet manager. Where the SM runs on a managed switch, the embedded manager exposes only a subset of these options through the switch CLI, and on some firmware levels routing engine selection is not in it. Moving the SM to a host gives the full option set and a config file you can version. Find out what you have before planning around it:

$ sminfo
sminfo: sm lid 1 sm guid 0x043f720300xxxxxx, activity count 74412 priority 14 state 3 SMINFO_MASTER

sminfo reports the SM’s port GUID, which for a switch’s management port is normally the same value ibswitches prints as the node GUID. A match there means the fabric is being managed from the switch rather than from a host.

What to claim afterwards

Be careful about the size of the claim. The evidence supports this: for this node allocation, in the read direction, the achievable aggregate is bounded by three inter-switch links, two of them at 95% of payload ceiling, and a network-only test with no storage in the path reproduces the bound to within 2%. Narrow, strong, checkable.

It does not support “the storage is fine”. The storage was never tested above 50.3 GiB/s on reads because the network would not deliver more. When the routing is fixed, expect a second ceiling behind the first. That is the ordinary shape of performance work.

It also does not support a general statement about the fabric. Change the node list and the numbers change. Quote any figure with the node count, the direction, the dataset size relative to cache, and the ISL distribution that produced it. Without those four the number is not reproducible, and someone will eventually try to reproduce it.

Vault and service accounts: rotation must reach the running application

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A secret manager can issue a new credential while the application continues to use the old one from a configuration file or a connection pool. The security outcome depends on the consumer, not just the successful rotation task.

Start with one service identity

Choose a noncritical application and record which database or API it accesses, the required privileges, how it receives credentials, and how it refreshes them. Separate deployment credentials from runtime credentials. A build system should not automatically inherit the application’s production data access.

HashiCorp Vault’s database secrets engine documents dynamic credentials and database-specific integrations. Supported plugins and credential behavior vary, so use the exact database and plugin documentation before implementation.

Define the credential contract

Property	Question
Scope	Which operations and resources are allowed?
Lifetime	How long may a credential remain valid?
Renewal	Who renews it and what happens on failure?
Revocation	How does the backend actually invalidate it?
Recovery	How does the application restart if Vault is unavailable?

The contract should also identify where credentials might be copied: environment variables, logs, crash dumps, temporary files, or deployment artifacts. Moving the original into Vault does not remove existing copies.

Test the consumer lifecycle

In a lab database, issue a least-privilege credential through the intended integration. Start the application, perform a permitted operation, and confirm that an out-of-scope operation is denied.

Allow renewal or rotation to occur through the actual application workflow. Confirm new connections use valid credentials and that the application handles the transition without accumulating connection failures. Then test expiry or revocation on the disposable identity and observe both new and existing connections according to the database’s behavior.

Do not assume a revoked lease instantly terminates every established session. Establish the backend-specific outcome and document any additional response step.

Protect bootstrap authority

The application needs an initial way to authenticate to Vault. Review that mechanism as carefully as the database credential. A permanent broad token embedded in an image can recreate the original problem at a more powerful layer.

Prefer workload identity methods supported by the environment, scope policies tightly, and verify the audit trail. Avoid displaying live secret values in routine validation output.

Plan for service loss

Test a temporary loss of the secret-management path in a lab. Distinguish an already running application from a cold restart. Record the maximum acceptable interruption and the approved recovery path. Do not silently extend credentials indefinitely to make an availability test pass.

A completed implementation includes a successful permitted operation, denied excess access, observed rotation, tested revocation, and understood outage behavior. The existence of a Vault policy is only the configuration portion of that evidence.

Offboarding: disabling the directory account is one step

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

An employee’s directory account can be disabled while an application session, SSH key, or API token remains usable. Offboarding is an access-removal workflow across systems, not a single identity-console action.

Inventory the access forms

Record the person’s interactive accounts, privileged identities, application roles, SSH credentials, API tokens, and delegated access. Include services they own so that access removal does not silently stop a business process.

Do not record secret values in the inventory. Store credential identifiers, owning systems, scopes, and rotation procedures.

Separate new access from existing access

Microsoft documents that Entra token and application-session behavior can produce a delay between revocation and effective loss of access. Application-issued sessions may require application-side handling. See Microsoft’s access-revocation guide.

Use a test identity to establish what happens in your application estate:

Access form	Validation
New SSO login	Denied after the intended control takes effect
Existing browser session	Ends within the documented interval/workflow
API token	Revoked or replaced at its issuing system
SSH key or certificate	No longer authorizes a new session
Active server session	Handled by the approved session procedure

A failed new login says nothing about a browser window already open elsewhere.

Handle SSH independently

Inspect central key or certificate issuance and local authorized-key paths. OpenSSH supports multiple authorization mechanisms, documented in sshd_config. Removing one visible key file is incomplete if another source still grants access.

A terminated employment relationship is not a reason to delete application data indiscriminately. Preserve records and transfer ownership according to the organization’s retention and business procedures. Keep operational ownership separate from the individual’s access rights.

Rehearse a full departure

Create a synthetic user with representative access and an active application session. Run the offboarding checklist in a test scope. Record each system’s completion time and the actual result of attempting access afterward.

Include a long-lived credential issued outside SSO. This catches a common design gap: the central identity system does its job while an independent service continues to trust an older credential.

Make exceptions visible

Some departures require temporary continuity for scheduled tasks or shared business records. Replace personal ownership with an appropriate service identity or named successor. An exception should specify its purpose, expiry, and approving owner; leaving a personal account active indefinitely is not a migration strategy.

Acceptance evidence

Keep the systems covered, access attempts, revocation timestamps, unresolved dependencies, and ownership transfers. Report incomplete items explicitly. “Account disabled” is a valid fact, but it is not a defensible summary of the entire access state.

Review service-account lifecycle to reduce the number of workflows tied to personal credentials before the next departure.

Privileged SSH: make the bastion a real boundary

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Putting a bastion in a network diagram does not force administrators to use it. If managed hosts still accept direct connections from broad networks or share long-lived keys, the new hop may add logging while leaving the original access path intact.

Define the authorized route

Document which administrator devices can reach the bastion, which hosts it can reach, and which identities are accepted downstream. Separate ordinary user access from privileged maintenance. Keep a tested emergency route for recovery, with explicit ownership and audit.

The boundary includes network policy and host authentication. Neither one substitutes for the other.

Inspect effective OpenSSH settings

OpenSSH documents configuration processing, Match rules, forwarding controls, and authorized principals in sshd_config. On a Linux lab server with an installed OpenSSH daemon:

sudo sshd -t
sudo sshd -T

The first checks configuration validity; the second displays effective settings for the default context. Where Match blocks are used, test the intended connection context with the version-supported -C options as well. Keep the output restricted because it reveals security configuration.

Do not assume that the last line in a configuration file overrides earlier values. Includes and first-obtained-value behavior make visual inspection alone unreliable.

Test the access matrix

Use a pilot host and test four cases: approved administrator via bastion, unrelated user via bastion, direct connection from an ordinary client network, and the emergency path. Define the expected result for each.

Test both authentication and privilege elevation. An account that can log in but cannot perform the approved task is an operational failure; an account that can become root everywhere may be broader than intended.

If certificates are used, test expired and unauthorized principals with disposable credentials. Certificate issuance policy and signer protection become part of the access boundary.

Review forwarding and sessions

Agent, TCP, and other forwarding features can extend authority beyond the apparent login route. Disable unnecessary features using the supported controls and test the workflows that remain. OpenSSH also notes that forwarding restrictions are not complete confinement for a user who can execute arbitrary code on the host.

Record session start, identity, destination, and privileged actions according to the organization’s logging policy. Do not promise full command attribution merely because SSH authentication logs exist.

Roll out without losing recovery

Keep a working rescue session and independently verified console access during the pilot. Apply changes to a small group, establish a new session through the intended route, and verify both allowed and denied behavior before expanding.

The result should be a route enforced by network and host policy, with bounded downstream authority and a working recovery path. See management-network isolation for the infrastructure beneath that route.

Microsoft Entra admin access: require the authentication method you intend

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

An administrator can satisfy a broad MFA policy with a method that does not meet the organization’s intended resistance to phishing. The policy needs to specify the required assurance, and the rollout needs to prove both allowed access and denied fallback.

Inspect authentication strength

Microsoft Entra Conditional Access provides authentication strengths that constrain acceptable method combinations. Microsoft documents a policy for administrator roles using phishing-resistant MFA. See the administrator policy guide and how strengths are evaluated.

Confirm current licensing, supported methods, tenant configuration, and dependencies before rollout. Do not assume a policy named “strong MFA” actually requires the intended methods.

Prepare registration and recovery

Select a pilot group and ensure users have registered the approved authenticators before enforcement. Test an alternative recovery route that does not recreate the same failure dependency. Losing a device should not leave the only administrator unable to operate the tenant.

Microsoft’s emergency access guidance is the starting point for protecting and monitoring emergency accounts. Follow current requirements rather than copying an old blanket MFA exclusion.

Test a policy matrix

Scenario	Expected result
Pilot administrator with approved method	Access succeeds
Pilot administrator using a disallowed method	Required step-up or denial
Unregistered pilot administrator	Documented enrollment/recovery outcome
Emergency procedure	Authorized access and visible audit evidence
Ordinary user outside scope	Intended existing behavior

Run report-only analysis where supported, then a controlled enforced pilot. Report-only results cannot establish that an application actually accepts the final login flow.

Review exclusions as carefully as inclusions

List excluded users, groups, workloads, and applications. Give each exception an owner and review date. A broad exclusion for automation may accidentally cover interactive administrators if account types and group membership are poorly controlled.

Check the interaction with other Conditional Access policies. The effective result comes from the policy set, not the one screen currently being reviewed. Inspect sign-in details for the test session to confirm which controls applied.

Stronger authentication does not remove excessive privileges, insecure administrator workstations, or stolen active sessions. Keep privileged work separate from ordinary browsing and email, minimize standing authority, and document session-revocation procedures.

A valid acceptance report includes policy scope, exclusions, allowed and denied test outcomes, sign-in evidence, and the tested recovery path. Avoid presenting enrollment counts as equivalent to enforced protection.

Next, examine privileged SSH access and offboarding across sessions and tokens.

Missing security telemetry: monitor the silence as well as the alerts

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A collector answering health checks can still receive no useful events. A source removed from service discovery may disappear from every dashboard. Security monitoring needs a separate definition of which sources should exist and how recently each should have produced evidence.

Distinguish three failure states

A known target can be down. A known target can be reachable but stale. A target can be missing from discovery entirely. One uptime graph does not cover all three.

Use an inventory of required sensors and log producers. Include ownership, expected event cadence, and maintenance rules. Compare observed sources against that inventory rather than deriving the entire expectation from whatever happened to report today.

Express freshness explicitly

Prometheus documents absent_over_time and time functions in its query reference. For an exporter that you implement to publish the most recent event time, illustrative expressions are:

(time() - security_last_event_timestamp_seconds{source="storage-audit"}) > 900

absent_over_time(security_last_event_timestamp_seconds{source="storage-audit"}[15m])

security_last_event_timestamp_seconds is an example metric, not a built-in metric supplied by a storage vendor or Prometheus. The first expression detects old values; the second detects no samples for the explicitly named source. Neither automatically enumerates every missing source in a changing fleet.

The 15-minute window is illustrative. Select the threshold from the system’s real event behavior and response requirement. A genuinely idle source needs a heartbeat or controlled validation event rather than an assumption that user activity is continuous.

Test the alert chain

In a lab, stop the event producer while leaving the exporter running. Confirm the stale-event alert. Then stop the exporter and confirm missing-sample or target-down behavior. Finally remove the source from discovery and verify that inventory reconciliation still identifies it.

These tests isolate different failure modes. If the third condition becomes invisible, the monitor still has an absence problem.

Carry enough context to act

An alert should name the producer, last event time, last successful collection, owning team, and a short diagnostic procedure. A generic “no data” notification encourages responders to guess whether the event source, network, parser, or query failed.

Preserve maintenance windows with expiry. A permanent silence created for a temporary outage can make the monitoring system look healthy for months.

Check timestamps and delivery

Clock skew can make a producer appear fresh or stale incorrectly. Compare event time with ingest time and monitor clock synchronization independently. Confirm that alert routing reaches a staffed destination using an approved test notification workflow.

The completed control is not a query that evaluates successfully. It is an expected source inventory, a freshness signal, and a demonstrated response when each part stops working. Apply the same approach to storage audit pipelines.

Detection engineering: keep a test that must match and one that must not

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A detection rule has two obligations: identify the behavior it was written for and leave expected activity alone. Testing only the first produces false positives; testing only the second can remove the detection entirely.

Write the detection contract

For each rule, record the behavior, required telemetry, expected event fields, and the response owner. Describe the scope narrowly enough to test. “Detect suspicious traffic” is not a specification. “Alert on this synthetic request pattern in this inspected HTTP path” is.

Keep one approved matching fixture, one ordinary nonmatching fixture, and any fixture added after a false positive. Use synthetic captures or appropriately sanitized authorized data. Store hashes and provenance so later reviewers know exactly what was exercised.

Use the engine’s parser

Suricata and Snort have documented configuration tests and offline capture modes. See Suricata’s CLI reference and Snort’s traffic inspection guide. Validate the complete configuration used for the test, not just a detached rule string.

The test record should contain:

rule_id: locally assigned identifier
engine_version: exact installed version
configuration_hash: recorded SHA-256
fixture_hash: recorded SHA-256
expected: alert present or alert absent
observed: matching identifiers and count

This is a schema example, not measured output. Record the actual values from the run.

Assert the right event

Do not accept “one or more alerts” as proof. Assert the intended rule identifier and, when relevant, the source, destination, or application field. A completely unrelated signature can otherwise make a broken test look successful.

For suppression and threshold rules, define the expected count or range explicitly. Replay timing matters: an accelerated fixture may not behave like the original traffic window. Record the timing method and engine options.

Separate detection from delivery

An offline fixture proves engine behavior. A live benign event proves the collection and routing path. A ticket or notification received by the intended responder proves delivery. Maintain all three where the control depends on all three.

If the product is deployed inline, add an observed client transaction result. A drop action in a file does not prove the forwarding path enforced it.

Make exceptions testable

When narrowing a rule to fix a false positive, add that legitimate example to the fixture library before changing the rule. Rerun both the old match and the new nonmatch. An exception should identify its owner, scope, and review date rather than becoming an undocumented global bypass.

Keep fixture size modest and access controlled. Packet captures can contain credentials and personal data even when the original investigation seemed routine.

What to release

Release the rule, its tests, configuration changes, and results together. Keep the previous working version available for rollback. This turns a ruleset update into an engineering change with known behavior rather than a hopeful increase in the number of enabled signatures.

Encrypted traffic: decide what each security sensor can actually establish

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Encryption changes the evidence available to a network sensor. It does not make all monitoring useless, and it does not justify pretending that a packet inspection engine sees application content it cannot decrypt.

Build a visibility table

For each important service, record where encryption starts and ends, which network paths are observed, and which endpoint or application logs are available.

Evidence source	Useful question	Important limit
Network flow	Which endpoints communicated?	Usually no application payload
TLS metadata	What handshake details were observed?	Fields vary by protocol and privacy features
Endpoint telemetry	Which process opened the connection?	Depends on agent coverage and trust
Application audit	Which authenticated operation occurred?	Depends on application logging

Zeek’s TLS log documentation describes protocol metadata that may be available. Do not assume every connection exposes a hostname or certificate in the same way.

Test two observation points

Use a lab application with a known request and a unique harmless marker. Observe the client-to-service encrypted path, then inspect authorized server-side application logs. Identify which source can establish the marker and which can establish only the connection.

If there is an approved TLS termination proxy, document both sides. A sensor after termination may see plaintext traffic for that segment while missing traffic that takes a different route. The presence of a proxy does not establish universal inspection.

Suricata’s HTTP keyword documentation describes application buffers used by HTTP rules. A payload rule requiring those buffers cannot be assumed to match opaque encrypted contents.

Evaluate decryption as an architecture change

TLS inspection introduces certificate trust, private-key protection, application compatibility, and privacy obligations. Establish which traffic classes may be inspected and how the inspection infrastructure is administered. Test certificate-pinned applications and machine-to-machine clients before a rollout.

Do not make “decrypt everything” a substitute for identifying business requirements. Some flows may be better covered by endpoint detection and application audit, especially where interception changes the trust model or breaks the workload.

Correlate without overstating certainty

Join network and endpoint records using time, addresses, ports, and host identity. Account for NAT and address reuse. A shared egress address can represent many users; it is not a user identifier.

In an analyst exercise, write each conclusion beside its evidence source. “Host A connected to service B” may be established. “User C downloaded file D” may require application audit. Keeping that distinction visible prevents a plausible story from becoming an unsupported incident finding.

Repeat the fixture when routing, TLS termination, or endpoint tooling changes. Retain a list of unobserved paths and assign owners. A security architecture with documented limits is more actionable than a dashboard claiming complete coverage without a reproducible test.

Wazuh file integrity monitoring: start with files someone will investigate

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Monitoring every changing file on a busy server produces activity, but not necessarily useful evidence. A file integrity monitoring deployment should begin with a small set of paths whose unexpected modification has a clear owner and consequence.

Choose a meaningful baseline

Start with authentication configuration, service definitions, privileged automation, and selected application configuration. Exclude noisy runtime data only after understanding what would be lost. A blanket exclusion for an entire application tree can remove the very files an attacker would alter.

Wazuh documents scheduled and real-time integrity monitoring and platform-dependent capabilities in its FIM guide. Confirm which mode the target operating system and configured paths support.

Use a harmless configuration fixture

Before applying a broad production policy, create a dedicated directory containing a synthetic text file. Add only that directory to the lab agent’s configured monitoring scope using the documented configuration method.

Record the baseline, then make three changes separately: edit the contents, change permissions, and remove the test file. Wait for the configured detection mode and interval after each operation.

Test	Evidence expected
Content edit	Correct path and changed integrity information
Permission change	Relevant attribute change, if enabled and supported
Removal	Deletion event with the same file identity/path context
Agent offline	Missing-agent or freshness signal

Do not claim attribution unless the configured platform feature actually supplies it. A file change event can establish that a change happened without establishing the process or person responsible.

Follow the alert to an owner

Inspect the event on the manager and in the analyst view. Confirm hostname, path, timestamp, and the distinction between the original event and ingestion time. A correctly collected event assigned to no one is not an operational detection.

Write a response note for each monitored path family. An unexpected SSH configuration change may require reviewing a deployment record and testing effective settings. An application template change may belong to the application team. Different changes should not all trigger the same emergency response.

Handle legitimate deployments

Associate maintenance windows with an approved change, but preserve the events. Suppressing all file integrity activity during deployment can conceal an unrelated change. Prefer scoped interpretation and review over discarding the evidence.

After rebuilding an agent or reinstalling a host, confirm that the new baseline is intentional. A newly initialized baseline is not proof that the host matches a trusted image.

Control volume before expanding

Measure events per host, storage growth, and analyst effort on a representative subset. Expand only when expected events arrive and ordinary maintenance remains understandable.

Finish by repeating the harmless fixture after a policy change or manager upgrade. Pair FIM with missing telemetry monitoring so a quiet dashboard cannot conceal a disconnected source.

Snort 3 rule validation: a loaded rule is not a tested detection

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A rule can parse successfully and never match the traffic it was meant to detect. Another can match so broadly that moving it into an inline policy interrupts legitimate work. Snort 3 changes need evidence at both boundaries.

Pin the inputs

Keep the Snort version, Lua configuration, rule file, fixture PCAP, and expected signature identifiers in the change record. Pin the fixture by checksum. A result without those inputs is difficult to reproduce after a ruleset update.

The official guide separates configuration, reading traffic, and rule actions. Use the options for the installed Snort 3 package; Snort 2 configuration examples are not interchangeable.

Validate syntax and behavior separately

For a lab installation with configuration at the example path:

snort -T -c /etc/snort/snort.lua
snort -c /etc/snort/snort.lua -r ./approved-match.pcap -A alert_fast
snort -c /etc/snort/snort.lua -r ./approved-nonmatch.pcap -A alert_fast

Supply your own approved synthetic fixtures and confirm that the relevant rule file is included by the configuration. The commands do not create the fixtures. Output location and available logging modules should be verified against the installation.

The first run should load successfully. The matching fixture should produce the intended rule identifier. The nonmatching fixture should not produce that identifier. Other unrelated alerts need separate interpretation rather than being counted as a test failure automatically.

Test the rule’s assumptions

A fixture should represent the actual protocol and normalization the rule expects. A string in a raw packet is not necessarily the same as a value in an HTTP inspector’s buffer. Include benign variations that previously caused false positives, such as a similar URI or legitimate application payload.

Treat negative fixtures as part of the rule’s specification. They explain what the rule must leave alone and protect that behavior when the signature is tightened later.

Prove enforcement on the final path

Offline alert output does not prove a client transaction was blocked. For an IPS policy, use a controlled live path, a harmless match, and an explicit expected client result. Check the configured action and capture mode together.

Do not infer physical fail-open or fail-closed behavior from a rule action. Those depend on the deployed forwarding and failure design. Rehearse rollback and ordinary application traffic as described in the inline rollout article; the engineering tests apply even though the configuration syntax differs.

Keep the result small and auditable

An acceptance record can be a table of fixture hashes, expected signature IDs, observed IDs, and pass/fail outcomes. Add client evidence for enforcement tests. This is much stronger than “Snort started” and cheap enough to repeat with every rule change.

Zeek: turn a connection into an investigation trail

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A network alert becomes more useful when an analyst can establish what happened immediately before and after it. Zeek supplies protocol and connection context that can support that investigation, even when a signature alone gives little explanation.

Begin with a known capture

On a host with Zeek installed, process an approved synthetic PCAP from a clean output directory:

mkdir -p ./zeek-lab-output
cd ./zeek-lab-output
zeek -r ../approved-test.pcap LogAscii::use_json=T

This writes local logs. It does not send traffic or validate the location of a production sensor. Zeek documents the workflow in its quick start.

Choose a fixture whose endpoints and expected protocols are known. A large unexplained capture is a poor first test because it makes missing evidence difficult to recognize.

Follow one connection

Inspect a compact view of JSON connection records:

jq -c '{uid,origin:."id.orig_h",destination:."id.resp_h",
  port:."id.resp_p",proto,service,duration}' conn.log

The connection-log reference explains the fields and the connection UID. Use the UID to locate related protocol records where that field is present. Preserve sensor identity and time context when combining records from multiple systems.

Ask a specific question: did this host query a name and then connect to an unexpected service? The DNS answer and the subsequent destination can support a hypothesis. They do not prove that the same process or user caused both actions without additional endpoint evidence.

Know what encryption removes

TLS can leave useful connection and handshake metadata while hiding application contents. Visibility varies with protocol and configuration. A missing HTTP log for an encrypted connection is not, by itself, a sensor failure.

Likewise, traffic that never traverses the observation point cannot appear in the logs. Capture loss, asymmetric routing, and mirror oversubscription can create partial records. Investigate those conditions before interpreting absence as proof that communication did not happen.

Build a repeatable analyst exercise

Prepare a small scenario containing a DNS lookup, an allowed web request, and a connection to a lab-only destination. Ask a second operator to reconstruct the sequence using the logs and the documented queries. Record which claims can be established and which require endpoint or identity records.

Then repeat on the live monitored path using approved harmless traffic. Verify that the logs reach the analyst’s system within the agreed window and remain queryable after rotation.

Avoid turning metadata into certainty

A large transfer is not automatically exfiltration. A new destination is not automatically malicious. Use inventory, business purpose, endpoint process information, and access records to test the hypothesis.

A useful deployment delivers dependable context and explicit blind spots. See encrypted traffic visibility for the complementary sensor-placement review.

Suricata IPS: turn detection into blocking without guessing about failure

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Changing a signature from alerting to dropping is only one part of an IPS rollout. Traffic must traverse the enforcement path, legitimate work must continue, and the organization must understand what happens when the sensor or capture process fails.

State the enforcement design

Suricata documents inline operation at Layer 2 and Layer 3. Its IPS concept uses drop or reject actions for unwanted traffic. AF_PACKET inline deployments require an explicitly paired interface design. See IPS concepts and Linux inline setup.

These links describe Suricata 8.0.1; use the equivalent pages for the installed supported release before configuring production. The design tests below do not depend on a particular interface name or release default.

Establish the failure policy before installation

Decide whether the protected service should lose connectivity or pass traffic without inspection when the enforcement path fails. Hardware bypass, queue behavior, routing, and process failure can produce different outcomes. “Fail open” in a slide deck is not an observed result.

Document a bypass procedure and the person authorized to invoke it. It should be reachable through a management path independent of the traffic being inspected.

Use three traffic classes

Class	Expected outcome
Ordinary business transaction	Allowed, within agreed latency
Benign fixture matching the test drop rule	Blocked and logged
Similar fixture that should not match	Allowed

Run these in an isolated lab first. Use a harmless synthetic signature rather than malware. Confirm the blocked transaction fails at the client and is recorded by the sensor. An alert alone does not prove enforcement.

Record the exact rule set and configuration. Start production rollout with a small, justified set of blocking rules after observing them in alerting mode. Retain an explicit rollback version rather than editing rules under pressure.

Exercise operational failures

In the lab, stop the process, remove one test link, and simulate loss of management access one at a time. Inspect client behavior for each condition. Repeat with a long-lived connection as well as new connections; the outcomes may differ.

Under representative traffic, measure packet drops, application error rates, and latency percentiles. An inline system with acceptable average latency can still cause intermittent application timeouts.

Keep exceptions narrow and expiring

A false positive should produce a scoped exception with the affected service, rule identifier, owner, and review date. Disabling an entire ruleset because one application fails makes the enforcement policy hard to understand.

Before declaring the deployment accepted, rerun the three traffic classes through the final path and demonstrate rollback. Preserve the results with the change record. Pair this with detection regression tests so future rule updates do not silently alter the policy.

Suricata IDS: prove the sensor can see the traffic you care about

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A Suricata service can be running with a current ruleset and still inspect the wrong traffic. The first acceptance test is visibility: the intended packets reach the sensor, the expected rule fires, and the event reaches the analyst.

Define the observation point

Draw the path from a representative client to the protected service. Mark switches, virtual switches, firewalls, and encryption boundaries. A SPAN port observing an internet uplink may miss traffic between two servers on the same VLAN. A mirror that carries only one direction can impair stream interpretation.

Record the capture interface, expected networks, link capacity, and whether traffic can bypass the observation point. This is the sensor’s coverage statement. Avoid describing it as coverage of the entire enterprise.

Validate configuration and a known capture

On a lab host with Suricata 7.x and its matching configuration, use the documented test and offline modes:

suricata -T -c /etc/suricata/suricata.yaml
mkdir -p ./suricata-lab-output
suricata -r ./approved-test.pcap -c /etc/suricata/suricata.yaml \
  -l ./suricata-lab-output

The PCAP is an approved fixture you supply; it is not downloaded or generated by these commands. It should contain synthetic traffic matching a known enabled test signature. Consult the command-line reference for your package’s options and permissions.

A configuration test establishes syntax and loading, not detection correctness. The offline run establishes behavior on a fixture, not live capture visibility.

Inspect the event, then the live path

With EVE alerts enabled in the configuration:

jq -c 'select(.event_type == "alert") |
  {timestamp,src_ip,dest_ip,signature:.alert.signature}' \
  ./suricata-lab-output/eve.json

Suricata’s EVE documentation describes the event output. Confirm the expected signature identifier and endpoints rather than accepting any alert as success.

Next generate the approved benign test on the actual monitored path. Find its event locally and in the SIEM. Record the delay and the rule revision. If it appears offline but not live, investigate placement, capture filtering, and packet delivery before changing the detection rule.

Measure loss under ordinary load

Watch capture and decoder statistics during representative busy periods. Compare the switch mirror’s capacity with the traffic it aggregates. A mirror receiving two directions from several links can be oversubscribed even when the sensor’s nominal interface speed looks adequate.

Do not publish a throughput guarantee from a small fixture. Keep the configuration, rule count, traffic mix, CPU allocation, and observed drop statistics together.

Acceptance evidence

Retain configuration-test output, fixture identity, expected alert, live-path alert, SIEM arrival, and coverage limitations. That establishes a functioning IDS path. Blocking requires a separate IPS rollout.

Storage auditing: prove an event reaches an investigator

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A storage platform can record an event locally while the security team sees nothing. Events may be excluded, delayed, dropped by a collector, parsed incorrectly, or retained for less time than investigators expect. Test the path end to end with a known operation.

Define the event contract

Choose the questions the audit trail must answer: who accessed a sensitive path, which client was involved, what operation occurred, whether it succeeded, and when. Identify which of those fields the deployed platform and protocol actually emit.

For VAST, PowerScale/Isilon, and WEKA, verify release-specific audit support and configuration with the vendor. Do not assume the same event coverage across NFS, SMB, S3, and management APIs. File data access and administrative changes are separate event families.

Create a controlled evidence marker

Use an approved synthetic filename such as audit-validation-2026-09-09.txt in a test directory. Perform a read, a write, and a denied operation using distinct test identities. Record local UTC times and the client address without inserting confidential data into filenames.

Follow each expected event through the pipeline:

Stage	Check
Storage producer	Operation is in the configured audit scope
Export or collector	Event leaves the source without an unexplained gap
Transport	Authentication and delivery behave as intended
SIEM	Fields remain searchable and correctly typed
Analyst workflow	Saved query finds the test within the agreed delay

A file modification does not necessarily produce every event type in this example. Set expectations from the product’s documented event model first.

Keep clocks and identifiers usable

Preserve the event’s original timestamp and the ingest timestamp. Their difference helps distinguish a delayed producer from a delayed collector. Retain the source identity, cluster identifier, and protocol alongside normalized fields.

Network telemetry can provide corroboration. Zeek’s connection log records connection context, but it is not a substitute for a storage authorization record. Encrypted or unobserved traffic further limits what a network sensor can establish.

Test the pipeline’s absence signal

Schedule a harmless recurring validation event where practical and monitor its freshness. A reachable collector with no recent storage events is not necessarily healthy. Prometheus documents absent_over_time for detecting missing series; an inventory-based check is also needed for sources that disappear from discovery entirely.

Choose a delay threshold from expected traffic and operating hours. A quiet archive should not generate an alert merely because no user accessed it overnight.

Protect the evidence itself

Restrict access to raw logs, define retention, and record who may alter collector configuration. Audit logs can expose paths, usernames, and business activity. Forward them to a boundary that a routine storage administrator cannot silently erase.

Keep the test event IDs and the investigator query with the deployment record. Repeat after storage upgrades, parser changes, and collector migrations.

NFS security: root squashing does not authenticate every user

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

An export restricted to a subnet can still trust more client authority than its owner intended. Before tightening filesystem permissions, establish how the server decides which user made each request.

Distinguish the security flavors

The Linux NFS export manual documents sec=sys and Kerberos flavors. krb5 provides Kerberos authentication, krb5i adds integrity, and krb5p adds privacy. It also documents root squashing, which maps client root requests to an anonymous identity rather than treating them as server root. See exports(5).

Root squashing does not turn client-supplied AUTH_SYS user IDs into independently authenticated identities. That matters when a client is shared, unmanaged, or compromised. Evaluate the trustworthiness of the client host as part of the export’s security boundary.

Inspect what the client actually mounted

On a Linux NFS client:

findmnt -t nfs,nfs4
nfsstat -m
id
klist

The last command requires Kerberos client tools and reports the current ticket context. Its absence does not diagnose the server. Compare negotiated mount options with server export settings and the application’s actual execution identity.

Do not assume that an entry in a proposed mount configuration describes an already established mount. Inspect the running system.

Test identities, not just connectivity

Use a dedicated export with synthetic files. Test an authorized user, an unrelated user, and a client root process. Define which read, create, rename, and permission-changing operations should succeed before running the test.

If Kerberos is required, include a fresh session without a usable ticket and a session with the intended credentials. Long-running jobs need a documented credential-lifetime strategy. A successful interactive mount does not prove a batch job will continue working several hours later.

Evaluate directory permissions as well as file permissions. A user may be unable to alter file contents but still be allowed to rename or remove a directory entry.

Treat migration as an interoperability test

VAST, PowerScale, WEKA, and Linux NFS servers do not expose identical configuration interfaces. Check the supported security flavors, release constraints, identity integration, and client requirements for each platform. Do not paste Linux /etc/exports syntax into an appliance runbook.

Kerberos introduces dependencies on time, name resolution, principals, and key distribution. Benchmark the application’s real workload with the required security settings rather than disabling privacy to obtain a more attractive throughput number.

Acceptance criteria

Retain the negotiated security flavor, identity used, expected-versus-observed operation matrix, and long-running job result. Document residual client trust explicitly when AUTH_SYS remains necessary. Network segmentation and managed clients can reduce exposure, but they should not be described as cryptographic user authentication.

Replication and failback: choose a clean recovery point before reversing direction

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Replication can faithfully transfer an unwanted change. That is useful for availability and dangerous when an operator mistakes the remote copy for an independently protected history. The recovery plan must say which failure it covers.

Classify the event before promotion

A failed primary site, a deleted directory, and a compromised administrator require different decisions. Site failure may justify promoting the latest consistent replica. Corruption may require an older recovery point. Compromise also requires deciding whether identities, configuration, and the destination remain trustworthy.

Write the intended decision path before the incident:

Event	First recovery question
Hardware or site outage	Is the latest replica complete and usable?
Accidental change	Which earlier point contains the required data?
Suspected compromise	Which point and administrative boundary can be trusted?
Return to primary	Which site now owns authoritative writes?

These are operating decisions. The replication engine cannot infer them from a successful transfer.

Make authority explicit

A promotion procedure should identify who can stop writes, declare the source fenced, select the recovery point, and open the destination to applications. Include the behavior of clients with stale connections and old DNS answers.

For PowerScale SyncIQ with SmartLock, use Dell’s documented source/target and failback constraints. The OneFS replication limitations demonstrate why a generic “reverse replication” step is inadequate. Other platforms require their own supported workflow.

Rehearse the full round trip

In an isolated environment, create a baseline dataset, replicate it, and record checksums. Simulate source unavailability without deleting the source. Promote through the supported procedure, make a controlled change at the recovery site, and confirm that only the intended site accepts writes.

Now rehearse the return. Preserve the destination’s new data, re-establish the intended relationship, and confirm which copy wins. A test ending immediately after promotion misses the part that can overwrite valid recovery-site work.

Measure application consequences

Time the last accepted source write, recovery-point availability, promotion, application readiness, and return to steady operation. Record any accepted data loss against the service’s agreed recovery-point objective. Measure user-visible downtime against the recovery-time objective.

Use the same credentials and network routes expected during an actual outage, except where the lab intentionally substitutes test identities. If a privileged engineer manually repaired five undocumented dependencies, those are findings, not a passing automated recovery.

Keep history outside the propagation path

Maintain a recovery strategy that can survive harmful changes being replicated. That may include protected historical copies and separate administrative authority, selected for the product and threat model. Document the retention window and the maximum time the business expects to take to detect corruption.

See administrator-compromise recovery for the authority review that complements the data path.

S3 Object Lock: test versions, not just object names

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

An object disappearing from a normal listing does not prove its protected contents were destroyed. Conversely, enabling a bucket feature does not prove that every object version has the intended retention. S3 recovery tests need version-aware evidence.

Understand the unit of protection

Amazon S3 Object Lock applies to object versions. AWS documents governance and compliance retention modes, along with legal holds. A new version or a delete marker can exist above a protected version. Governance bypass requires specific authority and a bypass request; compliance retention has a different boundary. See AWS Object Lock documentation.

These are Amazon S3 semantics. An S3-compatible product needs its own compatibility review. Do not assume identical retention behavior because both systems accept an S3 client.

Inspect a known test object

With AWS CLI installed and a read-only role authorized for a dedicated lab bucket, replace the example names and version identifier:

aws s3api get-object-lock-configuration --bucket example-recovery-lab
aws s3api list-object-versions --bucket example-recovery-lab --prefix test.txt
aws s3api get-object-retention --bucket example-recovery-lab \
  --key test.txt --version-id EXAMPLE_VERSION_ID
aws s3api get-object-legal-hold --bucket example-recovery-lab \
  --key test.txt --version-id EXAMPLE_VERSION_ID

Capture the exact version ID. An access-denied response means the inspection is incomplete; it does not mean retention is absent. Likewise, a bucket default is not a substitute for examining the version you intend to recover.

Design positive and negative tests

Upload synthetic data under an approved short test retention. Confirm that the protected version cannot be permanently removed by the ordinary application role. Test any governance-bypass role separately, with a disposable object and an explicit expected result.

Then verify that the recovery role can retrieve the intended historical version even when it is no longer the current one. Compare its content against a local test checksum. Do not perform these deletion exercises against actual backup objects.

Preserve the other dependencies

Object retention does not preserve an application’s encryption keys, restore catalog, account access, or business knowledge about which point is clean. Keep those in the recovery design. A backup system should also document its required permissions and supported bucket configuration before retention is enforced.

Monitor failed writes, cleanup failures, and capacity growth after rollout. Protection that interferes with a backup product’s lifecycle can create a different outage while appearing to improve security.

Evidence that answers the question

The useful report contains the bucket configuration, version identifier, retention state, tested identities, denied destructive operation, and successful version-specific retrieval. “Object Lock enabled” is only one line in that report.

Compare this model with PowerScale SmartLock; the two are related design problems, not interchangeable implementations.

WEKA encryption: include the key service in the recovery plan

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

Encryption creates a dependency that storage capacity planning often misses: the availability and integrity of the key-management service. A healthy storage cluster can still be unusable if the recovery environment cannot authenticate to the KMS or obtain the required key material.

Establish the release and encryption state

The WEKA 4.2 documentation describes an external KMS for encrypted filesystems and documents the read-only weka security kms command. It identifies Vault and KMIP integrations; support details must be checked against the deployed release. See security management and the API/CLI reference.

On an authorized management host with the matching WEKA CLI, inspect:

weka version
weka fs
weka security kms

These are inventory commands, not a recovery test. Treat their output as sensitive configuration information and store it accordingly.

Draw the boot and recovery order

Record how the KMS itself starts after a site outage. Identify its storage, DNS, certificates, authentication backend, and recovery custodians. If those services depend entirely on the encrypted filesystem, there is a circular dependency to resolve.

A recovery worksheet should include the filesystem, KMS endpoint, key identifier, trusted CA, authentication method, supported recovery procedure, and owner. It should not contain private keys, tokens, or unseal material.

Decide which copies of configuration can be held in an emergency package and which secrets require separate custodians. An offline document that says “log into the vault” is incomplete if the vault is what failed.

Use a lab to test key-service loss

Create a disposable encrypted filesystem with a test key and representative data. Establish a baseline read and recovery operation. Under the vendor’s supported test procedure, simulate KMS unavailability in the lab and observe existing access, new mounts, and recovery behavior separately.

Do not infer one behavior from another. Cached key material and lifecycle operations can create different dependencies. Do not delete or disable production keys to see what happens.

Restore the lab KMS path and repeat the original operation. Record any manual steps, certificate failures, authentication changes, and elapsed time. The result should identify which operations require the KMS, not simply whether the cluster stayed up.

Separate key rotation from data movement

Key rewrapping, key replacement, and bulk data re-encryption are not interchangeable terms. Use the release-specific workflow and inspect the resulting configuration. A successful rotation task should be followed by access and recovery tests using the intended recovery environment.

The acceptance criterion is straightforward: protected data remains confidential, authorized workloads can use it, and the documented recovery path works when the normal site is unavailable. See Snap-To-Object recovery for the associated data-side exercise.

WEKA Snap-To-Object: prove recovery without the source cluster

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A WEKA snapshot visible on the source cluster is not the same as a recoverable copy outside that cluster. Snap-To-Object is useful precisely because the recovery design can separate those dependencies, but the separation has to be demonstrated.

Identify the recovery artifacts

WEKA documents Snap-To-Object as storing snapshot data and filesystem metadata in an object store, with a reference locator used for recovery. Its documentation specifies recovery to the same or a higher WEKA version and describes interactions with tiering. See Snap-To-Object.

Record the snapshot identity, successful upload state, locator, destination bucket, source release, and intended recovery release. Keep recovery credentials and encryption material in their approved secret-management systems, not in the inventory document.

The snapshot documentation distinguishes local history from external backup. Use that distinction in monitoring labels so operators do not mistake snapshot creation for off-cluster completion.

Rehearse with an independent destination

Prepare a small filesystem containing several large files, many small files, nested directories, and representative permission settings. Generate a manifest of expected paths and checksums before taking the snapshot. Use synthetic data if the recovery environment has a different security boundary.

Upload through the supported workflow and confirm completion. Recover using the documented locator on a separate test cluster with an explicitly compatible release. The exercise should not need to query the original cluster for an undocumented missing value.

Check the recovered namespace, read files throughout the tree, and compare checksums and ownership. Then run an application-level test. Being able to list filenames does not prove the backing data is available or that a workload can use it.

Include object-store failure modes

Dependency	Test question
Bucket access	Does the recovery identity have exactly the required access?
Encryption	Can the recovery environment obtain the necessary keys?
Locator	Is it retained independently of the failed cluster?
Lifecycle	Could object expiration remove required data?
Network	Is recovery throughput adequate across the actual route?

Do not add object-lock or lifecycle settings to an active WEKA bucket by analogy with a generic backup product. Confirm compatibility with the WEKA release and workflow, including cleanup and tiering behavior, before making retention changes.

Measure usable recovery

Track when the filesystem becomes available and when representative reads finish. Data or metadata fetched on demand can make initial availability look much better than workload recovery. Measure the first full working-set access separately from a later warm-cache run.

Retain the completed-upload evidence, recovery release, manifest comparison, and application results. That bundle establishes a recovery capability. A dashboard showing that uploads are scheduled establishes only intent.

Continue with WEKA and KMS recovery.

PowerScale multiprotocol access: prove who the same user becomes

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A user can appear correctly restricted from a Windows workstation and still gain unexpected access through a Linux client. The files did not change; the identity and authorization path did. This is a central security test for a PowerScale estate serving both SMB and NFS.

Map identity before editing permissions

OneFS supports directory services, access zones, and identity mapping across protocol identities. Dell describes combining identities into an access token and configuring mappings within individual zones. See identity management and user mapping.

For one affected file, record the client protocol, zone, authentication provider, user SID or UID, groups, and effective file permissions. Do not start with a recursive permission change. That destroys evidence and can widen access across a much larger tree.

Construct a small access matrix

Use a test share and export containing synthetic data. Choose an owner, a same-team reader, and an unrelated user. Exercise each from both client types.

Operation	Owner	Reader	Unrelated user
Read existing file	Allow	Allow	Deny
Create new file	Allow	Deny	Deny
Rename or delete	Allow	Deny	Deny
Change permissions	Explicit policy	Deny	Deny

This is an example policy, not a default OneFS behavior. Adapt it to the application. Test the directory as well as the file, because creation, rename, and deletion involve directory authorization.

Inspect the client view

On a Linux client with the relevant utilities installed, these are read-only starting points:

id
findmnt -t nfs,nfs4
ls -ln /mnt/security-lab
getfacl /mnt/security-lab/example.txt

getfacl is a client-side view; it does not necessarily expose every server-side Windows ACL semantic. Pair it with OneFS effective-identity and ACL inspection for the installed release. On Windows, inspect the actual connected identity and both share and file permissions.

Repeat the access test after a group membership change using a new authenticated session. An existing session or cached token can preserve old access and make the timing look inconsistent.

What a migration must preserve

Before moving a directory tree, capture ownership, inherited ACL behavior, identity mappings, and application service identities. A byte-identical copy can still be an authorization failure if SID-to-UID mapping changes at the destination.

Accept the migration only when the expected allowed and denied operations agree on both protocols. Keep the test users and synthetic dataset for later directory or OneFS upgrades. This small regression suite is more useful than a one-time screenshot of permissions.

Related: NFS authentication boundaries.

PowerScale SmartLock: a successful replica can have the wrong protection

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

On a Dell PowerScale or older Isilon estate, SnapshotIQ, SyncIQ, and SmartLock answer different questions. A point-in-time view, a remote copy, and a write-once retention policy should not be represented by one green status in an operations dashboard.

The particularly useful failure case is a replica whose data arrived but whose intended retention protection did not.

Read the source-to-target rules

Dell’s OneFS 9.9 backup guide documents that replicating a SmartLock enterprise directory before creating the target SmartLock directory can produce a normal target directory and a successful job. It also states that directory configuration is not carried across merely because WORM file state is replicated. See SmartLock replication limitations.

Those are release-specific rules, not permission to improvise on an existing compliance cluster. Consult the installed release’s guide before designing or changing the topology.

Separate four acceptance questions

Question	Evidence
Did data arrive?	Destination file contents and replication completion
Is the destination a SmartLock domain?	Destination domain configuration
Is this file committed with the expected retention?	File state and expiration
Can the service recover there?	Controlled recovery and client access test

A checksum answers only the first row. A successful SyncIQ run does not replace the remaining rows.

Build a representative test

Use an isolated source directory and a separate target path. Include a committed test file, an uncommitted file, and the directory settings the application depends on. Keep the retention window short enough for the approved lab exercise; retention changes can be irreversible within that window.

Record the source and target domain types before replication. Run the supported policy, inspect the destination independently, and confirm that a client with ordinary write access cannot alter the committed test file. Verify the expected expiration rather than assuming every file inherited the same policy.

Then evaluate failback using Dell’s matrix for the exact domain pair. A recovery plan that describes only copying forward leaves the harder operational decision unresolved.

Treat time and workflow as dependencies

Dell documents additional planning for compliance-mode failover, including cluster configuration and compliance clocks. Its failover guide should be part of the change record.

Keep retention requirements separate from product feature names. Choosing a mode does not establish the organization’s regulatory compliance. The engineering deliverable is a verified source-to-target protection contract, a recovery procedure, and documented ownership of exceptions.

For the wider recovery sequence, read replication and clean failback.

VAST indestructible snapshots: test the authority to undo protection

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A read-only snapshot stops a client from overwriting its historical contents. It does not automatically stop a storage administrator from removing the snapshot. For VAST, the distinction to examine is the protection attached to snapshots and policies, together with the authority that can change that protection.

Establish the documented boundary

VAST documents an Indestructibility feature that protects flagged snapshots and protection policies against deletion and retention reduction. Its architecture documentation also describes a controlled, time-limited support unlock. That exception belongs in the security model: the customer representatives and verification procedure behind it matter as much as the checkbox. See VAST’s overview and the platform white paper.

Confirm the documentation for the installed VAST software release. Do not infer that a feature name proves protection against every administrator, every support workflow, or the loss of the entire site.

Write a protection inventory

For every protected path, record its policy, snapshot frequency, retention, replication destination, and responsible team. Record whether protection was explicitly enabled and whether the most recent recovery point completed. A configured schedule and an existing snapshot are separate facts.

Review item	Evidence to collect
Scope	Protected path and a file known to reside beneath it
History	Actual recovery-point timestamps
Retention	Expiration shown for a specific protected snapshot
Authority	Roles allowed to change policies or request unlock
Off-site survival	Recovery point visible at the intended peer

Export the inventory to an access-controlled evidence location. A screenshot without a snapshot identifier is difficult to reconcile later.

Exercise a disposable policy

Create a small test dataset through the normal client protocol. Take a protected recovery point with an approved test retention. Modify the live test data and verify that the earlier version remains readable through the supported recovery workflow.

Use an ordinary administrative role to attempt a forbidden policy or retention change on this test scope. Record the denial and the identity used. Then inspect the configuration again: an error message is less persuasive than a confirmed unchanged protection state.

Separately review the support-unlock contacts and approval process with the customer owner. Do not initiate an unlock merely to prove the support channel exists. A tabletop review can establish who is authorized and how a compromised mailbox would be handled.

Plan for capacity and recovery

Retained history occupies capacity as the live dataset changes. Model a high-change incident window as well as routine daily churn. An unexpectedly long retention period can create an operational problem precisely because ordinary deletion is restricted.

Finally, restore to an isolated destination and validate representative file contents, permissions, and application behavior. Snapshot survival is the first result; usable recovery is the second. Keep both in the acceptance report, alongside the identity boundary around recovery.

Recovery after an administrator account is compromised

2026-09-09T00:00:00-04:00

Editorial archive date; first published and reviewed 9 September 2026. Examples are proposed checks, not measured customer results.

A backup job can finish successfully while the recovery design remains vulnerable to one stolen administrator account. The question is whether that account can disable protection, remove recovery points, change retention, or obtain the credentials needed to destroy the other copies.

This is an architecture review and a proposed exercise, not a report of a customer incident. Start with one critical service and trace the actual dependencies needed to bring it back.

Draw the recovery dependency chain

An example service needs directory authentication, DNS, a virtualization platform, a storage share, database keys, and an application configuration repository. If the backup console authenticates only against the compromised directory, restoring the data does not solve the access problem. If the key server runs only on the failed storage, the recovery order is circular.

Make a worksheet with a row for each dependency:

Dependency	Normal authority	Recovery authority	Independent copy
Backup catalog	Backup service account	Recovery operator	Export outside production
Encryption keys	KMS administrators	Documented recovery custodians	Supported KMS recovery material
Storage configuration	Storage administrators	Restricted emergency account	Protected configuration export
Application secrets	Application identity	Approved recovery workflow	Recoverable secret store

The entries above are a design example. Replace them with named owners and tested procedures. Do not put credentials in the worksheet.

Test the administrator boundary

Use a disposable dataset under a short, approved protection policy. Enumerate what the ordinary storage administrator, backup administrator, and recovery operator can each do. Test permission denial against that dataset rather than attempting destructive operations against production backups.

A useful negative test asks whether an everyday administrator can shorten retention or remove the final recovery point. A useful positive test asks whether the designated recovery operator can restore the protected copy without the normal production login path. Both are needed: a system that denies everyone is resistant to deletion but may also be impossible to recover.

CISA’s ransomware guidance recommends protected backups and recovery testing. The dependency worksheet and role tests here are an engineering method for turning that objective into observable evidence.

Measure service recovery

Record the start time, chosen recovery point, data restoration completion, application checks, and business acceptance. A mounted filesystem is an intermediate milestone. The application owner should read representative records, perform a controlled write, and confirm access boundaries.

Keep the recovered service isolated until the incident team accepts the source data and credentials. Restoring an old image can restore persistence or a vulnerable configuration along with the application.

What to retain

Keep the dependency map, permission-test results, recovery-point identifier, elapsed time, and remaining manual steps. Repeat the exercise when identity, encryption, backup tooling, or storage topology changes. Those changes can invalidate recovery without causing a single backup job to fail.

Continue with replication and failback and a restore benchmark.

Class file major	Java release	Class file major	Java release
49	5	60	16
50	6	61	17
51	7	62	18
52	8	63	19
53	9	64	20
54	10	65	21
55	11	66	22
56	12	67	23
57	13	68	24
58	14	69	25
59	15

Class file major	Java release	Class file major	Java release
49	5	60	16
50	6	61	17
51	7	62	18
52	8	63	19
53	9	64	20
54	10	65	21
55	11	66	22
56	12	67	23
57	13	68	24
58	14	69	25
59	15

Class file major	Java release	Class file major	Java release
49	5	60	16
50	6	61	17
51	7	62	18
52	8	63	19
53	9	64	20
54	10	65	21
55	11	66	22
56	12	67	23
57	13	68	24
58	14	69	25
59	15