Writing · 132 articles

AI and HPC infrastructure articles.

Setup, troubleshooting and monitoring for the people running HPC and AI systems.

Start with a problem-based guide, use the lab tools, choose a category, or browse by topic.

AI Infrastructure · 12 articles

Benchmarking · 11 articles

Data Infrastructure · 2 articles

Detection · 9 articles

Enterprise IT · 8 articles

Node-local state and the reimage that erases it

A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is knowing which changes are node...

Enterprise IT

Fabric · 16 articles

An MPI troubleshooting playbook: symptom to cause

Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguishes them, and most of the time ...

Fabric

InfiniBand error counters: read deltas, not totals

Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta over a known window tells you wh...

Fabric

GPU Infrastructure · 7 articles

Genomics · 1 articles

HPC · 3 articles

HPC Architecture · 1 articles

HPC Operations · 9 articles

Identity · 4 articles

Infrastructure Management · 1 articles

Kubernetes · 2 articles

Lab notes · 1 articles

Operations · 5 articles

Six shell idioms that quietly corrupt a data pipeline

Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and fails, and an environment cha...

Operations

The green check that hides a broken result

In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents are wrong. A taxonomy of how ...

Operations

Parallel Storage · 7 articles

Recovery · 4 articles

Research Infrastructure · 3 articles

Storage · 16 articles

Why du and df disagree about a compressed filesystem

du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real ratio means asking the allocator,...

Storage

Deploying a GPFS ECE cluster, end to end

Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisions that are expensive to rever...

Storage

Storage security · 7 articles

Systems Management · 3 articles

Written by Lenny Shovsky. Browse by topic. Research infrastructure services. Subscribe by RSS.