PyTorch DataLoader bottlenecks on HPC GPU clusters
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
Setup, troubleshooting and monitoring for the people running HPC and AI systems.
132 articles. Choose a category below.
Start with a problem-based guide, use the lab tools, choose a category, or browse by topic.
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
Operate NVIDIA AI infrastructure through controlled drivers and containers, GPU Operator boundaries, DCGM monitoring, scheduler allocation, Triton release management, and tested recovery.
Set up and qualify a Run:ai research platform through dependency checks, project policy, distributed jobs, storage recovery and monitoring that distinguishes allocation from progress.
Integrate NVIDIA Triton scoring with batch and streaming systems using bounded GPU queues, microbatch latency budgets, versioned outputs, idempotent replay, and event-time correctness.
Map distributed training onto NVIDIA GPU servers using a memory ledger, NVLink and PCIe topology, ConnectX and InfiniBand placement, NCCL transport evidence, and recoverable sharded check...
Size adapter fine-tuning infrastructure using GPU memory, activation peaks, host offload, PCIe transfers, scratch capacity, runtime compatibility, and validation of the final serving arti...
Compare retrieval, adaptation, and pretraining by GPU residency, serving topology, storage traffic, training allocations, interconnect requirements, and the failure each approach can actu...
Plan retraining as an infrastructure workload with measured GPU-hour budgets, queue limits, data staging, checkpoint recovery, interference controls, and compatible serving promotion.
An infrastructure-led training pipeline covering NGC containers, host-driver compatibility, GPU allocation, checkpoint bandwidth, reproducible runs, and complete inference release bundles.
Build an inference acceptance test that joins model quality with NVIDIA GPU memory, Triton request timing, batch behavior, concurrency, precision changes, and replica-loss capacity.
Design a training input path that keeps NVIDIA GPUs supplied while preserving point-in-time correctness, dataset versions, host-memory limits, storage capacity, and recovery behavior.
How model requirements translate into NVIDIA GPU memory, server topology, TensorRT compatibility, Triton capacity, storage, power, and a defensible hardware acceptance test.
Build an AMD GPU investigation from a HIP reproducer to wavefront debugging, timeline tracing and architecture-specific cache and compute analysis, then verify the result outside the tools.
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thirties result on a real trainin...
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and that is when static routing, ...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constraint. Test from the bottom and...
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster gets declared broken.
HPL-MxP reports a much larger number than HPL on the same machine. Knowing exactly where that factor comes from is the difference between a useful measurement and a marketing slide.
Investigate AMD CPU performance using topology, uProf event sampling and IBS, with explicit distinctions between core activity, cache domains, memory locality and data-fabric traffic.
HPCG reaches a small percentage of peak and that is the entire point. It measures the parts of the machine that actually limit most scientific codes.
HPL is a poor predictor of application performance and an excellent acceptance test. Those two statements are both true, and confusing them is how people end up tuning the wrong thing.
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on top of them.
Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to DRAM.
Diagnose parallel GPU analysis through Spark plans, RAPIDS task concurrency, host and device spill, shuffle traffic, Nsight traces, and controlled scaling with verified output equivalence.
Design storage and node resources for NVIDIA RAPIDS analytics, connecting Parquet layout, small-file overhead, GPU executor placement, host memory, transfer cost, and spill behavior.
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
A small fixture library makes rule changes reviewable and keeps false-positive fixes from silently removing coverage.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
A selective baseline and a tested response path make file-change alerts useful.
Use matching and nonmatching captures, explicit actions, and live enforcement tests to validate a rule change.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
An inline sensor is part of the availability path. Validate forwarding, selective blocking, and failure behavior separately.
Start with capture visibility and a harmless detection fixture before counting alerts.
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live configuration — displayed to users,...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is knowing which changes are node...
Check record content, sender alignment, and real message headers before declaring mail authentication healthy.
A namespace is an organizational boundary. Verify the controls that make it an access boundary.
A scan result becomes useful evidence when the profile, content version, scope, and exceptions are retained.
Use small batches, explicit stop conditions, and application checks to avoid reporting installation as remediation.
Labels, supported booleans, and application behavior should be understood before a new allow rule is introduced.
BMCs, hypervisors, storage controllers, and provisioning services belong in the privileged-access model.
Trace Spectrum-X behavior through switch marking, NIC firmware, runtime QoS and SPC-X algorithm parameters, with a versioned inventory instead of a universal tuning script.
Configure and qualify a Spectrum-X multi-plane deployment with an explicit rail/plane map, a release-matched NIC profile, and per-plane application and failure tests.
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguishes them, and most of the time ...
Investigate network CPU hotspots with queue and IRQ evidence, explicit ownership of affinity policy, and controlled tests that distinguish kernel packet processing from RDMA progress.
Different tools report link speed in different units, half the field is per-lane, and a link that negotiated two lanes instead of four is Active and silent. On a synchronized collective, ...
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives across the entire fabric while ...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than the fabric, and the first thi...
Every endpoint you add consumes address space and forwarding table entries. Both are finite, both are countable in advance, and neither fails gracefully.
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a log line most people never read.
Doubling your port count is the easy part. The parts that surprise people are the renumbering, the cabling, and what it does to the subnet manager.
Four commands do the work, and the one people skip is the one that makes the update real. The order matters, and the PSID matters more than the version number.
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta over a known window tells you wh...
Trace memory-registration errors through resource limits, userspace providers, containers and process scale instead of treating every verbs failure as a network fault.
Investigate RoCE congestion with matched workload intervals, queue mappings and counter deltas before changing thresholds or blaming packet loss.
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-only corroboration test. Includes...
Debug UCX from the installed build through MPI selection, device exposure and GPU-memory support, with bounded experiments and communication evidence from actual ranks.
Build a supported NVIDIA container path around Pyxis and Enroot, immutable images, cache placement, GPU constraints and multi-node acceptance tests.
Move from application timelines to targeted kernel counters, accounting for replay, cache state, occupancy and shared-resource effects before changing CUDA code.
Build a GPU monitoring pipeline that preserves device identity, sampling meaning and job boundaries, then handles unsupported or missing values honestly.
Reduce CUDA failures to a reproducible case, separate host and device errors, choose the right sanitizer, and preserve optimized behavior when a debug build hides the fault.
Qualify fixed MIG layouts, device isolation, profile naming and accounting before offering GPU slices to research users through Slurm.
Investigate GPU faults with a shared event timeline, stable device identity, generation-specific recovery guidance and a workload-based return-to-service test.
Plan GPU-node upgrades around the running kernel, module trust, userspace libraries and workload tests, with an image-level rollback that can actually boot.
How to find out that the expensive stage of a variant annotation pipeline is spending almost all of its time on records with nothing to annotate, how to remove that work in the right orde...
The environment module system fails quietly by design, and almost every one of its failures presents as missing or broken software. This walks through nine mechanisms, the command that di...
How to compute CPU and memory efficiency from Slurm's accounting database so the number survives a sceptical reader, which two field choices produce confidently wrong answers that nobody ...
Deploy a release-pinned Slinky Slurm cluster with explicit controller persistence, accounting, identity and GPU boundaries, then debug both Kubernetes and Slurm state.
Compare separate worker pools, Slurm on Kubernetes and coordinated scheduling, then test GPU ownership, identity, storage and failure recovery before sharing capacity.
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport at scale.
Separate allocation, process launch, rendezvous, communication and application progress when a distributed MPI or NVIDIA GPU workload appears to hang.
Reconstruct a failed job across allocation and step records, application output, cgroup limits and node events, then design a small reproduction that distinguishes causes.
Trace a pending Slurm job through resource shape, association limits, priority, reservations and backfill without changing cluster policy to repair one submission.
Build a Slurm admission test around CPU placement, memory enforcement, GPU device access and process cleanup, then carry that evidence through image and driver changes.
Qualify Slurm controller high availability through shared-state checks, dependency failure tests, accounting separation and evidence from real job transitions.
A practical release process for BCM images: separate desired configuration from running state, inspect synchronization scope, and qualify a reboot before rolling changes outward.
Design a restart contract around Slurm signals, durable checkpoints and bounded retries so a requeued training or simulation job resumes useful work rather than repeating it.
Control array concurrency, input discovery and result publication so large parameter sweeps finish predictably without overwhelming shared metadata services.
Secret issuance, application renewal, and credential revocation are separate steps to verify.
Test existing sessions and independent credentials as well as new sign-ins.
A jump host is useful only when direct access, forwarding, and downstream authority are deliberately controlled.
An MFA requirement and a phishing-resistant authentication requirement are not identical policies.
Build read-only Redfish collection around resource discovery, sensor identity, timestamps and explicit missing-data states, then validate it against scheduler and node observations.
Build and debug GPU pods with multiple RDMA networks by checking host readiness, CNI attachments, device allocation and the application transport independently.
Configure GPU batch admission with Kueue queues and hardware flavors, then distinguish quota waits from placement failures and distributed application startup problems.
Problem-based guides, local tools, sample reports and illustrative GPU and storage investigations, available without a signup.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long transfer it can be wrong by m...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you have a job whose behaviour ca...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the script that wrote it exits zero a...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and fails, and an environment cha...
In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents are wrong. A taxonomy of how ...
Diagnose a missing GPFS storage path by tracing IPoIB neighbor discovery, ARP filtering and source routing. Preserve peer traffic, verify both RDMA paths and make the fix survive interfac...
Diagnose write failures despite free filesystem capacity by separating mount identity, fileset inode space, quota scope, pool headroom and application behavior.
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global changes.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
Containment decisions should stop harmful activity while preserving evidence and an authorized recovery path.
Remote availability, historical recovery, and returning service to the primary site require separate decisions.
A recovery copy is only useful if the people and systems needed to restore it survive the same incident.
Choose a research container workflow by comparing identity, privilege, filesystems, image delivery, scheduler integration and reproducibility rather than image format alone.
Design a node-local staging workflow with capacity checks, immutable inputs, bounded transfers, transactional result publication and explicit recovery after interruption.
Build a reproducible Intel compiler investigation with matching symbols, explicit optimization levels, Intel Distribution for GDB and controlled thread and runtime settings.
Set up a BeeGFS mirror canary, distinguish reachability from consistency, and qualify recovery through target identity, resync scope and application-level data checks.
Map namespace identity and path independence, inspect ANA state, and validate failure and restoration with application-level data checks instead of relying on path counts.
Before pulling tens of terabytes out of someone else's bucket, you want to know whose invoice it lands on. The configuration API that would answer the question is usually denied to you — ...
Set an ACL on the cache and it reverts, because the home side is authoritative and revalidation overwrites what you did. Add chmod silently destroying NFSv4 ACLs and you have permissions ...
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a checksum the provider will actually...
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real ratio means asking the allocator,...
Plan NVMe firmware work around verified device identity, slot capabilities, activation requirements and workload recovery, with a rollback assessment before touching the device.
Keeping compute nodes in their own cluster and remote-mounting the filesystem is the right default for almost every site. The setup is four commands. The parts that bite are the ones that...
Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisions that are expensive to rever...
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and almost nobody measures it.
Investigate Lustre client evictions using client, server and network timelines, then validate application recovery before returning affected nodes to service.
Build an NVMe inventory that distinguishes controllers, namespaces, paths and filesystems, then use it to scope namespace changes and replacement work accurately.
Evaluate cuFile and GPUDirect Storage with direct-path evidence, fallback detection and application-level measurements, including cases where the data loader remains the bottleneck.
Interpret NVMe SMART and error records with device identity, units and interval deltas, then connect them to kernel events and application-visible failures.
Evaluate Linux NFS connection parallelism with controlled mount state, RPC evidence and application measurements, while preserving the workload’s durability and access requirements.
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and aggregate CPU idle is meaning...
Client trust, user identity, integrity, and encryption are separate choices in an NFS design.
A delete marker can hide a protected version. Recovery tooling must know which version it needs.
Encrypted data and recoverable data are different outcomes. Test the key dependency without risking production keys.
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
Retention, administrative separation, support unlock, and recovery are different parts of the same protection design.
Plan firmware maintenance by hardware compatibility and failure domain, track transfer and activation separately, then verify boot, fabric, storage and scheduled workloads before readmiss...
Build Redfish automation around discovered capabilities, verified identity, asynchronous task tracking, conditional updates and recovery from uncertain request outcomes.
Investigate PCIe link width, speed and AER events across the endpoint, switches and root port, then validate the result with GPU, NIC and storage workloads.
Written by Lenny Shovsky. Browse by topic. Research infrastructure services. Subscribe by RSS.