Offboarding: disabling the directory account is one step
Test existing sessions and independent credentials as well as new sign-ins.
The same 119 articles as the main index, grouped by subject rather than by category. A topic appears here when it carries more than one article; everything else is listed at the foot of the page.
Test existing sessions and independent credentials as well as new sign-ins.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport a...
Choose a research container workflow by comparing identity, privilege, filesystems, image delivery, scheduler integration and reproducibility rather than ima...
A scan result becomes useful evidence when the profile, content version, scope, and exceptions are retained.
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
Build Redfish automation around discovered capabilities, verified identity, asynchronous task tracking, conditional updates and recovery from uncertain reque...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents...
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
A recovery copy is only useful if the people and systems needed to restore it survive the same incident.
A practical release process for BCM images: separate desired configuration from running state, inspect synchronization scope, and qualify a reboot before rol...
Plan GPU-node upgrades around the running kernel, module trust, userspace libraries and workload tests, with an image-level rollback that can actually boot.
Build an AMD GPU investigation from a HIP reproducer to wavefront debugging, timeline tracing and architecture-specific cache and compute analysis, then veri...
Investigate AMD CPU performance using topology, uProf event sampling and IBS, with explicit distinctions between core activity, cache domains, memory localit...
Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to...
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and...
Investigate AMD CPU performance using topology, uProf event sampling and IBS, with explicit distinctions between core activity, cache domains, memory localit...
Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to...
Move from application timelines to targeted kernel counters, accounting for replay, cache state, occupancy and shared-resource effects before changing CUDA c...
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Plan retraining as an infrastructure workload with measured GPU-hour budgets, queue limits, data staging, checkpoint recovery, interference controls, and com...
Build an inference acceptance test that joins model quality with NVIDIA GPU memory, Triton request timing, batch behavior, concurrency, precision changes, an...
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Design a restart contract around Slurm signals, durable checkpoints and bounded retries so a requeued training or simulation job resumes useful work rather t...
An infrastructure-led training pipeline covering NGC containers, host-driver compatibility, GPU allocation, checkpoint bandwidth, reproducible runs, and comp...
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster...
Four commands do the work, and the one people skip is the one that makes the update real. The order matters, and the PSID matters more than the version number.
Map distributed training onto NVIDIA GPU servers using a memory ledger, NVLink and PCIe topology, ConnectX and InfiniBand placement, NCCL transport evidence,...
Build a supported NVIDIA container path around Pyxis and Enroot, immutable images, cache placement, GPU constraints and multi-node acceptance tests.
Choose a research container workflow by comparing identity, privilege, filesystems, image delivery, scheduler integration and reproducibility rather than ima...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
Move from application timelines to targeted kernel counters, accounting for replay, cache state, occupancy and shared-resource effects before changing CUDA c...
Evaluate cuFile and GPUDirect Storage with direct-path evidence, fallback detection and application-level measurements, including cases where the data loader...
Reduce CUDA failures to a reproducible case, separate host and device errors, choose the right sanitizer, and preserve optimized behavior when a debug build ...
Qualify fixed MIG layouts, device isolation, profile naming and accounting before offering GPU slices to research users through Slurm.
Plan GPU-node upgrades around the running kernel, module trust, userspace libraries and workload tests, with an image-level rollback that can actually boot.
An infrastructure-led training pipeline covering NGC containers, host-driver compatibility, GPU allocation, checkpoint bandwidth, reproducible runs, and comp...
Design a node-local staging workflow with capacity checks, immutable inputs, bounded transfers, transactional result publication and explicit recovery after ...
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a check...
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
Before pulling tens of terabytes out of someone else's bucket, you want to know whose invoice it lands on. The configuration API that would answer the questi...
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a check...
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Build a GPU monitoring pipeline that preserves device identity, sampling meaning and job boundaries, then handles unsupported or missing values honestly.
Operate NVIDIA AI infrastructure through controlled drivers and containers, GPU Operator boundaries, DCGM monitoring, scheduler allocation, Triton release ma...
Plan retraining as an infrastructure workload with measured GPU-hour budgets, queue limits, data staging, checkpoint recovery, interference controls, and com...
Reconstruct a failed job across allocation and step records, application output, cgroup limits and node events, then design a small reproduction that disting...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and...
Build a reproducible Intel compiler investigation with matching symbols, explicit optimization levels, Intel Distribution for GDB and controlled thread and r...
Reduce CUDA failures to a reproducible case, separate host and device errors, choose the right sanitizer, and preserve optimized behavior when a debug build ...
Check record content, sender alignment, and real message headers before declaring mail authentication healthy.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives a...
Investigate RoCE congestion with matched workload intervals, queue mappings and counter deltas before changing thresholds or blaming packet loss.
Different tools report link speed in different units, half the field is per-lane, and a link that negotiated two lanes instead of four is Active and silent. ...
Containment decisions should stop harmful activity while preserving evidence and an authorized recovery path.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
Plan firmware maintenance by hardware compatibility and failure domain, track transfer and activation separately, then verify boot, fabric, storage and sched...
Plan NVMe firmware work around verified device identity, slot capabilities, activation requirements and workload recovery, with a rollback assessment before ...
Four commands do the work, and the one people skip is the one that makes the update real. The order matters, and the PSID matters more than the version number.
Diagnose a missing GPFS storage path by tracing IPoIB neighbor discovery, ARP filtering and source routing. Preserve peer traffic, verify both RDMA paths and...
Diagnose write failures despite free filesystem capacity by separating mount identity, fileset inode space, quota scope, pool headroom and application behavior.
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Keeping compute nodes in their own cluster and remote-mounting the filesystem is the right default for almost every site. The setup is four commands. The par...
Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisio...
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and...
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster...
Compare separate worker pools, Slurm on Kubernetes and coordinated scheduling, then test GPU ownership, identity, storage and failure recovery before sharing...
Plan retraining as an infrastructure workload with measured GPU-hour budgets, queue limits, data staging, checkpoint recovery, interference controls, and com...
Evaluate cuFile and GPUDirect Storage with direct-path evidence, fallback detection and application-level measurements, including cases where the data loader...
Design a training input path that keeps NVIDIA GPUs supplied while preserving point-in-time correctness, dataset versions, host-memory limits, storage capaci...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
Investigate Lustre client evictions using client, server and network timelines, then validate application recovery before returning affected nodes to service.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Build read-only Redfish collection around resource discovery, sensor identity, timestamps and explicit missing-data states, then validate it against schedule...
Diagnose a missing GPFS storage path by tracing IPoIB neighbor discovery, ARP filtering and source routing. Preserve peer traffic, verify both RDMA paths and...
Diagnose write failures despite free filesystem capacity by separating mount identity, fileset inode space, quota scope, pool headroom and application behavior.
Use matching and nonmatching captures, explicit actions, and live enforcement tests to validate a rule change.
Start with capture visibility and a harmless detection fixture before counting alerts.
Integrate NVIDIA Triton scoring with batch and streaming systems using bounded GPU queues, microbatch latency budgets, versioned outputs, idempotent replay, ...
Build an inference acceptance test that joins model quality with NVIDIA GPU memory, Triton request timing, batch behavior, concurrency, precision changes, an...
Different tools report link speed in different units, half the field is per-lane, and a link that negotiated two lanes instead of four is Active and silent. ...
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives a...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Every endpoint you add consumes address space and forwarding table entries. Both are finite, both are countable in advance, and neither fails gracefully.
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
Doubling your port count is the easy part. The parts that surprise people are the renumbering, the cabling, and what it does to the subnet manager.
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
Trace memory-registration errors through resource limits, userspace providers, containers and process scale instead of treating every verbs failure as a netw...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Map distributed training onto NVIDIA GPU servers using a memory ledger, NVLink and PCIe topology, ConnectX and InfiniBand placement, NCCL transport evidence,...
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and...
Use matching and nonmatching captures, explicit actions, and live enforcement tests to validate a rule change.
An inline sensor is part of the availability path. Validate forwarding, selective blocking, and failure behavior separately.
Set an ACL on the cache and it reverts, because the home side is authoritative and revalidation overwrites what you did. Add chmod silently destroying NFSv4 ...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
Compare separate worker pools, Slurm on Kubernetes and coordinated scheduling, then test GPU ownership, identity, storage and failure recovery before sharing...
A namespace is an organizational boundary. Verify the controls that make it an access boundary.
Build a Slurm admission test around CPU placement, memory enforcement, GPU device access and process cleanup, then carry that evidence through image and driv...
Map namespace identity and path independence, inspect ANA state, and validate failure and restoration with application-level data checks instead of relying o...
Build an AMD GPU investigation from a HIP reproducer to wavefront debugging, timeline tracing and architecture-specific cache and compute analysis, then veri...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
Plan NVMe firmware work around verified device identity, slot capabilities, activation requirements and workload recovery, with a rollback assessment before ...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and...
Investigate AMD CPU performance using topology, uProf event sampling and IBS, with explicit distinctions between core activity, cache domains, memory localit...
Investigate Lustre client evictions using client, server and network timelines, then validate application recovery before returning affected nodes to service.
Build an NVMe inventory that distinguishes controllers, namespaces, paths and filesystems, then use it to scope namespace changes and replacement work accura...
Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to...
A practical release process for BCM images: separate desired configuration from running state, inspect synchronization scope, and qualify a reboot before rol...
Design a restart contract around Slurm signals, durable checkpoints and bounded retries so a requeued training or simulation job resumes useful work rather t...
Trace memory-registration errors through resource limits, userspace providers, containers and process scale instead of treating every verbs failure as a netw...
Build a reproducible Intel compiler investigation with matching symbols, explicit optimization levels, Intel Distribution for GDB and controlled thread and r...
Interpret NVMe SMART and error records with device identity, units and interval deltas, then connect them to kernel events and application-visible failures.
Reduce CUDA failures to a reproducible case, separate host and device errors, choose the right sanitizer, and preserve optimized behavior when a debug build ...
Use small batches, explicit stop conditions, and application checks to avoid reporting installation as remediation.
Evaluate Linux NFS connection parallelism with controlled mount state, RPC evidence and application measurements, while preserving the workload’s durability ...
Control array concurrency, input discovery and result publication so large parameter sweeps finish predictably without overwhelming shared metadata services.
Plan GPU-node upgrades around the running kernel, module trust, userspace libraries and workload tests, with an image-level rollback that can actually boot.
Build read-only Redfish collection around resource discovery, sensor identity, timestamps and explicit missing-data states, then validate it against schedule...
A selective baseline and a tested response path make file-change alerts useful.
Client trust, user identity, integrity, and encryption are separate choices in an NFS design.
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
Investigate Lustre client evictions using client, server and network timelines, then validate application recovery before returning affected nodes to service.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Test existing sessions and independent credentials as well as new sign-ins.
An MFA requirement and a phishing-resistant authentication requirement are not identical policies.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
Build a GPU monitoring pipeline that preserves device identity, sampling meaning and job boundaries, then handles unsupported or missing values honestly.
Interpret NVMe SMART and error records with device identity, units and interval deltas, then connect them to kernel events and application-visible failures.
In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents...
Build read-only Redfish collection around resource discovery, sensor identity, timestamps and explicit missing-data states, then validate it against schedule...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport a...
Separate allocation, process launch, rendezvous, communication and application progress when a distributed MPI or NVIDIA GPU workload appears to hang.
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
Separate allocation, process launch, rendezvous, communication and application progress when a distributed MPI or NVIDIA GPU workload appears to hang.
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster...
Map distributed training onto NVIDIA GPU servers using a memory ledger, NVLink and PCIe topology, ConnectX and InfiniBand placement, NCCL transport evidence,...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Evaluate Linux NFS connection parallelism with controlled mount state, RPC evidence and application measurements, while preserving the workload’s durability ...
Client trust, user identity, integrity, and encryption are separate choices in an NFS design.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Build a supported NVIDIA container path around Pyxis and Enroot, immutable images, cache placement, GPU constraints and multi-node acceptance tests.
Compare separate worker pools, Slurm on Kubernetes and coordinated scheduling, then test GPU ownership, identity, storage and failure recovery before sharing...
Plan firmware maintenance by hardware compatibility and failure domain, track transfer and activation separately, then verify boot, fabric, storage and sched...
Operate NVIDIA AI infrastructure through controlled drivers and containers, GPU Operator boundaries, DCGM monitoring, scheduler allocation, Triton release ma...
Integrate NVIDIA Triton scoring with batch and streaming systems using bounded GPU queues, microbatch latency budgets, versioned outputs, idempotent replay, ...
Diagnose parallel GPU analysis through Spark plans, RAPIDS task concurrency, host and device spill, shuffle traffic, Nsight traces, and controlled scaling wi...
Design storage and node resources for NVIDIA RAPIDS analytics, connecting Parquet layout, small-file overhead, GPU executor placement, host memory, transfer ...
Map distributed training onto NVIDIA GPU servers using a memory ledger, NVLink and PCIe topology, ConnectX and InfiniBand placement, NCCL transport evidence,...
Size adapter fine-tuning infrastructure using GPU memory, activation peaks, host offload, PCIe transfers, scratch capacity, runtime compatibility, and valida...
Compare retrieval, adaptation, and pretraining by GPU residency, serving topology, storage traffic, training allocations, interconnect requirements, and the ...
Plan retraining as an infrastructure workload with measured GPU-hour budgets, queue limits, data staging, checkpoint recovery, interference controls, and com...
An infrastructure-led training pipeline covering NGC containers, host-driver compatibility, GPU allocation, checkpoint bandwidth, reproducible runs, and comp...
Build an inference acceptance test that joins model quality with NVIDIA GPU memory, Triton request timing, batch behavior, concurrency, precision changes, an...
Design a training input path that keeps NVIDIA GPUs supplied while preserving point-in-time correctness, dataset versions, host-memory limits, storage capaci...
How model requirements translate into NVIDIA GPU memory, server topology, TensorRT compatibility, Triton capacity, storage, power, and a defensible hardware ...
Build a Slurm admission test around CPU placement, memory enforcement, GPU device access and process cleanup, then carry that evidence through image and driv...
Move from application timelines to targeted kernel counters, accounting for replay, cache state, occupancy and shared-resource effects before changing CUDA c...
Build a GPU monitoring pipeline that preserves device identity, sampling meaning and job boundaries, then handles unsupported or missing values honestly.
Design a restart contract around Slurm signals, durable checkpoints and bounded retries so a requeued training or simulation job resumes useful work rather t...
Investigate RoCE congestion with matched workload intervals, queue mappings and counter deltas before changing thresholds or blaming packet loss.
Evaluate cuFile and GPUDirect Storage with direct-path evidence, fallback detection and application-level measurements, including cases where the data loader...
Reduce CUDA failures to a reproducible case, separate host and device errors, choose the right sanitizer, and preserve optimized behavior when a debug build ...
Qualify fixed MIG layouts, device isolation, profile naming and accounting before offering GPU slices to research users through Slurm.
Plan GPU-node upgrades around the running kernel, module trust, userspace libraries and workload tests, with an image-level rollback that can actually boot.
Design a node-local staging workflow with capacity checks, immutable inputs, bounded transfers, transactional result publication and explicit recovery after ...
Map namespace identity and path independence, inspect ANA state, and validate failure and restoration with application-level data checks instead of relying o...
Plan NVMe firmware work around verified device identity, slot capabilities, activation requirements and workload recovery, with a rollback assessment before ...
Build an NVMe inventory that distinguishes controllers, namespaces, paths and filesystems, then use it to scope namespace changes and replacement work accura...
Interpret NVMe SMART and error records with device identity, units and interval deltas, then connect them to kernel events and application-visible failures.
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a check...
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Every endpoint you add consumes address space and forwarding table entries. Both are finite, both are countable in advance, and neither fails gracefully.
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
Evaluate Linux NFS connection parallelism with controlled mount state, RPC evidence and application measurements, while preserving the workload’s durability ...
Control array concurrency, input discovery and result publication so large parameter sweeps finish predictably without overwhelming shared metadata services.
Set an ACL on the cache and it reverts, because the home side is authoritative and revalidation overwrites what you did. Add chmod silently destroying NFSv4 ...
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
How to find out that the expensive stage of a variant annotation pipeline is spending almost all of its time on records with nothing to annotate, how to remo...
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives a...
Investigate RoCE congestion with matched workload intervals, queue mappings and counter deltas before changing thresholds or blaming packet loss.
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport a...
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
Build an AMD GPU investigation from a HIP reproducer to wavefront debugging, timeline tracing and architecture-specific cache and compute analysis, then veri...
Investigate AMD CPU performance using topology, uProf event sampling and IBS, with explicit distinctions between core activity, cache domains, memory localit...
Combine process-level sampling, core counters and socket-level memory observations to investigate Intel CPU performance without assigning every cache miss to...
How to find out that the expensive stage of a variant annotation pipeline is spending almost all of its time on records with nothing to annotate, how to remo...
Move from application timelines to targeted kernel counters, accounting for replay, cache state, occupancy and shared-resource effects before changing CUDA c...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Diagnose write failures despite free filesystem capacity by separating mount identity, fileset inode space, quota scope, pool headroom and application behavior.
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Diagnose parallel GPU analysis through Spark plans, RAPIDS task concurrency, host and device spill, shuffle traffic, Nsight traces, and controlled scaling wi...
Design storage and node resources for NVIDIA RAPIDS analytics, connecting Parquet layout, small-file overhead, GPU executor placement, host memory, transfer ...
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport a...
Separate allocation, process launch, rendezvous, communication and application progress when a distributed MPI or NVIDIA GPU workload appears to hang.
Diagnose a missing GPFS storage path by tracing IPoIB neighbor discovery, ARP filtering and source routing. Preserve peer traffic, verify both RDMA paths and...
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
Trace memory-registration errors through resource limits, userspace providers, containers and process scale instead of treating every verbs failure as a netw...
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
A recovery copy is only useful if the people and systems needed to restore it survive the same incident.
Plan firmware maintenance by hardware compatibility and failure domain, track transfer and activation separately, then verify boot, fabric, storage and sched...
Build Redfish automation around discovered capabilities, verified identity, asynchronous task tracking, conditional updates and recovery from uncertain reque...
Build read-only Redfish collection around resource discovery, sensor identity, timestamps and explicit missing-data states, then validate it against schedule...
BMCs, hypervisors, storage controllers, and provisioning services belong in the privileged-access model.
A scan result becomes useful evidence when the profile, content version, scope, and exceptions are retained.
Labels, supported booleans, and application behavior should be understood before a new allow rule is introduced.
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives a...
Investigate RoCE congestion with matched workload intervals, queue mappings and counter deltas before changing thresholds or blaming packet loss.
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
Build a supported NVIDIA container path around Pyxis and Enroot, immutable images, cache placement, GPU constraints and multi-node acceptance tests.
Validate hybrid and bind MPI container models through launcher, ABI, process manager, fabric libraries and device access, then prove the intended transport a...
Choose a research container workflow by comparing identity, privilege, filesystems, image delivery, scheduler integration and reproducibility rather than ima...
Compare separate worker pools, Slurm on Kubernetes and coordinated scheduling, then test GPU ownership, identity, storage and failure recovery before sharing...
Separate allocation, process launch, rendezvous, communication and application progress when a distributed MPI or NVIDIA GPU workload appears to hang.
Reconstruct a failed job across allocation and step records, application output, cgroup limits and node events, then design a small reproduction that disting...
Trace a pending Slurm job through resource shape, association limits, priority, reservations and backfill without changing cluster policy to repair one submi...
Build a Slurm admission test around CPU placement, memory enforcement, GPU device access and process cleanup, then carry that evidence through image and driv...
Plan firmware maintenance by hardware compatibility and failure domain, track transfer and activation separately, then verify boot, fabric, storage and sched...
Design a node-local staging workflow with capacity checks, immutable inputs, bounded transfers, transactional result publication and explicit recovery after ...
Diagnose write failures despite free filesystem capacity by separating mount identity, fileset inode space, quota scope, pool headroom and application behavior.
Build a GPU monitoring pipeline that preserves device identity, sampling meaning and job boundaries, then handles unsupported or missing values honestly.
Design a restart contract around Slurm signals, durable checkpoints and bounded retries so a requeued training or simulation job resumes useful work rather t...
Trace memory-registration errors through resource limits, userspace providers, containers and process scale instead of treating every verbs failure as a netw...
How to compute CPU and memory efficiency from Slurm's accounting database so the number survives a sceptical reader, which two field choices produce confiden...
Qualify fixed MIG layouts, device isolation, profile naming and accounting before offering GPU slices to research users through Slurm.
Control array concurrency, input discovery and result publication so large parameter sweeps finish predictably without overwhelming shared metadata services.
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Retention, administrative separation, support unlock, and recovery are different parts of the same protection design.
Design a node-local staging workflow with capacity checks, immutable inputs, bounded transfers, transactional result publication and explicit recovery after ...
Map namespace identity and path independence, inspect ANA state, and validate failure and restoration with application-level data checks instead of relying o...
Plan NVMe firmware work around verified device identity, slot capabilities, activation requirements and workload recovery, with a rollback assessment before ...
Investigate Lustre client evictions using client, server and network timelines, then validate application recovery before returning affected nodes to service.
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
Build an NVMe inventory that distinguishes controllers, namespaces, paths and filesystems, then use it to scope namespace changes and replacement work accura...
Evaluate cuFile and GPUDirect Storage with direct-path evidence, fallback detection and application-level measurements, including cases where the data loader...
Interpret NVMe SMART and error records with device identity, units and interval deltas, then connect them to kernel events and application-visible failures.
Evaluate Linux NFS connection parallelism with controlled mount state, RPC evidence and application measurements, while preserving the workload’s durability ...
Control array concurrency, input discovery and result publication so large parameter sweeps finish predictably without overwhelming shared metadata services.
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Design storage and node resources for NVIDIA RAPIDS analytics, connecting Parquet layout, small-file overhead, GPU executor placement, host memory, transfer ...
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
Design a training input path that keeps NVIDIA GPUs supplied while preserving point-in-time correctness, dataset versions, host-memory limits, storage capaci...
Keeping compute nodes in their own cluster and remote-mounting the filesystem is the right default for almost every site. The setup is four commands. The par...
Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisio...
A small fixture library makes rule changes reviewable and keeps false-positive fixes from silently removing coverage.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
An inline sensor is part of the availability path. Validate forwarding, selective blocking, and failure behavior separately.
Start with capture visibility and a harmless detection fixture before counting alerts.
Remote availability, historical recovery, and returning service to the primary site require separate decisions.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
Operate NVIDIA AI infrastructure through controlled drivers and containers, GPU Operator boundaries, DCGM monitoring, scheduler allocation, Triton release ma...
Integrate NVIDIA Triton scoring with batch and streaming systems using bounded GPU queues, microbatch latency budgets, versioned outputs, idempotent replay, ...
Compare retrieval, adaptation, and pretraining by GPU residency, serving topology, storage traffic, training allocations, interconnect requirements, and the ...
Build an inference acceptance test that joins model quality with NVIDIA GPU memory, Triton request timing, batch behavior, concurrency, precision changes, an...
How model requirements translate into NVIDIA GPU memory, server topology, TensorRT compatibility, Triton capacity, storage, power, and a defensible hardware ...
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
Build a reproducible Intel compiler investigation with matching symbols, explicit optimization levels, Intel Distribution for GDB and controlled thread and r...
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
Encrypted data and recoverable data are different outcomes. Test the key dependency without risking production keys.
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
Tags carrying a single article, linked straight to it.
Written by Lenny Shovsky. Subscribe by RSS.