Offboarding: disabling the directory account is one step
Test existing sessions and independent credentials as well as new sign-ins.
Identity
The same 72 articles as the main index, grouped by subject rather than by category. A topic appears here when it carries more than one article; everything else is listed at the foot of the page.
Test existing sessions and independent credentials as well as new sign-ins.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
A scan result becomes useful evidence when the profile, content version, scope, and exceptions are retained.
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents...
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
A recovery copy is only useful if the people and systems needed to restore it survive the same incident.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
Before pulling tens of terabytes out of someone else's bucket, you want to know whose invoice it lands on. The configuration API that would answer the questi...
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a check...
Check record content, sender alignment, and real message headers before declaring mail authentication healthy.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
Different tools report link speed in different units, half the field is per-lane, and a link that negotiated two lanes instead of four is Active and silent. ...
Containment decisions should stop harmful activity while preserving evidence and an authorized recovery path.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Keeping compute nodes in their own cluster and remote-mounting the filesystem is the right default for almost every site. The setup is four commands. The par...
Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisio...
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and...
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Two of the columns nccl-tests prints look like bandwidth. Only one of them is comparable across job sizes, and picking the wrong one is how a healthy cluster...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Use matching and nonmatching captures, explicit actions, and live enforcement tests to validate a rule change.
Start with capture visibility and a harmless detection fixture before counting alerts.
Different tools report link speed in different units, half the field is per-lane, and a link that negotiated two lanes instead of four is Active and silent. ...
The MPI layer looks identical on both. Everything underneath it is different, and on the Ethernet side a single hop configured wrongly degrades collectives a...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Every endpoint you add consumes address space and forwarding table entries. Both are finite, both are countable in advance, and neither fails gracefully.
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
Doubling your port count is the easy part. The parts that surprise people are the renumbering, the cabling, and what it does to the subnet manager.
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
A read test smaller than the server's buffer pool measures memory. An under-driven benchmark is indistinguishable from a ceiling. Counters are not rates, and...
Use matching and nonmatching captures, explicit actions, and live enforcement tests to validate a rule change.
An inline sensor is part of the availability path. Validate forwarding, selective blocking, and failure behavior separately.
Set an ACL on the cache and it reverts, because the home side is authoritative and revalidation overwrites what you did. Add chmod silently destroying NFSv4 ...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
Each of these is valid shell that runs without error and produces a wrong answer: an empty field separator that looks like a tab, a grep that prints zero and...
Use small batches, explicit stop conditions, and application checks to avoid reporting installation as remediation.
A selective baseline and a tested response path make file-change alerts useful.
Client trust, user identity, integrity, and encryption are separate choices in an NFS design.
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Streaming bandwidth is the easiest number to produce and the least likely to describe an AI training workload. The metadata rate usually matters more, and al...
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Test existing sessions and independent credentials as well as new sign-ins.
An MFA requirement and a phishing-resistant authentication requirement are not identical policies.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
In a mature estate the dangerous failures are not crashes. They are the runs where exit status is zero, the log looks clean, the file exists and the contents...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Client trust, user identity, integrity, and encryption are separate choices in an NFS design.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
A transfer that reports success has told you about HTTP status codes, not about bytes. Verifying tens of terabytes is tractable, but only if you pick a check...
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Completion order correlates with size, so the first items to finish are the small ones. Any rate extrapolated from early completions is biased, and on a long...
A cron job that redirects its own output has disabled the only notification channel cron has. Add a second copy of the crontab that nobody is reading and you...
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Every endpoint you add consumes address space and forwarding table entries. Both are finite, both are countable in advance, and neither fails gracefully.
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
A whole class of Unix configuration is "every file in this directory counts". Edit one in place and leave a .bak beside it, and the backup becomes live confi...
A fix applied to a running node survives until that node is reimaged, and then it is gone with no event to connect the two. The decision that prevents it is ...
Set an ACL on the cache and it reverts, because the home side is authoritative and revalidation overwrites what you did. Add chmod silently destroying NFSv4 ...
du reports a file's apparent size, so on a transparently compressed filesystem it measures the data you wrote, not the space you occupy. Getting the real rat...
Investigate Lustre metadata latency, per-target capacity, slow clients and AI pipeline stalls with scoped checks, workload evidence, and an operations handoff.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
nvidia-smi reporting 100% means a kernel was resident, not that the tensor cores did useful work. Model FLOPs utilization is the honest number, and a mid-thi...
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
Choose and test Lustre stripe layouts for AI datasets, shared HPC files and checkpoints. Inspect actual file layouts and compare performance without global c...
How to find out that the expensive stage of a variant annotation pipeline is spending almost all of its time on records with nothing to annotate, how to remo...
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
Design Dell PowerScale and Isilon storage for AI and HPC pipelines: map client access, dataset layout, compute placement, capacity, and recovery requirements.
Troubleshoot PowerScale and Isilon NFS performance from Linux clients through SmartConnect to OneFS, separating metadata, data, network, and background work.
An SMB login and an NFS identity can reach the same files through different mappings. Test both paths.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
The textfile collector parses a metrics file as a single unit. One misplaced TYPE declaration and node_exporter drops every series in the file, while the scr...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
Diagnose slow PyTorch data loading on HPC clusters with per-rank timing, worker sweeps, storage checks, and reproducible GPU throughput measurements.
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
Every layer of an MPI stack has a ceiling set by the layer beneath it. Test from the top and you will spend a week tuning something that was never the constr...
Port counters are cumulative since the last reset, so a large number may be five years of history and a small one may be happening right now. Only a delta ov...
Point-to-point latency and bandwidth are the first numbers to establish on a new cluster, because every distributed result you collect afterwards is built on...
Measure distributed AI checkpoint stalls, storage contention, and restart time. Validate recoverable training state before tuning asynchronous saves.
A recovery copy is only useful if the people and systems needed to restore it survive the same incident.
A scan result becomes useful evidence when the profile, content version, scope, and exceptions are retained.
Labels, supported booleans, and application behavior should be understood before a new allow rule is introduced.
A single flow takes a single path, so a pair test is indifferent to almost everything that is wrong with a fabric. Fan-out puts many flows on it at once, and...
Setting routing_engine does not mean the engine you named is the engine you got. ftree in particular declines the job and hands it back, and it does so in a ...
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
A quiet dashboard can mean no incidents, a broken producer, or a source that vanished from inventory.
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Retention, administrative separation, support unlock, and recovery are different parts of the same protection design.
Throughput is one component of recovery. Metadata, identity, keys, and application validation can dominate elapsed time.
How to tell a parallel filesystem read ceiling apart from a static-routing collapse on the inter-switch links, using per-ISL counter deltas and a network-onl...
An enabled audit setting is the start of an evidence pipeline, not proof that usable records exist.
Keeping compute nodes in their own cluster and remote-mounting the filesystem is the right default for almost every site. The setup is four commands. The par...
Erasure Code Edition turns commodity servers with internal NVMe into a parallel filesystem with no external RAID. The install is well documented. The decisio...
A small fixture library makes rule changes reviewable and keeps false-positive fixes from silently removing coverage.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
An inline sensor is part of the availability path. Validate forwarding, selective blocking, and failure behavior separately.
Start with capture visibility and a harmless detection fixture before counting alerts.
Remote availability, historical recovery, and returning service to the primary site require separate decisions.
Check destination WORM state and retention explicitly instead of treating a successful replication job as proof.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
Hangs, stragglers, silent TCP fallback, launcher failures and results that change between runs. Each has a small set of causes and a command that distinguish...
A second rail is bought as a doubling and usually arrives as something well short of it. The reasons are enumerable, most of them are in the host rather than...
Encrypted data and recoverable data are different outcomes. Test the key dependency without risking production keys.
A completed upload is a milestone. Recovery also needs the locator, compatible software, credentials, and usable application data.
Combine network, endpoint, and application evidence instead of assuming one sensor sees inside every connection.
Use connection identifiers to connect evidence while keeping the limits of passive visibility explicit.
Tags carrying a single article, linked straight to it.
Written by Lenny Shovsky. Subscribe by RSS.