Sizing NVIDIA GPUs for LoRA and QLoRA: memory, PCIe, and the full training node
Size adapter fine-tuning infrastructure using GPU memory, activation peaks, host offload, PCIe transfers, scratch capacity, runtime compatibility, and validation of the final serving artifact.
LoRA or QLoRA may let an adaptation job fit on fewer NVIDIA GPUs. The rest of the server still matters. Once the base model fits, host RAM, PCIe transfers, preprocessing CPUs or checkpoint storage can become the limiting resource.
Account for memory and transfers, then run a representative case on the intended server. Leave room for evaluation and recovery as well as steady-state training; the adapter download size tells you little about those demands.
Specify the device and server configuration precisely
Record the exact GPU SKU and installed memory, not just “A100” or “H100.” State whether the job uses one GPU, a supported GPU partition, or a connected group, and verify how the runtime sees those devices. The CUDA 12.4.1 Hopper guide provides the architectural context; it does not size a particular adapter workload.
Use representative maximum sequence lengths and supervised-token distributions for the fit test. Include one complete optimizer step, validation, and checkpoint saving. Leave explicit headroom instead of treating the last successful allocation as the operating limit. Compare useful tokens per second and total time to the required quality when evaluating a larger device.
Check the kernels and precision modes actually used. A quantized base can save storage and memory without making every operation execute as four-bit arithmetic. Confirm the adapter library, quantization implementation, framework, CUDA runtime, and device architecture as a tested combination.
Account for offload as host and transfer demand
Offloading can move pressure from device memory to host RAM and the CPU-to-GPU path. Estimate bytes transferred per step and measure the transfers in a short trace. The Nsight Systems 2024.2 guide describes CUDA tracing that can help reveal waits and copies; the proposed investigation is to correlate those events with your training-step boundaries.
Place CPU workers and pinned buffers with awareness of the GPU’s NUMA and PCIe attachment. Include other jobs on the node when testing host bandwidth. A configuration that succeeds on an otherwise empty server may stall when several jobs pin memory or stage data simultaneously.
Budget node-local scratch for dataset shards, temporary exports, and training checkpoints. Preserve a durable completed checkpoint outside the node if node loss is in the recovery requirement. Keep training allocations and interactive serving allocations separately accounted so that a fine-tuning experiment cannot silently consume the service’s latency margin.
These infrastructure checks set the feasible experiment envelope. The adapter, data, and quality controls below determine whether any configuration inside that envelope produces a useful model.
Understand what is updated
LoRA represents an update to a weight matrix using two smaller matrices. For a matrix with dimensions d by k, an adapter rank r introduces approximately r times the sum of d and k trainable parameters, instead of d times k for a full update to that matrix. The base weights remain frozen.
The LoRA paper was first submitted in June 2021. Its central contribution is a low-rank adaptation method, not a universal claim that every task needs the same rank or that every layer should receive an adapter.
QLoRA, submitted in May 2023, combines a frozen quantized base with trainable low-rank adapters. The paper describes NF4, double quantization, and paged optimizers. A four-bit base representation does not imply that every intermediate, adapter, gradient, or optimizer state is stored in four bits.
Build a memory ledger
List persistent and transient allocations separately. The persistent portion includes base weights, quantization metadata, adapters, adapter gradients, and optimizer state. The transient portion includes activations, attention workspaces, temporary dequantized values, communication buffers where applicable, and framework overhead.
As an illustrative lower-bound calculation, seven billion parameters at four bits occupy 3.5 billion bytes before metadata and other allocations. That is about 3.26 GiB. It is not a complete estimate of whether a seven-billion-parameter training job fits on a particular device.
Sequence length and microbatch size can change the activation budget dramatically. Measure representative long sequences, not just a batch of short examples. Keep memory headroom for evaluation and checkpoint operations, which may follow different allocation paths from steady-state training.
Record allocated and reserved memory as different observations. A framework allocator retaining memory is not automatically a leak; growth across repeated identical steps deserves investigation.
Choose adapter placement with an experiment
Identify the modules receiving adapters and verify that their names match the actual model architecture. A configuration copied from another model can target the wrong components or none at all.
Print the trainable parameter count and inspect several named parameters. Confirm that base weights are frozen and intended adapter weights receive gradients. Run a tiny update and check that adapter values actually change.
Treat rank, scaling, dropout, and target modules as experimental choices. A higher rank increases flexibility and cost, but improvement is task dependent. Compare a small number of justified configurations under the same data and evaluation budget before expanding the search.
Keep a full fine-tuning comparison only when resources and the project question warrant it. The useful decision is whether the adapter meets the task requirement, not whether one method wins every possible benchmark.
Verify tokenization and loss masking
Inspect decoded examples after the complete formatting path. Confirm role markers, separators, end tokens, truncation behavior, and which tokens contribute to the loss. A plausible-looking text file can become a very different training sequence after templating.
For supervised response training, decide whether the task should train on prompt tokens, response tokens, or another explicit span. Verify masks numerically and visually on a few examples. An answer truncated away by the maximum length can leave a record with no useful supervised target.
If packing multiple examples into a sequence, verify boundary and attention behavior for the implementation in use. Count useful supervised tokens as well as total processed tokens. Otherwise padding or masked prompt text can make a throughput result look more productive than it is.
Use validation examples from sources not represented by near duplicates in training. Adapter efficiency does not protect against data leakage.
Tune the optimizer using measured behavior
Start with a conservative task-appropriate configuration and a short run. Track training loss, held-out quality, gradient behavior, and output examples. A decreasing training loss is not sufficient evidence of useful adaptation.
Gradient accumulation increases the effective batch without requiring the same activation memory as a single larger microbatch, but the run still needs to preserve the intended loss normalization and update schedule. Record both microbatch size and effective batch size.
Activation checkpointing trades recomputation for memory. Offloading trades device pressure for host memory and transfer cost. Paged optimizer behavior can depend on memory pressure. Measure these choices on the intended hardware instead of assuming a configuration that fits will also train efficiently.
Save a reproducible adapter bundle
An adapter needs the exact compatible base revision, tokenizer, formatting template, adapter configuration, and runtime assumptions. Preserve those identifiers with the adapter weights. A base model family name does not identify a particular set of parameters.
For training continuation, retain optimizer and scheduler state in addition to the adapter. For deployment, retain the input contract and generation settings used during evaluation. Distinguish these two purposes so that a minimal serving export is not mistaken for a resumable checkpoint.
Hash important artifacts and test loading them in a clean environment. Document any custom code required by the loader and review it as executable software.
Evaluate the serving representation separately
Serving with an adapter, merging into higher-precision base weights, and quantizing a merged model are different artifact paths. Numerical behavior and runtime support can differ. Validate the final path using held-out task cases, general regression cases, long inputs, and invalid-input handling.
Measure quality and latency together. If a merged version removes adapter overhead but damages a critical slice after quantization, a smaller file is not sufficient justification for promotion.
Publish a result with base revision, adapter placement, rank, data version, supervised-token policy, context length, hardware, peak memory, runtime, and evaluation outcome. Such a record tells another engineer what was demonstrated. “Fine-tunes on one GPU” leaves almost every important condition unstated.