AI Infrastructure

RAG or fine-tuning: what each choice demands from NVIDIA infrastructure

Compare retrieval, adaptation, and pretraining by GPU residency, serving topology, storage traffic, training allocations, interconnect requirements, and the failure each approach can actually repair.

When an AI system gives poor answers, adding retrieval and retraining the model solve different problems. They also place different demands on the infrastructure. Retrieval needs a maintained data and query path; adaptation needs GPU jobs and versioned datasets; large pretraining needs sustained distributed capacity and substantial checkpoint storage.

Find the failure first, then cost the complete NVIDIA platform needed to address it. Compare answer quality alongside GPU residency, storage, networking and the work of keeping the system recoverable.

Draw the resource topology before choosing the technique

A retrieval-based service can have separate parsing, embedding, indexing, reranking, and generation stages. Some stages may use NVIDIA GPUs; others may remain CPU- or storage-bound. Map their owners, placement, memory, and peak rates separately. Scaling generation replicas does not repair an index whose refresh process cannot keep up.

Estimate embedding refresh traffic from changed content, not only total corpus size. Include temporary index copies, metadata, authorization data, and the time needed to retire an old index safely. Measure whether embedding jobs compete with interactive requests for the same device or storage path.

A fine-tuning path adds its own dataset snapshots, training allocation, checkpoints, and validation service. A distributed pretraining path adds topology constraints: several GPUs merely available somewhere in a cluster are not equivalent to a suitable connected allocation.

Assign NVIDIA software to a specific stage

Triton is a candidate service layer for supported embedding, reranking, or generation backends; it is not a retrieval index or an authorization system. Its 24.04 model configuration reference describes backend, tensor, batch, and instance configuration. Qualify each stage’s supported shapes and memory demand, then qualify their combined request path.

TensorRT and TensorRT-LLM provide distinct deployment entry points for suitable supported models. Establish model, operator, precision, and runtime compatibility before budgeting gains. The TensorRT 8.6 OSS documentation illustrates that deployment depends on a specific software stack and that the OSS repository represents only part of the full release. It does not imply every checkpoint converts without additional work.

For a hypothetical design, keep the generator resident on its allocated GPU while batching embedding refresh work elsewhere. Compare that separation against co-location under the real latency objective. Include cache memory and a replica failure. If the model does not fit with the required concurrency, first revisit the model, precision, context policy, and admitted workload; adding a multi-GPU dependency is a substantial infrastructure decision.

Separate knowledge access from learned behavior

Consider an illustrative assistant that answers questions about equipment procedures. A procedure changes frequently, and the model gives an old torque specification. The first question is whether the correct procedure was available to the answer-generation step.

If retrieval returned an obsolete revision, changing the generator weights does not repair document lifecycle management. If retrieval returned the right revision but omitted the relevant table, inspect parsing and chunking. If the necessary table was visible and the answer still ignored it, investigate instruction following, reasoning, and output validation.

A different failure is consistent use of the wrong response format or inability to perform a domain-specific transformation. More documents may not solve that behavior. Supervised examples and a suitable adaptation method may be a better experiment.

Use retrieval when evidence should remain external

Retrieval-augmented generation supplies relevant material at answer time. The original RAG paper, submitted in May 2020, combined a parametric generator with a dense index. Enterprise implementations can differ substantially from that research architecture; the common design concern is how external evidence reaches generation.

External evidence is attractive when facts change, when an answer needs traceable sources, or when different users are entitled to different documents. It also creates responsibilities: ingestion, parsing, indexing, revision selection, access enforcement, and deletion propagation.

Enforce authorization before protected text reaches generation, and avoid caches that cross user permission boundaries. A citation is not an access control. An instruction telling a model to ignore unauthorized material is not a substitute for keeping that material out of the request.

Evaluate retrieval independently

Build a query set with known relevant evidence. Measure whether the retrieval stage finds that evidence and whether the returned version is valid for the question’s date and user. Inspect lexical search, vector search, and any reranking separately when they are components of the system.

Test difficult document structures: tables, repeated headers, attachments, scans, superseded procedures, and references spanning sections. Chunk size should follow the information needed to answer, not a universal number of tokens.

Then evaluate generation with the correct evidence supplied directly. If quality remains poor under this controlled condition, search tuning alone will not close the gap. Conversely, if generation succeeds with correct evidence but the end-to-end system fails, focus the next experiment on retrieval and orchestration.

Fine-tune for a measurable behavior change

Supervised fine-tuning adjusts a pretrained model using task examples. Good candidates include a repeatable extraction format, a specialized transformation, or consistent application of a rubric that the base model does not follow reliably.

Prepare examples with clear target behavior and representative edge cases. Separate training data from evaluation by source or entity where needed. Audit generated training examples instead of assuming synthetic volume produces correct supervision.

Keep a base-model comparison and evaluate general capability as well as task-specific gains. A fine-tuned model can improve one workflow while becoming less reliable on others. If the application needs broad behavior, preserve regression tests for that behavior.

Do not treat a fine-tuned checkpoint as a reliable database of changing facts. Learned associations can be difficult to update or remove precisely, and a model may still invent unsupported details. Retrieval and fine-tuning can be combined when their roles are independently justified.

Distinguish continued pretraining from supervised adaptation

Continued pretraining uses a general prediction objective on additional domain material. It may help when the underlying language or domain distribution is poorly represented, but it requires a careful corpus policy, compute budget, and evaluation plan. It is not interchangeable with teaching a small set of preferred answer examples.

Training from scratch is a larger decision still. It introduces tokenizer design, corpus acquisition and filtering, long-running distributed optimization, checkpoint recovery, and broad capability evaluation. Possessing a large document collection does not establish that this path is economical or that the rights and quality of the collection are suitable.

The 2017 Transformer paper establishes the architecture’s historical foundation. It does not supply a business case for building a new foundation model. That case needs evidence that adaptation and existing models cannot satisfy the actual requirement.

Compare complete systems under a fixed task

Use a controlled comparison that includes the same accepted outputs, data boundary, and failure policy. One candidate may combine a smaller model with better retrieval; another may use a larger adapted model. Measure their full behavior, rather than attributing the result solely to parameter count.

Observed failure First controlled experiment Evidence needed before expanding scope
Correct source never retrieved Supply the source directly Generator succeeds with sufficient evidence
Wrong document revision Repair version filtering Queries select the valid revision consistently
Output schema repeatedly wrong Validate and test supervised adaptation Reduced repair rate on held-out inputs
Domain language poorly understood Compare domain-adapted candidates Improvement beyond retrieval and prompting
Broad capability absent Compare stronger base models A demonstrated task gap with feasible alternatives

These are diagnostic suggestions, not guarantees about which technique will win.

Account for operational consequences

Retrieval adds ingestion and query-time dependencies. Fine-tuning adds dataset management, training, and per-version evaluation. Continued pretraining adds much larger compute and corpus obligations. All paths require serving, monitoring, and recovery.

Measure end-to-end latency, indexing freshness, access correctness, answer quality, and operator workload. A cheap generator with a complicated retrieval path may cost more to operate than a simpler alternative. A highly accurate fine-tune may be unsuitable if its training data cannot be maintained.

Choose the smallest intervention that resolves a demonstrated failure and survives the acceptance tests. Keep the cases that exposed the original failure. Rerunning them after a model or data change is the quickest way to see whether that problem was actually fixed.

All articles