AI model selection as infrastructure design: sizing NVIDIA GPU services
How model requirements translate into NVIDIA GPU memory, server topology, TensorRT compatibility, Triton capacity, storage, power, and a defensible hardware acceptance test.
A model that wins a benchmark can still be a poor fit for the service you need to run. It may need more GPU memory than the available servers have, miss the latency target under load, or leave too little capacity for a failed replica. Model selection is also a decision about the infrastructure and people who will support it.
Work through the workload and the NVIDIA platform together. A large model spread across tightly coupled GPUs creates different placement and recovery problems from a classifier that fits on one device. Include those costs while comparing quality, before the hardware is ordered.
Translate the model shortlist into a hardware shortlist
Build a capacity sheet for each candidate: weight representation, runtime workspace, activations, concurrent requests, and, for autoregressive generation, the key/value cache. Include maximum admitted input and output lengths. Parameter count alone cannot establish how many requests fit.
Compare actual server configurations rather than GPU family names. A100 and H100 are relevant data-center candidates, but memory capacity, form factor, power envelope, and GPU interconnect depend on the exact product and server design. A PCIe system and an SXM system are different infrastructure choices. Verify slot layout, power delivery, cooling, and the path from each GPU to its network adapter.
The CUDA 12.4.1 Ampere and Hopper tuning guides document architecture-specific behavior. Use them to identify compatibility and optimization questions, then measure the application. Architecture capability is not a guarantee that a particular framework invokes an efficient kernel.
Choose the NVIDIA serving path before procurement
For supported workloads, evaluate TensorRT as an inference optimization path and Triton Inference Server as the service layer. TensorRT, Triton, and the language-specific runtime have different roles; installing one does not make an unsupported model graph deployable. Preserve a working framework baseline and validate exported or compiled candidates against it.
The Triton 24.04 optimization guide describes batching and instance-count tradeoffs. The proposed procurement test is an end-to-end load run with the intended batch policy, concurrency, input distribution, and tail-latency limit. Increase instance count only while memory and useful throughput remain acceptable.
Include host CPU, RAM, NICs, storage, and a replica-loss scenario in that test. Record requests completed within the objective per server, not just a peak model-throughput figure. A smaller model with adequate quality may free a GPU for redundancy or another service; a more demanding model must justify the extra infrastructure with a measurable task benefit.
Write the task as an input and an accepted output
“Use AI for support” is too broad to evaluate. “Assign each incoming ticket to one of twelve queues, with uncertain cases sent to an operator” is a testable task. Record the input fields available at decision time, the output schema, the consumer of the output, and the action that follows it.
Separate tasks that have different failure consequences. Drafting a proposed response and automatically changing an account are separate capabilities even if the same language model could produce both outputs. An extraction system may need exact field values and evidence spans; a search system may need useful ranking; a forecasting system needs errors measured at a specified horizon.
Describe what the system should do when information is missing. An abstention is an output with an operating cost, not an embarrassing case to delete from the evaluation. A candidate that can identify uncertainty usefully may be preferable to one with slightly better average accuracy and uncontrolled confident errors.
Establish a simple baseline
Before comparing large models, build the least complicated credible solution. For structured classification this might be logistic regression or a boosted tree. For document routing it might be keywords plus an existing search index. For a bounded extraction task it might be a parser and a review queue.
These are engineering controls on the experiment. If a more expensive candidate cannot beat the baseline under the real acceptance rules, additional model capacity has not justified its operating cost. The 2016 XGBoost paper is useful historical context: scalable tree learning was already a serious option for structured data well before the current interest in generative systems. It does not establish that boosted trees win every tabular task.
Compare against the current human or software process as well. Measure its error rate, turnaround time, and workload. Otherwise the project may optimize a model metric while making the actual service slower.
Apply hard gates before scoring preferences
A weighted score can hide a disqualifying property. Treat mandatory requirements as pass/fail gates and document who can approve an exception.
| Criterion | Evidence to request | Selection consequence |
|---|---|---|
| Data handling | Actual hosting path, retention settings, operator access | Exclude incompatible processing arrangements |
| Usage rights | License for the exact weights and intended use | Exclude unresolved restrictions |
| Output behavior | Representative schema and failure tests | Require validation or reject the candidate |
| Capacity | Measured latency at expected concurrency | Size the service or remove it from the shortlist |
| Recovery | Available version, deployment artifact, rollback procedure | Reject an unmaintainable dependency |
A downloadable checkpoint is not automatically unrestricted software. Model weights, tokenizer files, training data, and serving components can carry different terms. Record the actual artifacts reviewed rather than relying on the label “open.”
Build an evaluation set that resembles deployment
Sample work by source, length, language, complexity, and business importance. Include normal inputs, incomplete inputs, and inputs the system should decline. Keep entire related entities together when splitting data so that near duplicates or repeated customers do not create misleading independence.
Separate development cases from a final held-out evaluation. Prompts, thresholds, preprocessing, and model choice are all fitted decisions when they are adjusted in response to examples. Repeatedly looking at a held-out score turns it into another development signal. Preserve an untouched set for the final comparison, then collect new evidence after deployment.
For classification, publish a confusion matrix and the performance of important slices. For generation, define a rubric with concrete examples of acceptable and unacceptable answers. Blind the reviewers to candidate identity where practical. Record disagreements and resolve rubric ambiguities before treating small score differences as meaningful.
Price the error and the review work
Suppose, purely as an illustration, 10,000 documents arrive daily. Candidate A saves two seconds per document but creates 100 additional manual exceptions. If an exception takes five minutes, the added review is 500 minutes per day; the time saved is about 333 minutes. The faster model has increased the total labor requirement.
Use a cost model that includes serving, review, reprocessing, storage, operational support, and the impact of mistakes. A threshold should reflect those consequences. Do not assign a dollar figure to an uncertain harm just to make a spreadsheet appear precise; identify the uncertainty and test the operational response instead.
Where errors affect people or expensive actions, inspect which populations or work categories experience the errors. A good aggregate can coexist with an unusable result on the slice that matters most.
Measure performance under the intended workload
Use the expected distribution of request sizes and simultaneous users. Report median and tail latency, rejected requests, timeouts, and throughput within the latency objective. For generated output, distinguish time to first output from time to completion. A short demonstration answer does not represent a long document workload.
Include validation and downstream work in the measurement. If an answer requires a repair call or operator correction, count that. If a model needs more retrieval context to meet quality requirements, include that context in its memory and latency profile.
Choose a capacity margin intentionally. The smallest deployment that survives an average load test may have no room for retries, a failed replica, or a sudden burst.
Select a model you can operate
For the finalists, rehearse a version change, a failed dependency, and a rollback. Identify who owns evaluation cases, who can change the model, and who responds when the service produces valid-looking but incorrect output.
Record the exact model revision, preprocessing, prompt or feature pipeline, runtime, and acceptance thresholds. Schedule a review trigger such as a changed task, changed input distribution, revoked artifact availability, or measured quality regression. The 2017 ML Test Score provides a production-readiness rubric; the decision gates here are a proposed way to apply that concern to selection.
Leave enough detail for the next engineer to rerun the comparison and understand the choice. When the workload or input data changes, that record will show which assumptions need checking again.