Research infrastructure services

HPC and AI benchmark and acceptance testing

I measure what a cluster delivers for your workloads and give your team a report it can use with vendors, procurement and future operators.

Agree on the question before running tests

Acceptance testing is most useful when the expected behavior is written down before delivery. We select workloads, configuration and thresholds that reflect what the cluster was bought to do. A headline bandwidth or FLOPS result alone may leave the important application paths untested.

The service covers one cluster against an agreed acceptance plan. Results retain hardware and software identity, placement, input sizes and test settings, giving your team a basis for comparison after a vendor change or future upgrade.

Include the relevant layers

The test set can include IOR and mdtest for storage, OSU and NCCL for communication, and application tests. I separate client cache, concurrency and placement effects from the behavior being measured and investigate gaps between isolated tests and the complete workload.

For GPU systems, the plan can include training throughput, step time, memory use, data loading, checkpoint save and restore, and scaling across devices and nodes. Loss or agreed quality metrics remain part of interpreting a training result; a faster run that changed the work is not an equivalent comparison.

Scope AI training and pretraining explicitly

Selected pretraining or continued-pretraining benchmarks can use agreed datasets and model architectures such as Llama, Qwen, Mistral or DeepSeek. Language encoders, vision transformers or diffusion workloads can be considered where relevant. Hardware, licenses and test goals determine the variants and sizes.

Before testing, we agree precision, batch size, sequence length, run duration and acceptance targets. Fine-tuning and inference tests can also be included where needed. The fixed-price service covers the accepted test window; full pretraining to completion or production model development requires a separate scope.

Receive a report that supports a decision

The engagement includes a vendor-facing report, findings and a readout. It identifies settings, observations and unresolved limits, so a result can be reproduced and a proposed remedy can be checked.

Retesting after vendor changes is quoted separately, as are remediation, travel and ongoing support. A brief qualification call confirms the cluster, workload, access, test window and delivery date. No paid architecture assessment is required before booking this service.

Engagement options

Choose a format below to see current rates and the ordering options for that engagement. If you are unsure which fits, we can work through the scope together.

Explore all services · About Lenny Shovsky