How to Size GPU Servers for AI Inference
Choosing the right gpu server for inference is one of the most consequential infrastructure decisions an organization can make right now. Unlike model training — which happens once, runs for days on a cluster, and can tolerate modest latency — inference is a real-time, revenue-critical operation. Every query to a chatbot, every image classification call, every AI-assisted document review runs through your inference tier hundreds or thousands of times per day. Size too small and you miss your latency SLA. Size too large and you are paying for idle HBM that could fund other initiatives.
This guide walks through the key variables that govern inference sizing, maps those variables to specific HPE platforms, and gives you a decision framework you can use before you talk to a vendor. If you are already evaluating HPE hardware, the NVIDIA AI on HPE guide is a useful companion read once you have finished here.
Why Inference Sizing Differs from Training Sizing
Training workloads are compute-bound: you need raw FLOPS, high-speed interconnects like NVLink, and large collective-communication bandwidth across many GPUs. You can tolerate a few minutes of extra run time per epoch.
Inference workloads are almost always memory-bandwidth-bound, particularly for large language models (LLMs). At typical serving batch sizes — often just 1 to 8 concurrent requests for latency-sensitive APIs — the GPU spends most of its cycles reading model weights from HBM, not performing matrix multiplications. As a result:
- HBM capacity determines which models fit on a single GPU or node without splitting across devices.
- HBM bandwidth determines how fast tokens are generated per second at a given batch size.
- PCIe vs. NVLink topology determines whether you can efficiently shard models across multiple GPUs.
- CPU and host memory determine how many concurrent requests can be queued and how fast KV-cache can be managed.
Conflating training sizing rules with inference sizing is the most common mistake organizations make when they first spec out AI infrastructure.
The Four Numbers You Must Know Before Sizing
Before you pick a server, collect these four numbers for your target workload. Everything else follows from them.
1. Model size in parameters (and precision) A 70-billion-parameter model in FP16 consumes roughly 140 GB of GPU memory for weights alone. In INT4/FP4 quantization that drops to approximately 35 GB, though with some accuracy tradeoff. Knowing your model size and acceptable quantization level sets your minimum memory floor.
2. Maximum allowable latency For interactive user-facing applications, time-to-first-token (TTFT) should stay under 200 milliseconds and sustained output should reach at least 30 tokens per second (TPS) per request for a fluid user experience. For batch or offline jobs — document summarization, overnight analytics — you can relax both constraints and optimize purely for throughput.
3. Peak concurrent users / requests per second (RPS) Inference servers handle many requests simultaneously through continuous batching in frameworks like vLLM, TensorRT-LLM, and TGI. Estimate your peak RPS, then add a 30–40% overhead buffer so the server does not saturate during traffic spikes.
4. KV-cache memory headroom The key-value cache for a long-context model (128K tokens) can easily add tens of gigabytes of memory per active request. Multiply your expected context length by your peak concurrent-request count and add that to the model-weight footprint. If the total exceeds a single GPU's HBM, you need either multi-GPU tensor parallelism or a GPU with larger memory.
Mapping Model Scale to GPU Tier
Not every workload needs an H200. Matching GPU tier to model size and latency class saves significant budget without sacrificing service quality.
| Model Scale | Example Models | Recommended GPU Tier | HPE Platform Fit |
|---|---|---|---|
| Sub-7B parameters | Llama 3.1-8B, Mistral 7B, Phi-3 mini | NVIDIA L4 (24 GB GDDR6) | HPE ProLiant DL380a Gen12 (up to 16× L4) |
| 7B–30B parameters | Llama 3.1-8B INT8, Mixtral 8x7B | NVIDIA L40S (48 GB GDDR6) | HPE ProLiant DL380a Gen12 (up to 10× L40S) |
| 30B–70B parameters | Llama 3.1-70B, Falcon 40B | NVIDIA H100 NVL / H200 NVL (80–141 GB HBM) | HPE ProLiant DL380a Gen12, HPE Cray XD670 |
| 70B–405B parameters | Llama 3.1-405B, GPT-class frontier | NVIDIA H200 SXM, B300 HGX | HPE Cray XD670, HPE Compute XD690 |
| Frontier / multimodal | Large vision-language, reasoning models | NVIDIA Blackwell Ultra (B300) | HPE Compute XD690 |
The L4 is purpose-built for cost-efficient inference of smaller models and handles NVIDIA's INT8 Tensor Cores well. The L40S adds GDDR6X memory and FP8 Tensor Cores, making it the workhorse for mid-size models where NVLink is not required. Step up to H100 NVL or H200 NVL when model sharding or long-context windows push memory requirements above a single L40S.
HPE Platforms Built for AI Inference
HPE offers a tiered portfolio that matches the model-scale table above. Here is how the current lineup lines up:
HPE ProLiant DL380a Gen12 The flagship enterprise inference node for organizations that need density without a dedicated AI supercomputer. The 4U chassis supports up to sixteen single-wide NVIDIA L4 GPUs or up to ten double-wide cards (L40S, H100 NVL, H200 NVL, or RTX PRO 6000 Blackwell Server Edition). It runs dual Intel Xeon 6 processors with up to 144 cores and PCIe Gen5 I/O. In MLPerf Inference v5.1 benchmarks, HPE ProLiant Compute Gen12 systems achieved seven number-one rankings, including top results for DLRM-v2 recommendation workloads and competitive Llama 3.1-8B server-scenario throughput among eight-GPU PCIe systems. This is the right platform for enterprises that want a standard rack server they can procure through existing IT channels without a specialized data-center build-out.
HPE Cray XD670 A 5U chassis housing eight NVIDIA H100 or H200 SXM Tensor Core GPUs in an HGX configuration with NVLink interconnect. Minimum system memory is 1 TB for H100 configurations and 2 TB for H200 configurations. The XD670 is the logical choice when model parallelism across multiple GPUs is required — for example, a Llama 3.1-70B deployment where no single GPU has enough HBM — or when you need the 4.8 TB/s memory bandwidth of H200 HBM3e to hit aggressive per-token latency targets for large models.
HPE Compute XD690 The newest Blackwell-generation node, shipping from October 2025, featuring eight NVIDIA Blackwell Ultra (B300 HGX) GPUs in a 10U air-cooled chassis with 12 redundant 3,200-watt power supplies. The B300 delivers 50% higher FP4 performance and double the GPU-to-CPU bandwidth compared to the prior Blackwell generation, making it the right call for organizations deploying frontier models at scale or requiring FP4 inference for maximum token throughput.
NVIDIA RTX PRO 6000 Blackwell Server Edition in ProLiant For organizations with smaller model footprints but demanding multi-tenant inference workloads — healthcare imaging, legal document review, enterprise knowledge management — the RTX PRO 6000 Blackwell Server Edition offers 96 GB of GDDR7 memory per card and up to 6× inference performance improvement compared with the L40S. The DL380a Gen12 supports up to eight of these cards, making it an excellent value platform for agentic AI pipelines where multiple specialized models run concurrently. You can explore the full range of HPE servers available from Uniqcli to compare configuration options.
Latency vs. Throughput: Choosing Your Optimization Target
Many teams conflate latency and throughput, then size incorrectly for both. They are distinct and sometimes competing objectives.
Latency-optimized sizing minimizes TTFT and per-token decode time. For this mode:
- Use fewer GPUs with higher per-GPU bandwidth (H200 SXM outperforms two H100 PCIe cards in latency even though aggregate FLOPS may be comparable).
- Keep batch sizes small (1–4 concurrent requests per GPU context).
- Use NVLink-connected GPU configs to avoid PCIe bottlenecks when model sharding is needed.
- Target GPU memory utilization at 70–75% to leave headroom for KV-cache growth.
Throughput-optimized sizing maximizes tokens generated per second across all concurrent users. For this mode:
- Use continuous batching frameworks (vLLM, TRT-LLM) to pack more requests per GPU.
- Allow larger batch sizes; the GPU becomes more compute-efficient as batch size rises.
- Add GPUs horizontally rather than paying for higher-bandwidth single GPUs.
- Quantization to FP8 or INT4/FP4 meaningfully improves throughput with modest accuracy loss for most production tasks.
For regulated sectors like federal and healthcare — where Uniqcli specializes — latency guarantees are often contractually specified. Build your sizing scenario around the worst-case TTFT SLA first, then validate throughput capacity for your peak concurrent-user count. You can review Uniqcli's broader guides library for sector-specific deployment patterns.
Networking, Storage, and Cooling Considerations
GPU server sizing does not stop at the GPU. Three supporting systems can become bottlenecks that nullify expensive GPU investment:
Networking Large inference clusters that disaggregate model weights from compute — or that use speculative decoding across multiple nodes — need low-latency, high-bandwidth interconnects. NVIDIA Quantum-2 InfiniBand or NVIDIA Spectrum-X Ethernet are the reference fabrics for HPE AI Factory configurations. HPE Aruba Networking CX switches provide the data-center spine when InfiniBand is not required for smaller deployments. Insufficient network bandwidth between nodes in a tensor-parallel inference cluster will cap your effective throughput regardless of GPU count.
Storage Model checkpoints for a 70B parameter model in FP16 occupy roughly 140 GB on disk; loading this into GPU memory at startup takes minutes if served from spinning disk. NVMe-backed local storage or a high-throughput all-flash shared storage tier (HPE Alletra or comparable) significantly reduces cold-start time and supports fast model-swapping in multi-model serving scenarios.
Cooling and power The H200 HGX configuration in the Cray XD670 requires 6× 3,600-watt titanium power supplies per node. The B300-based XD690 draws 12× 3,200-watt supplies. Before finalizing a server count, audit your facility for power distribution capacity, available circuit amperage, and cooling BTU headroom. HPE offers direct liquid cooling (DLC) options on the XD670 to reduce heat rejection to the room, which matters significantly in co-location or government facilities with constrained CRAC capacity. Plan rack power density before you submit a purchase order, not after.
Quantization and Software Stack: The Multiplier You Control
Hardware is only half the equation. The inference software stack and quantization strategy dramatically affect how much useful work a given GPU delivers.
- FP16 is the baseline precision for most deployed models. It preserves accuracy but requires full memory footprint.
- FP8 (supported on H100 and newer) typically reduces memory footprint by roughly half versus FP16 with minimal accuracy regression on most transformer architectures.
- INT4 / FP4 (supported natively on Blackwell B300) can push throughput 2–4× versus FP16 on the same hardware, with accuracy tradeoffs that vary by model and task. HPE ProLiant DL380a Gen12 with RTX PRO 6000 Blackwell Server Edition supports FP4 inference natively.
- Speculative decoding uses a small draft model to propose tokens that a larger model verifies in parallel, reducing effective latency for autoregressive generation without requiring additional training.
NVIDIA TensorRT-LLM is the recommended serving engine for NVIDIA GPUs and integrates directly with HPE-validated configurations. vLLM is the dominant open-source alternative with broad model coverage. Both support paged KV-cache management, which is the single most important software feature for reducing memory waste in high-concurrency inference deployments. Request a configuration consultation to discuss which stack best fits your model and latency requirements before committing to hardware.
A Practical Sizing Checklist
Use this checklist when preparing an inference sizing request:
- Model parameters and precision: What is the model family, parameter count, and acceptable quantization (FP16, FP8, INT4)?
- Context length: What is the maximum input + output context window the application requires?
- Peak RPS: What is the expected peak requests per second at busiest hour?
- Latency SLA: Is there a contractual or product requirement for TTFT and sustained TPS?
- Throughput target: What aggregate tokens-per-second does the application require at peak load?
- Redundancy requirement: Is N+1 failover needed? Is the workload mission-critical?
- Data residency: Must inference run on-premises (federal, healthcare, sovereign AI) or is cloud burst acceptable?
- Facility constraints: What is available rack power (kW per rack) and cooling capacity?
- Growth horizon: What is the expected model footprint and user load 18–24 months out?
Completing this checklist before engaging a reseller saves weeks of back-and-forth and prevents the common mistake of sizing for today's smallest model rather than the next generation.
How Uniqcli Helps
Uniqcli is an authorized HPE and HPE Aruba Networking partner with deep experience sizing infrastructure for federal agencies, SLED institutions, healthcare systems, and enterprise organizations. Our team has worked through inference deployments across the full HPE AI server portfolio — from ProLiant DL380a Gen12 nodes running RTX PRO 6000 Blackwell cards for departmental AI to multi-node Cray XD670 and XD690 clusters for large-scale production inference.
We do not push a single platform. We walk through your workload parameters, model roadmap, facility constraints, and budget before recommending a configuration — and we back that recommendation with HPE's validated reference architectures and benchmark data. If you are ready to size a deployment or want a second opinion on an existing spec, request a quote or reach out to our team directly. We are also happy to connect you with HPE specialists for complex federal or sovereign AI deployments.