BLIS: Planning LLM Serving Capacity Before Committing GPUs

Buying more GPUs is one response to an inference service that misses its latency targets. It is not necessarily the right response. Requests can spend time waiting for admission, competing for prefill capacity, following an unbalanced route, or contending with cache constraints. Additional compute does not, by itself, establish which of those problems determines serving performance.[1]

BLIS approaches that uncertainty as a simulation problem. Described in Simulate Before You Scale: High-Fidelity LLM Inference Planning Without GPUs, it is a CPU-only, discrete-event simulator that models the serving pipeline without requiring access to the target GPUs. Its scope includes admission, routing, scheduling, prefill, decode, KV-cache management, and preemption.[1]

The reported results connect prediction accuracy to deployment decisions: 6.7% median end-to-end error across 36 experiments, execution approximately 200 times faster than real runs, and a search that identified a four-GPU Llama-2-70B deployment meeting a 300 ms mean time-to-first-token objective.[1] For infrastructure teams, the important question is how to use that evidence without treating simulation as a substitute for measurement.

1. Model the serving system, not just the GPU

A kernel benchmark answers a bounded question about execution performance. Capacity planning asks a broader one: can a particular combination of hardware, parallelism, and serving policies handle a workload while meeting its latency objective? BLIS treats those choices as a joint configuration problem rather than assuming that faster individual operations necessarily produce a better deployment.[1]

That distinction matters because a request passes through several interacting controls. Admission determines whether work enters the system. Routing determines where it goes. Scheduling determines when it receives execution resources. Prefill processes the prompt, while decode produces output tokens incrementally. KV-cache management and preemption affect the system's ability to keep active requests resident and progressing.[1]

BLIS represents these stages in a discrete-event simulation. Its value is not simply that it includes more components than an isolated model test. It is that it models components whose interactions can change the outcome. Request concurrency, cache capacity, and preemption can combine to reduce decode throughput under load. Admission delays, routing imbalance, and prefill contention can contribute to high time to first token.[1]

The simulator is also deterministic: repeated simulations of the same configuration produce reproducible results. Because it runs on CPUs, evaluating candidate deployment policies does not require occupying production accelerators or constructing every candidate setup on physical hardware.[1]

For an engineering team, this suggests a useful separation of tasks. Simulation can support comparisons among candidate system designs. Hardware testing can then determine whether the selected design actually meets its requirements. The first task narrows uncertainty; the second remains necessary to resolve it.

2. Interpret speed and accuracy together

The BLIS evaluation covered six models, three GPU types, and 36 experiments. Across that evaluation, the simulator achieved a 6.7% median end-to-end error relative to real execution and ran approximately 200 times faster.[1] Neither figure is as useful alone as the combination.

Speed makes a broader search practical. When evaluating a candidate is much faster than running it on the target system, teams can compare more configurations before committing accelerator time. Accuracy determines whether those comparisons provide meaningful guidance. A simulator that runs quickly but misrepresents serving behavior could simply accelerate the selection of an unsuitable configuration.

The reported median error supports BLIS as a planning tool, but it does not establish a maximum error. It also does not mean that every configuration will fall within 6.7% of measured behavior. Teams should therefore avoid treating that number as a guaranteed tolerance around every prediction. The brief reports a median across the evaluated experiments, not a universal bound.[1]

The evaluation's scope also matters. Six models and three GPU types provide evidence across multiple settings, but they do not demonstrate equivalent accuracy for every model family, accelerator generation, context length, serving framework, or workload distribution.[1] A proposed deployment outside the tested conditions still needs its own validation.

The practical interpretation is straightforward: use simulation to eliminate weak candidates and identify promising ones, then measure those candidates on hardware. A prediction close to an SLO boundary deserves particular scrutiny because prediction error could change the deployment decision. That is a reason to validate carefully, not a reason to discard simulation's search advantage.

3. What the four-GPU result actually establishes

The paper's Llama-2-70B configuration search is its clearest capacity-planning example. BLIS evaluated 420 serving configurations and identified a four-GPU deployment that met a 300 ms mean time-to-first-token SLO. According to the paper, competing tools either required twice as many GPUs or recommended configurations that failed under load.[1]

This is more informative than predicting latency for a single, already selected deployment. The simulator was used to choose among hundreds of alternatives, and the resulting choice connected resource count to a defined service objective. In that reported search, BLIS found a configuration that competing approaches did not match successfully at the same GPU count.[1]

The qualification is important. This does not establish that BLIS can halve GPU requirements for every model or service. It does not show that four GPUs are sufficient for arbitrary Llama-2-70B traffic. The supported conclusion concerns the paper's reported search and its stated latency target.[1]

The metric also needs to remain attached to the result. A 300 ms mean TTFT objective concerns average time to first token. It is not a P90 latency guarantee, a full-response latency target, or a general statement about throughput. Replacing that precise objective with a broader claim about serving performance would overstate the evidence.

For infrastructure buyers, the lesson is to compare deployment configurations against the same workload and service objective, rather than comparing GPU counts in isolation. Parallelism, scheduling, routing, and admission behavior can materially affect serving capacity. BLIS includes those dimensions because hardware quantity alone does not describe how effectively a system turns its resources into acceptable request latency.[1]

4. Simulation can discover policies, not just size deployments

BLIS was also used to evolve serving policies entirely in simulation. The paper reports that two policies were subsequently validated on real hardware and merged into llm-d, an open-source platform for distributed LLM serving.[1] This extends the simulator's role beyond choosing hardware: it can help explore changes to how the serving system operates.

The first result concerns admission control. A simulated admission policy reduced critical-tier P90 time to first token by up to 82%.[1] P90 describes the latency below which 90% of requests fall. The result therefore concerns tail behavior for a specified traffic tier, not an equivalent reduction in average latency across all requests.

Admission control matters because allowing work into the system changes which requests compete for resources. The result shows that this control can materially protect latency-sensitive traffic under contention. It does not establish that every workload, traffic tier, or admission policy will receive the same benefit.[1]

The second result concerns flow control. A policy developed in simulation reduced end-to-end P50 latency by 32% to 48% on overloaded workloads.[1] P50 is the median, so this result describes an improvement in typical full-request latency under overload. It should not be conflated with the admission policy's critical-tier P90 TTFT improvement.

Together, the results show why serving optimization cannot be reduced to model execution speed. The underlying model and GPUs can remain unchanged while policies alter user-visible latency. The subsequent hardware validation is particularly important: these were not only favorable simulated outcomes.[1] For operators, the evidence supports testing admission and flow-control behavior as explicit optimization variables, while retaining the workload qualifications attached to the reported gains.

5. Cache behavior makes fixed benchmarks incomplete

Traditional inference benchmarks commonly report tokens per second, time to first token, or inter-token latency for a fixed model, hardware configuration, and workload. Those measurements remain useful. BLIS addresses a different question: how performance changes when multiple system-level choices interact.[1]

Mixed serving workloads make that question difficult. Prefill and decode represent different stages of request execution, and requests at those stages compete for compute, memory, and scheduling opportunities. A configuration that performs well for one batch size or prompt-length distribution may fail when the workload changes.[1] A fixed benchmark therefore cannot, on its own, characterize every operating condition relevant to a deployment.

KV-cache behavior adds another constraint. The cache stores attention-related state for active requests. Its capacity and movement affect how many requests can remain resident and how often the system must preempt or otherwise manage them. BLIS explicitly models both KV-cache management and request preemption.[1]

That inclusion helps explain why nominal compute capacity is an incomplete planning signal. A system may have sufficient theoretical compute yet miss its latency objective because cache state, scheduling, or request placement prevents work from progressing effectively.[1] The useful question is not only how quickly the GPU can execute an operation, but whether the serving system can keep the right requests moving through the pipeline.

The implication is not to replace benchmarking with simulation. Instead, teams can use each at the appropriate level. Benchmarks characterize measured behavior under defined conditions. Simulation explores how candidate configurations might behave as those conditions and control policies change. Hardware validation then tests whether the chosen combination delivers the expected service outcome.

6. Build a predict, search, and validate workflow

The research supports a planning workflow built around simulation followed by empirical validation. The following sequence is a practical interpretation of the reported results, not a claim that every deployment will reproduce the paper's gains.

Define the objective before comparing configurations. Specify whether the decision depends on mean TTFT, critical-tier P90 TTFT, median end-to-end latency, or another measured requirement. The BLIS results use different metrics for different experiments. Keeping those distinctions explicit prevents a configuration from appearing successful against an objective it was never evaluated to meet.[1]

Treat serving policies as configuration variables. Hardware layout and parallelism belong in the search, but so do scheduling, admission, routing, cache behavior, and preemption. BLIS's scope reflects the fact that these choices interact.[1] Holding all policies fixed while changing only GPU count can leave relevant alternatives unexplored.

Use simulation to produce a shortlist. The reported execution advantage makes broad comparison more practical, while deterministic behavior supports reproducible runs.[1] A useful planning output is a set of promising candidates with explicit objectives and workload assumptions, rather than an unqualified declaration that one configuration is universally optimal.

Validate the selected candidates on hardware. The 6.7% median prediction error makes this step essential. Validation should check the actual metric driving the decision and the load conditions under which the service must operate. The paper's comparison included configurations recommended by competing tools that failed under load, illustrating why a recommendation alone is insufficient.[1]

Revisit the plan as conditions change. Workload-dependent results should not become permanent guarantees. If the workload changes, teams should reconsider whether the selected admission, routing, scheduling, and flow-control behavior still serves the original objective. Simulation is most useful as part of an ongoing planning process, not merely as a one-time sizing exercise.

7. What this means for teams running their own inference infrastructure

For teams operating their own inference stack, BLIS shifts the planning conversation from accelerator count to deployment behavior. The research suggests that hardware allocation and serving-policy design should be evaluated together, before a team assumes that a latency problem requires more GPUs.[1]

The strongest evidence is specific: a 420-configuration Llama-2-70B search produced a four-GPU deployment meeting a 300 ms mean TTFT SLO, and two policies developed in simulation were validated on hardware and integrated into llm-d.[1] Those results connect simulation to both capacity selection and operational control. They do not establish universal GPU savings or fixed latency improvements.

For engineers, the opportunity is to investigate more candidate configurations without reproducing each one on accelerators. For infrastructure buyers, it is to ask for a clearer relationship among requested GPU capacity, workload assumptions, serving policies, and the service objective. A resource proposal becomes more useful when it explains not just what hardware is needed, but why that configuration is expected to meet the target.

The boundary remains physical validation. BLIS's reported accuracy and speed make it a way to narrow the search, not a replacement for running the chosen system. Its central lesson is a disciplined sequence: model the complete serving pipeline, compare configurations and policies, validate the strongest candidates, and revise the plan when operating conditions change.[1] That approach makes GPU allocation a more informed systems decision rather than a proxy for unresolved serving behavior.