LLM Enhancements
FlashOffloading: Making Host DRAM Part of the Long-Context Inference Path
FlashOffloading extends KV-cache capacity into host DRAM, reporting higher throughput under a 30 ms per-token latency target.

Long-context inference can run out of memory capacity before it runs out of compute. As prompts grow and more requests run concurrently, their key-value (KV) caches consume accelerator high-bandwidth memory (HBM). The serving system can then lose the ability to admit additional work even while computational resources remain available. FlashOffloading addresses that mismatch by making host dynamic random-access memory (DRAM) the main KV-cache tier across both prefill and decode.[2]
The important question is not simply whether KV state can move off the accelerator. It is whether the serving system can use that additional capacity without allowing transfers and cache management to overwhelm its latency budget. FlashOffloading combines separate prefill and decode caches, a reverse sparse index, and a kernel that overlaps loading with computation.[2]
The reported outcome is promising but configuration-specific: up to 1.38× higher prefill throughput and 1.91× higher decode throughput under a 30 ms time-per-output-token (TPOT) service-level objective. Those results come from GLM-5.2 with MTP3 speculative decoding on an Ascend910C platform with disaggregated prefill and decode.[2] For infrastructure teams, the architectural lesson matters as much as the headline gains.
1. HBM capacity can limit serving before compute does
The KV cache holds key and value tensors generated for previously processed tokens. During autoregressive generation, the model reuses those tensors rather than recomputing the earlier sequence. This reuse is central to decoding, but it also creates a growing memory obligation: longer sequences require more KV state, and concurrent requests add to the total resident footprint.[2]
FlashOffloading identifies that growth as a constraint on serving concurrency. Once the cache reaches the capacity available in HBM, the system can become unable to support additional requests even though the accelerator still has computational resources available. In that situation, adding usable cache capacity is potentially a way to expose more work to the accelerator, not merely a way to accommodate a longer individual prompt.[2]
Offloading introduces a different constraint. KV state stored in host DRAM must be made available when computation needs it. Increasing the amount of addressable cache does not automatically increase useful throughput: transfers, lookup work, and scheduling overhead can consume the latency headroom that additional concurrency was supposed to exploit.[2]
This is why the evaluation's latency condition is important. A throughput improvement measured without a service constraint would leave open the possibility that the system simply accepted slower generation. FlashOffloading instead reports throughput under a 30 ms TPOT objective, tying the capacity expansion to a specified generation-latency target.[2]
For buyers and operators, the relevant framing is therefore capacity constrained serving under a latency budget. The system is not evidence that host memory universally replaces accelerator memory. It is evidence for a particular architecture that makes a larger memory tier useful in the tested configuration.
2. One offloading strategy, two phase-specific caches
FlashOffloading treats KV-cache management as an end-to-end serving problem. It fully offloads the KV cache across both prefill and decode, using host DRAM as the main storage tier while maintaining selective HBM residency. That distinction matters: full offloading does not mean eliminating accelerator-side caching. The architecture explicitly includes a rolling HBM cache for prefill and a separate hot cache for decode.[2]
During prefill, the system processes the prompt and produces KV entries. Requiring the complete prompt-derived cache to remain in HBM would preserve the capacity bottleneck for long inputs. FlashOffloading instead maintains a rolling subset in accelerator memory while the broader KV cache resides in host DRAM.[2]
This arrangement makes the prefill stage part of the offloading design rather than treating it as a preliminary step that must independently fit the entire cache into HBM. That is especially relevant to the evaluated disaggregated platform, where prefill and decode are separate serving stages.[2]
Decode has a different role. It repeatedly accesses KV state associated with active requests as generation proceeds. FlashOffloading uses a hot cache to retain immediately useful data in HBM, with the aim of reducing the frequency and cost of transfers from host memory.[2]
The two caches therefore serve different phases without abandoning a common hierarchy. Host DRAM provides the broader capacity tier, while HBM holds the subset intended to support the current computation efficiently. The design acknowledges that prompt processing and token generation do not have identical access characteristics.[2]
The available evidence does not expose enough implementation detail to reconstruct the rolling-cache policy or hot-cache behavior. It would be inappropriate to infer a particular replacement algorithm, residency threshold, or cache size. What is established is the division of responsibilities: phase-specific accelerator caching over a host-resident KV store.[2]
3. Lookup and transfer overhead determine whether capacity helps
A larger cache tier is useful only if the system can access it efficiently. FlashOffloading includes two mechanisms aimed at the overhead surrounding offloaded KV data: a reverse sparse index and a compute-load-overlapped kernel.[2]
The paper describes the reverse sparse index as providing zero extra miss-lookup overhead. The scope of that claim matters. It means that identifying offloaded KV entries on a cache miss does not require an additional lookup stage beyond the architecture's normal indexing path. It does not mean that a miss has no cost, or that retrieving data from host DRAM is instantaneous.[2]
That distinction separates metadata overhead from data movement. Avoiding an extra lookup stage can keep cache management from adding another dependency, but the KV data still needs to be available for computation. FlashOffloading addresses that second issue through its overlapping kernel.[2]
The compute-load-overlapped kernel overlaps KV loading with model computation. Its objective is to hide some transfer latency behind work that already occupies the accelerator, reducing the direct contribution of offloading to decode latency. The claimed mechanism is overlap, not the elimination of transfers.[2]
Taken together, the cache hierarchy, index, and kernel address different parts of the offloading problem. Cache organization controls which data remains in HBM. Indexing avoids an additional miss-lookup stage. Overlap reduces how much loading delays computation.[2]
The available results do not isolate the numerical contribution of each component. Teams should therefore evaluate FlashOffloading as an integrated serving architecture rather than attributing its reported throughput gain to the index, hot cache, or kernel alone.
4. What the throughput results actually establish
FlashOffloading's reported evaluation combines several specific choices: GLM-5.2, an Ascend910C accelerator platform, prefill and decode disaggregation, and MTP3 speculative decoding. MTP3 denotes a three-token multi-token-prediction configuration used as part of speculative generation.[2]
Under a 30 ms TPOT service-level objective, the reported maximum improvements are:[2]
| Serving phase | Reported maximum throughput improvement |
|---|---|
| Prefill | 1.38× |
| Decode | 1.91× |
TPOT measures the time required to produce output tokens during generation. Reporting throughput under this constraint makes the results more relevant to latency-sensitive serving than unconstrained throughput figures would be. The measurements indicate that the tested architecture can improve throughput while meeting the stated per-token target.[2]
However, the numbers are maximum improvements, not guarantees across all workloads. They do not establish the same gain for every prompt length, batch size, concurrency level, or cache-residency pattern. They also should not be interpreted as hardware-independent speedups.[2]
Decode shows the larger reported improvement. That is consistent with an architecture that includes a decode hot cache and overlapping loads, but the available source does not attribute the difference to a particular mechanism. Explaining the 1.91× result as the effect of one component would go beyond the evidence.[2]
The presence of speculative decoding is also significant. The results demonstrate KV offloading in a configuration that already uses MTP3, rather than presenting offloading and speculative generation as mutually exclusive approaches. They do not quantify how much improvement either technique contributes independently, or establish a general rule for combining their benefits.[2]
5. Host DRAM becomes an active serving tier
The central systems implication is a change in the role of host memory. In FlashOffloading, host DRAM is not merely a destination for infrequently accessed state. It is the main KV-cache tier, supported by selective HBM residency and mechanisms intended to keep offloaded data usable within the generation-latency budget.[2]
This makes cache capacity a property of the serving hierarchy rather than only the accelerator. The design aims to relieve the admission pressure created by growing KV state while continuing to use HBM where immediate access matters. Its value depends on coordinating those tiers, not simply on placing more data in host memory.[2]
That distinction helps explain why offloading must be assessed at the system level. A transfer mechanism alone cannot establish serving performance. Neither can a cache-capacity figure. Lookup behavior, transfers, cache residency, and scheduling overhead all influence whether additional stored context translates into useful work under a latency objective.[2]
Speculative decoding does not remove this need. Although it can reduce the number of expensive target-model decoding steps, long prompts and concurrent requests still require KV-state management. FlashOffloading's MTP3 evaluation shows the reported performance in a configuration that includes both speculative generation and host-DRAM offloading.[2]
For infrastructure planning, the architectural proposition is specific: some HBM-constrained deployments may benefit from treating host memory as an active extension of the KV-cache hierarchy. The brief does not establish a cost saving, an energy reduction, or a universal advantage over fully HBM-resident serving. Those remain separate questions requiring additional evidence.
6. The missing evidence matters for procurement and reproduction
The available record identifies the principal mechanisms and headline results, but it does not provide the complete experimental table, baseline implementation details, prompt-length distribution, host-memory bandwidth, DRAM capacity, or energy measurements. Those omissions limit both reproducibility and direct comparison with other cache-management approaches.[2]
Baseline details are particularly important when interpreting a relative throughput improvement. Without the implementation context, an infrastructure team cannot determine how closely the comparison resembles its own serving stack. The reported ratio is useful evidence about the tested system, but it is not a substitute for an equivalent local baseline.
Workload details are equally consequential. The reported maximum gains do not establish universal improvements across prompt lengths, batch sizes, concurrency levels, and residency patterns. A deployment decision should therefore test the workload characteristics that drive the team's own KV footprint and latency requirements, rather than relying on the highest reported ratio.[2]
The missing host-memory specifications also deserve attention. FlashOffloading relies on host DRAM as its main cache tier and on overlapping KV loading with computation. Without the reported bandwidth and capacity details, the available evidence is insufficient to reproduce that memory configuration or infer equivalent performance on another platform.[2]
Finally, the record does not support a direct ranking against alternative KV-cache compression, paging, or persistent-cache systems. Their relative merits would require comparable workloads, baselines, and service constraints. Nor does the throughput result establish an energy or economic outcome, since the necessary measurements are absent.[2]
These limitations do not negate the architectural contribution. They define its evidentiary boundary: an integrated offloading design with reported gains in one named configuration, not a general performance guarantee for host-memory KV storage.
7. What this means for teams running their own inference infrastructure
For self-managed inference teams, FlashOffloading suggests starting with a bottleneck diagnosis rather than an offloading purchase decision. If KV growth prevents additional requests from being admitted while accelerator compute remains available, the system addresses a relevant problem. If that is not the observed constraint, these results alone do not establish that offloading should be the next optimization.[2]
A practical evaluation should follow three priorities.
First, preserve the service objective. Compare throughput while enforcing the latency requirement that matters to the deployment. FlashOffloading's result is tied to a 30 ms TPOT objective. A local test should not assume that meeting that target establishes suitability for a different latency budget.[2]
Second, assess prefill and decode together. The contribution is broader than a decode-only cache extension. Its rolling prefill cache and decode hot cache address both phases, so an evaluation should examine whether capacity and data movement remain controlled throughout the serving path.[2]
Third, make configuration differences explicit. The reported measurements use GLM-5.2, Ascend910C, MTP3, and disaggregated prefill and decode. Changes to the model, platform, speculative-decoding configuration, or workload require validation rather than an assumed transfer of the published speedup.[2]
Before using the headline gains in capacity or procurement plans, teams should also request the missing baseline, workload, and host-memory details. Those are necessary inputs for determining whether the architecture fits the intended deployment and for making a defensible comparison with alternatives.
FlashOffloading's strongest lesson is architectural: KV-cache capacity, lookup overhead, and data movement need to be managed as one serving problem. Host DRAM becomes useful not simply because it can hold offloaded state, but because the surrounding cache and execution design can make that state available within a latency budget. For teams whose primary constraint is HBM capacity rather than raw compute, that is a concrete direction to investigate, with the reported gains serving as motivation for validation rather than a forecast.[2]
Sources
- [1]Probe and Skip: Self-Predictive Token Skipping for Efficient Long-Context LLM Inference | Proceedings of the 34th ACM International Conference on Multimedia
- [2]FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving
- [3]Supermicro, IBM, and NVIDIA KV Cache Solution Cuts ...
- [4]BasisKV: Value Latents for Communication and Key Routing
- [5]TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference
- [6]Asymptotic Optimality of cμ-Rule for LLM Inference Scheduling under Memory Constraint
- [7]Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode
- [8]Quality-aware cost minimization for multi-tier LLM routing in ...
- [9]Characterizing KV-Cache Performance in Large Language ...
- [10]Learning a Single Token to Replace Long System Prompts in LLMs
Researched from the sources above and fact-checked against them before publishing.
See what this means for your infrastructure.
Book a discovery call and get a benchmark projection for your hardware and model.
SCHEDULE A CALL