Adaptive DHiSparse Makes KV Residency a Runtime Decision

Long-context inference can reach a memory limit before it reaches useful GPU throughput. The accumulated key-value cache competes for GPU capacity, and every decoding step may require access to a substantial history. For infrastructure teams, that creates a difficult operating constraint: available compute is not enough if the serving system cannot manage the KV state needed to use it.[2]

Adaptive DHiSparse addresses that constraint by combining hierarchical sparse KV storage, load-adaptive GPU residency, and speculative decoding. Its paper, Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy, was submitted to arXiv on October 9, 2026. Rather than assigning the cache one fixed GPU budget, the system changes how much history remains resident as load changes.[2]

The reported results include 793 output tokens per second per decode GPU for GLM-5.3 with DFlash2 at 100K context and concurrency 128. A separate evaluation at 500K context and concurrency 512 reports 1.85× the output of resident serving when both systems use DSpark.[2]

Those results warrant attention, but the more useful engineering lesson is architectural: KV residency, retrieval, and speculation need to be evaluated together, under explicit concurrency and latency constraints.

1. The bottleneck is capacity as well as traffic

The paper states its central deployment problem directly: “Long-context serving runs out of KV memory before the hardware reaches its useful throughput.” That framing distinguishes a serving-capacity problem from a simple shortage of arithmetic throughput. Adding compute does not necessarily address the limiting resource when the system cannot keep enough relevant cache data close to the accelerator.[2]

Long-context decoding also creates a memory-traffic problem. Each generated token may require access to a large accumulated KV cache. A serving design therefore has to address both where that history lives and how much of it must move during decoding. Treating placement as a fixed configuration leaves less room to respond when request concurrency changes.[2]

Adaptive DHiSparse builds on HiSparse, which stores history in host memory and fetches selected entries. That hierarchy avoids requiring the entire history to remain on the GPU. However, the paper identifies two limitations in HiSparse: it uses the same cache budget at every system load, and it does not support draft-model speculative decoding systems such as DSpark and DFlash2.[2]

These limitations concern different parts of the serving path, but they intersect operationally. A fixed cache budget can leave available memory unused at low load, while speculation introduces another mechanism whose usefulness depends on cache access and data movement. Adaptive DHiSparse changes the placement policy and integrates speculation with sparse retrieval rather than treating either as an independent optimization.[2]

The resulting question is not simply whether the cache fits. It is how much history should remain resident at the current load, and what output the system can sustain within its latency requirement.

2. A hierarchy with protected and optional residency

Adaptive DHiSparse divides KV storage between GPU memory and host memory. Host memory provides the larger backing tier, while GPU memory holds the required working set and, when capacity permits, additional history. Entries that are not resident can be obtained through fused exact retrieval when needed.[2]

The distinction between required and optional residency is central. The required working set is protected. Optional entries occupy available GPU capacity, but the system can release them under pressure. This gives residency a different role from a permanently reserved cache allocation: it becomes a runtime control variable that responds to load.[2]

At low load, retaining more history can reduce host fetches. At higher load, releasing optional entries makes the memory available without abandoning the protected working set. The system therefore does not have to choose one static cache footprint intended to cover every concurrency level.[2]

The paper reports a concrete example during a load step. Optional residency reduced host fetches from 14.2% to 1.4% of selected rows at low load, while preserving the ability to release memory under pressure.[2] The denominator matters: these percentages describe selected rows, not the entire stored history, total memory traffic, or end-to-end latency.

That result supports a specific interpretation. Spare GPU memory can be useful even when it is not necessary to hold the required working set, because retaining additional history can avoid subsequent retrieval. Conversely, that extra residency need not become a permanent commitment that prevents the system from accommodating greater pressure.[2]

For operators, the implication is to examine residency behavior across changing load, not only at one steady-state concurrency. A single snapshot would not capture both the low-load retrieval benefit and the system's ability to relinquish optional memory.

3. Speculation becomes part of the cache design

Speculative decoding uses a draft model to propose multiple tokens and a target model to verify them. Adaptive DHiSparse supports this process with DFlash2 and DSpark while the target model accesses selected KV entries through a memory hierarchy.[2]

The integration matters because speculation and cache management affect the same serving path. Draft proposals can increase useful output, but their verification still interacts with memory movement and cache access. A hierarchy that reduces GPU residency requirements must therefore remain useful when serving includes speculative work.[2]

Adaptive DHiSparse addresses two coupled mechanisms: reducing the KV data that must stay on the GPU and increasing output through draft-model proposals. Its contribution is not merely placing these mechanisms side by side. The paper explicitly evaluates speculative decoding over the load-adaptive sparse hierarchy, addressing a capability missing from HiSparse.[2]

The comparison baselines help distinguish what the evaluations establish. Comparing Adaptive DHiSparse with HiSparse measures the improvement over the earlier sparse-cache serving approach. Comparing it with non-speculative adaptive serving more directly examines the contribution associated with speculation inside the adaptive setup.[2]

A third comparison, against resident serving that also uses DSpark, holds the presence of speculation constant while changing the KV-management approach. These comparisons answer related but different questions. Their output multipliers should not be multiplied together or presented as interchangeable measures of one universal gain.[2]

For infrastructure evaluation, this is a useful discipline: identify which mechanism changes between configurations before interpreting the result. An improvement over a non-speculative baseline means something different from an improvement over a baseline that already uses the same speculative system.

4. What the 100K-context results establish

For GLM-5.3 with DFlash2 at 100K context and concurrency 128, Adaptive DHiSparse achieved 793 output tokens per second per decode GPU. The paper reports that output as 1.45× HiSparse and 1.27× non-speculative adaptive serving.[2]

These are configuration-specific measurements. The model, speculative system, context length, concurrency, and throughput denominator all belong with the result. In particular, output tokens per second per decode GPU is not the same metric as single-request generation speed or an unspecified cluster-wide throughput figure.

Reported measurement	Result
GLM-5.3 with DFlash2, 100K context, concurrency 128	793 output tokens per second per decode GPU
Output relative to HiSparse at that configuration	1.45×
Output relative to non-speculative adaptive serving	1.27×
Concurrency under the same latency bounds as HiSparse	Up to 4×
Output under those latency-bound comparisons	2.3× to 3.7×

All measurements in the table are reported by the Adaptive DHiSparse paper.[2]

The latency-bound comparisons are particularly relevant to serving teams. When held to the same latency bounds as HiSparse, Adaptive DHiSparse sustained up to four times the concurrency and 2.3× to 3.7× the output, depending on the per-token latency bound.[2]

That dependence is part of the finding, not a detail to remove from a headline. The system does not have one throughput multiplier that applies independently of its latency constraint. Different bounds produce different reported advantages.[2]

The results support evaluating useful output at an acceptable latency rather than maximizing an unconstrained throughput number. They also keep concurrency in view: the ability to serve more simultaneous requests within a latency bound is distinct from making one request faster. Infrastructure buyers should preserve those distinctions when comparing serving systems.

5. The 500K test asks a different baseline question

The second reported configuration uses DeepSeek-V4.1-Flash with DSpark at 500K context and concurrency 512. Adaptive DHiSparse delivered 1.85× the output of resident serving with DSpark.[2]

This extends the evaluation to a longer context and higher concurrency than the GLM-5.3 experiment. More importantly, the baseline also uses speculation. The result therefore cannot be explained simply as adding a draft model to a non-speculative serving system. Both configurations use DSpark, while Adaptive DHiSparse changes how KV state is managed.[2]

That makes the comparison relevant to teams already using speculative decoding. It demonstrates a reported benefit from the load-adaptive hierarchy relative to resident serving in the tested configuration, rather than only a benefit from introducing speculation itself.[2]

It does not, however, establish a scaling curve between 100K and 500K contexts. The model, speculative system, concurrency, and comparison baseline differ across the two evaluations. Reading the results as a controlled context-length sweep would attribute more to the evidence than the reported configurations support.

The same caution applies to general deployment claims. The available results identify specific model and speculative-decoding combinations. They do not establish superiority for every model, hardware platform, context length, or workload.[2]

For an infrastructure buyer, the practical use of this result is as a focused evaluation hypothesis: test whether an adaptive KV hierarchy improves output over the existing resident speculative-serving path at the context lengths and concurrency levels that matter. The reported 1.85× result is evidence for that tested configuration, not a guaranteed uplift elsewhere.

6. How this differs from adjacent optimization work

Other papers submitted during the same period target different parts of inference efficiency. Their results provide context, but they should not be treated as directly comparable serving benchmarks.

Read What Matters: Query-Adaptive Quantization for KV Caches, submitted October 8, 2026, introduces ReadKV. It uses a progressive code that allows stored keys and values to be reconstructed at different precisions depending on the query.[8] Its central control is read precision, whereas Adaptive DHiSparse focuses on runtime residency and retrieval within a hierarchy.[2][8]

Across six base models, ReadKV reports that reading four bits on average from an eight-bit cache increased C4 perplexity by at most 0.66%. It used about one quarter of the logical reads and half the retained capacity of a 16-bit cache.[8]

ReadKV also reports 39% lower latency than the tested TurboQuant codec on an NVIDIA A10G, using a restricted eight-bit reader with a two-bit mean payload-read budget. That measurement is for an 8K-token, batch-one, single-layer workload.[8] It should not be equated with Adaptive DHiSparse's long-context serving output under concurrency constraints.

When Lower Reconstruction Loss Hurts, also submitted October 8, introduces Distributionally Robust Quantization. This post-hoc refinement minimizes worst-case reconstruction loss over a constrained set of input activation distributions.[7] It addresses low-bit quantization rather than load-adaptive KV placement.

SSR: Sparse Segment Reduction for Ternary GEMM Acceleration, submitted October 6, reports 2.1× to 11.3× speedups over RSR++ for ternary GEMM at 45% to 95% sparsity. It also reports 3.5× to 6.3× end-to-end speedups and 4.9% memory saving for Llama-3 1B inference.[3]

The distinction is architectural: these papers address query-dependent precision, quantization refinement, and ternary computation. Adaptive DHiSparse addresses the interaction of long-context KV state, memory placement, concurrency, and speculation. The brief does not establish combined gains from deploying these techniques together.[2][3][7][8]

7. What this means for teams running their own inference infrastructure

Adaptive DHiSparse suggests that GPU cache residency should be evaluated as an operating policy, not only as a capacity setting. Its required-versus-optional distinction allows the system to use spare memory at low load and release optional history when pressure rises.[2]

For teams evaluating this approach, the reported experiments point to four quantities that should remain visible throughout testing:

• Context length: Preserve the context configuration alongside every result, including whether it resembles the reported 100K or 500K evaluations.
• Request concurrency: Measure behavior as load changes, rather than assuming one cache allocation remains appropriate everywhere.
• Per-token latency bounds: Compare output under the same bound. The reported 2.3× to 3.7× output range depends on that constraint.
• Host-fetch activity: Track retrieval alongside output. The reported reduction from 14.2% to 1.4% concerns selected rows and helps explain the benefit of optional residency.[2]

Baseline selection is equally important. A comparison against resident serving with speculation asks whether the hierarchy adds value to an already speculative system. A comparison against non-speculative adaptive serving asks what speculation contributes within the adaptive design. Neither replaces the other.[2]

For purchasing decisions, the evidence supports a targeted benchmark, not an unconditional throughput or cost claim. The reported results do not establish a universal performance multiplier or an infrastructure cost reduction. Teams should test the relevant model, speculative path, context distribution, and latency requirement before translating an output gain into a deployment decision.

The broader lesson is clear: long-context serving performance depends on how effectively a system manages KV state as demand changes. Adaptive DHiSparse makes that management dynamic and integrates it with speculative decoding. For self-hosted inference teams, its value is a concrete demonstration that residency policy can be part of the throughput strategy, provided the gains are measured against the right baseline and within the service's latency constraints.[2]