ALL INSIGHTS

LLM Enhancements

FOSCAR Makes the Case for Low-Bit KV Caches as a Decode Optimization

FOSCAR pairs low-bit KV caches with specialized attention kernels, reporting H100 speedups and higher long-context serving throughput.

November 9, 20269 min read
FOSCAR Makes the Case for Low-Bit KV Caches as a Decode Optimization

A smaller KV cache is not necessarily a faster KV cache. Reducing the precision of cached keys and values lowers the amount of data an inference system must store and move, but decoding still requires that compressed representation to participate in attention. Unpacking, reconstruction, scaling, and zero-point handling can consume the time saved on memory traffic. For infrastructure teams, that separates an attractive compression ratio from a useful performance optimization.[2]

FOSCAR, presented in the paper “FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding,” tackles that separation directly. Submitted on November 6, 2026, and updated on November 9, it presents an algorithm and kernel co-design for turning low-bit KV storage into lower decode-attention latency. Its evaluation includes both NVIDIA H100 kernels and integration into SGLang.[2]

The reported results are substantial: geometric-mean kernel speedups reaching 3.36× over FlashInfer BF16, plus peak aggregate serving throughput improvements reaching 170% over a 2-bit baseline in evaluated long-context workloads. Those are different measurements with different implications. The useful story is how FOSCAR connects cache representation, attention execution, and serving performance, and where the available evidence stops.[2]

1. Why fewer cache bits do not guarantee faster decoding

The KV cache grows with sequence length and batch concurrency. During autoregressive decoding, attention must read cached state, making that state an important source of memory traffic. Quantizing keys and values to 2 or 4 bits addresses the storage and transfer side of this problem.[2]

But the stored representation is only one part of the execution path. An attention implementation must unpack compressed data, reconstruct the numerical information needed for computation, and handle the associated scales and offsets. A reduction in bytes moved does not eliminate those operations. If their cost becomes dominant, compression can deliver memory savings without a corresponding latency reduction.[2]

FOSCAR explicitly identifies this implementation gap. Its contribution is not simply another low-bit cache format. It is a decode-attention design intended to make that format computationally useful, so the reduction in memory traffic survives the work required to consume the cache.[2]

This distinction matters when evaluating inference infrastructure. Memory capacity, attention-kernel latency, and serving throughput answer different questions. A cache may reduce memory requirements without accelerating its attention kernel. A faster attention kernel may still produce a smaller improvement at the serving level, where scheduling, batching, cache management, and token generation remain part of the execution path.[2]

FOSCAR evaluates both kernels and serving integration, which gives its performance claim more scope than a compression-only result. It does not, however, make the metrics interchangeable. The kernel measurements establish how its attention implementation compares with specified alternatives. The SGLang measurements test whether the approach also produces throughput gains in the evaluated long-context serving configurations.[2]

2. Affine folding targets the cost of using compressed data

The central technique is affine folding. Low-bit KV attention introduces work to recover quantized values and apply their affine scale and offset. FOSCAR aims to fold those transformations into the attention computation rather than treating dequantization as an independent preprocessing stage.[2]

The architectural objective is straightforward: avoid allowing the mechanics of the compressed representation to consume the benefit of moving fewer bytes. Instead of considering cache quantization and attention execution as separate engineering tasks, FOSCAR designs them together.[2]

That is an important shift in how to interpret low-bit performance. The nominal precision describes the stored data, but it does not fully describe the cost of using that data. Two implementations can both operate on a low-bit cache while taking different execution paths through unpacking, reconstruction, and attention. FOSCAR's reported improvements over a 2-bit serving baseline reinforce the importance of implementation beyond bit width alone.[2]

The available source record does not expose the complete mathematical derivation or all kernel details. It therefore does not support a detailed account of how each transformation maps to instructions, intermediate representations, or particular hardware resources. Affine folding should be understood here as the paper's reported design direction, not as a fully reconstructed implementation recipe.[2]

What the evidence does establish is the intended connection between mechanism and outcome: FOSCAR targets 2-bit and 4-bit KV caches, designs decode attention around them, and evaluates both kernel execution and serving integration. The contribution is the combined path, not a claim that reducing precision alone automatically accelerates decoding.[2]

3. The H100 results show gains against both BF16 and FP8

FOSCAR reports the following geometric-mean decode-attention kernel speedups on NVIDIA H100 hardware:[2]

FOSCAR kernelOver FlashInfer BF16Over TensorRT-LLM FP8
INT23.36×2.03×
INT42.83×1.71×

The FlashInfer comparison establishes a substantial improvement over the cited BF16 implementation. Because BF16 is a higher-bandwidth baseline than the low-bit formats under discussion, that result is consistent with the value of reducing cache traffic while controlling the overhead of consuming compressed data.[2]

The TensorRT-LLM comparison adds a more demanding reference point. FP8 already reduces precision and memory traffic relative to BF16. FOSCAR's reported advantage therefore is not limited to comparison with an uncompressed KV cache. Both its INT2 and INT4 kernels also outperform the cited FP8 implementation under the paper's evaluation conditions.[2]

The use of a geometric mean is significant. These numbers summarize performance across multiple evaluated cases rather than presenting a single best-case speedup. A geometric mean reduces the influence of one unusually favorable workload, making it a more useful aggregate than an isolated peak result.[2]

An aggregate still does not reveal the full distribution. The available record does not provide the complete benchmark matrix, including individual sequence lengths, head configurations, and attention shapes. It is therefore not possible to determine from these figures alone which cases benefited most, how much performance varied, or how closely the evaluated cases match a particular deployment.[2]

The defensible conclusion is strong but bounded: on H100, FOSCAR reports substantial geometric-mean attention-kernel speedups against both named baselines. These figures are not measurements of total model execution or request completion time, and they should not be presented as universal serving speedups.[2]

4. SGLang integration tests whether kernel gains survive serving

An isolated kernel result leaves an important systems question unanswered: does the improvement remain useful inside a serving runtime? FOSCAR addresses that question by integrating with SGLang, an open-source LLM serving framework.[2]

That integration matters because serving includes more than attention execution. Request scheduling, paged KV-cache management, batching, and token generation surround the kernel. Measuring within the framework tests whether the low-bit execution path produces a benefit despite those additional runtime costs.[2]

For evaluated long-context serving workloads, FOSCAR reports increases in peak aggregate throughput of up to 170% over a 2-bit baseline and up to 341% over a 16-bit baseline. The comparison points must remain attached to their respective numbers.[2]

A 170% increase corresponds arithmetically to 2.70 times the baseline throughput. A 341% increase corresponds to 4.41 times the baseline. Those conversions explain the reported percentages; they are not additional measurements. In particular, a 341% increase should not be described as 3.41 times the baseline.[2]

The 2-bit comparison is especially relevant to the paper's central argument. It shows a reported serving benefit against another low-bit configuration, rather than only against a higher-precision cache. That supports the importance of the execution path, although it does not isolate the contribution of every implementation difference.[2]

Long context is also a material condition, not a footnote. Longer contexts increase the cached attention state read during decoding, making reductions in bytes transferred potentially more consequential. The reported throughput gains apply to that evaluated setting.[2]

Finally, these are “up to” improvements in peak aggregate throughput. They are not geometric means like the kernel figures, and they do not establish the same improvement for every request or operating point. The available evidence supports a serving-level benefit, not a universal multiplier for deployment planning.[2]

5. Related work explores different parts of low-bit inference

FOSCAR sits alongside other efforts to make reduced precision useful across the inference stack. The distinctions matter because similar throughput language can hide different hardware, numerical formats, and optimization targets.

BitDecoding-FP4, titled “Native FP4 KV Cache Decoding via Algorithm-System Co-Design,” reports native FP4 KV decoding on NVIDIA B300 systems running Qwen3-4B and Qwen3-30B-A3B. It reports 2.4 to 3.0× higher peak serving throughput than BF16 KV. Like FOSCAR, it links a low-bit representation with an execution design rather than treating storage compression as the complete solution.[3]

It is not a direct performance ranking against FOSCAR. BitDecoding-FP4 uses FP4, different hardware, and different model configurations. Its throughput figures cannot be placed beside FOSCAR's H100 kernel speedups as though they measured the same workload or execution boundary.[2][3]

StrataFlux addresses a broader combination of memory hierarchy, weight quantization, and KV-cache quantization. It evaluates 2 to 4-bit weights and 2 to 3-bit KV configurations across dedicated GPUs and unified-memory platforms.[7]

For Llama-3.1-8B-Instruct with 2-bit weights on an RTX 5090, StrataFlux reports reducing WikiText-2 perplexity from 9.46 to 8.40 while retaining 95.4% of the quantized base throughput. At 128K context, its 3-bit-weight and 2-bit-KV configuration reportedly achieves comparable perplexity and speed to vLLM with 4-bit AWQ weights and 8-bit KV, using approximately half the GPU memory.[7]

Those results answer different questions from FOSCAR's decode-attention evaluation. Together, the papers illustrate several optimization targets: native low-bit execution, quality and memory tradeoffs, and attention-kernel speed. They should be evaluated on those terms rather than collapsed into a single low-bit leaderboard.[2][3][7]

6. What the available evidence does not establish

The research record provides FOSCAR's hardware, precisions, named kernel baselines, integration framework, and headline performance results. It does not provide enough experimental detail to reproduce the complete evaluation or establish deployment equivalence.[2]

Missing details include the exact number of evaluated models, context lengths, batch sizes, GPU memory configurations, and quality measurements. The available record also does not establish whether scheduler settings were identical across all serving baselines.[2]

These omissions constrain interpretation in specific ways. Without the workload matrix, teams cannot map the reported geometric mean onto their own attention shapes. Without serving configuration details, they cannot determine how much of the throughput difference depends on runtime choices. Without quality measurements, they cannot assess the numerical tradeoff associated with the reported low-bit configurations from this record alone.[2]

None of those limitations negates the reported results. They separate evidence that an approach works under evaluated conditions from evidence that it will deliver a particular benefit in another environment. FOSCAR provides the former; the available brief does not establish the latter.[2]

For technical evaluation, the appropriate response is to keep each claim attached to its measurement boundary. H100 attention-kernel speedups support conclusions about the cited kernel comparisons. SGLang peak aggregate throughput supports conclusions about the evaluated long-context serving configurations. Neither automatically establishes unchanged quality, identical gains across contexts, or a fixed reduction in infrastructure requirements.[2]

The strongest takeaway remains the implementation result: low-bit KV storage can become a decode-speed optimization when the attention path is designed around the representation. The exact benefit for a new deployment remains a validation question.[2]

7. What this means for teams running their own inference infrastructure

For teams operating their own inference stack, FOSCAR suggests evaluating KV quantization as an execution-path decision, not merely a cache-format decision. The relevant question is not only how many bits the cache uses, but how the serving system performs attention over those bits.[2]

An evaluation should preserve the distinctions in the paper. Measure cache memory requirements separately from attention latency and serving throughput. Compare against a relevant reduced-precision implementation where possible, rather than relying only on a BF16 reference. FOSCAR's comparisons against TensorRT-LLM FP8 and a 2-bit serving baseline show why those reference points are informative.[2]

Workload alignment should come before extrapolation. Because the serving gains are reported for long-context workloads, teams should establish how their own context lengths, concurrency, and runtime configuration affect the result. Quality also needs explicit validation, since the available FOSCAR record does not expose the measurements needed to resolve that question.[2]

Infrastructure buyers should likewise distinguish a promising engineering direction from a capacity forecast. A geometric-mean kernel speedup is not a guaranteed serving multiplier. An “up to” peak-throughput improvement is not a guaranteed improvement at every operating point. Deployment decisions need measurements at the boundary the team actually intends to optimize.

FOSCAR's contribution is nevertheless consequential. It connects a compressed KV representation, specialized GPU attention kernels, and a serving runtime, with reported gains at both the kernel and serving levels. The lesson is not that 2-bit caches are automatically fast. It is that compression becomes a performance optimization only when the computation path can exploit it.[2]

Sources

  1. [1][2611.05304] LoopVQ: Sub 1-bit KV Cache Quantization for ...
  2. [2]FOSCAR: Accelerating Decode Attention over 2- and 4-Bit KV Caches through Affine Folding
  3. [3]BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
  4. [4]Quantized large language models in biomedical natural language processing: evaluation and recommendation - npj Health Systems
  5. [5]MORTIS: Compiler-Guided KV-Cache Management for LLM Agent Programs
  6. [6][2611.07542] FlashOffloading: Low-Latency KV Cache Offloading ...
  7. [7]StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference
  8. [8]Asymptotic Optimality of cμ-Rule for LLM Inference Scheduling under Memory Constraint
  9. [9]ChainFlow: Few-Step Latent Flow Matching for Speculative Decoding
  10. [10]Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Researched from the sources above and fact-checked against them before publishing.

See what this means for your infrastructure.

Book a discovery call and get a benchmark projection for your hardware and model.

SCHEDULE A CALL