Insights / LLM Enhancements
Research on making models do more with the same hardware.
New articles are published regularly, researched from current papers, releases and industry reporting, then fact-checked against their sources.

MosaicKV and the Case for Scheduling KV Cache Across Tenants
MosaicKV coordinates KV-cache placement, tenant memory allocation, and prefetching. Its architecture is clear; performance gains remain unquantified.
READ ARTICLE
IgniteServe Moves LLM Cold Starts Off the Serial Path
IgniteServe treats LLM startup as state-aware orchestration, reporting lower latency for both fully cold and cache-resident starts.

Latent Sparse Attention: What Reported Long-Context Gains Mean for Inference Infrastructure
LSA reports large attention speedups at 10 million tokens. For inference teams, end-to-end latency and quality are the decisive tests.

D-REAL and the Case for Training Speculative Drafters Around Acceptance Length
D-REAL reports up to 9.4% higher serving throughput by training speculative drafters directly for acceptance length.

Verify Before You Fetch: KV-Cache Capacity Without Unproven Speedup Claims
Verified KV-cache compression targets long-context memory limits. Its serving value depends on quality, retrieval traffic, and latency.

BridgeKV and the Case for Splitting Inference Across Models
BridgeKV translates a large model’s prompt cache for small-model decoding. Its measured speedup is promising, but deployment questions remain.

MINE and the Case for Entropy-Coded LLM Inference
MINE reports single-stream LLM speedups by decoding compressed weights on the GPU, making compression an execution choice, not just storage.

FOSCAR Makes the Case for Low-Bit KV Caches as a Decode Optimization
FOSCAR pairs low-bit KV caches with specialized attention kernels, reporting H100 speedups and higher long-context serving throughput.

FlashOffloading: Making Host DRAM Part of the Long-Context Inference Path
FlashOffloading extends KV-cache capacity into host DRAM, reporting higher throughput under a 30 ms per-token latency target.

LoopVQ and the Case for Architecture-Aware KV-Cache Compression
LoopVQ reports major serving gains with sub-1-bit KV-cache quantization, but infrastructure teams need more than headline speedups.

BlockLock: Making Batch Geometry a Control for Reproducible LLM Serving
BlockLock reports higher throughput with reproducible outputs by fixing batch geometry for selected requests, not the entire workload.

FastKVComp and the Real Cost of Moving KV State
FastKVComp targets the compression bottleneck in disaggregated inference, showing why fewer KV bytes are only part of the performance equation.

BasisKV: Rethinking KV-Cache Compression for Distributed Inference
BasisKV reports up to 1.42× faster 8-GPU inference by treating KV-cache compression as a distributed attention problem.

Speculative Decoding Needs an Energy Budget, Not Just a Longer Draft
Speculative decoding can increase energy use and slow serving. Acceptance, batching, and verification cost determine when it helps.

Self-Indexing Attention: Shared Retrieval and Compression for Long-Context Inference
Self-Indexing Attention reports large attention-operator speedups at 64K context. Here is what infrastructure teams should verify next.

MCF Makes KV-Cache Placement a Stage-Specific Inference Decision
MCF combines temporary KV replication with query-centric decode to address communication costs in distributed long-context inference.

GPU-Resident Speculative Decoding: A Systems Approach to Faster Browser Inference
A WebGPU paper reports substantial decoding gains over WebLLM, highlighting GPU residency as a practical inference optimization.

Morph: Making Request-Adaptive LLM Execution Work for Serving
Morph pairs request-specific attention and SSM execution with selective batching and prefix caching, reporting up to 3.87× higher throughput.

DISCO and the Communication Bottleneck in Distributed Speculative Decoding
DISCO targets vocabulary-scale transfers in distributed speculative decoding while preserving exact sampling. Production gains remain unquantified.

BellowsKV: Rethinking the KV Cache Across Transformer Layers
BellowsKV compresses KV caches across layers. Its accuracy results are promising, but serving gains still need measurement.

AKV Makes KV-Cache Retention a Serving Decision for LLM Agents
AKV reports lower agent inference costs through selective KV-cache reuse. Here is what the results mean for self-hosted serving stacks.

When Visual KV Caches Go Stale: Dynamic Re-selection for Multimodal Inference
Attention drift and recomputation budgets offer a runtime approach to preserving quality in compressed visual KV caches.

RIFT: Compressing the Draft Cache Without Losing Speculative Decoding Effectiveness
RIFT combines cache selection, adaptive refresh, and drafter training to reduce draft-side KV state while preserving acceptance length.

Specter: Using Speculative Decoding to Hide MoE Expert Transfer Latency
Specter uses speculative decoding to anticipate MoE expert demand and overlap PCIe transfers with GPU work in single-request inference.

Inference Speedups Do Not Travel Without Their Hardware Contract
Apple Silicon results show why inference speedups must be validated against the hardware, runtime, and workload teams actually deploy.

UlaCache and the Case for Unified KV-Cache Addressing
UlaCache unifies GPU and CPU KV-cache addressing, reporting up to 2.51× peak decoding throughput under matched memory budgets.

DLite Reframes Speculative Decoding as a State-Management Problem
DLite reuses target-model hidden states for local speculation, targeting draft-cache overhead and acceptance loss in agentic inference.

LLM-CoOpt and the Case for Joint Model and Hardware Optimization
LLM-CoOpt targets inference bottlenecks across algorithms and mixed hardware, but missing benchmarks limit deployment conclusions.

Prism Attention: What a Reported 3.4× Decode Speedup Means for Long-Context Inference
Prism Attention reports 3.4× faster decoding at 128K context. Here is what infrastructure teams should evaluate before adopting it.

Energy per Token Is Becoming a First-Class Metric for LLM Inference
Energy per token adds a practical dimension to inference optimization, connecting quantization, cache reuse, and workload-aware benchmarking.

EdgeFlow: Rethinking How KV Caches Move Across Inference Nodes
EdgeFlow combines KV-cache transfer and recomputation to reduce migration time across distributed inference nodes.

Token Window Pacing: Rethinking Admission Control for Serverless LLM Inference
Token Window Pacing reports higher throughput and steadier latency by regulating token workload rather than admitting whole requests or batches.

BELT Connects KV-Cache Compression to Faster Long-Context Decoding
BELT combines low-rank KV-cache compression, quantization, and GPU kernels. Here is what its reported gains mean for inference teams.

BLIS: Planning LLM Serving Capacity Before Committing GPUs
BLIS simulates LLM serving on CPUs, helping teams compare GPU configurations and serving policies before hardware validation.

Low-Bit Quantization Needs More Than a Better Calibration Score
DRQ refines low-bit model weights for robustness to activation shift while preserving the existing inference path.

Adaptive DHiSparse Makes KV Residency a Runtime Decision
Adaptive DHiSparse combines load-adaptive KV residency with speculative decoding to improve long-context serving under memory pressure.

Persistent KV State: What a 50-Million-Token Stream Means for Inference Infrastructure
Persistent KV storage shifts reused context from GPU recomputation to NVMe retrieval. Here is what the reported gains mean for inference teams.

RaReCache Moves KV-Cache Optimization Across Model Boundaries
RaReCache uses smaller-model prefill and selective recomputation to cut first-token latency, with important deployment questions still open.

H100 vs H200: Memory, Bandwidth and What It Means for LLM Inference
The H100 and H200 share Hopper compute. The difference is memory. Here is what 141 GB of HBM3e at 4.8 TB/s changes for LLM inference, and how to decide.
H100 · H200 · GPU comparison · LLM inference · HGX

What Is LLM Inference? How It Works, What Limits It, and How to Optimize It
How LLM inference works, from prefill and decode to the KV cache, why memory bandwidth sets the speed limit, which metrics matter, and which optimizations have evidence behind them.
LLM inference · inference optimization · KV cache · GPU memory bandwidth

SparseDecoding: Why Pruning Should Account for How LLMs Generate
SparseDecoding reports up to 1.48× faster decoding on A100 GPUs. Here is what the evidence supports and what infrastructure teams should verify.

SPIN Targets the Hidden Indexing Cost of Sparse Attention
SPIN predicts important KV blocks instead of scoring the full cache, reporting throughput and latency gains in vLLM serving.

MOIRA and the Case for Reading Less KV Cache
MOIRA targets long-context decode costs with adaptive KV-cache reads. Its reported gains are promising, but deployment evidence needs verification.

TaSQ and the Deployment Case for 1-Bit KV-Cache Compression
TaSQ reports larger batches and higher peak throughput with 1-bit KV-cache compression. Here is what its evidence means for inference teams.

SpecScale and the Shift From Token Throughput to Useful Reasoning
SpecScale targets wasted work in multi-path LLM reasoning through pruning, deduplication, and deferred verification. Its reported gains point to a broader scheduling problem for inference teams.

DRelay: Why Better Draft Context Matters for Parallel Speculative Decoding
DRelay reports higher decoding speedups by improving draft context and prefix-aware repair. Its SGLang results are promising, but infrastructure teams still need workload-level validation.

SparseEngine Makes Long-Context Inference a State Management Problem
SparseEngine treats sparse KV state as an engine-level concern. Its reported gains make a case for evaluating cache lifecycle and cross-request reuse alongside decoding speed.

PulseInfer and the I/O Problem Behind Long-Context Decoding
PulseInfer treats sparse KV-cache offloading as an I/O scheduling problem. Its reported gains highlight why long-context inference needs more than additional memory capacity.

Spexis Makes Speculation a Parallelism Axis for Multi-GPU Inference
Spexis adds speculative parallelism to vLLM-based multi-GPU serving, reporting up to 34% speedup. Its lookahead scheduler weighs speculation quality and memory pressure without increasing KV-cache usage.

KV-Cache Optimization Is Becoming a Memory-Systems Discipline
Recent research targets how KV caches are compressed, retrieved and staged across memory tiers. The opportunity depends on the actual serving bottleneck.

SpecStream: Coordinating KV Offloading and Speculative Decoding for Long-Context Inference
SpecStream overlaps KV-cache transfers, verification and drafting for long-context inference, showing why throughput per GPU matters alongside serving.
