Insights / LLM Enhancements

Research on making models do more with the same hardware.

New articles are published regularly, researched from current papers, releases and industry reporting, then fact-checked against their sources.

MosaicKV and the Case for Scheduling KV Cache Across Tenants
Nov 16, 2026 10 min read

MosaicKV and the Case for Scheduling KV Cache Across Tenants

MosaicKV coordinates KV-cache placement, tenant memory allocation, and prefetching. Its architecture is clear; performance gains remain unquantified.

READ ARTICLE
IgniteServe Moves LLM Cold Starts Off the Serial Path
Nov 15, 2026 10 min read

IgniteServe Moves LLM Cold Starts Off the Serial Path

IgniteServe treats LLM startup as state-aware orchestration, reporting lower latency for both fully cold and cache-resident starts.

Latent Sparse Attention: What Reported Long-Context Gains Mean for Inference Infrastructure
Nov 14, 2026 9 min read

Latent Sparse Attention: What Reported Long-Context Gains Mean for Inference Infrastructure

LSA reports large attention speedups at 10 million tokens. For inference teams, end-to-end latency and quality are the decisive tests.

D-REAL and the Case for Training Speculative Drafters Around Acceptance Length
Nov 13, 2026 10 min read

D-REAL and the Case for Training Speculative Drafters Around Acceptance Length

D-REAL reports up to 9.4% higher serving throughput by training speculative drafters directly for acceptance length.

Verify Before You Fetch: KV-Cache Capacity Without Unproven Speedup Claims
Nov 12, 2026 10 min read

Verify Before You Fetch: KV-Cache Capacity Without Unproven Speedup Claims

Verified KV-cache compression targets long-context memory limits. Its serving value depends on quality, retrieval traffic, and latency.

BridgeKV and the Case for Splitting Inference Across Models
Nov 11, 2026 9 min read

BridgeKV and the Case for Splitting Inference Across Models

BridgeKV translates a large model’s prompt cache for small-model decoding. Its measured speedup is promising, but deployment questions remain.

MINE and the Case for Entropy-Coded LLM Inference
Nov 10, 2026 9 min read

MINE and the Case for Entropy-Coded LLM Inference

MINE reports single-stream LLM speedups by decoding compressed weights on the GPU, making compression an execution choice, not just storage.

FOSCAR Makes the Case for Low-Bit KV Caches as a Decode Optimization
Nov 9, 2026 9 min read

FOSCAR Makes the Case for Low-Bit KV Caches as a Decode Optimization

FOSCAR pairs low-bit KV caches with specialized attention kernels, reporting H100 speedups and higher long-context serving throughput.

FlashOffloading: Making Host DRAM Part of the Long-Context Inference Path
Nov 8, 2026 9 min read

FlashOffloading: Making Host DRAM Part of the Long-Context Inference Path

FlashOffloading extends KV-cache capacity into host DRAM, reporting higher throughput under a 30 ms per-token latency target.

LoopVQ and the Case for Architecture-Aware KV-Cache Compression
Nov 7, 2026 9 min read

LoopVQ and the Case for Architecture-Aware KV-Cache Compression

LoopVQ reports major serving gains with sub-1-bit KV-cache quantization, but infrastructure teams need more than headline speedups.

BlockLock: Making Batch Geometry a Control for Reproducible LLM Serving
Nov 6, 2026 9 min read

BlockLock: Making Batch Geometry a Control for Reproducible LLM Serving

BlockLock reports higher throughput with reproducible outputs by fixing batch geometry for selected requests, not the entire workload.

FastKVComp and the Real Cost of Moving KV State
Nov 5, 2026 9 min read

FastKVComp and the Real Cost of Moving KV State

FastKVComp targets the compression bottleneck in disaggregated inference, showing why fewer KV bytes are only part of the performance equation.

BasisKV: Rethinking KV-Cache Compression for Distributed Inference
Nov 4, 2026 9 min read

BasisKV: Rethinking KV-Cache Compression for Distributed Inference

BasisKV reports up to 1.42× faster 8-GPU inference by treating KV-cache compression as a distributed attention problem.

Speculative Decoding Needs an Energy Budget, Not Just a Longer Draft
Nov 3, 2026 9 min read

Speculative Decoding Needs an Energy Budget, Not Just a Longer Draft

Speculative decoding can increase energy use and slow serving. Acceptance, batching, and verification cost determine when it helps.

Self-Indexing Attention: Shared Retrieval and Compression for Long-Context Inference
Nov 2, 2026 10 min read

Self-Indexing Attention: Shared Retrieval and Compression for Long-Context Inference

Self-Indexing Attention reports large attention-operator speedups at 64K context. Here is what infrastructure teams should verify next.

MCF Makes KV-Cache Placement a Stage-Specific Inference Decision
Nov 1, 2026 10 min read

MCF Makes KV-Cache Placement a Stage-Specific Inference Decision

MCF combines temporary KV replication with query-centric decode to address communication costs in distributed long-context inference.

GPU-Resident Speculative Decoding: A Systems Approach to Faster Browser Inference
Oct 31, 2026 9 min read

GPU-Resident Speculative Decoding: A Systems Approach to Faster Browser Inference

A WebGPU paper reports substantial decoding gains over WebLLM, highlighting GPU residency as a practical inference optimization.

Morph: Making Request-Adaptive LLM Execution Work for Serving
Oct 30, 2026 9 min read

Morph: Making Request-Adaptive LLM Execution Work for Serving

Morph pairs request-specific attention and SSM execution with selective batching and prefix caching, reporting up to 3.87× higher throughput.

DISCO and the Communication Bottleneck in Distributed Speculative Decoding
Oct 29, 2026 9 min read

DISCO and the Communication Bottleneck in Distributed Speculative Decoding

DISCO targets vocabulary-scale transfers in distributed speculative decoding while preserving exact sampling. Production gains remain unquantified.

BellowsKV: Rethinking the KV Cache Across Transformer Layers
Oct 28, 2026 9 min read

BellowsKV: Rethinking the KV Cache Across Transformer Layers

BellowsKV compresses KV caches across layers. Its accuracy results are promising, but serving gains still need measurement.

AKV Makes KV-Cache Retention a Serving Decision for LLM Agents
Oct 27, 2026 9 min read

AKV Makes KV-Cache Retention a Serving Decision for LLM Agents

AKV reports lower agent inference costs through selective KV-cache reuse. Here is what the results mean for self-hosted serving stacks.

When Visual KV Caches Go Stale: Dynamic Re-selection for Multimodal Inference
Oct 26, 2026 9 min read

When Visual KV Caches Go Stale: Dynamic Re-selection for Multimodal Inference

Attention drift and recomputation budgets offer a runtime approach to preserving quality in compressed visual KV caches.

RIFT: Compressing the Draft Cache Without Losing Speculative Decoding Effectiveness
Oct 25, 2026 9 min read

RIFT: Compressing the Draft Cache Without Losing Speculative Decoding Effectiveness

RIFT combines cache selection, adaptive refresh, and drafter training to reduce draft-side KV state while preserving acceptance length.

Specter: Using Speculative Decoding to Hide MoE Expert Transfer Latency
Oct 24, 2026 9 min read

Specter: Using Speculative Decoding to Hide MoE Expert Transfer Latency

Specter uses speculative decoding to anticipate MoE expert demand and overlap PCIe transfers with GPU work in single-request inference.

Inference Speedups Do Not Travel Without Their Hardware Contract
Oct 23, 2026 9 min read

Inference Speedups Do Not Travel Without Their Hardware Contract

Apple Silicon results show why inference speedups must be validated against the hardware, runtime, and workload teams actually deploy.

UlaCache and the Case for Unified KV-Cache Addressing
Oct 22, 2026 9 min read

UlaCache and the Case for Unified KV-Cache Addressing

UlaCache unifies GPU and CPU KV-cache addressing, reporting up to 2.51× peak decoding throughput under matched memory budgets.

DLite Reframes Speculative Decoding as a State-Management Problem
Oct 21, 2026 9 min read

DLite Reframes Speculative Decoding as a State-Management Problem

DLite reuses target-model hidden states for local speculation, targeting draft-cache overhead and acceptance loss in agentic inference.

LLM-CoOpt and the Case for Joint Model and Hardware Optimization
Oct 20, 2026 9 min read

LLM-CoOpt and the Case for Joint Model and Hardware Optimization

LLM-CoOpt targets inference bottlenecks across algorithms and mixed hardware, but missing benchmarks limit deployment conclusions.

Prism Attention: What a Reported 3.4× Decode Speedup Means for Long-Context Inference
Oct 19, 2026 9 min read

Prism Attention: What a Reported 3.4× Decode Speedup Means for Long-Context Inference

Prism Attention reports 3.4× faster decoding at 128K context. Here is what infrastructure teams should evaluate before adopting it.

Energy per Token Is Becoming a First-Class Metric for LLM Inference
Oct 18, 2026 10 min read

Energy per Token Is Becoming a First-Class Metric for LLM Inference

Energy per token adds a practical dimension to inference optimization, connecting quantization, cache reuse, and workload-aware benchmarking.

EdgeFlow: Rethinking How KV Caches Move Across Inference Nodes
Oct 17, 2026 9 min read

EdgeFlow: Rethinking How KV Caches Move Across Inference Nodes

EdgeFlow combines KV-cache transfer and recomputation to reduce migration time across distributed inference nodes.

Token Window Pacing: Rethinking Admission Control for Serverless LLM Inference
Oct 16, 2026 9 min read

Token Window Pacing: Rethinking Admission Control for Serverless LLM Inference

Token Window Pacing reports higher throughput and steadier latency by regulating token workload rather than admitting whole requests or batches.

BELT Connects KV-Cache Compression to Faster Long-Context Decoding
Oct 15, 2026 9 min read

BELT Connects KV-Cache Compression to Faster Long-Context Decoding

BELT combines low-rank KV-cache compression, quantization, and GPU kernels. Here is what its reported gains mean for inference teams.

BLIS: Planning LLM Serving Capacity Before Committing GPUs
Oct 14, 2026 9 min read

BLIS: Planning LLM Serving Capacity Before Committing GPUs

BLIS simulates LLM serving on CPUs, helping teams compare GPU configurations and serving policies before hardware validation.

Low-Bit Quantization Needs More Than a Better Calibration Score
Oct 13, 2026 10 min read

Low-Bit Quantization Needs More Than a Better Calibration Score

DRQ refines low-bit model weights for robustness to activation shift while preserving the existing inference path.

Adaptive DHiSparse Makes KV Residency a Runtime Decision
Oct 12, 2026 9 min read

Adaptive DHiSparse Makes KV Residency a Runtime Decision

Adaptive DHiSparse combines load-adaptive KV residency with speculative decoding to improve long-context serving under memory pressure.

Persistent KV State: What a 50-Million-Token Stream Means for Inference Infrastructure
Oct 11, 2026 9 min read

Persistent KV State: What a 50-Million-Token Stream Means for Inference Infrastructure

Persistent KV storage shifts reused context from GPU recomputation to NVMe retrieval. Here is what the reported gains mean for inference teams.

RaReCache Moves KV-Cache Optimization Across Model Boundaries
Oct 10, 2026 9 min read

RaReCache Moves KV-Cache Optimization Across Model Boundaries

RaReCache uses smaller-model prefill and selective recomputation to cut first-token latency, with important deployment questions still open.

H100 vs H200: Memory, Bandwidth and What It Means for LLM Inference
Oct 9, 2026 6 min read

H100 vs H200: Memory, Bandwidth and What It Means for LLM Inference

The H100 and H200 share Hopper compute. The difference is memory. Here is what 141 GB of HBM3e at 4.8 TB/s changes for LLM inference, and how to decide.

H100 · H200 · GPU comparison · LLM inference · HGX

What Is LLM Inference? How It Works, What Limits It, and How to Optimize It
Oct 9, 2026 7 min read

What Is LLM Inference? How It Works, What Limits It, and How to Optimize It

How LLM inference works, from prefill and decode to the KV cache, why memory bandwidth sets the speed limit, which metrics matter, and which optimizations have evidence behind them.

LLM inference · inference optimization · KV cache · GPU memory bandwidth

SparseDecoding: Why Pruning Should Account for How LLMs Generate
Oct 9, 2026 9 min read

SparseDecoding: Why Pruning Should Account for How LLMs Generate

SparseDecoding reports up to 1.48× faster decoding on A100 GPUs. Here is what the evidence supports and what infrastructure teams should verify.

SPIN Targets the Hidden Indexing Cost of Sparse Attention
Oct 8, 2026 9 min read

SPIN Targets the Hidden Indexing Cost of Sparse Attention

SPIN predicts important KV blocks instead of scoring the full cache, reporting throughput and latency gains in vLLM serving.

MOIRA and the Case for Reading Less KV Cache
Oct 7, 2026 10 min read

MOIRA and the Case for Reading Less KV Cache

MOIRA targets long-context decode costs with adaptive KV-cache reads. Its reported gains are promising, but deployment evidence needs verification.

TaSQ and the Deployment Case for 1-Bit KV-Cache Compression
Oct 6, 2026 10 min read

TaSQ and the Deployment Case for 1-Bit KV-Cache Compression

TaSQ reports larger batches and higher peak throughput with 1-bit KV-cache compression. Here is what its evidence means for inference teams.

SpecScale and the Shift From Token Throughput to Useful Reasoning
Oct 5, 2026 9 min read

SpecScale and the Shift From Token Throughput to Useful Reasoning

SpecScale targets wasted work in multi-path LLM reasoning through pruning, deduplication, and deferred verification. Its reported gains point to a broader scheduling problem for inference teams.

DRelay: Why Better Draft Context Matters for Parallel Speculative Decoding
Oct 4, 2026 10 min read

DRelay: Why Better Draft Context Matters for Parallel Speculative Decoding

DRelay reports higher decoding speedups by improving draft context and prefix-aware repair. Its SGLang results are promising, but infrastructure teams still need workload-level validation.

SparseEngine Makes Long-Context Inference a State Management Problem
Oct 3, 2026 10 min read

SparseEngine Makes Long-Context Inference a State Management Problem

SparseEngine treats sparse KV state as an engine-level concern. Its reported gains make a case for evaluating cache lifecycle and cross-request reuse alongside decoding speed.

PulseInfer and the I/O Problem Behind Long-Context Decoding
Oct 2, 2026 10 min read

PulseInfer and the I/O Problem Behind Long-Context Decoding

PulseInfer treats sparse KV-cache offloading as an I/O scheduling problem. Its reported gains highlight why long-context inference needs more than additional memory capacity.

Spexis Makes Speculation a Parallelism Axis for Multi-GPU Inference
Oct 1, 2026 10 min read

Spexis Makes Speculation a Parallelism Axis for Multi-GPU Inference

Spexis adds speculative parallelism to vLLM-based multi-GPU serving, reporting up to 34% speedup. Its lookahead scheduler weighs speculation quality and memory pressure without increasing KV-cache usage.

KV-Cache Optimization Is Becoming a Memory-Systems Discipline
Sep 30, 2026 10 min read

KV-Cache Optimization Is Becoming a Memory-Systems Discipline

Recent research targets how KV caches are compressed, retrieved and staged across memory tiers. The opportunity depends on the actual serving bottleneck.

SpecStream: Coordinating KV Offloading and Speculative Decoding for Long-Context Inference
Sep 30, 2026 10 min read

SpecStream: Coordinating KV Offloading and Speculative Decoding for Long-Context Inference

SpecStream overlaps KV-cache transfers, verification and drafting for long-context inference, showing why throughput per GPU matters alongside serving.