KV-Cache Optimization for Efficient LLM Inference
Autoregressive transformers generate one token at a time. Caching the attention keys and values of earlier tokens means they are computed once instead of on every step; the cost is GPU memory, which grows with context length. This line of study is about shrinking and managing that cache.
Focus areas
- Studying Key-Value (KV) cache mechanisms in autoregressive transformer inference, and how caching prior attention states avoids redundant recomputation and reduces latency in LLMs.
- Exploring memory-reduction techniques including Multi-Query Attention (MQA), Grouped-Query Attention (GQA) and quantized KV-cache representations to lower GPU memory use in long-context generation.
- Analyzing trade-offs between cache eviction strategies (sliding window, attention-sink retention) and generation quality for extended context lengths.
Topics
Transformer AttentionMemory-Bound InferencePyTorch