Efficient Memory Management for Large Language Model Serving with PagedAttention

Introduced PagedAttention inspired by OS virtual memory paging, eliminating KV-cache memory fragmentation and enabling 2-4x higher serving throughput.

The core innovation powering vLLM, the industry standard high-throughput open-source LLM inference serving engine.

Assumptions

• KV cache memory can be partitioned into non-contiguous physical memory blocks indexed by page tables without impacting attention correctness

Limitations

• Page size tuning balances internal fragmentation vs page table overhead; complex CUDA memory management