KV cache growth

Rough bytes for one sequence: 2 × layers × seq × kv_heads × head_dim × dtype_bytes. Assumes fp16/bf16 cache (2 bytes).

8192 32 8 128
Estimated KV cache (one request)
—