Optimizations

Speculative decoding — across models

A small draft guesses ahead; the big model verifies in one pass. Speedup needs a cheap draft that's often right.

Throughput: without vs with speculation (tok/s)

Draft acceptance rate (%)

Deep dive:

Model

Acceptance & tokens/target-pass vs k

Throughput vs k

Token source (best k)

Acceptance by prompt type

KV-cache quantization — does it actually help?

Model

The KV cache itself: memory vs context — full vs 4-bit

At scale the cache overtakes the weights (measured → projected)

Precision trade-off: how low can you go?

Quantizing the cache to fewer bits saves proportionally more memory — but at some point it wrecks the output. Memory is exact (bits ÷ 16); fidelity is measured (how much of the generated text still matches the full-precision run).

Cache memory vs output fidelity, by KV precision

Across models — KV-cache footprint (exact, from config)

Computed directly from each model's architecture (layers × KV-heads × head-dim), so it's exact rather than a noisy measurement. Shows which models carry a heavier cache and how 8-/4-bit shrink it.

KV cache at 32k tokens: full vs 8-bit vs 4-bit (per model)

Full KV cache vs context length (per model)

Static batching — across models

Decoding many sequences at once fills the GPU; throughput climbs, per-request speed drops, memory grows.

Aggregate throughput vs batch size

Per-stream throughput vs batch size

Peak GPU memory vs batch size

Quantization — bf16 / 8-bit / 4-bit

Model

Same model at different precisions. Lower precision = faster & smaller; perplexity shows the quality cost.

How much did optimizations help?

Speed vs memory, by optimization stack (right = faster · lower = less memory · bottom-right is best)

Speculative decoding — across models

Throughput: without vs with speculation (tok/s)

Draft acceptance rate (%)

Deep dive:

Acceptance & tokens/target-pass vs k

Throughput vs k

Token source (best k)

Acceptance by prompt type

KV-cache quantization — does it actually help?

The KV cache itself: memory vs context — full vs 4-bit

At scale the cache overtakes the weights (measured → projected)

Precision trade-off: how low can you go?

Cache memory vs output fidelity, by KV precision

Across models — KV-cache footprint (exact, from config)

KV cache at 32k tokens: full vs 8-bit vs 4-bit (per model)

Full KV cache vs context length (per model)

Static batching — across models

Aggregate throughput vs batch size

Per-stream throughput vs batch size

Peak GPU memory vs batch size

Quantization — bf16 / 8-bit / 4-bit

Speed vs quality (Pareto)

Perplexity (lower better)

Throughput (tok/s)

GPU memory (MB)