Continuous batching from first principles
Continuous batching from first principles Continuous batching Published November 25, 2025 Update on GitHub Upvote 444 Rémi Ouazan Reboul ror Arthur Zucker ArthurZ Luc Georges mcpotato TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput. If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency.
This Research is relevant to the technology intelligence record because it involves GitHub, Hugging Face, Meta, Claude. The source article should remain the factual reference for follow-up coverage.
- Continuous batching Published November 25, 2025 Update on GitHub Upvote 444 Rémi Ouazan Reboul ror Arthur Zucker ArthurZ Luc Georges mcpotato TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput.
- If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency.
- That's because at the heart of it, all LLMs are just fancy next token predictors.
- An LLM first processes your entire prompt to produce one new token.
- Then it keeps adding tokens one by one, each time reading everything that came before, until it decides generation is over.
- This generation process is computationally expensive: it requires passing the input through billions of parameters for each token generated.