Imagine four people booking a ride to the same destination. If each person takes a separate taxi, four vehicles make the same trip. But if they travel together in a bus, everyone reaches the destination in a single trip. The bus carries more people at once and makes better use of the resources.
Batching in AI works in a similar way. Instead of processing one request at a time, a server groups multiple requests together so the GPU can process them more efficiently. This helps serve more users while making better use of expensive computing resources. But how those requests are grouped and scheduled also affects how quickly users get responses and how much it costs to serve them.
There are three common approaches to batching: static batching, dynamic batching, and continuous batching. Let’s understand what batching means technically and how these approaches differ.
What is Batching?
Batching is the process of running multiple inputs through a model in a single forward pass instead of processing each input individually. In an LLM, each input is a sequence of tokens, so the model processes a batch of sequences together. Conceptually, instead of an input tensor shaped [1, seq_len], the model processes a batch shaped [batch_size, seq_len].
This allows the GPU to perform computations across multiple requests in parallel, improving hardware utilization and throughput. However, LLMs generate responses token by token, and different requests can take different amounts of time to finish. How the server groups requests, handles incoming traffic, and frees up resources as responses finish determines which batching approach works best.
Why LLMs Need Batching?
The answer is simple, during generation, the GPU is limited by how fast it can read memory, not how fast it can do the math. Batching lets one memory read serve many requests.
LLM inference has two phases:
- Prefill: The model reads your whole prompt in one forward pass. All prompt tokens are processed in parallel, so this phase keeps the GPU’s compute units busy.
- Decode: The model generates one token at a time. Each new token needs a full forward pass, which means reading every model weight from GPU memory again.
Decode is where the waste happens. Some rough numbers for a 7B model in FP16 on an H100:
- The weights take about 14 GB.
- H100 memory bandwidth is about 3.35 TB/s, so reading 14 GB takes roughly 4 ms.
- That caps a single request at around 240 tokens per second.
- Each token needs about 14 GFLOPs (2 × parameters). Doing that in 4 ms is around 3.5 TFLOPS, while the H100 offers close to 1,000 TFLOPS of dense BF16.
So at batch size 1, the GPU’s compute sits more than 99% idle. It spends almost all its time loading the weights. If you run 32 sequences in the same steps, the weights are loaded once and you get 32 tokens instead of 1, nearly the same step time. That’s the whole case of batching higher throughput and lower cost per token, without buying more GPUs.
There is a limit. Every sequence in the batch keeps its own KV cache in GPU memory, and that cache grows with every token. Batch size ends up bounded by memory, which becomes important when we get to continuous batching.
Also Read: The Four Caches in LLM Serving
Static Batching
Static batching groups a fixed set of requests, runs them together, and returns results only when every request in the group is finished. Nothing new can join until the whole batch is done.
The model.generate() call from earlier is static batching. It’s the default behavior of most hand-written inference loops.
The problem is that LLM outputs vary wildly in length. Say a batch of 4 requests ends up generating 20, 50, 100 and 500 tokens. Then the batch runs for 500 decode steps, because it waits for the longest one. The 20-token request is done at step 20, but its slot sits idle (or computes padding) for 480 more steps. Useful work is 20 + 50 + 100 + 500 = 670 token-steps out of 4 × 500 = 2,000 possible. That’s about 33% utilization.A request that arrives at step 1 has to wait 499 steps before it can even start.
This is called head-of-line blocking: one long request holds up everyone behind it.
To compare the three strategies fairly, let’s build a small simulator. It skips the model and only tracks decode steps, which is enough to see how scheduling affects GPU usage and latency. A new request arrives every 10 steps, and the GPU can run 4 sequences at once.
Code:
from collections import deque
# Each request: (id, arrival_step, output_tokens)
# A new request arrives every 10 decode steps, with very different output lengths.
lengths = [120, 30, 60, 300, 40, 90, 25, 150, 50, 70, 35, 80]
REQUESTS = [(f”R{i}”, i * 10, n) for i, n in enumerate(lengths)]
SLOTS = 4 # max sequences the GPU runs at once
def summarize(name, total_steps, useful, finish):
latency = [finish[rid] – arr for rid, arr, _ in REQUESTS]
util = useful / (total_steps * SLOTS)
print(f”{name:
Two lines carry the whole story. The batch waits until it has 4 requests, and every request gets its result at step + longest, no matter how short it was.
When static batching is fine: offline jobs where you control the inputs and outputs are similar in length, like summarizing 10,000 documents with a fixed output cap.
Dynamic Batching
Dynamic batching changes how a batch is formed, not how it runs. Requests wait in a queue, and the server launches a batch when either of two things happens first:
- The batch reaches its maximum size.
- The oldest request in the queue has waited longer than a set timeout.
This fixes one problem with static batching. Under light traffic, a request no longer waits forever for the batch to fill up. Under heavy traffic, batches still fill quickly and the GPU stays busy.
NVIDIA Triton Inference Server is the classic example. You turn it on in the model’s config.pbtxt:
Code:
max_batch_size: 8
dynamic_batching {
preferred_batch_size: [ 4, 8 ]
max_queue_delay_microseconds: 100
}
This tells Triton to aim for batches of 4 or 8, and to never hold a request more than 100 microseconds waiting for others to join.
Here is the same idea added to our simulator:
Code:
def dynamic_batching(requests, max_wait=15):
pending = deque(requests)
step, useful, finish = 0, 0, {}
while pending:
step = max(step, pending[0][1])
deadline = pending[0][1] + max_wait
# launch when the batch is full OR the oldest request has waited max_wait
while sum(r[1]
Look at the line finish[rid] = step + longest. It’s the same as in static batching. Once a batch starts, it still runs until its longest request is done.
That’s why dynamic batching works so well for models where every request costs about the same: BERT classifiers, embedding models, image models. One forward pass, done. For LLM generation, the straggler problem is still there. Worse, launching a half-full batch early can lock the GPU into running 2 sequences for 120 steps while new requests pile up outside. We’ll see this in the numbers shortly.
Continuous Batching
Continuous batching makes scheduling decisions at every decode step instead of once per batch. When a sequence emits its end-of-sequence token, it’s returned right away and its slot is freed. A waiting request takes that slot on the very next step. The batch never drains; it keeps refilling.
You’ll also see it called iteration-level scheduling (from the Orca paper, OSDI 2022, which introduced the idea) or in-flight batching (NVIDIA TensorRT-LLM’s name for it).
Back to the 4-request example with 20, 50, 100 and 500 tokens. With continuous batching, the 20-token request returns at step 20, and a new request starts in that slot at step 21. The GPU stays close to full, and new requests start almost immediately.
Here is the last piece of the simulator:
Code:
def continuous_batching(requests):
waiting = deque(requests)
running = {} # id -> tokens left
step, useful, finish = 0, 0, {}
while waiting or running:
# every step: fill free slots with requests that have arrived
while waiting and len(running)
The difference is that the scheduling loop runs inside the decode loop. Each request’s finish time is its own, not the batch’s.
What makes it work in practice:
The idea is simple, but real systems need a few extra pieces:
- Mixing prefill and decode: New requests need prefill while existing ones generate tokens. Orca uses selective batching to handle both efficiently.
- KV cache management: vLLM’s PagedAttention stores KV caches in fixed-size blocks, reducing memory fragmentation and allowing more sequences to fit in a batch.
- Chunked prefill: Systems like Sarathi-Serve split long prompts into smaller chunks and interleave them with decode steps to reduce delays for running requests.
Static vs Dynamic vs Continuous Batching
Running the full simulator on the same 12 requests gives this:
Static steps=560 slot_util=47% avg_latency=402 first_two=[330, 320]
Dynamic steps=665 slot_util=39% avg_latency=418 first_two=[135, 125]
Continuous steps=330 slot_util=80% avg_latency=118 first_two=[120, 30]
A few things stand out:
- Continuous batching finishes the same work in 41% fewer steps (330 vs 560) and cuts average latency from 402 steps to 118. The GPU slots are busy 80% of the time instead of 47%.
- Dynamic batching helps the first requests get results about 2.5x sooner than static, since it doesn’t wait for a full batch. But it does worse overall, because it launched half-empty batches that then had to run to completion. That’s the trade-off you get when a request-level batcher meets variable-length generation.
- R1 asked for only 30 tokens. With static batching it waited 320 steps for its answer. With continuous batching it waited exactly 30.
The simulator ignores prefill cost and memory limits, so real numbers will differ, but the pattern holds on real serving engines.
Static Dynamic Continuous When a batch starts Once a fixed group is ready When full or the timeout hits Every decode step New requests join mid-run? No No Yes Finished sequences Wait for the whole batch Wait for the whole batch Returned at once, slot reused Scheduling level Per batch Per batch Per iteration (token) Best fit Offline jobs, similar output lengths Fixed-cost models (BERT, embeddings, vision) LLM serving with variable outputs Where you’ll see it Plain model.generate() loops Triton dynamic batcher vLLM, TGI, TensorRT-LLM, SGLang
If you remember one line: static batching waits for everyone, dynamic batching waits less to start but still waits for everyone to finish, and continuous batching doesn’t wait for anyone. That’s why every serious LLM serving stack today is built on continuous batching, and why swapping a hand-rolled generate() server for vLLM is often the cheapest speedup you’ll ever get.
Conclusion
Batching exists because LLM decode is memory-bound. Reading the weights once and using them for many sequences is the single biggest lever you have on cost per token. The three strategies are really a story of moving the scheduling decision to a finer level. Static batching decides once per group. Dynamic batching decides once per batch but stops waiting for a full group. Continuous batching decides on every token, so short requests never get stuck behind long ones. For most teams, the practical takeaway is simple: don’t build your own batching loop for LLM serving. Use an engine like vLLM, TGI or TensorRT-LLM, then spend your time tuning the knobs that matter, such as maximum concurrent sequences, tokens per step, and how much GPU memory goes to the KV cache.
I am a Data Science Trainee at Analytics Vidhya, passionately working on the development of advanced AI solutions such as Generative AI applications, Large Language Models, and cutting-edge AI tools that push the boundaries of technology. My role also involves creating engaging educational content for Analytics Vidhya’s YouTube channels, developing comprehensive courses that cover the full spectrum of machine learning to generative AI, and authoring technical blogs that connect foundational concepts with the latest innovations in AI. Through this, I aim to contribute to building intelligent systems and share knowledge that inspires and empowers the AI community.
Login to continue reading and enjoy expert-curated content.
Keep Reading for Free