In this article, you will learn how to match an LLM inference optimization technique to the actual bottleneck in your workload by working through a six-question decision tree.
Topics we will cover include:
- What each major inference optimization technique, from quantization to prefill-decode disaggregation, assumes about your workload.
- How to work through a decision tree that routes you to the right technique based on your model and traffic.
- How the techniques compare on trade-offs, and which symptoms show that a chosen technique isn’t working.
Introduction
LLM inference gets slow and expensive as traffic grows. Requests queue behind long generations, GPU memory fills with cached attention state, and the hardware spends much of its time waiting on memory reads.
Inference optimization is the set of techniques that help here. Some techniques target memory, some target compute, and some target how requests are scheduled against each other. Each technique assumes something specific about your workload, so the right choice depends on which assumptions hold for yours. This article covers:
- What each major inference optimization technique assumes about your workload
- A decision tree that routes you to a technique
- How the techniques compare on latency, throughput, and implementation cost
- The symptoms that show a technique isn’t working
We start with what each technique assumes.
What Each Inference Optimization Technique Assumes
LLM inference has many optimization techniques, but each one addresses a different type of bottleneck. Before using one, it helps to understand the workload it is designed for.
Technique
What It Assumes
Quantization / sparsity / distillation
The model itself is too large for the hardware or the latency budget, and shrinking it is worth some accuracy risk
Prefix caching
Many requests share an identical starting sequence of tokens, so recomputing that prefix is wasted work
Continuous (in-flight) batching
Requests arrive concurrently with unpredictable output lengths, and waiting for the slowest one in a batch wastes the GPU
PagedAttention
KV cache memory is fragmented or over-provisioned, capping batch size below what the hardware could otherwise hold
Speculative decoding
Decode is the bottleneck, tokens are generated one at a time, and a cheaper draft model can guess several of them correctly
Prefill-decode disaggregation
Prefill and decode are contending for the same GPU, and that contention is hurting both time-to-first-token and inter-token latency at once
Tensor / pipeline parallelism
The model doesn’t fit in a single device’s memory, or per-device compute is the ceiling on throughput
KV cache offloading
Useful cache entries are being evicted too early because GPU memory can’t hold everything you’d like to keep
Offline batch inference
Requests don’t need an interactive response, and total throughput outweighs per-request latency
Choose the technique based on the bottleneck in your workload to avoid unnecessary complexity.
Working Through a Decision Tree to Choose a Technique
The tree below has six branching questions. Each one narrows the field based on a specific property of your model or your traffic. Every question ends the same way: a direct answer if the condition holds, and a pointer to the next question if it doesn’t.
Question 1: Does the Model Fit Comfortably on Your Target Hardware?
Before moving on to the serving stack, check whether the model itself is the problem. A model that barely fits in GPU memory leaves almost no room for KV cache, which caps your batch size no matter how good your scheduler is.
If the model is too large, or inference is compute-bound even at batch size one, start with model-level compression:
- Quantization reduces the precision of weights and activations.
- Structured sparsity removes less important weights to reduce computation.
- Distillation trains a smaller model to mimic a larger model.
These techniques are mostly independent of the serving stack, and you may want to consider them first because they determine how much hardware capacity remains.
If the model already fits with headroom to spare, move to Question 2.
Question 2: Do Your Requests Share Repeated Prompt Structure?
Chat applications resend the full conversation history on every turn. Agent frameworks reuse the same system prompt across thousands of calls. Retrieval-augmented generation (RAG) pipelines often prepend the same instructions to different retrieved contexts.
If a meaningful share of your traffic has an identical prefix, whether that’s a system prompt or a growing chat history, prefix caching is the highest-leverage technique available. It skips recomputing the KV cache for tokens the model has already processed, and the gains compound as conversations get longer.
⚠️ Prefix caching only works when the shared prefix matches exactly, character for character. Put stable, reusable content first and variable content at the end of the prompt.
If your prompts are mostly unique with little shared structure, move to Question 3.
Question 3: Are Requests Arriving Concurrently With Variable Output Lengths?
A support chatbot that returns a one-sentence answer and a code assistant that returns four hundred lines rarely finish at the same time, even if they started together. Under naive batching, the whole batch waits on the slowest request.
If your traffic mixes short and long generations and arrives in overlapping bursts, continuous batching fixes the idle time this creates. Finished sequences leave the batch immediately, allowing new requests to start without waiting for a batch boundary. Most serving engines enable this by default.
If your traffic is already well-batched and GPU idle time is not a problem, move to Question 4.
Question 4: Is KV Cache Fragmentation Limiting Your Batch Size?
Some engines reserve a fixed, contiguous block of memory per request sized for the maximum possible sequence length, even when most requests are much shorter. That reserved space sits unused for the life of the request and blocks other requests from using it.
If KV cache is wasting memory or limiting batch size, PagedAttention can help. It splits the cache into fixed-size pages, allowing memory to be allocated in smaller chunks instead of large contiguous blocks. Most modern serving engines already support it, so check whether it is enabled before adding new configuration.
If memory is not the constraint, move to Question 5.
Question 5: Is Per-Token Decode Latency the Metric You’re Focusing On?
If your goal is faster token streaming for each user rather than higher overall throughput, consider speculative decoding.
A smaller draft model predicts several tokens ahead, and the main model verifies them in a single pass. When the draft model predicts correctly, multiple sequential decode steps can be completed together, reducing inter-token latency.
The trade-off is the extra model and verification overhead. If you don’t have a suitable draft model or its predictions are accepted too rarely, the added complexity may not pay off. In that case, move to Question 6.
Question 6: Are Prefill and Decode Contending for the Same GPU at Scale?
This is mainly a production-scale concern because it changes the serving architecture.
If long prefills are delaying decode requests and hurting both time-to-first-token and inter-token latency, prefill-decode disaggregation can separate the two phases onto dedicated hardware. Prefill nodes process prompts, then pass the KV cache to decode nodes optimized for token generation.
This adds KV-cache transfer and routing overhead, so it makes sense only when the traffic justifies that complexity. For smaller workloads or non-interactive requests, offline batch inference is often simpler. If none of these conditions apply, start with continuous batching and prefix caching.
Mapping the Decision Tree to an Optimization Technique
Working through the tree lands you on one of the techniques in the following table:
Technique
Best For
Trade-off
Quantization / sparsity / distillation
Models too large for the target hardware or latency budget
Some accuracy loss, and quantized activations are harder to get right than quantized weights
Prefix caching
Chat, agents, and RAG workloads with repeated prompt structure
Breaks on any character-level mismatch in the shared prefix
Continuous batching
Concurrent traffic with variable output lengths
Minimal downside; mostly a matter of whether it’s enabled
PagedAttention
Workloads bottlenecked by KV cache fragmentation
Adds a page table layer to attention computation
Speculative decoding
Latency-sensitive single-user generation
Needs a draft model with a reasonably high acceptance rate to pay off
Prefill-decode disaggregation
Large-scale traffic where prefill and decode contend for the same GPU
Network cost of moving the KV cache between nodes, plus routing complexity
Tensor / pipeline parallelism
Models that don’t fit on a single device
Pipeline bubbles and inter-device communication overhead
KV cache offloading
Long-context or high-concurrency workloads running out of GPU memory for cache
Fetching offloaded cache back adds latency
Offline batch inference
Non-interactive workloads prioritizing total throughput
Not usable for anything with a strict response-time requirement
Recognizing Common Inference Optimization Pitfalls
A technique that you think will work may behave quite differently in practice. The signals below tell you when that is happening and what to check first.
Symptom
Likely Cause
Fix
Prefix cache hit rate is near zero despite repeated prompts
Non-deterministic serialization, or dynamic content placed early in the prompt
Front-load static content and fix key ordering in serialized context
Batch size won’t grow even with memory to spare
PagedAttention isn’t enabled, or page size is misconfigured
Check the serving engine’s batching and cache flags directly rather than assuming defaults
Speculative decoding is slower than plain decoding
Draft model’s acceptance rate is too low for the target model
Pick a draft model closer in behavior to the target, or drop speculative decoding for this workload
Disaggregation made latency worse, not better
Workload is too small, or most requests are short and cache-heavy
Fall back to local prefill and revisit disaggregation only past a defined traffic threshold
Throughput gains from quantization didn’t show up
Only weights were quantized while activations stayed at full precision, and the GPU still converts back before compute
Profile where time is spent before assuming quantization alone will fix throughput
If a technique consistently underperforms, check its underlying assumption before adding more complexity.
Wrapping Up
Choosing an inference optimization technique starts with identifying your bottleneck. Model size, prompt overlap, traffic concurrency, memory fragmentation, and the balance between throughput and per-user latency each point toward a different answer, and most systems will need a combination of a few of these rather than relying on just one.
If you’re interested in learning more, you may find The Roadmap to Mastering LLM Inference Optimization useful.
Happy learning!
About Bala Priya C
Bala Priya is a developer, educator, and technical writer based in India. She writes about programming, data science, machine learning, and AI, with a focus on making complex technical ideas practical and approachable.
With over five years of experience in technical writing, she creates tutorials, how-to guides, and deep dives that help developers build, solve problems, and understand complex technical concepts. Her technical interests include LLM applications, agentic AI, graph algorithms, computer vision, explainable AI, and ML systems.
When she’s not writing or coding, Bala is usually reading classic literature, epic fantasy, or speculative fiction. She also enjoys talking about books, learning new things, and sharing what she learns with the developer community.