As LLM applications grow more complex, inference cost and latency become increasingly important. A single request can contain thousands or even millions of tokens from system instructions, conversation history, retrieved documents, tool definitions, and user input. Reprocessing the same information again and again wastes both time and compute.
Caching helps avoid this repeated work. But LLM caching isn’t a single technique. Different caches operate at different stages of the serving stack and solve different problems. In this article, we’ll explore four key techniques: KV caching, prefix caching, prompt caching, and semantic caching.
1. KV Cache: Remembering What the Model Has Already Processed
Let’s start with the cache that is fundamental to every modern autoregressive LLM inference: the KV cache.
When the LLM generates a response, it doesn’t produce the entire response in one shot. It generates one token at a time autoregressively. For example, if the model is generating the sentence “Quantum computing is a new approach to computation,” the model might generate it approximately as: “Quantum” → “computing” → “is” → “a” → “new” → “approach” → … and so on.
At every generation step, the Transformer uses its attention mechanism to determine how the new token should interact with the tokens that came before it. As part of this attention computation, the model produces Key (K) and Value (V) tensors for the tokens it has processed. These tensors are useful for subsequent tokens because future tokens need to attend to the previous context.
Without caching, the model would repeatedly recompute the K/V representations associated with the earlier tokens from scratch. As the generated sequence becomes longer, this repeated work becomes increasingly expensive and highly time consuming. And no one likes a slow response.
The KV cache solves this by storing those previously computed K/V tensors in memory, typically GPU memory which we here call it KV Cache. When the next token needs to be generated, the model can reuse the cached K/V states instead of recomputing them.
The key idea is simple: compute the K/V states once, store them, and reuse them during next upcoming decoding steps.
Consider a prompt containing “I love LLMs.” During the initial prefill phase, the model processes the prompt and produces K/V states for those tokens. Those states are placed into the KV cache. When the model begins generating the response, the cached states can be reused while the newly generated token contributes its own K/V states.
This is one of the reasons KV caching is so important for autoregressive inference. Instead of repeatedly reconstructing the attention state of the entire conversation at every decoding step, the serving system maintains that state and incrementally appends into that state.
There is, however, an important limitation: a traditional KV cache is generally associated with an active sequence or request. Once that request is finished, its KV state isn’t automatically useful to an unrelated future request. And that leads us to the next technique.
If you want to read about KV Caching and how it works in detail: https://www.analyticsvidhya.com/blog/2025/11/kv-caching-guide/
2. Prefix Cache: Reusing the Beginning of Another Request
In an active LLM request, the KV cache helps the model avoid recomputing tokens it has already processed. But what happens when a completely new request arrives with the same beginning as an earlier request? The model normally has no reason to recompute that shared prefix from scratch—but without prefix caching, that is exactly what happens.
This is where prefix caching comes in.
How Prefix Caching Works
Suppose an application sends the following prompt:
You are an AI assistant for Acme. Follow these company policies…Use these tools when necessary…What is the refund policy?
A second user might send:
You are an AI assistant for Acme.Follow these company policies…Use these tools when necessary…How do I cancel my subscription?
The questions are different, but a large portion of the prompt is identical. The system prompt, policies, instructions, and tool definitions may all be shared.
Instead of processing this entire prefix again, a prefix cache allows the serving system to reuse the KV states that were already computed for the shared portion.
From Tokens to Cache Blocks
Prefix caching typically works by dividing the prompt into fixed-size blocks of tokens. Each completed block corresponds to a portion of the KV cache. For example, imagine a simplified prompt divided into four-token blocks:
The serving system can associate each block with a hash derived from the block’s contents and its position in the prefix. These hashes allow a new request to determine whether the corresponding KV block already exists in the cache. This is important because we don’t want to compare entire prompts character by character every time. Instead, the system can efficiently identify previously computed blocks and determine which portions of the new request can be reused.
Now a New Request Arrives
Consider a second request:
[A B C D] [E F G H] [I J Y Z] [Q R S T]
The first two blocks are identical to the previous request, while the remaining blocks are different.
The cache lookup therefore looks conceptually like this:
Block 0 → CACHE HIT ✓Block 1 → CACHE HIT ✓Block 2 → CACHE MISS ✗Block 3 → CACHE MISS ✗
The serving system can reuse the K and V states for Blocks 0 and 1 instead of recomputing them. Only the uncached portion needs to go through the model’s computation.
Prefix Cache in Action
The following animation visualizes this entire process from splitting the prompt into blocks, hashing them, storing their KV states, finding matching blocks in a new request, reusing cache hits, and finally evicting old blocks when the cache becomes full.
The key part to watch is the transition from CACHE HIT → REUSE. The second request doesn’t need to start from zero: it can pick up from the already-computed KV states of its shared prefix.
What Exactly Is Being Cached?
It is worth making one distinction here. Prefix caching does not simply store the text:
[A B C D]
and return it when the same text appears again.
The useful thing being stored is the model’s computed KV state associated with those tokens. When the prefix is encountered again, those states can be loaded and reused during inference. This is why prefix caching can significantly reduce the amount of prefill computation required for workloads where many requests share a common beginning.
What Happens When the Cache Is Full?
KV cache memory is finite. If the serving system continuously adds new blocks, eventually there will not be enough GPU memory to keep everything. This is where eviction comes into play. A common strategy is LRU (Least Recently Used) eviction. When space is needed, blocks that have not been used recently are removed first, making room for newly computed blocks.
Conceptually:
Cache: [OLD] [OLD] [A] [B] [C] [D] ↑ LRU Need space ↓ Evict old blocks ↓ [NEW] [NEW] [A] [B] [C] [D]
So prefix caching isn’t simply “store everything forever.” A real serving system has to continuously manage which KV blocks are worth keeping and which can be discarded.
What About Images and Multimodal Prompts?
The same idea becomes more interesting with multimodal models.
Consider:
“What is shown in this image?”+ Image A
and later:
“What is shown in this image?”+ Image B
The textual portion is identical, but the image is different. A cache therefore cannot treat the requests as identical simply because their text matches. The multimodal input also needs to be represented correctly when determining whether a cached computation is reusable.
This becomes an important consideration for systems serving vision-language models, where prompts may contain text, images, audio, or other multimodal inputs.
Prefix Cache vs. KV Cache
The two are closely related, but they solve different problems. KV caching primarily helps within an ongoing autoregressive generation: “I’ve already processed these tokens for this request, so don’t recompute their K/V states.”
Prefix caching extends the idea across different requests: “I’ve already processed this exact prefix for another request, so reuse those K/V states.”
KV Cache
Prefix Cache
Scope
Current request
Across requests
Reuses
Previous tokens’ KV states
Previously computed prefix KV blocks
Main benefit
Faster decoding
Faster prefill
Requires same prefix?
Within same sequence
Yes, for cache hits
In systems such as vLLM, prefix caching is implemented using block-based KV-cache management, hashing, cache lookup, and eviction mechanisms. The exact implementation details are more involved than the conceptual model presented here, but the underlying idea remains the same: identify a previously computed prefix and reuse its KV blocks instead of performing the same computation again.
3. Prompt Cache: Letting the LLM Provider Cache the Prompt
Now consider a slightly different scenario from above ones. Instead of hosting the model yourself, you’re using an LLM through an API provider. Your application might repeatedly send a very large system prompt containing documentation, instructions, tool definitions, examples, and other context. The user query only changes every time, but perhaps tens of thousands of tokens of the prompt remains exactly the same in the history.
Processing that repeated context again and again can be wasteful. Some LLM providers therefore offer prompt caching, where frequently reused portions of a prompt can be cached on their infrastructure. When a subsequent request contains the same cacheable content, the provider can reuse the previously processed state rather than treating the entire prompt as new input.
The important point is that the cache is generally managed by the provider. Your application sends the prompt according to the provider’s caching mechanism, while the provider handles storing and reusing the cached representation.
Provider
Model
Cache hit
Cache write
No cache
openai
GPT-5.6 Sol1
0.1x
1.25x
1x
anthropic
Claude Opus 52
0.1x
1.25x (5 min) / 2x (1 hour)
1x
google
Gemini 3.1 Pro3
0.1x + storage fee
1x + storage fee
1x
kimi
Kimi K34
0.1x
1x (automatic)
1x
xai
Grok 4.55
0.15x
1x (automatic)
1x
deepseek
DeepSeek V4 Pro6
0.008x
1x (automatic)
1x
Depending on the provider, prompt caching can reduce both latency and input-processing costs. The exact behavior, cache lifetime, minimum token requirements, and pricing are provider-specific, so these details should always be checked against the particular API you’re using.
At this point, you might be wondering: isn’t prompt caching basically the same thing as prefix caching?
Conceptually, there is indeed a lot of overlap. Both are designed to exploit the repeated prompt content, and both can involve reusing previously computed model state. The difference is primarily in the serving layer and terminology used by the system.
Inference engines and self-hosted serving infrastructure commonly use prefix caching, while LLM providers typically expose prompt caching as an API feature. Rather than thinking of them as two completely unrelated algorithms, it is more accurate to think of them as closely related caching strategies exposed at different layers of the LLM stack.
What Breaks Prompt Caching?
Prompt caching works best when the cacheable portion of the prompt stays stable. Several things can cause cache misses:
- Dynamic tool lists: Adding/removing tools or connecting MCP servers changes the tool definitions and therefore the prompt prefix.
- Dynamic system prompts: Including changing values such as the current time, Git branch, or open files can invalidate the cached prefix.
- Context compaction/summarization: Replacing conversation history with a summary changes the prompt, so the new context may need to be processed again.
- TTL expiry: Cached content can expire. If a user returns after the cache lifetime, the context has to be processed again.
- Non-deterministic serialization: Different JSON key ordering, whitespace, float formatting, etc. can produce different prompt representations and prevent cache matching.
The key idea: prompt caching depends on a stable cacheable prefix. Even small changes can turn a cache hit into a cache miss.
Must follow Stable Prefixes
Prompt caching works best when the beginning of your prompt remains unchanged. A cache hit generally requires the cacheable prefix to match the previous request according to the provider’s matching rules. Change something early in the prefix, and the reusable portion after that point may no longer be available.
- Order content by stability. Put the most stable content first, system instructions, tool definitions, and relatively stable conversation history while keeping volatile information such as the latest tool results or dynamic context toward the end.
- Avoid surprises at the top. Don’t inject timestamps, request IDs, random values, or frequently changing user metadata into the beginning of the prompt. A small change near the start can prevent reuse of a large portion of the cache.
- Prefer append-only history. Avoid rewriting earlier messages whenever possible. If the existing prefix changes, the model may need to process everything after that change again.
4. Semantic Cache: When You Don’t Need the LLM at All
The previous three caching mechanisms are primarily concerned with reusing model computation. Semantic caching takes a different approach. Instead of asking whether we can avoid processing these tokens again, it asks whether we have already answered this question.
Suppose a user asks, “What is the capital of France?” The request goes to the LLM and the model responds, “Paris.” A semantic cache can store this interaction. Later, another user might ask, “Which city is France’s capital?” The two questions are not identical at the text level, but their meanings are extremely similar.
A traditional cache based on exact string matching would treat these as two different queries. A semantic cache instead converts the query into an embedding, which represents the meaning of the text as a vector. The new query can then be compared against embeddings of previously cached queries using a similarity search.
If the similarity exceeds a configured threshold, the system can decide that the new question is sufficiently similar to a previous question. Instead of calling the LLM again, it can return the previously generated answer.
This is why semantic caching can potentially produce much larger savings than the other caches. A successful semantic-cache hit can eliminate the entire LLM inference request.
Of course, this comes with an important trade-off: similar doesn’t always mean equivalent. For example, “What is Apple’s revenue?” and “What was Apple’s revenue in 2025?” are related questions but require different answers. Therefore, a semantic cache needs a carefully chosen similarity threshold and often additional validation logic. An overly aggressive cache can return an answer that is relevant to the question but not actually correct for the specific request.
How the Four Caches Fit Together
These techniques become much easier to understand when we look at them as different layers of optimization rather than four competing caches. A typical conceptual flow starts with a semantic-cache lookup. If there is no sufficiently similar previous answer, the request proceeds toward the model, where repeated prompt prefixes may be reused through prefix or provider-level prompt caching. During inference, the KV cache then helps make autoregressive decoding efficient.
The exact architecture will differ between inference engines and API providers, but the important idea is that multiple caching mechanisms can coexist in the same application. They are not necessarily alternatives to one another.
What Exactly Is Being Cached?
Cache
What is being reused?
Main purpose
KV Cache
Key/Value attention states
Speeds up decoding
Prefix Cache
Computation/KV state for a shared prefix
Avoids repeated prefill
Prompt Cache
Provider-managed prompt processing
Reduces repeated prompt cost/latency
Semantic Cache
Previous query + generated answer
Avoids the LLM call
The first three are therefore mostly about avoiding computation. Semantic caching is about avoiding inference altogether.
A Real-World Example
Consider an AI customer-support agent. Every request might contain a large system prompt with company policies, product documentation, tool definitions, and instructions. The actual user question is appended at the end. Suppose thousands of users interact with this system every day.
The KV cache helps each individual generation by keeping previously computed K/V states available during token-by-token decoding. The prefix cache can then take advantage of the fact that many requests share the same beginning. Instead of repeatedly processing the same system instructions and documentation from scratch, the inference engine can reuse the cached prefix state.
If the application uses an API provider that offers prompt caching, the provider can similarly reuse the repeated prompt content on its infrastructure and potentially reduce the cost and latency associated with processing that repeated context.
Finally, the semantic cache can catch cases where different users ask essentially the same question. If a previous answer is sufficiently similar and safe to reuse, the system can return that answer without invoking the LLM at all.
This means a single application can potentially benefit from multiple caching mechanisms simultaneously.
The Mental Model to Remember
If you’re working with LLM serving, don’t think of caching as a single optimization. Think of it as a series of opportunities to avoid work that has already been done.
- KV Cache asks: “Have I already computed the K/V states for these previous tokens?”
- Prefix Cache asks: “Have I already processed this with the same prompt prefix for another request?”
- Prompt Cache asks: “Can the provider reuse this repeated prompt processing?”
- Semantic Cache asks: “Have I already answered a question with essentially the same meaning?”
The first three primarily help you do less model computation. Semantic caching can help you avoid model computation entirely.
Conclusion
LLM inference costs rise partly because applications repeatedly process the same information. A system prompt may remain unchanged across thousands of requests, a conversation may contain hundreds of previously processed tokens, and users may repeatedly ask essentially identical questions.
Caching lets us exploit these repetitions. At the lowest level, KV caching prevents the model from repeatedly rebuilding attention state during generation. In the request level, prefix caching allows shared prompt computation to be reused across requests. At the API layer, prompt caching allows providers to optimize repeated prompt processing. And at the application layer, semantic caching can recognize that an answer already exists and avoid inference completely.
The deeper idea behind all four is the same: Don’t pay twice for work you don’t need to do twice. Save you money while tokenmaxxing.
GenAI Intern @ Analytics Vidhya | Final Year @ VIT Chennai
Passionate about AI and machine learning, I’m eager to dive into roles as an AI/ML Engineer or Data Scientist where I can make a real impact. With a knack for quick learning and a love for teamwork, I’m excited to bring innovative solutions and cutting-edge advancements to the table. My curiosity drives me to explore AI across various fields and take the initiative to delve into data engineering, ensuring I stay ahead and deliver impactful projects.
Login to continue reading and enjoy expert-curated content.
Keep Reading for Free