5. Self-Hosting: KV Cache & Context

Understanding the memory cost of long prompts, concurrent users, and persistent agent runs—and how modern inference servers manage it.

When self-hosting an open-weight model, fitting the model’s weights into memory is only the beginning. The inference runtime also needs memory for temporary activations, model buffers, multimodal inputs, and—most importantly for long conversations—the key-value cache, usually shortened to KV cache.

The KV cache allows an autoregressive model to reuse computations from tokens it has already processed. It is essential for efficient text generation, but it can consume a substantial amount of GPU or unified memory as prompts become longer and the number of simultaneous requests increases.

For an agent, the practical challenge is not simply “How large is the model?” It is:

How many tokens will each active run contain?
How many runs will be served concurrently?
Which attention architecture does the model use?
At what precision will the KV cache be stored?
How efficiently does the inference runtime allocate and reuse cache memory?

The Context Window

The context window is the maximum sequence length that a model and serving configuration can process in one request. It includes more than the visible user message.

A request may contain:

Developer and system instructions.
The user’s messages.
Previous assistant responses.
Tool definitions.
Tool calls and tool results.
Retrieved documents.
Images or other multimodal inputs represented internally as tokens.
Tokens generated in the current response.

If a model supports a maximum sequence length of 128,000 tokens, that does not necessarily mean the application can submit 128,000 input tokens and then generate an unlimited answer. The input and generated output generally share the available sequence budget.

A simplified relationship is:

input tokens + generated tokens ≤ configured maximum sequence length

The exact rules depend on the model and runtime.

📐 Tokens Are Not Words

A token may be a complete word, part of a word, punctuation, whitespace, code fragment, or another unit selected by the model’s tokenizer. The number of words represented by a fixed token budget varies by language, writing style, code content, and tokenizer.

Rules such as “8,000 tokens equals 6,000 words” are rough English-language estimates, not reliable capacity guarantees. Count tokens using the tokenizer for the actual model whenever possible.

Models Are Stateless, but Applications Are Not

A model call does not inherently remember earlier, separate requests. The application must provide the information needed for the current decision, either by sending it again or by using a serving mechanism that retains reusable inference state.

This does not mean that a well-designed agent must resend every historical message forever. The application can maintain authoritative state outside the model and construct a smaller, task-relevant context for each step.

That external state might include:

The current objective.
A structured plan or task record.
Completed actions.
Important tool results.
Generated files and artifact references.
Pending approvals.
Errors and retry history.
Relevant retrieved documents.

The context supplied to the model is therefore best understood as a temporary working view of the agent’s state—not the complete or authoritative record of everything that has happened.

Prompt Processing and Token Generation

Inference has two important phases:

Prefill: The model processes the input prompt and constructs attention state for its tokens.
Decode: The model generates new tokens one at a time while reusing the previously computed state.

Long prompts make the prefill phase more computationally expensive. Long generated responses extend the decode phase and grow the KV cache further.

This distinction explains why a server may process a short chat response quickly but take much longer to begin answering after receiving a large document. The runtime must first process that document during prefill.

What Is the KV Cache?

Most current autoregressive language models use transformer attention. For each previously processed token, attention layers calculate internal key and value tensors.

Without caching, the model would repeatedly recalculate those tensors for all earlier tokens every time it generated a new token. The KV cache stores them so that each new decoding step can reuse the earlier results.

Conceptually:

The model processes the prompt.
Each attention layer stores key and value tensors for the prompt tokens.
The model generates a new token.
The new token’s key and value tensors are added to the cache.
The process repeats until generation stops.

The cache accelerates generation, but its size generally grows with the number of cached tokens.

What Determines KV-Cache Size?

KV-cache requirements depend on the model’s architecture, not simply its total parameter count.

Important factors include:

Number of cached tokens: Longer sequences require more cache memory.
Number of layers: Each attention layer may maintain its own key and value tensors.
Number of key-value heads: Models using fewer KV heads can require substantially less cache memory.
Head dimension: Wider attention representations consume more memory.
Cache precision: FP16, BF16, FP8, INT8, INT4, and other formats have different storage requirements.
Attention type: Full, grouped-query, multi-query, sliding-window, local, and chunked attention have different cache characteristics.
Batch size and concurrency: Each active sequence needs its own non-shared continuation state.
Beam search or multiple candidates: Several active branches may multiply cache use.

For a conventional transformer layer, a simplified approximation is:

KV bytes per token ≈ 2 × layers × KV heads × head dimension × bytes per value

The factor of two represents the key and value tensors. This is only a starting estimate. Hybrid architectures, latent attention, cache compression, alignment requirements, and runtime metadata can change the actual result.

🧮 Parameter Count Is Not Enough

Two models with the same parameter count can have very different KV-cache requirements. A model using grouped-query or multi-query attention may need far less cache memory than a model with the same number of query and key-value heads.

Use the model configuration and serving runtime’s memory estimator rather than assuming that all 8B, 70B, or 400B models behave alike.

Grouped-Query and Multi-Query Attention

Earlier transformer designs commonly used a separate set of key-value heads for every query head. Many newer models use:

Grouped-query attention: Several query heads share one set of key-value heads.
Multi-query attention: All query heads share a much smaller set of key-value representations.

These architectures reduce KV-cache storage and memory bandwidth requirements, which can improve long-context and high-concurrency serving.

This is one reason that context-memory calculations should be based on the model’s actual attention configuration rather than its headline parameter count.

Full Attention, Sliding Windows, and Hybrid Models

Not every layer must retain attention state for every earlier token.

Some models use sliding-window attention, in which selected layers attend only to a recent window of tokens. Once that window is full, the cache for those layers can stop growing with the total sequence length.

Other models combine:

Full-attention layers.
Local or sliding-window layers.
Chunked attention.
State-space or recurrent components.
Other compressed or latent attention mechanisms.

These hybrid architectures can reduce the memory growth associated with long contexts, but they do not eliminate context costs. Full-attention layers may still grow with sequence length, and processing a very long prompt still requires compute.

The Context Limit and KV Capacity Are Different

Several limits are often confused:

Model context limit: The sequence length for which the model was designed or configured.
Server maximum sequence length: The maximum request length allowed by the inference server.
Allocated KV capacity: The amount of memory currently made available for active sequence state.
Application context policy: The limit imposed by the application for cost, latency, or quality reasons.

A model may advertise a very large context window while the local server is configured for a smaller value because the available hardware cannot support the full length at an acceptable concurrency.

Conversely, configuring a very high maximum context length does not mean every request immediately consumes the full theoretical cache allocation. This depends on the runtime and cache-management strategy.

Dynamic and Static Cache Allocation

Inference frameworks commonly provide different cache-allocation strategies.

Dynamic Cache

A dynamic cache grows as the sequence grows. This avoids reserving the maximum capacity for short requests, but changing tensor shapes can be less convenient for some compilers and execution backends.

Static Cache

A static cache reserves capacity up to a configured maximum length. This can provide predictable memory use and may enable compilation or graph optimizations, but unused capacity can waste memory when most requests are short.

Static allocation does not inherently make a server immune to out-of-memory failures. The combined requirements of weights, cache capacity, temporary workspaces, concurrent requests, and other processes must still fit within the available memory.

Paged KV-Cache Allocation

Modern high-throughput inference servers often divide KV memory into fixed-size blocks or pages rather than assigning each request one large contiguous memory region.

This approach can:

Reduce memory fragmentation.
Allocate cache capacity as a sequence grows.
Reuse freed blocks efficiently.
Support many requests with different sequence lengths.
Make scheduling and preemption easier.
Enable sharing of identical cached prefixes.

vLLM’s PagedAttention is a prominent example. It manages KV data in blocks that can be stored non-contiguously, conceptually resembling virtual-memory paging.

🧱 Paged Allocation Changes the Failure Mode

With a block-based server, a single long request does not necessarily allocate one enormous contiguous cache region. However, the physical memory remains finite. When demand exceeds capacity, the server must queue, preempt, swap, evict reusable cache blocks, reject work, or reduce concurrency.

Concurrency Is Often the Real Memory Wall

For a local interactive chat, one active sequence may be the primary KV-cache consumer. In a production server, the larger challenge is usually the combined cache of many simultaneous requests.

A simplified capacity estimate is:

total active KV memory ≈ KV bytes per token × cached tokens across all active sequences

For example, ten concurrent requests averaging 16,000 cached tokens can require roughly the same KV storage as one request containing 160,000 cached tokens, before considering scheduling and implementation overhead.

Actual serving capacity also depends on whether sequences share prefixes, whether some requests are waiting rather than active, and how the server batches prompt processing and generation.

Continuous Batching

Traditional static batching waits for a group of requests and processes them together. This is inefficient for generation because different responses finish at different times.

Continuous batching allows a serving runtime to add new requests and remove completed requests between decoding iterations. This improves accelerator utilization and throughput, especially when serving users with different prompt and response lengths.

However, higher throughput can increase aggregate KV-cache demand because more sequences remain active at once. Server configuration must balance:

Throughput.
Time to first token.
Per-token latency.
Maximum sequence length.
Concurrent request capacity.
Available KV-cache memory.

Prefix Caching

Many requests share the same initial content, such as:

A long system prompt.
Tool definitions.
A policy document.
A common few-shot example set.
The initial portion of a conversation.

Prefix caching allows the runtime to reuse the KV state for an identical prefix instead of recomputing it for every request.

This can reduce prefill latency and duplicated computation. In some serving systems, identical prefixes can also share physical KV blocks until their sequences diverge.

Prefix caching has important limitations:

The prefix must match according to the runtime’s caching rules.
Changing a timestamp, user identifier, tool ordering, or whitespace may prevent reuse.
The cache consumes finite memory and may be evicted.
It does not reduce the cost of generating new continuation tokens.
It does not increase the model’s logical context limit.

⚡ Prefix Caching Saves Compute, Not Unlimited Context

Reusing a cached system prompt can make repeated requests faster, but the cached tokens still form part of the sequence seen by the model. Prefix caching does not allow a request to exceed the model’s maximum context length.

Session Caching and Conversation Reuse

A local runtime may retain KV state between turns in a conversation. When the user adds a new message, the runtime can reuse the unchanged prefix and process only the newly appended tokens.

This is efficient while the prompt remains append-only. Reusing the cache becomes more complicated when the application:

Edits an earlier message.
Inserts retrieved context in the middle of the prompt.
Changes the system instructions.
Reorders tool definitions.
Removes old messages.
Creates a new summary that replaces earlier history.

After the first changed token, subsequent cached state may need to be discarded and recomputed unless the runtime already has a matching cached branch.

KV-Cache Quantization

The model weights and the KV cache can use different numerical formats. A model might use 4-bit quantized weights while retaining its KV cache in FP16 or another higher-precision format.

KV-cache quantization stores keys and values at lower precision, potentially reducing memory requirements and increasing the number of tokens or concurrent requests that fit on the hardware.

Possible formats depend on the runtime and accelerator and may include FP8, INT8, INT4, or other quantized representations.

The trade-offs can include:

Additional quantization and dequantization work.
Possible reductions in generation quality.
Different levels of hardware support.
Runtime-specific compatibility constraints.
Less benefit for short sequences where cache memory is not the bottleneck.

Cache quantization should be tested against representative long-context tasks. A configuration that preserves quality for casual chat may behave differently for exact retrieval, code generation, or long agent trajectories.

KV-Cache Offloading

Some runtimes can move part of the KV cache from accelerator memory to CPU memory or another storage tier.

Offloading can make longer contexts or higher concurrency possible when GPU memory is limited. The cost is additional data movement, which may increase latency and reduce throughput.

Offloading is therefore a capacity strategy rather than free memory. Its usefulness depends heavily on:

CPU memory capacity.
Host-to-device bandwidth.
Interconnect latency.
Batching strategy.
How often the offloaded cache must be accessed.
The expected performance target.

Out-of-Memory Failures Are Not Always Sudden Cache “Explosions”

KV-cache use generally grows predictably with active sequence length and concurrency. An out-of-memory error may occur when the combined memory demand exceeds capacity, but production runtimes can respond in several ways:

Queue the request until capacity becomes available.
Reject a request that exceeds configured limits.
Preempt or pause a lower-priority sequence.
Recompute evicted cache state later.
Swap cache blocks to CPU memory.
Reduce the batch size.
Stop generation before the maximum requested output.
Return a controlled resource-exhaustion error.

A raw CUDA out of memory crash usually indicates that resource limits, admission control, runtime configuration, or error handling need improvement. A well-operated service should fail predictably rather than allowing one request to terminate the entire process.

⚠️ Long Context Can Fail Before VRAM Is Exhausted

A request may remain technically within the memory limit but still be impractical because prefill latency, attention compute, cache bandwidth, or response quality becomes unacceptable.

“Fits in memory” is not the same as “meets the application’s latency, throughput, and accuracy requirements.”

Long Context Is Not Perfect Memory

A large advertised context window does not guarantee that the model will use every token equally well. Performance may decline when:

The relevant information is buried among large amounts of unrelated text.
Several sources contain conflicting facts.
The prompt contains repeated or distracting instructions.
The agent must track many similar identifiers.
Important information appears far from the current decision.
The model has not been evaluated at the configured sequence length.

Long-context capability should therefore be tested on the actual tasks your agent performs. Do not assume that increasing the server limit automatically improves the result.

Strategies for Managing Agent Context

Long-running agents should treat context as a managed resource rather than an append-only transcript.

1. Set Explicit Input and Output Budgets

Configure limits for:

Maximum input tokens.
Maximum generated tokens.
Maximum combined sequence length.
Maximum tokens retrieved per tool call.
Maximum tool-output size.
Maximum tokens retained in the working history.

Reserve space for the model’s response rather than allowing retrieved material to consume the entire context window.

2. Use Token-Aware Admission Control

Before accepting a request, estimate whether the server has sufficient capacity for:

The prompt.
The requested output.
Current active sequences.
Temporary prefill memory.
The selected cache precision.

For shared servers, limits should consider aggregate demand rather than evaluating each request in isolation.

3. Compact Tool Results

Raw tool output can dominate an agent’s context. A web page, database dump, compiler log, or API response may contain far more information than the model needs.

Tools should return:

Structured fields.
Relevant excerpts.
Stable identifiers.
Source references.
Explicit error types.
Pagination or continuation tokens.

Store the full result externally and place only the relevant subset in the model’s working context.

4. Maintain Structured Task State

Do not rely on the model to reconstruct the complete task from a long chat transcript. Maintain a structured state object containing items such as:

The objective.
Constraints.
Completed steps.
Unresolved questions.
Important facts and their sources.
Artifact locations.
Pending approvals.
Remaining resource budget.

This state can be rendered into a compact context block for each model call.

5. Use Selective Truncation

When history must be shortened, remove low-value content rather than blindly deleting the oldest messages.

Potential candidates include:

Verbose acknowledgements.
Repeated instructions.
Obsolete plans.
Large raw tool results already stored elsewhere.
Failed attempts whose lessons have been captured in structured state.
Intermediate assistant prose that is no longer operationally relevant.

Preserve governing instructions, unresolved user requirements, important evidence, active commitments, and identifiers needed for subsequent tool calls.

6. Summarize With Verification

Summarization can compress a long trajectory, but it is lossy. A summary may omit a constraint, alter a number, or remove the provenance of a fact.

A durable checkpoint should preferably contain structured fields such as:

Confirmed facts.
Unverified assumptions.
Decisions made.
Actions completed.
Actions still pending.
Errors encountered.
Source and artifact references.
Exact values that must not be paraphrased.

Important facts should be validated against authoritative state rather than trusted solely because they appear in a model-generated summary.

7. Use Retrieval Instead of Permanent Prompt Stuffing

Large document collections should normally remain outside the prompt. Retrieve only the material relevant to the current decision.

Retrieval can use:

Keyword or full-text search.
Embedding-based semantic search.
Metadata filters.
SQL queries.
Knowledge graphs.
Application-specific indexes.
Hybrid combinations of these approaches.

Retrieval-augmented generation, or RAG, does not require a vector database. The correct retrieval system depends on the structure of the information and the queries the agent must answer.

8. Retrieve in Stages

Instead of inserting many complete documents, an agent can:

Search titles, metadata, or short passages.
Select the most promising source.
Retrieve a larger section from that source.
Extract the exact evidence needed.
Store citations or identifiers in structured state.

This reduces context use and makes the evidence trail easier to inspect.

9. Use External Artifacts

For long-running tasks, store intermediate work as files, database records, code branches, notebooks, or other artifacts rather than repeatedly embedding the entire content in the prompt.

The context can contain a concise inventory such as:

{
"artifacts": [
{
"id": "research_notes_v3",
"type": "document",
"location": "/workspace/research-notes.md",
"status": "verified"
},
{
"id": "analysis_output",
"type": "dataset",
"location": "/workspace/results.parquet",
"status": "generated"
}
]
}

The agent can retrieve the relevant portion when it needs to inspect or modify an artifact.

10. Start a Fresh Context at Stable Checkpoints

For extended tasks, it may be more reliable to begin a new model session after a verified checkpoint rather than preserving an increasingly noisy transcript.

The new context can include:

The original objective.
Current structured state.
A verified progress summary.
References to external artifacts.
Outstanding tasks.
Current permissions and limits.

This sacrifices direct KV-cache reuse for a cleaner and often smaller working context.

Context Editing and Cache Reuse Can Conflict

Removing, reordering, or summarizing earlier messages reduces the logical context, but it may invalidate the existing KV cache after the point where the prompt changed.

This creates a trade-off:

Keep the prompt append-only: Maximize cache reuse, but allow the context to grow.
Compact or rewrite the prompt: Reduce future context size, but pay for a new prefill.

For long-running agents, periodic compaction is often worthwhile despite the recomputation cost. The correct interval depends on prompt length, cache capacity, model quality, and how frequently the agent reuses the session.

Configuring a Local Inference Server

Exact settings differ among llama.cpp, Ollama, vLLM, SGLang, MLX, TensorRT-LLM, and other runtimes, but the same planning questions apply.

Before deployment, determine:

The model’s supported and validated context length.
The server’s configured maximum sequence length.
The cache data type.
How much accelerator memory is reserved for weights and KV blocks.
The maximum number of active sequences.
Whether prefix caching is enabled.
Whether cache offloading or swapping is available.
How the runtime handles preemption and resource exhaustion.
The maximum prompt and output length exposed to clients.
Whether multimodal inputs require additional memory.

A setting such as --ctx-size or --max-model-len normally defines a sequence limit or cache-planning parameter. It should not be described as a universal guarantee that a process cannot encounter an out-of-memory condition.

Capacity Testing

Do not validate a self-hosted agent using only a short, single-user conversation. Test realistic worst cases.

A useful load test should vary:

Prompt length.
Requested output length.
Number of concurrent users.
Shared versus unique prefixes.
Tool-output size.
Cache precision.
Batching and scheduling settings.
Long-running versus newly arriving requests.
Multimodal inputs where supported.

Measure:

Time to first token.
Prompt-processing throughput.
Generation throughput.
Per-request latency.
KV-cache utilization.
Queue time.
Preemption and eviction frequency.
Out-of-memory or rejected-request rate.
Task quality at different context lengths.

📊 Plan for Tokens in Flight

For a shared inference service, the most useful capacity measure is often not the maximum context of one request. It is the number of tokens in flight across all active prompts and generations while meeting the required latency target.

A Practical Self-Hosting Checklist

Leave memory headroom beyond the model weights.
Calculate KV requirements from the actual model configuration.
Budget for concurrency, not only one maximum-length request.
Set separate limits for prompt length and generated output.
Use block-based or paged cache management where appropriate.
Enable prefix caching for stable, commonly repeated prefixes.
Test lower-precision KV caches before relying on them in production.
Use offloading only after measuring its latency impact.
Store authoritative task state outside the model context.
Compact large tool results before returning them to the model.
Use retrieval and external artifacts instead of permanent prompt stuffing.
Apply admission control and return controlled resource errors.
Load-test long prompts and concurrent agent runs.
Monitor cache utilization, queueing, latency, and failed allocations.

The Updated Mental Model

The original simplified view was:

Model weights + growing conversation = VRAM exhaustion

A more accurate modern view is:

Weights + active KV blocks + runtime buffers + concurrency + attention architecture + cache precision + scheduling policy = serving memory requirement

Context management is also not simply deleting old chat messages. It is the engineering discipline of deciding which information belongs in:

The current model context.
A reusable prefix cache.
Structured task state.
A retrieval system.
A persistent application database.
An external file or artifact.
An audit log.

A reliable agent keeps each type of information in the system best suited to manage it.