4. VRAM, RAM, and Compute Power

Understanding the memory hierarchy behind local AI—and why model size is only the beginning of the story.

When choosing hardware for conventional applications, developers often focus heavily on CPU speed and total system RAM. Local AI workloads change those priorities because neural-network inference repeatedly processes enormous matrices and moves large quantities of data through memory.

For most GPU-accelerated language-model workloads, three hardware characteristics deserve particular attention:

• Memory capacity: Can the model and its runtime state fit?
• Memory bandwidth: How quickly can model data reach the processing cores?
• Compute throughput: How quickly can those cores perform the required mathematics?

These factors interact. A GPU with enormous compute capability cannot run a workload that does not fit in memory, while a GPU with plenty of memory can still be slow if its bandwidth or compute resources are insufficient.

VRAM: Your Fastest Memory Pool

On a discrete GPU, VRAM is the high-speed memory located directly on the graphics card. NVIDIA GPUs, for example, use dedicated GDDR or HBM memory designed to provide far more bandwidth than conventional desktop system RAM.

For local LLM inference, VRAM commonly stores:

• The model’s quantized or full-precision weights
• The KV cache used for active context
• Temporary tensors and intermediate calculations
• Inference-engine workspace
• Multimodal components such as vision encoders

It is therefore misleading to say that a “20GB model requires a 20GB GPU.” If the weights consume nearly all available VRAM, there may be insufficient memory left for the context window and runtime.

A better rule is:

Model weights must fit with enough headroom for everything the inference engine needs while the model is running.

The Model Is Only Part of the Memory Budget

One of the most important concepts in modern LLM hardware planning is that model weights and runtime memory are different things.

Imagine that a quantized model occupies 18GB on disk. Loading it into a 24GB GPU does not necessarily leave a simple 6GB context budget.

The inference engine may also need memory for:

• KV-cache blocks
• CUDA or other backend workspaces
• Temporary activations
• Attention operations
• Sampling buffers
• Additional model components

Always leave memory headroom rather than sizing a system around a model that barely fits.

The KV Cache: The Other Major Memory Consumer

During autoregressive generation, a transformer repeatedly refers to information computed for earlier tokens. Recalculating everything from scratch for every new token would be extremely inefficient.

Instead, inference engines maintain a Key-Value cache, or KV cache.

The KV cache grows as the model processes more tokens. Its exact size depends on the architecture, number of layers, attention design, data type, context length, and number of simultaneous sequences.

This creates an important relationship:

Longer context → larger KV cache → greater memory consumption.

For an AI agent, this matters because contexts can grow quickly. The model may receive system instructions, tool definitions, retrieved documents, conversation history, plans, and previous tool outputs in the same request.

A model that works comfortably with an 8,000-token context may require substantially more memory when operating with tens of thousands of tokens.

KV-Cache Quantization

Model weights are not the only part of inference that can use lower numerical precision.

Some modern inference engines can also quantize the KV cache, reducing the amount of memory required for long contexts and allowing more tokens or concurrent requests to fit into the same accelerator memory.

As with model quantization, there can be trade-offs involving compatibility, performance, and output quality.

This is another reason hardware requirements cannot be calculated accurately from parameter count alone.

System RAM: More Than a Slow Fallback

On a conventional PC, system RAM is attached to the CPU rather than the discrete GPU. DDR4 and DDR5 memory typically provide much less bandwidth than high-end GPU memory.

However, system RAM remains extremely useful for local AI.

Inference engines such as llama.cpp can divide a model between CPU memory and GPU memory. Instead of requiring the entire model to reside in VRAM, selected layers can run on the GPU while the remainder stays in system memory.

This is commonly called CPU/GPU hybrid inference or partial GPU offloading.

It can make a major difference in what models a machine is capable of running.

Example

Suppose you have a model whose weights require approximately 30GB but your graphics card has only 24GB of VRAM.

You might:

• Place as much of the model as possible on the GPU.
• Keep the remaining layers in system RAM.
• Execute different parts of the model on the appropriate processor.

The model can now run without requiring a second GPU.

The trade-off is performance: CPU memory has lower bandwidth, and work performed on the CPU is typically slower than the equivalent GPU execution.

Offloading Does Not Have One Fixed Performance Penalty

It is tempting to say that a model running entirely in VRAM achieves one token rate while an offloaded model always falls to another specific speed. In practice, there is no universal ratio.

Performance depends on:

• The percentage of model layers placed on the GPU
• CPU performance
• System-memory bandwidth
• GPU performance
• PCIe configuration
• Model architecture
• Quantization format
• Context length
• Inference engine

A model with only a few layers remaining on the CPU may still feel reasonably responsive. A model that runs primarily on a slow CPU may be dramatically slower.

For experimentation, this flexibility is extremely useful. For latency-sensitive agents, keeping more of the workload on the accelerator is generally preferable.

Why PCIe Matters

In a conventional desktop workstation, the CPU and discrete GPU have separate memory pools connected through PCI Express (PCIe).

PCIe is fast by ordinary computer standards, but its bandwidth is much lower than the internal memory bandwidth available between a high-end GPU and its own VRAM.

This distinction is important:

• GPU ↔ VRAM: Extremely high bandwidth.
• CPU ↔ System RAM: Lower bandwidth.
• CPU/System RAM ↔ GPU: Must communicate through PCIe.

Workloads that repeatedly move large tensors between system RAM and GPU VRAM can therefore incur significant overhead.

Good inference engines attempt to minimize unnecessary transfers.

Capacity vs Bandwidth

Two graphics cards can have the same amount of VRAM while providing very different LLM performance.

This happens because memory capacity and memory bandwidth measure different things.

Memory capacity answers:

“How much information can I keep on the accelerator?”

Memory bandwidth answers:

“How quickly can I move that information between memory and the processing cores?”

An accelerator with 32GB of VRAM can hold a larger workload than a 16GB accelerator, but that does not automatically mean it will generate tokens twice as quickly.

Why LLM Generation Is Often Memory-Bandwidth Bound

When a language model generates one token at a time for a single interactive user, the accelerator repeatedly needs to access a large portion of the model’s weights.

The amount of arithmetic performed for each weight can be relatively small compared with the amount of data that must be moved from memory.

As a result, single-sequence token generation is often strongly limited by memory bandwidth.

This explains why memory bandwidth is such an important specification when comparing hardware for local LLM inference.

But Compute Power Still Matters

It would be incorrect to conclude that compute performance is irrelevant.

LLM inference has multiple phases with different hardware characteristics.

1. Prompt Processing — Prefill

Before a model generates its first new token, it must process the input prompt.

If your agent sends a long document, thousands of tokens of conversation history, or a large collection of retrieved information, the model has substantial work to perform before output generation begins.

This stage is often called prefill.

Prefill can make much greater use of the GPU’s parallel compute resources than one-token-at-a-time generation.

2. Token Generation — Decode

After the prompt has been processed, the model generates new tokens sequentially.

This stage is commonly called decode.

For a single interactive request, decode frequently becomes more sensitive to memory bandwidth because the model repeatedly streams weights through the accelerator while generating relatively small amounts of new output at each step.

Why This Distinction Matters for Agents

An AI agent may alternate between very different workloads.

For example:

1. The agent receives a 30-page document.
2. The model processes the large input.
3. It generates a short tool request.
4. The tool returns several thousand tokens of information.
5. The model processes that new context.
6. It generates another short action.

The large input-processing phases benefit strongly from compute performance, while the short sequential generation phases may be more sensitive to memory bandwidth.

This is why measuring only “tokens per second” does not completely describe agent performance.

TFLOPs, TOPS, and Marketing Numbers

GPU specifications often advertise TFLOPs or TOPS as measures of computational throughput.

These numbers can be useful, but they must be interpreted carefully.

Different metrics may represent:

• FP32 computation
• FP16 or BF16 tensor operations
• FP8 computation
• FP4 computation
• Integer operations
• Sparse versus dense performance

A GPU advertising an enormous AI TOPS figure does not necessarily deliver proportionally higher LLM generation speed than another GPU.

The model, precision, inference engine, kernel implementation, memory bandwidth, batch size, and workload all influence real-world performance.

Do not compare AI hardware using a single TFLOPs or TOPS number.

Batch Size Changes the Equation

The claim that LLM inference is “memory-bandwidth bound” is particularly relevant to low-batch interactive generation.

When an inference server handles many sequences simultaneously, it can reuse model weights across more calculations and keep the GPU’s compute units busier.

As batch size and concurrency increase, compute throughput becomes increasingly important.

This creates two very different optimization goals:

• Personal local agent: Optimize latency and single-user token generation.
• Production inference server: Optimize total throughput across many simultaneous users.

The best GPU for one scenario is not necessarily the best GPU for the other.

Memory Bandwidth: A Modern Example

Consider two generations of high-end NVIDIA consumer GPUs:

• RTX 4090: 24GB of GDDR6X memory and approximately 1,008 GB/s of memory bandwidth.
• RTX 5090: 32GB of GDDR7 memory and approximately 1,792 GB/s of memory bandwidth.

The newer GPU therefore improves two characteristics that matter greatly for local LLMs: capacity and bandwidth.

However, even this comparison does not predict an exact token-generation improvement. Software optimization, model precision, architecture, context, and workload still matter.

What About Apple Unified Memory?

Apple Silicon uses a different memory architecture.

Instead of giving the CPU one pool of system RAM and the GPU another pool of dedicated VRAM, Apple provides a unified memory pool accessible to both.

This changes the CPU/GPU memory discussion considerably.

A large-memory Apple Silicon system can make far more memory available to GPU workloads than most consumer discrete GPUs, although the operating system and other applications also use that same memory.

Because the architecture is unified, it does not have exactly the same CPU-RAM-to-discrete-GPU-VRAM transfer model as a conventional NVIDIA desktop.

This is one reason Apple Silicon can be attractive for running very large quantized models locally.

Don’t Forget System RAM

Even if you intend to run models primarily on an NVIDIA GPU, system RAM still matters.

It may be used for:

• CPU-offloaded model layers
• Loading and preparing model files
• Vector databases
• Embedding workloads
• Document processing
• Web browsers and development environments
• Agent tools
• Containers and virtual machines

An agent does much more than execute an LLM. The rest of your software stack needs memory too.

This is why a workstation with a powerful GPU but very little system RAM can still become frustrating to use.

A Better Way to Think About AI Hardware

Instead of asking only, “How powerful is this GPU?”, evaluate the complete system.

Diagnosing a Slow Local Model

If your local model is slower than expected, ask these questions in order:

1. Does the complete workload fit in accelerator memory?
2. Is part of the model running on the CPU?
3. Is the context window consuming excessive KV-cache memory?
4. Is the inference engine using the correct GPU backend?
5. What quantization format is being used?
6. Is the bottleneck prompt processing or token generation?
7. Is another application consuming GPU or system memory?
8. Are you optimizing for one user or many concurrent requests?

This diagnostic approach is much more useful than simply comparing GPU model numbers.