2. Local AI Hardware: Minimum & Recommended Specs

A practical guide to choosing hardware for running open-weight models and AI agents locally.

If you plan to run AI agents on hardware you control, choosing the right machine requires more than looking at a model’s parameter count. The most important constraints are usually memory capacity, memory bandwidth, model precision, context length, and expected concurrency.

For local inference, the first question is usually:

“Can the complete inference workload fit into fast memory?”

That workload may include the model weights, the model’s KV cache, multimodal encoders, runtime overhead, and—in production systems—multiple simultaneous requests.

The hardware tiers below are therefore approximate. They are intended to help you choose a practical starting point rather than guarantee that every model of a particular parameter count will fit.

Why Quantization Changes Everything

Models are normally trained and distributed using relatively high-precision numerical formats. For inference, their weights can often be converted to lower-precision representations—a process known as quantization.

For example, a model stored at approximately 4 bits per weight can require dramatically less memory than the same model stored using 16-bit values.

As a rough conceptual estimate:

• 8B parameters at 16-bit: approximately 16GB for weights alone.
• 8B parameters at 4-bit: approximately 4–6GB in a practical quantized format.
• 32B parameters at 4-bit: often roughly 18–22GB depending on the quantization format and architecture.
• 70B parameters at 4-bit: often roughly 40–45GB.

These figures describe model weights, not total runtime memory.

Quantization makes large models practical on much smaller machines, but lower precision can introduce some loss of model quality. Different inference engines also support different quantization formats and hardware combinations.

VRAM Is Not the Whole Story

It is common to hear that “VRAM determines what model you can run.” That is useful shorthand, but it is incomplete.

Your actual memory budget includes several components:

• Model weights: Usually the largest fixed allocation.
• KV cache: Memory used to retain information from previously processed tokens.
• Context length: Longer prompts generally require a larger KV cache.
• Batch size and concurrency: Serving multiple requests can substantially increase memory use.
• Runtime overhead: The inference engine itself requires memory.
• Multimodal components: Vision and audio models may require additional encoders or projectors.

This is why two people running the “same 32B model” can report very different memory requirements.

Context Windows Can Consume Gigabytes

Agent systems often use much larger contexts than ordinary chat applications. A single agent request might contain:

• System instructions
• Tool definitions
• Conversation history
• Retrieved documents
• Previous tool results
• Plans and intermediate state

All of this context must be processed by the model, and the associated KV cache consumes memory.

Therefore, when selecting hardware for agents, do not size the machine so that the model barely fits. Leave memory headroom for the context window and runtime.

Apple Unified Memory vs Dedicated GPU Memory

Apple Silicon takes a different approach from a conventional PC with an NVIDIA or AMD graphics card.

On a traditional GPU workstation, the CPU has system RAM while the GPU has a separate pool of high-speed VRAM. Ideally, most or all of the model runs in GPU memory.

Apple Silicon uses unified memory, meaning the CPU and GPU can access the same memory pool. This allows some Mac configurations to run models whose weights would not fit on a typical consumer GPU.

For example, high-memory Mac Studio systems can offer far more unified memory than the 24GB–32GB found on high-end consumer graphics cards.

However, capacity and speed are different metrics. A machine may have enough memory to load a very large model without delivering the token generation speed of a high-end NVIDIA GPU or multi-GPU inference server.

Choose Apple Silicon primarily when large unified-memory capacity, compact hardware, low complexity, or macOS integration fits your workload—not simply because a model can technically be loaded into memory.

Should You Use Multiple GPUs?

Multiple GPUs can increase the amount of accelerator memory available to a model, but their memory does not behave exactly like one giant pool of VRAM.

The inference engine must divide the model or workload across the GPUs. Performance then depends on factors such as:

• How the model is partitioned
• PCIe bandwidth
• GPU-to-GPU communication
• Motherboard slot layout
• CPU PCIe lanes
• Inference-engine support

Two 24GB GPUs may make it possible to run a model that cannot fit on either card individually, but they do not automatically behave like one 48GB GPU with identical performance characteristics.

For experimentation, multi-GPU consumer workstations can offer a large amount of memory for the money. For production systems, higher-memory professional accelerators may provide a simpler and more reliable architecture.

Why Memory Bandwidth Matters

LLM inference repeatedly moves enormous quantities of model data through memory. For many local inference workloads, performance is therefore strongly influenced by memory bandwidth.

This explains why accelerator specifications that receive less attention in gaming—such as VRAM capacity and memory bandwidth—can matter enormously for LLM inference.

A GPU with more compute cores is not automatically faster for every language model if the workload is primarily limited by how quickly weights can be moved through memory.

Why System RAM Still Matters

Even when you have a dedicated GPU, system memory remains useful.

Some inference engines can split a model between GPU memory and ordinary RAM. This technique—often called CPU offloading or partial GPU offloading—can make it possible to run models larger than your VRAM capacity.

The trade-off is speed. System RAM has much lower bandwidth than modern GPU memory, and data movement across PCIe adds additional overhead.

CPU offloading is therefore excellent for experimentation and occasional large-model use, but it may not provide the responsiveness required for a production agent.

Why Fast Storage Matters

Model files are large. Even relatively modest local-model collections can consume hundreds of gigabytes once you begin storing several models, quantization levels, embedding models, and multimodal components.

A quantized 70B-class model alone can occupy more than 40GB.

Fast NVMe storage helps with:

• Loading models into memory
• Switching between models
• Downloading and unpacking large checkpoints
• Managing multiple quantizations
• Vector databases and retrieval workloads

A conventional HDD can still be used for archival storage, but frequently used models should generally live on an SSD.

Do You Actually Need a GPU?

No. Frameworks such as llama.cpp can run models on CPUs, GPUs, or a mixture of both.

CPU-only inference can be perfectly adequate for:

• Learning how local LLMs work
• Testing orchestration code
• Small background tasks
• Batch processing where latency is unimportant
• Very small quantized models

However, interactive agents make repeated model calls, so slow inference compounds quickly. If an agent needs twenty reasoning or tool-use steps, reducing each step from several seconds to a fraction of that time can dramatically change the user experience.

Buying Hardware: Prioritize in This Order

If the machine is primarily intended for local AI development, a useful purchasing order is:

1. Enough accelerator memory for your target models
2. High memory bandwidth
3. Enough system RAM for offloading and development tools
4. Fast NVMe storage with sufficient capacity
5. CPU performance

For LLM inference, buying a more expensive CPU while sacrificing GPU memory is often the wrong trade-off.