2. Understanding Model Parameters

Decoding model names such as 8B, 35B-A3B, and 675B-A41B—and understanding what they mean for capability, memory, speed, and agent deployment.

When browsing open-weight models, you will frequently see numbers attached to model names, such as Ministral 3 8B, Gemma 3 27B, Qwen3.5-35B-A3B, or Mistral Large 3 675B-A41B. The letter B means billion and usually refers to the number of trainable parameters in the model.

Parameter counts remain useful, but they no longer tell the whole story. Modern models differ in architecture, training data, post-training, context length, reasoning methods, quantization, multimodal capabilities, and the number of parameters used for each token. A larger parameter count does not automatically mean that a model will be more capable—or more suitable for your agent.

What Is a Parameter?

A language model is a large mathematical function implemented as a neural network. Inside that network are layers containing numerical values called parameters. Most parameters are weights that determine how strongly different pieces of information influence one another as data passes through the network.

During training, an optimization process repeatedly adjusts these parameters so that the model becomes better at predicting tokens and performing the other objectives included in its training process. Post-training may then teach the model to follow instructions, reason through problems, use tools, process images, or behave according to human preferences.

The final parameters encode patterns learned during training, but it is misleading to describe them as a simple database of facts. Knowledge and capabilities are distributed across the network, and the model reconstructs responses dynamically rather than retrieving complete sentences from individual parameters.

When you download an open-weight model, most of the download consists of files containing these trained numerical values, together with configuration files and tokenizer information.

Dense Models and Mixture-of-Experts Models

Understanding a model’s architecture is now as important as knowing its headline parameter count.

Dense Models

In a dense model, nearly all of the model’s parameters participate in processing each generated token. A model described simply as “8B” will therefore use roughly eight billion parameters for each token, although some components may operate differently depending on the architecture.

Examples of current dense model families include:

Ministral 3: Available as dense 3B, 8B, and 14B models.
Gemma 3: Available in sizes including 1B, 4B, 12B, and 27B.
Many compact specialist models: Often designed for local assistants, extraction, classification, coding, vision, or edge deployment.

Mixture-of-Experts Models

A mixture-of-experts, or MoE, model contains multiple groups of parameters called experts. For each token, a routing system activates only a subset of those experts.

Model names may therefore show two different numbers:

Total parameters: The complete number of parameters contained in the model.
Active parameters: The approximate number used to process each token.

For example, Qwen3.5-35B-A3B contains approximately 35 billion total parameters but activates about 3 billion for each token. Similarly, Qwen3.5-397B-A17B contains approximately 397 billion total parameters while activating about 17 billion per token. Mistral Large 3 contains approximately 675 billion total parameters and uses about 41 billion active parameters per token.

🧩 Total Parameters vs Active Parameters

Active parameter count is useful for estimating the amount of computation required for each token. However, the total parameter count is generally more relevant when estimating how much storage and aggregate memory are needed to load the complete model.

An MoE model with 397B total and 17B active parameters may have per-token computation closer to a much smaller dense model, but it does not normally fit into the memory required by a 17B model. Serving systems can distribute, offload, or selectively load experts, but doing so introduces additional engineering and performance trade-offs.

Does a Larger Parameter Count Mean a Smarter Model?

Within the same model family and training recipe, larger variants often perform better because they have more capacity. Across different model families, however, parameter count is a poor standalone predictor of quality.

A smaller, newer model may outperform a much larger, older model because of improvements in:

Training data quality and coverage.
Model architecture and attention mechanisms.
Training duration and compute efficiency.
Instruction tuning and reinforcement learning.
Reasoning-oriented post-training.
Tool-use and structured-output training.
Tokenizer design and multilingual coverage.
Retrieval, prompting, and the surrounding agent system.

Benchmark scores also do not guarantee that a model will work well in your application. The most meaningful comparison is an evaluation using the prompts, tools, documents, languages, failure cases, and latency constraints found in your actual workflow.

A More Useful View of Model Sizes

The following categories are approximate. They describe typical deployment patterns rather than fixed capability boundaries.

Very Small Models: Below Approximately 3B Parameters

Very small models are increasingly useful for focused tasks, including:

Classification and intent detection.
Query rewriting and routing.
Entity and field extraction.
Autocomplete and short-form generation.
On-device or edge applications.
Filtering information before it is sent to a larger model.

These models may run on phones, embedded devices, laptops, CPUs, integrated GPUs, or modest discrete GPUs, depending on the runtime and quantization format. They can be highly effective when the task is narrow and outputs are tightly constrained.

Small Local Models: Approximately 3B to 14B

This range includes models such as Ministral 3 3B, 8B, and 14B and smaller Gemma variants. Quantized versions can often run on consumer laptops and gaming PCs.

Well-trained models in this range can support:

Summarization and rewriting.
Retrieval-augmented question answering.
Document extraction and transformation.
Routine coding assistance.
Simple tool calling.
Local conversational assistants.

Their limitations become more visible in unfamiliar domains, ambiguous instructions, long autonomous workflows, difficult planning, and tasks requiring consistent recovery from errors.

Medium Models: Approximately 20B to 70B

This category includes dense models such as Gemma 3 27B as well as MoE models whose total and active parameter counts may fall into different categories.

Models in this range can offer a strong balance of capability, customization, and deployment cost. With suitable post-training, they may perform well at coding, tool use, document analysis, multilingual work, and multi-step reasoning.

Running them locally usually requires a high-memory workstation, one or more powerful GPUs, or a computer with substantial unified memory. Quantization makes local deployment more practical, but the desired context length and generation speed remain important constraints.

Large and Frontier-Scale Open-Weight Models

Modern large open-weight models may contain hundreds of billions of total parameters. Many use sparse MoE architectures so that only part of the model is active for each token.

Examples include:

Qwen3.5-122B-A10B: Approximately 122B total parameters and 10B active parameters.
Qwen3.5-397B-A17B: Approximately 397B total parameters and 17B active parameters.
Mistral Large 3: Approximately 675B total parameters and 41B active parameters.

These models generally require multi-GPU servers, distributed inference, expert parallelism, aggressive quantization, parameter offloading, or specialist hosting platforms. Although active parameter counts can reduce per-token computation, storing and serving all of the experts remains a major infrastructure challenge.

Parameter counts for proprietary models are often not disclosed. Avoid treating unofficial estimates about commercial models as established facts: architecture, active parameter count, training compute, and serving optimizations may all be unknown.

🧠 There Is No Universal “Agent Parameter Threshold”

Agent reliability does not begin at a fixed size such as 30B. Current small models can successfully call tools when the task is narrow, the tool schemas are clear, and outputs are validated. Larger or more capable models are usually more resilient when instructions are ambiguous, plans are long, tools return unexpected results, or the agent must recover from its own mistakes.

For production agents, evaluate the model’s actual success rate rather than assuming that a particular parameter count guarantees reliable reasoning.

Memory Requirements: A Better Rule of Thumb

The memory required to store model weights depends primarily on the number of parameters and the number of bits used to represent each parameter.

A useful first approximation is:

Weight memory in bytes ≈ number of parameters × bits per parameter ÷ 8

Using decimal gigabytes, one billion parameters requires approximately:

2 GB at 16-bit precision before runtime overhead.
1 GB at 8-bit precision before runtime overhead.
0.5 GB at 4-bit precision before runtime overhead.

This gives the theoretical storage required for the weights alone. Real deployments normally need additional memory for quantization metadata, model buffers, temporary tensors, the inference engine, multimodal components, and the key-value cache used to hold context.

Approximate Weight Memory

Model Size	16-bit Weights	8-bit Weights	4-bit Weights
3B	About 6 GB	About 3 GB	About 1.5 GB
8B	About 16 GB	About 8 GB	About 4 GB
14B	About 28 GB	About 14 GB	About 7 GB
27B	About 54 GB	About 27 GB	About 13.5 GB
70B	About 140 GB	About 70 GB	About 35 GB
122B	About 244 GB	About 122 GB	About 61 GB
397B	About 794 GB	About 397 GB	About 199 GB

These figures cover only the raw parameter values. A practical deployment needs additional headroom, and actual quantized file sizes vary by format and quantization method.

Why Context Length Also Consumes Memory

Loading the weights is only part of the memory requirement. During generation, transformer-based models commonly maintain a key-value cache, or KV cache, containing information about previously processed tokens.

The KV cache grows with:

The number of tokens in the prompt and generated response.
The number of simultaneous requests or conversations.
The model’s layers, attention dimensions, and architecture.
The precision used for the cache.
Whether the model uses techniques such as grouped-query or multi-query attention.

This means that a model that fits comfortably with a short prompt may run out of memory when asked to process a very long document or serve several users at once. Advertised maximum context length does not mean that the full context can be used efficiently on every hardware configuration.

VRAM, Unified Memory, and System RAM

For GPU inference, VRAM is usually the fastest place to hold model weights and the KV cache. However, it is not correct to say that any model spilling into system memory becomes useless.

Discrete GPUs: Keeping the complete workload in VRAM generally provides the best performance. Moving data repeatedly across the CPU–GPU connection can create a severe bottleneck.
Apple silicon and other unified-memory systems: The CPU and GPU use a shared memory pool, which can make large local models practical without separate VRAM. Performance still depends on memory bandwidth and the amount of memory available to the operating system.
CPU-only inference: Quantized models can run entirely in system RAM. Generation is often slower than on a capable GPU, but it may still be suitable for offline, low-volume, or privacy-sensitive workloads.
Hybrid offloading: Some runtimes place selected layers or experts on the GPU and keep the remainder in system memory. This trades speed for the ability to run a larger model.

A reported speed such as “one token per second” cannot be predicted from parameter count alone. It depends on the model architecture, hardware, memory bandwidth, quantization, context length, batch size, inference engine, and the number of model layers assigned to each device.

Parameters Do Not Determine the Full Hardware Requirement

Before downloading or deploying a model, consider all of the following:

Total parameter count: Drives weight storage and aggregate memory requirements.
Active parameter count: Helps indicate the computation performed per token in an MoE model.
Precision and quantization: Strongly affect memory use, speed, and sometimes quality.
Context length: Longer contexts increase KV-cache memory and processing time.
Architecture: Dense, MoE, recurrent, hybrid-attention, multimodal, and other architectures have different serving characteristics.
Batch size and concurrency: Serving multiple users requires substantially more memory than running one local chat.
Prompt processing speed: Reading a long prompt and generating new tokens place different demands on the hardware.
Inference software: llama.cpp, MLX, Transformers, vLLM, SGLang, TensorRT-LLM, and other runtimes may produce very different results on the same hardware.

Selecting a Model for an Agent

Do not choose an agent model based on parameter count alone. Start by defining the tasks the agent must perform and measuring candidate models against representative examples.

For each model, test:

Whether it chooses the correct tool.
Whether it supplies valid tool arguments.
Whether it follows the required output schema.
Whether it uses retrieved evidence correctly.
Whether it notices and recovers from tool errors.
Whether it can maintain the task state over several steps.
Whether its latency and memory use meet your requirements.
The percentage of tasks completed successfully without human repair.

A small model may be the best option for a high-volume routing or extraction step, while a larger reasoning model is used only for difficult decisions. This multi-model approach can be more efficient than assigning every stage of an agent workflow to the largest model available.

📏 The Most Useful Metric

For an agent, the most useful unit of comparison is not parameters, tokens per second, or price per million tokens in isolation. It is the cost, latency, and resource use per successfully completed task.