4. Model Formats: GGUF, AWQ, EXL2

Quantisation is one of the key technologies that makes modern AI models practical to run locally. Understanding model formats, quantisation levels, and inference engines will help you choose the right model for your hardware.

When an AI organisation releases a model, its weights are often provided in a relatively high-precision format such as FP16 or BF16. A 70-billion-parameter model stored using 16-bit weights requires roughly 140 GB just for the model weights, before accounting for the context, KV cache, runtime overhead, and other memory requirements.

That is considerably more memory than is available on most consumer GPUs.

The solution is quantisation: representing the model’s weights using fewer bits. Instead of storing every weight with 16 bits of numerical precision, quantised models may use 8-bit, 6-bit, 5-bit, 4-bit, or even lower-precision representations.

Quantisation can dramatically reduce memory requirements while usually retaining much of the original model’s capability. The trade-off is that more aggressive quantisation can introduce greater quality loss, and the effect varies between models and tasks.

🧮 A Simple Memory Estimate

A rough first approximation is:

Model memory ≈ Parameters × Bits per parameter ÷ 8

For a 70B model:

FP16: approximately 140 GB
8-bit: approximately 70 GB
6-bit: approximately 52.5 GB
4-bit: approximately 35 GB

These figures are only approximations. Real memory consumption is higher because the inference runtime also needs memory for the KV cache, temporary buffers, metadata, context, and other runtime structures.

Format and Quantisation Are Different Things

One of the most important distinctions to understand is that a model format and a quantisation method are not the same thing.

A format defines how model data and metadata are stored and represented for an inference system. Quantisation describes how the numerical weights have been reduced in precision.

For example:

GGUF └── Q4_K_M safetensors └── BF16 AWQ └── 4-bit weights GPTQ └── 4-bit weights

Consequently, when choosing a model you need to consider both how the model is stored and how the weights have been quantised.

Why There Are So Many Formats

Different inference engines have different requirements and are optimised for different hardware.

A format that works extremely well for a CPU-oriented runtime may not be the best choice for a high-throughput NVIDIA GPU server.

The important question is therefore not:

“Which format is the best?”

It is:

“Which format and quantisation are best supported by the inference engine and hardware I intend to use?”

Flexible Local Inference

GGUF

GGUF is a model file format developed for the llama.cpp ecosystem. It has become one of the most widely used formats for local LLM inference.

GGUF files can contain model weights together with metadata such as tokenizer information and chat-template information, making them convenient for local inference applications.

Best For: CPU inference, Apple Silicon, consumer systems, and mixed CPU/GPU inference.
The Advantage: It is highly portable and is supported by a large ecosystem of local inference applications.
Hardware Flexibility: Depending on the runtime, model layers can be distributed between CPU and GPU when the model does not fit entirely into dedicated GPU memory.
Common Quantisations: You may encounter names such as Q4_K_M, Q5_K_M, Q6_K, and Q8_0.

GGUF is particularly attractive for beginners because it provides a relatively straightforward route from a downloaded model file to local inference.

GPU-Oriented Quantisation

AWQ

AWQ (Activation-aware Weight Quantization) is a low-bit weight-quantisation method designed with efficient GPU inference in mind.

AWQ is commonly encountered in GPU-oriented inference ecosystems and can be particularly useful when serving models through high-performance inference frameworks.

Best For: NVIDIA GPU inference and server-oriented deployments.
The Advantage: Efficient low-bit inference with good preservation of model quality.
Typical Environment: GPU inference servers and high-throughput applications.
Important: Support depends on the inference engine and the particular model architecture.

AWQ is therefore less about making a model universally portable and more about providing an efficient representation for compatible GPU inference stacks.

Specialised NVIDIA Inference

EXL2

EXL2 is a quantisation format associated with the ExLlamav2 inference ecosystem and is designed primarily for efficient NVIDIA GPU inference.

It became particularly popular among local-AI enthusiasts who wanted highly efficient inference from large quantised models on NVIDIA hardware.

Best For: NVIDIA GPUs using compatible ExLlama-based inference software.
The Advantage: Very efficient GPU inference and flexible quantisation choices.
The Catch: It is considerably more specialised than GGUF and is not a universal model format.
Hardware: It is primarily intended for GPU inference rather than CPU-oriented or Apple Silicon workflows.

EXL2 can be an excellent choice for a compatible NVIDIA setup, but it should be selected because your inference engine supports it — not simply because the file has a high tokens-per-second reputation.

GPTQ

Established GPU Quantisation

GPTQ

GPTQ is another widely encountered low-bit quantisation approach, particularly in GPU inference ecosystems.

GPTQ models are commonly distributed through repositories containing quantised weights designed for specific inference frameworks.

Best For: Compatible NVIDIA GPU inference.
Typical Precision: Frequently encountered in 4-bit configurations.
Advantage: Mature ecosystem and broad historical adoption.
Consideration: Always check current support for the particular model architecture and inference engine.

Safetensors

Original / High Precision

Safetensors

Safetensors is a secure and efficient format for storing model tensors. It is widely used for distributing original or higher-precision model weights on Hugging Face.

You will frequently encounter files such as:

model-00001-of-00004.safetensors model-00002-of-00004.safetensors ...

These are often sharded model weights. The fact that there are several files does not necessarily mean that there are several different models.

Safetensors is particularly common when using frameworks such as Hugging Face Transformers, rather than being a specialised quantised format like GGUF, AWQ, or EXL2.

MLX and Apple Silicon

Apple Silicon

MLX

MLX is Apple’s machine-learning framework designed around Apple Silicon’s unified memory architecture.

The MLX ecosystem includes tools for running and converting language models specifically for Apple hardware.

Best For: Apple Silicon Macs.
Key Advantage: Designed around Apple’s unified memory architecture.
Typical Use: Local inference and experimentation on M-series Macs.
Important: MLX model files are intended for MLX-compatible software and are not interchangeable with every other model format.

For Mac users, therefore, the decision is not necessarily “GGUF or nothing.” GGUF through llama.cpp-based software and MLX-based inference are both important parts of the current Apple Silicon local-AI ecosystem.

Understanding Quantisation Names

When browsing Hugging Face, you may encounter filenames such as:

ModelName-7B-Q4_K_M.gguf ModelName-14B-Q5_K_M.gguf ModelName-32B-Q6_K.gguf

The names can look cryptic, but they usually contain useful information.

For example:

7B — approximately 7 billion parameters.
Q4 — approximately 4-bit quantisation.
Q5 — approximately 5-bit quantisation.
Q6 — approximately 6-bit quantisation.
Q8 — approximately 8-bit quantisation.
K_M — identifies a particular quantisation scheme within the GGUF ecosystem.

These labels should not be interpreted as precise measurements of the final memory requirement. Metadata, quantisation details, runtime overhead, and the KV cache all contribute to actual memory usage.

Quality vs. Memory vs. Speed

Quantisation involves a trade-off.

In general:

Higher precision ↓ More memory ↓ Potentially higher fidelity Lower precision ↓ Less memory ↓ Potentially greater quality loss

But the relationship is not simply “more bits = better model.” Modern quantisation techniques are designed to preserve important information efficiently, and a well-designed 4-bit model can perform remarkably close to its higher-precision counterpart on many tasks.

The best choice depends on the model, quantisation method, task, and inference engine.

Model Size vs. Available VRAM

One of the first questions when selecting a local model is whether it will fit into your hardware.

Suppose you have a GPU with 16 GB of VRAM. A 32B model in a roughly 4-bit representation might require around 16 GB just for its quantised weights, before accounting for the KV cache and runtime overhead.

That means a nominally “16 GB” model may still not comfortably fit into a 16 GB GPU.

⚠️ Don’t Aim for 100% VRAM Utilisation

It is usually better to leave some memory headroom rather than selecting a model whose weight file exactly matches your available VRAM.

You need additional memory for:

KV cache.
Context processing.
Runtime buffers.
CUDA or other GPU runtime allocations.
Other applications using the GPU.

A model that technically loads may still perform poorly if the system is constantly operating at the edge of its memory capacity.

CPU/GPU Offloading

One of the advantages of flexible local inference systems is that the entire model does not always need to reside in dedicated GPU memory.

Some runtimes can keep part of a model in GPU memory while placing other portions in system RAM.

This makes models that are technically larger than your GPU’s VRAM usable, but there is an important trade-off:

CPU/GPU offloading generally increases memory flexibility at the expense of performance.

Moving data between system memory and the GPU can introduce a significant bottleneck. Therefore, “it runs” and “it runs quickly” are two very different requirements.

Mixture-of-Experts Models

Modern model families increasingly include Mixture-of-Experts (MoE) architectures.

An MoE model may contain a very large total number of parameters but activate only a subset of those parameters for each token.

For example, a model might be described using both:

Total parameters: 100B+ Active parameters: 10B+

This can make inference more computationally efficient than a conventional dense model with the same total parameter count.

However, the total model weights still need to be stored. Therefore, don’t assume that an MoE model with a small number of active parameters will require memory equivalent to a dense model containing only that many parameters.

Choosing the Format for Your Setup

Hardware / Goal	Good Starting Point
Apple Silicon Mac	GGUF or MLX, depending on the application
CPU-only inference	GGUF with a suitable quantisation
Consumer NVIDIA GPU	GGUF, AWQ, GPTQ, EXL2, or another format supported by your chosen runtime
High-throughput NVIDIA server	Formats supported by the selected serving framework, commonly including AWQ or other GPU-optimised representations
Hugging Face Transformers development	Often safetensors / native model format, with the appropriate precision or quantisation library
Ollama	Use models supported by Ollama; many are ultimately based on GGUF-style local inference
LM Studio	Primarily GGUF-based local model workflows

Don’t Choose a Format in Isolation

The correct decision is a three-way relationship:

┌──────────────┐ │ Model │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Format │ │ Quantisation │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Runtime │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ Hardware │ └──────────────┘

A theoretically excellent quantisation is of little use if your preferred inference engine cannot load it.

A Practical Decision Process

🔍 Before Downloading a Model

Identify your hardware.
How much RAM and GPU VRAM do you have?
Choose your inference engine.
For example, Ollama, LM Studio, llama.cpp, Transformers, vLLM, ExLlama, or MLX.
Check supported formats.
Don’t download a format simply because it is popular.
Estimate model memory.
Use parameter count and bits-per-weight as an initial approximation.
Leave memory headroom.
Remember that the weights are not the only memory consumer.
Choose a quantisation level.
Balance quality, memory requirements, and inference speed.
Benchmark the actual model.
Measure tokens per second, latency, memory usage, and task quality on your own system.

Which Should You Choose?

For someone beginning with local AI, GGUF is usually the easiest format to understand, particularly when using llama.cpp-based software, Ollama, or LM Studio.

For NVIDIA GPU users building higher-performance inference systems, AWQ, GPTQ, EXL2, and other GPU-oriented formats may be appropriate depending on the inference engine and model.

For Apple Silicon users, GGUF and MLX are both important options.

For developers working directly with the Hugging Face ecosystem, safetensors and the model’s native Transformers representation may be the most appropriate starting point.

The Key Principle

Quantisation is not simply a trick for making a huge model smaller. It is part of the engineering decision that determines whether a model can run efficiently on your particular hardware.

The practical local-AI stack is:

Model → Precision → Quantisation → Format → Inference Engine → Hardware

Once you understand that relationship, choosing between GGUF, AWQ, EXL2, GPTQ, MLX, safetensors, and other formats becomes much less mysterious.

The goal is not to find the smallest possible file. The goal is to find the best-quality model that fits comfortably into your available memory and runs efficiently on your chosen inference engine.

Previous Topic

Back to Lesson

Next Topic