1. Navigating Huggingface

Hugging Face is the central marketplace and community for open and open-weight AI models. Learn how to find, evaluate, download, and run models locally — and how to choose the right model for your hardware and application.

If you plan to run AI models locally, your journey will often begin at Hugging Face. It hosts a huge ecosystem of models, datasets, demos, libraries, and tools from organisations such as Meta, Google, Microsoft, Mistral AI, Qwen, NVIDIA, DeepSeek, and independent researchers.

However, choosing a model is only the first step. A model that looks ideal on paper may be impractical to run on your computer because of its memory requirements, model format, quantisation, context length, or compatibility with your chosen inference engine.

The Model Hub

The Hugging Face Model Hub is where model weights and their supporting files are published. Models can range from very small models that run comfortably on a laptop to extremely large models requiring multiple high-end GPUs.

When evaluating a model for local use, don’t look only at its parameter count. You should also check its architecture, quantisation options, context window, licensing, supported inference frameworks, tool-calling capabilities, and hardware requirements.

Base, Instruct, and Reasoning Models

One of the most important distinctions when selecting a model is whether you are downloading a base model or a model that has been post-trained to follow instructions.

Base Models

Base models are primarily trained to predict the next token in a sequence. They are useful as foundations for further training and experimentation, but they are generally not the easiest choice for a conversational assistant or autonomous agent.

For example, given the beginning of a sentence such as The capital of France is, a base model can naturally continue the text with Paris. Given an instruction such as Explain the capital of France, however, it has not necessarily been specifically trained to behave as an assistant.

Instruct and Chat Models

Instruct, Chat, or similarly post-trained models have been trained to follow user instructions and participate in conversational interactions.

For most local applications — including chatbots, coding assistants, RAG systems, and agents — an appropriate instruct model is usually the better starting point.

However, don’t assume that every instruct model has identical capabilities. If your application needs structured JSON output, tool calling, long-context document processing, or reliable instruction following, check that the specific model and runtime support those capabilities.

Reasoning Models

A newer category is the reasoning model. These models are designed to spend additional computation on difficult problems such as mathematics, programming, planning, and multi-step reasoning.

Reasoning models can be considerably more capable on some tasks, but they may also be slower, consume more tokens, and behave differently from conventional chat models. For a simple conversational assistant, a smaller instruct model may therefore provide a better balance of speed and capability.

Parameters Are Not the Whole Story

You will frequently see models described as 3B, 7B, 14B, 32B, or 70B. The B means billions of parameters.

As a general rule, larger models can provide greater capability, but they require more memory and are usually slower to run. Parameter count is therefore an important consideration, but it should not be treated as a simple measure of model quality.

Modern model families can also use architectures such as Mixture of Experts (MoE). An MoE model may contain a very large total number of parameters while activating only a subset of them for each token. Consequently, both total parameters and active parameters can matter when estimating inference performance.

Understanding Quantisation

If you intend to run a model locally, quantisation is one of the most important concepts to understand.

Model weights are commonly stored using numerical formats such as FP32, FP16, BF16, INT8, or lower-bit representations. Quantisation reduces the amount of memory required to store and process the weights, usually with some trade-off in model quality.

For example, a model available in an 8-bit or 4-bit representation can require substantially less memory than the same model stored in full or half precision.

Choosing a Model for Your Hardware

Before downloading a model, work backwards from your hardware.

• CPU-only: Smaller quantised models can run locally, although generation may be relatively slow.
• Integrated or shared memory: Systems with unified memory can sometimes run models that would not fit into a conventional GPU’s dedicated VRAM.
• Consumer GPU: GPU VRAM becomes a major constraint. Quantised models can make considerably larger models practical.
• Multiple GPUs: Larger models can be distributed across GPUs, depending on the inference engine and model architecture.
• Apple Silicon: Unified memory can be particularly useful for local inference, with applications such as llama.cpp and MLX providing hardware-specific approaches.

Remember that the model weights are not the only thing consuming memory. Context length, the KV cache, runtime overhead, batching, and other processes can significantly increase the amount of memory required during inference.

Model Format vs. Inference Engine

A model on Hugging Face is not necessarily something you simply double-click and run. You need an inference engine — software that loads the model and performs the computation required to generate tokens.

Common approaches include:

• llama.cpp: A highly portable C/C++ inference framework widely used for running quantised models, particularly GGUF models.
• Ollama: A user-friendly local model runner that simplifies downloading and serving models through a local API.
• Hugging Face Transformers: A Python ecosystem for loading and running many models directly, with support for a wide range of architectures and hardware configurations.
• vLLM: A high-performance inference and serving framework particularly suited to GPU-based deployment, APIs, and concurrent requests.
• MLX / MLX-LM: Apple’s machine-learning ecosystem for Apple Silicon, with support for efficient local inference and model conversion.
• Other specialised runtimes: Frameworks such as TensorRT-LLM and other optimised inference systems may be appropriate when maximum GPU throughput is the priority.

The important principle is that the model and the runtime are separate decisions. A model can be excellent but still be inconvenient to run if your preferred inference engine does not support its architecture or model format.

Finding the Right Files

The Files and versions section of a Hugging Face repository shows the actual files associated with a model.

A full-precision or half-precision model may contain multiple large .safetensors files. This is normal. Large models are frequently divided into multiple shards because the complete set of weights can be many gigabytes or more.

For local inference, you may instead find a separate repository containing a quantised version of the model, often with files such as .gguf.

Chat Templates Matter

Modern instruct models often expect prompts to be formatted using a particular chat template. This defines how system, user, and assistant messages are represented internally.

For example, two models may receive the same conversation but require completely different token sequences. Good inference tools can often read the model’s chat-template metadata and apply it automatically.

This is especially important when building agents. Incorrect prompt formatting can result in poor instruction following, malformed tool calls, or unexpectedly weak performance even when the underlying model is capable.

Tool Calling and Agents

If your goal is to build an AI agent, don’t choose a model solely because it performs well in ordinary chat benchmarks.

Look specifically for evidence that the model supports:

• Tool or function calling
• Structured output
• Reliable instruction following
• Long-context processing
• Code generation, if required
• Reasoning and planning, if required

There is also an important distinction between a model being capable of producing tool-call-like text and a complete inference stack reliably supporting structured tool calls. Always test the combination of model + template + runtime + agent framework rather than assuming compatibility from the model name alone.

Understanding Licences

Availability on Hugging Face does not automatically mean that a model is unrestricted.

Always read the licence and the model’s specific terms before using it in a commercial application.

• Apache 2.0: Generally permissive and suitable for many commercial and modified-use scenarios, subject to the licence terms.
• MIT: A permissive licence commonly allowing commercial use and modification, subject to its conditions.
• Model-specific licences: Some major model families use their own licences with additional conditions, restrictions, or acceptable-use requirements.
• Research or non-commercial licences: These may restrict commercial deployment or redistribution.

Do not assume that “open weights” means “open source”. The distinction matters. A model may make its weights available for download while imposing conditions on how those weights can be used or redistributed.

Evaluate Before You Commit

Model benchmarks are useful, but they should not be the only basis for choosing a local model.

For a local application, consider running a small practical test using the actual tasks your application will perform.

• How accurately does it answer your questions?
• How well does it follow instructions?
• How reliable are its structured outputs?
• Does it call tools correctly?
• How quickly does it generate tokens on your hardware?
• How much RAM or VRAM does it consume?
• How does performance change with a large context?
• Does the licence permit your intended use?

This gives you a more useful measure than a benchmark score alone: how well does this particular model perform on this particular computer for this particular job?

A Practical Local-Model Checklist

The Key Principle

Choosing a local AI model is no longer simply a question of finding the model with the largest number of parameters.

The best model for your project is the one that provides the right combination of capability, memory usage, inference speed, context length, runtime compatibility, tool support, and licensing for your particular hardware and application.

Think of local AI as a complete stack:

Model → Format → Quantisation → Inference Engine → Hardware → Application

Understanding how these layers interact is the key to successfully running modern AI models locally.