2. The Easy Way: Ollama & LM Studio

Modern local AI tools hide most of the complexity of model management and inference. Ollama and LM Studio make it possible to download, configure, test, and serve capable language models locally without building a PyTorch environment from scratch.

Historically, running an open-weight AI model locally could involve installing Python packages, matching CUDA and GPU drivers, downloading model checkpoints, understanding tokenizers, configuring memory settings, and resolving dependency conflicts.

Today, much of that complexity can be handled by dedicated local inference applications. Instead of writing code to load model weights directly, you can install a model runner, download a compatible model, and start generating text within minutes.

Two particularly accessible options are Ollama and LM Studio. They overlap considerably, but they are designed around different workflows.

ollama run llama3.2

Terminal → Ollama → Local Model → Local API → Your Application

LM Studio → Download Model → Configure Runtime → Local Server → Your Application

Ollama vs. LM Studio

Neither application is universally “better”. The choice depends largely on how you intend to work with local models.

Which Should You Use?

Use Ollama if:

• You are comfortable using a terminal.
• You are building applications or AI agents.
• You want a simple local API for your software to call.
• You want models to run as a background service.
• You intend to use Linux servers, containers, or remote machines.
• You want model management to be largely automated.
• You are experimenting with frameworks that already support Ollama.

Use LM Studio if:

• You prefer a graphical desktop application.
• You want to browse and compare models visually.
• You want greater visibility into individual model files and quantisations.
• You want to test a model interactively before integrating it into an application.
• You want an easy way to adjust inference parameters through a graphical interface.
• You want to expose a locally running model through an OpenAI-compatible API.

Understanding What Happens When You Run a Model

It is useful to understand what these applications are actually doing behind the scenes.

When you run a local model, the process broadly looks like this:

1. Model selection: You choose a model and, where applicable, a particular quantisation.
2. Download: The model files are stored locally on your computer.
3. Loading: The inference engine loads the model weights into system RAM, GPU VRAM, unified memory, or a combination of these.
4. Prompt processing: Your input is converted into tokens and processed by the model.
5. Inference: The model predicts tokens sequentially.
6. Generation: The generated tokens are converted back into text.
7. API access: If a local server is enabled, other applications can send requests to the running model.

This is fundamentally different from using a hosted AI API. With a hosted service, the model runs on somebody else’s infrastructure. With Ollama or LM Studio, the inference computation takes place on your own machine.

Hardware Still Matters

Although Ollama and LM Studio make running models much easier, they cannot remove the underlying hardware requirements.

Your computer still needs enough memory to hold the model and its runtime state.

Quantisation: Why Model Downloads Have Different Sizes

When browsing models, you may encounter several versions of what appears to be the same model.

These may differ because the weights have been quantised to different numerical precisions.

For example, a model may have 4-bit, 5-bit, 6-bit, 8-bit, and higher-precision versions. Lower-bit versions generally use less memory and can be faster, while higher-precision versions can preserve more of the original model’s numerical information.

For most local users, the goal is not to find the largest possible model. It is to find the largest or most capable model that runs comfortably on your hardware at an acceptable speed.

The Importance of Context Length

One setting that is easy to overlook is the context length.

The context is the information available to the model while generating its response. It can include the system prompt, conversation history, retrieved documents, tool definitions, and the model’s generated tokens.

A larger context window can be extremely useful for agents and RAG applications, but it can also increase memory consumption.

Therefore, don’t automatically configure the maximum context length advertised by a model. Start with a practical value for your application and increase it when you actually need it.

Inference Settings

Both Ollama and LM Studio expose settings that influence how a model generates responses.

Common settings include:

• Temperature: Controls the degree of randomness in generation.
• Top-p: Restricts token selection to a probability mass.
• Maximum output tokens: Limits how much the model can generate.
• Context length: Determines how much information can be retained in the active context.
• Seed: Can sometimes be used to make generation more reproducible.
• GPU offloading: Determines how computation or model layers are distributed between available hardware, where supported.

For reliable agents, deterministic or relatively low-randomness settings are often preferable to highly creative settings. The optimal configuration depends on the model and task, however, so these settings should be tested rather than treated as universal rules.

The Goal: A Local API

If you are learning about local models because you ultimately want to build an AI agent, the chat interface is only part of the story.

The important component is the local inference API.

Your application can send a request to the local server, including the conversation, system instructions, and potentially tool definitions. The model processes the request and returns its response to your application.

The architecture therefore looks something like this:

┌───────────────────┐ │ Your AI Agent │ └─────────┬─────────┘ │ │ HTTP / API ▼ ┌───────────────────┐ │ Local API Server │ │ Ollama / LM Studio│ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Local Inference │ │ Engine │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ Local Model │ │ + Quantisation │ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ │ CPU / GPU / RAM │ │ / Unified Memory │ └───────────────────┘ 

OpenAI-Compatible APIs

One of the most useful developments in local AI is the availability of OpenAI-compatible APIs.

Many applications are written to communicate with an API that follows the general conventions of OpenAI’s API. Local model servers can provide compatible endpoints, allowing developers to switch between a cloud model and a locally running model with relatively small configuration changes.

For example, an application might normally be configured with:

Cloud API ↓ Hosted AI Model 

and then be reconfigured to use:

Local API ↓ Ollama / LM Studio ↓ Local AI Model 

This is particularly powerful for learning. You can develop an application using a local model without necessarily paying for API calls during every experiment.

Local Does Not Automatically Mean Private or Free

Running a model locally has major privacy and cost advantages, but it is important to be precise about what this means.

• Local inference: The model computation can take place on your own hardware.
• No per-token API charge: You are not paying a cloud provider for every generated token, although your hardware consumes electricity and has a purchase cost.
• Greater data control: Data sent only to your local model does not need to be transmitted to a model provider.
• Internet access is still possible: Your application, tools, plugins, web search, model downloads, and other components may still communicate with external services.

If privacy is a requirement, examine the entire application architecture, not simply whether the language model is running locally.

From Chatbot to Agent

A local model becomes much more interesting when it is connected to software tools.

A simple local chatbot looks like:

User → Model → Response 

An agent adds an orchestration layer:

User ↓ Agent ↓ Local Model ↓ Tool Selection ↓ External Tool / Database / Code / File ↓ Tool Result ↓ Local Model ↓ Final Response 

This is where capabilities such as structured output, tool calling, context management, and reliable instruction following become particularly important.

Consequently, when selecting a local model for an agent, don’t ask only, “How good is this model at answering questions?” Ask instead, “How reliably can this model participate in the complete agent loop on my hardware?”

A Practical Starting Point

The Key Principle

Ollama and LM Studio remove much of the mechanical complexity of local AI, but they do not eliminate the need to understand what is happening underneath.

Think of them as the bridge between AI models and your computer.

Instead of manually managing model files, Python environments, tokenizers, GPU libraries, and inference code, you can concentrate on learning how to use the model:

Model → Local Runtime → API → Application → Agent

Once you understand this architecture, you can move between different models, runtimes, and hardware configurations without having to rebuild your entire application from scratch.