3. Working with Models as Chatbots

Before building an autonomous agent, learn how to interact with your local model, understand its behaviour, and control the inputs and settings that influence its responses.

Once you have a model running locally through Ollama, LM Studio, or another inference engine, resist the temptation to immediately build a complex agent. Your first task should be to talk to the model directly.

Manual experimentation gives you an opportunity to discover how the model behaves before an orchestration framework introduces another layer of complexity. You can test its instruction following, coding ability, reasoning behaviour, context handling, structured output, and tool-calling capabilities.

It also helps you distinguish between problems caused by the model and problems caused by your agent software.

The System Message

A modern chat interaction is normally constructed from a sequence of messages, commonly including a system message, user messages, and assistant responses.

The system message provides high-level instructions about how the model should behave. For example:

You are a senior Python developer. Answer questions about Python programming. Return concise, technically accurate answers.

A more specialised system instruction might define a role, constraints, output format, or interaction protocol:

You are a software engineering assistant. When asked to produce code: - Return valid Python. - Do not invent libraries. - Follow the requested output format. - If information is missing, state what is required.

However, the system message is not an absolute control mechanism. Models can misunderstand instructions, produce malformed output, or prioritise competing instructions incorrectly. Its effectiveness depends on the model’s post-training, chat template, context, and inference runtime.

Chat Templates Matter

One important concept that is easy to miss when working with local models is the chat template.

The messages you see in a chat interface are not necessarily sent to the model as plain text. The inference engine converts the conversation into the particular token structure expected by that model.

Different model families can use different templates for representing system, user, assistant, and tool messages.

This means that a model can appear to perform badly simply because it is being prompted using the wrong template.

💡 The Chat Stack

Think of a chat interaction as several layers:

System Message ↓ Conversation Messages ↓ Chat Template ↓ Tokens ↓ Model ↓ Generated Tokens ↓ Assistant Response

When troubleshooting a local model, remember that a problem may originate at any of these layers rather than in the model weights themselves.

Controlling Generation

Local inference tools expose parameters that influence how the model selects its next tokens. These parameters are particularly useful when experimenting with models for agents.

However, there is no universal combination of settings that is “correct” for every model. Different model families are trained and calibrated differently, and some reasoning-oriented models may respond poorly to settings designed for conventional chat models.

🌡️ Temperature

Temperature controls how strongly the model favours high-probability tokens during sampling.

Lower values generally make responses more predictable. Higher values generally allow more variation.

A useful conceptual range is:

  • Low temperature: More predictable and constrained output.
  • Medium temperature: A balance between consistency and variation.
  • Higher temperature: Greater variation and potentially more creative responses.

For agents: Lower temperatures are often a sensible starting point when the model needs to produce structured output or make tool-selection decisions.

However, temperature 0 does not guarantee perfectly deterministic behaviour. Determinism can also depend on the inference engine, hardware, sampling implementation, model architecture, and other settings.

🎲 Top-P

Top-P, also known as nucleus sampling, restricts token selection to a dynamically sized set of tokens whose cumulative probability reaches the specified threshold.

For example, a Top-P value of 0.9 means that the sampler considers the smallest group of candidate tokens whose combined probability reaches approximately 90%.

Top-P is therefore another sampling control, rather than simply “removing the least likely 10% of words.”

For agents: It is often best to leave Top-P at its model/runtime default while adjusting temperature, unless you have a specific reason to change it. Combining very aggressive sampling restrictions can sometimes produce worse rather than better results.

📏 Maximum Output Tokens

The maximum output token setting limits the amount of text the model can generate in a response.

This is different from the model’s context length. Context length describes the total amount of information the model can process, while the output limit controls how much new text it can generate for a particular response.

For agents, an output limit can provide a useful safety mechanism. If the model is expected to produce a short tool call but instead begins generating a long response, the limit can prevent it from consuming unlimited generation time.

However, setting the limit too low can also cause legitimate responses, JSON objects, or tool calls to be truncated.

🧠 Context Length

The context length determines how much tokenised information the model can consider during an interaction.

This can include:

  • The system instructions.
  • The user’s current request.
  • Previous conversation history.
  • Retrieved documents.
  • Tool definitions.
  • Previous tool calls and results.
  • The model’s generated response.

Increasing context length can therefore be useful for RAG systems and agents, but it can also substantially increase memory requirements, particularly when the model maintains a large KV cache.

Structured Output

If you are building an agent, asking the model to “please return JSON” is not always enough.

Modern inference systems can provide more reliable mechanisms for structured output, including JSON schemas or grammar-constrained generation, depending on the model and runtime.

For example, an agent might be instructed to return:

{ "action": "search", "query": "latest AI news" }

A robust application should still validate the result rather than assuming that the model will always produce valid JSON.

The ideal architecture is:

Model Output ↓ Parse ↓ Validate ↓ Accept / Reject ↓ Execute Tool

This is much safer than allowing arbitrary model-generated text to be interpreted directly as an instruction to your application.

Tool Calling

Modern instruct models can also be trained or configured to interact with tools.

Instead of generating a natural-language answer, the model can request that an application execute a function.

Conceptually:

User: What's the weather in London? ↓ Model: Call weather_tool location = "London" ↓ Application: Executes weather_tool ↓ Tool: Returns weather data ↓ Model: Produces final answer

This is the foundation of many modern agent architectures.

But tool calling introduces another important compatibility requirement. You need to consider the capabilities of the entire stack:

  • Does the model support tool calling?
  • Does the model’s chat template represent tool calls correctly?
  • Does the inference engine support the required format?
  • Does your agent framework understand the model’s tool-call output?
  • Does your application validate the requested tool and its arguments?

A model that can produce text resembling a function call is not necessarily equivalent to a model/runtime combination with reliable native tool-calling support.

Reasoning Models Require Different Expectations

Some newer models are specifically optimised for reasoning, planning, mathematics, programming, or other multi-step tasks.

These models can behave differently from conventional chat models. They may use additional internal computation or generate intermediate reasoning-related tokens before producing an answer.

Consequently, don’t assume that the same temperature, token limits, prompting strategy, or output expectations that work well for a conventional instruct model will work equally well for a reasoning model.

When testing a reasoning model, pay particular attention to:

  • Response latency.
  • Total token consumption.
  • Maximum output limits.
  • Tool-calling behaviour.
  • Performance on multi-step tasks.
  • How the model behaves when it reaches context limits.

Testing the Model Before Building the Agent

Before writing your orchestration code, treat the chat interface as a laboratory.

Create a series of repeatable tests that represent what your eventual agent will need to do.

🧪 A Useful Model Test

  1. Test basic instruction following.
    Give the model a simple instruction and check whether it follows it accurately.
  2. Test structured output.
    Ask it to return information using a defined JSON structure.
  3. Test edge cases.
    Give it incomplete, ambiguous, or deliberately difficult input.
  4. Test tool selection.
    Present several possible tools and see whether it selects the appropriate one.
  5. Test tool arguments.
    Check whether the generated arguments have the correct names, types, and values.
  6. Test long context.
    Provide increasingly large amounts of information and observe when performance begins to deteriorate.
  7. Repeat the tests.
    Run the same prompts multiple times when using non-zero sampling settings to understand how stable the results are.

Simulating the Agent Loop

You can perform a surprisingly effective agent test without writing an agent.

Start by writing the system instructions that you expect your eventual agent to use. Then manually act as the orchestrator.

For example:

SYSTEM: You are an assistant with access to a calculator tool. USER: What is 127 × 38? MODEL: { "tool": "calculator", "arguments": { "expression": "127 * 38" } } ORCHESTRATOR: The calculator returned: 4826 USER: [Return the tool result to the model] MODEL: 127 × 38 = 4826.

This simple exercise exposes problems before they become buried inside hundreds of lines of agent code.

Evaluate the Whole System

It is important to remember that an agent is not just a model.

A useful mental model is:

┌───────────────┐ │ Model │ └───────┬───────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ Prompting Tool Calling Structured Output │ │ │ └──────────────┼──────────────┘ ▼ ┌───────────────┐ │ Orchestrator │ └───────┬───────┘ │ ▼ External Tools

A failure can therefore originate in the model, prompt, chat template, inference settings, tool definition, parser, orchestrator, or external tool.

Manual testing helps isolate these components before they become difficult to debug.

A Practical Starting Configuration

There is no universal “best” configuration for every local model, but a sensible starting approach is:

  • Use an appropriate instruct or reasoning model for your task.
  • Start with the model’s recommended defaults.
  • Use a relatively low temperature for structured agent tasks.
  • Leave Top-P at its default initially.
  • Set a sensible maximum output length.
  • Choose a context length appropriate to your application rather than automatically selecting the maximum.
  • Use structured output or schema validation where your runtime supports it.
  • Validate every model-generated tool call before executing it.
  • Test the same prompts repeatedly to understand the model’s reliability.

The Key Principle

The purpose of chatting with a local model is not simply to see whether it can answer questions.

You are learning how the model behaves as a component inside a larger software system.

Before building an autonomous agent, you should understand:

Prompt → Chat Template → Model → Sampling → Structured Output → Tool Call → Validation → Tool → Model

Once you understand this loop, the transition from a simple chatbot to an agent becomes much easier to reason about — and much easier to debug.

Previous Topic
Back to Lesson
Next Topic