5. Basic Troubleshooting & Configs

⚠️ Issue: “CUDA OutOfMemoryError” or the model immediately crashes

The Cause: The inference process requires more GPU memory than is currently available.

This can happen because of:

• The model weights are too large for the available VRAM.
• The quantisation is too high-precision.
• The context length is too large.
• The KV cache requires more memory than expected.
• Another application is already consuming GPU memory.
• The inference engine is allocating additional runtime buffers.

The Fix:

• Reduce the model size: Move from a larger model to a smaller one.
• Use a more aggressive quantisation: For example, move from an 8-bit representation to a suitable 4-bit or 5-bit version.
• Reduce context length: A very large context can significantly increase memory requirements.
• Reduce GPU offloading: Where your runtime supports CPU/GPU splitting, moving some layers into system RAM can allow the model to load, although it may reduce performance substantially.
• Close other GPU applications: Games, browsers, image-generation software, video editors, and other applications may consume VRAM.
• Check the runtime configuration: Different inference engines have different memory-management strategies.

Don’t assume that a model file’s size is the exact amount of VRAM required. The runtime needs additional memory beyond the stored weights.

⚠️ Issue: The model fits initially but crashes when the conversation gets longer

The Cause: The model’s memory requirements can increase as the active context grows.

The KV cache stores information required to efficiently continue generation from previous tokens. Larger context windows therefore generally require additional memory.

This is particularly important for agents, where the context may contain:

• Long system instructions.
• Conversation history.
• Retrieved documents.
• Tool definitions.
• Previous tool calls.
• Tool results.

The Fix:

• Reduce the context length if it is unnecessarily large.
• Reduce the amount of conversation history being retained.
• Summarise older messages rather than continually appending them.
• Limit the size of retrieved documents in RAG applications.
• Reduce the number or size of tool definitions included in every request.
• Use a smaller model or more efficient quantisation if necessary.

Remember that maximum context length and actual context usage are different things. Configuring a large maximum context does not mean every request will use that amount, but the runtime may reserve or allocate memory based on its configuration.

⚠️ Issue: Generation is extremely slow

The Cause: There are several possible causes. The model may be running primarily on the CPU, partially offloaded to system RAM, using an inefficient configuration, or simply be too large for your hardware.

First check:

• Is the correct GPU being detected?
• Is GPU acceleration enabled?
• How much GPU memory is being used?
• Is system RAM usage unusually high?
• Is the model partially offloaded to CPU memory?
• What model and quantisation are you using?
• Is the context unusually large?

Don’t diagnose performance using GPU utilisation alone. A GPU can show relatively low utilisation while still being the limiting component, and utilisation can vary significantly during prompt processing and token generation.

Instead, measure tokens per second, time to first token, prompt-processing speed, memory consumption, and end-to-end latency.

The Fix:

• Use an appropriate GPU-accelerated runtime.
• Ensure your drivers and inference software support your hardware.
• Use a model that fits more comfortably into GPU memory.
• Reduce CPU/GPU memory transfers where possible.
• Try a smaller or more efficiently quantised model.
• Reduce unnecessary context.

⚠️ Issue: The model works, but the first response takes a long time

The Cause: This can be different from slow token generation.

The system may be spending time:

• Loading the model into memory.
• Initialising the inference engine.
• Processing a large prompt.
• Building the KV cache.
• Moving model data between CPU and GPU memory.

The Fix:

• Check whether the delay occurs only on the first request.
• Keep the model loaded when repeated requests are expected.
• Reduce unnecessarily large system prompts.
• Reduce excessive context history.
• Check whether the runtime is repeatedly unloading and reloading the model.

For agents, time to first token and tokens per second are separate performance measurements and should be monitored separately.

⚠️ Issue: The model produces repetitive text or gets stuck in a loop

The Cause: Repetition can have several causes, including sampling settings, prompt construction, model limitations, context problems, or incorrect stopping behaviour.

The Fix:

• Try reducing the temperature.
• Check Top-P and other sampling settings.
• Check repetition-related parameters supported by your inference engine.
• Make sure the conversation history is not being duplicated accidentally.
• Check that tool results are being inserted into the conversation correctly.
• Verify that the model is receiving the correct chat template.
• Test the model directly outside the agent framework.

Do not assume that increasing randomness will automatically fix repetition. First determine whether the repetition originates from sampling, prompting, or an application-level loop.

⚠️ Issue: The model outputs strange tokens, markup, or garbage characters

The Cause: This can occur when the model is being prompted using an incompatible format or when the runtime is incorrectly interpreting the model’s expected chat structure.

Modern instruct models can rely on specific chat templates and special tokens to distinguish system, user, assistant, and tool messages.

The Fix:

• Check that the correct chat template is being applied.
• Use the template recommended by the model author.
• Check whether your inference engine automatically reads the model’s template metadata.
• Don’t manually add special tokens unless the runtime documentation specifically requires it.
• Check that your tokenizer matches the model.
• Test the model using the inference application’s native chat interface.

Older tutorials often recommend manually inserting tags such as [INST] or other special markers. With modern runtimes, this can actually make things worse if the application already applies the appropriate template automatically.

⚠️ Issue: The model doesn’t stop generating

The Cause: The inference engine may not be recognising the appropriate end-of-turn or end-of-generation condition.

Possible causes include:

• Incorrect chat template.
• Incorrect or missing end-of-sequence handling.
• Incorrect stop sequences.
• A model/runtime compatibility problem.
• The model simply failing to recognise that the requested response is complete.

The Fix:

• Use the model’s recommended chat template.
• Allow the inference engine to manage special tokens where possible.
• Check the model’s documented stop or end-of-turn tokens.
• Set a sensible maximum output length.
• Check that your agent loop isn’t accidentally requesting another response after receiving a valid result.

Stop sequences are model- and runtime-dependent. Do not copy a stop token from another model family simply because it worked elsewhere.

⚠️ Issue: The model keeps hallucinating or confidently inventing information

The Cause: Hallucination is not simply a temperature problem. Language models generate plausible continuations and can produce incorrect information even at very low sampling temperatures.

The Fix:

• Use retrieval or external data sources when factual accuracy is important.
• Ask the model to distinguish known information from uncertainty.
• Provide relevant source material in the context.
• Use tools for calculations, database queries, current information, and other tasks where external verification is available.
• Validate important model-generated results programmatically.
• Reduce the model’s responsibilities rather than expecting it to know everything.

For an agent, the most effective solution is often not “make the model smarter” but give the model access to reliable tools and require the application to validate important outputs.

⚠️ Issue: The model ignores tools or outputs tool calls as ordinary text

The Cause: Tool calling requires compatibility across several layers.

• The model must support the required tool-calling behaviour.
• The chat template must represent tools correctly.
• The inference runtime must support the model’s tool format.
• The API must expose tools in the expected schema.
• The agent framework must understand the returned tool call.

The Fix:

1. Test the model’s tool-calling capability directly.
2. Check the model documentation.
3. Check the inference runtime’s supported tool-calling formats.
4. Use a simple tool with only a few parameters.
5. Validate the generated tool arguments.
6. Only then add multiple tools and more complicated orchestration.