1. Cloud vs Local Inference: Choosing Where Your Agent Runs
Where should your agent do its thinking: on a provider’s infrastructure, on hardware you control, or somewhere in between?
Once you understand what your agent needs to do, one of the most important architectural decisions is where inference happens. Inference is the process of running a trained AI model on an input—such as a prompt, conversation history, image, or tool result—to produce an output.
Today, this is no longer a simple choice between “the cloud” and “a GPU under your desk.” Developers can use hosted model APIs, managed deployments of open-weight models, private cloud or VPC infrastructure, on-premise servers, and models running directly on laptops or edge devices.
For most projects, however, it is still useful to begin with two broad categories: hosted inference and self-hosted inference.
☁️ Hosted Inference
A third-party provider runs the model and exposes it through an API. Examples include model providers and inference platforms such as OpenAI, Anthropic, Google, Groq, and cloud services that host open-weight models.
- Pro: Minimal infrastructure. You can build a sophisticated agent without purchasing or maintaining AI accelerators. The provider handles model serving, scaling, hardware failures, and much of the underlying optimization.
- Pro: Access to highly capable models. Hosted APIs provide convenient access to large reasoning, multimodal, coding, and agent-oriented models that may require substantial infrastructure to self-host.
- Pro: Easy scaling. A prototype processing ten requests per day can often use the same API architecture when it grows to thousands of requests, although quotas, rate limits, and capacity planning still matter.
- Pro: Rapid model upgrades. Developers can often adopt newer models without purchasing new hardware or rebuilding an inference cluster.
- Con: Usage-based costs. Costs commonly scale with input tokens, output tokens, model choice, tool usage, cached context, and other features. Agentic systems can generate many model calls, so poorly controlled loops can become expensive.
- Con: External dependency. Your application depends on the provider’s availability, latency, pricing, quotas, model lifecycle, and API policies.
- Con: Data leaves your infrastructure. Requests must normally be transmitted to the provider. This does not automatically mean the data is used for model training—many business and API products provide contractual privacy, retention, residency, or zero-data-retention controls—but organizations must evaluate the provider’s terms and compliance requirements for their use case.
🖥️ Self-Hosted & Local Inference
You run an open-weight or otherwise deployable model on hardware you control. This could mean a laptop, workstation, company server, private data center, or dedicated cloud GPU instance.
- Pro: Greater control over data. Sensitive prompts and outputs can remain inside infrastructure you manage, which may simplify certain privacy, sovereignty, or security requirements.
- Pro: Greater deployment control. You choose the model version, inference engine, quantization level, update schedule, logging policy, network access, and surrounding security controls.
- Pro: Predictable infrastructure economics. Instead of paying per request, you primarily pay for hardware or compute capacity, electricity, operations, and engineering. At sufficiently high utilization, this can be economical.
- Pro: Offline and edge operation. Smaller models can run without a continuous internet connection, making local inference useful for private workstations, industrial systems, field deployments, and latency-sensitive applications.
- Con: Hardware and operational requirements. Larger models may require significant GPU memory, system RAM, storage, power, cooling, and engineering expertise.
- Con: Model capability trade-offs. A model that comfortably fits on a laptop may not perform as well on a difficult reasoning or tool-use task as a much larger hosted model.
- Con: You operate the infrastructure. Model serving, monitoring, security patches, concurrency, scaling, upgrades, and failures become your responsibility.
Local Does Not Necessarily Mean “One Giant GPU”
Modern inference software makes self-hosting far more flexible than it once was. Models can be quantized—represented using lower-precision numbers—to reduce memory requirements and often improve inference speed, usually with some trade-off in model quality.
Inference engines can also divide work across CPUs and GPUs or multiple GPUs. Tools such as llama.cpp, for example, support multiple quantization levels and CPU/GPU hybrid inference, while production-oriented serving engines such as vLLM support features including quantization and distributed inference.
This means the practical question is not simply, “Can I run this model locally?” A better question is:
“Can I run the model I need, at the quality, context length, concurrency, and response speed my application requires?”
The Latency Factor
Latency is especially important for agents because a single user request may trigger many sequential operations:
Model → Tool → Model → Tool → Model → Final Answer
If each step takes several seconds, the delays accumulate. An agent performing ten sequential model calls can feel slow even when each individual request seems reasonably fast.
When evaluating inference performance, consider more than a single “tokens per second” number:
- Time to First Token (TTFT): How long the user waits before generation begins.
- Output throughput: How quickly the model generates tokens after it starts responding.
- Input processing: Large prompts and long conversation histories can take significant time to process.
- Network latency: Hosted APIs require network communication, although optimized inference providers can still deliver extremely low response times.
- Queueing and concurrency: A system that is fast for one user may slow down when many requests arrive simultaneously.
- Tool latency: Database queries, web requests, browsers, code execution, and external APIs may take longer than the model itself.
For agent systems, end-to-end task completion time is usually more meaningful than raw token-generation speed.
Context Length, Memory, and the KV Cache
Agent workloads often involve large contexts: system instructions, conversation history, retrieved documents, tool schemas, intermediate results, and accumulated state. Long context windows increase both computational work and memory requirements.
During inference, models also maintain a KV cache containing information from previously processed tokens. Longer contexts and more simultaneous users require more memory, so a model that fits comfortably for a single short conversation may require substantially more resources when serving long-running agents concurrently.
Hosted providers manage most of this infrastructure for you. With self-hosted inference, context size and concurrency become important parts of capacity planning.
Cost Is More Than “API vs Electricity”
A common mistake is to assume that cloud inference is expensive while local inference is free. Neither statement is generally true.
Hosted inference costs can include:
- Input and output tokens
- Premium model tiers
- Stored or cached context
- Tool calls or additional platform services
- High-volume or reserved capacity
Many providers now support techniques such as prompt or context caching, which can reduce the cost of repeatedly processing the same large prompt or document set.
Self-hosted inference costs can include:
- GPUs, accelerators, or cloud GPU rental
- Electricity and cooling
- Idle hardware capacity
- Engineering and maintenance time
- Monitoring, security, and deployment infrastructure
- Redundancy and scaling capacity
The correct comparison is therefore total cost of ownership, not simply the price of an API token versus the price of electricity.
Privacy and Security: A More Nuanced View
Self-hosting can provide strong control over where information is processed, but it does not automatically make a system secure. You are still responsible for access control, encryption, logging, patching, network security, backups, and protection against attacks such as prompt injection or malicious tool use.
Likewise, using a hosted API does not automatically mean surrendering ownership of your data. Enterprise and API providers may offer encryption, configurable retention, contractual privacy protections, regional processing, or zero-data-retention options.
The correct question is therefore not simply “cloud or private?” but:
“Does this deployment architecture satisfy the security, privacy, retention, residency, and compliance requirements of this specific application?”
The Hybrid Approach
Increasingly, production systems use more than one model and more than one inference location.
An agent might use:
- A small local model for classification, routing, or extracting structured information.
- A fast hosted model for routine tool-use decisions.
- A larger reasoning model only when the task is genuinely difficult.
- A private self-hosted model when sensitive data must remain inside a controlled environment.
This strategy is sometimes called model routing. Instead of forcing every task through the same model, the application selects an appropriate model based on factors such as complexity, latency, privacy, and cost.
Hybrid deployment can also extend beyond model routing. Organizations may run the application and sensitive data inside their own environment while accessing selected models remotely, or deploy open-weight models across on-premise, private-cloud, and managed infrastructure.
Choosing an Inference Strategy
Instead of asking, “Is cloud or local better?”, evaluate your application across several dimensions:
- Capability: How difficult are the tasks the model must solve?
- Latency: How quickly must the agent respond?
- Throughput: How many simultaneous requests must it handle?
- Context: How much information must the model process at once?
- Privacy: What information is allowed to leave your infrastructure?
- Reliability: Can the application tolerate an external service outage?
- Cost: What will the system cost at realistic production volume?
- Operations: Does your team want to manage inference infrastructure?
For experimentation, hosted APIs often reduce infrastructure work and make it easy to compare capable models. For production, the best architecture may remain fully hosted, become fully self-hosted, or combine multiple deployment strategies.
engineering trade-off between capability, speed, privacy, reliability, operational complexity, and total
cost. Modern agent systems increasingly choose the right model—and the right deployment environment—for
each part of the job.