Two voice agents can run the same model and behave completely differently on a busy Monday. The difference is not the weights. It is who provisioned the capacity, whose queue the request waits in, and whose phone rings when the model step stops responding while callers are on the line.
- Concurrency is the sizing unit Voice workloads are measured in simultaneous calls, not daily volume, and the peak minute is what capacity has to survive.
- Variance matters more than median A shared service usually has a good typical response and a tail you cannot control; a private pool has variance you own entirely.
- The cost curves cross One arrangement is near-linear with usage, the other is mostly fixed, so the cheaper option depends on volume and utilization.
The comparison is often framed as a philosophical one about control. In practice it is four concrete operational questions with measurable answers, and the answers change depending on how many people call you at once.
It is worth being precise about scope. This is about where the model that decides what to say is executed. Where the rest of the system runs is a separate arrangement with its own considerations.
The Question Is Who Runs the Inference
Both arrangements execute the same kind of computation. A prompt goes in, tokens come out, and the agent speaks them. What differs is everything around that: who bought the capacity, who scales it, and who is accountable when it stops working.
The request crosses the network to a provider's shared service. Capacity is theirs, scaling is theirs, and usage is metered.
The model executes on hardware you rent or own. The serving process is yours, and capacity is exactly what you provisioned.
Neither arrangement makes an agent better at conversation. The model decides that, and which model to pick is a separate question answered in the behaviors that make a model good on a call.
What the choice changes is the shape of every operational problem the model step can have: how it degrades, how fast it recovers, who can act, and what it costs at the volume you actually run.
How Capacity Behaves When Calls Arrive Together
Call traffic is not smooth. It clusters around a campaign going out, the hour after an outage somewhere else, the first thirty minutes of a Monday. Capacity has to survive the peak minute, not the daily average.
On a shared hosted service, scaling is elastic within whatever limits apply to your account. You do not usually hit hardware saturation. You hit a rate limit, or you get a slower first token because other tenants are busy at the same moment.
On capacity you run, the pool is fixed until somebody changes it. Requests beyond what the pool can serve concurrently queue, and the degradation is not gentle: response times climb slowly with load and then sharply once the serving batch is saturated.
- Size on concurrent calls, not on calls per day, because ten thousand calls spread over a week is a different machine from a hundred at once.
- Count tokens per turn rather than per call, since each turn is an independent request against the same pool.
- Use the busiest minute you have ever recorded as the floor, not the mean.
- Leave headroom for the extra traffic a slow turn generates, because callers repeat themselves when nothing happens.
- Remember that a turn abandoned mid-generation has already consumed the capacity that produced it.
Voice has one advantage over other workloads here: concurrency is physically bounded by how many lines are open, which is a number telephony records already contain. Sizing does not have to be guesswork.
Where the Latency Is Actually Spent
Both arrangements pay the same underlying costs for reading a prompt and generating tokens. The differences sit in what surrounds that work, and they are not where most people expect.
| Where the time goes | Hosted endpoint | Endpoint you run |
|---|---|---|
| Reaching the model | A round trip over the public internet plus session setup | A hop inside your own network, usually shorter |
| Waiting for capacity | Queueing you cannot see or measure directly | Your own queue, visible and sized by you |
| First token | Shaped by prompt length and the service's current load | Shaped by prompt length and your batching settings |
| Run-to-run variance | Partly determined by other tenants sharing the service | Determined only by your own concurrency |
| Cold starts | Rare, since the service stays warm | Real, if instances are allowed to scale to zero |
The network hop is the row people assume dominates, and it usually does not. It is a fixed cost measured in tens of milliseconds against a model step measured in hundreds, and the wider chain it sits in is laid out in where the delay in a spoken turn comes from.
Variance is the row that matters more. A shared service can give an excellent typical response with an occasional slow one you had no warning about. A private pool trades that for variance you cause yourself, which is worse when you misjudge capacity and better because you can see it coming.
Callers do not average their experience. They remember the one turn where nothing happened for three seconds, which is why the slow end of the distribution deserves more attention than the middle of it.
Who Owns a Failure While a Caller Is on the Line
When the model step fails mid-call, the caller hears silence and then, if nothing is done, a dropped conversation. The technical symptom is similar either way. What differs completely is what your team can do about it in the next sixty seconds.
- A provider-side incident: you have a status page, a support channel and no levers. Recovery time is somebody else's decision.
- An incident in capacity you run: you have every lever, the pager and the obligation to use both at an unhelpful hour.
- Capacity exhaustion: on a shared service it usually surfaces as rejected requests; on your own it surfaces as everything getting slower at once.
- A behavior change: a hosted model can be updated beneath you, while a build you hold changes only when you change it.
- A configuration mistake: entirely yours in both cases, and statistically the most common cause of the other four.
Neither arrangement is safer in the abstract, and teams pick the wrong one by imagining the incident they would rather have instead of the one they are equipped for. The useful question is narrow: at three in the morning, what can the person on call actually do?
Both arrangements should also have somewhere to go when the model step is unavailable, which is a design decision rather than a hosting one and is covered in what happens when the model a call depends on stops answering.
The Two Cost Curves Have Different Shapes
The important difference is not which is cheaper. It is that one cost is proportional to use and the other is mostly paid whether or not anyone calls.
Metered usage scales down as well as up. A quiet week costs less, a pilot with fifty calls costs almost nothing, and the bill is straightforwardly a function of how many tokens moved through it.
Capacity you run is dominated by the fixed side. Accelerators are paid for whether they are busy or idle, so cost per call falls as volume rises and is punishing at low volume. Utilization, not the hardware price, is what determines whether that arrangement looks sensible.
That is why the curves cross rather than one simply being lower. Where they cross depends on tokens per call, how evenly traffic is spread across the day, and what the fixed side genuinely includes.
- Idle hours count against you: a pool sized for the Monday peak is mostly unused on a Thursday evening.
- Engineering time belongs on the fixed side, including the evaluation work that a change to the build requires.
- Redundancy multiplies the fixed side, because a second pool for resilience is a second pool to pay for.
- Growth is smooth on one curve and stepped on the other, since capacity is bought in units rather than continuously.
Reducing what the model step costs within either arrangement is a separate exercise with its own levers, described in bringing down what the model step costs per call.
Deciding From Your Own Call Profile
The decision becomes straightforward once four numbers exist, and most teams argue about it before collecting any of them.
- Measure peak concurrent callsThe most lines open at the same moment, taken from telephony records rather than estimated from monthly totals.
- Measure tokens per turnPrompt and output together, on real conversations, since prompt length usually dominates and is the part teams forget to count.
- Work out honest utilizationWhat proportion of the day capacity would actually be doing work, including nights, weekends and seasonal quiet periods.
- Decide what on-call can ownWhether the people who would be paged can genuinely operate an inference service, or would be waiting on a status page either way.
- Keep the model access behind one interfaceSo the answer can change later without rewriting the agent, and so both options can be measured against the same fixtures.
Plenty of systems end up running both at once, and not as a fallback arrangement. Different turns have different requirements, and sending them to different destinations on purpose is described in how one agent uses more than one model.
Whether the weights themselves can be downloaded and what obligations arrive with them is the other half of this decision, set out in what an open-weight model asks of the team running it.
Conclusion
Choosing between self-hosted and API LLMs for voice agents is an operational decision, not simply a model comparison. Hosted APIs reduce infrastructure management and can suit variable demand, while self-hosted inference offers more direct control over deployment, capacity, and incident response.
Evaluate both approaches against realistic peak traffic, tokens per turn, latency percentiles, reliability requirements, and total cost of ownership. The best setup is the one your team can operate predictably when real callers are waiting, and a hybrid architecture may be the right choice when different turns have different needs.
Frequently Asked Questions
Sometimes, and mostly for a reason people do not expect. The saved network hop is small against the model step itself. The larger effect is variance: a private pool responds only to your own load, while a shared service can be slow at moments you did not cause and cannot see coming. Whether that helps depends on whether you sized the pool correctly.
It follows from peak concurrent calls and tokens per turn, not from calls per day. Every open line generates an independent request each turn, so the sizing unit is simultaneous conversations at the busiest minute you have recorded. Telephony records already hold that number, so this is measurable rather than a guess.
There is no universal crossing point, because it depends on tokens per call, how evenly traffic spreads across the day and what you count on the fixed side. Work it out from your own numbers, and include engineering time and any redundant capacity. A pool sized for the peak and idle most of the night changes the answer considerably.
Yes, and many systems do. It requires the agent to reach the model through a single interface so the destination is configuration rather than code, and it requires the same evaluation fixtures to run against both. Doing that also makes changing your mind later a small piece of work instead of a rebuild.
It changes which parties process it, which is one input to a data protection review rather than the whole of it. The rest of the system still handles audio, transcripts and records. What is required in a given case depends on jurisdiction, sector and the legal basis you rely on, so the applicable position is one to establish with counsel.
