Three hosted families dominate most shortlists for a voice agent: OpenAI's models, Anthropic's Claude models and Google's Gemini models.
The interesting differences between them are not about which writes better prose. They are about the shape of the interface a phone call has to connect to, where the contract is signed, and what happens to call audio afterwards.
- Two shapes of audio interface Some families expose a direct audio-in, audio-out session. Others expose text only, which makes the call a chained pipeline by necessity.
- Where you sign matters These families are reachable through their own APIs and through major cloud platforms, and that changes the contract, not just the endpoint.
- Caching shapes differ A voice system re-sends its system prompt on every turn, so how a provider caches a repeated prefix is a per-turn cost and latency question.
The behaviors that make any model good on a call are the same across all three, and they are the wrong place to start a provider comparison because they are not provider-specific. What is provider-specific is structural, and it is visible in each one's own documentation.
This is a snapshot taken from those documents. Model lineups and pricing move constantly, so the durable comparison is on the interfaces and policies rather than on whichever release is current when you read this.
The Dimensions a Call Cares About
Most provider comparisons are written for text applications, so they compare the things a text application notices. A call notices a different set, and four of them are documented facts rather than matters of taste.
- The audio interface: whether the provider accepts and returns speech directly, or only text.
- The commercial surface: which clouds and platforms the models can be bought and run through.
- Data handling: what the provider states about training on your data and how long it keeps it.
- Prompt caching: how a repeated prefix is reused, since a call re-sends its instructions every turn.
- The behaviors on the turn itself, which are real but are not a provider-level property.
That last point is worth separating out. Brevity, interruption recovery and willingness to decline vary between individual releases far more than they follow a provider, and they are set out in the behaviors that make a model good on a call.
Nothing here declares a winner, because the dimensions pull in different directions. A family that fits your contracting constraints may be the one with the interface shape you did not want.
Two Different Shapes of Audio Interface
This is the largest structural difference between the three, and it decides how the rest of the system is built rather than merely which endpoint gets called.
OpenAI documents two architectures for voice agents. One is speech to speech through its Realtime API, where a single model interprets audio, decides what to do and responds in speech within one session. The other is a chained pipeline, which its guidance recommends when each stage needs to be visible or replaceable.
Google's Gemini Live API takes the direct-audio approach too. It accepts raw audio and returns raw audio over a stateful connection, performs voice activity detection on the incoming stream by default, and cancels and discards the in-flight response when it detects an interruption.
Anthropic's model documentation states that its current models support text and image input and text output. Audio is not among the listed modalities, so a call built on that family is a chained pipeline with separately chosen recognition and synthesis in front and behind.
What Each Shape Gives Up
| Dimension | Direct audio session | Chained pipeline |
|---|---|---|
| Number of moving parts | One provider handles the whole turn | Three components chosen and joined by you |
| Visibility of the transcript | Intermediate text may not be the unit of work | Text exists between every stage and can be inspected |
| Replacing one stage | The session is the product, so stages move together | Any stage can be swapped without touching the others |
| Applying rules mid-turn | Harder to intercept before speech is produced | A rule or approval step can sit between the stages |
Neither column is the right answer generally. A call that has to log a transcript, run a policy check and redact a value before anything is spoken has a strong reason to keep the stages apart.
How the Call Reaches the Model
A voice deployment starts on a phone network, and how that audio gets to a model is a real engineering question rather than a detail. The families differ here in documented ways.
OpenAI states that its Realtime API supports connecting to the public phone network, PBX systems, desk phones and other SIP endpoints directly. That removes a translation layer for a team whose audio is already arriving as SIP.
Google documents the Live API as a stateful connection carrying raw audio frames, which suits a media bridge holding the call. Its documentation also notes a duration limit on audio-only sessions, with session-management techniques available to extend beyond it. On a phone line that needs a resumption strategy designed in advance rather than discovered on a long call.
How telephony connects to the rest of the system is a separate subject, handled in how SIP trunking carries calls into a voice agent.
Where You Buy It and What You Sign
Each of these families is reachable through more than one commercial surface, and that is often more consequential than the technical difference between them.
The direct route, with the newest capabilities usually arriving there first and the contract held with the model provider.
Anthropic documents Claude on Amazon Bedrock, Google Cloud Vertex AI and Microsoft Foundry. Google offers Gemini through its own API and Vertex AI, and Microsoft offers OpenAI models on Azure.
The paper, the billing relationship, the regions available and which security review the deployment has to pass.
For a regulated buyer this is frequently the deciding dimension. An organization that has already completed a review of one cloud platform can often reach a model through it with far less friction than signing a new agreement.
Whether any particular residency, retention or contractual arrangement satisfies your obligations depends on your sector, jurisdiction and the data on the call, and is a question for counsel rather than a documentation page.
What Happens to the Call Data Afterwards
Call audio and transcripts are frequently the most sensitive material an organization handles, so what a provider states about training and retention belongs in the comparison rather than in a later review.
All three publish a default position that paid or commercial use is not training material, and the shapes of those statements differ enough to be worth reading individually.
- OpenAI states that data sent to its API is not used to train or improve its models unless you explicitly opt in, that abuse monitoring logs are retained for a bounded period by default, and that eligible customers may be approved for zero data retention.
- Google draws the line at the tier: content submitted to unpaid services may be used to improve its products and seen by human reviewers, while for paid services prompts and responses are not used to improve its products.
- Anthropic states that by default it will not use inputs or outputs from its commercial products to train its models, that conversation content is not retained by default for most API features, and that zero data retention is available by arrangement.
The tier distinction is the one that catches teams out. A prototype built against a free tier and a production deployment on a paid one can sit under materially different terms, and the code looks identical either way.
Read each of these from the provider's current documentation rather than a summary, because they are revised. How the surrounding platform handles the same material is covered in how call data is protected across a voice deployment.
Caching, Because the Prompt Is Re-Sent Every Turn
A voice agent sends its instructions, its business rules and its accumulated conversation to the model on every single turn. That repeated prefix is paid for each time, in tokens and in the time the model spends reading before it can start speaking.
All three families offer some way to reuse a repeated prefix, and the mechanisms are shaped differently enough to change how a prompt should be assembled.
OpenAI states that prompt caching is enabled by default for supported models, that reuse requires the entire rendered prefix to match, and that a minimum cacheable length applies.
Anthropic documents a cache control field that can be applied automatically at the top level or placed on individual blocks, with a short default cache lifetime that refreshes on use and a longer one available.
Google documents implicit caching as enabled by default on its recent models with nothing required from the developer, and minimum token thresholds that vary by model.
The shared consequence is a design rule rather than a preference. Put everything stable at the front of the prompt and everything that changes at the back, because a prefix that shifts on every turn cannot be reused by any of these mechanisms.
Short-lived caches interact awkwardly with call patterns too. A cache measured in minutes serves a busy line well and does nothing for the first call after a quiet hour, which is exactly when a slow first turn is most visible.
What else drives the model bill on a spoken turn is covered in lowering what the model step costs per call.
Running a Fair Comparison on Your Own Calls
Everything above narrows the field on structure. What remains is behavior, and behavior only reveals itself against your own callers, your own prompt and the material you actually intend to ship.
- Decide the interface shape firstA direct audio session and a chained pipeline are different systems, so comparing candidates across that line compares architectures rather than models.
- Hold everything else constantSame prompt, same retrieved material, same fixtures. A prompt tuned for one candidate quietly becomes a test of your tuning.
- Measure the opening of the replyTime to the first word and the length of the first sentence, with the spread reported rather than an average.
- Run long conversationsPut the instruction you care about near the end of a fifteen-turn call, where drift actually shows up.
- Re-run before you commitHosted releases change underneath you, so a result from last quarter is evidence about last quarter.
That last step is the real cost of a hosted family and the honest counterweight to not operating any infrastructure. The trade in full is set out in who runs the inference and what that changes, and the open-weight side of the same shortlist is in the open-weight families teams evaluate for voice.
Expect no clean winner. Splitting turns between families is a design with its own rules, described in how one agent uses more than one model, and what happens when one of them fails mid-call sits in handling a model failure while a caller is on the line.
Conclusion: Choosing the Right AI Voice Model
OpenAI, Claude and Gemini can all serve as the language intelligence behind enterprise voice agents, but the architecture around each one matters. OpenAI and Gemini offer direct real-time audio interfaces for supported models, while a Claude-based phone agent typically uses separate speech recognition and synthesis components.
Choose the architecture that matches your call handling, telephony integration, transcript visibility, privacy obligations and deployment constraints. Then compare realistic calls using the same test fixtures and measure end-to-end latency, interruptions, answer quality and cost rather than choosing by brand alone.
Finally, verify the selected model's current API features, commercial terms and data retention settings before production. Provider capabilities change, so repeat the evaluation when you change models or release a major voice-agent update.
Frequently Asked Questions
There is no general answer, because they differ in shape rather than in quality. One team is decided by which cloud platform it has already cleared for procurement, another by whether it needs the transcript between stages, another by which languages its callers use. Narrow on those structural points first, then test what survives against real call audio.
Both are supported architectures and they suit different deployments. A direct audio session reduces the number of parts you assemble. A chained pipeline keeps text between every stage, which is what you want if a rule, a redaction or an approval has to run before anything is spoken. Decide from the call, not from novelty.
All three publish a default position that commercial or paid API use is not used for training, and each states it in its own terms with its own exceptions and retention behavior. The tier matters in at least one case, where free and paid usage sit under different terms. Read the current documentation and confirm the position for your deployment with counsel.
It can matter a great deal, though rarely to the code. The route changes the contract, the billing relationship, the regions available and which security review the deployment has to pass. Newer capabilities also tend to appear on a provider's own API first, which is worth checking for anything you depend on.
More often than feels necessary, and on a schedule rather than a hunch. Hosted releases change without the deployment changing, so a result from last quarter is evidence about last quarter. Keeping the fixtures and the harness ready is what makes a regular re-run realistic.
