A proof of concept exists to make one decision under a deadline: whether to commit to this supplier. It is not a rehearsal for launch, and treating it as one is how trials end with a signed contract and no more certainty than the demo provided.
- One decision, not a rehearsal A trial answers whether to commit. Readiness for live traffic is a later exercise with a different standard.
- Hold the setup constant Same documents, same calls, same phone path, same effort. Otherwise the comparison measures configuration, not capability.
- Write the decision rule first Agree what result would mean no before the first call runs, or the result will be read to fit the decision already preferred.
The useful question a trial answers is narrow. Given our calls, our documents and the people who would operate this, does the thing behave well enough to be worth the commitment, and where exactly does it stop behaving.
Almost everything that makes a trial predictive comes down to two disciplines: running it on calls you actually receive, and holding the setup identical between suppliers so the comparison is about the product rather than about who spent longer configuring it.
What a Trial Has to Decide, and What It Cannot
A trial is a purchasing instrument. It runs for a few weeks, on a fraction of the traffic, with a configuration nobody has had time to refine, and every conclusion has to be read with those limits attached.
Whether the agent handles your real call reasons, how it behaves when it does not know, and how much work the setup took.
Likely resolution rates, typical call length and how much of the conversation ends up needing a person.
Behavior at full volume, reliability across a year, or how the supplier responds when something goes wrong at an awkward hour.
Full integration into every system. Integration scope is a contract question and will consume the whole trial if allowed to.
The third card deserves respect rather than a workaround. A trial that tries to prove reliability ends up too long, too expensive and still inconclusive, and the questions it was trying to answer belong in what the supplier is asked to evidence and commit to in writing.
Those written commitments are a separate exercise, set out in asking suppliers to evidence rather than assert.
Choosing the Calls the Trial Runs On
The single biggest determinant of whether a trial predicts anything is which calls it runs on. A supplier will offer a scenario. Use yours instead, and choose it deliberately rather than by convenience.
- The highest volume reason Whatever your line actually receives most. If the agent cannot carry this one, nothing else in the trial matters.
- One awkward reason A call where people speak over each other, read out long references, or change their mind halfway through.
- One that must reach a person A reason where the correct outcome is a transfer, so the handover and what travels with it can be observed.
- One the documents do not cover A question with no approved answer, to see whether the agent says so or invents something plausible.
- One low-stakes edge A wrong number, a silent caller, an answering machine, or somebody asking for a department that no longer exists.
Five reasons is usually enough, and fewer than five is usually too few to be persuasive. The fourth is the one that separates suppliers most sharply, because a system with no honest way to say it does not know will fill the gap confidently.
Behavior in that situation is a design property rather than an accident, which is why it is worth probing directly. The mechanics are covered in keeping a voice agent from inventing answers.
What Has to Be Held Constant Across Suppliers
A trial run differently for each supplier compares the trials, not the products. The list of things to fix is short and every item on it changes the result if left free.
| Hold constant | Why it matters | What drifts if you do not |
|---|---|---|
| The source documents | Answer quality mostly reflects the material behind it | One supplier is judged on a better prepared knowledge set |
| The call reasons and scripts | Difficulty varies enormously between reasons | The comparison rewards whoever got the easier calls |
| The phone path | Carrier audio differs from a browser session | Audio quality differences get read as product differences |
| Who configures it | Supplier-built setups hide how hard setup is | You buy a configuration you cannot reproduce yourself |
| The time allowed | More tuning time produces better results anywhere | The comparison measures attention, not capability |
The fourth row is the one most often waived, and it is the most expensive to waive. If the supplier's own team builds the trial configuration, you learn what their experts can do, not what your team will be able to do in month three.
A reasonable compromise is to let the supplier demonstrate first and then have your own operations person rebuild the same thing unaided, with the time it takes recorded. That number predicts the cost of every future change.
The Failure Modes a Demo Hides
Demos are not dishonest so much as selected. They show the path that works, on the phrasing that works, for one turn longer than the point where things usually go wrong.
- The second and third turn. Most demos end while the conversation is still simple, and difficulty compounds after the first exchange.
- The browser shortcut. A test call in a browser skips carrier codecs, network conditions and real transfer behavior, so it flatters audio and timing.
- The rehearsed phrasing. Callers do not speak in the words the prompt was written against, and the gap shows up as repeated requests to confirm.
- The silent middle. Time spent waiting on a lookup is invisible on a slide and very audible on a call.
- The handover. A transfer that connects is not the same as a transfer that arrives with the conversation attached.
A browser test call is genuinely useful for checking a script before a number goes live, and Pulastya AI provides one for that purpose. Run the scored part of a trial over the real phone path anyway, because the two are answering different questions.
The handover point is worth watching closely rather than ticking. What should arrive with the person is set out in what travels with a caller when a live call is handed over. You can also try an agent on a test call to see the difference for yourself.
What to Measure, and Who Does the Scoring
A trial produces a small number of decision-relevant outcomes, not a research dataset. Keep the list short enough that a person can score every call by hand, because hand scoring is what makes the result defensible.
- Did the caller get what they rang for, judged by someone listening to the call rather than by a system flag.
- Was anything recorded about the caller wrong, and would that error have caused a problem downstream.
- Did the caller have to repeat themselves, and how often within a single call.
- When the agent transferred, did the person receiving it have to start the conversation again.
- How long did each change to the agent take, and who was able to make it.
The last item is not about the calls at all, and it is the one that predicts the second year. An agent that only the supplier can change has a different cost profile from one an operations lead can adjust the same afternoon.
Platforms differ sharply on this point, and it is measurable during a trial rather than after it. A four-step setup aimed at business teams rather than developers, which is how Pulastya AI is configured, is a different proposition from one where each edit becomes a development request.
Scoring should be done by someone who handles these calls today, without knowing which supplier produced which recording. A vendor-supplied summary of its own performance is evidence about the summary, not about the calls.
Call length and turn counts collected here also feed the commercial comparison, since they are the reference profile described in what actually makes up a per-minute figure.
Deciding at the End of the Trial
Most trials end ambiguously because nobody agreed in advance what a poor result would look like. Writing the rule down beforehand is a five-minute task that changes the quality of the decision entirely.
- Set the rule before the first call Name the outcome on each measure that would mean no, and have the people who will live with the decision sign up to it.
- Separate fixable from structural A weak answer on one call reason is usually configuration. An inability to reach a person cleanly is not.
- Price the gap, do not ignore it Where something fell short, cost the work to close it and add that to the commercial comparison rather than to a wish list.
- Decide what happens if you stop Agree what you keep: the documents, the recordings, the scored results and the call records produced during the trial.
A trial that ends in no is not a failed trial. It has bought a decision cheaply, and the scored recordings it produced are reusable material for the next supplier.
Once a supplier is chosen, the standard changes from is this worth buying to is this safe to put in front of customers, which is a different discipline described in the checks a chosen system passes before live traffic. A structured scoring template for the selection itself is set out in the evaluation criteria to score suppliers against.
Conclusion
An AI voice agent proof of concept should provide evidence for a purchasing decision, not another polished demonstration. Test the call reasons your business actually receives, keep the documents and phone conditions consistent across suppliers, and have your own team score the recordings. Pay particular attention to accurate answers, honest handling of unknown questions, clean transfers, and the effort needed to update the agent.
Agree on pass-or-stop criteria before the trial begins, then use the results to commit, request a priced improvement, or move on to another supplier. Retain your call scripts, recordings, transcripts, and scores whichever way you decide. A promising trial justifies the next stage of evaluation; it does not replace the separate checks needed before an AI voice agent handles live customer calls.
Frequently Asked Questions
Long enough to collect a scoreable number of calls on each chosen reason, and short enough that the configuration does not quietly get refined into something unrepresentative. Set the end date at the start and extend only for a reason you can name. A trial without an end date becomes an unpaid deployment.
Ideally both, in that order. Let the supplier show what is possible, then have your own operations person rebuild the same setup unaided and record how long it took. The second number tells you what every future change will cost, and it is not something a supplier-built configuration can reveal.
Around five is the usual working answer: the highest volume reason, one awkward one, one that must reach a person, one your documents do not cover, and one edge case such as a wrong number or a silent caller. Fewer than that rarely convinces anyone, and many more dilutes the attention available for scoring.
It is useful for walking through a script quickly, but it is not the same call. A browser session skips the carrier path, real network conditions and live transfer behavior, all of which change audio quality and timing. Use it to iterate, then score the trial over the phone path you would actually use.
Agree it beforehand rather than afterwards: the documents you prepared, the call recordings and transcripts produced, and the scored results. Those are the expensive part of the exercise and they carry over to the next supplier, which makes a second trial considerably cheaper than the first.
