Voice agents on the phone line: latency, handoffs and honesty

Voice agents have crossed a threshold. The best ones now hold a natural phone conversation, understand accents and background noise reasonably well, and can book appointments or answer account questions without the caller pressing a single button. They are also less forgiving than text agents in every way. A two-second pause that is fine in chat feels broken on a call, and a caller cannot scroll back to reread a confusing answer. Here is what we have learned putting voice agents on real phone lines for service businesses.

Latency is the whole experience

People expect a reply to begin within about 700 to 900 milliseconds of finishing their sentence. Beyond roughly 1.5 seconds, callers start repeating themselves or asking "hello?", which the system then has to handle as well. Every component in the pipeline eats into that budget:

• End-of-speech detection, deciding the caller has actually finished rather than paused, typically 200 to 400 milliseconds.
• Speech recognition of the final words, usually fast when streaming.
• Model response, measured to the first token rather than the full reply.
• Speech synthesis of the first phrase, again streamed.
• Telephony transport, which adds its own delay depending on carrier and region.

We design to a budget and measure each stage separately on every call. Practical tactics include streaming everything, keeping system prompts compact, choosing a fast model tier for the conversational layer, hosting in the region closest to the callers, and starting speech with a short, natural acknowledgment while a slower tool call completes. "Let me check that for you" buys a second or two honestly; dead air does not.

Designing for interruptions and messy speech

Real callers interrupt, change their minds, spell names letter by letter, read out order numbers with pauses, and talk to someone else in the room. The agent needs to stop speaking immediately when the caller starts, discard the rest of its planned reply, and respond to what was actually said.

A few design choices make a large difference:

1. Confirm critical details by reading them back, especially numbers, dates and email addresses, and accept corrections gracefully.
2. Keep replies short. Two sentences at a time is plenty on the phone; long lists should become a follow-up text message.
3. Use constrained collection for structured data. When collecting a date of birth, tell the recognizer to expect a date.
4. Offer a way out at any point. "You can say 'agent' at any time to reach a person" should be true and should work instantly.

On a phone call, clarity beats cleverness. The best voice agents sound less impressive and get more done.

Handoffs that do not start from zero

The fastest way to make callers hate a voice agent is to transfer them to a human who asks them to repeat everything. Every handoff we build passes context along with the call: the caller's verified identity, the reason for the call, what the agent already tried, and a two-line summary. It appears on the human agent's screen before they pick up.

We also decide handoff rules explicitly rather than leaving them to the model's judgment. Typical triggers include a caller asking for a person, two failed attempts to understand the same request, any mention of a complaint, emergency or legal matter, and any request outside the agent's approved scope. Outside business hours, the agent offers a callback slot instead of a transfer and books it directly.

Honesty, consent and compliance

Callers should know they are speaking with an automated system. We disclose it in the greeting, in plain words, and we do not give the agent a fake human backstory. In our experience disclosure has little effect on completion rates, and it avoids a far worse outcome when a caller discovers it later.

Other obligations vary by jurisdiction and industry, so we review them with the client's counsel for each deployment. Typical items include:

• Call recording notices and consent where required, with recordings retained according to policy.
• Restrictions on outbound automated calls, which are much stricter than inbound in many places.
• Identity verification before discussing account details, using the same checks human agents follow.
• Redaction of card numbers and other sensitive data from transcripts and logs.

We generally recommend starting with inbound calls only. Outbound voice automation carries more regulatory risk and more potential to annoy people.

Measuring whether it works

Containment rate, the share of calls completed without a human, is the metric vendors like to quote. On its own it is misleading, since hanging up on callers also improves containment. We track a balanced set:

• Task completion, verified against the system of record, such as appointments actually booked.
• Transfer rate and the reasons for transfer.
• Caller abandonment during the agent portion of the call.
• Repeat calls from the same number within 48 hours, a strong signal of unresolved issues.
• Median and 95th-percentile response latency.

For a multi-location clinic group, a scheduling-focused agent now completes about 58% of booking and rescheduling calls end to end, with median response latency around 850 milliseconds. Front-desk staff handle fewer routine calls during peak morning hours, and after-hours callbacks are booked instead of lost to voicemail.

Voice work lives in our voice agents practice and usually shares its knowledge and tools with a text-based customer support chatbot, so both channels give the same answers. Scheduling and CRM connections are standard workflow automation.

Hear what your line could sound like

Tell us which calls you want to automate and roughly how many you receive each week. We will scope the agent, the handoffs and the compliance review, and send a fixed-price quote within 24 hours.