Voice AI Systems Built Beyond the Platform Ceiling

The Platform Ceiling Is Real

Voice AI platforms are everywhere in 2026. Retell, Vapi, Bland, and a dozen others let you spin up a phone agent in an afternoon. They work well for appointment booking, simple FAQ routing, and outbound confirmation calls. Then the requirements get harder. The insurance intake call needs to track fifteen data fields across a branching conversation. The medical triage bot needs to recognize drug names that sound like common English words. The payments flow needs PCI-DSS isolation with no card data touching the LLM. And the platform that got you to the demo breaks under the weight of the real use case.

Production voice agent implementations grew 340% year-over-year in 2025. But S&P Global's survey of over 1,000 enterprises found that 42% abandoned the majority of their AI initiatives before production, up from 17% the year before. McDonald's spent two years and 100+ restaurants with IBM before ending their drive-through voice AI in June 2024 after the system could not handle accents, background noise, or order corrections. Wendy's FreshAI with Google Cloud delivered a 22-second speed improvement by investing in natural language handling for pauses, corrections, and complex customizations. The difference was engineering depth applied to real-world audio.

We build voice systems for the use cases where platforms hit their ceiling. Custom pipelines from best-of-breed components, architected for the latency budget, vocabulary, and regulatory constraints of each engagement.

Latency Is the Conversation Killer

Users abandon voice interactions when the system takes more than 800 milliseconds to respond. The standard voice AI pipeline runs speech-to-text, then an LLM for reasoning, then text-to-speech. Each step adds 100-300ms plus network overhead, pushing total round-trip latency to 1-2 seconds. That gap between when the caller stops talking and when the system starts responding is where trust dies.

We engineer latency out of the pipeline at every layer. On the ASR side, Deepgram nova-3 delivers sub-300ms streaming transcription with 5-7% median word error rate in production, compared to Whisper's batch-oriented architecture that processes audio in 30-second chunks. On the TTS side, Cartesia Sonic produces first audio in approximately 40ms, and ElevenLabs Flash v2.5 in approximately 75ms. Between them, the LLM inference step is where most latency hides: we use speculative response generation, where the system begins formulating likely responses while the caller is still speaking, and streams the first audio tokens before the full response is generated.

Telephony infrastructure adds its own latency. Twilio's Media Streams WebSocket connections introduce jitter that does not appear in lab testing. Their newer ConversationRelay product reduces this overhead. Genesys AudioHook has similar characteristics. SIP trunk selection, codec transcoding, and jitter buffer tuning each contribute 20-50ms that compound silently. We profile the full audio path from microphone to speaker, not just the AI inference step, because that is where real-world latency lives.

Domain Vocabulary Breaks General-Purpose ASR

General-purpose speech recognition optimizes for conversational English. It handles "I'd like to schedule an appointment" well. It does not handle "atorvastatin 40 milligrams QD" or "force majeure clause in section 12.3(b)" or "part number XKCD-4419-REV-C" without help. Custom vocabulary features like phrase boosting exist in most ASR providers, but they are fragile when boosted terms are phonetically similar to common words.

For regulated industries where recognition errors have downstream consequences, we fine-tune ASR models on domain-specific corpora. Medical voice recognition requires training on clinical notes and physician dictations. Legal dictation requires exposure to archaic phrasing and citation formats. Financial services calls reference ticker symbols and proprietary product names that no general model has seen. Google Research demonstrated in 2023 that augmenting training data with synthetic accented speech improved recognition accuracy for underrepresented accents without degrading performance on well-represented ones. We apply the same principle to domain vocabulary: augment the training distribution with the exact terms the system will encounter, validated against real audio from the deployment environment.

Conversation State Is an Engineering Problem, Not a Prompt Problem

Simple voice agents work by feeding the full conversation history into an LLM context window and relying on the model to track what has been discussed. This works for short, linear interactions. It fails on complex multi-turn conversations where the caller provides information out of order, corrects previous statements, or the conversation branches based on collected data.

We build structured dialog management that separates conversation state from the language model. Required fields, validation rules, branching logic, and escalation triggers are defined in a state machine that the LLM cannot override. The language model handles natural language understanding and generation within the boundaries the state machine sets. This means the system can track that the caller already provided their date of birth in turn three even if they mention a different date in turn seven, that an insurance claim requires all fifteen data fields before routing to adjudication, and that the system cannot commit to a price, a deadline, or a coverage determination without explicit authorization.

For healthcare voice AI, this architecture is not optional. HIPAA requires that the system never disclose protected health information to unauthorized parties, which means the dialog manager must enforce access controls at the conversation level, not rely on prompt instructions that the LLM might ignore under adversarial pressure or context window drift.

The Nuance Migration Nobody Is Talking About

Nuance's on-premise speech stack reaches end of sustaining support around June 2026. Roughly 30% of Nuance's customer base is still running on-premise. Microsoft is pushing migration to Dynamics 365 Contact Center and Azure AI services. HCLTech launched a "Nuance Migration Factory" for at-scale transitions. But the real challenge is not swapping ASR engines. It is preserving the dialog logic embedded in VXML grammars that have been refined over years of production use.

Enterprise IVR trees encode business rules, exception handling, and edge cases documented nowhere except the VXML itself. A rip-and-replace migration that starts fresh with an LLM-based system will spend months rediscovering edge cases the old system already handled. We approach Nuance migrations as dialog logic extraction followed by architecture modernization: parse existing VXML flows, extract the state machine and business rules into a portable representation, then implement on modern infrastructure. Realistic timeline is 18-24 months for complex contact centers, not the 90-day plans vendor marketing suggests.

Regulatory Compliance Is Not an Add-On

The FCC confirmed in 2024 that TCPA restrictions on artificial or prerecorded voices apply to AI-generated speech. Any voice AI making outbound calls requires prior express consent for informational calls and prior express written consent for marketing calls. Violations carry $500-$1,500 per call in statutory damages with strict liability, meaning no intent required. The one-to-one consent requirement, postponed to April 2026, requires individual consent per seller, closing the lead-generation loophole that many voice AI vendors have been operating through.

Beyond TCPA, voice AI deployments face PCI-DSS requirements when handling payment data (card numbers must never reach the LLM or appear in logs), HIPAA requirements for healthcare interactions (BAAs, minimum necessary access, audit trails), and state-level requirements from Colorado's AI Act (effective June 2026) and Illinois' BIPA. We architect compliance into the voice pipeline from the beginning: consent capture and recording mechanisms, PCI-scoped audio routing that isolates card data from the AI processing path, call recording consent handling that adapts to the caller's jurisdiction, and audit trails that document exactly what the system said and why.

Build, Buy, or Compose

The build-vs-buy framing is outdated for voice AI in 2026. The real decision is what to compose. Open-source frameworks like Pipecat (from Daily) and LiveKit Agents provide vendor-neutral pipeline orchestration. ASR, LLM, and TTS components can be swapped independently. Telephony infrastructure connects through standard SIP trunks or WebRTC. The question is where your use case needs custom engineering versus where an off-the-shelf component is genuinely sufficient.

We help teams answer this honestly. If the use case is appointment scheduling with standard English speakers and no regulatory constraints, a platform at $0.07-0.09 per minute is the right answer. We will tell you that. If the use case involves domain-specific vocabulary, multi-turn state management, regulated data, or call volumes where per-minute pricing becomes the dominant cost driver, custom composition eliminates vendor lock-in. At 50,000 minutes per month, the spread between platform pricing and a self-hosted pipeline can exceed $5,000 monthly. A custom build at $50K-300K upfront eliminates compounding per-minute charges entirely.

Every engagement starts with actual requirements: latency budget, vocabulary complexity, regulatory exposure, call volume projections, and existing telephony infrastructure. We design the architecture, select components, build customizations, and deliver a system the team owns without per-minute vendor dependency.