精选

← 返回 OPC日报

ElevenLabs Blog · 产品 ·

Voice AI infrastructure: The 2026 stack explained

Introducing Eleven v4Meet Eleven v4, our most emotive model yet.Discover Skip to contentProductsSolutionsCustomersResourcesEnterprisePricingLog inSign upContact salesLog in Learn moreContact salesBlogResourcesVoice AI infrastructure: The 2026 stack explainedWritten byJack LimebearPublishedNov 6, 2026ListenListen to this article0:000:000:001.0xLearn moreContact salesOn this pageIntroductionHow voice AI infrastructure fits togetherSpeech recognition: Turning audio into textTurn taking: Knowing when to speak and when to stopThe language model: Reasoning and generating the replyOrchestration: Routing work between agents and humansTools and MCP: Letting the agent actKnowledge and memory: Facts and caller contextSpeech synthesis: Giving the agent a voiceTelephony and transport: Connecting the agent to callersGuardrails: Keeping the agent on taskHow ElevenAgents does itTesting and experiments: Proving the agent worksObservability: Measuring what happens in productionScale, reliability, and human handoffWhat to look for in production-ready voice AI infrastructure in 2026Frequently asked questions about voice AI infrastructureVoice AI infrastructure is the set of models, connections, tools, and services that lets an AI agent listen to a call, decide what to do, take action in other systems, and answer out loud. And, with leading providers like ElevenLabs, all of this happens in a fraction of a second.A production voice agent runs on a stack of layers that includes speech recognition, turn-taking, a large language model, orchestration, tools, knowledge retrieval, speech synthesis, telephony, and monitoring.This guide explains the AI agent architecture behind a working voice agent. Each section converts what a layer does, why it matters in the wider infrastructure, and how ElevenAgents implements it, with Eleven v4 as the example voice layer.Build at scale. Talk to the ElevenAgents teamFirst name*Last name*Company website*Business email*Let's goLooking for help? Visit our Help CenterHow voice AI infrastructure fits togetherA voice agent turns audio into text, reasons based on that input, generates an answer, and then turns that response back into audio that the customer hears. That cascade of actions is made up by a robust stack of AI agent architecture.Here are the main layers that contribute to voice AI infrastructure in 2026:Speech recognition: Automatic Speech Recognition (ASR) technology converts the caller's audio into text for processing.Turn taking: Turn taking decides when the caller has finished and when to stop talking, allowing for more fluid, responsive conversation.Language model(s): Interprets the transcribed request and generates a reply, after pulling necessary context.Orchestration: An AI orchestration layer routes the conversation between prompts, subagents, tools, and humans.Tools and MCP: Connect the agent to external systems, such as CRM platforms or contact centers, to perform tasks.Knowledge and memory: Supply facts and caller context via Retrieval Augmented Generation.Speech synthesis: Uses Text to Speech to transform the text generation reply into audio.Transport and telephony: Carries audio between the agent and the caller.Guardrails, testing, and monitoring: Keeps the agent on task and measurable.Together, these components work to create real-time conversations between AI agents and calling customers.Voice AI architecture diagramThe flow isn’t a straight line. ElevenAgents starts speaking after it receives “enough words and a comma” from the language model, meaning that audio begins playing before the full reply exists. Toll calls and retrieval run in the middle of a turn. Interruptions can arrive and are handled naturally by the agent while it generates speech. Two ways to build voice AI architecture ElevenAgents supports two voice modes. In Custom mode, you pick the language model, the Text to Speech model, and the speech recognition settings yourself.In Realtime mode, you select an Eleven Realtime model and ElevenLabs manages those settings for you. Eleven Realtime v1 Mini is the default and has the lowest latency and cost. Eleven Realtime v1 Max reasons before it responds, which suits more complex instructions and cases where multi-step tool use is required.The system prompt, chosen voice, tools, attached knowledge base, and turn settings work the same in both modes. Custom mode is the better choice when you need a specific language model, a custom LLM, or a specific TTS model (for performance or latency optimization.)Speech recognition: Turning audio into textSpeech recognition, also called ASR or speech to text, is the first layer of the stack.What it doesThe model transcribes the caller's audio continuously while they speak. Doing so continuously allows the rest of the pipeline to start working before the sentence ends, saving on latency. Advanced models will handle everything from thick accents and different speaking speeds to background noise.Why it matters for voice AI infrastructureTranscription is arguably one of the most important layers in the stack because any errors here will permeate through the rest of the stages. Mishear a booking reference? Wrong customer record actioned. Get a medication’s name wrong? Potential disaster. In its Trust Framework for Healthcare, ElevenLabs shows how high the stakes get with one misheard syllable: hydroxyzine versus hydralazine.How ElevenAgents does itElevenAgents uses Scribe v2 Realtime, which transcribes speech in approximately 150 ms. ElevenLabs trained it for noisy backgrounds, diverse accents, and identifiers such as names, emails, and IDs. On the FLEURS multilingual benchmark across 30 languages, ElevenLabs reported the lowest word error rate of any low-latency ASR model. You enable it under the Advanced section of the agent configuration. In the same white paper mentioned above, ElevenLabs reports that Scribe v2 Medical leads every model on the public Eka Medical ASR benchmark at a 7.0% word error rate as of publication.Accurate (and rapid) transcription gives the next layers the clean input they need without adding significant latency.Turn taking: Knowing when to speak and when to stopTurn taking is the set of rules that decides when the agent replies and when it listens. Being able to effectively understand turns in conversation and act upon them is what separates a conversation from a legacy IVR phone menu.What it doesThere are a few functions that a turn taking component provides to voice AI infrastructure:Voice activity detection (VAD): VAD classifies incoming audio as speech or silence, dozens of times per second.End-of-turn detection: Judges whether the caller has finished, using tone, rhythm, and word context.Barge-in: Lets the caller interrupt, at which point the agent stops talking to allow the user to continue.Backchannels: Short acknowledgments such as "yeah" that do not take the turn.Why it mattersA gap of about 200 ms is the typical gap between speakers in a human conversation. An agent that answers too early cuts callers off, while one that waits too long creates silences that make the call feel broken or unresponsive. For example, consider a caller who says "I'd like to book Friday afternoon, but..." and pauses. A silence timer replies at the pause and misses the qualification. A model that reads the words and the rhythm waits, allowing the user to finish, "It can’t be between 3 and 5 p.m., as I’ll be busy.”One agent misqualifies the caller’s intention and schedules the meeting at the wrong time. The other, with effective turn taking abilities, makes sure to avoid the specified time period. How ElevenAgents does itElevenAgents runs a proprietary turn-taking model that analyzes tone, rhythm, and word context instead of relying on silence timers alone. When a caller interrupts, the signal that stops the agent lands in under 200 ms. ElevenLabs measures a median of ~150 ms from request to audible speech with Eleven v4 Turbo on ElevenAgents. The behavior is configurable via Conversation flow:SettingWhat it controlsTurn eagernessEager responds at the earliest opportunity. Normal is the balanced default. Patient waits longer and suits phone numbers, addresses, and emails.Speculative turnStarts the language model reply before the caller is confirmed finished. It cuts latency and can raise LLM cost.InterruptionsTurn them off when every word must be heard, such as legal disclaimers and safety instructions.Ignore listWords such as "right" that should not interrupt the agent.Skip turn toolKeeps the agent quiet when the caller says "give me a minute".Turn timeoutTakes a turn after 1 to 30 seconds of silence. Use 5 to 10 seconds for casual conversations and 10 to 30 when callers need time to think.Soft timeoutPlays filler speech such as "Hhmmmm...yeah." when the model is slow. Range 0.5 to 8 seconds, with 3 seconds recommended.Noisy environment detectionAn Enterprise setting that detects background voices and asks the caller to move somewhere quieter, at most once per conversation.Workflows can change turn eagerness mid-call. Your client also receives a vad_score event, a 0 to 1 probability that the user is speaking. Well-tuned turn-taking decides whether callers experience the agent as attentive or abrupt. For more on the concepts, see our full guide on turn taking.The language model: Reasoning and generating the replyThe large language model (LLM) is the reasoning layer. It reads the transcript, the instructions, and the conversation so far, then writes the response or decides to call a tool.What it doesA scripted voice bot recognizes a handful of commands. An LLM interprets requests phrased many different ways, holds context across turns, and decides when to retrieve information or hand off.Why it mattersModels differ in reasoning depth, speed, and cost. For a live voice conversation, response time matters as much as answer quality, so the model choice is a latency decision and a cost decision.How ElevenAgents does itElevenAgents gives you one interface to models from Google, OpenAI, and Anthropic, plus open models hosted by ElevenLabs. You can also connect your own LLM by specifying an endpoint and storing credentials as secrets. ElevenLabs passes third-party LLM costs through at the provider's published rate with no markup.You can choose the model per subagent. The Enterprise Conversational Agent Playbook, Volume 2 recommends reserving higher-performance models for complex reasoning and using cheaper, low-latency models for tasks such as intent classification and FAQs.Providers have outages. LLM cascading handles them: if the primary model returns an API error, times out, or sends an empty response, ElevenAgents tries a backup sequence. The default timeout from primary to backup is 4 seconds, adjustable from 2 to 15 seconds, and the system makes at least 3 attempts across the available models. A custom LLM used as the primary model retries itself at least 3 times, unless you define your own backup order. The right model per task keeps the agent both fast and affordable.Orchestration: Routing work between agents and humansAgent orchestration coordinates the prompts, subagents, tools, and people involved in finishing a task. It decides what happens next at every point in the call.What it doesOne agent with one long prompt grows hard to manage as responsibilities pile up. Orchestration splits the work into stages, each with its own instructions, model, and tools, and moves the caller between them.Why it mattersSmaller prompts are easier to test and cheaper to run, because each interaction loads less text. Splitting a flow lets a cheap model handle triage and FAQs and hands off to a stronger model only when a request needs it.How ElevenAgents does itWorkflows in ElevenAgents are a visual builder for branching conversation graphs. The nodes are:Subagent: Changes or overrides the base agent, including its prompt, LLM, voice, tools, and knowledge base.Say: Speaks one message, either literal text or an LLM-generated line.Dispatch tool: A dedicated step that guarantees a tool call, with separate success and failure paths.Update state: Assigns dynamic variables as the flow passes through.Agent transfer: Hands the call to a different agent.Transfer to number: Moves the call from the AI agent to a human over the phone system.End call: Closes the conversation.Edges between nodes carry natural-language conditions that the LLM evaluates in real time. A support flow might start with a routing subagent on a fast model, send billing questions to a billing subagent on a stronger model, and finish with a transfer-to-number node when the caller asks for a person. After launch, the Workflow tab in analytics shows entries, average time spent, terminations, and edge distribution for each node.Orchestration turns a single agent into a system you can design, measure, and improve one node at a time.Tools and MCP: Letting the agent actFunction calling, also called tool calling, lets an agent request actions from external software that you have connected to your tech stack. It turns a question-answering interface into an agentic AI system that is able to complete work and interact with other tools on your behalf.What it doesTake a caller who asks to move tomorrow's appointment to Friday. The agent retrieves the booking, checks Friday's availability, updates the scheduling system, and confirms the change. Each step is a tool call that takes action.Why it mattersTools are where conversations produce outcomes. Each tool needs clear inputs and outputs, and the agent needs to know whether a call succeeded before it tells the caller it did.How ElevenAgents does itElevenAgents supports client tools that run in the user's application, webhook tools that call external APIs, code tools that run custom JavaScript in a sandbox on ElevenLabs infrastructure, MCP tools, and built-in system tools. The system tools include end call, language detection, agent transfer, transfer to number, skip turn, play keypad touch tone, and voicemail detection. You can add ambient audio while a tool runs and control whether callers can interrupt during execution.The Model Context Protocol (MCP) is an open standard for how applications give context and tools to language models. ElevenAgents connects to MCP servers over SSE and streamable HTTP. Workspace admins opt in first, since MCP is disabled by default for every workspace. You then choose an approval mode for each server: Always Ask, Fine-Grained Tool Approval, or No Approval. Client events such as client_tool_call and mcp_tool_call report tool activity, and the MCP event reports four states: loading, awaiting approval, success, and failure. Reliable tools give the agent real-world reach, and clear approval rules keep that reach under control.Knowledge and memory: Facts and caller contextRetrieval-augmented generation (RAG) gives the agent facts from your own content, such as an internal fact database or wider client history log. Memory and personalization give it context about the caller that allows your agent to improve personalization and utility for your business.What it doesA language model has a training cutoff and no access to your documentation or records. RAG searches a knowledge base for each query, retrieves the most relevant passages, and supplies them to the model to give it the extra context needed for a useful response. Personalization data tells the agent who is calling and what is relevant to them.Why it mattersGrounding replies in approved content keeps answers accurate. Retrieval speed matters too, as every millisecond it adds is another that your customers will be waiting for. Personalization functions also stop callers from having to repeat information across different calls, as your agents will have the CRM records on hand at the start of each new session.How ElevenAgents does itYou add documents to the knowledge base as files (PDF, DOCX, TXT, MD, HTML, or EPUB, up to 20 MB each), URLs, or pasted text. A document uses full context when its extracted text fits in the prompt, up to roughly 300,000 characters. Larger documents use RAG. Retrieval can run every turn or on demand, and the default usage mode is Auto, which uses RAG when it is enabled and the document is indexed.ElevenLabs works on retrieval speed directly. Its engineering article, How we reduced RAG latency, documents a drop in median RAG latency from 326 ms to 155 ms using models self-hosted on infrastructure ElevenLabs already operates.For caller context, ElevenAgents uses dynamic variables written as {{ var_name }}, which fill messages, system prompts, and tools. For inbound calls over Twilio, SIP trunk, WhatsApp, and SMS, ElevenAgents can fetch initiation data from your server before the conversation starts. Your server looks up the caller by caller_id and returns the variables, for example, from your CRM, with a response of up to 256 KB. After the call, post-call webhooks deliver evaluation results and extracted data to your external systems.Together, retrieval and personalization give the agent both the right facts and the right caller.Speech synthesis: Giving the agent a voiceText to speech (TTS) turns the language model's reply into spoken audio. It is the layer the caller actually hears.What it doesThe model converts text into audio with natural pacing and emotional delivery. In a live call it streams audio in chunks so playback starts before the full reply is generated.Why it mattersLatency and delivery both shape how callers rate the call. Two numbers matter for real-time work: the time to generate audio and the time before the caller hears the first sound. Names, drug names, and technical terms also need correct pronunciation, or callers lose confidence in the agent.How Eleven v4 powers the voice layerEleven v4 is ElevenLabs' most emotive Text to Speech model yet, built on an entirely new architecture. Eleven v4 Turbo is the low-latency variant for real-time use, such as agents, optimized with ElevenAgents as one system.Eleven v4 TurboFigureMedian inference latencyAbout 100 msMedian time to first speechAbout 150 msLanguages90+ElevenLabs measured the latency figures in September 2026 over WebSocket streaming with network latency removed. Eleven v4 ranked first on the Artificial Analysis Provider Voice Arena Leaderboard in September 2026*, and about 75% of listeners preferred it in blind tests**.* Artificial Analysis, Provider Voice Arena Leaderboard, Sept 2026** Based on head-to-head user preference testing against Cartesia Sonic 3.6, Inworld TTS-2, Google Gemini 3.8 Flash-Lite TTS, and Google Gemini 3.8 Flash TTS.Eleven v4 supports inline audio tags such as [laughs] for delivery and IPA phoneme support for pronunciation. In healthcare agents, that pronunciation control covers medical terms. ElevenAgents adds agent-level controls on top. Multi-voice support lets one agent switch between voices for multi-character conversations. Speed ranges from 0.7x to 1.2x. Each additional language can use its own voice. Pronunciation dictionaries take IPA or CMU notation for specific words. A fast first sound and an accurate, expressive voice make the agent feel like a conversation partner.Telephony and transport: Connecting the agent to callersThe transport layer carries audio between the caller and the agent. Depending on which telephony channel you use, the integrations you’ll include in your voice AI infrastructure here will change.What it doesPhone calls travel over a telephone network and SIP, while browser and app conversations travel over WebRTC or WebSocket. A website widget wraps the browser connection in a ready-made interface.Why it mattersEach channel has different setup needs and different audio constraints, so decide your channels early. A phone-first deployment needs trunking and number management. A product deployment needs an SDK.How ElevenAgents does itSIP trunking: Connects your phone system directly to ElevenLabs, inbound and outbound. It w

查看来源原文