{"id":2619,"date":"2026-11-16T08:45:07","date_gmt":"2026-11-16T08:45:07","guid":{"rendered":"https:\/\/www.capanicus.com\/blog\/?p=2619"},"modified":"2026-11-16T09:07:05","modified_gmt":"2026-11-16T09:07:05","slug":"ai-voice-agent-freeswitch","status":"publish","type":"post","link":"https:\/\/www.capanicus.com\/blog\/ai-voice-agent-freeswitch\/","title":{"rendered":"How to Build a Scalable Real-Time AI Voice Agent with FreeSWITCH"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Callers no longer want to press 1 for billing and 2 for support. Instead, they want to say what they need and get a useful answer. A real-time AI voice agent with FreeSWITCH makes that possible on the phone lines you already run.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The key idea is a clean split of responsibilities. FreeSWITCH manages the call: SIP signaling, media, codecs, transfers, and recording. Meanwhile, external AI services manage the conversation. They turn speech into text, decide what to say, and then turn the reply back into speech.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">By the end of this guide, you will know how to connect the two step by step. You will also learn to keep latency low, support barge-in, hand off to humans, and scale to thousands of concurrent calls.<\/span><\/p>\n<h2><b>Why Use FreeSWITCH for AI Voice Agents?<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">FreeSWITCH is an <a href=\"https:\/\/www.capanicus.com\/blog\/freeswitch-vs-asterisk\/\">open-source telephony platform<\/a> designed for high call concurrency. In fact, it already handles the hard parts of voice infrastructure that an AI model should never touch:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\"><a href=\"https:\/\/www.capanicus.com\/blog\/sip-trunking-architecture\/\">SIP trunking<\/a> and WebRTC endpoints<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Codec negotiation and transcoding (G.711, G.722, Opus)<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">RTP media handling and call recording<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Dialplan routing, queues and call transfers<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Event Socket Layer (ESL) for instant call control from external apps<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Because the intelligence lives <a href=\"https:\/\/www.capanicus.com\/blog\/integrate-ai-voice-agents-pbx\/\">outside the PBX<\/a>, you can upgrade your speech or language models without touching telephony. Likewise, you can tune telephony without re-training anything. As a result, that separation makes a <a href=\"https:\/\/www.capanicus.com\/ai-voice-agent-development\">FreeSWITCH AI voice agent<\/a> production-ready rather than a demo.<\/span><\/p>\n<h2><b>Core Architecture of an AI Voice Agent with FreeSWITCH<\/b><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2623\" src=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Core-Architecture-of-an-AI-Voice-Agent-with-FreeSWITCH.webp\" alt=\"Core Architecture of an AI Voice Agent with FreeSWITCH\" width=\"1000\" height=\"562\" srcset=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Core-Architecture-of-an-AI-Voice-Agent-with-FreeSWITCH.webp 1000w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Core-Architecture-of-an-AI-Voice-Agent-with-FreeSWITCH-300x169.webp 300w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Core-Architecture-of-an-AI-Voice-Agent-with-FreeSWITCH-768x432.webp 768w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Every real-time voice agent is built from five layers that pass audio and text within a continuous loop.<\/span><\/p>\n<h3><b>Telephony Layer (FreeSWITCH)<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">FreeSWITCH receives the call from a SIP trunk, carrier, or WebRTC client and answers it. Then it negotiates codecs and routes the call to an AI context in the dialplan. It also controls the call for its entire life.<\/span><\/p>\n<h3><b>Real-Time Audio Streaming Bridge<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">A streaming module forks the live call audio to your bridge service over a WebSocket, frame by frame. The bridge is your own application. It connects FreeSWITCH to the AI services and also sends synthesized speech back into the call.<\/span><\/p>\n<h3><b>Speech-to-Text, LLM and Text-to-Speech Layers<\/b><\/h3>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Streaming STT<\/b><span style=\"font-weight: 400;\"> converts caller audio into partial and final transcripts, with endpointing to detect when the caller has finished speaking.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>The LLM<\/b><span style=\"font-weight: 400;\"> reads the transcript, conversation history, and business rules, then decides what to say and which backend tools to call.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Streaming TTS<\/b><span style=\"font-weight: 400;\"> converts the reply into audio and starts playback before the full reply has been synthesized.<\/span><\/li>\n<\/ul>\n<h3><b>Architecture Diagram: End-to-End Call Flow<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Audio flows from the caller through FreeSWITCH to the bridge. Then it goes around the STT, LLM, and TTS loop and back into the call. FreeSWITCH executes transfers to human agents when the bridge requests them over ESL.<\/span><\/p>\n<h2><b>Step-by-Step: How to Build the AI Voice Agent<\/b><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2624\" src=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Step-by-Step-How-to-Build-the-AI-Voice-Agent.webp\" alt=\"Step-by-Step How to Build the AI Voice Agent\" width=\"1000\" height=\"562\" srcset=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Step-by-Step-How-to-Build-the-AI-Voice-Agent.webp 1000w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Step-by-Step-How-to-Build-the-AI-Voice-Agent-300x169.webp 300w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Step-by-Step-How-to-Build-the-AI-Voice-Agent-768x432.webp 768w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Module and provider choices vary, but the build sequence stays the same.<\/span><\/p>\n<h3><b>Step 1: Prepare FreeSWITCH, SIP Trunk and AI Providers<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Before writing code, make sure you have:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A recent, stable FreeSWITCH release with a SIP trunk or WebRTC endpoint that can receive test calls.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Accounts with streaming STT, LLM, and TTS providers, or self-hosted models<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">A server for the bridge service on the same private network as FreeSWITCH<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Event Socket Layer (ESL) enabled and bound to a private interface.<\/span><\/li>\n<\/ul>\n<h3><b>Step 2: Install an Audio Streaming Module<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Core FreeSWITCH does not stream call audio to WebSockets out of the box. Instead, the usual approach is a community module such as <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">mod_audio_fork<\/code> or <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">mod_audio_stream<\/code>. Then compile it against your FreeSWITCH version and load it in <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">modules.conf.xml<\/code>. Choose a module that supports bidirectional audio if you want to stream TTS back without writing temporary files.<\/span><\/p>\n<h3><b>Step 3: Route Calls to the AI in the Dialplan<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Create a dialplan extension that answers the call, starts the audio fork to your bridge, and holds the call open:<\/span><\/p>\n<pre style=\"background: #f6f8fa; border: 1px solid #e1e4e8; border-left: 4px solid #fb8500; border-radius: 6px; padding: 14px 16px; margin: 16px 0; overflow-x: auto; white-space: pre; font-family: Consolas,Monaco,'Courier New',monospace; font-size: 14px; line-height: 1.55; color: #24292e;\"><code><span style=\"color: #008000; font-weight: bold;\">&lt;extension<\/span> <span style=\"color: #687822;\">name=<\/span><span style=\"color: #ba2121;\">\"ai_voice_agent\"<\/span><span style=\"color: #008000; font-weight: bold;\">&gt;<\/span>\n  <span style=\"color: #008000; font-weight: bold;\">&lt;condition<\/span> <span style=\"color: #687822;\">field=<\/span><span style=\"color: #ba2121;\">\"destination_number\"<\/span> <span style=\"color: #687822;\">expression=<\/span><span style=\"color: #ba2121;\">\"^1000$\"<\/span><span style=\"color: #008000; font-weight: bold;\">&gt;<\/span>\n    <span style=\"color: #008000; font-weight: bold;\">&lt;action<\/span> <span style=\"color: #687822;\">application=<\/span><span style=\"color: #ba2121;\">\"set\"<\/span> <span style=\"color: #687822;\">data=<\/span><span style=\"color: #ba2121;\">\"api_on_answer=uuid_audio_fork ${uuid} start wss:\/\/bridge.internal:8443\/call mono 16k\"<\/span><span style=\"color: #008000; font-weight: bold;\">\/&gt;<\/span>\n    <span style=\"color: #008000; font-weight: bold;\">&lt;action<\/span> <span style=\"color: #687822;\">application=<\/span><span style=\"color: #ba2121;\">\"answer\"<\/span><span style=\"color: #008000; font-weight: bold;\">\/&gt;<\/span>\n    <span style=\"color: #008000; font-weight: bold;\">&lt;action<\/span> <span style=\"color: #687822;\">application=<\/span><span style=\"color: #ba2121;\">\"park\"<\/span><span style=\"color: #008000; font-weight: bold;\">\/&gt;<\/span>\n  <span style=\"color: #008000; font-weight: bold;\">&lt;\/condition&gt;<\/span>\n<span style=\"color: #008000; font-weight: bold;\">&lt;\/extension&gt;<\/span><\/code><\/pre>\n<p><span style=\"font-weight: 400;\">The exact command syntax depends on the module you install, so check its documentation.<\/span><\/p>\n<h3><b>Step 4: Build the WebSocket Bridge Service<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">The bridge is an asynchronous service written in Python, Node.js, or Go. It accepts the WebSocket from FreeSWITCH and reads binary PCM frames. Then it runs two tasks in parallel: one sends caller audio to STT, while the other streams replies.<\/span><\/p>\n<pre style=\"background: #f6f8fa; border: 1px solid #e1e4e8; border-left: 4px solid #fb8500; border-radius: 6px; padding: 14px 16px; margin: 16px 0; overflow-x: auto; white-space: pre; font-family: Consolas,Monaco,'Courier New',monospace; font-size: 14px; line-height: 1.55; color: #24292e;\"><code><span style=\"color: #008000; font-weight: bold;\">import<\/span> <span style=\"color: #00f; font-weight: bold;\">asyncio<\/span><span style=\"color: #666;\">,<\/span> <span style=\"color: #00f; font-weight: bold;\">websockets<\/span>\n\n<span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">def<\/span> <span style=\"color: #00f;\">handle_call<\/span>(ws):\n    stt <span style=\"color: #666;\">=<\/span> <span style=\"color: #008000; font-weight: bold;\">await<\/span> open_stt_stream()  <span style=\"color: #3d7b7b; font-style: italic;\"># your STT client<\/span>\n\n    <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">def<\/span> <span style=\"color: #00f;\">from_caller<\/span>():\n        <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">for<\/span> frame <span style=\"color: #a2f; font-weight: bold;\">in<\/span> ws:\n            <span style=\"color: #008000; font-weight: bold;\">if<\/span> <span style=\"color: #008000;\">isinstance<\/span>(frame, <span style=\"color: #008000;\">bytes<\/span>):\n                <span style=\"color: #008000; font-weight: bold;\">await<\/span> stt<span style=\"color: #666;\">.<\/span>send_audio(frame)\n\n    <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">def<\/span> <span style=\"color: #00f;\">to_caller<\/span>():\n        <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">for<\/span> text <span style=\"color: #a2f; font-weight: bold;\">in<\/span> stt<span style=\"color: #666;\">.<\/span>final_transcripts():\n            <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">for<\/span> chunk <span style=\"color: #a2f; font-weight: bold;\">in<\/span> tts_stream(llm_stream(text)):\n                <span style=\"color: #008000; font-weight: bold;\">await<\/span> send_audio(ws, chunk)  <span style=\"color: #3d7b7b; font-style: italic;\"># module-specific format<\/span>\n\n    <span style=\"color: #008000; font-weight: bold;\">await<\/span> asyncio<span style=\"color: #666;\">.<\/span>gather(from_caller(), to_caller())\n\n<span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">def<\/span> <span style=\"color: #00f;\">main<\/span>():\n    <span style=\"color: #008000; font-weight: bold;\">async<\/span> <span style=\"color: #008000; font-weight: bold;\">with<\/span> websockets<span style=\"color: #666;\">.<\/span>serve(handle_call, <span style=\"color: #ba2121;\">\"0.0.0.0\"<\/span>, <span style=\"color: #666;\">8443<\/span>, ssl<span style=\"color: #666;\">=<\/span>ssl_ctx):\n        <span style=\"color: #008000; font-weight: bold;\">await<\/span> asyncio<span style=\"color: #666;\">.<\/span>Future()\n\nasyncio<span style=\"color: #666;\">.<\/span>run(main())<\/code><\/pre>\n<p><span style=\"font-weight: 400;\">The helper functions and <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">ssl_ctx<\/code> (your TLS certificate) are placeholders for your own code. However, one rule matters: nothing in the audio path may block. For example, a slow CRM lookup must never stop audio from flowing.<\/span><\/p>\n<h3><b>Step 5: Connect Streaming STT, LLM and TTS<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Send audio to STT as soon as it arrives, and use its endpointing to detect the end of a turn. Then pass the final transcript, call history, and caller context to the LLM. Expose narrow tools such as <a href=\"https:\/\/www.capanicus.com\/contact-center-crm-integration\">CRM lookup<\/a>, appointment booking, order status, and ticket creation. However, keep permissions, payments, and other sensitive logic in your backend, not in the model. Finally, stream the LLM&#8217;s first sentence to TTS immediately and send audio chunks back as they arrive.<\/span><\/p>\n<h3><b>Step 6: Add Barge-In and Human Transfer<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Keep listening while the agent speaks. When voice activity detection hears the caller, the bridge stops sending TTS audio. It also clears anything already queued. For FreeSWITCH playback, send <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">uuid_break &lt;uuid&gt; all<\/code> over ESL. For audio sent over the WebSocket, use the streaming module&#8217;s own clear command instead. Also tune VAD sensitivity so background noise or speakerphone echo does not trigger false interruptions.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For handoff, the bridge first stops the audio fork. Then it saves a short conversation summary as a channel variable with <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">uuid_setvar<\/code>. Finally, it sends <code style=\"background: #f3f4f6; color: #b3261e; padding: 2px 5px; border-radius: 4px; font-family: Consolas,Monaco,monospace; font-size: 0.92em;\">uuid_transfer<\/code> to move the call to a human queue.<\/span><\/p>\n<h3><b>Step 7: Test the Agent Under Real Call Load<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Test with real phone lines, not just WebRTC, because PSTN audio is narrowband and noisier. Also use SIPp to generate concurrent calls and log STT, LLM, and TTS latency for every turn. Then fix the slowest stage first.<\/span><\/p>\n<h2><b>How to Keep Voice Agent Latency Under 800 ms<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Callers judge a voice agent by how fast it responds. Therefore, aim for the first audible word within 500 to 800 milliseconds after the caller stops speaking. That budget covers network travel, STT finalization, LLM time to first token, TTS time to first audio, and playback.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Stream every stage.<\/b><span style=\"font-weight: 400;\"> Never wait for a full transcript, full reply, or full audio file.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Tune endpointing.<\/b><span style=\"font-weight: 400;\"> The pause STT waits for before closing a turn is often the largest single delay. So keep it as short as accuracy allows.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Use 16-bit linear PCM in mono.<\/b><span style=\"font-weight: 400;\"> Use 8 kHz for PSTN calls and 16 kHz for wideband or WebRTC calls.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Send small frames.<\/b><span style=\"font-weight: 400;\"> Around 20 ms per frame suits interactive voice.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Keep the bridge close.<\/b><span style=\"font-weight: 400;\"> Run it on the same private network as FreeSWITCH.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Pre-synthesize static prompts<\/b><span style=\"font-weight: 400;\"> such as greetings and hold messages.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Track per-turn telemetry<\/b><span style=\"font-weight: 400;\"> for each stage so you know where the time goes.<\/span><\/li>\n<\/ul>\n<h2><b>How to Scale FreeSWITCH AI Voice Agents<\/b><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2625\" src=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Scale-FreeSWITCH-AI-Voice-Agents.webp\" alt=\"Scale FreeSWITCH AI Voice Agents\" width=\"1000\" height=\"562\" srcset=\"https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Scale-FreeSWITCH-AI-Voice-Agents.webp 1000w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Scale-FreeSWITCH-AI-Voice-Agents-300x169.webp 300w, https:\/\/www.capanicus.com\/blog\/wp-content\/uploads\/2026\/11\/Scale-FreeSWITCH-AI-Voice-Agents-768x432.webp 768w\" sizes=\"auto, (max-width: 1000px) 100vw, 1000px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Scalability depends on the whole pipeline, not just the PBX. Therefore, scale each layer independently.<\/span><\/p>\n<h3><b>Scaling FreeSWITCH Nodes<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Run several FreeSWITCH nodes behind a <a href=\"https:\/\/www.capanicus.com\/blog\/voip-scalability-challenges\/\">SIP load balancer<\/a> such as Kamailio or OpenSIPS. Store call metadata in Redis or a shared database, and manage configuration centrally. Also remove unhealthy nodes automatically. Finally, match SIP trunk capacity to your concurrent-call target.<\/span><\/p>\n<h3><b>Scaling the Bridge Layer<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Run bridge instances behind a WebSocket-aware load balancer and cap sessions per instance based on CPU, memory, and bandwidth. Use asynchronous I\/O and connection pools for every provider. In addition, add backpressure so an overloaded instance rejects new calls cleanly instead of degrading live ones.<\/span><\/p>\n<h3><b>Scaling STT, LLM and TTS Independently<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">STT and TTS are media-heavy, while LLM load depends on tokens and context. Give each its own rate limits and autoscaling. Also use regional endpoints, provider failover, and model routing. For example, send simple intents to a fast model and complex requests to a stronger one. Finally, move summaries and CRM updates to background queues.<\/span><\/p>\n<h2><b>Securing Your AI Voice Agent: Media, Data and Compliance<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Voice agents handle personal data, so security belongs in the first design, not a later sprint.<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Use WSS, not plain WebSocket, for audio streams. Also accept bridge connections only from FreeSWITCH nodes, using tokens or IP allowlists.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Keep ESL on localhost or a private network. ESL has no native encryption, so tunnel it through a VPN or stunnel if it must cross hosts.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Store API keys in a secrets manager and rotate them regularly.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Take card details through a PCI-compliant DTMF flow, never through the AI, and redact sensitive data from transcripts and logs.<\/span><\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><span style=\"font-weight: 400;\">Define retention periods, obtain consent where recording or AI processing requires it, and log every tool call for audits.<\/span><\/li>\n<\/ul>\n<h2><b>Human Handoff and Failure Handling<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">An AI agent should never be a dead end. Instead, transfer the call to a human when the caller asks for a person or fails verification repeatedly. Also transfer when the caller raises a high-risk or out-of-scope request, sounds frustrated, or when tools keep failing.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Plan for outages too. If STT, the LLM, or TTS stops responding, play a fallback prompt. Then route the caller to a live queue or a <a href=\"https:\/\/www.capanicus.com\/blog\/ivr-services-solutions-development-the-phone-system-for-modern-business\/\">conventional IVR<\/a>. The bridge should detect a dropped WebSocket and transfer the call rather than leave the caller in silence.<\/span><\/p>\n<h2><b>Conclusion<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">In short, a scalable real-time AI voice agent with FreeSWITCH comes down to four decisions. Let FreeSWITCH own the call, and stream audio through a non-blocking bridge. Then stream every AI stage to stay under 800 ms, and scale each layer independently. Finally, add barge-in, clear human handoff, and strong security. As a result, you have an agent callers actually want to talk to.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">If you are planning an AI voice agent or a <a href=\"https:\/\/www.capanicus.com\/asterisk-freeswitch-development\">FreeSWITCH deployment<\/a>, our team can help you design, build, and scale it for production traffic. <a href=\"https:\/\/www.capanicus.com\/contact-us\">Talk to Capanicus<\/a> about your project.<\/span><\/p>\n<h2><b>Frequently Asked Questions<\/b><\/h2>\n<h3><b>Can FreeSWITCH run an AI voice agent on its own?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">No. FreeSWITCH handles calls and media, not speech recognition or reasoning. It streams call audio to a WebSocket bridge, which connects to STT, an LLM, and TTS.<\/span><\/p>\n<h3><b>What audio format should FreeSWITCH use for AI voice agents?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Use 16-bit linear PCM in mono: 8 kHz for PSTN calls and 16 kHz for wideband or WebRTC calls. Most streaming STT and TTS services accept it without extra conversion.<\/span><\/p>\n<h3><b>How does barge-in work in a FreeSWITCH voice agent?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">The bridge keeps listening while the agent speaks. When it detects caller speech, it stops playback and processes the new input. DTMF can trigger an interruption too.<\/span><\/p>\n<h3><b>How many concurrent calls can a FreeSWITCH AI voice agent handle?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">There is no fixed number. Capacity depends on hardware, codecs, bridge efficiency, and AI provider limits. Because each layer scales horizontally, you can add capacity where load-testing shows a bottleneck.<\/span><\/p>\n<h3><b>When should the AI transfer a call to a human agent?<\/b><\/h3>\n<p><span style=\"font-weight: 400;\">Transfer when the caller asks for a person, the request is high-risk or out of scope, verification fails, or the system fails. An obvious exit protects customer satisfaction.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Callers no longer want to press 1 for billing and 2 for support. Instead, they want to say what they need and get a useful answer. A real-time AI voice agent with FreeSWITCH makes that possible on the phone lines you already run. The key idea is a clean split of responsibilities. FreeSWITCH manages the<\/p>\n","protected":false},"author":1,"featured_media":2621,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[172,169,170,171],"class_list":["post-2619","post","type-post","status-publish","format-standard","has-post-thumbnail","category-uncategorized","tag-ai-ivr-for-freeswitch","tag-ai-voice-agent-with-freeswitch","tag-freeswitch-ai-integration","tag-freeswitch-ai-voice-bot"],"_links":{"self":[{"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/posts\/2619","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/comments?post=2619"}],"version-history":[{"count":2,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/posts\/2619\/revisions"}],"predecessor-version":[{"id":2626,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/posts\/2619\/revisions\/2626"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/media\/2621"}],"wp:attachment":[{"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/media?parent=2619"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/categories?post=2619"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.capanicus.com\/blog\/wp-json\/wp\/v2\/tags?post=2619"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}