Connecting AI voice agents
This guide walks you through answering calls with an AI voice agent: Signals streams the caller's audio to your server, your server relays it to the OpenAI Realtime API and plays the agent's replies, and the agent hands the call to a person when the caller needs one.
How an AI voice agent call works
- A caller dials a number routed to your voice application, and Signals requests the application's voice URL. See Receiving calls.
- Your server returns a document with a
<Connect><Stream>that names your WebSocket server. - Signals opens the WebSocket and sends
connected,startand then the caller's audio inmediamessages. - Your server forwards the caller's audio to OpenAI and sends the agent's audio back in
mediamessages, which Signals plays to the caller. - When your server closes the WebSocket, Signals runs the next verb in the document, such as a
<Dial>to a person.
The audio is G.711 µ-law at 8,000 Hz in both directions. The OpenAI Realtime API accepts that format as audio/pcmu, so your server passes the audio through without converting it. The Stream reference lists every message.
Return the stream from your voice application
Return this document from your voice application's voice URL. <Connect> holds the call on the stream until your server closes it. The <Say> and <Dial> run only after that, so the caller hears a short message and then reaches a person at +2348051234567:
Code
The call also moves on to the <Dial> when the connection drops, so the caller still reaches a person if your server stops.
Bridge the stream to OpenAI
Your WebSocket server accepts the connection from Signals, opens a second WebSocket to the OpenAI Realtime API with your OpenAI API key, and passes audio between the two. The server below listens on port 8080. Signals connects only to wss:// URLs, so serve it over TLS at wss://voice.example.com/agent, for example behind a reverse proxy.
Install the WebSocket library first: npm install ws for Node.js or pip install websockets for Python. These examples use ws 8.23 and websockets 17.4, and read your key from OPENAI_API_KEY.
The session asks OpenAI for audio/pcmu in both directions and for server_vad, which detects when the caller speaks. The response.create sent after it makes the agent greet the caller first. From then on, the server does three things:
- On
start, it keeps thestreamSid, because every message it sends to Signals carries it. - On each
mediamessage from Signals, it adds the audio to OpenAI's input withinput_audio_buffer.append. - On each
response.output_audio.deltafrom OpenAI, it sends the audio to Signals in amediamessage.
Keep your OpenAI API key on your server. Never put it in the OneML document or in the stream URL.
Stop the agent when the caller speaks
When the caller talks over the agent, OpenAI stops its reply and sends input_audio_buffer.speech_started. Audio the agent already sent is still queued at Signals, so the server sends clear, and Signals discards that audio at once:
Code
To tell OpenAI how much of its reply the caller heard, send a mark after each chunk of audio. Signals returns each mark as its audio finishes playing, and you can pass that point to OpenAI with conversation.item.truncate.
Hand the call to a person
The session gives the agent a transfer_to_agent function. When the agent calls it, OpenAI sends response.function_call_arguments.done with name set to transfer_to_agent, and the server closes the WebSocket. Signals ends the stream and runs the next verbs: the <Say>, then the <Dial>. The caller stays on the call while the person's phone rings, and the person sees +2342013300100 as the caller ID. Forwarding calls covers timeouts, outcomes and ringing several people.
To send the call to a place the document does not name, redirect the call with the Calls API instead. Signals stops the stream and requests the new URL. Take the call's SID from callSid in the start message:
Code
Next steps
- Stream: every message Signals and your endpoint exchange.
- Forwarding calls: caller IDs, outcomes and ringing several numbers.
- Calls API reference: updating and ending a call in progress.