Signals Docs
Guides

Connecting AI voice agents

This guide walks you through answering calls with an AI voice agent: Signals streams the caller's audio to your server, your server relays it to the OpenAI Realtime API and plays the agent's replies, and the agent hands the call to a person when the caller needs one.

How an AI voice agent call works

  1. A caller dials a number routed to your voice application, and Signals requests the application's voice URL. See Receiving calls.
  2. Your server returns a document with a <Connect><Stream> that names your WebSocket server.
  3. Signals opens the WebSocket and sends connected, start and then the caller's audio in media messages.
  4. Your server forwards the caller's audio to OpenAI and sends the agent's audio back in media messages, which Signals plays to the caller.
  5. When your server closes the WebSocket, Signals runs the next verb in the document, such as a <Dial> to a person.

The audio is G.711 µ-law at 8,000 Hz in both directions. The OpenAI Realtime API accepts that format as audio/pcmu, so your server passes the audio through without converting it. The Stream reference lists every message.

Return the stream from your voice application

Return this document from your voice application's voice URL. <Connect> holds the call on the stream until your server closes it. The <Say> and <Dial> run only after that, so the caller hears a short message and then reaches a person at +2348051234567:

XMLCode
<?xml version="1.0" encoding="UTF-8"?> <Response> <Connect> <Stream url="wss://voice.example.com/agent"/> </Connect> <Say>Connecting you to an agent.</Say> <Dial callerId="+2342013300100">+2348051234567</Dial> </Response>

The call also moves on to the <Dial> when the connection drops, so the caller still reaches a person if your server stops.

Bridge the stream to OpenAI

Your WebSocket server accepts the connection from Signals, opens a second WebSocket to the OpenAI Realtime API with your OpenAI API key, and passes audio between the two. The server below listens on port 8080. Signals connects only to wss:// URLs, so serve it over TLS at wss://voice.example.com/agent, for example behind a reverse proxy.

Install the WebSocket library first: npm install ws for Node.js or pip install websockets for Python. These examples use ws 8.23 and websockets 17.4, and read your key from OPENAI_API_KEY.

import WebSocket, { WebSocketServer } from "ws"; const OPENAI_URL = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"; const session = { type: "realtime", instructions: "You answer support calls for Acme Delivery. Keep each answer short. " + "When the caller asks for a person, or you cannot help, call transfer_to_agent.", audio: { input: { format: { type: "audio/pcmu" }, turn_detection: { type: "server_vad" } }, output: { format: { type: "audio/pcmu" }, voice: "marin" }, }, tools: [ { type: "function", name: "transfer_to_agent", description: "Hand the caller to a human support agent.", parameters: { type: "object", properties: {} }, }, ], }; const server = new WebSocketServer({ port: 8080 }); server.on("connection", (signals) => { let streamSid = null; const openai = new WebSocket(OPENAI_URL, { headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }, }); openai.on("open", () => { openai.send(JSON.stringify({ type: "session.update", session })); openai.send(JSON.stringify({ type: "response.create" })); }); signals.on("message", (data) => { const message = JSON.parse(data); if (message.event === "start") { streamSid = message.start.streamSid; } else if (message.event === "media" && openai.readyState === WebSocket.OPEN) { openai.send(JSON.stringify({ type: "input_audio_buffer.append", audio: message.media.payload })); } }); openai.on("message", (data) => { const event = JSON.parse(data); if (!streamSid) return; if (event.type === "response.output_audio.delta") { signals.send(JSON.stringify({ event: "media", streamSid, media: { payload: event.delta } })); } else if (event.type === "input_audio_buffer.speech_started") { signals.send(JSON.stringify({ event: "clear", streamSid })); } else if (event.type === "response.function_call_arguments.done" && event.name === "transfer_to_agent") { signals.close(); } }); signals.on("close", () => openai.close()); openai.on("close", () => signals.close()); signals.on("error", () => openai.close()); openai.on("error", () => signals.close()); });

The session asks OpenAI for audio/pcmu in both directions and for server_vad, which detects when the caller speaks. The response.create sent after it makes the agent greet the caller first. From then on, the server does three things:

  1. On start, it keeps the streamSid, because every message it sends to Signals carries it.
  2. On each media message from Signals, it adds the audio to OpenAI's input with input_audio_buffer.append.
  3. On each response.output_audio.delta from OpenAI, it sends the audio to Signals in a media message.

Keep your OpenAI API key on your server. Never put it in the OneML document or in the stream URL.

Stop the agent when the caller speaks

When the caller talks over the agent, OpenAI stops its reply and sends input_audio_buffer.speech_started. Audio the agent already sent is still queued at Signals, so the server sends clear, and Signals discards that audio at once:

JSONCode
{"event": "clear", "streamSid": "ST3f6c1a2b9d8e4f7a8b9c0d1e2f3a4b5c"}

To tell OpenAI how much of its reply the caller heard, send a mark after each chunk of audio. Signals returns each mark as its audio finishes playing, and you can pass that point to OpenAI with conversation.item.truncate.

Hand the call to a person

The session gives the agent a transfer_to_agent function. When the agent calls it, OpenAI sends response.function_call_arguments.done with name set to transfer_to_agent, and the server closes the WebSocket. Signals ends the stream and runs the next verbs: the <Say>, then the <Dial>. The caller stays on the call while the person's phone rings, and the person sees +2342013300100 as the caller ID. Forwarding calls covers timeouts, outcomes and ringing several people.

To send the call to a place the document does not name, redirect the call with the Calls API instead. Signals stops the stream and requests the new URL. Take the call's SID from callSid in the start message:

TerminalCode
curl -X POST "https://api.signals.pressone.africa/2010-04-01/Accounts/$SIGNALS_ACCOUNT_SID/Calls/$CALL_SID.json" \ -u "$SIGNALS_ACCOUNT_SID:$SIGNALS_AUTH_TOKEN" \ --data-urlencode "Url=https://voice.example.com/handoff"

Next steps

Last modified on