# Connecting AI voice agents

This guide walks you through answering calls with an AI voice agent: Signals streams the caller's audio to your server, your server relays it to the OpenAI Realtime API and plays the agent's replies, and the agent hands the call to a person when the caller needs one.

## How an AI voice agent call works

1. A caller dials a number routed to your voice application, and Signals requests the application's voice URL. See [Receiving calls](/programmable-voice/receiving-calls).
2. Your server returns a document with a `<Connect><Stream>` that names your WebSocket server.
3. Signals opens the WebSocket and sends `connected`, `start` and then the caller's audio in `media` messages.
4. Your server forwards the caller's audio to OpenAI and sends the agent's audio back in `media` messages, which Signals plays to the caller.
5. When your server closes the WebSocket, Signals runs the next verb in the document, such as a `<Dial>` to a person.

The audio is G.711 µ-law at 8,000 Hz in both directions. The OpenAI Realtime API accepts that format as `audio/pcmu`, so your server passes the audio through without converting it. The [Stream](/programmable-voice/oneml/stream#websocket-messages) reference lists every message.

## Return the stream from your voice application

Return this document from your voice application's voice URL. `<Connect>` holds the call on the stream until your server closes it. The `<Say>` and `<Dial>` run only after that, so the caller hears a short message and then reaches a person at `+2348051234567`:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://voice.example.com/agent"/>
  </Connect>
  <Say>Connecting you to an agent.</Say>
  <Dial callerId="+2342013300100">+2348051234567</Dial>
</Response>
```

The call also moves on to the `<Dial>` when the connection drops, so the caller still reaches a person if your server stops.

## Bridge the stream to OpenAI

Your WebSocket server accepts the connection from Signals, opens a second WebSocket to the OpenAI Realtime API with your OpenAI API key, and passes audio between the two. The server below listens on port 8080. Signals connects only to `wss://` URLs, so serve it over TLS at `wss://voice.example.com/agent`, for example behind a reverse proxy.

Install the WebSocket library first: `npm install ws` for Node.js or `pip install websockets` for Python. These examples use `ws` 8.23 and `websockets` 17.4, and read your key from `OPENAI_API_KEY`.

<CodeTabs syncKey="lang">

```javascript title="Node.js"
import WebSocket, { WebSocketServer } from "ws";

const OPENAI_URL = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";

const session = {
  type: "realtime",
  instructions:
    "You answer support calls for Acme Delivery. Keep each answer short. " +
    "When the caller asks for a person, or you cannot help, call transfer_to_agent.",
  audio: {
    input: { format: { type: "audio/pcmu" }, turn_detection: { type: "server_vad" } },
    output: { format: { type: "audio/pcmu" }, voice: "marin" },
  },
  tools: [
    {
      type: "function",
      name: "transfer_to_agent",
      description: "Hand the caller to a human support agent.",
      parameters: { type: "object", properties: {} },
    },
  ],
};

const server = new WebSocketServer({ port: 8080 });

server.on("connection", (signals) => {
  let streamSid = null;
  const openai = new WebSocket(OPENAI_URL, {
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
  });

  openai.on("open", () => {
    openai.send(JSON.stringify({ type: "session.update", session }));
    openai.send(JSON.stringify({ type: "response.create" }));
  });

  signals.on("message", (data) => {
    const message = JSON.parse(data);
    if (message.event === "start") {
      streamSid = message.start.streamSid;
    } else if (message.event === "media" && openai.readyState === WebSocket.OPEN) {
      openai.send(JSON.stringify({ type: "input_audio_buffer.append", audio: message.media.payload }));
    }
  });

  openai.on("message", (data) => {
    const event = JSON.parse(data);
    if (!streamSid) return;
    if (event.type === "response.output_audio.delta") {
      signals.send(JSON.stringify({ event: "media", streamSid, media: { payload: event.delta } }));
    } else if (event.type === "input_audio_buffer.speech_started") {
      signals.send(JSON.stringify({ event: "clear", streamSid }));
    } else if (event.type === "response.function_call_arguments.done" && event.name === "transfer_to_agent") {
      signals.close();
    }
  });

  signals.on("close", () => openai.close());
  openai.on("close", () => signals.close());
  signals.on("error", () => openai.close());
  openai.on("error", () => signals.close());
});
```

```python title="Python"
import asyncio
import json
import os

from websockets.asyncio.client import connect
from websockets.asyncio.server import serve

OPENAI_URL = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"

SESSION = {
    "type": "realtime",
    "instructions": (
        "You answer support calls for Acme Delivery. Keep each answer short. "
        "When the caller asks for a person, or you cannot help, call transfer_to_agent."
    ),
    "audio": {
        "input": {"format": {"type": "audio/pcmu"}, "turn_detection": {"type": "server_vad"}},
        "output": {"format": {"type": "audio/pcmu"}, "voice": "marin"},
    },
    "tools": [
        {
            "type": "function",
            "name": "transfer_to_agent",
            "description": "Hand the caller to a human support agent.",
            "parameters": {"type": "object", "properties": {}},
        }
    ],
}


async def handle_call(signals):
    headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}
    async with connect(OPENAI_URL, additional_headers=headers) as openai:
        await openai.send(json.dumps({"type": "session.update", "session": SESSION}))
        await openai.send(json.dumps({"type": "response.create"}))
        stream_sid = None

        async def from_signals():
            nonlocal stream_sid
            async for raw in signals:
                message = json.loads(raw)
                if message["event"] == "start":
                    stream_sid = message["start"]["streamSid"]
                elif message["event"] == "media":
                    audio = message["media"]["payload"]
                    await openai.send(json.dumps({"type": "input_audio_buffer.append", "audio": audio}))

        async def from_openai():
            async for raw in openai:
                event = json.loads(raw)
                if stream_sid is None:
                    continue
                if event["type"] == "response.output_audio.delta":
                    media = {"payload": event["delta"]}
                    await signals.send(json.dumps({"event": "media", "streamSid": stream_sid, "media": media}))
                elif event["type"] == "input_audio_buffer.speech_started":
                    await signals.send(json.dumps({"event": "clear", "streamSid": stream_sid}))
                elif event["type"] == "response.function_call_arguments.done" and event["name"] == "transfer_to_agent":
                    return

        tasks = [asyncio.create_task(from_signals()), asyncio.create_task(from_openai())]
        await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
        for task in tasks:
            task.cancel()
    await signals.close()


async def main():
    async with serve(handle_call, "0.0.0.0", 8080) as server:
        await server.serve_forever()


asyncio.run(main())
```

</CodeTabs>

The session asks OpenAI for `audio/pcmu` in both directions and for `server_vad`, which detects when the caller speaks. The `response.create` sent after it makes the agent greet the caller first. From then on, the server does three things:

1. On `start`, it keeps the `streamSid`, because every message it sends to Signals carries it.
2. On each `media` message from Signals, it adds the audio to OpenAI's input with `input_audio_buffer.append`.
3. On each `response.output_audio.delta` from OpenAI, it sends the audio to Signals in a `media` message.

:::caution
Keep your OpenAI API key on your server. Never put it in the OneML document or in the stream URL.
:::

## Stop the agent when the caller speaks

When the caller talks over the agent, OpenAI stops its reply and sends `input_audio_buffer.speech_started`. Audio the agent already sent is still queued at Signals, so the server sends `clear`, and Signals discards that audio at once:

```json
{"event": "clear", "streamSid": "ST3f6c1a2b9d8e4f7a8b9c0d1e2f3a4b5c"}
```

To tell OpenAI how much of its reply the caller heard, send a `mark` after each chunk of audio. Signals returns each mark as its audio finishes playing, and you can pass that point to OpenAI with `conversation.item.truncate`.

## Hand the call to a person

The session gives the agent a `transfer_to_agent` function. When the agent calls it, OpenAI sends `response.function_call_arguments.done` with `name` set to `transfer_to_agent`, and the server closes the WebSocket. Signals ends the stream and runs the next verbs: the `<Say>`, then the `<Dial>`. The caller stays on the call while the person's phone rings, and the person sees `+2342013300100` as the caller ID. [Forwarding calls](/programmable-voice/forwarding-calls) covers timeouts, outcomes and ringing several people.

To send the call to a place the document does not name, redirect the call with the Calls API instead. Signals stops the stream and requests the new URL. Take the call's SID from `callSid` in the `start` message:

```bash
curl -X POST "https://api.signals.pressone.africa/2010-04-01/Accounts/$SIGNALS_ACCOUNT_SID/Calls/$CALL_SID.json" \
  -u "$SIGNALS_ACCOUNT_SID:$SIGNALS_AUTH_TOKEN" \
  --data-urlencode "Url=https://voice.example.com/handoff"
```

## Next steps

- [Stream](/programmable-voice/oneml/stream): every message Signals and your endpoint exchange.
- [Forwarding calls](/programmable-voice/forwarding-calls): caller IDs, outcomes and ringing several numbers.
- [Calls API reference](/api/voice-calls): updating and ending a call in progress.
