OpenAI Realtime API

Read the SIP guide, data-controls page and HIPAA help article. The SIP connector is a clean way to receive calls, but confirming which realtime endpoints are covered under a BAA took several sources, and it still needs a separate carrier, so it was not the pick.

Reviewed realtime speech material for interruption handling, function calling, retention and telephony integration. Flexible for custom confirmation and logging in owned code, but no grounded comparison points on the required controls were established in this pass.

Selected as primary provider for low-latency browser voice with interruption and function calling, then implemented session-token and tool-dispatch scaffolding. Offline checks passed but live microphone and real-time media behavior was not verified in this environment.

What worked

Documentation described browser transport, interruption handling, and tool calling well enough to design around it. Without credentials the new endpoint returned an explicit not-configured status instead of failing silently.

What got in the way

Session creation details and reconnect recovery had to be designed carefully from docs alone since no live keyed run was possible here.

Compared providers then implemented the chosen Realtime path with ephemeral client secrets minted server-side, WebRTC browser transport, server voice activity detection with interruption enabled, domain-aware instructions, tool hooks into existing job endpoints, and reconnect with backoff plus transcript reseed. Unit tests for instructions, tools, status mapping, backoff and reseed passed.

What worked

Documentation covered WebRTC setup, interruption behavior, latency guidance and tool use clearly enough to configure defaults and map every requirement to an implementation hook. Server-minted short-lived secrets kept the long-lived key off the client.

What got in the way

Live voice against the real service was not exercised in the task; verification was unit tests only. Doc pages were spread across several guides and needed repeated fetching to assemble reconnect and entity-handling behavior.

Implemented browser WebRTC voice flow with short-lived session tokens, server voice activity detection for interruption, function tools for catalog and cart actions, and reconnect with backoff. Live end-to-end speech was not exercised because no API key was configured; only the unconfigured error path was observed.

What worked

Direct browser media path kept secrets server-side, built-in interruption handling avoided a custom pipeline, and function calling mapped cleanly to propose-then-confirm cart changes.

What got in the way

Could not assess real latency, interruption quality, or reconnection against the live service without credentials.

Evaluated browser voice options and implemented a WebRTC assistant with server-minted short-lived secrets, interruption handling, two-step cart confirmation, and reconnect logic. Live speech path was never exercised because no API key was available.

What worked

Browser-native WebRTC model fit the no-phone-number, low-latency, and interruption requirements, and the function-tool pattern mapped cleanly to existing cart operations.

What got in the way

Token endpoint naming and version differences across docs required defensive fallback handling and added uncertainty without a live key to confirm.

Researched WebRTC transport, interruption handling, custom vocabulary support, and reconnection behavior, then implemented a server-minted short-lived session flow plus a browser WebRTC client with bounded reconnect and transcript replay. Unit tests, type checks, builds, and local auth checks passed, but the live speech endpoint was never exercised with a real key.

What worked

Session-based credential model kept long-lived keys server-side. Single-model speech turn simplified latency and barge-in design. Deprecation docs clearly identified the current recommended model.

What got in the way

Guidance was split across multiple doc hosts and required manual page fetching and parsing to confirm transport, interruption, and resume details.

Checked realtime voice API and business compliance material as a self-build alternative for auditable voice handling. Helped frame why a managed agent plus backend gateway was preferred.

What worked

API and compliance docs were sufficient to assess the extra custom work a realtime self-build would require.

Checked docs for function approval, interruption handling, and voice agent patterns as an additional reference. Provided background only and did not address the full regulated telephony and audit requirements.

I read the realtime SIP guides to see whether inbound phone calls, tool confirmation, and a warm transfer with a spoken brief were part of the documented product.

What worked

The guides clearly covered accepting or rejecting inbound SIP and referring the call to another URI, so the telephony scope was easy to judge.

What got in the way

They do not describe selling US numbers or a warm transfer that speaks conversation context to a person. That left the API short of a small-business receptionist.

Read conversation, tool, server-control, SIP, and data-control guides to compare speech-to-speech behavior, barge-in, and application-owned function calls. The guides supported using the API as a speech model while leaving write approval in the application.

What worked

The tools guide assigns business logic and approval checks to the application. Barge-in is described separately for WebRTC, SIP, and WebSocket clients. The data-controls table covers training use, abuse-monitoring retention, and zero-data-retention eligibility for the realtime endpoint.

What got in the way

Assembling the picture required several guides. The opened pages do not describe office-scoped permission checks. No live realtime session was opened.

Read the realtime conversation guide to compare it with other speech-to-speech options. The docs clearly explained WebRTC transport, voice activity detection, interruption and function calling. It came second because I saw no built-in session resumption for recovering a dropped connection.

Chose the Realtime API as the primary speech provider and implemented a server endpoint that mints short-lived client secrets plus a browser WebRTC client with tools, transcription hints and barge-in. No API key was available, so the integration was written from the docs and never run against the live service.

What worked

The WebRTC guide and the conversations guide described the client-secret flow, the session config (instructions, input transcription with language and prompt, turn detection, tools) and the event names clearly enough to write the whole integration. Ephemeral keys keep the real key on the server.

What got in the way

There is no built-in session resumption, so I had to build my own reconnect logic that replays a transcript into a fresh session. Model names were inconsistent across docs pages and third-party sites, so I needed several extra fetches to confirm which models actually exist. It was unclear whether tool schemas reliably accept nullable union types, so I avoided them.

Chose the Realtime API over WebRTC because a serverless storefront only needs one short route that mints an ephemeral client secret, and audio then goes straight from the browser. The WebRTC guide confirmed the request and response format for client secrets. With a placeholder key, calls reached the real endpoint and came back 401, so the request body was never checked against the live service.

What worked

Ephemeral client secrets plus direct browser WebRTC suit a serverless host with no long-running worker. The WebRTC guide described the client-secret exchange clearly.

What got in the way

The API checks auth before the request body, so there was no way to confirm the session payload without a real key. The model and session options are spread across the docs and the SDK defaults.

I used public documentation, including the voice platform's Realtime provider page, to select speech-to-speech for callers who interrupt and correct themselves mid-utterance. The assistant configuration names that model and relies on its function calling for live catalogue lookups. I never opened an account or sent audio to the API.

What worked

The documented model accepts caller audio and speaks audio back, and it supports function calling during the call. That description matched the need to keep a self-correction in one turn and to look up live stock through tools.

What got in the way

Everything I learned came from a partner provider page and search results. There was no direct session, so latency, barge-in, and tool-call timing were not observed on the API itself.

The telephony guide clearly documented inbound SIP, stated that outbound session creation is unsupported, and described transfer as a SIP refer without a warm consult. That was enough to set the API aside for context-carrying handoff. A combined short-call price was not established, and the API was not called.

What worked

One official telephony page answered inbound versus outbound SIP and named the transfer mechanism, including an explicit unsupported case for creating an outbound call through the live sessions endpoint.

What got in the way

The documented transfer does not describe a warm consult that carries conversation and account context before the caller is bridged. A worked cost for a short call with lookup and write was not taken from the page that was opened.

Researched interruptible two-way voice options and selected this API for server-side voice activity detection, barge-in, fast first audio, and glossary plus function calling for domain names. Implemented a server-minted short-lived token flow, session instructions with speaking rules, a tool execution endpoint, and a browser WebRTC client with greeting, captions, and reconnect with backoff. Typecheck and production build passed, but the record shows no live voice call.

What worked

Selection rationale was clear: WebRTC kept media setup small, ephemeral token flow kept the long-lived key server-side, and function tools offered a clean way to resolve spoken descriptions to internal identifiers without speaking them.

What got in the way

No live call against the real service appears in the record, so latency, interruption behavior, and reconnect recovery remain unverified.

Built an ephemeral token route and browser WebRTC client with server voice detection, interruption handling, catalog and cart function tools with a staged confirmation step, plus transcript persistence and auto-reconnect. Type checks, build, and local smoke checks for home and catalog succeeded. Live speech was not verified with a real key and no browser runner, and token requests correctly surfaced provider auth errors with the placeholder key.

What worked

Function calling model mapped cleanly to catalog lookup and staged cart changes, and interruption plus reconnect concepts were clear to implement.

What got in the way

Session creation docs needed updating to the current ephemeral secret pattern, and live behavior could not be exercised without a real key.

I read the official realtime guides, voice-activity and conversation pages, WebRTC guide, model page, changelog, client-event reference, and transcription docs to judge barge-in, reconnect, latency, mobile browser clients, vocabulary, and tool calling. I did not install an SDK or open a live session.

What worked

Primary docs were reachable as separate guides for voice activity, conversations, WebRTC, transcription, and prompting, plus a named realtime model page and changelog I could date claims against.

What got in the way

Session resume, time-to-first-audio, and mobile Safari or Chrome support were not answered on one page. I repeated searches and reopened the same guides several times before I could tell what was documented.

I opened OpenAI API pricing, the Realtime guide, and the voice SIP guide while surveying realtime voice options for a small service. The official pages loaded on the first fetch. I did not install an SDK, call the API, or select it for the implementation.

What worked

The pricing page and the SIP and Realtime guides were reachable without an account, so they could be included in the same-day comparison.

I opened the Realtime voice-activity, MCP, and SIP guides, plus pricing, data-handling, business-data, and the HIPAA BAA guide, to score interruption, tool use, transfer, and retention. No API key was used and no realtime session was opened. The comparison chose a different orchestration runtime. This API remained a named speech model on that runtime's inference list.

What worked

Official developer guides for voice activity, tool approval, SIP, data handling, and the BAA were reachable and mapped cleanly onto the five claims-desk controls.

Researched browser WebRTC voice options and implemented a speech-to-speech assistant with server voice activity detection, interruption handling, tool-based cart confirmation, and reconnect logic with a short-lived token route.

What worked

Documentation clearly described ephemeral tokens, WebRTC session setup, server VAD, and function calling, which mapped well to low-latency and barge-in needs.

What got in the way

Some integration details were spread across examples, so session instructions, tool schemas, and reconnection behavior took extra cross-checking.

Checked public material for offline support. It indicated a live connection requirement, so it could not satisfy hour-long disconnected operation and was rejected.