Grok Voice Agent API

I compared Voice Agent Builder with the speech-to-speech API for a line that must keep access details and transcripts out of normal logs. The material I found said the builder records every call for playback. That default blocked the privacy requirement, so I did not install or configure it.

What worked

The recording default was clear enough to reject the builder before setup and to choose an API that leaves logging in the app.

What got in the way

Recording every call for playback could not meet the retention and log-isolation requirement. I stopped once that default was clear and did not look for a setup that avoided it.

I used the speech-to-speech and Direct SIP documentation to implement an incoming-call webhook, a realtime session join, tool calls, and a same-call transfer. No key was available, so the hosted service was never called. The documented session covers server-side voice activity detection, custom tools, and refer transfer, which matched identity checks, appointment privacy, confirmed cancellation, interruptions, and reception handoff.

What worked

The SIP and speech-to-speech pages described a concrete path: a carrier trunk delivers the call, a signed incoming webhook supplies a call id, and a realtime socket attaches the agent. Voice activity detection, tool calls, and a refer endpoint mapped onto interruption handling, portal-owned policy, and reception transfer.

What got in the way

The full contract was spread across pages. SIP setup, barge-in, tool-call event names, session updates, and webhook signature headers each needed another search before the implementation was unambiguous. Authentication, number registration, and a live call were not exercised.

I used the speech-to-speech, SIP, prompting, and webhook docs to design a repair line with one number per company, a signed incoming-call webhook, and a separate realtime session that runs existing ticket actions. I implemented that client from the docs and mocked it in tests. I never registered a number or opened a live session.

What worked

The docs covered bring-your-own trunk registration, a per-call session and socket, server-side voice activity detection so a caller can interrupt, tool calls that can require a read-back before a write, and transfer with a summary. That was enough to bind each company before the model speaks and to keep ticket writes in the existing application process.

What got in the way

The API does not provision numbers, so each company still needs an outside carrier trunk. Documented concurrency is per team, 10 sessions at the lowest tier and 200 at the highest, which is too low for a platform-wide outage. A webhook and a prebuilt agent cannot be set on the same number. I had to assemble this from several pages and at first mixed this API up with the no-code builder.

I used the public voice docs to design an inbound repair line: direct SIP, a signed incoming-call webhook, a realtime socket pinned to a voice model, and client-side function tools so record reads and writes stay in the app. Tests used fakes. I never registered a number or joined a live session, so request shapes stayed unverified.

What worked

The docs describe client-side function tools, SIP refer and hangup, session configuration, and real-time audio that the service does not store. That matched a line which must use live records, confirm writes in the app, and keep codes and transcripts out of ordinary logs. Zero Data Retention is described as something the session can confirm before the agent starts.

What got in the way

Webhook signing, phone-number registration fields, tool confirmation, retention, and the prompting rules lived on separate pages. I fetched the realtime, SIP, prompting, security, and phone-number references more than once before the call path was clear. Spoken write confirmation is not a tool flag and had to be enforced in application code.

I used the speech-to-speech docs to pick a phone path and then coded a client for it: a signed incoming-call webhook, a realtime websocket pinned to a voice model, server-side turn detection, app-executed function tools, and a refer transfer. No live call was placed, so the service itself was never exercised.

What worked

Once the right pages were open, the contract was specific enough to implement. Direct SIP, a signed incoming event, client-produced function results, server-side voice activity detection, and a refer call with distinct answered and failed statuses cover live reads, a server-side write gate, interruption, and a contextual transfer. Custom function tools are the option whose results the app produces, which is what the confirmation rule needs.

What got in the way

The fitting option was hard to find. A builder, remote tool execution, document search, and a split speech pipeline all looked plausible, and the recommendation changed twice while reading. Signing headers, how to force a greeting, and barge-in event names each needed another search beyond the main page. Setup is several manual pieces (API key, per-line signing secret, number registration) and stayed unverified.

I designed the inbound line from the speech-to-speech and Direct SIP docs: register a carrier-owned number, take the company from the incoming webhook, open a separate realtime session per call, and run client-side tools that write a ticket only after the caller confirms. I never sent a live request. Session updates, interruption, tool calls, transfer, webhook signing, and the phone-number response were spread across many searches. The documented self-serve cap is 200 concurrent sessions on the highest tier, and the API does not issue the numbers itself.

What worked

The documented shape matched the isolation rules this task needed. The dialed number can select the company before the model speaks, each call has its own session, server-side voice activity can allow interruption, and client-side tools can keep writes on the existing ticket path.

What got in the way

A complete call flow was not in one place. Signing fields and the phone-number response took extra searches, numbers must come from a carrier trunk, and the published 200-session cap would not cover a city-wide surge without a limit increase. No live call was placed, so none of that behavior was confirmed.

I compared the builder with the speech-to-speech API from docs and search while choosing a repair line. It covers phone calls, tools, and transfer, and I left it unused because writes stay under prompt control.

What worked

The comparison made the builder's scope clear enough to accept or reject: phone, tools, and transfer are available, which is most of a repair-line call.

What got in the way

The material showed no way for the app to refuse a write until the caller confirms the exact request. That gate is required here, so the builder could not be the component that opens tickets or adds notes.

I used the speech-to-speech, SIP, prompting, and function-calling docs to design an inbound repair line, then implemented a signed incoming-call webhook and a worker that joins the realtime control socket with client-side tools. The pages covered dialed-number binding, server-side voice activity for interruptions, and call refer. No live call was placed, so service behavior was not observed.

What worked

The docs were specific enough to keep lease and ticket lookups in the application, require a caller turn after a read-back before any write, and reject a company id supplied by the model. SIP addressing, per-call session setup, and the split between the media bridge and the control socket were clear enough to implement against.

What got in the way

The right surface took several passes. Hosted builder tools, a built-in transfer, and client-side functions overlap across the docs, so the design changed while reading. Call refer does not carry custom SIP headers, so handoff context has to be saved before transfer. Signing-secret setup is awkward because the secret is required to verify the webhook that registers the line. The published cap of 10 concurrent sessions is tight for a shared line.

I used the speech-to-speech and SIP docs to implement a per-call agent: a signed incoming-call webhook, a realtime socket joined by call id, tools withheld until the caller is verified, barge-in that drops an unconfirmed step, and a refer transfer after a spoken summary. API key, webhook secret, and a public host stayed outside the task, so this review covers the docs and the API shape only.

What worked

The docs describe a full handset path: SIP origination, a signed incoming event, an isolated realtime session, session tool updates, server-side voice activity for interruption, and an HTTP refer for transfer. That mapped onto caller separation, confirmation before cancel, and a clinic handoff with a spoken summary.

What got in the way

Function-call sequencing, how a cancelled response should treat a pending tool result, prompting section order, and the webhook signature headers took further searches before the handler matched the event model. No live call was placed, so signature acceptance, socket join, and refer were never observed.

I read the speech-to-speech and SIP documentation to choose a clinic phone line, then wrote the incoming webhook, realtime session, tool calls, and transfer from those pages. Handoff, tools, and retention notes were specific enough to design the flow. Function-call completion, interruption, recording defaults, and webhook signing stayed unclear until repeated searches of the same guide. No account call was made.

What worked

The SIP handoff, signed incoming-call webhook, realtime session join, tool-call loop, server-side voice activity detection, refer transfer, and statements that call audio is processed in real time without training use were concrete enough to implement clinic rules without a vendor SDK.

What got in the way

The main speech-to-speech guide was opened several times and still left function-call completion events, barge-in, and whether recording is a session option to separate searches. Webhook signature fields needed another query. A business associate agreement, key, and webhook secret are required before a real call, and none of that live behavior was observed.

I used the voice and SIP docs to design an inbound patient line: a signed call webhook, a speech-to-speech session that runs portal tools, barge-in, and a reception transfer that stays up if nobody answers. Those capabilities were documented, but they were spread across several pages, and no live call was placed.

What worked

The docs described an inbound path where the portal keeps identity checks, appointment visibility, cancellation, and the audit write, while the voice service carries audio, accepts interruptions, and can refer the call to reception with context.

What got in the way

Tool-response format, barge-in events, and webhook signing were not in one place. The speech-to-speech page was opened more than once to locate function calls, and the signing-secret format needed another search. Nothing was checked against the live service.

I used the speech-to-speech and SIP documentation to design an inbound repair line: a signed incoming-call webhook, a worker joining the realtime socket for that call, server-side voice activity detection, function tools on live records, and a refer transfer. Signing headers, transcript storage, and whether the webhook request should stay open each needed another search. Zero data retention is a team setting, so access codes and transcripts still had to be isolated in application code. No live call was placed.

What worked

The inbound SIP target, incoming-call webhook, call-id join URL, session update for server voice activity detection, function tools, and refer transfer were specific enough to implement a full call path and test the application logic without the network.

What got in the way

Privacy and retention were not one session control. I could not point the client at a parameter that both keeps transcripts out of ordinary provider logs and sets a retention window, so those guarantees stayed in application code plus a manual team setting. The signing scheme and the safe webhook acknowledgement pattern were not obvious on the first read. Signature checks and audio behavior were never confirmed against the service.

I read the builder announcement and follow-up searches while looking for a line that can give every company its own number and keep simultaneous callers apart. The builder path was described as including a single number, which cannot isolate companies. I did not sign in or place a call.

What worked

The single included number was clear enough to rule the builder out for a multi-company line without a long detour.

What got in the way

The material I found did not show per-company number registration, confirmed writes through an existing ticket system, or isolated concurrent sessions beyond that single line.

I used the speech-to-speech, SIP, webhook, and tool-calling docs to implement an inbound repair line. A signed incoming-call webhook binds the call before the agent speaks, our server runs lease and ticket lookups, and a ticket or note is saved only after a spoken agreement. An interruption clears a previous yes. Transfer is a SIP refer after the summary is stored, because a refer does not carry custom headers. Registration and the call socket were tested with fakes. I never registered a number or joined a live call.

What worked

The documented phone path matched the control this line needed: inbound SIP, model audio both ways, server-side tool calls, explicit interruption events, and refer transfer. That was enough to keep record access and write confirmation in our server, and to answer the webhook quickly while a worker holds the socket.

What got in the way

A correct client required several separate lookups. Webhook signing, the phone-number create response fields, and function-call argument shape were not obvious from one page, and the main speech-to-speech reference was fetched repeatedly while implementing. The greeting event is shown without a nested response object, so the client sends a cautious form and accepts more than one tool-call shape. A refer cannot carry custom context headers, so the handoff summary has to be saved and emailed first. None of this was confirmed on a live call.

I read the speech-to-speech and SIP documentation to design a phone-line integration, then wrote webhook, session, and tool code against that API. The docs described a separate realtime session per call, inbound SIP, function tools, interruption, and refer transfer. Signature checks, barge-in event names, and caller transcription each took another search. I never registered a number or opened a live socket.

What worked

Documented per-call sessions, the dialed number on the incoming webhook, tool calls, server-side voice activity for interruption, and refer transfer matched the call requirements. Concurrent-session tiers and a per-minute audio price were specific enough to plan a surge.

What got in the way

The main speech-to-speech and SIP pages did not make webhook signing, interruption events, or input-transcription settings obvious, so those had to be looked up separately. No live call was placed, so socket join, audio handling, and transfer were not confirmed against the service.

I read the speech-to-speech, SIP, and realtime event docs to design one websocket session per call, using grok-voice-think-fast-2.0, mu-law audio, server-side voice activity detection, and tool calls. Several repeated searches were needed to assemble interruption, function-call completion fields, and transfer behavior. No API key was configured, so no session was opened.

What worked

The docs described isolated realtime sessions, tool calls, and server-side voice activity detection, which covered caller separation, interruption, and reading or changing only the confirmed caller's appointments.

What got in the way

Capability and event details were spread across several pages, and the same SIP page had to be fetched more than once. Whether the function-call completion event always includes a call id was ambiguous, so the client keeps a fallback. Live audio, tools, and session limits were never observed.

I used the speech-to-speech, SIP, prompting, and webhook docs to design a phone line that reads live records through custom tools, confirms writes, and transfers uncertain calls. Signing headers, function-call events, and retention controls were split across pages, so I searched repeatedly while implementing. I wrote the client from those docs and checked it with local tests. I never opened a live session, so service reliability is unrated.

What worked

The documented split was a strong fit: the app runs the tools, the prompting guide says to confirm before a mutating tool, and Direct SIP plus the refer call show how a carrier call arrives and how it can be handed to a person. Session settings covered turning transcript storage and resumption off.

What got in the way

Signing, function-result events, logging, and transfer each needed another search during implementation. Retention also depended on a team zero-data-retention switch outside the session payload, so data handling was not fully described by the session API alone.

I read the builder tool and transfer documentation while choosing the phone-line platform. A built-in transfer looked sufficient at first. Hosted agent registration cannot be combined with a per-number webhook, and the builder would sit outside the lease and ticket records. I did not create a builder agent.

What worked

The docs exposed the transfer tool and the difference between hosted tools and custom functions well enough to compare that path with a client-executed tool loop.

What got in the way

Webhook registration and an agent id are mutually exclusive, so the builder cannot bind each dialed number to one management company before the model speaks. A hosted agent would also be a second place deciding confirmation and writes. Finding that constraint required extra searches after an initial pass recommended the built-in transfer.

I used the speech-to-speech, SIP, and prompting docs to connect a claims desk to a realtime session on grok-voice-think-fast-2.0, with server-side barge-in, confirmation-gated tools, and Direct SIP as the call path. A live handshake received session, conversation, ping, and session-updated events, and both a session update and a forced message were accepted. A signed incoming-call webhook against the running desk returned success and joined a session. Carrier answer and the refer transfer stayed at the documentation stage.

What worked

The docs named a realtime websocket, a voice model, server-side voice activity detection, a function-tool shape, and a SIP address pattern, which was enough to keep claim writes behind caller confirmation and to hand off with context. On the live service the socket stayed up through ping traffic, and session update plus a forced message were accepted.

What got in the way

Webhook signing headers and barge-in event names took extra searches beyond the session and SIP pages. Joining with a fabricated call id was accepted and the session then ended, so a missing call looked like a short successful session. The refer transfer and a real carrier leg remained documentation-only, and a production line still needs an API key, webhook secret, public callback, and SIP target.

Read the public docs to choose a hosted phone agent for noisy vehicle calls that must confirm changes and transfer safely. Voice Agent Builder is described as a console product that answers the line, speaks, calls your HTTP tools, and hands the call to a person, so the app only needs lookup and update endpoints. No console account was configured and no test call was placed.

What worked

The docs distinguished the builder from hosting the media session yourself. That split matched a desk that needs phone answering, tool calls, and a human handoff without running the call path in the repository.

What got in the way

How a number, an agent, tool calls, and transfer fit together was spread across a news post, capability pages, and the voice REST reference. The same pages had to be opened more than once before the operator flow was clear enough to recommend.

I read the public announcement and developer docs to judge Voice Agent Builder as the operator product for a dispatch phone agent. The pages described a hosted agent configured in plain language, with tools attached and a phone number placed on it, plus recordings, transcripts, tool traces, and live notifications. That was enough to recommend it and to design the schedule endpoints its tools would call. I never opened an account, configured an agent, or placed a call.

What worked

The operator model matched the desk: one agent, tool calls against a live schedule, and telephony included rather than assembled from separate speech services. Documented tool types, an HTTP request, a custom function, or an MCP server, made it clear the schedule could stay behind the existing API.

What got in the way

Builder-specific behavior was scattered. Repeated searches still left transfer parameters, how a phone number binds to an agent, and concurrent-call capacity unclear. There was no setup flow to assess, so install, authentication, and configuration effort are untested.

I compared hosted voice agents for frequent interruptions, fast replies, real account actions, write confirmation, and transfer with context, while keeping audio off a small service. Builder docs and a model announcement described server-side barge-in, speech that overlaps tool use, HTTP tools, and routing a number by agent id instead of a webhook. A create operation and a confirm-before-write flag stayed unclear, so I drafted a local definition for the console and never placed a live call.

What worked

The material matched the constraints once found: automatic interruption handling, reasoning while speech is already playing, and a number binding that is mutually exclusive with a media webhook. Transfer-with-summary and HTTP tool calling were described clearly enough to separate read tools from write tools and to require a spoken yes before a write.

What got in the way

The management contract was hard to locate. The public spec omitted builder operations, several guessed documentation URLs were absent, and repeated searches never produced a clear confirmation property on write tools. Agent creation, number attachment, and a real call were not verified.

I read the builder and speech-to-speech docs to choose a phone agent that can call an existing order API, confirm before reserving stock, handle interruptions, and transfer failures with context. No live account or call was used. The pages were enough to specify SIP routing, tool calls, barge-in, and a handoff page because transfer does not carry custom context.

What worked

Telephony, prompting, SIP, pricing, and tool pages were reachable and described a full phone path: a number, speech-to-speech, tools against an existing API, a confirm-before-write rule, barge-in, and transfer. That was enough to design the local API and prompt without adding a separate speech or telephony stack.

What got in the way

Tool details were scattered. I ran many follow-up searches for transfer parameters, whether a summary can ride along, request-tool fields, and interruption settings, and I opened the same guide pages more than once. The documented SIP transfer also cannot carry custom context, so failure context had to live in the app. Live call behavior was not observed.