Bidirectional proxy for Grok's realtime speech-to-speech agent.
Relays xAI's realtime events verbatim in both directions between
the fal client and
wss://api.x.ai/v1/realtime?model=grok-voice-latest. Connect
with a fal realtime client (fal_client.realtime_async in
Python, fal.realtime.connect in JS) and exchange the event
dicts documented by xAI — the client library owns the wire
framing, so payloads are plain dicts, not JSON strings. The
upstream connection is opened as soon as the session starts, so
xAI's session.created reaches clients that connect and wait
without sending. The XAI_KEY credential is injected
server-side on the upstream handshake and never exposed to the
caller.
Session parameters:
session_timeout=3600 — matches xAI's 1-hour maximum
session length.buffering=1 — voice agents are latency-sensitive; do not
accumulate input frames.The client provides a convenient way to interact with the model API.
npm install --save @fal-ai/clientThe @fal-ai/serverless-client package has been deprecated in favor of @fal-ai/client. Install the new package and update your imports — see client setup.
Set FAL_KEY as an environment variable in your runtime.
export FAL_KEY="YOUR_API_KEY"This model has a real-time mode via WebSockets, supported by fal.realtime.connect.
import { fal } from "@fal-ai/client";
const connection = fal.realtime.connect("xai/grok-voice/realtime", {
onResult: (result) => {
console.log(result);
},
onError: (error) => {
console.error(error);
},
// Fetch short-lived JWT token from your backend
tokenProvider: async (app) => {
const response = await fetch("/api/fal/realtime-token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ app }),
});
return response.text();
},
tokenExpirationSeconds: 10,
});
connection.send({
type: "session.update"
});The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.
import { fal } from "@fal-ai/client";
fal.config({
credentials: "YOUR_FAL_KEY"
});When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.
Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.
You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.
You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.
We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.
import { fal } from "@fal-ai/client";
const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);The client will auto-upload the file for you if you pass a binary object (e.g. File, Data).
Read more about file handling in our file upload guide.
prompt stringSystem instructions describing the agent's persona and conversation context. Uses Grok's default persona when omitted. Default value: "null"
Automatic turn detection: any configured object (even empty) lets the agent detect when you stop speaking and reply on its own. Set null for manual turns; omit to keep the session default.
Server-side tools available to the agent. Web, X, and remote MCP tools are supported; every tool is disabled by default. An explicitly supplied object always reaches the session, so previously enabled tools can be cleared.
type string* requiredxAI realtime event type (e.g. 'session.update', 'input_audio_buffer.append', 'response.create', 'response.output_audio.delta'). Events are relayed verbatim in both directions.
voice EnumBuilt-in xAI voice for the agent's speech. Uses Grok's default voice when omitted.
Possible enum values: carina, zagan, helix, orion, luna, iris, altair, zenith, perseus, helios, lux, kepler, rigel, cosmo, celeste, ursa, sirius, lumen, castor, naksh, atlas, aurora, liora, ara, eve, leo, rex, sal
{
"type": "session.update"
}prompt stringSystem instructions describing the agent's persona and conversation context. Uses Grok's default persona when omitted. Default value: "null"
Automatic turn detection: any configured object (even empty) lets the agent detect when you stop speaking and reply on its own. Set null for manual turns; omit to keep the session default.
Server-side tools available to the agent. Web, X, and remote MCP tools are supported; every tool is disabled by default. An explicitly supplied object always reaches the session, so previously enabled tools can be cleared.
type string* requiredxAI realtime event type (e.g. 'session.update', 'input_audio_buffer.append', 'response.create', 'response.output_audio.delta'). Events are relayed verbatim in both directions.
voice EnumBuilt-in xAI voice for the agent's speech. Uses Grok's default voice when omitted.
Possible enum values: carina, zagan, helix, orion, luna, iris, altair, zenith, perseus, helios, lux, kepler, rigel, cosmo, celeste, ursa, sirius, lumen, castor, naksh, atlas, aurora, liora, ara, eve, leo, rex, sal
{
"type": "session.update"
}threshold floatVAD activation threshold; higher values require louder audio.
prefix_padding_ms integerAudio retained before detected speech, in milliseconds.
idle_timeout_ms integerIdle time after an assistant response before xAI starts a proactive turn, in milliseconds.
silence_duration_ms integerSilence duration that ends a detected turn, in milliseconds.
xAI-managed X search; off unless enabled is set true.
xAI-managed web search; off unless enabled is set true.
Remote MCP servers that xAI connects to and executes.
from_date stringEarliest post date, as ISO YYYY-MM-DD. Default value: "null"
enabled booleanWhether to include x_search in session.tools. Disabled by default; set true to let the agent search X posts.
enable_video_understanding booleanAllow the agent to inspect videos found on X.
enable_image_understanding booleanAllow the agent to inspect images found on X.
Exclude posts from at most 20 X handles.
Restrict results to posts from at most 20 X handles.
to_date stringLatest post date, as ISO YYYY-MM-DD. Default value: "null"
enabled booleanWhether to include web_search in session.tools. Disabled by default; set true to let the agent search the web.
enable_image_understanding booleanAllow the agent to inspect images found during web search.
Optional location used to localize search results.
Exclude results from at most five domains, without protocol or path.
Restrict results to at most five domains, without protocol or path.
server_description stringDescription of the capabilities exposed by this server. Default value: "null"
enabled booleanFalse omits this server. Default value: true
authorization stringSensitive authorization value forwarded only to xAI. Default value: "null"
Sensitive MCP request headers forwarded only to xAI.
Optional allow-list of tool names exposed from this server.
server_url string* requiredCaller-selected HTTP(S) MCP endpoint. xAI makes the remote connection and supports Streaming HTTP/SSE transports. Use at your own risk; fal does not connect to or SSRF-fetch this URL.
server_label string* requiredLabel xAI uses to identify and prefix this server's tools.
country stringCountry used to localize web-search results. Default value: "null"
city stringCity used to localize web-search results. Default value: "null"
timezone stringIANA timezone name, for example America/New_York. Default value: "null"
region stringRegion used to localize web-search results. Default value: "null"