Start Session
The client provides a convenient way to interact with the model API.
npm install @fal-ai/client@alpha @fal-ai/server-proxy@alphafal.realtime.open is currently available on the alpha release and may change in a minor version.
Keep FAL_KEY on your server and expose an authenticated proxy route to your browser. For a Next.js App Router application, create app/api/fal/proxy/route.ts:
import { route } from "@fal-ai/server-proxy/nextjs";
export const { GET, POST, PUT } = route;Set FAL_KEY in the server environment, and protect this route with your application's authentication before deploying it.
This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.
import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";
const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });
const session = fal.realtime.open(wma("fal-ai/abot-world-0"), {
receive: [
"video"
],
onMedia: (stream) => {
const element = document.querySelector("video");
if (element) element.srcObject = stream;
},
onData: (raw) => {
const message = JSON.parse(raw);
console.log(message);
},
onState: (state) => console.log("realtime:", state),
onError: (error) => console.error(error),
});
// Consult the session flow and message reference before sending control messages.
// Later, release the peer connection and network resources:
// await session.close();This world starts automatically. Wait for the first video frame before using the movement controls.
Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.
configure and wait for configured before sending another world update. Setup is optional and can be used once per session.reset and wait for reset_applied before sending another world update.prompt and wait for prompt_applied before sending another world update.World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.
The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.
When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.
Client contract for the WebRTC session created by the linked OpenAPI operation.
Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.
No send media tracks.
{
"height": 704,
"frameRate": 16,
"width": 1280
}type: "commands"payload: objectSemantic command-state report.
active contains commands currently held by the user and activated
contains edge-triggered commands since the previous report. A browser may
map a keyboard, gamepad, touch control, or agent action to these names.
activestring[]maxItems: 8
stringValues: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"
typestring* requiredConstant: "commands"
activatedstring[]maxItems: 8
stringValues: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"
{
"active": [
"forward"
],
"type": "commands",
"activated": []
}type: "configure"payload: objectOptionally replace the automatic default world, once per connection.
Await configured before submitting more controls. Use reset for
subsequent restarts. Image URL takes precedence over the preset.
presetstring | nullChoose a built-in world, or provide a starting image below.
Default: null
At most 100 characters when not null
promptstring | nullOptionally replace the starting world's description.
Default: null
At most 2000 characters when not null
image_urlstring | nullUse your own image instead of a built-in world.
Default: null
At most 4096 characters when not null
typestring* requiredConstant: "configure"
A huge real-looking alpine village beneath towering snow peaks. A broad level cleared stone path crosses a spacious sunny courtyard between timber chalets. Two villagers and a dog remain clearly beside the path. The route ends at a breathtaking mountain viewpoint.
{
"type": "configure",
"image_url": "https://v3b.fal.media/files/b/0aad61d2/OxN2bYXBwsBPbFtO0M81d_NN9HvttQ.png",
"prompt": "Photorealistic third-person exploration. Natural crisp morning light, believable snow, wood and stone, no illustration. A huge real-looking alpine village beneath towering snow peaks. A broad level cleared stone path crosses a spacious sunny courtyard between timber chalets. Two villagers and a dog remain clearly beside the path. The route ends at a breathtaking mountain viewpoint. Adult man with fair skin, wavy blond hair, dark green hiking jacket, navy trousers and red backpack. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}type: "keys"payload: objectLegacy physical-key report retained for existing WMA clients.
pressedstring[]maxItems: 32
stringtypestring* requiredConstant: "keys"
activatedstring[]maxItems: 32
string{
"type": "keys"
}type: "ping"payload: objectLatency probe; the server echoes ts back in a pong payload.
typestring* requiredConstant: "ping"
tsnumber | nullDefault: null
{
"type": "ping"
}type: "prompt"payload: objectpromptstring* requiredChange the scene's description while continuing this world.
At least 1 character · At most 2000 characters
typestring* requiredConstant: "prompt"
{
"type": "prompt",
"prompt": "A realistic outdoor world scene with a navigable path, natural lighting, detailed ground texture, and stable forward motion."
}type: "reset"payload: objectpresetstring | nullChoose a built-in world, or provide a starting image below.
Default: null
At most 100 characters when not null
promptstring | nullOptionally replace the starting world's description.
Default: null
At most 2000 characters when not null
image_urlstring | nullUse your own image instead of a built-in world.
Default: null
At most 4096 characters when not null
typestring* requiredConstant: "reset"
A huge real-looking alpine village beneath towering snow peaks. A broad level cleared stone path crosses a spacious sunny courtyard between timber chalets. Two villagers and a dog remain clearly beside the path. The route ends at a breathtaking mountain viewpoint.
{
"type": "reset",
"image_url": "https://v3b.fal.media/files/b/0aad61d2/OxN2bYXBwsBPbFtO0M81d_NN9HvttQ.png",
"prompt": "Photorealistic third-person exploration. Natural crisp morning light, believable snow, wood and stone, no illustration. A huge real-looking alpine village beneath towering snow peaks. A broad level cleared stone path crosses a spacious sunny courtyard between timber chalets. Two villagers and a dog remain clearly beside the path. The route ends at a breathtaking mountain viewpoint. Adult man with fair skin, wavy blond hair, dark green hiking jacket, navy trousers and red backpack. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}type: "stop"payload: objectAdditional properties: not allowed
typestring* requiredConstant: "stop"
{
"type": "stop"
}type: "configured"payload: objecttypestring* requiredConstant: "configured"
{
"type": "configured"
}type: "error"payload: objecttypestring* requiredConstant: "error"
retryableboolean | nullWhether retrying unchanged input may succeed; absent if unspecified.
Default: null
errorstring* required{
"type": "error",
"error": "string"
}type: "pong"payload: objectReply to a ping, echoing the client's timestamp for RTT measurement.
The echoed timestamp has two wire spellings: apps that handle ping
themselves echo ts; the fallback handler in
:class:registry.wma.sdk.Session replies with client_ts. Clients read
whichever is present. server_ts is the (unsynchronized) runner clock; it
only shows how the round trip divides once the client has an offset estimate.
typestring* requiredConstant: "pong"
tsnumber | nullDefault: null
server_tsnumber | nullDefault: null
client_tsnumber | nullDefault: null
{
"type": "pong"
}type: "prompt_applied"payload: objecttypestring* requiredConstant: "prompt_applied"
{
"type": "prompt_applied"
}type: "reset_applied"payload: objecttypestring* requiredConstant: "reset_applied"
{
"type": "reset_applied"
}type: "session_info"payload: objectpresetsobject[]* requiredobjectnamestring* requiredpromptstring* requiredturn_statusstring* requiredtypestring* requiredConstant: "session_info"
turnboolean* requiredframes_per_blockinteger* requiredframes_per_secondnumber* requiredresolutionobject* requiredheightinteger* requiredwidthinteger* requiredkey_orderstring[]* requiredstringmax_blocksinteger* requiredconflict_groupsstring[][]* requiredstring[]stringmodel_revisionstring* requiredmodel_idstring* requireddefault_promptstring* required{
"presets": [],
"turn_status": "string",
"type": "session_info",
"turn": true,
"frames_per_block": 0,
"frames_per_second": 0,
"resolution": {
"height": 0,
"width": 0
},
"key_order": [],
"max_blocks": 0,
"conflict_groups": [],
"model_revision": "string",
"model_id": "string",
"default_prompt": "string"
}type: "stats"payload: objectgeneration_fpsnumber* requiredpace_sleep_ms_avgnumber* requiredblock_secondsnumber* requiredqueue_depthinteger* requiredtypestring* requiredConstant: "stats"
queue_age_ms_p50number* requiredblock_indexinteger* requiredgpu_wait_fracnumber* requiredqueue_age_ms_maxnumber* required{
"generation_fps": 0,
"pace_sleep_ms_avg": 0,
"block_seconds": 0,
"queue_depth": 0,
"type": "stats",
"queue_age_ms_p50": 0,
"block_index": 0,
"gpu_wait_frac": 0,
"queue_age_ms_max": 0
}type: "stream_exhausted"payload: objectThe model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.
typestring* requiredConstant: "stream_exhausted"
{
"type": "stream_exhausted"
}