Loading API documentation
Start Session
The client provides a convenient way to interact with the model API.
npm install @fal-ai/client@alpha @fal-ai/server-proxy@alphafal.realtime.open is currently available on the alpha release and may change in a minor version.
Keep FAL_KEY on your server and expose an authenticated proxy route to your browser. For a Next.js App Router application, create app/api/fal/proxy/route.ts:
import { route } from "@fal-ai/server-proxy/nextjs";
export const { GET, POST, PUT } = route;Set FAL_KEY in the server environment, and protect this route with your application's authentication before deploying it.
This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.
import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";
const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });
const deviceStream = await navigator.mediaDevices.getUserMedia({
"video": {
"height": 704,
"width": 1280,
"frameRate": 24
},
"audio": false
});
const localStream = deviceStream;
const session = fal.realtime.open(wma("vast/eden-live-5b-video"), {
receive: [
"video"
],
localStream,
onMedia: (stream) => {
const element = document.querySelector("video");
if (element) element.srcObject = stream;
},
onData: (raw) => {
const message = JSON.parse(raw);
console.log(message);
},
onState: (state) => console.log("realtime:", state),
onError: (error) => console.error(error),
});
// Send any client message described below.
session.send({
"prompt": "Change to anime style",
"enable_prompt_expansion": true,
"type": "configure"
});
// Later, release the peer connection and network resources:
// localStream.getTracks().forEach((track) => track.stop());
// await session.close();The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.
When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.
Client contract for the WebRTC session created by the linked OpenAPI operation.
Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.
{
"height": 704,
"width": 1280,
"frameRate": 24
}{
"height": 704,
"width": 1280,
"frameRate": 24
}type: "configure"payload: objectAdditional properties: not allowed
Set the session's conditioning, once per connection; await configured.
Without image_url the world anchors on a reference generated from the
newest input frame and the prompt (FLUX.2 [klein] 4B edit), or on that
frame itself when generation fails.
promptstring | nullWhat the input video should look like. Defaults to a generic photoreal prompt.
Default: null
At most 2000 characters when not null
image_urlstring | nullRendering of the input's first frame in the target look (URL, or a data URI under ~45 KB); anchors the generated world. Without it, one is generated from the newest input frame and the prompt.
Default: null
At most 60000 characters when not null
enable_prompt_expansionbooleanRewrite the scene description (and later prompt changes) into a detailed one with an LLM, using the reference image when given. Adds about a second to session start and to each prompt change.
Default: false
seedinteger | nullNoise seed; the same inputs and seed reproduce the output.
Default: null
minimum: 0 when not null · maximum: 2147483647 when not null
typestring* requiredConstant: "configure"
{
"prompt": "Change to anime style",
"enable_prompt_expansion": true,
"type": "configure"
}type: "ping"payload: objectLatency probe; the server echoes ts back in a pong payload.
tsnumber | nullDefault: null
typestring* requiredConstant: "ping"
{
"type": "ping"
}type: "prompt"payload: objectAdditional properties: not allowed
Change the scene description; applied at the next chunk boundary.
promptstring* requiredText the model renders the scene with. Keeps the current world.
At least 1 character · At most 2000 characters
typestring* requiredConstant: "prompt"
{
"prompt": "string",
"type": "prompt"
}type: "reset"payload: objectAdditional properties: not allowed
Restart generation from a new anchor; unset fields keep their values,
and an explicit null clears image_url or seed.
The world is re-anchored on a reference generated from the newest input
frame unless image_url is set.
promptstring | nullWhat the input video should look like. Defaults to a generic photoreal prompt.
Default: null
At most 2000 characters when not null
image_urlstring | nullRendering of the input's first frame in the target look (URL, or a data URI under ~45 KB); anchors the generated world. Without it, one is generated from the newest input frame and the prompt.
Default: null
At most 60000 characters when not null
seedinteger | nullNoise seed; the same inputs and seed reproduce the output.
Default: null
minimum: 0 when not null · maximum: 2147483647 when not null
enable_prompt_expansionboolean | nullDefault: null
typestring* requiredConstant: "reset"
{
"type": "reset"
}type: "stop"payload: objectAdditional properties: not allowed
End the session.
typestring* requiredConstant: "stop"
{
"type": "stop"
}type: "configured"payload: objectgameobject | nullDefault: null
Must match at least one of the following:
objectpromptstring* requireddescriptionstring* requiredinstructionsstring[]* requiredstringtitlestring* requiredactionsstring[]* requiredstringcontrolsobject[]* requiredobjectlabelstring* requiredkeystring* requiredholdbooleanDefault: true
commandstring | nullSemantic command name.
Default: null
idstring* requiredthumbnail_urlstring | nullDefault: null
nulltypestring* requiredConstant: "configured"
{
"type": "configured"
}type: "error"payload: objectretryableboolean | nullDefault: null
errorstring* requiredtypestring* requiredConstant: "error"
{
"error": "string",
"type": "error"
}type: "pong"payload: objectReply to a ping, echoing the client's timestamp for RTT measurement.
The echoed timestamp has two wire spellings: apps that handle ping
themselves echo ts; the fallback handler in
:class:registry.wma.sdk.Session replies with client_ts. Clients read
whichever is present. server_ts is the (unsynchronized) runner clock; it
only shows how the round trip divides once the client has an offset estimate.
client_tsnumber | nullDefault: null
server_tsnumber | nullDefault: null
tsnumber | nullDefault: null
typestring* requiredConstant: "pong"
{
"type": "pong"
}type: "prompt_applied"payload: objecttypestring* requiredConstant: "prompt_applied"
{
"type": "prompt_applied"
}type: "reset_applied"payload: objectgameobject | nullDefault: null
Must match at least one of the following:
objectpromptstring* requireddescriptionstring* requiredinstructionsstring[]* requiredstringtitlestring* requiredactionsstring[]* requiredstringcontrolsobject[]* requiredobjectlabelstring* requiredkeystring* requiredholdbooleanDefault: true
commandstring | nullSemantic command name.
Default: null
idstring* requiredthumbnail_urlstring | nullDefault: null
nulltypestring* requiredConstant: "reset_applied"
{
"type": "reset_applied"
}type: "session_info"payload: objectsessionstring* requiredValues: "video", "game"
infoobject* requiredtypestring* requiredConstant: "session_info"
{
"session": "video",
"info": {},
"type": "session_info"
}type: "stats"payload: objectbillable_unitsnumber | nullSeconds of video billed so far this session (queued for playback); the final total settles when the session closes.
Default: null
pipelinedboolean | nullThe next chunk renders while this one generates (throughput mode).
Default: null
chunk_msnumber* requiredloop_lag_msnumber | nullServer event-loop lag, p95 over the stats window.
Default: null
generate_msnumber | nullModel time per chunk (moving average).
Default: null
prompt_msnumber | nullPer-chunk prompt update time with dynamic_prompt, unchanged chunks included (moving average).
Default: null
typestring* requiredConstant: "stats"
generation_fpsnumber* requiredchunkinteger* requiredlatency_msnumber | nullInput or control capture to frame handoff, median.
Default: null
input_latency_msnumber | nullGame control message arrival to the handoff of the first frame it changes, median.
Default: null
queue_framesinteger* requiredrealtime_factornumber* requiredGenerated frames per second / target fps.
render_msnumber | nullGame render time per chunk (moving average).
Default: null
{
"chunk_ms": 0,
"type": "stats",
"generation_fps": 0,
"chunk": 0,
"queue_frames": 0,
"realtime_factor": 0
}type: "stream_exhausted"payload: objectThe model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.
typestring* requiredConstant: "stream_exhausted"
{
"type": "stream_exhausted"
}