Claude

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install @fal-ai/client@alpha @fal-ai/server-proxy@alpha

Experimental realtime API

fal.realtime.open is currently available on the alpha release and may change in a minor version.

Set up the server proxy#

Keep FAL_KEY on your server and expose an authenticated proxy route to your browser. For a Next.js App Router application, create app/api/fal/proxy/route.ts:

import { route } from "@fal-ai/server-proxy/nextjs";

export const { GET, POST, PUT } = route;

Set FAL_KEY in the server environment, and protect this route with your application's authentication before deploying it.

Real-time via WebRTC#

This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.

import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";

const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });

const session = fal.realtime.open(wma("vast/eden-live-5b-game"), {
  receive: [
    "video"
  ],
  onMedia: (stream) => {
      const element = document.querySelector("video");
      if (element) element.srcObject = stream;
    },
  onData: (raw) => {
      const message = JSON.parse(raw);
      console.log(message);
    },
  onState: (state) => console.log("realtime:", state),
  onError: (error) => console.error(error),
});


// Consult the session flow and message reference before sending control messages.

// Later, release the peer connection and network resources:
// await session.close();

Session flow#

This world starts automatically. Wait for the first video frame before using the movement controls.

Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.

Send configure and wait for configured before sending another world update. Setup is optional and can be used once per session.
Send reset and wait for reset_applied before sending another world update.
Send prompt and wait for prompt_applied before sending another world update.

World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "width": 1280,
  "frameRate": 24,
  "height": 704
}

Client messages#

Game action

type: "action"

payload: object

A game UI action, e.g. a "Play again" or "Pause" button.

actionstring* required

One of the game's actions from session_info (restart, pause).

At least 1 character · At most 32 characters

typestring* required

Constant: "action"

Example

{
  "action": "restart",
  "type": "action"
}

Controls

type: "commands"

payload: object

Semantic command-state report.

active contains commands currently held by the user and activated contains edge-triggered commands since the previous report. A browser may map a keyboard, gamepad, touch control, or agent action to these names.

activestring[]

maxItems: 14

Array item

string

Values: "forward", "backward", "left", "right", "look_up", "look_down", "look_left", "look_right", "fire", "jump", "ability_1", "ability_2", "ability_3", "ability_4"

activatedstring[]

maxItems: 14

Array item

string

Values: "forward", "backward", "left", "right", "look_up", "look_down", "look_left", "look_right", "fire", "jump", "ability_1", "ability_2", "ability_3", "ability_4"

typestring* required

Constant: "commands"

Example

{
  "active": [
    "forward",
    "fire"
  ],
  "activated": [],
  "type": "commands"
}

Choose game

type: "configure"

payload: object

Pick the game and how it is presented, once per connection; await configured.

gamestring | null

Game id from /game/info. Defaults to the first game.

Default: null

When not null:

Values: "freeze", "spiders", "pulse", "deepwater", "kaiju", "arcadeduel", "desert", "snowball", "puppy", "cat", "drone", "herding"

hudstring

'baked' draws the game's HUD into the video; 'client' only sends game_state messages so the client draws its own UI; 'none' sends neither.

Default: "baked"

Values: "baked", "client", "none"

autopilotboolean

Let the game play itself; held player input overrides it.

Default: false

promptstring | null

Override the game's built-in scene description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Override the game's photoreal opening frame.

Default: null

At most 60000 characters when not null

seedinteger | null

Default: null

minimum: 0 when not null · maximum: 2147483647 when not null

condition_capturestring

Resolution the game frames the model conditions on are captured at: 'output', 'encoder' (the control encoder's input, which every capture is resized to) or 'encoder_half'. Smaller captures render faster; sizes are in /game/info.

Default: "output"

Values: "output", "encoder", "encoder_half"

dynamic_promptboolean

Before each chunk, re-encode the prompt as the scene description plus the game's description of what happened in that chunk (shots, hits, turns). Adds the text-encode time to input latency on chunks whose description changed.

Default: true

typestring* required

Constant: "configure"

Example

Freeze stepping stones across a lava lake and reach the far platform.

{
  "autopilot": false,
  "condition_capture": "output",
  "game": "freeze",
  "hud": "baked",
  "dynamic_prompt": true,
  "type": "configure"
}

Client input message

type: "input"

payload: object

Ordered raw input events for custom game clients.

Clicks on the baked HUD arrive as pointer presses (see MessageUI).

eventsobject[]* required

minItems: 1 · maxItems: 64

Array item

object

One raw input event; see GameController.push_input.

xnumber | null

Default: null

minimum: 0 when not null · maximum: 1 when not null

ynumber | null

Default: null

minimum: 0 when not null · maximum: 1 when not null

kindstring

Default: "key"

Values: "key", "action", "pointer"

namestring* required

At least 1 character · At most 32 characters

phasestring

Default: "press"

Values: "down", "up", "press"

typestring* required

Constant: "input"

{
  "events": [
    {
      "name": "string"
    }
  ],
  "type": "input"
}

Client keys message

type: "keys"

payload: object

Legacy physical-key report retained for existing WMA clients.

pressedstring[]

maxItems: 16

Array item

string

Values: "W", "A", "S", "D", "ARROWUP", "ARROWDOWN", "ARROWLEFT", "ARROWRIGHT", "F", " ", "E", "SHIFT", "R", "G", "P", "ESCAPE"

activatedstring[]

maxItems: 16

Array item

string

Values: "W", "A", "S", "D", "ARROWUP", "ARROWDOWN", "ARROWLEFT", "ARROWRIGHT", "F", " ", "E", "SHIFT", "R", "G", "P", "ESCAPE"

typestring* required

Constant: "keys"

{
  "type": "keys"
}

Client ping message

type: "ping"

payload: object

Latency probe; the server echoes ts back in a pong payload.

tsnumber | null

Default: null

typestring* required

Constant: "ping"

{
  "type": "ping"
}

Update prompt

type: "prompt"

payload: object

Change the scene description; applied at the next chunk boundary.

promptstring* required

Text the model renders the scene with. Keeps the current world.

At least 1 character · At most 2000 characters

typestring* required

Constant: "prompt"

{
  "prompt": "string",
  "type": "prompt"
}

Restart

type: "reset"

payload: object

Restart the round (and the model), optionally switching game or settings.

gamestring | null

Game id from /game/info. Defaults to the first game.

Default: null

When not null:

Values: "freeze", "spiders", "pulse", "deepwater", "kaiju", "arcadeduel", "desert", "snowball", "puppy", "cat", "drone", "herding"

promptstring | null

Override the game's built-in scene description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Override the game's photoreal opening frame.

Default: null

At most 60000 characters when not null

seedinteger | null

Default: null

minimum: 0 when not null · maximum: 2147483647 when not null

dynamic_promptboolean | null

Default: null

condition_capturestring | null

Default: null

When not null:

Values: "output", "encoder", "encoder_half"

hudstring | null

Default: null

When not null:

Values: "baked", "client", "none"

autopilotboolean | null

Default: null

typestring* required

Constant: "reset"

Example

Freeze stepping stones across a lava lake and reach the far platform.

{
  "game": "freeze",
  "type": "reset"
}

End session

type: "stop"

payload: object

End the session.

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server configured message

type: "configured"

payload: object

gameobject | null

Default: null

Must match at least one of the following:

GameInfo

object

actionsstring[]* required

Array item

string

thumbnail_urlstring | null

Default: null

descriptionstring* required

instructionsstring[]* required

Array item

string

promptstring* required

idstring* required

controlsobject[]* required

Array item

object

labelstring* required

holdboolean

Default: true

keystring* required

commandstring | null

Semantic command name.

Default: null

titlestring* required

Alternative 2

null

typestring* required

Constant: "configured"

{
  "type": "configured"
}

Server controls_state message

type: "controls_state"

payload: object

The current game's complete control set, sent after configured and reset_applied and whenever the game changes its controls mid-round.

Only controls with a command can be pressed with commands; clients show and bind those, and drop the rest of the vocabulary.

pointerstring | null

Mouse input the client should offer: 'click' presses buttons the baked HUD draws; absent offers none.

Default: null

When not null:

Values: "click", "capture"

controlsobject[]* required

Array item

object

labelstring* required

holdboolean

Default: true

keystring* required

commandstring | null

Semantic command name.

Default: null

typestring* required

Constant: "controls_state"

{
  "controls": [],
  "type": "controls_state"
}

Server error message

type: "error"

payload: object

retryableboolean | null

Default: null

errorstring* required

typestring* required

Constant: "error"

{
  "error": "string",
  "type": "error"
}

Server game_state message

type: "game_state"

payload: object

Game state at an output frame; frames are numbered from session start.

frameinteger* required

stateobject | null* required

Must match at least one of the following:

Alternative 1

object

Alternative 2

null

roundinteger* required

typestring* required

Constant: "game_state"

{
  "frame": 0,
  "state": {},
  "round": 0,
  "type": "game_state"
}

Server pong message

type: "pong"

payload: object

Reply to a ping, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ping themselves echo ts; the fallback handler in :class:registry.wma.sdk.Session replies with client_ts. Clients read whichever is present. server_ts is the (unsynchronized) runner clock; it only shows how the round trip divides once the client has an offset estimate.

tsnumber | null

Default: null

server_tsnumber | null

Default: null

client_tsnumber | null

Default: null

typestring* required

Constant: "pong"

{
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

typestring* required

Constant: "prompt_applied"

{
  "type": "prompt_applied"
}

Server reset_applied message

type: "reset_applied"

payload: object

gameobject | null

Default: null

Must match at least one of the following:

GameInfo

object

actionsstring[]* required

Array item

string

thumbnail_urlstring | null

Default: null

descriptionstring* required

instructionsstring[]* required

Array item

string

promptstring* required

idstring* required

controlsobject[]* required

Array item

object

labelstring* required

holdboolean

Default: true

keystring* required

commandstring | null

Semantic command name.

Default: null

titlestring* required

Alternative 2

null

typestring* required

Constant: "reset_applied"

{
  "type": "reset_applied"
}

Server session_info message

type: "session_info"

payload: object

sessionstring* required

Values: "video", "game"

infoobject* required

typestring* required

Constant: "session_info"

{
  "session": "video",
  "info": {},
  "type": "session_info"
}

Server stats message

type: "stats"

payload: object

realtime_factornumber* required

Generated frames per second / target fps.

generation_fpsnumber* required

queue_framesinteger* required

chunkinteger* required

pipelinedboolean | null

The next chunk renders while this one generates (throughput mode).

Default: null

generate_msnumber | null

Model time per chunk (moving average).

Default: null

render_msnumber | null

Game render time per chunk (moving average).

Default: null

input_latency_msnumber | null

Game control message arrival to the handoff of the first frame it changes, median.

Default: null

chunk_msnumber* required

latency_msnumber | null

Input or control capture to frame handoff, median.

Default: null

loop_lag_msnumber | null

Server event-loop lag, p95 over the stats window.

Default: null

prompt_msnumber | null

Per-chunk prompt update time with dynamic_prompt, unchanged chunks included (moving average).

Default: null

billable_unitsnumber | null

Seconds of video billed so far this session (queued for playback); the final total settles when the session closes.

Default: null

typestring* required

Constant: "stats"

{
  "realtime_factor": 0,
  "generation_fps": 0,
  "queue_frames": 0,
  "chunk": 0,
  "chunk_ms": 0,
  "type": "stats"
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

The model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.

typestring* required

Constant: "stream_exhausted"

{
  "type": "stream_exhausted"
}

vast/eden-live-5b-game

Table of contents

Table of contents

vast/eden-live-5b-game

Table of contents

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

Session flow#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Game action

Controls

Choose game

Client input message

Client keys message

Client ping message

Update prompt

Restart

End session

Server messages#

Server configured message

Server controls_state message

Server error message

Server game_state message

Server pong message

Server prompt_applied message

Server reset_applied message

Server session_info message

Server stats message

Server stream_exhausted message

Related Models