Claude

Lingbot World 2 fal Worlds API Docs | fal

About

Start Session

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install @fal-ai/client@alpha @fal-ai/server-proxy@alpha

Experimental realtime API

fal.realtime.open is currently available on the alpha release and may change in a minor version.

Set up the server proxy#

Keep FAL_KEY on your server and expose an authenticated proxy route to your browser. For a Next.js App Router application, create app/api/fal/proxy/route.ts:

import { route } from "@fal-ai/server-proxy/nextjs";

export const { GET, POST, PUT } = route;

Set FAL_KEY in the server environment, and protect this route with your application's authentication before deploying it.

Real-time via WebRTC#

This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.

import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";

const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });

const session = fal.realtime.open(wma("fal-ai/lingbot-world-2"), {
  receive: [
    "video"
  ],
  onMedia: (stream) => {
      const element = document.querySelector("video");
      if (element) element.srcObject = stream;
    },
  onData: (raw) => {
      const message = JSON.parse(raw);
      console.log(message);
    },
  onState: (state) => console.log("realtime:", state),
  onError: (error) => console.error(error),
});


// Consult the session flow and message reference before sending control messages.

// Later, release the peer connection and network resources:
// await session.close();

Session flow#

This world starts automatically. Wait for the first video frame before using the movement controls.

Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.

Send configure and wait for configured before sending another world update. Setup is optional and can be used once per session.
Send reset and wait for reset_applied before sending another world update.
Send prompt and wait for prompt_applied before sending another world update.

World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "height": 960,
  "width": 1664,
  "frameRate": 48
}

Client messages#

Movement

type: "commands"

payload: object

Semantic command-state report.

active contains commands currently held by the user and activated contains edge-triggered commands since the previous report. A browser may map a keyboard, gamepad, touch control, or agent action to these names.

activestring[]

maxItems: 17

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right", "quick_action_1", "quick_action_2", "quick_action_3", "quick_action_4", "quick_action_5", "quick_action_6", "quick_action_7", "quick_action_8", "quick_action_9"

activatedstring[]

maxItems: 17

Array item

string

typestring* required

Constant: "commands"

Example

{
  "active": [
    "forward"
  ],
  "activated": [],
  "type": "commands"
}

Customize world

type: "configure"

payload: object

Optionally replace the automatic default world, once per connection.

Await configured before submitting more controls. Use reset for subsequent restarts. Image URL takes precedence over the preset. Safety, prompt expansion, and the initial quick actions apply to the whole session.

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

enable_prompt_expansionboolean

Rewrite world descriptions, updates, and quick actions into detailed interactive prompts with an LLM.

Default: true

enable_safety_checkerboolean

Screen prompts and images with standard thresholds. When disabled, only illegal content is blocked.

Default: true

quick_actionsobject[]

Up to nine actions, each on its own number key.

maxItems: 9

Array item

object

One action performed while its number key is held.

keyinteger* required

Number key (1-9) that performs this action while held.

minimum: 1 · maximum: 9

titlestring* required

Label shown next to the action's button.

At least 1 character · At most 40 characters

descriptionstring* required

What happens while the key is held; combined with the world description.

At least 1 character · At most 1000 characters

typestring* required

Constant: "configure"

Example

A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views.

{
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "enable_prompt_expansion": true,
  "enable_safety_checker": true,
  "quick_actions": [],
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways.",
  "type": "configure"
}

Client keys message

type: "keys"

payload: object

Legacy physical-key report retained for existing WMA clients.

pressedstring[]

maxItems: 32

Array item

string

activatedstring[]

maxItems: 32

Array item

string

typestring* required

Constant: "keys"

{
  "type": "keys"
}

Client ping message

type: "ping"

payload: object

Latency probe; the server echoes ts back in a pong payload.

tsnumber | null

Default: null

typestring* required

Constant: "ping"

{
  "type": "ping"
}

Update world

type: "prompt"

payload: object

promptstring* required

Change the scene's description while continuing this world.

At least 1 character · At most 2000 characters

typestring* required

Constant: "prompt"

Example

{
  "prompt": "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds. Gentle ripples reflect the tree and sky.",
  "type": "prompt"
}

Quick actions

type: "quick_actions"

payload: object

Replace the whole quick-action mapping; keys left out are un-mapped.

An action changes generation once quick_actions_state reports it ready.

actionsobject[]* required

Up to nine actions, each on its own number key.

maxItems: 9

Array item

object

One action performed while its number key is held.

keyinteger* required

Number key (1-9) that performs this action while held.

minimum: 1 · maximum: 9

titlestring* required

Label shown next to the action's button.

At least 1 character · At most 40 characters

descriptionstring* required

What happens while the key is held; combined with the world description.

At least 1 character · At most 1000 characters

typestring* required

Constant: "quick_actions"

Example

Map dragon-rider actions to keys 1-3.

{
  "actions": [
    {
      "title": "Breathe fire",
      "description": "The dragon rears its head and breathes a roaring torrent of fire ahead.",
      "key": 1
    },
    {
      "title": "Roar",
      "description": "The dragon throws its head back and roars, scattering birds from the canopy.",
      "key": 2
    },
    {
      "title": "Beat wings",
      "description": "The dragon beats its huge wings, whipping up a storm of leaves from the trees below.",
      "key": 3
    }
  ],
  "type": "quick_actions"
}

Restart world

type: "reset"

payload: object

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "reset"

Example

{
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways.",
  "type": "reset"
}

End session

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server configured message

type: "configured"

payload: object

enable_prompt_expansionboolean* required

enable_safety_checkerboolean* required

Whether standard screening is in effect for this session.

promptstring* required

World description now in effect, after any prompt expansion.

typestring* required

Constant: "configured"

{
  "enable_prompt_expansion": true,
  "enable_safety_checker": true,
  "prompt": "string",
  "type": "configured"
}

Server error message

type: "error"

payload: object

errorstring* required

retryableboolean | null

Whether retrying unchanged input may succeed; absent if unspecified.

Default: null

typestring* required

Constant: "error"

{
  "error": "string",
  "type": "error"
}

Server pong message

type: "pong"

payload: object

Reply to a ping, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ping themselves echo ts; the fallback handler in :class:registry.wma.sdk.Session replies with client_ts. Clients read whichever is present. server_ts is the (unsynchronized) runner clock; it only shows how the round trip divides once the client has an offset estimate.

server_tsnumber | null

Default: null

tsnumber | null

Default: null

client_tsnumber | null

Default: null

typestring* required

Constant: "pong"

{
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

promptstring* required

World description now in effect, after any prompt expansion.

typestring* required

Constant: "prompt_applied"

{
  "prompt": "string",
  "type": "prompt_applied"
}

Server quick_actions_state message

type: "quick_actions_state"

payload: object

The quick-action mapping and which actions are ready to use.

actionsobject[]* required

Array item

object

titlestring* required

promptstring | null

Prompt attended while the key is held.

Default: null

keyinteger* required

readyboolean* required

Holding the key changes generation.

typestring* required

Constant: "quick_actions_state"

{
  "actions": [],
  "type": "quick_actions_state"
}

Server reset_applied message

type: "reset_applied"

payload: object

promptstring* required

World description now in effect, after any prompt expansion.

typestring* required

Constant: "reset_applied"

{
  "prompt": "string",
  "type": "reset_applied"
}

Server session_info message

type: "session_info"

payload: object

frames_per_secondnumber* required

supported_controlsstring[]

Array item

string

unsupported_featuresstring[]

Array item

string

key_orderstring[]* required

Array item

string

presetsobject[]* required

Array item

object

namestring* required

promptstring* required

turnboolean* required

turn_statusstring* required

quick_action_keysstring[]

Array item

string

model_revisionstring* required

typestring* required

Constant: "session_info"

conflict_groupsstring[][]* required

Array item

string[]

Array item

string

max_quick_actionsinteger

Default: 9

generation_speed_verifiedboolean

Default: false

default_promptstring* required

model_variantstring

Constant: "14b-causal-fast"

resolutionobject* required

heightinteger* required

widthinteger* required

max_blocksinteger* required

pixel_frames_per_blockinteger

Default: 12

model_idstring* required

source_revisionstring

Default: "1895d300d8ac936401689b26389f51cbd36530eb"

frames_per_blockinteger* required

first_block_pixel_framesinteger

Default: 9

{
  "frames_per_second": 0,
  "key_order": [],
  "presets": [],
  "turn": true,
  "turn_status": "string",
  "model_revision": "string",
  "type": "session_info",
  "conflict_groups": [],
  "default_prompt": "string",
  "model_variant": "14b-causal-fast",
  "resolution": {
    "height": 0,
    "width": 0
  },
  "max_blocks": 0,
  "model_id": "string",
  "frames_per_block": 0
}

Server stats message

type: "stats"

payload: object

queue_depthinteger* required

gpu_wait_fracnumber* required

queue_age_ms_p50number* required

native_timingobject | null

Default: null

Must match at least one of the following:

Alternative 1

object

Additional properties: number | integer

number | integer

Alternative 2

null

generation_fpsnumber* required

pace_sleep_ms_avgnumber* required

block_secondsnumber* required

queue_age_ms_maxnumber* required

block_indexinteger* required

typestring* required

Constant: "stats"

{
  "queue_depth": 0,
  "gpu_wait_frac": 0,
  "queue_age_ms_p50": 0,
  "generation_fps": 0,
  "pace_sleep_ms_avg": 0,
  "block_seconds": 0,
  "queue_age_ms_max": 0,
  "block_index": 0,
  "type": "stats"
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

The model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.

typestring* required

Constant: "stream_exhausted"

{
  "type": "stream_exhausted"
}

fal-ai/lingbot-world-2/start-session

Table of contents

Table of contents

fal-ai/lingbot-world-2/start-session

Table of contents

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

Session flow#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Movement

Customize world

Client keys message

Client ping message

Update world

Quick actions

Restart world

End session

Server messages#

Server configured message

Server error message

Server pong message

Server prompt_applied message

Server quick_actions_state message

Server reset_applied message

Server session_info message

Server stats message

Server stream_exhausted message