Lingbot World 2 fal Worlds API Docs

This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.

import { createFalClient } from "@fal-ai/client"; import { wma } from "@fal-ai/client/realtime"; const fal = createFalClient({ proxyUrl: "/api/fal/proxy" }); const session = fal.realtime.open(wma("robbyant/lingbot-world-2"), { receive: [ "video" ], onMedia: (stream) => { const element = document.querySelector("video"); if (element) element.srcObject = stream; }, onData: (raw) => { const message = JSON.parse(raw); console.log(message); }, onState: (state) => console.log("realtime:", state), onError: (error) => console.error(error), }); // Consult the session flow and message reference before sending control messages. // Later, release the peer connection and network resources: // await session.close();

Session flow#

This world starts automatically. Wait for the first video frame before using the movement controls.

Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.

Send configure and wait for configured before sending another world update. Setup is optional and can be used once per session.
Send reset and wait for reset_applied before sending another world update.
Send prompt and wait for prompt_applied before sending another world update.

World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "height": 960,
  "frameRate": 48,
  "width": 1664
}

Client messages#

Movement

type: "commands"

payload: object

Semantic command-state report.

active contains commands currently held by the user and activated contains edge-triggered commands since the previous report. A browser may map a keyboard, gamepad, touch control, or agent action to these names.

activestring[]

maxItems: 17

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right", "quick_action_1", "quick_action_2", "quick_action_3", "quick_action_4", "quick_action_5", "quick_action_6", "quick_action_7", "quick_action_8", "quick_action_9"

activatedstring[]

maxItems: 17

Array item

string

typestring* required

Constant: "commands"

Example

{
  "active": [
    "forward"
  ],
  "activated": [],
  "type": "commands"
}

Customize world

type: "configure"

payload: object

Optionally replace the automatic default world, once per connection.

Await configured before submitting more controls. Use reset for subsequent restarts. Image URL takes precedence over the preset. Safety, prompt expansion, and the initial quick actions apply to the whole session.

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

enable_prompt_expansionboolean

Rewrite world descriptions, updates, and quick actions into detailed interactive prompts with an LLM.

Default: true

enable_safety_checkerboolean

Screen prompts and images with standard thresholds. When disabled, only illegal content is blocked.

Default: true

quick_actionsobject[]

Up to nine actions, each on its own number key.

maxItems: 9

Array item

object

One action performed while its number key is held.

keyinteger* required

Number key (1-9) that performs this action while held.

minimum: 1 · maximum: 9

titlestring* required

Label shown next to the action's button.

At least 1 character · At most 40 characters

descriptionstring* required

What happens while the key is held; combined with the world description.

At least 1 character · At most 1000 characters

typestring* required

Constant: "configure"

Example

A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views.

{
  "quick_actions": [],
  "enable_prompt_expansion": true,
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "type": "configure",
  "enable_safety_checker": true,
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}

Client keys message

type: "keys"

payload: object

Legacy physical-key report retained for existing WMA clients.

pressedstring[]

maxItems: 32

Array item

string

activatedstring[]

maxItems: 32

Array item

string

typestring* required

Constant: "keys"

{
  "type": "keys"
}

Client ping message

type: "ping"

payload: object

Latency probe; the server echoes ts back in a pong payload.

typestring* required

Constant: "ping"

tsnumber | null

Default: null

{
  "type": "ping"
}

Update world

type: "prompt"

payload: object

promptstring* required

Change the scene's description while continuing this world.

At least 1 character · At most 2000 characters

typestring* required

Constant: "prompt"

Example

{
  "type": "prompt",
  "prompt": "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds. Gentle ripples reflect the tree and sky."
}

Quick actions

type: "quick_actions"

payload: object

Replace the whole quick-action mapping; keys left out are un-mapped.

An action changes generation once quick_actions_state reports it ready.

typestring* required

Constant: "quick_actions"

actionsobject[]* required

Up to nine actions, each on its own number key.

maxItems: 9

Array item

object

One action performed while its number key is held.

keyinteger* required

Number key (1-9) that performs this action while held.

minimum: 1 · maximum: 9

titlestring* required

Label shown next to the action's button.

At least 1 character · At most 40 characters

descriptionstring* required

What happens while the key is held; combined with the world description.

At least 1 character · At most 1000 characters

Example

Map dragon-rider actions to keys 1-3.

{
  "type": "quick_actions",
  "actions": [
    {
      "title": "Breathe fire",
      "description": "The dragon rears its head and breathes a roaring torrent of fire ahead.",
      "key": 1
    },
    {
      "title": "Roar",
      "description": "The dragon throws its head back and roars, scattering birds from the canopy.",
      "key": 2
    },
    {
      "title": "Beat wings",
      "description": "The dragon beats its huge wings, whipping up a storm of leaves from the trees below.",
      "key": 3
    }
  ]
}

Restart world

type: "reset"

payload: object

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "reset"

Example

{
  "type": "reset",
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}

End session

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server configured message

type: "configured"

payload: object

enable_prompt_expansionboolean* required

typestring* required

Constant: "configured"

enable_safety_checkerboolean* required

Whether standard screening is in effect for this session.

promptstring* required

World description now in effect, after any prompt expansion.

{
  "enable_prompt_expansion": true,
  "type": "configured",
  "enable_safety_checker": true,
  "prompt": "string"
}

Server error message

type: "error"

payload: object

typestring* required

Constant: "error"

retryableboolean | null

Whether retrying unchanged input may succeed; absent if unspecified.

Default: null

errorstring* required

{
  "type": "error",
  "error": "string"
}

Server pong message

type: "pong"

payload: object

Reply to a ping, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ping themselves echo ts; the fallback handler in :class:registry.wma.sdk.Session replies with client_ts. Clients read whichever is present. server_ts is the (unsynchronized) runner clock; it only shows how the round trip divides once the client has an offset estimate.

typestring* required

Constant: "pong"

server_tsnumber | null

Default: null

client_tsnumber | null

Default: null

tsnumber | null

Default: null

{
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

typestring* required

Constant: "prompt_applied"

promptstring* required

World description now in effect, after any prompt expansion.

{
  "type": "prompt_applied",
  "prompt": "string"
}

Server quick_actions_state message

type: "quick_actions_state"

payload: object

The quick-action mapping and which actions are ready to use.

typestring* required

Constant: "quick_actions_state"

actionsobject[]* required

Array item

object

titlestring* required

readyboolean* required

Holding the key changes generation.

keyinteger* required

promptstring | null

Prompt attended while the key is held.

Default: null

{
  "type": "quick_actions_state",
  "actions": []
}

Server reset_applied message

type: "reset_applied"

payload: object

typestring* required

Constant: "reset_applied"

promptstring* required

World description now in effect, after any prompt expansion.

{
  "type": "reset_applied",
  "prompt": "string"
}

Server session_info message

type: "session_info"

payload: object

model_variantstring

Constant: "14b-causal-fast"

model_revisionstring* required

max_blocksinteger* required

source_revisionstring

Default: "1895d300d8ac936401689b26389f51cbd36530eb"

pixel_frames_per_blockinteger

Default: 12

model_idstring* required

max_quick_actionsinteger

Default: 9

default_promptstring* required

first_block_pixel_framesinteger

Default: 9

presetsobject[]* required

Array item

object

namestring* required

promptstring* required

typestring* required

Constant: "session_info"

frames_per_secondnumber* required

quick_action_keysstring[]

Array item

string

unsupported_featuresstring[]

Array item

string

resolutionobject* required

heightinteger* required

widthinteger* required

key_orderstring[]* required

Array item

string

conflict_groupsstring[][]* required

Array item

string[]

Array item

string

supported_controlsstring[]

Array item

string

turnboolean* required

generation_speed_verifiedboolean

Default: false

turn_statusstring* required

frames_per_blockinteger* required

{
  "model_variant": "14b-causal-fast",
  "model_revision": "string",
  "max_blocks": 0,
  "model_id": "string",
  "default_prompt": "string",
  "presets": [],
  "type": "session_info",
  "frames_per_second": 0,
  "resolution": {
    "height": 0,
    "width": 0
  },
  "key_order": [],
  "conflict_groups": [],
  "turn": true,
  "turn_status": "string",
  "frames_per_block": 0
}

Server stats message

type: "stats"

payload: object

queue_age_ms_p50number* required

native_timingobject | null

Default: null

Must match at least one of the following:

Alternative 1

object

Additional properties: number | integer

number | integer

Alternative 2

null

queue_age_ms_maxnumber* required

generation_fpsnumber* required

typestring* required

Constant: "stats"

block_secondsnumber* required

gpu_wait_fracnumber* required

pace_sleep_ms_avgnumber* required

block_indexinteger* required

queue_depthinteger* required

{
  "queue_age_ms_p50": 0,
  "queue_age_ms_max": 0,
  "generation_fps": 0,
  "type": "stats",
  "block_seconds": 0,
  "gpu_wait_frac": 0,
  "pace_sleep_ms_avg": 0,
  "block_index": 0,
  "queue_depth": 0
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

The model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.

typestring* required

Constant: "stream_exhausted"

{
  "type": "stream_exhausted"
}

robbyant/lingbot-world-2

Table of contents

Table of contents

robbyant/lingbot-world-2

Table of contents

Table of contents

1. Calling the API

2. Authentication

3. Schema

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

Session flow#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Movement

Customize world

Client keys message

Client ping message

Update world

Quick actions

Restart world

End session

Server messages#

Server configured message

Server error message

Server pong message

Server prompt_applied message

Server quick_actions_state message

Server reset_applied message

Server session_info message

Server stats message

Server stream_exhausted message