# Lingbot World 2

> Explore interactive worlds in real time with movement and camera controls.


## Overview

- **Endpoint**: `https://fal.run/fal-ai/lingbot-world-2/start-session`
- **Model ID**: `fal-ai/lingbot-world-2/start-session`
- **Category**: worlds
- **Kind**: inference
**Description**: Lingbot World 2 by Robbyant turns a starting image into an interactive world. Explore the scene in real time with movement and camera controls in the fal Worlds playground, or integrate the realtime session API.

**Tags**: worlds, world-model, realtime, interactive, walkable, explorable, lingbot



## Pricing

- **Price**: $0.00125 per compute seconds

For more details, see [fal.ai pricing](https://fal.ai/pricing).

## Realtime API (WMA/WebRTC)

This endpoint publishes a WMA WebRTC contract. Open it with `fal.realtime.open`; do **not** call the session endpoint with `fal.run`, `fal.subscribe`, or the Queue API.

The browser negotiates through fal and then exchanges media with the model over WebRTC.

Client contract for the WebRTC session created by the linked OpenAPI operation.


### Install the experimental client

```bash
npm install @fal-ai/client@alpha @fal-ai/server-proxy@alpha
```

### Authentication and server proxy

Keep `FAL_KEY` on your server. Expose an authenticated proxy route to the browser. In a Next.js App Router application, create `app/api/fal/proxy/route.ts`:

```typescript
import { route } from "@fal-ai/server-proxy/nextjs";

export const { GET, POST, PUT } = route;
```

Protect that route with your application's authentication before deploying it.

### Open a browser session

```typescript
import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";

const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });

const session = fal.realtime.open(wma("fal-ai/lingbot-world-2"), {
  receive: [
    "video"
  ],
  onMedia: (stream) => {
      const element = document.querySelector("video");
      if (element) element.srcObject = stream;
    },
  onData: (raw) => {
      const message = JSON.parse(raw);
      console.log(message);
    },
  onState: (state) => console.log("realtime:", state),
  onError: (error) => console.error(error),
});


// Consult the session flow and message reference before sending control messages.

// Later, release the peer connection and network resources:
// await session.close();
```

## Session flow

This world starts automatically. Wait for the first video frame before using the movement controls.

Send ` commands ` with held controls in ` active ` and newly pressed controls in ` activated `. Send empty arrays when controls are released or focus leaves the world.

- Send ` configure ` and wait for ` configured ` before sending another world update. Setup is optional and can be used once per session.
- Send ` reset ` and wait for ` reset_applied ` before sending another world update.
- Send ` prompt ` and wait for ` prompt_applied ` before sending another world update.

World updates run one at a time and must not be replayed automatically. A ` error ` event is diagnostic and does not identify a failed update; read ` /error ` for details. The session ends on ` stream_exhausted `.


## Media contract

Tracks are described from the browser's perspective.

### Browser sends to the model

None.

### Browser receives from the model

- `video` — optional
  - Settings: `{"height":960,"frameRate":48,"width":1664}`

## Client-to-model control messages

### Movement

- **Wire type**: `commands`
- **Payload**: `object`

Semantic command-state report.

``active`` contains commands currently held by the user and ``activated``
contains edge-triggered commands since the previous report. A browser may
map a keyboard, gamepad, touch control, or agent action to these names.

- **`active`** (`string[]`, _optional_)
  - Constraints: maxItems=17
  - Array items: `string`
    - Values: `"forward"`, `"left"`, `"backward"`, `"right"`, `"look_up"`, `"look_left"`, `"look_down"`, `"look_right"`, `"quick_action_1"`, `"quick_action_2"`, `"quick_action_3"`, `"quick_action_4"`, `"quick_action_5"`, `"quick_action_6"`, `"quick_action_7"`, `"quick_action_8"`, `"quick_action_9"`
- **`activated`** (`string[]`, _optional_)
  - Constraints: maxItems=17
  - Array items: `string`
    - Values: `"forward"`, `"left"`, `"backward"`, `"right"`, `"look_up"`, `"look_left"`, `"look_down"`, `"look_right"`, `"quick_action_1"`, `"quick_action_2"`, `"quick_action_3"`, `"quick_action_4"`, `"quick_action_5"`, `"quick_action_6"`, `"quick_action_7"`, `"quick_action_8"`, `"quick_action_9"`
- **`type`** (`string`, _required_)
  - Constant: `"commands"`

**Example**

```json
{
  "active": [
    "forward"
  ],
  "activated": [],
  "type": "commands"
}
```

### Customize world

- **Wire type**: `configure`
- **Payload**: `object`

Optionally replace the automatic default world, once per connection.

Await ``configured`` before submitting more controls. Use ``reset`` for
subsequent restarts. Image URL takes precedence over the preset. Safety,
prompt expansion, and the initial quick actions apply to the whole session.

- **`preset`** (`string | null`, _optional_)
  Choose a built-in world, or provide a starting image below.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=100
    2. `null`
- **`prompt`** (`string | null`, _optional_)
  Optionally replace the starting world's description.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=2000
    2. `null`
- **`image_url`** (`string | null`, _optional_)
  Use your own image instead of a built-in world.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=4096
    2. `null`
- **`enable_prompt_expansion`** (`boolean`, _optional_)
  Rewrite world descriptions, updates, and quick actions into detailed interactive prompts with an LLM.
  - Default: `true`
- **`enable_safety_checker`** (`boolean`, _optional_)
  Screen prompts and images with standard thresholds. When disabled, only illegal content is blocked.
  - Default: `true`
- **`quick_actions`** (`object[]`, _optional_)
  Up to nine actions, each on its own number key.
  - Constraints: maxItems=9
  - Array items: `object`
    One action performed while its number key is held.
    - **`key`** (`integer`, _required_)
      Number key (1-9) that performs this action while held.
      - Constraints: minimum=1, maximum=9
    - **`title`** (`string`, _required_)
      Label shown next to the action's button.
      - Constraints: minLength=1, maxLength=40
    - **`description`** (`string`, _required_)
      What happens while the key is held; combined with the world description.
      - Constraints: minLength=1, maxLength=1000
- **`type`** (`string`, _required_)
  - Constant: `"configure"`

**Example**

```json
{
  "quick_actions": [],
  "enable_prompt_expansion": true,
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "type": "configure",
  "enable_safety_checker": true,
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}
```

### Client keys message

- **Wire type**: `keys`
- **Payload**: `object`

Legacy physical-key report retained for existing WMA clients.

- **`pressed`** (`string[]`, _optional_)
  - Constraints: maxItems=32
  - Array items: `string`
- **`activated`** (`string[]`, _optional_)
  - Constraints: maxItems=32
  - Array items: `string`
- **`type`** (`string`, _required_)
  - Constant: `"keys"`

**Example**

```json
{
  "type": "keys"
}
```

### Client ping message

- **Wire type**: `ping`
- **Payload**: `object`

Latency probe; the server echoes ``ts`` back in a ``pong`` payload.

- **`type`** (`string`, _required_)
  - Constant: `"ping"`
- **`ts`** (`number | null`, _optional_)
  - Default: `null`

**Example**

```json
{
  "type": "ping"
}
```

### Update world

- **Wire type**: `prompt`
- **Payload**: `object`

- **`prompt`** (`string`, _required_)
  Change the scene's description while continuing this world.
  - Constraints: minLength=1, maxLength=2000
- **`type`** (`string`, _required_)
  - Constant: `"prompt"`

**Example**

```json
{
  "type": "prompt",
  "prompt": "A serene lakeside scene with a lone tree standing in calm water, surrounded by distant snow-capped mountains under a bright blue sky with drifting white clouds. Gentle ripples reflect the tree and sky."
}
```

### Quick actions

- **Wire type**: `quick_actions`
- **Payload**: `object`

Replace the whole quick-action mapping; keys left out are un-mapped.

An action changes generation once ``quick_actions_state`` reports it ready.

- **`type`** (`string`, _required_)
  - Constant: `"quick_actions"`
- **`actions`** (`object[]`, _required_)
  Up to nine actions, each on its own number key.
  - Constraints: maxItems=9
  - Array items: `object`
    One action performed while its number key is held.
    - **`key`** (`integer`, _required_)
      Number key (1-9) that performs this action while held.
      - Constraints: minimum=1, maximum=9
    - **`title`** (`string`, _required_)
      Label shown next to the action's button.
      - Constraints: minLength=1, maxLength=40
    - **`description`** (`string`, _required_)
      What happens while the key is held; combined with the world description.
      - Constraints: minLength=1, maxLength=1000

**Example**

```json
{
  "type": "quick_actions",
  "actions": [
    {
      "title": "Breathe fire",
      "description": "The dragon rears its head and breathes a roaring torrent of fire ahead.",
      "key": 1
    },
    {
      "title": "Roar",
      "description": "The dragon throws its head back and roars, scattering birds from the canopy.",
      "key": 2
    },
    {
      "title": "Beat wings",
      "description": "The dragon beats its huge wings, whipping up a storm of leaves from the trees below.",
      "key": 3
    }
  ]
}
```

### Restart world

- **Wire type**: `reset`
- **Payload**: `object`

- **`preset`** (`string | null`, _optional_)
  Choose a built-in world, or provide a starting image below.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=100
    2. `null`
- **`prompt`** (`string | null`, _optional_)
  Optionally replace the starting world's description.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=2000
    2. `null`
- **`image_url`** (`string | null`, _optional_)
  Use your own image instead of a built-in world.
  - Default: `null`
  - anyOf alternatives:
    1. `string`
      - Constraints: maxLength=4096
    2. `null`
- **`type`** (`string`, _required_)
  - Constant: `"reset"`

**Example**

```json
{
  "type": "reset",
  "image_url": "https://v3b.fal.media/files/b/0aad61cc/2Yx7Ey7Pi2QblGQXyf4yY_XdmbNwp8.png",
  "prompt": "Photorealistic third-person exploration. Natural sunset light, believable stone and vegetation, true photographic materials. A spectacular Mediterranean cliff city above the sea. A broad level stone promenade between terraced gardens leads to an open lookout, with small groups of visitors beside the path, distant boats, warm limestone architecture and immense clear ocean views. Adult woman with warm brown skin, dark curly ponytail, burnt-orange windbreaker, ivory hiking trousers and dark boots. Third-person follow camera directly behind the character, complete body clearly visible at lower center, facing away along the open route. Keep the avatar visible and recognizable in every future view. Move smoothly forward together with the avatar; stable upright horizon, no cuts, no first-person conversion. Navigation-ready: minimum six-metre-wide unobstructed level route, floor visible to the vanishing point, all obstacles and other characters beside the path, open large turns, no stairs or narrow doorways."
}
```

### End session

- **Wire type**: `stop`
- **Payload**: `object`

- **`type`** (`string`, _required_)
  - Constant: `"stop"`
- Additional properties: not allowed

**Example**

```json
{
  "type": "stop"
}
```

## Model-to-client control messages

### Server configured message

- **Wire type**: `configured`
- **Payload**: `object`

- **`enable_prompt_expansion`** (`boolean`, _required_)
- **`type`** (`string`, _required_)
  - Constant: `"configured"`
- **`enable_safety_checker`** (`boolean`, _required_)
  Whether standard screening is in effect for this session.
- **`prompt`** (`string`, _required_)
  World description now in effect, after any prompt expansion.

**Example**

```json
{
  "enable_prompt_expansion": true,
  "type": "configured",
  "enable_safety_checker": true,
  "prompt": "string"
}
```

### Server error message

- **Wire type**: `error`
- **Payload**: `object`

- **`type`** (`string`, _required_)
  - Constant: `"error"`
- **`retryable`** (`boolean | null`, _optional_)
  Whether retrying unchanged input may succeed; absent if unspecified.
  - Default: `null`
- **`error`** (`string`, _required_)

**Example**

```json
{
  "type": "error",
  "error": "string"
}
```

### Server pong message

- **Wire type**: `pong`
- **Payload**: `object`

Reply to a ``ping``, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ``ping``
themselves echo ``ts``; the fallback handler in
:class:`registry.wma.sdk.Session` replies with ``client_ts``. Clients read
whichever is present. ``server_ts`` is the (unsynchronized) runner clock; it
only shows how the round trip divides once the client has an offset estimate.

- **`type`** (`string`, _required_)
  - Constant: `"pong"`
- **`server_ts`** (`number | null`, _optional_)
  - Default: `null`
- **`client_ts`** (`number | null`, _optional_)
  - Default: `null`
- **`ts`** (`number | null`, _optional_)
  - Default: `null`

**Example**

```json
{
  "type": "pong"
}
```

### Server prompt_applied message

- **Wire type**: `prompt_applied`
- **Payload**: `object`

- **`type`** (`string`, _required_)
  - Constant: `"prompt_applied"`
- **`prompt`** (`string`, _required_)
  World description now in effect, after any prompt expansion.

**Example**

```json
{
  "type": "prompt_applied",
  "prompt": "string"
}
```

### Server quick_actions_state message

- **Wire type**: `quick_actions_state`
- **Payload**: `object`

The quick-action mapping and which actions are ready to use.

- **`type`** (`string`, _required_)
  - Constant: `"quick_actions_state"`
- **`actions`** (`object[]`, _required_)
  - Array items: `object`
    - **`title`** (`string`, _required_)
    - **`ready`** (`boolean`, _required_)
      Holding the key changes generation.
    - **`key`** (`integer`, _required_)
    - **`prompt`** (`string | null`, _optional_)
      Prompt attended while the key is held.
      - Default: `null`

**Example**

```json
{
  "type": "quick_actions_state",
  "actions": []
}
```

### Server reset_applied message

- **Wire type**: `reset_applied`
- **Payload**: `object`

- **`type`** (`string`, _required_)
  - Constant: `"reset_applied"`
- **`prompt`** (`string`, _required_)
  World description now in effect, after any prompt expansion.

**Example**

```json
{
  "type": "reset_applied",
  "prompt": "string"
}
```

### Server session_info message

- **Wire type**: `session_info`
- **Payload**: `object`

- **`model_variant`** (`string`, _optional_)
  - Constant: `"14b-causal-fast"`
- **`model_revision`** (`string`, _required_)
- **`max_blocks`** (`integer`, _required_)
- **`source_revision`** (`string`, _optional_)
  - Default: `"1895d300d8ac936401689b26389f51cbd36530eb"`
- **`pixel_frames_per_block`** (`integer`, _optional_)
  - Default: `12`
- **`model_id`** (`string`, _required_)
- **`max_quick_actions`** (`integer`, _optional_)
  - Default: `9`
- **`default_prompt`** (`string`, _required_)
- **`first_block_pixel_frames`** (`integer`, _optional_)
  - Default: `9`
- **`presets`** (`object[]`, _required_)
  - Array items: `object`
    - **`name`** (`string`, _required_)
    - **`prompt`** (`string`, _required_)
- **`type`** (`string`, _required_)
  - Constant: `"session_info"`
- **`frames_per_second`** (`number`, _required_)
- **`quick_action_keys`** (`string[]`, _optional_)
  - Array items: `string`
- **`unsupported_features`** (`string[]`, _optional_)
  - Array items: `string`
- **`resolution`** (`object`, _required_)
  - **`height`** (`integer`, _required_)
  - **`width`** (`integer`, _required_)
- **`key_order`** (`string[]`, _required_)
  - Array items: `string`
- **`conflict_groups`** (`string[][]`, _required_)
  - Array items: `string[]`
    - Array items: `string`
- **`supported_controls`** (`string[]`, _optional_)
  - Array items: `string`
- **`turn`** (`boolean`, _required_)
- **`generation_speed_verified`** (`boolean`, _optional_)
  - Default: `false`
- **`turn_status`** (`string`, _required_)
- **`frames_per_block`** (`integer`, _required_)

**Example**

```json
{
  "model_variant": "14b-causal-fast",
  "model_revision": "string",
  "max_blocks": 0,
  "model_id": "string",
  "default_prompt": "string",
  "presets": [],
  "type": "session_info",
  "frames_per_second": 0,
  "resolution": {
    "height": 0,
    "width": 0
  },
  "key_order": [],
  "conflict_groups": [],
  "turn": true,
  "turn_status": "string",
  "frames_per_block": 0
}
```

### Server stats message

- **Wire type**: `stats`
- **Payload**: `object`

- **`queue_age_ms_p50`** (`number`, _required_)
- **`native_timing`** (`object | null`, _optional_)
  - Default: `null`
  - anyOf alternatives:
    1. `object`
      - Additional properties: number | integer
    2. `null`
- **`queue_age_ms_max`** (`number`, _required_)
- **`generation_fps`** (`number`, _required_)
- **`type`** (`string`, _required_)
  - Constant: `"stats"`
- **`block_seconds`** (`number`, _required_)
- **`gpu_wait_frac`** (`number`, _required_)
- **`pace_sleep_ms_avg`** (`number`, _required_)
- **`block_index`** (`integer`, _required_)
- **`queue_depth`** (`integer`, _required_)

**Example**

```json
{
  "queue_age_ms_p50": 0,
  "queue_age_ms_max": 0,
  "generation_fps": 0,
  "type": "stats",
  "block_seconds": 0,
  "gpu_wait_frac": 0,
  "pace_sleep_ms_avg": 0,
  "block_index": 0,
  "queue_depth": 0
}
```

### Server stream_exhausted message

- **Wire type**: `stream_exhausted`
- **Payload**: `object`

The model has no more frames to send and the current peer is closing.
Clients must negotiate a new session to continue.

- **`type`** (`string`, _required_)
  - Constant: `"stream_exhausted"`

**Example**

```json
{
  "type": "stream_exhausted"
}
```

## Additional Resources

### Documentation

- [Model Playground](https://fal.ai/models/fal-ai/lingbot-world-2/start-session)
- [API Documentation](https://fal.ai/models/fal-ai/lingbot-world-2/start-session/api)
- [OpenAPI Schema](https://fal.ai/api/openapi/queue/openapi.json?endpoint_id=fal-ai/lingbot-world-2/start-session)
- [AsyncAPI Contract](https://fal.ai/api/apps/fal-ai/lingbot-world-2/asyncapi.json)

### fal.ai Platform

- [Platform Documentation](https://fal.ai/docs/documentation)
- [Python Client](https://fal.ai/docs/api-reference/client-libraries/python)
- [JavaScript Client](https://fal.ai/docs/api-reference/client-libraries/javascript)

### Other agent-readable surfaces

This file covers one model. To find anything else:

- [Platform overview](https://fal.ai/llms.txt): Entry points and representative endpoint IDs
- [Documentation index](https://fal.ai/docs/llms.txt): Every documentation page
- [Full documentation text](https://fal.ai/docs/llms-full.txt): The whole documentation inlined
- Any other model: `https://fal.ai/models/<endpoint-id>/llms.txt`