Abot World 0 API Docs

This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.

import { createFalClient } from "@fal-ai/client"; import { wma } from "@fal-ai/client/realtime"; const fal = createFalClient({ proxyUrl: "/api/fal/proxy" }); const session = fal.realtime.open(wma("fal-ai/abot-world-0"), { receive: [ "video" ], onMedia: (stream) => { const element = document.querySelector("video"); if (element) element.srcObject = stream; }, onData: (raw) => { const message = JSON.parse(raw); console.log(message); }, onState: (state) => console.log("realtime:", state), onError: (error) => console.error(error), }); // Consult the session flow and message reference before sending control messages. // Later, release the peer connection and network resources: // await session.close();

Session flow#

This world starts automatically. Wait for the first video frame before using the movement controls.

Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.

Send configure and wait for configured before sending another world update. Setup is optional and can be used once per session.
Send reset and wait for reset_applied before sending another world update.
Send prompt and wait for prompt_applied before sending another world update.

World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "height": 704,
  "width": 1280,
  "frameRate": 16
}

Client messages#

Movement

type: "commands"

payload: object

Semantic command-state report.

active contains commands currently held by the user and activated contains edge-triggered commands since the previous report. A browser may map a keyboard, gamepad, touch control, or agent action to these names.

typestring* required

Constant: "commands"

activatedstring[]

maxItems: 8

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"

activestring[]

maxItems: 8

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"

Example

{
  "type": "commands",
  "activated": [],
  "active": [
    "forward"
  ]
}

Customize world

type: "configure"

payload: object

Optionally replace the automatic default world, once per connection.

Await configured before submitting more controls. Use reset for subsequent restarts. Image URL takes precedence over the preset.

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "configure"

Example

Explore sunlit temple ruins, towering columns, and paths reclaimed by vines.

{
  "type": "configure",
  "preset": "example"
}

Client keys message

type: "keys"

payload: object

Legacy physical-key report retained for existing WMA clients.

typestring* required

Constant: "keys"

activatedstring[]

maxItems: 32

Array item

string

pressedstring[]

maxItems: 32

Array item

string

{
  "type": "keys"
}

Client ping message

type: "ping"

payload: object

Latency probe; the server echoes ts back in a pong payload.

typestring* required

Constant: "ping"

tsnumber | null

Default: null

{
  "type": "ping"
}

Update world

type: "prompt"

payload: object

promptstring* required

Change the scene's description while continuing this world.

At least 1 character · At most 2000 characters

typestring* required

Constant: "prompt"

Example

{
  "type": "prompt",
  "prompt": "A realistic outdoor world scene with a navigable path, natural lighting, detailed ground texture, and stable forward motion."
}

Restart world

type: "reset"

payload: object

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "reset"

Example

Explore sunlit temple ruins, towering columns, and paths reclaimed by vines.

{
  "type": "reset",
  "preset": "example"
}

End session

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server configured message

type: "configured"

payload: object

typestring* required

Constant: "configured"

{
  "type": "configured"
}

Server error message

type: "error"

payload: object

typestring* required

Constant: "error"

errorstring* required

retryableboolean | null

Whether retrying unchanged input may succeed; absent if unspecified.

Default: null

{
  "type": "error",
  "error": "string"
}

Server pong message

type: "pong"

payload: object

Reply to a ping, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ping themselves echo ts; the fallback handler in :class:registry.wma.sdk.Session replies with client_ts. Clients read whichever is present. server_ts is the (unsynchronized) runner clock; it only shows how the round trip divides once the client has an offset estimate.

typestring* required

Constant: "pong"

client_tsnumber | null

Default: null

server_tsnumber | null

Default: null

tsnumber | null

Default: null

{
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

typestring* required

Constant: "prompt_applied"

{
  "type": "prompt_applied"
}

Server reset_applied message

type: "reset_applied"

payload: object

typestring* required

Constant: "reset_applied"

{
  "type": "reset_applied"
}

Server session_info message

type: "session_info"

payload: object

key_orderstring[]* required

Array item

string

frames_per_blockinteger* required

presetsobject[]* required

Array item

object

promptstring* required

namestring* required

turn_statusstring* required

model_revisionstring* required

conflict_groupsstring[][]* required

Array item

string[]

Array item

string

model_idstring* required

max_blocksinteger* required

typestring* required

Constant: "session_info"

resolutionobject* required

heightinteger* required

widthinteger* required

frames_per_secondnumber* required

default_promptstring* required

turnboolean* required

{
  "key_order": [],
  "frames_per_block": 0,
  "presets": [],
  "turn_status": "string",
  "model_revision": "string",
  "conflict_groups": [],
  "model_id": "string",
  "max_blocks": 0,
  "type": "session_info",
  "resolution": {
    "height": 0,
    "width": 0
  },
  "frames_per_second": 0,
  "default_prompt": "string",
  "turn": true
}

Server stats message

type: "stats"

payload: object

queue_age_ms_p50number* required

gpu_wait_fracnumber* required

queue_depthinteger* required

generation_fpsnumber* required

block_indexinteger* required

block_secondsnumber* required

typestring* required

Constant: "stats"

queue_age_ms_maxnumber* required

pace_sleep_ms_avgnumber* required

{
  "queue_age_ms_p50": 0,
  "gpu_wait_frac": 0,
  "queue_depth": 0,
  "generation_fps": 0,
  "block_index": 0,
  "block_seconds": 0,
  "type": "stats",
  "queue_age_ms_max": 0,
  "pace_sleep_ms_avg": 0
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

The model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.

typestring* required

Constant: "stream_exhausted"

{
  "type": "stream_exhausted"
}

fal-ai/abot-world-0/start-session

Table of contents

Table of contents

fal-ai/abot-world-0/start-session

Table of contents

Table of contents

1. Calling the API

2. Authentication

3. Schema

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

Session flow#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Movement

Customize world

Client keys message

Client ping message

Update world

Restart world

End session

Server messages#

Server configured message

Server error message

Server pong message

Server prompt_applied message

Server reset_applied message

Server session_info message

Server stats message

Server stream_exhausted message