fal-ai/abot-world-0/start-session

Inference

Private

Schema AsyncAPI

LLMs

Playground API

About

Start Session

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install @fal-ai/client@alpha @fal-ai/server-proxy@alpha

Experimental realtime API

fal.realtime.open is currently available on the alpha release and may change in a minor version.

Set up the server proxy#

Keep FAL_KEY on your server and expose an authenticated proxy route to your browser. For a Next.js App Router application, create app/api/fal/proxy/route.ts:

import { route } from "@fal-ai/server-proxy/nextjs";

export const { GET, POST, PUT } = route;

Set FAL_KEY in the server environment, and protect this route with your application's authentication before deploying it.

Real-time via WebRTC#

This deployment publishes a WMA WebRTC contract. The example uses fal.realtime.open; its media handlers, capture constraints, and control message are generated from the linked AsyncAPI document.

import { createFalClient } from "@fal-ai/client";
import { wma } from "@fal-ai/client/realtime";

const fal = createFalClient({ proxyUrl: "/api/fal/proxy" });

const session = fal.realtime.open(wma("fal-ai/abot-world-0"), {
  receive: [
    "video"
  ],
  onMedia: (stream) => {
      const element = document.querySelector("video");
      if (element) element.srcObject = stream;
    },
  onData: (raw) => {
      const message = JSON.parse(raw);
      console.log(message);
    },
  onState: (state) => console.log("realtime:", state),
  onError: (error) => console.error(error),
});


// Consult the session flow and message reference before sending control messages.

// Later, release the peer connection and network resources:
// await session.close();

Session flow#

This world starts automatically. Wait for the first video frame before using the movement controls.

Send commands with held controls in active and newly pressed controls in activated. Send empty arrays when controls are released or focus leaves the world.

Send configure and wait for configured before sending another world update. Setup is optional and can be used once per session.
Send reset and wait for reset_applied before sending another world update.
Send prompt and wait for prompt_applied before sending another world update.

World updates run one at a time and must not be replayed automatically. A error event is diagnostic and does not identify a failed update; read /error for details. The session ends on stream_exhausted.

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "width": 1280,
  "height": 704,
  "frameRate": 16
}

Client messages#

Movement

type: "commands"

payload: object

Semantic command-state report.

active contains commands currently held by the user and activated contains edge-triggered commands since the previous report. A browser may map a keyboard, gamepad, touch control, or agent action to these names.

activestring[]

maxItems: 8

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"

activatedstring[]

maxItems: 8

Array item

string

Values: "forward", "left", "backward", "right", "look_up", "look_left", "look_down", "look_right"

typestring* required

Constant: "commands"

Example

{
  "active": [
    "forward"
  ],
  "activated": [],
  "type": "commands"
}

Customize world

type: "configure"

payload: object

Optionally replace the automatic default world, once per connection.

Await configured before submitting more controls. Use reset for subsequent restarts. Image URL takes precedence over the preset.

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "configure"

Example

Explore sunlit temple ruins, towering columns, and paths reclaimed by vines.

{
  "preset": "example",
  "type": "configure"
}

Client keys message

type: "keys"

payload: object

Legacy physical-key report retained for existing WMA clients.

pressedstring[]

maxItems: 32

Array item

string

activatedstring[]

maxItems: 32

Array item

string

typestring* required

Constant: "keys"

{
  "type": "keys"
}

Client ping message

type: "ping"

payload: object

Latency probe; the server echoes ts back in a pong payload.

tsnumber | null

Default: null

typestring* required

Constant: "ping"

{
  "type": "ping"
}

Update world

type: "prompt"

payload: object

promptstring* required

Change the scene's description while continuing this world.

At least 1 character · At most 2000 characters

typestring* required

Constant: "prompt"

Example

{
  "prompt": "A realistic outdoor world scene with a navigable path, natural lighting, detailed ground texture, and stable forward motion.",
  "type": "prompt"
}

Restart world

type: "reset"

payload: object

presetstring | null

Choose a built-in world, or provide a starting image below.

Default: null

At most 100 characters when not null

promptstring | null

Optionally replace the starting world's description.

Default: null

At most 2000 characters when not null

image_urlstring | null

Use your own image instead of a built-in world.

Default: null

At most 4096 characters when not null

typestring* required

Constant: "reset"

Example

Explore sunlit temple ruins, towering columns, and paths reclaimed by vines.

{
  "preset": "example",
  "type": "reset"
}

End session

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server configured message

type: "configured"

payload: object

typestring* required

Constant: "configured"

{
  "type": "configured"
}

Server error message

type: "error"

payload: object

errorstring* required

retryableboolean | null

Whether retrying unchanged input may succeed; absent if unspecified.

Default: null

typestring* required

Constant: "error"

{
  "error": "string",
  "type": "error"
}

Server pong message

type: "pong"

payload: object

Reply to a ping, echoing the client's timestamp for RTT measurement.

The echoed timestamp has two wire spellings: apps that handle ping themselves echo ts; the fallback handler in :class:registry.wma.sdk.Session replies with client_ts. Clients read whichever is present. server_ts is the (unsynchronized) runner clock; it only shows how the round trip divides once the client has an offset estimate.

client_tsnumber | null

Default: null

tsnumber | null

Default: null

server_tsnumber | null

Default: null

typestring* required

Constant: "pong"

{
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

typestring* required

Constant: "prompt_applied"

{
  "type": "prompt_applied"
}

Server reset_applied message

type: "reset_applied"

payload: object

typestring* required

Constant: "reset_applied"

{
  "type": "reset_applied"
}

Server session_info message

type: "session_info"

payload: object

turnboolean* required

default_promptstring* required

resolutionobject* required

widthinteger* required

heightinteger* required

model_idstring* required

typestring* required

Constant: "session_info"

conflict_groupsstring[][]* required

Array item

string[]

Array item

string

model_revisionstring* required

presetsobject[]* required

Array item

object

namestring* required

promptstring* required

key_orderstring[]* required

Array item

string

turn_statusstring* required

frames_per_blockinteger* required

frames_per_secondnumber* required

max_blocksinteger* required

{
  "turn": true,
  "default_prompt": "string",
  "resolution": {
    "width": 0,
    "height": 0
  },
  "model_id": "string",
  "type": "session_info",
  "conflict_groups": [],
  "model_revision": "string",
  "presets": [],
  "key_order": [],
  "turn_status": "string",
  "frames_per_block": 0,
  "frames_per_second": 0,
  "max_blocks": 0
}

Server stats message

type: "stats"

payload: object

block_secondsnumber* required

queue_depthinteger* required

generation_fpsnumber* required

queue_age_ms_maxnumber* required

typestring* required

Constant: "stats"

gpu_wait_fracnumber* required

pace_sleep_ms_avgnumber* required

queue_age_ms_p50number* required

block_indexinteger* required

{
  "block_seconds": 0,
  "queue_depth": 0,
  "generation_fps": 0,
  "queue_age_ms_max": 0,
  "type": "stats",
  "gpu_wait_frac": 0,
  "pace_sleep_ms_avg": 0,
  "queue_age_ms_p50": 0,
  "block_index": 0
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

The model has no more frames to send and the current peer is closing. Clients must negotiate a new session to continue.

typestring* required

Constant: "stream_exhausted"

{
  "type": "stream_exhausted"
}

fal-ai/abot-world-0/start-session

Table of contents

1. Calling the API

2. Authentication

3. Schema

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

Session flow#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Movement

Customize world

Client keys message

Client ping message

Update world

Restart world

End session

Server messages#

Server configured message

Server error message

Server pong message

Server prompt_applied message

Server reset_applied message

Server session_info message

Server stats message

Server stream_exhausted message