alibaba/happy-oyster

Realtime interactive world model — generate a world from a prompt, then explore it or direct its story as live video.

Inference

Commercial use

About

Happy Oyster is an interactive world model: you create a world (a durable, explorable environment generated from a prompt), then run travels — live, realtime video sessions inside it that you steer as they stream. This API is the control plane; the realtime video is delivered over WebRTC by the world engine's browser SDK (@happy-oyster/js-sdk) using short-lived credentials minted here. Media never transits your backend, and your fal key never ships to the browser.

Install both halves — the fal client for the control plane (server-side) and the world engine SDK for the stream (browser):

npm install @fal-ai/client @happy-oyster/js-sdk

1. Create a world (server)

Control-plane calls authenticate with your fal key, so they belong on your server. Build a world in adventure (explore/play) or directing (story) mode, then poll until it's ready (builds take ~30–120s):

import { fal } from "@fal-ai/client";

const { data: created } = await fal.run(
  "alibaba/happy-oyster/worlds/create",
  {
    input: {
      mode: "adventure",
      prompt: "A detective adventure in a rainy cyberpunk city",
      perspective: "first_person",
    },
  }
);

let world = created;
while (world.status === "generating") {
  await new Promise((resolve) => setTimeout(resolve, 4000));
  ({ data: world } = await fal.run(
    "alibaba/happy-oyster/worlds/build-status",
    { input: { encrypted_world_id: created.encrypted_world_id } }
  ));
}
// world.status is now "ready" (or "failed")

Worlds are durable — keep encrypted_world_id and travel through the same world again later.

2. Mint session credentials (server)

A session exchanges your fal key for short-lived realtime credentials: a one-time ticket, a refreshable token, and the engine's api_base_url. Token grants bill at mint for the seconds granted, so request what you expect to stream and refresh to extend (step 5) — don't front-load a long grant. Hand the connection object to your browser client — it contains no long-lived secrets:

const { data: session } = await fal.run("alibaba/happy-oyster/session", {
  input: {
    session_params: {
      encrypted_world_id: world.encrypted_world_id,
      token_expire_seconds: 60,
    },
  },
});
// session.session_id           — keep server-side for heartbeats
// session.heartbeat_interval_sec
// session.connection           — { api_base_url, ticket, token, token_expires_in }

3. Connect and stream (browser)

The SDK opens the WebRTC connection straight to the world engine and plays the live video into your <video> element:

import { HappyOysterEngine } from "@happy-oyster/js-sdk";

const engine = new HappyOysterEngine({
  APIHost: new URL(connection.api_base_url).host,
});
engine.updateToken(connection.token);

const travel = engine.createTravel({
  ticket: connection.ticket,
  videoElement: document.querySelector("video"),
});
travel.on("statusChanged", (status) => console.log("travel:", status));

const { encryptedTravelId } = await travel.start();

4. Drive the travel

Adventure mode takes movement commands — components are sent on change and "None" releases them:

// walk forward while panning right, then stop
await travel.sendCommand({
  translation: "Front",
  rotation: "Mouse_Right",
  interaction: "None",
});
await travel.sendCommand({
  translation: "None",
  rotation: "None",
  interaction: "None",
});

Directing mode takes free-text instructions plus transport control:

await travel.sendInstruct({
  content: "A giant robotic dinosaur suddenly appears",
});
await travel.pause();
await travel.resume();
await travel.rewind({ rewindToSec: 12 }); // pause first; snaps to 4s blocks

Gate your UI with travel.can("pause") etc. so you never offer an action the current travel state rejects.

5. Keep it alive (server)

Two loops run while the travel streams. Heartbeat the session at the returned interval, and bind the travel id so the platform can clean up if your client disappears:

await fal.run("alibaba/happy-oyster/session/bind-travel", {
  input: { session_id: session.session_id, encrypted_travel_id: encryptedTravelId },
});

setInterval(async () => {
  const { data } = await fal.run("alibaba/happy-oyster/session/heartbeat", {
    input: { session_id: session.session_id },
  });
  if (!data.alive) {
    // session expired — end the travel and start a fresh session
  }
}, session.heartbeat_interval_sec * 1000);

Tokens expire after token_expires_in seconds; refresh before expiry and inject the new one without interrupting the stream:

const { data: refreshed } = await fal.run("alibaba/happy-oyster/tokens/issue", {
  input: { expire_in_seconds: 60 },
});
engine.updateToken(refreshed.token); // browser side

6. End and fetch the replay

await travel.end(); // browser

// server: release the session, then poll for the composed replay
await fal.run("alibaba/happy-oyster/session/close", {
  input: { session_id: session.session_id },
});
const { data: artifacts } = await fal.run(
  "alibaba/happy-oyster/travels/artifacts",
  { input: { encrypted_travel_id: encryptedTravelId } }
);
// artifacts.video.original.url once compose_status is "ready"

Exact request/response shapes for every route are documented in the Schema sections below, in flow order.

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install --save @fal-ai/client

Migrate to @fal-ai/client

The @fal-ai/serverless-client package has been deprecated in favor of @fal-ai/client. Please check the migration guide for more information.

Setup your API Key#

Set FAL_KEY as an environment variable in your runtime.

export FAL_KEY="YOUR_API_KEY"

Submit a request#

The client API handles the API submit protocol. It will handle the request status updates and return the result when the request is completed.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("alibaba/happy-oyster/worlds/create", {
  input: {
    mode: "adventure"
  },
  logs: true,
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") {
      update.logs.map((log) => log.message).forEach(console.log);
    }
  },
});
console.log(result.data);
console.log(result.requestId);

2. Authentication#

The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.

API Key#

In case your app is running in an environment where you cannot set environment variables, you can set the API Key manually as a client configuration.

import { fal } from "@fal-ai/client";

fal.config({
  credentials: "YOUR_FAL_KEY"
});

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Queue#

Long-running requests

For long-running requests, such as training jobs or models with slower inference times, it is recommended to check the Queue status and rely on Webhooks instead of blocking while waiting for the result.

Submit a request#

The client API provides a convenient way to submit requests to the model.

import { fal } from "@fal-ai/client";

const { request_id } = await fal.queue.submit("alibaba/happy-oyster/worlds/create", {
  input: {
    mode: "adventure"
  },
  webhookUrl: "https://optional.webhook.url/for/results",
});

Fetch request status#

You can fetch the status of a request to check if it is completed or still in progress.

import { fal } from "@fal-ai/client";

const status = await fal.queue.status("alibaba/happy-oyster/worlds/create", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b",
  logs: true,
});

Get the result#

Once the request is completed, you can fetch the result. See the Output Schema for the expected result format.

import { fal } from "@fal-ai/client";

const result = await fal.queue.result("alibaba/happy-oyster/worlds/create", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b"
});
console.log(result.data);
console.log(result.requestId);

4. Files#

Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.

Data URI (base64)#

You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.

Hosted files (URL)#

You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.

Uploading files#

We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.

import { fal } from "@fal-ai/client";

const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);

Auto uploads

The client will auto-upload the file for you if you pass a binary object (e.g. File, Data).

Read more about file handling in our file upload guide.

5. Schema#

POST /worlds/create#

Create World

Input

mode ModeEnum* required

Experience mode: adventure (explore/play) or directing (story).

Possible enum values: adventure, directing

prompt string

Natural-language world description. Required for simple creation unless upload_mode is scenario_role.

event_style Enum

Event style for generated events.

Possible enum values: normal, dramatic

ref_world_id string

Encrypted ID of a template world to derive this world from.

sync boolean

When true the upstream build is awaited (up to ~120s) before returning; otherwise poll /worlds/build-status.

perspective Enum

Camera perspective. Required in adventure mode.

Possible enum values: first_person, third_person

upload_mode Enum

Adventure creation sub-mode; default first_frame.

Possible enum values: first_frame, scenario_role

scene_prompt string

Scene description (scenario_role).

role_prompt string

Role description (scenario_role).

scene_image_url string

Scene reference image URL (scenario_role).

role_image_url string

Role reference image URL (scenario_role).

resolution Enum

Video resolution. Required in directing mode.

Possible enum values: 480p, 720p

layout Enum

Camera movement style (directing mode).

Possible enum values: Stable, Fast

narrative Enum

Narrative style (directing mode).

Possible enum values: Calm, Dramatic, Normal

script_list object

Structured script (subjects + up to 45 acts); selects scriptlist creation. Directing mode only.

first_frame_image_url string

First-frame anchor image URL.

input_image_urls list<string>

Additional reference image URLs.

{
  "mode": "adventure",
  "prompt": "A detective adventure in a rainy cyberpunk city"
}

Output

encrypted_world_id string* required

status string* required

generating, ready, or failed.

first_frame string

First-frame image URL once available.

name string

World title once built.

mode Enum

Possible enum values: adventure, directing

POST /worlds/build-status#

Open to any holder of the world id — clients must poll build readiness before they can enter (travel) a shared world.

Poll every few seconds until status is ready (builds take ~30–120s).

Input

encrypted_world_id string* required

Encrypted world ID returned by /worlds/create.

{
  "encrypted_world_id": ""
}

Output

encrypted_world_id string* required

status string* required

generating, ready, or failed.

first_frame string

First-frame image URL once available.

name string

World title once built.

mode Enum

Possible enum values: adventure, directing

POST /worlds/detail#

World Detail

Input

encrypted_world_id string* required

Encrypted world ID returned by /worlds/create.

{
  "encrypted_world_id": ""
}

Output

encrypted_world_id string* required

status string* required

generating, ready, or failed.

first_frame string

First-frame image URL once available.

name string

World title once built.

mode Enum

Possible enum values: adventure, directing

prompt string

creation_model string

script_list object

created_at string

updated_at string

POST /session#

Start Session

The connection object carries the one-time ticket, the short-lived token, and the api_base_url to hand to @happy-oyster/js-sdk in the browser.

Input

session_params Session Params

Mutable session parameters shared between client and runner. The app validates and consumes app-specific keys.

offer SDPOffer

Optional WebRTC SDP offer. Only used by apps that terminate media on the runner; control-plane apps ignore it.

{}

Output

session_id string* required

Server-issued session identifier.

session_timeout_sec float* required

Seconds of heartbeat silence after which the session expires.

heartbeat_interval_sec float* required

Recommended interval between heartbeat calls, in seconds.

answer SDPMessage

WebRTC SDP answer; present only when an offer was sent.

connection Connection

App-specific connection material returned by on_connect.

POST /session/heartbeat#

Session Heartbeat

Call every ~5 seconds while the travel runs. alive: false means the session expired — start a fresh one.

Input

session_id string* required

Session identifier from POST /session.

{
  "session_id": ""
}

Output

session_id string* required

alive boolean* required

False when the session is unknown or expired. Sessions are not resumable: create a fresh session instead of retrying.

heartbeat_interval_sec float* required

POST /session/bind-travel#

Bind Travel

Report the travel id from travel.start() so the platform can end the travel if your client disappears.

Input

session_id string* required

Session ID from POST /session.

encrypted_travel_id string* required

Travel to associate with the session; it is ended upstream when the session closes or expires.

{
  "session_id": "",
  "encrypted_travel_id": ""
}

Output

session_id string* required

bound boolean* required

False when the session is unknown or expired.

POST /session/close#

Close Session

Input

session_id string* required

Session identifier from POST /session.

{
  "session_id": ""
}

Output

session_id string* required

closed boolean* required

False when the session was already gone (call is idempotent).

POST /tokens/issue#

Mint a fresh temporary token (e.g. for updateToken on expiry). Tickets are not reissued here; start a new /session for that.

Refresh before expiry and inject via engine.updateToken().

Input

expire_in_seconds integer

Temporary token validity. Prefer the maximum so the token outlives the session; refresh via this endpoint on expiry. Default value: 1800

{
  "expire_in_seconds": 1800
}

Output

token string* required

Short-lived Open Platform API-key token.

expires_in integer* required

Requested validity in seconds.

api_base_url string* required

Open Platform base URL for the client SDK (apiBaseUrl).

POST /travels/status#

Travel Status

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

client_stream_status string

Optional client RTC playback status heartbeat (DISCONNECTED / CONNECTING / CONNECTED / PLAYING / BUFFERING / PAUSED / RECONNECTING).

client_stream_status_time_ms integer

Client status change timestamp (ms epoch).

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

status string* required

init, pending, running, failed, or completed.

rtc_status string

update_time string

user_instructions list<object>

chapters list<object>

character_actions list<string>

environment_actions list<string>

POST /travels/pause#

Pause Travel

Programmatic control for server-side clients — interactive browsers should use the SDK's equivalent instead, so its local state stays in sync with the stream.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

status string* required

POST /travels/resume#

Resume Travel

Programmatic control for server-side clients — interactive browsers should use the SDK's equivalent instead, so its local state stays in sync with the stream.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

status string* required

POST /travels/rewind#

Rewind Travel

Programmatic control for server-side clients — interactive browsers should use the SDK's equivalent instead, so its local state stays in sync with the stream.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

rewind_to_sec float* required

Target position in seconds; the server may round it down.

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

status string* required

resumed_at_sec float

POST /travels/instruct#

Instruct Travel

Programmatic control for server-side clients — interactive browsers should use the SDK's equivalent instead, so its local state stays in sync with the stream.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

content string* required

Directing-mode text instruction (e.g. a story beat).

{
  "encrypted_travel_id": "",
  "content": "A giant robotic dinosaur suddenly appears"
}

Output

encrypted_travel_id string* required

content string* required

accepted boolean* required

POST /travels/end#

End Travel

Programmatic control for server-side clients — interactive browsers should use the SDK's equivalent instead, so its local state stays in sync with the stream.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

fail_code string

Pass TRAVEL_NO_STREAM_AUTO_END when ending because no stream arrived in time; the travel is then marked failed and produces no replay artifact.

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

status string* required

ended_at string

duration_sec integer

error_code string

error_message string

POST /travels/artifacts#

Travel Artifacts

Replay video variants for an ended travel; poll while compose_status is processing.

Input

encrypted_travel_id string* required

Encrypted travel ID from the client's enter-travel.

{
  "encrypted_travel_id": ""
}

Output

encrypted_travel_id string* required

compose_status string

ready, partial, or processing.

video Video

Variants keyed original / withWatermark / withInstruction / withInstructionAndWatermark, each with url/status/resolution.

POST /info#

Info

Output

api_base_url string* required

modes Modes* required

adventure_commands Adventure Commands* required

ticket_expires_in integer* required

max_token_expire_seconds integer* required

session_timeout_sec float* required

heartbeat_interval_sec float* required

Other types#

SDPOffer#

sdp string* required

Session description protocol payload.

type string

SDP message type; must be offer. Default value: "offer"

SDPMessage#

sdp string* required

Session description protocol payload.

type string* required

SDP message type: offer or answer.

alibaba/happy-oyster

Table of contents

1. Calling the API

2. Authentication

3. Queue

4. Files

5. Schema

About

1. Create a world (server)

2. Mint session credentials (server)

3. Connect and stream (browser)

4. Drive the travel

5. Keep it alive (server)

6. End and fetch the replay

1. Calling the API#

Install the client#

Migrate to @fal-ai/client

Setup your API Key#

Submit a request#

2. Authentication#

API Key#

Protect your API Key

3. Queue#

Long-running requests

Submit a request#

Fetch request status#

Get the result#

4. Files#

Data URI (base64)#

Hosted files (URL)#

Uploading files#

Auto uploads

5. Schema#

POST /worlds/create#

Input

Output

POST /worlds/build-status#

Input

Output

POST /worlds/detail#

Input

Output

POST /session#

Input

Output

POST /session/heartbeat#

Input

Output

POST /session/bind-travel#

Input

Output

POST /session/close#

Input

Output

POST /tokens/issue#

Input

Output

POST /travels/status#

Input

Output

POST /travels/pause#

Input

Output

POST /travels/resume#

Input

Output

POST /travels/rewind#

Input

Output

POST /travels/instruct#

Input

Output

POST /travels/end#

Input

Output

POST /travels/artifacts#

Input

Output

POST /info#

Output