H3 Max Reference Director fal Worlds API Docs

import { createFalClient } from "@fal-ai/client"; import { wma } from "@fal-ai/client/realtime"; const fal = createFalClient({ proxyUrl: "/api/fal/proxy" }); const session = fal.realtime.open(wma("minimax/h3-max/reference-director"), { receive: [ "video", "audio" ], onMedia: (stream) => { const element = document.querySelector("video"); if (element) element.srcObject = stream; }, onData: (raw) => { const message = JSON.parse(raw); console.log(message); }, onState: (state) => console.log("realtime:", state), onError: (error) => console.error(error), }); // Send any client message described below. session.send({ "dialogue_words_per_second": 2, "resolution": "768p", "aspect_ratio": "16:9", "reference_audio": [], "reference_audio_urls": [], "type": "configure", "reference_image_urls": [], "memory": 12, "num_inference_steps": 10, "prompt": "Photorealistic live-action cinema. Natural light and real materials, clear faces, grounded physical movement. Nora: adult woman, olive skin, long dark braid, burgundy bomber jacket, cream trousers; precise, competitive, generous. Milo: adult man, light skin, sandy-blond tight curls, small mustache, mustard sweater, dark jeans; distractible, athletic, optimistic. Keep the same faces, hair, outfits and body proportions throughout. Show each person separately and make their interaction readable. Nora and Milo jog along an open harbor promenade toward a small departing ferry. Milo carries a closed canvas bag. Nora points toward the gangway. A continuous adventurous short with occasional clear motivated cuts. Natural environmental sound, light music, no narration.", "mxfp8_ref2va_attention": false, "protocol_version": 1, "prompt_version": 1, "reference_images": [ { "url": "https://v3b.fal.media/files/b/0aad61a3/c9LOOpJnQOyGHiFyqe2nL_epSrZDee.png", "name": "Nora" }, { "url": "https://v3b.fal.media/files/b/0aad61a5/DlurMst_q9n1wsmoqwBYs_W2gpRpM6.png", "name": "Milo" } ], "seed": 631011, "pin_audio": false }); // Later, release the peer connection and network resources: // await session.close();

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "frameRate": 24
}

audio

optional

Client messages#

Client configure message

type: "configure"

payload: object

Additional properties: not allowed

promptstring* required

At least 1 character · At most 50000 characters

resolutionstring

Default: "480p"

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

aspect_ratiostring

Default: "16:9"

Values: "16:9", "9:16", "1:1"

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Vision-conditioning detail for image references: high (768px), medium (512px) or low (256px). Omitted keeps the reference's previous fidelity, or high for a new reference. Ignored for audio references.

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

reference_image_urlsstring[]

Default: []

maxItems: 9

Array item

string

scriptobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

minItems: 1 · maxItems: 64

Array item

object

Additional properties: not allowed

offsetnumber* required

Seconds from the start of the first video generated under this script; fractions are allowed and quantize to the 24 fps frame grid. Text and audio placed here start at this moment. A beat carrying reference_images makes those identities available from the whole second at or before this offset, so its own prompt can show them immediately.

minimum: 0

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

audio_urlstring | null

Default: null

At least 1 character when not null

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

Alternative 2

null

reference_audio(object & object)[]

Default: []

maxItems: 3

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

reference_audio_urlsstring[]

Default: []

maxItems: 3

Array item

string

pin_audioboolean

Default: false

memoryinteger

Number of prior segment prompts retained as context for future prompt expansion.

Default: 12

minimum: 1 · maximum: 50

audio_bitrateinteger | null

Session audio target in bits/s: 96000, 128000, or 192000. Explicit values use Opus audio mode on direct WebRTC; LiveKit uses its native mode with the same bitrate target. Null preserves transport defaults (WebRTC: 96000, voip). Immutable; reconnect to compare.

Default: null

When not null:

Values: 96000, 128000, 192000

seedinteger | null

Default: null

num_inference_stepsinteger

Default: 10

minimum: 1 · maximum: 50

mxfp8_ref2va_attentionboolean

Use MXFP8 attention for dense generation steps.

Default: false

forced_cut_max_segmentsinteger | null

Most segments in one unbroken continuation run before a forced hard cut opens a fresh shot (Ref2VA burn-in mitigation). 0 disables forced cuts; omitted keeps the deployment default. Pinned for the whole session.

Default: null

minimum: 0 when not null · maximum: 64 when not null

dialogue_words_per_secondnumber

Spoken words per second of new screen time the planner budgets dialogue against. The default keeps delivery calm; raise it for deliberately fast talkers, lower it for sparser dialogue.

Default: 2

minimum: 0.5 · maximum: 6

typestring* required

Constant: "configure"

protocol_versioninteger

Constant: 1

prompt_versioninteger* required

minimum: 1

Example

Nora and Milo jog along an open harbor promenade toward a small departing ferry. Milo carries a closed canvas bag. Nora points toward the gangway.

{
  "dialogue_words_per_second": 2,
  "resolution": "768p",
  "aspect_ratio": "16:9",
  "reference_audio": [],
  "reference_audio_urls": [],
  "type": "configure",
  "reference_image_urls": [],
  "memory": 12,
  "num_inference_steps": 10,
  "prompt": "Photorealistic live-action cinema. Natural light and real materials, clear faces, grounded physical movement. Nora: adult woman, olive skin, long dark braid, burgundy bomber jacket, cream trousers; precise, competitive, generous. Milo: adult man, light skin, sandy-blond tight curls, small mustache, mustard sweater, dark jeans; distractible, athletic, optimistic. Keep the same faces, hair, outfits and body proportions throughout. Show each person separately and make their interaction readable. Nora and Milo jog along an open harbor promenade toward a small departing ferry. Milo carries a closed canvas bag. Nora points toward the gangway. A continuous adventurous short with occasional clear motivated cuts. Natural environmental sound, light music, no narration.",
  "mxfp8_ref2va_attention": false,
  "protocol_version": 1,
  "prompt_version": 1,
  "reference_images": [
    {
      "url": "https://v3b.fal.media/files/b/0aad61a3/c9LOOpJnQOyGHiFyqe2nL_epSrZDee.png",
      "name": "Nora"
    },
    {
      "url": "https://v3b.fal.media/files/b/0aad61a5/DlurMst_q9n1wsmoqwBYs_W2gpRpM6.png",
      "name": "Milo"
    }
  ],
  "seed": 631011,
  "pin_audio": false
}

Client ping message

type: "ping"

payload: object

Additional properties: not allowed

typestring* required

Constant: "ping"

tsnumber* required

{
  "type": "ping",
  "ts": 0
}

Client prompt message

type: "prompt"

payload: object

Additional properties: not allowed

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

reference_images(object & object)[] | null

Default: null

Must match at least one of the following:

Alternative 1

(object & object)[]

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

Alternative 2

null

reference_image_urlsstring[] | null

Default: null

Must match at least one of the following:

Alternative 1

string[]

maxItems: 9

Array item

string

Alternative 2

null

scriptobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

minItems: 1 · maxItems: 64

Array item

object

Additional properties: not allowed

offsetnumber* required

minimum: 0

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

audio_urlstring | null

Default: null

At least 1 character when not null

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

Alternative 2

null

audio_behaviorstring

'replace' cuts to this audio at the next chunk; 'queue' appends it.

Default: "replace"

Values: "replace", "queue"

replanboolean

When a prompt is included: true (default) busts the planned prompt queue so the new direction applies at the next undispatched chunk; false appends the direction after the already-planned chunks.

Default: true

reference_audio(object & object)[] | null

Default: null

Must match at least one of the following:

Alternative 1

(object & object)[]

maxItems: 3

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

Additional properties: allowed

urlunknown

Alternative 2

null

reference_audio_urlsstring[] | null

Default: null

Must match at least one of the following:

Alternative 1

string[]

maxItems: 3

Array item

string

Alternative 2

null

typestring* required

Constant: "prompt"

prompt_versioninteger* required

minimum: 1

script_modestring

'replace' cuts to the script at the next chunk; 'append' queues it.

Default: "replace"

Values: "replace", "append"

{
  "type": "prompt",
  "prompt_version": 1
}

Client stop message

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server audio_applied message

type: "audio_applied"

payload: object

An audio update was accepted; it conditions the next eligible chunk.

starts_at_chunk_index is the first chunk that can carry it (the next undispatched chunk for replace; for queue the chunk where the preceding sources end, if already known).

transcribedboolean* required

duration_secondsnumber* required

exclusiveMinimum: 0

prompt_versioninteger* required

minimum: 1

sourcestring* required

remaining_secondsnumber* required

minimum: 0

typestring* required

Constant: "audio_applied"

queued_sourcesinteger* required

minimum: 0

behaviorstring* required

Values: "replace", "queue"

{
  "transcribed": true,
  "duration_seconds": 5e-324,
  "prompt_version": 1,
  "source": "string",
  "remaining_seconds": 0,
  "type": "audio_applied",
  "queued_sources": 0,
  "behavior": "replace"
}

Server audio_exhausted message

type: "audio_exhausted"

payload: object

Accepted audio ran out inside this chunk; the rest is silence.

Later chunks continue without audio conditioning (checkpoint continuation on the same family) until new audio is accepted.

typestring* required

Constant: "audio_exhausted"

chunk_indexinteger* required

minimum: 0

silent_secondsnumber* required

minimum: 0

source_versioninteger | null

Default: null

minimum: 0 when not null

{
  "type": "audio_exhausted",
  "chunk_index": 0,
  "silent_seconds": 0
}

Server audio_pending message

type: "audio_pending"

payload: object

typestring* required

Constant: "audio_pending"

prompt_versioninteger* required

minimum: 1

behaviorstring* required

Values: "replace", "queue"

{
  "type": "audio_pending",
  "prompt_version": 1,
  "behavior": "replace"
}

Server audio_rejected message

type: "audio_rejected"

payload: object

reasonstring* required

Values: "invalid_audio", "content_policy", "preparation_failed", "queue_full", "stale_prompt_version"

typestring* required

Constant: "audio_rejected"

errorstring* required

prompt_versioninteger* required

minimum: 1

{
  "reason": "invalid_audio",
  "type": "audio_rejected",
  "error": "string",
  "prompt_version": 1
}

Server chunk message

type: "chunk"

payload: object

routestring* required

Constant: "k10-reference"

next_generation_estimate_secondsnumber* required

minimum: 0

native_playable_frame_countinteger | null

Default: null

exclusiveMinimum: 0 when not null

trimmed_context_framesinteger* required

minimum: 0

buffer_depth_secondsnumber* required

minimum: 0

buffer_depth_chunksinteger* required

minimum: 0

dispatchobject* required

overhead_msnumber* required

minimum: 0

wall_msnumber* required

minimum: 0

classified_msnumber* required

minimum: 0

phases_msobject* required

Additional properties: number

number

typestring* required

Constant: "chunk"

script_versioninteger | null

Default: null

minimum: 1 when not null

scheduling_lead_msnumber* required

requested_duration_secondsinteger* required

minimum: 5 · maximum: 15

script_offset_secondsinteger | null

Default: null

minimum: 0 when not null

script_end_keyframeboolean | null

Default: null

generated_frame_countinteger* required

exclusiveMinimum: 0

prompt_versioninteger* required

minimum: 1

scheduling_slack_msnumber* required

generation_secondsnumber* required

minimum: 0

hard_cutboolean

Default: false

presented_frame_countinteger | null

Default: null

exclusiveMinimum: 0 when not null

chunk_indexinteger* required

minimum: 0

playback_secondsnumber* required

exclusiveMinimum: 0

{
  "route": "k10-reference",
  "next_generation_estimate_seconds": 0,
  "trimmed_context_frames": 0,
  "buffer_depth_seconds": 0,
  "buffer_depth_chunks": 0,
  "dispatch": {
    "overhead_ms": 0,
    "wall_ms": 0,
    "classified_ms": 0,
    "phases_ms": {}
  },
  "type": "chunk",
  "scheduling_lead_ms": 0,
  "requested_duration_seconds": 5,
  "generated_frame_count": 1,
  "prompt_version": 1,
  "scheduling_slack_ms": 0,
  "generation_seconds": 0,
  "chunk_index": 0,
  "playback_seconds": 5e-324
}

Server chunk_metrics message

type: "chunk_metrics"

payload: object

unitsstring* required

Constant: "ms"

routestring* required

Constant: "k10-reference"

chunk_indexinteger* required

minimum: 0

phases_msobject* required

Additional properties: number

number

gaugesobject* required

Additional properties: number

number

typestring* required

Constant: "chunk_metrics"

chunk_consumable_interval_msnumber | null

Default: null

minimum: 0 when not null

chunk_consumable_ready_msnumber | null

Default: null

minimum: 0 when not null

{
  "units": "ms",
  "route": "k10-reference",
  "chunk_index": 0,
  "phases_ms": {},
  "gauges": {},
  "type": "chunk_metrics"
}

Server configured message

type: "configured"

payload: object

chunk_durationinteger | null

Default: null

minimum: 5 when not null · maximum: 15 when not null

prompt_versioninteger* required

minimum: 1

resolutionstring | null

Default: null

When not null:

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

enable_safety_checkerboolean* required

accelerationstring | null

Default: null

When not null:

Values: "none", "regular"

has_initial_imageboolean | null

Default: null

aspect_ratiostring | null

Default: null

When not null:

Values: "16:9", "9:16", "1:1"

has_initial_audioboolean | null

Default: null

typestring* required

Constant: "configured"

audio_bitrateinteger | null

Default: null

When not null:

Values: 96000, 128000, 192000

memoryinteger | null

Default: null

minimum: 1 when not null · maximum: 50 when not null

{
  "prompt_version": 1,
  "enable_safety_checker": true,
  "type": "configured"
}

Server deadline_missed message

type: "deadline_missed"

payload: object

late_by_secondsnumber* required

minimum: 0

typestring* required

Constant: "deadline_missed"

chunk_indexinteger* required

minimum: 0

behaviorstring* required

Constant: "freeze_video_and_silence_audio_until_ready"

{
  "late_by_seconds": 0,
  "type": "deadline_missed",
  "chunk_index": 0,
  "behavior": "freeze_video_and_silence_audio_until_ready"
}

Server error message

type: "error"

payload: object

codestring* required

Values: "balance_unavailable", "content_policy", "configuration_timeout", "generation_failed", "generation_timeout", "immutable_settings", "initialization_timeout", "invalid_initial_image", "invalid_initial_audio", "invalid_initial_script", "invalid_input", "invalid_message", "not_configured", "stale_prompt_version"

typestring* required

Constant: "error"

errorstring* required

detailobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

Array item

object

Alternative 2

null

prompt_versioninteger | null

Default: null

minimum: 1 when not null

{
  "code": "balance_unavailable",
  "type": "error",
  "error": "string"
}

Server pong message

type: "pong"

payload: object

client_tsnumber* required

typestring* required

Constant: "pong"

{
  "client_ts": 0,
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

A direction (or script) is admitted; the script fields report scripts.

script_origin_chunk_index is the first chunk generated under the script when that is already fixed (a replacement binds to the next undispatched chunk); an appended script reports None until it starts.

script_beatsinteger | null

Default: null

minimum: 1 when not null

script_queuedinteger | null

Default: null

minimum: 0 when not null

typestring* required

Constant: "prompt_applied"

script_modestring | null

Default: null

When not null:

Values: "replace", "append"

script_origin_chunk_indexinteger | null

Default: null

minimum: 0 when not null

prompt_versioninteger* required

minimum: 1

{
  "type": "prompt_applied",
  "prompt_version": 1
}

Server prompt_pending message

type: "prompt_pending"

payload: object

typestring* required

Constant: "prompt_pending"

prompt_versioninteger* required

minimum: 1

{
  "type": "prompt_pending",
  "prompt_version": 1
}

Server prompt_rejected message

type: "prompt_rejected"

payload: object

reasonstring* required

Values: "content_policy", "preparation_failed", "stale_prompt_version", "invalid_script", "infeasible_timing", "invalid_audio", "invalid_image", "queue_full"

typestring* required

Constant: "prompt_rejected"

errorstring | null

Default: null

prompt_versioninteger* required

minimum: 1

{
  "reason": "content_policy",
  "type": "prompt_rejected",
  "prompt_version": 1
}

Server reference_state message

type: "reference_state"

payload: object

typestring* required

Constant: "reference_state"

prompt_versioninteger* required

referencesobject[]* required

Array item

object

revisioninteger* required

kindstring* required

Values: "image", "audio"

idstring* required

namestring | null

Default: null

{
  "type": "reference_state",
  "prompt_version": 0,
  "references": []
}

Server session_info message

type: "session_info"

payload: object

billable_units_per_playable_secondobject

Default: {"1080p":1.2,"544p":1.04,"704p":1.15,"768p":1.2,"640p":1.1,"480p":1,"1440p":1.2}

Additional properties: number

number

pricing_statusstring

Default: "provisional-experiment"

session_limit_scopestring

Default: "configured"

min_num_inference_stepsinteger

Default: 1

dialogue_words_per_secondnumber

Default: 2

appstring

Constant: "minimax-h3-max-reference-director"

max_num_inference_stepsinteger

Default: 50

scriptsboolean

Default: true

typestring* required

Constant: "session_info"

min_chunk_durationinteger

Default: 5

backend_selectionstring

Default: "resident-k10-reference"

default_resolutionstring

Default: "480p"

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

prompt_deck_sizeinteger

Default: 6

accelerationsstring[]

Default: ["none"]

Array item

string

forced_cut_max_segmentsinteger

resolutionsstring[]

Default: ["480p","544p","640p","704p","768p","1080p","1440p"]

Array item

string

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

reference_audio_modestring

Default: "sequential-content-windows"

continuation_context_framesinteger

Default: 39

protocol_versioninteger

Constant: 1

controller_machine_typestring

Default: "GPU-B300/GPU-B200 (8 GPUs)"

max_session_secondsnumber | null

Default: null

fpsinteger

Default: 24

default_num_inference_stepsinteger

Default: 10

default_chunk_durationinteger

Default: 10

max_chunk_durationinteger

Default: 15

one_session_per_machineboolean

Default: true

default_pin_audioboolean

Default: false

{
  "app": "minimax-h3-max-reference-director",
  "type": "session_info",
  "protocol_version": 1
}

Server session_metrics message

type: "session_metrics"

payload: object

unitsstring* required

Constant: "ms"

history_sizeinteger* required

minimum: 0

finalboolean

Default: false

gaugesobject* required

Additional properties: number

number

phasesobject* required

Additional properties: object

object

total_msnumber* required

minimum: 0

max_msnumber* required

minimum: 0

countinteger* required

minimum: 0

p50_msnumber* required

minimum: 0

p95_msnumber* required

minimum: 0

typestring* required

Constant: "session_metrics"

session_wall_msnumber* required

minimum: 0

history_limitinteger* required

minimum: 1

{
  "units": "ms",
  "history_size": 0,
  "gauges": {},
  "phases": {},
  "type": "session_metrics",
  "session_wall_ms": 0,
  "history_limit": 1
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

reasonstring* required

Values: "stopped", "session_limit"

typestring* required

Constant: "stream_exhausted"

chunksinteger* required

minimum: 0

{
  "reason": "stopped",
  "type": "stream_exhausted",
  "chunks": 0
}

minimax/h3-max/reference-director

Table of contents

Table of contents

minimax/h3-max/reference-director

Table of contents

Table of contents

1. Calling the API

2. Authentication

3. Schema

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Client configure message

Client ping message

Client prompt message

Client stop message

Server messages#

Server audio_applied message

Server audio_exhausted message

Server audio_pending message

Server audio_rejected message

Server chunk message

Server chunk_metrics message

Server configured message

Server deadline_missed message

Server error message

Server pong message

Server prompt_applied message

Server prompt_pending message

Server prompt_rejected message

Server reference_state message

Server session_info message

Server session_metrics message

Server stream_exhausted message

Related Models