H3 Max API Docs

import { createFalClient } from "@fal-ai/client"; import { wma } from "@fal-ai/client/realtime"; const fal = createFalClient({ proxyUrl: "/api/fal/proxy" }); const session = fal.realtime.open(wma("minimax/h3-max/reference-director"), { receive: [ "video", "audio" ], onMedia: (stream) => { const element = document.querySelector("video"); if (element) element.srcObject = stream; }, onData: (raw) => { const message = JSON.parse(raw); console.log(message); }, onState: (state) => console.log("realtime:", state), onError: (error) => console.error(error), }); // Send any client message described below. session.send({ "protocol_version": 1, "reference_audio": [], "prompt": "A traveler explores", "dialogue_words_per_second": 2, "reference_images": [], "prompt_version": 1, "resolution": "480p", "num_inference_steps": 10, "aspect_ratio": "16:9", "reference_image_urls": [], "type": "configure", "mxfp8_ref2va_attention": false, "reference_audio_urls": [], "pin_audio": false, "memory": 12 }); // Later, release the peer connection and network resources: // await session.close();

2. Authentication#

The browser connects through your server proxy, which reads FAL_KEY from the server environment. Never put that key in browser code.

API Key#

Authenticate users before allowing them to call your proxy route so other people cannot spend against your account.

Protect your API Key

When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.

3. Schema#

Media#

Client contract for the WebRTC session created by the linked OpenAPI operation.

Tracks are described from the browser's perspective. Send tracks are captured by the browser and sent to the model; receive tracks come back from the model.

Send

No send media tracks.

Receive

video

optional

{
  "frameRate": 24
}

audio

optional

Client messages#

Client configure message

type: "configure"

payload: object

Additional properties: not allowed

promptstring* required

At least 1 character · At most 50000 characters

resolutionstring

Default: "480p"

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

aspect_ratiostring

Default: "16:9"

Values: "16:9", "9:16", "1:1"

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Vision-conditioning detail for image references: high (768px), medium (512px) or low (256px). Omitted keeps the reference's previous fidelity, or high for a new reference. Ignored for audio references.

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

reference_image_urlsstring[]

Default: []

maxItems: 9

Array item

string

scriptobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

minItems: 1 · maxItems: 64

Array item

object

Additional properties: not allowed

offsetnumber* required

Seconds from the start of the first video generated under this script; fractions are allowed and quantize to the 24 fps frame grid. Text and audio placed here start at this moment. A beat carrying reference_images makes those identities available from the whole second at or before this offset, so its own prompt can show them immediately.

minimum: 0

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

audio_urlstring | null

Default: null

At least 1 character when not null

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

Alternative 2

null

reference_audio(object & object)[]

Default: []

maxItems: 3

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

reference_audio_urlsstring[]

Default: []

maxItems: 3

Array item

string

pin_audioboolean

Default: false

memoryinteger

Number of prior segment prompts retained as context for future prompt expansion.

Default: 12

minimum: 1 · maximum: 50

audio_bitrateinteger | null

Session audio target in bits/s: 96000, 128000, or 192000. Explicit values use Opus audio mode on direct WebRTC; LiveKit uses its native mode with the same bitrate target. Null preserves transport defaults (WebRTC: 96000, voip). Immutable; reconnect to compare.

Default: null

When not null:

Values: 96000, 128000, 192000

seedinteger | null

Default: null

num_inference_stepsinteger

Default: 10

minimum: 1 · maximum: 50

mxfp8_ref2va_attentionboolean

Use MXFP8 attention for dense generation steps.

Default: false

forced_cut_max_segmentsinteger | null

Most segments in one unbroken continuation run before a forced hard cut opens a fresh shot (Ref2VA burn-in mitigation). 0 disables forced cuts; omitted keeps the deployment default. Pinned for the whole session.

Default: null

minimum: 0 when not null · maximum: 64 when not null

dialogue_words_per_secondnumber

Spoken words per second of new screen time the planner budgets dialogue against. The default keeps delivery calm; raise it for deliberately fast talkers, lower it for sparser dialogue.

Default: 2

minimum: 0.5 · maximum: 6

typestring* required

Constant: "configure"

protocol_versioninteger

Constant: 1

prompt_versioninteger* required

minimum: 1

Example

Begin a continuous scene without references.

{
  "protocol_version": 1,
  "reference_audio": [],
  "prompt": "A traveler explores",
  "dialogue_words_per_second": 2,
  "reference_images": [],
  "prompt_version": 1,
  "resolution": "480p",
  "num_inference_steps": 10,
  "aspect_ratio": "16:9",
  "reference_image_urls": [],
  "type": "configure",
  "mxfp8_ref2va_attention": false,
  "reference_audio_urls": [],
  "pin_audio": false,
  "memory": 12
}

Client ping message

type: "ping"

payload: object

Additional properties: not allowed

typestring* required

Constant: "ping"

tsnumber* required

{
  "type": "ping",
  "ts": 0
}

Client prompt message

type: "prompt"

payload: object

Additional properties: not allowed

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

reference_images(object & object)[] | null

Default: null

Must match at least one of the following:

Alternative 1

(object & object)[]

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

Alternative 2

null

reference_image_urlsstring[] | null

Default: null

Must match at least one of the following:

Alternative 1

string[]

maxItems: 9

Array item

string

Alternative 2

null

scriptobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

minItems: 1 · maxItems: 64

Array item

object

Additional properties: not allowed

offsetnumber* required

minimum: 0

promptstring | null

Default: null

At least 1 character when not null · At most 50000 characters when not null

audio_urlstring | null

Default: null

At least 1 character when not null

reference_images(object & object)[]

Default: []

maxItems: 9

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

Alternative 2

null

audio_behaviorstring

'replace' cuts to this audio at the next chunk; 'queue' appends it.

Default: "replace"

Values: "replace", "queue"

replanboolean

When a prompt is included: true (default) busts the planned prompt queue so the new direction applies at the next undispatched chunk; false appends the direction after the already-planned chunks.

Default: true

reference_audio(object & object)[] | null

Default: null

Must match at least one of the following:

Alternative 1

(object & object)[]

maxItems: 3

Array item

object & object

Must match all of the following:

NamedReference

object

Additional properties: not allowed

namestring | null

Default: null

At most 64 characters when not null

urlstring* required

At least 1 character

fidelitystring | null

Default: null

When not null:

Values: "high", "medium", "low"

idstring | null

Default: null

pattern: ^ref-[1-9][0-9]*$ when not null

Requirement 2

object

urlunknown

Alternative 2

null

reference_audio_urlsstring[] | null

Default: null

Must match at least one of the following:

Alternative 1

string[]

maxItems: 3

Array item

string

Alternative 2

null

typestring* required

Constant: "prompt"

prompt_versioninteger* required

minimum: 1

script_modestring

'replace' cuts to the script at the next chunk; 'append' queues it.

Default: "replace"

Values: "replace", "append"

Example

Update ref-1 after configuring Mira's arrival; replace the example URL with your asset.

{
  "script_mode": "replace",
  "reference_images": [
    {
      "url": "https://example.com/new-coat.jpg"
    }
  ],
  "type": "prompt",
  "prompt_version": 2,
  "audio_behavior": "replace",
  "replan": true
}

Client stop message

type: "stop"

payload: object

Additional properties: not allowed

typestring* required

Constant: "stop"

{
  "type": "stop"
}

Server messages#

Server audio_applied message

type: "audio_applied"

payload: object

An audio update was accepted; it conditions the next eligible chunk.

starts_at_chunk_index is the first chunk that can carry it (the next undispatched chunk for replace; for queue the chunk where the preceding sources end, if already known).

transcribedboolean* required

remaining_secondsnumber* required

minimum: 0

typestring* required

Constant: "audio_applied"

behaviorstring* required

Values: "replace", "queue"

sourcestring* required

prompt_versioninteger* required

minimum: 1

queued_sourcesinteger* required

minimum: 0

duration_secondsnumber* required

exclusiveMinimum: 0

{
  "transcribed": true,
  "remaining_seconds": 0,
  "type": "audio_applied",
  "behavior": "replace",
  "source": "string",
  "prompt_version": 1,
  "queued_sources": 0,
  "duration_seconds": 5e-324
}

Server audio_exhausted message

type: "audio_exhausted"

payload: object

Accepted audio ran out inside this chunk; the rest is silence.

Later chunks continue without audio conditioning (checkpoint continuation on the same family) until new audio is accepted.

source_versioninteger | null

Default: null

minimum: 0 when not null

chunk_indexinteger* required

minimum: 0

silent_secondsnumber* required

minimum: 0

typestring* required

Constant: "audio_exhausted"

{
  "chunk_index": 0,
  "silent_seconds": 0,
  "type": "audio_exhausted"
}

Server audio_pending message

type: "audio_pending"

payload: object

typestring* required

Constant: "audio_pending"

prompt_versioninteger* required

minimum: 1

behaviorstring* required

Values: "replace", "queue"

{
  "type": "audio_pending",
  "prompt_version": 1,
  "behavior": "replace"
}

Server audio_rejected message

type: "audio_rejected"

payload: object

reasonstring* required

Values: "invalid_audio", "content_policy", "preparation_failed", "queue_full", "stale_prompt_version"

typestring* required

Constant: "audio_rejected"

prompt_versioninteger* required

minimum: 1

errorstring* required

{
  "reason": "invalid_audio",
  "type": "audio_rejected",
  "prompt_version": 1,
  "error": "string"
}

Server chunk message

type: "chunk"

payload: object

routestring* required

Constant: "k10-reference"

scheduling_slack_msnumber* required

next_generation_estimate_secondsnumber* required

minimum: 0

requested_duration_secondsinteger* required

minimum: 5 · maximum: 15

trimmed_context_framesinteger* required

minimum: 0

script_end_keyframeboolean | null

Default: null

hard_cutboolean

Default: false

native_playable_frame_countinteger | null

Default: null

exclusiveMinimum: 0 when not null

prompt_versioninteger* required

minimum: 1

buffer_depth_chunksinteger* required

minimum: 0

script_offset_secondsinteger | null

Default: null

minimum: 0 when not null

script_versioninteger | null

Default: null

minimum: 1 when not null

typestring* required

Constant: "chunk"

dispatchobject* required

classified_msnumber* required

minimum: 0

wall_msnumber* required

minimum: 0

phases_msobject* required

Additional properties: number

number

overhead_msnumber* required

minimum: 0

scheduling_lead_msnumber* required

generation_secondsnumber* required

minimum: 0

generated_frame_countinteger* required

exclusiveMinimum: 0

buffer_depth_secondsnumber* required

minimum: 0

presented_frame_countinteger | null

Default: null

exclusiveMinimum: 0 when not null

chunk_indexinteger* required

minimum: 0

playback_secondsnumber* required

exclusiveMinimum: 0

{
  "route": "k10-reference",
  "scheduling_slack_ms": 0,
  "next_generation_estimate_seconds": 0,
  "requested_duration_seconds": 5,
  "trimmed_context_frames": 0,
  "prompt_version": 1,
  "buffer_depth_chunks": 0,
  "type": "chunk",
  "dispatch": {
    "classified_ms": 0,
    "wall_ms": 0,
    "phases_ms": {},
    "overhead_ms": 0
  },
  "scheduling_lead_ms": 0,
  "generation_seconds": 0,
  "generated_frame_count": 1,
  "buffer_depth_seconds": 0,
  "chunk_index": 0,
  "playback_seconds": 5e-324
}

Server chunk_metrics message

type: "chunk_metrics"

payload: object

routestring* required

Constant: "k10-reference"

chunk_consumable_interval_msnumber | null

Default: null

minimum: 0 when not null

typestring* required

Constant: "chunk_metrics"

phases_msobject* required

Additional properties: number

number

unitsstring* required

Constant: "ms"

gaugesobject* required

Additional properties: number

number

chunk_indexinteger* required

minimum: 0

chunk_consumable_ready_msnumber | null

Default: null

minimum: 0 when not null

{
  "route": "k10-reference",
  "type": "chunk_metrics",
  "phases_ms": {},
  "units": "ms",
  "gauges": {},
  "chunk_index": 0
}

Server configured message

type: "configured"

payload: object

prompt_versioninteger* required

minimum: 1

has_initial_imageboolean | null

Default: null

aspect_ratiostring | null

Default: null

When not null:

Values: "16:9", "9:16", "1:1"

typestring* required

Constant: "configured"

accelerationstring | null

Default: null

When not null:

Values: "none", "regular"

resolutionstring | null

Default: null

When not null:

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

enable_safety_checkerboolean* required

memoryinteger | null

Default: null

minimum: 1 when not null · maximum: 50 when not null

audio_bitrateinteger | null

Default: null

When not null:

Values: 96000, 128000, 192000

chunk_durationinteger | null

Default: null

minimum: 5 when not null · maximum: 15 when not null

has_initial_audioboolean | null

Default: null

{
  "prompt_version": 1,
  "type": "configured",
  "enable_safety_checker": true
}

Server deadline_missed message

type: "deadline_missed"

payload: object

late_by_secondsnumber* required

minimum: 0

typestring* required

Constant: "deadline_missed"

chunk_indexinteger* required

minimum: 0

behaviorstring* required

Constant: "freeze_video_and_silence_audio_until_ready"

{
  "late_by_seconds": 0,
  "type": "deadline_missed",
  "chunk_index": 0,
  "behavior": "freeze_video_and_silence_audio_until_ready"
}

Server error message

type: "error"

payload: object

detailobject[] | null

Default: null

Must match at least one of the following:

Alternative 1

object[]

Array item

object

Alternative 2

null

typestring* required

Constant: "error"

prompt_versioninteger | null

Default: null

minimum: 1 when not null

codestring* required

Values: "balance_unavailable", "content_policy", "configuration_timeout", "generation_failed", "generation_timeout", "immutable_settings", "initialization_timeout", "invalid_initial_image", "invalid_initial_audio", "invalid_initial_script", "invalid_input", "invalid_message", "not_configured", "stale_prompt_version"

errorstring* required

{
  "type": "error",
  "code": "balance_unavailable",
  "error": "string"
}

Server pong message

type: "pong"

payload: object

client_tsnumber* required

typestring* required

Constant: "pong"

{
  "client_ts": 0,
  "type": "pong"
}

Server prompt_applied message

type: "prompt_applied"

payload: object

A direction (or script) is admitted; the script fields report scripts.

script_origin_chunk_index is the first chunk generated under the script when that is already fixed (a replacement binds to the next undispatched chunk); an appended script reports None until it starts.

script_origin_chunk_indexinteger | null

Default: null

minimum: 0 when not null

script_modestring | null

Default: null

When not null:

Values: "replace", "append"

script_beatsinteger | null

Default: null

minimum: 1 when not null

typestring* required

Constant: "prompt_applied"

script_queuedinteger | null

Default: null

minimum: 0 when not null

prompt_versioninteger* required

minimum: 1

{
  "type": "prompt_applied",
  "prompt_version": 1
}

Server prompt_pending message

type: "prompt_pending"

payload: object

typestring* required

Constant: "prompt_pending"

prompt_versioninteger* required

minimum: 1

{
  "type": "prompt_pending",
  "prompt_version": 1
}

Server prompt_rejected message

type: "prompt_rejected"

payload: object

reasonstring* required

Values: "content_policy", "preparation_failed", "stale_prompt_version", "invalid_script", "infeasible_timing", "invalid_audio", "invalid_image", "queue_full"

typestring* required

Constant: "prompt_rejected"

prompt_versioninteger* required

minimum: 1

errorstring | null

Default: null

{
  "reason": "content_policy",
  "type": "prompt_rejected",
  "prompt_version": 1
}

Server reference_state message

type: "reference_state"

payload: object

referencesobject[]* required

Array item

object

namestring | null

Default: null

idstring* required

kindstring* required

Values: "image", "audio"

revisioninteger* required

typestring* required

Constant: "reference_state"

prompt_versioninteger* required

{
  "references": [],
  "type": "reference_state",
  "prompt_version": 0
}

Server session_info message

type: "session_info"

payload: object

billable_units_per_playable_secondobject

Default: {"704p":1.15,"544p":1.04,"640p":1.1,"1080p":1.2,"1440p":1.2,"768p":1.2,"480p":1}

Additional properties: number

number

protocol_versioninteger

Constant: 1

fpsinteger

Default: 24

appstring

Constant: "minimax-h3-max-reference-director"

forced_cut_max_segmentsinteger

max_chunk_durationinteger

Default: 15

resolutionsstring[]

Default: ["480p","544p","640p","704p","768p","1080p","1440p"]

Array item

string

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

scriptsboolean

Default: true

dialogue_words_per_secondnumber

Default: 2

accelerationsstring[]

Default: ["none"]

Array item

string

pricing_statusstring

Default: "provisional-experiment"

one_session_per_machineboolean

Default: true

prompt_deck_sizeinteger

Default: 6

max_session_secondsnumber | null

Default: null

continuation_context_framesinteger

Default: 39

session_limit_scopestring

Default: "configured"

typestring* required

Constant: "session_info"

reference_audio_modestring

Default: "sequential-content-windows"

default_pin_audioboolean

Default: false

max_num_inference_stepsinteger

Default: 50

default_chunk_durationinteger

Default: 10

default_num_inference_stepsinteger

Default: 10

min_num_inference_stepsinteger

Default: 1

controller_machine_typestring

Default: "GPU-B300/GPU-B200 (8 GPUs)"

backend_selectionstring

Default: "resident-k10-reference"

min_chunk_durationinteger

Default: 5

default_resolutionstring

Default: "480p"

Values: "480p", "544p", "640p", "704p", "768p", "1080p", "1440p"

{
  "protocol_version": 1,
  "app": "minimax-h3-max-reference-director",
  "type": "session_info"
}

Server session_metrics message

type: "session_metrics"

payload: object

session_wall_msnumber* required

minimum: 0

history_limitinteger* required

minimum: 1

typestring* required

Constant: "session_metrics"

finalboolean

Default: false

unitsstring* required

Constant: "ms"

gaugesobject* required

Additional properties: number

number

history_sizeinteger* required

minimum: 0

phasesobject* required

Additional properties: object

object

p50_msnumber* required

minimum: 0

total_msnumber* required

minimum: 0

p95_msnumber* required

minimum: 0

max_msnumber* required

minimum: 0

countinteger* required

minimum: 0

{
  "session_wall_ms": 0,
  "history_limit": 1,
  "type": "session_metrics",
  "units": "ms",
  "gauges": {},
  "history_size": 0,
  "phases": {}
}

Server stream_exhausted message

type: "stream_exhausted"

payload: object

reasonstring* required

Values: "stopped", "session_limit"

chunksinteger* required

minimum: 0

typestring* required

Constant: "stream_exhausted"

{
  "reason": "stopped",
  "chunks": 0,
  "type": "stream_exhausted"
}

minimax/h3-max/reference-director/start-session

Table of contents

Table of contents

minimax/h3-max/reference-director/start-session

Table of contents

Table of contents

1. Calling the API

2. Authentication

3. Schema

About

1. Calling the API#

Install the client#

Experimental realtime API

Set up the server proxy#

Real-time via WebRTC#

2. Authentication#

API Key#

Protect your API Key

3. Schema#

Media#

Send

Receive

Client messages#

Client configure message

Client ping message

Client prompt message

Client stop message

Server messages#

Server audio_applied message

Server audio_exhausted message

Server audio_pending message

Server audio_rejected message

Server chunk message

Server chunk_metrics message

Server configured message

Server deadline_missed message

Server error message

Server pong message

Server prompt_applied message

Server prompt_pending message

Server prompt_rejected message

Server reference_state message

Server session_info message

Server session_metrics message

Server stream_exhausted message