fal-ai/ltx23-trainer-v2/extend-prefix

LTX 2.3 Trainer — Forward Video Extension (`/extend-prefix`)

Overview

The `/extend-prefix` endpoint trains a LoRA for the LTX 2.3 model that continues a video forward in time. During training, the first N frames of each clip are kept as a clean "prefix" and the model learns to generate everything that follows. At inference you supply an opening clip and the model produces the continuation in the learned style.

Key features:

• Learns to extend video forward from a clean opening (prefix) window.
• Optional joint audio extension: when enabled, audio is continued in sync with the video from the same opening window.
• Trains on plain videos (plus optional captions) — the prefix is carved from each clip automatically.
• Validation previews continue a supplied clip forward.

Dataset Format

Provide a single `.zip` archive (linked via `training_data_url`) of plain videos:

• Videos: `.mp4`, `.mov`, `.avi`, `.mkv`
• Captions: a `.txt` with the same base name as each video (optional but recommended).

Images are rejected (there is nothing to extend over time on a still). If you enable audio extension, every training clip — and every validation clip — must contain an audio track. Aim for at least 10 clips. Files in subfolders are fine — clips with the same name in different subfolders are kept distinct automatically.

Minimum clip length: with `auto_scale_input` off (the default), each video must already have at least `number_of_frames` frames (default 89, ≈ 3.7 s at 24 fps). Shorter clips are silently skipped, and if every clip is too short the request fails (422, "All training videos are too short to be trainable"). Turn on `auto_scale_input` to resample shorter clips to the target frame count instead.

With `with_audio` enabled, audio is required on every clip. Joint audio extension requires every training clip and every validation clip to contain an audio track; if any lacks one, the request is rejected (HTTP 422). With `with_audio` off, audio is ignored and the output is silent.

Input Parameters Reference

Dataset

`training_data_url` (required)

Type: `string`

URL to the `.zip` archive of videos.

`trigger_phrase`

Type: `string` Default: `""`

Phrase prepended to captions during training; include it at inference.

Training Parameters

`rank`

Type: `integer` (`8`, `16`, `32`, `64`, `128`) Default: `32`

LoRA capacity.

`number_of_steps`

Type: `integer` Default: `2000` (range `100`–`20000`)

Number of optimization steps.

`learning_rate`

Type: `number` Default: `0.0002`

Optimization step size.

`conditioning_frames`

Type: `integer` Default: `9` (range `9`–`121`)

Number of leading frames kept as the clean prefix the model continues from. Must be `≡ 1 (mod 8)` (e.g. 9, 17, 25). It must be short enough that there is a continuation left to generate — i.e. the prefix cannot cover the whole `number_of_frames` (or `validation_number_of_frames`) clip.

Video Configuration

`number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Frames per training clip. Must satisfy `frames % 8 == 1` (other values are snapped down to the nearest valid count) and must be larger than the prefix window.

`frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

Target frames per second.

`resolution`

Type: `string` (`low`, `medium`, `high`) Default: `medium`

`aspect_ratio`

Type: `string` (`16:9`, `1:1`, `9:16`) Default: `1:1`

`auto_scale_input`

Type: `boolean` Default: `false`

Fit videos to the target frame count and frame rate.

`split_input_into_scenes`

Type: `boolean` Default: `true`

Split long clips into scenes before training.

`split_input_duration_threshold`

Type: `number` Default: `30.0` (range `1.0`–`60.0`)

Duration above which a clip is eligible for scene splitting.

Audio Configuration

`with_audio`

Type: `boolean` Default: `false`

Set `true` to jointly extend audio and video — the model continues both from the same opening window so they stay in sync. Requires every training clip and every validation clip to contain an audio track. Default (`false`) produces a video-only, silent extension.

`audio_normalize`

Type: `boolean` Default: `true`

Peak-normalize audio for consistent loudness (used when audio extension is enabled).

`audio_preserve_pitch`

Type: `boolean` Default: `true`

Preserve pitch when fitting audio to video duration.

Validation

`validation`

Type: `array` Default: `[]` (max 2 entries)

Validation samples, each an object with:

• `prompt` (`string`) — the text prompt.
• `video_url` (`string`, required) — a video to extend forward. Its first `conditioning_frames` frames are used as the prefix.

The validation clip must be at least `conditioning_frames` long (at the validation frame rate).

`validation_negative_prompt`

Type: `string` Default: a built-in quality negative prompt.

`validation_number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Must be larger than the prefix window so there is a continuation to preview.

`validation_frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

`validation_resolution`

Type: `string` Default: `high`

`validation_aspect_ratio`

Type: `string` Default: `1:1`

`stg_scale`

Type: `number` Default: `1.0` (range `0.0`–`3.0`)

`debug_dataset`

Type: `boolean` Default: `false`

Return an archive of the preprocessed data for inspection.

Outputs

• `lora_file` — the trained LoRA weights (`.safetensors`).
• `config_file` — JSON describing the trigger phrase and training type.
• `video` — combined validation reel (when validation samples were provided).
• `debug_dataset` — preprocessed-data archive, only when `debug_dataset` is enabled.

Billing

A successful run is billed `max(100, number_of_steps)` billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.

How the Training Works

Pipeline Overview

1. Preprocessing — the archive is extracted, clips fit to the resolution bucket (optionally scene-split), and audio prepared when audio extension is enabled.
2. Training — for each clip, the first `conditioning_frames` frames are held clean as the prefix and the model learns to generate the rest. With audio extension on, the matching opening audio window conditions the audio so it continues in sync. Validation previews run at intervals.
3. Output — the LoRA, config, and validation reel are uploaded.

What Happens to Your Data

• Archive extraction: the `.zip` is unpacked; macOS `__MACOSX` metadata folders are ignored.
• Video fitting: clips are resized to fill the resolution bucket and center-cropped; with `auto_scale_input` they are resampled to the target frame rate/count.
• Prefix carving: the leading `conditioning_frames` frames of each clip are kept clean as conditioning; the remainder is the generation target. This happens automatically — you do not need to mark anything.
• Audio window: when audio extension is on, the matching opening seconds of audio are used as the audio prefix so audio and video stay aligned.
• Captions: the trigger phrase (if set) is prepended.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 clips that contain the kind of forward motion/continuation you want learned.
• Clips should be long enough that there is a meaningful continuation beyond the prefix window.
• For audio extension, ensure every clip carries clean, in-sync audio.

Caption Best Practices

• Describe the action/continuation plainly, optionally with a trigger phrase.
• Keep captions consistent across clips.

Good caption: `a car accelerating down a straight road` Weak caption: `car`

Trigger Phrases

• Use a distinctive trigger phrase to invoke a particular continuation style; include it in every caption and at inference.

Inference Format Matching

At inference, supply an opening clip at least `conditioning_frames` long, use the same caption style and trigger phrase, and match the audio setting (audio-on extension expects an audio track).

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/clips.zip",
  "trigger_phrase": "",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "conditioning_frames": 9,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "with_audio": false,
  "validation": [
    { "prompt": "a car accelerating down a road", "video_url": "https://example.com/opening.mp4" }
  ]
}

Diagnosing Issues

• Continuation drifts or stops matching the prefix: add more representative clips; try a slightly longer `conditioning_frames` for more context.
• Validation rejected for short clip: provide a validation clip at least `conditioning_frames` long, or lower `conditioning_frames`.
• Audio extension rejected: when `with_audio` is true, every training and validation clip must have an audio track.
• Overfitting: fewer steps, lower `rank`, more clips.

Validation Prompt Tips

• Use a fresh opening clip to gauge generalization.
• Keep the prompt aligned with the continuation you expect.

Common Pitfalls

• `conditioning_frames` not `≡ 1 (mod 8)` (use 9, 17, 25, ...).
• A prefix window so long it covers the whole clip (nothing left to generate).
• Enabling audio extension with silent clips.