fal-ai/ltx23-trainer-v2/t2v

LTX 2.3 Trainer — Text-to-Video (`/t2v`)

Overview

The `/t2v` endpoint trains a LoRA adapter for the LTX 2.3 video model on your own clips, so the model learns a new subject, character, object, or visual style that you can then summon at inference time with a text prompt. There is no conditioning asset — the model generates a full video from text alone.

Key features:

• Learns a subject, object, or style from a small set of your own videos (or images).
• Optional joint audio training: if your clips have sound, the LoRA can learn the matching soundtrack too. Audio is auto-detected (audio is used only if every clip has a track) by default.
• Trains on videos or on still images.
• Optional trigger phrase to activate the learned concept on demand.
• Validation previews generated during training so you can watch the LoRA take shape.

Dataset Format

Provide a single `.zip` archive (linked via `training_data_url`) containing your clips and, optionally, captions:

• Videos: `.mp4`, `.mov`, `.avi`, `.mkv`
• Images: `.png`, `.jpg`, `.jpeg`
• Captions: a `.txt` file with the same base name as each media file (e.g. `clip01.mp4` + `clip01.txt`). Captions are optional but strongly recommended.

The archive must contain only videos OR only images — mixed datasets are rejected. Aim for at least 10 files; more is generally better. Files in subfolders are fine — clips with the same name in different subfolders are kept distinct automatically.

Minimum clip length: with `auto_scale_input` off (the default), each video must already have at least `number_of_frames` frames (default 89, ≈ 3.7 s at 24 fps). Shorter clips are silently skipped, and if every clip is too short the request fails (422, "All training videos are too short to be trainable"). Turn on `auto_scale_input` to resample shorter clips to the target frame count instead. This applies to video datasets only; images are single-frame and always usable.

Input Parameters Reference

Dataset

`training_data_url` (required)

Type: `string`

URL to the `.zip` archive of training clips (and optional captions). See "Dataset Format" above.

`trigger_phrase`

Type: `string` Default: `""`

A phrase prepended to every caption during training. At inference, including this phrase activates the learned concept. Leave empty when teaching a general style you always want applied.

Training Parameters

`rank`

Type: `integer` (one of `8`, `16`, `32`, `64`, `128`) Default: `32`

LoRA capacity. Higher values can capture more detail but use more memory and are more prone to overfitting on small datasets.

`number_of_steps`

Type: `integer` Default: `2000` (range `100`–`20000`, in steps of 100)

How many optimization steps to run. More steps means more learning but also more time and a higher chance of overfitting.

`learning_rate`

Type: `number` Default: `0.0002`

How aggressively the model updates each step. The default is a sensible starting point; raise it cautiously and lower it if results look unstable or degraded.

Video Configuration

`number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Frames per training clip. Must satisfy `frames % 8 == 1` (e.g. 9, 17, 25, 33, 89, 121); other values are snapped down to the nearest valid count automatically. Has no effect on image datasets.

`frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

Target frames per second for training clips. LTX 2.3's native rate is 24.

`resolution`

Type: `string` (one of `low`, `medium`, `high`) Default: `medium`

Training resolution bucket. Combined with `aspect_ratio` this picks the exact pixel size:

`aspect_ratio`

Type: `string` (one of `16:9`, `1:1`, `9:16`) Default: `1:1`

Aspect ratio for training clips. See the table above.

`auto_scale_input`

Type: `boolean` Default: `false`

When true, videos are automatically fit to the target frame count and frame rate. No effect on image datasets.

`split_input_into_scenes`

Type: `boolean` Default: `true`

When true, videos longer than the duration threshold are automatically split into separate scenes (shots) before training.

`split_input_duration_threshold`

Type: `number` Default: `30.0` (range `1.0`–`60.0`)

Videos longer than this many seconds are eligible for scene splitting.

Audio Configuration

`with_audio`

Type: `boolean` or null Default: `null` (auto-detect)

Controls joint audio-video training, all-or-nothing across the dataset. With `null` (default, auto-detect), audio is enabled only when every clip has an audio track — if even one clip is silent, the whole run trains video-only. Set `true` to force audio training (the request fails with a 422 if any clip lacks an audio track), or `false` to ignore audio even when present.

`audio_normalize`

Type: `boolean` Default: `true`

Peak-normalize audio for consistent loudness across the dataset.

`audio_preserve_pitch`

Type: `boolean` Default: `true`

When fitting audio to video duration, keep the original pitch (time-stretch) instead of trimming or padding.

Validation

Validation generates preview videos at intervals during training so you can monitor progress. Previews are single-stage approximations and will not perfectly match final production-quality inference.

`validation`

Type: `array` Default: `[]` (max 2 entries)

A list of validation prompts. Each entry is an object with:

• `prompt` (`string`) — the text prompt to preview.

`validation_negative_prompt`

Type: `string` Default: a long built-in quality negative prompt.

Negative prompt applied to all validation previews.

`validation_number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Frame count for validation videos. Snapped to `frames % 8 == 1`.

`validation_frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

Frame rate for validation videos.

`validation_resolution`

Type: `string` Default: `high`

Resolution bucket for validation videos (same buckets as `resolution`).

`validation_aspect_ratio`

Type: `string` Default: `1:1`

Aspect ratio for validation videos.

`stg_scale`

Type: `number` Default: `1.0` (range `0.0`–`3.0`)

Spatio-Temporal Guidance scale for validation previews. `0.0` disables it; `1.0` is recommended.

Outputs

• `lora_file` — the trained LoRA weights (`.safetensors`). This is the main artifact.
• `config_file` — a small JSON describing the trigger phrase and training type, for setting up inference.
• `video` — a combined preview reel of all validation samples (when validation prompts were provided).
• `debug_dataset` — a downloadable archive of your preprocessed data, only present when `debug_dataset` is enabled.

`debug_dataset`

Type: `boolean` Default: `false`

When enabled, returns an archive of the preprocessed training data so you can verify your videos, images, and captions were processed correctly before committing to a longer run.

Billing

A successful run is billed `max(100, number_of_steps)` billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.

How the Training Works

Pipeline Overview

1. Preprocessing
   1. Your archive is downloaded and extracted.
   2. Media is matched to captions by file name.
   3. Each clip is resized and cropped to the chosen resolution bucket and (optionally) split into scenes.
   4. Audio is detected (or you force it on/off) and prepared.
2. Training — the LoRA is trained for `number_of_steps`, running validation previews at intervals.
3. Output — the final LoRA, config, and validation reel are uploaded.

What Happens to Your Data

• Archive extraction: the `.zip` is unpacked; macOS `__MACOSX` metadata folders are ignored.
• File matching: each media file is paired with the `.txt` of the same base name. Files without a caption train on an empty caption (or just the trigger phrase, if set).
• Video fitting: each clip is resized to fill the target resolution while keeping aspect ratio, then center-cropped to the exact bucket dimensions. With `auto_scale_input`, clips are resampled to the target frame rate and frame count.
• Scene splitting: with `split_input_into_scenes`, clips longer than the threshold are cut into separate shots, each becoming its own training sample.
• Captions: the trigger phrase (if any) is prepended to each caption.
• Audio: when audio is enabled, each clip's soundtrack is prepared (normalized and pitch-fit as configured) and learned jointly with the video.

LoRA Training

A LoRA is a small set of adapter weights layered on top of the frozen base model. Training only updates these adapters, so the result is a compact file you load alongside the base LTX 2.3 model at inference. The base model's general capabilities are preserved; the LoRA nudges it toward your subject or style.

Audio Support

If all your videos have sound and audio is enabled, the LoRA learns the audio-video relationship, so generated videos can come with a matching soundtrack. Audio is all-or-nothing across the dataset: if any clip is silent, the whole run trains video-only. Leave `with_audio` at its default to let the trainer enable audio only when every clip has a track.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 clips; 20–50 varied clips often work better for a robust concept.
• Keep quality high: sharp, well-lit, representative footage. The LoRA reproduces whatever artifacts are common in your data.
• For a subject/character/object, show it from multiple angles and in different contexts.
• For a style, include varied content all sharing the same look.

Caption Best Practices

• Describe what is actually in each clip, plainly and specifically.
• Subject/object training: caption the subject with your trigger phrase plus a plain description, e.g. `tronl0g0 a glowing blue logo spinning on a desk`.
• Style training: describe the content and let the style be learned implicitly; optionally use a style trigger phrase.

Good caption: `a red sports car drives along a coastal highway at sunset` Weak caption: `car` (too sparse to anchor the concept)

Trigger Phrases

• For a specific subject/object/character, pick a rare, distinctive trigger token (e.g. `tronlog0`) so it does not collide with words the model already knows, and include it in every caption.
• For an always-on style, you can skip the trigger phrase entirely.

Scene Splitting and Captions

If `split_input_into_scenes` is on, one long video becomes several shorter clips that all share the original caption. If different parts of the video show different things, the shared caption may not describe each split accurately. For precise captions, pre-split your clips and disable scene splitting.

Inference Format Matching

Prompt the trained LoRA the same way you captioned it. If you trained with a trigger phrase, include that phrase at inference. If your captions were short and descriptive, short descriptive prompts will behave most predictably.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/my_dataset.zip",
  "trigger_phrase": "tronlog0",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "validation": [
    { "prompt": "tronlog0 spinning slowly on a dark background" }
  ]
}

Diagnosing Issues

• Overfitting (validation previews look like exact copies of training clips, or ignore the prompt): reduce `number_of_steps`, lower `rank`, or add more varied data.
• Underfitting (the concept barely appears): increase `number_of_steps`, raise `rank`, or improve caption quality.
• No effect at all: confirm the trigger phrase is in your captions and used in validation prompts.
• Training fails on the dataset: check that the archive contains only videos or only images, and that captions match media base names.

Validation Prompt Tips

• Use prompts in the same style as your captions, including the trigger phrase.
• Pick prompts that exercise what you care about (e.g. the subject in a new setting) so you can judge generalization, not memorization.

Common Pitfalls

• Mixing images and videos in one archive (not supported).
• Expecting audio when your clips are silent.
• Forgetting the trigger phrase at inference.