fal-ai/ltx23-trainer-v2/a2v

LTX 2.3 Trainer — Audio-to-Video (`/a2v`)

Overview

The `/a2v` endpoint trains a LoRA for the LTX 2.3 video model that generates video driven by a start image plus a conditioning audio track. The model conditions on the first frame (the start image) and the audio, and learns to produce a video that matches the sound — for example talking-head / lip-sync-style motion, or audio-reactive animation.

Key features:

• Learns image + audio → video generation.
• The start image is held as the first frame; the audio is frozen as conditioning (not generated, only used to drive the video).
• Optional scene splitting after the start-image/audio assembly.
• Validation previews require both a start image and a conditioning audio track.

Dataset Format

Provide a single `.zip` archive (linked via `training_data_url`) where each training example is a group of files sharing a base name:

• `<name>_start.png` (or `.jpg`/`.jpeg`) — the start image (becomes the first frame). Alternatively a `<name>_start.mp4` (or `.mov`/`.avi`/`.mkv`) whose first frame is used.
• `<name>_audio.wav` (or `.mp3`, `.ogg`, `.m4a`, `.aac`, `.flac`) — the conditioning audio. (If a `<name>_start.mp4` already contains an audio track, a separate audio file is not required.)
• `<name>_end.mp4` (or `.mov`/`.avi`/`.mkv`) — the target video the model learns to produce.
• `<name>.txt` — caption (optional only if a `trigger_phrase` is set; otherwise required).

Each group needs a start image/video, an audio source, and an `_end` target. File names must be unique across the archive. Source clips may have different native sizes; preprocessing resizes and center-crops each sample to the chosen training resolution.

Minimum clip length: with `auto_scale_input` off (the default), each target (`_end`) video must have at least `number_of_frames` frames (default 89, ≈ 3.7 s at 24 fps) — the synthesized training clip inherits the target's length. Shorter targets are silently skipped, and if every one is too short the request fails (422). Turn on `auto_scale_input` to resample instead.

Example layout:

clip01_start.png   clip01_audio.wav   clip01_end.mp4   clip01.txt
clip02_start.png   clip02_audio.mp3   clip02_end.mp4   clip02.txt

Audio is required on every clip. Each group must provide an audio source (a separate `_audio` file, or a `_start` video that carries an audio track). If any group has no usable audio, the request is rejected (HTTP 422).

Input Parameters Reference

Dataset

`training_data_url` (required)

Type: `string`

URL to the `.zip` archive of audio-to-video groups.

`trigger_phrase`

Type: `string` Default: `""`

Phrase prepended to captions during training; include it at inference.

Training Parameters

`rank`

Type: `integer` (`8`, `16`, `32`, `64`, `128`) Default: `32`

LoRA capacity.

`number_of_steps`

Type: `integer` Default: `2000` (range `100`–`20000`)

Number of optimization steps.

`learning_rate`

Type: `number` Default: `0.0002`

Optimization step size.

Video Configuration

`number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Frames per training clip. Must satisfy `frames % 8 == 1`; other values are snapped down to the nearest valid count.

`frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

Target frames per second.

`resolution`

Type: `string` (`low`, `medium`, `high`) Default: `medium`

`aspect_ratio`

Type: `string` (`16:9`, `1:1`, `9:16`) Default: `1:1`

`auto_scale_input`

Type: `boolean` Default: `false`

Fit videos to the target frame count and frame rate.

`split_input_into_scenes`

Type: `boolean` Default: `false`

When true, synthesized training videos above the duration threshold are split into scenes after the start-image/audio assembly. Off by default for this mode.

`split_input_duration_threshold`

Type: `number` Default: `30.0` (range `1.0`–`60.0`)

Duration above which a synthesized clip is eligible for scene splitting.

Audio Configuration

`audio_normalize`

Type: `boolean` Default: `true`

Peak-normalize the conditioning audio for consistent loudness across the dataset.

`audio_preserve_pitch`

Type: `boolean` Default: `true`

Preserve pitch when fitting audio to the video duration (instead of trimming/padding).

Validation

`validation`

Type: `array` Default: `[]` (max 2 entries)

Validation samples, each an object with:

• `prompt` (`string`) — the text prompt.
• `image_url` (`string`, required) — the start image used as the first frame.
• `audio_url` (`string`, required) — the conditioning audio track that drives the video.

`validation_negative_prompt`

Type: `string` Default: a built-in quality negative prompt.

`validation_number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

`validation_frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

`validation_resolution`

Type: `string` Default: `high`

`validation_aspect_ratio`

Type: `string` Default: `1:1`

`stg_scale`

Type: `number` Default: `1.0` (range `0.0`–`3.0`)

`debug_dataset`

Type: `boolean` Default: `false`

Return an archive of the preprocessed data for inspection.

Outputs

• `lora_file` — the trained LoRA weights (`.safetensors`).
• `config_file` — JSON describing the trigger phrase and training type.
• `video` — combined validation reel (when validation samples were provided).
• `debug_dataset` — preprocessed-data archive, only when `debug_dataset` is enabled.

Billing

A successful run is billed `max(100, number_of_steps)` billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.

How the Training Works

Pipeline Overview

1. Preprocessing — the archive is extracted and grouped by base name; each group's start image is placed as the first frame, the conditioning audio is attached, and the target video is fit to the resolution bucket. Groups above the threshold are optionally scene-split afterward.
2. Training — the LoRA trains for `number_of_steps`, conditioning on the first frame and on the (frozen) audio, learning to generate the target video. Validation previews run at intervals.
3. Output — the LoRA, config, and validation reel are uploaded.

What Happens to Your Data

• Archive extraction: the `.zip` is unpacked; macOS `__MACOSX` metadata folders are ignored.
• Grouping: files are grouped by base name into (start image/video, audio, target video, caption). If a start video with an audio track is provided, its first frame becomes the start image and its track becomes the conditioning audio.
• Assembly: the start image is written into the first frame of each target clip, and its audio track is attached. (Audio is normalized and fit to the clip during preprocessing, per `audio_normalize` / `audio_preserve_pitch`.)
• Video fitting: target clips are resized to fill the resolution bucket and center-cropped.
• Captions: the trigger phrase (if set) is prepended.

How Conditioning Works

The first frame (your start image) is held fixed and the audio is used only as a driving signal — it is not regenerated. The LoRA learns to produce video that begins from the supplied image and moves in a way consistent with the audio.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 groups; more variety in speakers/sounds/motions helps.
• Start images should be clean, sharp, and representative of the first frame you want at inference.
• The audio track and target motion should genuinely correspond (e.g. matching speech and lip movement).

Caption Best Practices

• Describe the scene and motion plainly, optionally with a trigger phrase.
• Keep captions consistent in style across groups.

Good caption: `a person speaking to camera in a bright office` Weak caption: `talking`

Trigger Phrases

• Use a distinctive trigger phrase to invoke the behavior cleanly; include it in every caption and at inference.

Inference Format Matching

At inference, supply both a start image and a conditioning audio track, and use the same caption style and trigger phrase you trained with.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/a2v_groups.zip",
  "trigger_phrase": "",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "validation": [
    {
      "prompt": "a person speaking to camera",
      "image_url": "https://example.com/face.png",
      "audio_url": "https://example.com/speech.wav"
    }
  ]
}

Diagnosing Issues

• Motion does not match the audio: ensure your training pairs genuinely correspond; try lowering `learning_rate` if motion looks unstable, or add more well-matched data.
• Color drift / artifacts over time: lower the `learning_rate` and/or add more varied, clean data.
• Overfitting: fewer steps, lower `rank`, more groups.
• Dataset errors: each group needs a start image/video, an audio source, and an `_end` target with matching base names.

Validation Prompt Tips

• Provide a fresh start image and audio track (not from training) to gauge generalization.
• Make the audio length reasonable relative to `validation_number_of_frames` / `validation_frame_rate`.

Common Pitfalls

• Missing audio source for a group (no `_audio` file and no audio track in the start video).
• Start image that does not match the target footage style.
• Forgetting to supply both image and audio at inference.