fal-ai/ltx23-trainer-v2/interpolate

LTX 2.3 Trainer — Keyframe Interpolation (`/interpolate`)

Overview

The `/interpolate` endpoint trains a LoRA for the LTX 2.3 model that generates the video between keyframes. During training, the first and last frames (and optionally a middle frame) of each clip are kept clean and the model learns to generate the in-between motion. At inference you supply a start image and an end image (and optionally a middle image), and the model produces a smooth video connecting them.

Key features:

• Learns first + last (optionally + middle) keyframe → video interpolation.
• Optional middle keyframe for first+middle+last interpolation.
• Trains on plain video clips (plus optional captions) — keyframes are taken from each clip automatically.
• Video-only (no audio is learned).

Dataset Format

Provide a single `.zip` archive (linked via `training_data_url`) of plain videos:

• Videos: `.mp4`, `.mov`, `.webm`, `.mkv`, `.avi`
• Captions: a `.txt` with the same base name as each video (optional but recommended).

Images are rejected (there is nothing to interpolate over time on a still). The first and last frames of each clip (and the middle, if enabled) become the kept keyframes; the model learns to generate the rest. Aim for at least 10 clips. File names must be unique across the archive.

Minimum clip length: with `auto_scale_input` off (the default), each video should have at least `number_of_frames` frames (default 89, ≈ 3.7 s at 24 fps). Shorter clips are dropped during preprocessing; if no clip is long enough, the run fails with no usable training data. Turn on `auto_scale_input` to resample shorter clips to the target frame count instead.

Input Parameters Reference

Dataset

`training_data_url` (required)

Type: `string`

URL to the `.zip` archive of videos.

`trigger_phrase`

Type: `string` Default: `""`

Phrase prepended to captions during training; include it at inference.

Training Parameters

`rank`

Type: `integer` (`8`, `16`, `32`, `64`, `128`) Default: `32`

LoRA capacity.

`number_of_steps`

Type: `integer` Default: `2000` (range `100`–`20000`)

Number of optimization steps.

`learning_rate`

Type: `number` Default: `0.0002`

Optimization step size.

`include_middle_keyframe`

Type: `boolean` Default: `false`

When true, a middle keyframe is also kept (first + middle + last → video) instead of just first + last. When enabled, every validation sample must also provide a middle keyframe image.

Video Configuration

`number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

Frames per training clip. Must satisfy `frames % 8 == 1`; other values are snapped down to the nearest valid count.

`frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

Target frames per second.

`resolution`

Type: `string` (`low`, `medium`, `high`) Default: `medium`

`aspect_ratio`

Type: `string` (`16:9`, `1:1`, `9:16`) Default: `1:1`

`auto_scale_input`

Type: `boolean` Default: `false`

Fit videos to the target frame count and frame rate.

`split_input_into_scenes`

Type: `boolean` Default: `true`

Split long clips into scenes before training.

`split_input_duration_threshold`

Type: `number` Default: `30.0` (range `1.0`–`60.0`)

Duration above which a clip is eligible for scene splitting.

Validation

`validation`

Type: `array` Default: `[]` (max 2 entries)

Validation samples, each an object with:

• `prompt` (`string`) — the text prompt.
• `start_image_url` (`string`, required) — the first-frame keyframe image.
• `end_image_url` (`string`, required) — the last-frame keyframe image.
• `middle_image_url` (`string`, optional) — a middle keyframe image; required when `include_middle_keyframe` is true.

`validation_negative_prompt`

Type: `string` Default: a built-in quality negative prompt.

`validation_number_of_frames`

Type: `integer` Default: `89` (range `9`–`121`)

`validation_frame_rate`

Type: `integer` Default: `24` (range `8`–`60`)

`validation_resolution`

Type: `string` Default: `high`

`validation_aspect_ratio`

Type: `string` Default: `1:1`

`stg_scale`

Type: `number` Default: `1.0` (range `0.0`–`3.0`)

`debug_dataset`

Type: `boolean` Default: `false`

Return an archive of the preprocessed data for inspection.

Outputs

• `lora_file` — the trained LoRA weights (`.safetensors`).
• `config_file` — JSON describing the trigger phrase and training type.
• `video` — combined validation reel (when validation samples were provided).
• `debug_dataset` — preprocessed-data archive, only when `debug_dataset` is enabled.

Billing

A successful run is billed `max(100, number_of_steps)` billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.

How the Training Works

Pipeline Overview

1. Preprocessing — the archive is extracted, clips fit to the resolution bucket (optionally scene-split), and a temporal pattern is built per clip that keeps the first and last (and optionally middle) frames clean.
2. Training — the kept keyframes are held fixed and the model learns to generate the in-between frames. Validation previews run at intervals.
3. Output — the LoRA, config, and validation reel are uploaded.

What Happens to Your Data

• Archive extraction: the `.zip` is unpacked; macOS metadata and hidden files are ignored.
• Video fitting: clips are resized to fill the resolution bucket and center-cropped; with `auto_scale_input` they are resampled to the target frame rate/count.
• Keyframe selection: the first and last frames (and the middle, if enabled) of each clip are kept clean as the keyframes; the rest is the generation target. This happens automatically.
• Captions: the trigger phrase (if set) is prepended.

How Interpolation Training Works

The model is trained to fill in the motion between fixed keyframes. The trained LoRA specializes the base model at producing smooth, plausible transitions consistent with your footage. At inference you supply the start and end images (and optionally a middle image) plus a prompt.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 clips whose start→end motion is representative of the transitions you want learned.
• Clips should contain clear, coherent motion between their first and last frames.
• Keyframes (first/last frames) should be clean and sharp, since they anchor generation.

Caption Best Practices

• Describe the motion/transition plainly, optionally with a trigger phrase.
• Keep captions consistent across clips.

Good caption: `a flower blooming from bud to full bloom` Weak caption: `flower`

Trigger Phrases

• A distinctive trigger phrase helps invoke a particular interpolation style; include it in every caption and at inference.

Inference Format Matching

At inference, supply start and end images (and a middle image if you trained with `include_middle_keyframe`), and use the same caption style and trigger phrase.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/motion_clips.zip",
  "trigger_phrase": "",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "include_middle_keyframe": false,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "validation": [
    {
      "prompt": "a flower blooming",
      "start_image_url": "https://example.com/bud.png",
      "end_image_url": "https://example.com/bloom.png"
    }
  ]
}

Diagnosing Issues

• Transitions are jumpy or implausible: add more clips with smooth motion; ensure keyframes are clean.
• Middle keyframe missing error: when `include_middle_keyframe` is true, every validation sample needs `middle_image_url`.
• Overfitting: fewer steps, lower `rank`, more clips.

Validation Prompt Tips

• Use fresh keyframe images to gauge generalization.
• Keep the prompt aligned with the transition you expect between the keyframes.

Common Pitfalls

• Uploading still images instead of clips (rejected).
• Enabling `include_middle_keyframe` but omitting `middle_image_url` in validation.
• Keyframes that are blurry or unrepresentative.