LTX 2.3 Trainer — Masked Video-to-Video ()
Overview
The endpoint trains a LoRA for the LTX 2.3 model that regenerates only the masked region of a target video, guided by both the kept (unmasked) pixels and a separate reference/control video. It combines inpainting (regenerate just the masked area) with video-to-video control (a reference clip steers what goes there). You provide triplets — a reference clip, a target clip, and a mask — and the LoRA learns the masked, reference-guided transformation.
This is the right endpoint when you want a localized, control-driven edit: replace a region of footage with content guided by a reference, while everything outside the mask stays untouched.
Key features:
- Trains a LoRA that regenerates a masked region using both the kept pixels and a reference video.
- Standard mask convention: WHITE = the region to regenerate, BLACK = keep unchanged.
- Masks can be a single image (every frame) or a video mask (per-frame).
- Video-only (no audio is learned).
Provide a single archive (linked via ) of triplets sharing a base name:
- — the reference / control video that guides the regenerated region.
- — the target video.
- — the mask. Either an image (, , , , ) applied to all frames, or a video mask (per-frame). WHITE marks the region to regenerate; BLACK is kept.
- — optional caption.
Video formats: , , , , . Every target needs both a matching reference and a . A video mask is normalized to its clip's frame count automatically — a shorter mask freeze-holds its last frame, and a longer mask is trimmed. (An image mask is applied to every frame.) File names must be unique across the archive. Aim for at least 10 triplets.
Minimum clip length: with off (the default), the target () and reference () clips of each triplet must each have at least frames (default 89, ≈ 3.7 s at 24 fps). A triplet with either clip too short is skipped, and if none qualifies the request fails (422). Turn on to resample shorter clips instead.
Example layout:
Dataset
(required)
Type:
URL to the archive of reference/target/mask triplets.
Type:
Default:
Phrase prepended to captions during training; include it at inference.
Training Parameters
Type: (, , , , )
Default:
LoRA capacity.
Type:
Default: (range –)
Number of optimization steps. This composite transformation typically benefits from a higher step count, hence the higher default.
Type:
Default:
Optimization step size.
Video Configuration
Type:
Default: (range –)
Frames per training clip. Must satisfy ; other values are snapped down to the nearest valid count.
Type:
Default: (range –)
Target frames per second.
Type: (, , )
Default:
| Resolution | 16:9 | 1:1 | 9:16 |
|---|
| low | 512×288 | 512×512 | 288×512 |
| medium | 768×448 | 768×768 | 448×768 |
| high | 960×544 | 960×960 | 544×960 |
Type: (, , )
Default:
Type:
Default:
Fit videos to the target frame count and frame rate.
Type:
Default:
Off for this mode: scene splitting would desync the reference/target/mask triplet. Provide pre-split clips instead.
Type:
Default: (range –)
Duration threshold for scene splitting (only relevant if splitting were enabled).
Validation
Type:
Default: (max 2 entries)
Validation samples, each an object with:
- () — the text prompt.
- (, required) — the source video; its unmasked region is kept pixel-faithful and the masked region is regenerated.
- (, required) — the mask (image or video). WHITE = regenerate, BLACK = keep.
- (, required) — the reference (control) video guiding the regenerated region.
Type:
Default: a built-in quality negative prompt.
Type:
Default: (range –)
Type:
Default: (range –)
Type:
Default:
Type:
Default:
Type:
Default: (range –)
Type:
Default:
Return an archive of the preprocessed data for inspection.
Outputs
- — the trained LoRA weights ().
- — JSON describing the trigger phrase and training type.
- — combined validation reel (when validation samples were provided).
- — preprocessed-data archive, only when is enabled.
Billing
A successful run is billed billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.
How the Training Works
Pipeline Overview
- Preprocessing — the archive is extracted, reference/target/mask triplets and captions matched, masks converted to the internal convention (a video mask normalized to the target's frame count), and clips fit to the resolution bucket.
- Training — the model regenerates only the masked region of the target, guided by both the kept pixels and the reference video; the unmasked region is held as conditioning. Validation previews run at intervals.
- Output — the LoRA, config, and validation reel are uploaded.
What Happens to Your Data
- Archive extraction: the is unpacked; macOS metadata and hidden files are ignored.
- Triplet matching: each target is paired with its reference, , and optional . Any missing piece causes a clear error.
- Mask handling: your WHITE = edit / BLACK = keep mask is converted to the internal convention automatically. Image masks apply to every frame; video masks are aligned to the target's frame count.
- Video fitting: reference and target are resized to fill the resolution bucket and center-cropped, staying aligned.
- Captions: the trigger phrase (if set) is prepended.
How It Works
This is a LoRA trained to perform a reference-conditioned transformation: instead of generating from text alone, it conditions on a reference clip supplied at inference and produces a transformed result. The trained file specializes the base model on your reference→target mapping. Here it also respects a mask: only the masked region is regenerated, guided by both the surrounding kept pixels and the reference clip. At inference you supply a source video, a mask, and a reference video plus a prompt.
Tips for Getting Good Results
Dataset Quality
- Use at least 10 well-aligned triplets representative of the masked, reference-guided edit you want.
- Make masks cleanly cover the edit area with a small margin so edges blend.
- Reference and target should be aligned so the reference clearly drives the masked region.
Mask Best Practices
- Remember: WHITE = regenerate, BLACK = keep.
- Use an image mask for a static region; a video mask for a moving region.
- Masks are resized to match automatically.
Caption Best Practices
- Describe what should appear in the regenerated region and the overall scene, optionally with a trigger phrase.
- Keep captions consistent across triplets.
Good caption:
Weak caption:
At inference, supply a source video, a mask (WHITE=edit), and a reference video, and use the same caption style and trigger phrase.
Recommended Starting Configuration
Diagnosing Issues
- Regenerated region ignores the reference: ensure triplets are aligned and the reference clearly relates to the masked content; add more examples.
- Wrong area edited: check mask polarity (WHITE = regenerate).
- Mask/clip desync: avoid scene splitting; provide matched, pre-split triplets.
- Overfitting: fewer steps, lower , more triplets.
Validation Prompt Tips
- Use a fresh source/mask/reference set to gauge generalization.
- Describe the content you expect in the masked region.
Common Pitfalls
- Inverted mask polarity.
- Missing a , , or for a triplet.
- Relying on scene splitting (disabled here).
fal-ai/ltx23-trainer-v2/v2v-masked