fal-ai/ltx23-trainer-v2/ic-lora/av2av-masked

LTX 2.3 Trainer — Masked Audio+Video IC-LoRA (/ic-lora/av2av-masked)

Overview

The /ic-lora/av2av-masked endpoint trains an IC-LoRA (In-Context LoRA) for the LTX 2.3 model that regenerates the masked region of a target video (guided by the kept pixels and a video reference) while jointly generating audio from an audio reference. An IC-LoRA is a small adapter that conditions on references supplied at inference rather than generating from text alone; this variant adds a mask to keep the video edit localized and pairs it with reference-driven audio generation. It combines masked video-to-video editing with audio generation. You teach it that mapping by providing triplets — a reference clip (video + audio), a target clip (video + audio), and a mask — and the LoRA learns the masked, reference-guided audiovisual transformation.

Use this endpoint when you want a localized, control-driven edit that also produces sound, for example:

• Replace a region of footage with reference-guided content and generate matching audio.
• Restyle a masked area while jointly generating its accompanying sound.
• Any masked, reference-guided audiovisual edit you can demonstrate with paired examples, masks, and audio.

Key features:

• Trains an IC-LoRA that regenerates a masked video region (guided by kept pixels + a video reference) and jointly generates audio from an audio reference.
• Standard mask convention: WHITE = the region to regenerate, BLACK = keep unchanged.
• Masks can be a single image (every frame) or a video mask (per-frame).
• Validation previews produce a combined video that carries the generated audio.

Dataset Format

Provide a single .zip archive (linked via training_data_url) of triplets sharing a base name, where the reference and target both carry audio:

• <name>_start.<ext> — the reference clip (video + audio). Its video guides the regenerated region; its audio track is the audio reference.
• <name>_end.<ext> — the target clip (video + audio). Its audio is the audio generation target.
• <name>_mask.<ext> — the mask. Either an image (.png, .jpg, .jpeg, .bmp, .webp) applied to all frames, or a video mask (per-frame). WHITE marks the region to regenerate; BLACK is kept.
• <name>.txt — optional caption.

Video formats: .mp4, .mov, .webm, .mkv, .avi. Both the reference and target of every triplet must contain an audio track — silent clips are rejected. Every target needs a matching _start and _mask. A video mask is normalized to its clip's frame count automatically — a shorter mask freeze-holds its last frame, and a longer mask is trimmed. (An image mask is applied to every frame.) File names must be unique across the archive. Aim for at least 10 triplets.

Minimum clip length: with auto_scale_input off (the default), the target (_end) and reference (_start) clips of each triplet must each have at least number_of_frames frames (default 89, ≈ 3.7 s at 24 fps). A triplet with either clip too short is skipped, and if none qualifies the request fails (422). Turn on auto_scale_input to resample shorter clips instead.

Example layout:

sample01_start.mp4   sample01_end.mp4   sample01_mask.png   sample01.txt
sample02_start.mp4   sample02_end.mp4   sample02_mask.mp4   sample02.txt

Input Parameters Reference

Dataset

training_data_url (required)

Type: string

URL to the .zip archive of reference/target/mask triplets (with audio).

trigger_phrase

Type: string Default: ""

Phrase prepended to captions during training; include it at inference.

Training Parameters

rank

Type: integer (8, 16, 32, 64, 128) Default: 32

IC-LoRA capacity.

number_of_steps

Type: integer Default: 2000 (range 100–20000)

Number of optimization steps.

learning_rate

Type: number Default: 0.0002

Optimization step size.

Video Configuration

number_of_frames

Type: integer Default: 89 (range 9–121)

Frames per training clip. Must satisfy frames % 8 == 1; other values are snapped down to the nearest valid count.

frame_rate

Type: integer Default: 24 (range 8–60)

Target frames per second.

resolution

Type: string (low, medium, high) Default: medium

aspect_ratio

Type: string (16:9, 1:1, 9:16) Default: 1:1

auto_scale_input

Type: boolean Default: false

Fit videos to the target frame count and frame rate.

split_input_into_scenes

Type: boolean Default: false

Off for this mode: scene splitting would desync the reference/target/mask triplet. Provide pre-split clips instead.

split_input_duration_threshold

Type: number Default: 30.0 (range 1.0–60.0)

Duration threshold for scene splitting (only relevant if splitting were enabled).

Audio Configuration

audio_normalize

Type: boolean Default: true

Peak-normalize audio for consistent loudness across the dataset.

audio_preserve_pitch

Type: boolean Default: true

Preserve pitch when fitting audio to video duration.

Validation

validation

Type: array Default: [] (max 2 entries)

Validation samples, each an object with:

• prompt (string) — the text prompt.
• video_url (string, required) — the source video; its unmasked region is kept pixel-faithful and the masked region is regenerated.
• mask_url (string, required) — the mask (image or video). WHITE = regenerate, BLACK = keep.
• reference_video_url (string, required) — the reference video (its audio track is also used as the audio reference) guiding the regenerated audio+video.
• reference_audio_url (string, optional) — a separate reference audio; if omitted, the audio is taken from the reference video's own track — in which case that reference video must contain an audio track, or the validation sample is rejected with a 422.

validation_negative_prompt

Type: string Default: a built-in quality negative prompt.

validation_number_of_frames

Type: integer Default: 89 (range 9–121)

validation_frame_rate

Type: integer Default: 24 (range 8–60)

validation_resolution

Type: string Default: high

validation_aspect_ratio

Type: string Default: 1:1

stg_scale

Type: number Default: 1.0 (range 0.0–3.0)

debug_dataset

Type: boolean Default: false

Return an archive of the preprocessed data for inspection.

Outputs

• lora_file — the trained IC-LoRA weights (.safetensors).
• config_file — JSON describing the trigger phrase and training type.
• video — combined validation reel; the preview video carries the generated audio track.
• debug_dataset — preprocessed-data archive, only when debug_dataset is enabled.

Billing

A successful run is billed max(100, number_of_steps) billable units. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units.

How the Training Works

Pipeline Overview

1. Preprocessing — the archive is extracted, reference/target/mask triplets and captions matched, both clips verified to have audio, masks converted to the internal convention (a video mask normalized to the target's frame count), and clips fit to the resolution bucket. The target's audio is extracted from its own track; the reference's audio from the reference clip.
2. Training — the model regenerates the masked video region (guided by kept pixels + the video reference) and jointly generates audio (conditioned on the audio reference). Validation previews run at intervals.
3. Output — the IC-LoRA, config, and validation reel (with generated audio) are uploaded.

What Happens to Your Data

• Archive extraction: the .zip is unpacked; macOS metadata and hidden files are ignored.
• Triplet matching: each <name>_end target is paired with its <name>_start reference, <name>_mask, and optional <name>.txt. Any missing piece causes a clear error.
• Audio requirement: both clips of every triplet must carry audio; silent triplets are rejected.
• Mask handling: your WHITE = edit / BLACK = keep mask is converted to the internal convention automatically. Image masks apply to every frame; video masks are aligned to the target's frame count.
• Video fitting: reference and target are resized to fill the resolution bucket and center-cropped, staying aligned.
• Captions: the trigger phrase (if set) is prepended.

What an IC-LoRA Is

An IC-LoRA performs an in-context transformation using references supplied at inference. Here it respects a mask (only the masked video region is regenerated) and jointly generates audio from an audio reference. The trained file specializes the base model at your masked, reference-guided audiovisual edit. At inference you supply a source video, a mask, and a reference video (and optionally a separate reference audio) plus a prompt, and the LoRA regenerates the masked region together with matching audio.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 well-aligned triplets, each with clean, in-sync audio.
• Make masks cleanly cover the edit area with a small margin so edges blend.
• Reference and target should be aligned so the references clearly drive the regenerated content.

Mask Best Practices

• Remember: WHITE = regenerate, BLACK = keep.
• Use an image mask for a static region; a video mask for a moving region.
• Masks are resized to match automatically.

Caption Best Practices

• Describe the regenerated content, the overall scene, and the desired sound plainly, optionally with a trigger phrase.
• Keep captions consistent across triplets.

Good caption: repl4ce the screen content with rolling waves and ocean sounds Weak caption: screen

Inference Format Matching

At inference, supply a source video, a mask (WHITE=edit), and a reference video (optionally a separate reference audio), and use the same caption style and trigger phrase.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/av2av_masked_triplets.zip",
  "trigger_phrase": "repl4ce",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "validation": [
    {
      "prompt": "repl4ce the screen with ocean waves",
      "video_url": "https://example.com/source.mp4",
      "mask_url": "https://example.com/mask.png",
      "reference_video_url": "https://example.com/ref.mp4"
    }
  ]
}

Diagnosing Issues

• Regenerated region or audio ignores the references: ensure triplets are aligned and the references clearly relate to the regenerated content; add more examples.
• Wrong area edited: check mask polarity (WHITE = regenerate).
• Dataset rejected for silence: both clips of every triplet must contain audio.
• Mask/clip desync: avoid scene splitting; provide matched, pre-split triplets.
• Overfitting: fewer steps, lower rank, more triplets.

Validation Prompt Tips

• Use a fresh source/mask/reference set to gauge generalization.
• Describe the content and sound you expect in the regenerated region.

Common Pitfalls

• Silent reference or target clips (rejected).
• Inverted mask polarity.
• Missing a _start, _end, or _mask for a triplet.
• Relying on scene splitting (disabled here).