fal Assets is now live!
Docs
Log-inSign-up

fal-ai/ltx23-trainer-v2/av2av

Train a LoRA for a joint audio+video transformation, conditioned on a reference clip (its video and audio) to produce a matching target clip.
Training
Commercial use
Schema
LLMs
PlaygroundAPI

Table of contents

About

Av2Av

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install --save @fal-ai/client

Setup your API Key#

Set FAL_KEY as an environment variable in your runtime.

export FAL_KEY="YOUR_API_KEY"

Submit a request#

The client API handles the API submit protocol. It will handle the request status updates and return the result when the request is completed.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/ltx23-trainer-v2/av2av", {
  input: {
    training_data_url: ""
  },
  logs: true,
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") {
      update.logs.map((log) => log.message).forEach(console.log);
    }
  },
});
console.log(result.data);
console.log(result.requestId);

2. Authentication#

The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.

API Key#

In case your app is running in an environment where you cannot set environment variables, you can set the API Key manually as a client configuration.
import { fal } from "@fal-ai/client";

fal.config({
  credentials: "YOUR_FAL_KEY"
});

3. Queue#

Submit a request#

The client API provides a convenient way to submit requests to the model.

import { fal } from "@fal-ai/client";

const { request_id } = await fal.queue.submit("fal-ai/ltx23-trainer-v2/av2av", {
  input: {
    training_data_url: ""
  },
  webhookUrl: "https://optional.webhook.url/for/results",
});

Fetch request status#

You can fetch the status of a request to check if it is completed or still in progress.

import { fal } from "@fal-ai/client";

const status = await fal.queue.status("fal-ai/ltx23-trainer-v2/av2av", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b",
  logs: true,
});

Get the result#

Once the request is completed, you can fetch the result. See the Output Schema for the expected result format.

import { fal } from "@fal-ai/client";

const result = await fal.queue.result("fal-ai/ltx23-trainer-v2/av2av", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b"
});
console.log(result.data);
console.log(result.requestId);

4. Files#

Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.

Data URI (base64)#

You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.

Hosted files (URL)#

You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.

Uploading files#

We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.

import { fal } from "@fal-ai/client";

const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);

Read more about file handling in our file upload guide.

5. Schema#

Input#

training_data_url string* required

URL to a .zip archive of your training data. The exact file layout depends on the training mode — see this endpoint's documentation for the required structure. You can include a .txt caption alongside each media file (same base name).

rank RankEnum

The rank of the LoRA adaptation. Higher values increase capacity but use more memory. Default value: "32"

Possible enum values: 8, 16, 32, 64, 128

number_of_steps integer

The number of training steps. Default value: 2000

learning_rate float

Learning rate for optimization. Default value: 0.0002

number_of_frames integer

Number of frames per training sample. Must satisfy frames % 8 == 1 (e.g., 1, 9, 17, 25, 33, 41, 49, 57, 65, 73, 81, 89, 97). Default value: 89

frame_rate integer

Target frames per second for the training video (LTX-2.3 native is 24). Default value: 24

resolution ResolutionEnum

Resolution to use for training. Higher resolutions require more memory. Default value: "medium"

Possible enum values: low, medium, high

aspect_ratio AspectRatioEnum

Aspect ratio to use for training. Default value: "1:1"

Possible enum values: 16:9, 1:1, 9:16

trigger_phrase string

A phrase that will trigger the LoRA style. Will be prepended to captions during training. Default value: ""

auto_scale_input boolean

If true, videos will be automatically scaled to the target frame count and fps. This option has no effect on image datasets.

split_input_into_scenes boolean

Off for av2av: the paired target/reference prep ignores scene splitting (it would desync the _start/_end pair). Provide pre-split clips instead.

split_input_duration_threshold float

The duration threshold in seconds. If a video is longer than this, it will be split into scenes. Default value: 30

debug_dataset boolean

When enabled, the trainer returns a downloadable archive of your preprocessed training data for manual inspection. Use this to verify that your videos, images, and captions were processed correctly before committing to a full training run.

audio_normalize boolean

Peak-normalize audio for consistent levels across the dataset. Default value: true

audio_preserve_pitch boolean

Preserve pitch when fitting audio to video duration (vs trimming). Default value: true

validation list<AV2AVValidation>

A list of validation prompts, each with a reference video (+audio).

validation_negative_prompt string

A negative prompt to use for validation. Note: validation previews are single-stage approximations of the production two-stage (distilled) inference, so preview quality and guidance differ from final inference. Default value: "blurry, out of focus, overexposed, underexposed, low contrast, washed out colors, excessive noise, grainy texture, poor lighting, flickering, motion blur, distorted proportions, unnatural skin tones, deformed facial features, asymmetrical face, missing facial features, extra limbs, disfigured hands, wrong hand count, artifacts around text, inconsistent perspective, camera shake, incorrect depth of field, background too sharp, background clutter, distracting reflections, harsh shadows, inconsistent lighting direction, color banding, cartoonish rendering, 3D CGI look, unrealistic materials, uncanny valley effect, incorrect ethnicity, wrong gender, exaggerated expressions, wrong gaze direction, mismatched lip sync, silent or muted audio, distorted voice, robotic voice, echo, background noise, off-sync audio, incorrect dialogue, added dialogue, repetitive speech, jittery movement, awkward pauses, incorrect timing, unnatural transitions, inconsistent framing, tilted camera, flat lighting, inconsistent tone, cinematic oversaturation, stylized filters, or AI artifacts."

validation_number_of_frames integer

The number of frames in validation videos. Default value: 89

validation_frame_rate integer

Target frames per second for validation videos (LTX-2.3 native is 24). Default value: 24

validation_resolution ValidationResolutionEnum

The resolution to use for validation. Default value: "high"

Possible enum values: low, medium, high

validation_aspect_ratio ValidationAspectRatioEnum

The aspect ratio to use for validation. Default value: "1:1"

Possible enum values: 16:9, 1:1, 9:16

stg_scale float

STG (Spatio-Temporal Guidance) scale. 0.0 disables STG. Recommended value is 1.0. Default value: 1

reference_downscale_factor integer

Spatially downscale the reference VIDEO by this factor before encoding (coarse/low-res control proxy -> full-res output). 1 = same resolution (default). Reference width/height and width/height divided by the factor must be divisible by 32. Default value: 1

reference_temporal_scale_factor integer

Temporally downsample the reference VIDEO by this factor (lower fps) before encoding. 1 = same fps (default). Default value: 1

{
  "training_data_url": "",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "number_of_frames": 89,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "1:1",
  "trigger_phrase": "",
  "auto_scale_input": false,
  "split_input_duration_threshold": 30,
  "audio_normalize": true,
  "audio_preserve_pitch": true,
  "validation": [],
  "validation_negative_prompt": "blurry, out of focus, overexposed, underexposed, low contrast, washed out colors, excessive noise, grainy texture, poor lighting, flickering, motion blur, distorted proportions, unnatural skin tones, deformed facial features, asymmetrical face, missing facial features, extra limbs, disfigured hands, wrong hand count, artifacts around text, inconsistent perspective, camera shake, incorrect depth of field, background too sharp, background clutter, distracting reflections, harsh shadows, inconsistent lighting direction, color banding, cartoonish rendering, 3D CGI look, unrealistic materials, uncanny valley effect, incorrect ethnicity, wrong gender, exaggerated expressions, wrong gaze direction, mismatched lip sync, silent or muted audio, distorted voice, robotic voice, echo, background noise, off-sync audio, incorrect dialogue, added dialogue, repetitive speech, jittery movement, awkward pauses, incorrect timing, unnatural transitions, inconsistent framing, tilted camera, flat lighting, inconsistent tone, cinematic oversaturation, stylized filters, or AI artifacts.",
  "validation_number_of_frames": 89,
  "validation_frame_rate": 24,
  "validation_resolution": "high",
  "validation_aspect_ratio": "1:1",
  "stg_scale": 1,
  "reference_downscale_factor": 1,
  "reference_temporal_scale_factor": 1
}

Output#

video File

Combined validation video preview (video-producing endpoints), if any.

audio File

Combined validation audio preview for the audio-only-output endpoints (/v2a, /a2a, /t2a, /audio-extend-prefix, /audio-extend-suffix, /audio-inpaint), if any.

lora_file File* required

URL to the trained LoRA weights (.safetensors).

config_file File* required

Configuration used for setting up inference endpoints.

debug_dataset File

Downloadable archive of the preprocessed training data, when debug_dataset is enabled.

{
  "lora_file": {
    "url": "",
    "content_type": "image/png",
    "file_name": "z9RV14K95DvU.png",
    "file_size": 4404019
  },
  "config_file": {
    "url": "",
    "content_type": "image/png",
    "file_name": "z9RV14K95DvU.png",
    "file_size": 4404019
  }
}

Other types#

OutpaintValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the full-canvas video whose surround should be outpainted.

AV2AVValidation#

prompt string* required

The prompt to use for validation.

reference_video_url string* required

URL to the reference video (its audio track is also used as the audio reference) that conditions the generated audio+video.

reference_audio_url string

Optional URL to a separate reference audio. If omitted, the audio is taken from reference_video_url's own track.

A2AValidation#

prompt string* required

The prompt to use for validation.

reference_audio_url string* required

URL to the reference audio that conditions the generated audio.

T2VValidation#

prompt string* required

The prompt to use for validation.

V2VMaskedValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the source video. Its UNMASKED region is kept pixel-faithful and the masked region is regenerated.

mask_url string* required

URL to the mask (image or video). Standard convention: WHITE marks the region to regenerate/edit, BLACK is kept unchanged.

reference_video_url string* required

URL to the reference (control) video that guides the regenerated region.

A2VValidation#

prompt string* required

The prompt to use for validation.

image_url string* required

URL to the start image used as the first frame (a2v is image+audio -> video, so the model conditions on this frame and the audio rather than generating the first frame).

audio_url string* required

URL to the conditioning audio track that drives the generated video.

V2VValidation#

prompt string* required

The prompt to use for validation.

reference_video_url string* required

URL to reference video for IC-LoRA validation. This is the input video that will be transformed.

InterpolateValidation#

prompt string* required

The prompt to use for validation.

start_image_url string* required

URL to the first-frame keyframe image.

end_image_url string* required

URL to the last-frame keyframe image.

middle_image_url string

Optional URL to a middle keyframe image; required when the run sets include_middle_keyframe=true (first+middle+last interpolation).

File#

url string* required

The URL where the file can be downloaded from.

content_type string

The mime type of the file.

file_name string

The name of the file. It will be auto-generated if not provided.

file_size integer

The size of the file in bytes.

AV2AVMaskedValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the source video. Its UNMASKED region is kept pixel-faithful and the masked region is regenerated.

mask_url string* required

URL to the mask (image or video). WHITE marks the region to regenerate, BLACK is kept unchanged.

reference_video_url string* required

URL to the reference video (its audio track is also used as the audio reference) guiding the regenerated audio+video.

reference_audio_url string

Optional URL to a separate reference audio. If omitted, the audio is taken from reference_video_url's own track.

ExtendValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the video the model should temporally extend.

T2AValidation#

prompt string* required

The prompt to use for validation.

AudioInpaintValidation#

prompt string* required

The prompt to use for validation.

audio_url string* required

URL to the audio clip to inpaint.

time_ranges list<array>* required

Time ranges (in seconds, [start, end]) to REGENERATE/inpaint; the rest of the audio is kept. e.g. [[1.0, 2.0]].

I2VValidation#

prompt string* required

The prompt to use for validation.

image_url string* required

An image to use as the first frame for image-to-video validation.

InpaintValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the source video to inpaint.

mask_url string* required

URL to the inpainting mask (image or video). Standard convention: WHITE marks the region to regenerate/edit, BLACK is kept unchanged. Resolution need not match the video; it is resized automatically.

V2AValidation#

prompt string* required

The prompt to use for validation.

video_url string* required

URL to the silent video the model should generate audio for.

AudioExtendValidation#

prompt string* required

The prompt to use for validation.

audio_url string* required

URL to the audio clip the model should temporally extend.

Related Models

LTX 2.3 Trainer (V2) - Audio+Video Reference Transformation Training API Docs | fal