minimax/h3/flf2v/trainer

MiniMax H3 Trainer — First-Last-Frame-to-Video-Audio (/first-last-frame-to-video-audio)

Overview

The /first-last-frame-to-video-audio endpoint trains a LoRA adapter for the MiniMax H3 (Hailuo-03) video+audio model with the full keyframe-signature mechanism: per training sample, the conditioning is drawn from {first frame, last frame, first+last frames, none} with configurable probabilities. This is the endpoint that teaches H3's first-to-last mode — generating a video that starts on one image and lands on another — while also covering first-frame-only, last-frame-only, and unconditioned generation.

Key features:

• Trains all of H3's keyframe conditioning signatures in one run, with a probability knob per mode.
• Joint video + audio training: clips with an audio track teach the LoRA the matching sound; clips without audio train against silence. Audio handling is automatic and per-clip.
• Captions are optional — clips without a .txt train against the trigger phrase (or a blank prompt).
• Supports sidecar keyframe images: <stem>.first_frame.png and <stem>.last_frame.png next to a clip replace the clip's own first/last frames as conditioning images.

Dataset Format

Provide a single .zip archive (linked via training_data_url) containing your clips and, optionally, captions and sidecar keyframes:

• Videos: .mp4, .mov, .avi, .mkv
• Images: .png, .jpg, .jpeg
• Captions (optional): a .txt file with the same base name as each media file (e.g. clip01.mp4 + clip01.txt). Clips without a caption train on the trigger phrase, or a blank prompt if none is set.
• Sidecar keyframes (optional): clip01.first_frame.png and/or clip01.last_frame.png (also .jpg/.jpeg, any case) next to clip01.mp4 replace the clip's own first/last frames as conditioning images. Either, both, or neither may be present per clip; missing ones fall back to the clip's actual frames.

The archive must contain only videos OR only images — mixed datasets are rejected (sidecar keyframes don't count as dataset images). Aim for at least 10 files; more is generally better. Files in subfolders are fine — clips with the same name in different subfolders are kept distinct, and each sidecar binds to the clip in its own folder.

Minimum clip length: with auto_scale_input off (the default), each video must already have at least number_of_frames frames (default 73, ≈ 3.0 s at 24 fps). Shorter clips are silently skipped, and if every clip is too short the request fails (422). Turn on auto_scale_input to resample shorter clips to the target frame count instead.

Input Parameters Reference

Dataset

training_data_url (required)

Type: string

URL to the .zip archive of training clips (and optional captions/sidecars). See "Dataset Format" above.

trigger_phrase

Type: string Default: ""

A phrase prepended to every caption during training (and used as the whole caption for clips without one). At inference, including this phrase activates the learned concept.

Conditioning

Per training sample, one conditioning signature is drawn: both frames with probability first_last_frame_conditioning_p, else last frame only with last_frame_conditioning_p, else first frame only with first_frame_conditioning_p, else no conditioning (pure text-to-video) with the remaining probability mass. The three probabilities must sum to at most 1.0.

first_frame_conditioning_p

Type: number Default: 0.2 (range 0.0–1.0)

Probability of conditioning on the first frame only.

last_frame_conditioning_p

Type: number Default: 0.2 (range 0.0–1.0)

Probability of conditioning on the last frame only.

first_last_frame_conditioning_p

Type: number Default: 0.4 (range 0.0–1.0)

Probability of conditioning on BOTH the first and last frames — the API's first-to-last keyframe mode. With the defaults, 20% of samples train unconditioned.

Training Parameters

rank

Type: integer (one of 8, 16, 32, 64, 128) Default: 32

LoRA capacity. Higher values can capture more detail but use more memory and are more prone to overfitting on small datasets.

number_of_steps

Type: integer Default: 2000 (range 1–6000)

How many optimization steps to run. More steps means more learning but also more time and a higher chance of overfitting. Note that billing has a 100-step floor (see Billing).

learning_rate

Type: number Default: 0.0002

How aggressively the model updates each step. The default is a sensible starting point; raise it cautiously and lower it if results look unstable or degraded.

Video Configuration

number_of_frames

Type: integer Default: 73 (range 22–124)

Frames per training clip. H3's video VAE requires frames % 17 == 5 — valid counts are 22, 39, 56, 73, 90, 107, 124. Other values are adjusted with 17 * (value // 17) + 5 and clamped to the supported range rather than using nearest-value rounding; for example, 100 becomes 90 while 123 becomes 124. Has no effect on image datasets.

frame_rate

Type: integer Default: 24 (range 8–60)

Target frames per second for training clips. H3's native output rate is a fixed 24 fps, so keep the default unless you have a specific reason to deviate.

resolution

Type: string (one of low, medium, high) Default: medium

Training resolution bucket. Combined with aspect_ratio this picks the exact pixel size:

H3's native output is 2K; LoRA training at these sub-native resolutions is standard practice across the video-trainer family and carries over to full-resolution inference.

aspect_ratio

Type: string (one of 21:9, 16:9, 4:3, 1:1, 3:4, 9:16) Default: 16:9

Aspect ratio for training clips — the same set the H3 API supports. See the table above.

auto_scale_input

Type: boolean Default: false

When true, videos are automatically fit to the target frame count and frame rate. No effect on image datasets.

split_input_into_scenes

Type: boolean Default: true

When true, videos longer than the duration threshold are automatically split into separate scenes (shots) before training. Note: sidecar keyframes are dropped for clips that get split (the keyframes can't be matched to a specific scene) — disable this if your dataset uses sidecars on long clips. Splitting also weakens last-frame semantics: each scene's "last frame" is a cut point, not your intended endpoint.

split_input_duration_threshold

Type: number Default: 30.0 (range 1.0–60.0)

Videos longer than this many seconds are eligible for scene splitting.

Outputs

• lora_file — the trained LoRA weights (.safetensors). This is the main artifact.
• config_file — a small JSON containing the trigger phrase as instance_prompt and training_type set to fl2va, for setting up inference.
• debug_dataset — a downloadable archive of your preprocessed data, only present when debug_dataset is enabled.

The endpoint does not accept validation samples and does not return preview inference; evaluate the LoRA by loading it into the inference endpoint.

debug_dataset

Type: boolean Default: false

When enabled, returns an archive of the preprocessed training data so you can verify your videos, captions, and keyframes were processed correctly before committing to a longer run. The archive includes preprocessing_report.json, a bounded summary of any dataset fallbacks.

strict_dataset

Type: boolean Default: false

Preprocessing always logs a summary when it uses a blank prompt, substitutes clip endpoints for missing first/last sidecars, or substitutes silence for an unreadable or absent video soundtrack. Enable this option to reject those fallbacks with a 422 instead. Captions longer than 4,096 tokenizer tokens are rejected with a 422 in either mode, before the GPU-heavy encoders load.

Billing

A successful run is billed max(100, number_of_steps) step units at $0.01000 per step (category: training). The default 2,000-step run bills 2,000 units = $20.00; the 6,000-step maximum bills $60.00; requests under 100 steps are floored to 100 units = $1.00. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units. If the training node is interrupted, the run resumes from its latest checkpoint under the same request — you are billed once, on completion.

How the Training Works

Pipeline Overview

1. Preprocessing
   1. Your archive is downloaded and extracted; sidecar keyframes are detected and set aside.
   2. Media is matched to captions by file name (missing captions fall back to the trigger phrase or a blank prompt).
   3. Each clip is resized and cropped to the chosen resolution bucket and (optionally) split into scenes.
   4. Video frames, audio, captions, and both conditioning keyframes are pre-encoded (video VAE, audio VAE, and H3's Qwen3-VL text encoder — which also sees the keyframe images for image-aware text conditioning).
2. Training — the LoRA is trained for number_of_steps; per sample, a conditioning signature (first / last / both / none) is drawn with your configured probabilities. Resume checkpoints are written continuously.
3. Output — the final LoRA and config are uploaded.

Keyframe Signatures

When a sample draws a conditioning signature, the corresponding keyframe images — sidecars if present, else the clip's actual first/last frames — are encoded and packed into the training sequence at a near-clean signal level. The model learns to generate the clip given those anchors:

• first — animate forward from a starting image (image-to-video).
• last — generate footage that lands on a target image.
• first+last — interpolate: start on one image, end on the other. This is H3's first-to-last inference mode.
• none — pure text-to-video, keeping prompt-only generation intact.

Mixing signatures in one run (the default probabilities) produces a LoRA that works across all of H3's conditioning modes.

LoRA Training

A LoRA is a small set of adapter weights layered on top of the frozen base model — here, H3's attention projections. Training only updates these adapters, so the result is a compact file you load alongside the base H3 model at inference. The base model's general capabilities are preserved; the LoRA nudges it toward your subject or style.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 clips; 20–50 varied clips often work better for a robust concept.
• Make the first and last frames count: clips whose endpoints clearly show your subject teach the keyframe mapping best. Use sidecar keyframes when a clip's natural endpoints are weak (motion blur, occlusion, mid-action cuts).
• For interpolation-style effects (transformations, reveals, disintegrations), pick clips where the start and end states are visually distinct — that contrast is exactly what the first+last signature learns.
• If the soundtrack matters, make sure clips carry clean audio — the LoRA learns sound and picture together.

Choosing the Probabilities

• Defaults (0.2 / 0.2 / 0.4, 20% unconditioned) give a balanced all-mode LoRA.
• Targeting only first-to-last inference? Shift mass to first_last_frame_conditioning_p (e.g. 0.1 / 0.1 / 0.7).
• Keep a nonzero unconditioned share unless you never prompt without images — it stabilizes text following.

Scene Splitting, Captions, and Sidecars

If split_input_into_scenes is on, one long video becomes several shorter clips that share the original caption — any sidecar keyframes for that video are dropped, and each scene's first/last frames become arbitrary cut points. For this endpoint especially, pre-split your clips and disable scene splitting so the endpoints of every training sample are intentional.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/my_dataset.zip",
  "trigger_phrase": "tronl0g0",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "first_frame_conditioning_p": 0.2,
  "last_frame_conditioning_p": 0.2,
  "first_last_frame_conditioning_p": 0.4,
  "number_of_frames": 73,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "16:9",
  "split_input_into_scenes": false
}

Diagnosing Issues

• First-to-last generations don't land on the end image: raise first_last_frame_conditioning_p, verify your clips' last frames (or .last_frame sidecars) actually show the target state, and check that scene splitting is off.
• Overfitting (generated videos are exact copies of training clips): reduce number_of_steps, lower rank, or add more varied data.
• Underfitting (the concept barely appears): increase number_of_steps, raise rank, or improve captions.
• 422 about conditioning probabilities: the three probabilities sum above 1.0.
• Sidecars seem ignored: check the logs for "Ignoring sidecar keyframes" — the clip was probably scene-split; disable split_input_into_scenes.

Common Pitfalls

• Mixing images and videos in one archive (not supported).
• Leaving scene splitting on with sidecar keyframes or intentional endpoint framing (endpoints become arbitrary cut points).
• Frame counts that ignore the % 17 == 5 rule (they are snapped automatically — check the logs if durations look off).
• Forgetting the trigger phrase at inference.