minimax/h3/i2v/trainer

MiniMax H3 Trainer — Image-to-Video-Audio (/image-to-video-audio)

Overview

The /image-to-video-audio endpoint trains a LoRA adapter for the MiniMax H3 (Hailuo-03) video+audio model with first-frame conditioning: during training, each sample is conditioned on its opening frame with probability first_frame_conditioning_p (default 0.5), teaching the LoRA to animate from a starting image. The result works in both H3's image-to-video and text-to-video modes.

Key features:

• Learns a subject, motion pattern, or style from your own videos, with first-frame conditioning trained in.
• Joint video + audio training: clips with an audio track teach the LoRA the matching sound; clips without audio train against silence. Audio handling is automatic and per-clip.
• Captions are optional — clips without a .txt train against the trigger phrase (or a blank prompt).
• Supports sidecar keyframe images: ship <stem>.first_frame.png next to a clip to condition on that image instead of the clip's literal first frame.

Dataset Format

Provide a single .zip archive (linked via training_data_url) containing your clips and, optionally, captions and sidecar keyframes:

• Videos: .mp4, .mov, .avi, .mkv
• Images: .png, .jpg, .jpeg
• Captions (optional): a .txt file with the same base name as each media file (e.g. clip01.mp4 + clip01.txt). Clips without a caption train on the trigger phrase, or a blank prompt if none is set.
• Sidecar keyframes (optional): an image named <stem>.first_frame.png (also .jpg/.jpeg, any case) next to clip01.mp4 replaces the clip's own first frame as the conditioning image. Useful when the ideal conditioning image differs from the literal opening frame (e.g. a clean product shot vs. a motion-blurred frame).

The archive must contain only videos OR only images — mixed datasets are rejected (sidecar keyframes don't count as dataset images). Aim for at least 10 files; more is generally better. Files in subfolders are fine — clips with the same name in different subfolders are kept distinct, and each sidecar binds to the clip in its own folder.

Minimum clip length: with auto_scale_input off (the default), each video must already have at least number_of_frames frames (default 73, ≈ 3.0 s at 24 fps). Shorter clips are silently skipped, and if every clip is too short the request fails (422). Turn on auto_scale_input to resample shorter clips to the target frame count instead.

Input Parameters Reference

Dataset

training_data_url (required)

Type: string

URL to the .zip archive of training clips (and optional captions/sidecars). See "Dataset Format" above.

trigger_phrase

Type: string Default: ""

A phrase prepended to every caption during training (and used as the whole caption for clips without one). At inference, including this phrase activates the learned concept.

Conditioning

first_frame_conditioning_p

Type: number Default: 0.5 (range 0.0–1.0)

Probability that a training sample is conditioned on its first frame (or its sidecar keyframe). Higher values weight the LoRA toward image-to-video behavior; the remaining probability mass trains unconditioned text-to-video. At 0.0 no image conditioning is trained.

Training Parameters

rank

Type: integer (one of 8, 16, 32, 64, 128) Default: 32

LoRA capacity. Higher values can capture more detail but use more memory and are more prone to overfitting on small datasets.

number_of_steps

Type: integer Default: 2000 (range 1–6000)

How many optimization steps to run. More steps means more learning but also more time and a higher chance of overfitting. Note that billing has a 100-step floor (see Billing).

learning_rate

Type: number Default: 0.0002

How aggressively the model updates each step. The default is a sensible starting point; raise it cautiously and lower it if results look unstable or degraded.

Video Configuration

number_of_frames

Type: integer Default: 73 (range 22–124)

Frames per training clip. H3's video VAE requires frames % 17 == 5 — valid counts are 22, 39, 56, 73, 90, 107, 124. Other values are adjusted with 17 * (value // 17) + 5 and clamped to the supported range rather than using nearest-value rounding; for example, 100 becomes 90 while 123 becomes 124. Has no effect on image datasets.

frame_rate

Type: integer Default: 24 (range 8–60)

Target frames per second for training clips. H3's native output rate is a fixed 24 fps, so keep the default unless you have a specific reason to deviate.

resolution

Type: string (one of low, medium, high) Default: medium

Training resolution bucket. Combined with aspect_ratio this picks the exact pixel size:

H3's native output is 2K; LoRA training at these sub-native resolutions is standard practice across the video-trainer family and carries over to full-resolution inference.

aspect_ratio

Type: string (one of 21:9, 16:9, 4:3, 1:1, 3:4, 9:16) Default: 16:9

Aspect ratio for training clips — the same set the H3 API supports. See the table above.

auto_scale_input

Type: boolean Default: false

When true, videos are automatically fit to the target frame count and frame rate. No effect on image datasets.

split_input_into_scenes

Type: boolean Default: true

When true, videos longer than the duration threshold are automatically split into separate scenes (shots) before training. Note: sidecar keyframes are dropped for clips that get split (the keyframe can't be matched to a specific scene) — disable this if your dataset uses sidecars on long clips.

split_input_duration_threshold

Type: number Default: 30.0 (range 1.0–60.0)

Videos longer than this many seconds are eligible for scene splitting.

Outputs

• lora_file — the trained LoRA weights (.safetensors). This is the main artifact.
• config_file — a small JSON containing the trigger phrase as instance_prompt and training_type set to i2va, for setting up inference.
• debug_dataset — a downloadable archive of your preprocessed data, only present when debug_dataset is enabled.

The endpoint does not accept validation samples and does not return preview inference; evaluate the LoRA by loading it into the inference endpoint.

debug_dataset

Type: boolean Default: false

When enabled, returns an archive of the preprocessed training data so you can verify your videos, captions, and keyframes were processed correctly before committing to a longer run. The archive includes preprocessing_report.json, a bounded summary of any dataset fallbacks.

strict_dataset

Type: boolean Default: false

Preprocessing always logs a summary when it uses a blank prompt, substitutes the clip's first frame for a missing sidecar, or substitutes silence for an unreadable or absent video soundtrack. Enable this option to reject those fallbacks with a 422 instead. Captions longer than 4,096 tokenizer tokens are rejected with a 422 in either mode, before the GPU-heavy encoders load.

Billing

A successful run is billed max(100, number_of_steps) step units at $0.01000 per step (category: training). The default 2,000-step run bills 2,000 units = $20.00; the 6,000-step maximum bills $60.00; requests under 100 steps are floored to 100 units = $1.00. Requests that fail before training completes (input-validation errors / HTTP 422, or dataset-download failures) are billed 0 units. If the training node is interrupted, the run resumes from its latest checkpoint under the same request — you are billed once, on completion.

How the Training Works

Pipeline Overview

1. Preprocessing
   1. Your archive is downloaded and extracted; sidecar keyframes are detected and set aside.
   2. Media is matched to captions by file name (missing captions fall back to the trigger phrase or a blank prompt).
   3. Each clip is resized and cropped to the chosen resolution bucket and (optionally) split into scenes.
   4. Video frames, audio, captions, and conditioning keyframes are pre-encoded (video VAE, audio VAE, and H3's Qwen3-VL text encoder).
2. Training — the LoRA is trained for number_of_steps; per sample, first-frame conditioning is applied with probability first_frame_conditioning_p. Resume checkpoints are written continuously.
3. Output — the final LoRA and config are uploaded.

Keyframe Conditioning

When a sample is conditioned, the conditioning image — the sidecar <stem>.first_frame.png if present, else the clip's own first frame — is encoded and packed into the training sequence at a near-clean signal level, so the model learns to generate the clip given that image. This is the exact mechanism behind H3's image-to-video inference mode; training it with your data teaches the LoRA how your subject moves when animated from a still.

LoRA Training

A LoRA is a small set of adapter weights layered on top of the frozen base model — here, H3's attention projections. Training only updates these adapters, so the result is a compact file you load alongside the base H3 model at inference. The base model's general capabilities are preserved; the LoRA nudges it toward your subject or style.

Tips for Getting Good Results

Dataset Quality

• Use at least 10 clips; 20–50 varied clips often work better for a robust concept.
• Keep quality high: sharp, well-lit, representative footage. The LoRA reproduces whatever artifacts are common in your data.
• Make the first frame count: since conditioning uses the opening frame, clips that start with a clear view of your subject teach the mapping best. Use sidecar keyframes when the natural first frame is weak (motion blur, occlusion).
• If the soundtrack matters, make sure clips carry clean audio — the LoRA learns sound and picture together.

Caption Best Practices

Captions are optional here, but they still steer the LoRA when present:

• With captions: describe what happens in the clip, e.g. tronl0g0 the logo assembles itself from glowing particles.
• Without captions: set a trigger_phrase so all clips share a consistent anchor; fully caption-less training (blank prompts) works but gives you less prompt control at inference.

Conditioning Probability

• 0.5 (default) balances image-to-video and text-to-video ability.
• Raise toward 0.8–1.0 when the LoRA will only ever be used for image-to-video.
• Lower toward 0.2–0.3 when text-prompted generation matters more but you still want first-frame support.

Scene Splitting, Captions, and Sidecars

If split_input_into_scenes is on, one long video becomes several shorter clips that share the original caption — and any sidecar keyframe for that video is dropped. For precise captions and reliable sidecar binding, pre-split your clips and disable scene splitting.

Recommended Starting Configuration

json
{
  "training_data_url": "https://example.com/my_dataset.zip",
  "trigger_phrase": "tronl0g0",
  "rank": 32,
  "number_of_steps": 2000,
  "learning_rate": 0.0002,
  "first_frame_conditioning_p": 0.5,
  "number_of_frames": 73,
  "frame_rate": 24,
  "resolution": "medium",
  "aspect_ratio": "16:9",
  "split_input_into_scenes": false
}

Diagnosing Issues

• Underfitting (the concept barely appears): increase number_of_steps, raise rank, or improve captions.
• Image-to-video generations ignore the conditioning image: confirm first_frame_conditioning_p is well above 0 and that your clips' first frames actually show the subject.
• Sidecars seem ignored: check the logs for "Ignoring sidecar keyframes" — the clip was probably scene-split; disable split_input_into_scenes.

Common Pitfalls

• Mixing images and videos in one archive (not supported).
• Sidecar keyframes on long clips with scene splitting enabled (sidecars get dropped).
• Frame counts that ignore the % 17 == 5 rule (they are snapped automatically — check the logs if durations look off).
• Forgetting the trigger phrase at inference.