Try New Grok Imagine here!
Log-inSign-up

Lipsync Unknown

pipio/lipsync
A diffusion-based model that generates professional-grade lip-sync for talking head videos from video or image input and their corresponding audio.
Inference
Private
Schema
LLMs
PlaygroundAPI

About

Run

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install --save @fal-ai/client

Setup your API Key#

Set FAL_KEY as an environment variable in your runtime.

export FAL_KEY="YOUR_API_KEY"

Submit a request#

The client API handles the API submit protocol. It will handle the request status updates and return the result when the request is completed.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("pipio/lipsync", {
  input: {
    video: {
      url: ""
    },
    audio: {
      url: ""
    }
  },
  logs: true,
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") {
      update.logs.map((log) => log.message).forEach(console.log);
    }
  },
});
console.log(result.data);
console.log(result.requestId);

2. Authentication#

The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.

API Key#

In case your app is running in an environment where you cannot set environment variables, you can set the API Key manually as a client configuration.
import { fal } from "@fal-ai/client";

fal.config({
  credentials: "YOUR_FAL_KEY"
});

3. Queue#

Submit a request#

The client API provides a convenient way to submit requests to the model.

import { fal } from "@fal-ai/client";

const { request_id } = await fal.queue.submit("pipio/lipsync", {
  input: {
    video: {
      url: ""
    },
    audio: {
      url: ""
    }
  },
  webhookUrl: "https://optional.webhook.url/for/results",
});

Fetch request status#

You can fetch the status of a request to check if it is completed or still in progress.

import { fal } from "@fal-ai/client";

const status = await fal.queue.status("pipio/lipsync", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b",
  logs: true,
});

Get the result#

Once the request is completed, you can fetch the result. See the Output Schema for the expected result format.

import { fal } from "@fal-ai/client";

const result = await fal.queue.result("pipio/lipsync", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b"
});
console.log(result.data);
console.log(result.requestId);

4. Files#

Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.

Data URI (base64)#

You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.

Hosted files (URL)#

You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.

Uploading files#

We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.

import { fal } from "@fal-ai/client";

const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);

Read more about file handling in our file upload guide.

5. Schema#

Input#

use_custom_prompt boolean

If enabled, use the custom_prompt field instead of the automatically generated prompt based on the video content.

custom_prompt string

Custom text prompt describing the edited video. Only used when use_custom_prompt is enabled. Default value: "A high quality video."

video File-Input* required

URL or path to the input video file that will be edited.

audio File-Input* required

URL or path to the input audio file that will be used for lipsync.

frame_rate integer

Frame rate (fps) of the edited video. Special values: -1 for native fps. Default value: -1

height integer

Height in pixels of the edited video. Special values: -1 for native height, -2 for native height/2, -3 for native height/3, etc. Default value: -1

width integer

Width in pixels of the edited video. Special values: -1 for native width, -2 for native width/2, -3 for native width/3, etc. Default value: -1

num_frames integer

Number of frames to process from the input video. Determines the length of the output video when no edits are present. Default value: 100

vae_chunk_size integer

For long inference, size of the chunk to encode/decode. Must be a multiple of 8n+1. Set to a very large value to disable chunking. Default value: 65

vae_overlap_window_width integer

For long inference, size of the overlap window in between vae encode/decode chunks. 0 means no overlap. Default value: 16

frame_block_width integer

For long inference, the transformer will see the video in blocks of this width. Default value: 136

feed_forward_num_splits integer

Number of chunks to split the feed-forward layer into during processing. Higher values reduce memory usage but may increase processing time. Default value: 2

face_id_cond_strength integer

Face identity conditioning strength, should be an either -1 or an integer in range [0-16]. If it's set to -1 we choose an appropriate value. the Higher values increase subject identity preservation but may reduce lipsync accuracy. Default value: -1

appearance_cond_strength integer

Appearance conditioning strength, should be an integer in range [1-16]. Higher values increase the alignment of fully synthetic frames with the conditioning, at the cost of reduced creativity. Default value: 1

edit_addition_start_frames list<integer>

List of start frame indices (0-based) where new content should be added. Must have the same length as edit_addition_durations.

edit_addition_durations list<integer>

List of durations (in frames) for each addition edit. Must have the same length as edit_addition_start_frames.

edit_removal_ranges list<integer>

List of frame index pairs [start, end] (0-based, inclusive) specifying ranges to remove from the video. Must come in pairs, and the number of pairs must match edit_removal_bridge_durations.

edit_removal_bridge_durations list<integer>

List of bridge lengths (in frames) for removal edits. Must match the number of removal pairs in edit_removal_ranges. A value of 0 means a jump cut with no transition.

seed integer

Random seed for reproducible video generation. Use the same seed with identical inputs to get consistent results. Default value: 42

{
  "custom_prompt": "A high quality video.",
  "video": {
    "url": "",
    "content_type": "image/png",
    "file_name": "z9RV14K95DvU.png",
    "file_size": 4404019
  },
  "audio": {
    "url": "",
    "content_type": "image/png",
    "file_name": "z9RV14K95DvU.png",
    "file_size": 4404019
  },
  "frame_rate": -1,
  "height": -1,
  "width": -1,
  "num_frames": 100,
  "vae_chunk_size": 65,
  "vae_overlap_window_width": 16,
  "frame_block_width": 136,
  "feed_forward_num_splits": 2,
  "face_id_cond_strength": -1,
  "appearance_cond_strength": 1,
  "seed": 42
}

Output#

video File-Output* required

The generated video file.

{
  "video": {
    "url": "",
    "content_type": "image/png",
    "file_name": "z9RV14K95DvU.png",
    "file_size": 4404019
  }
}

Other types#