fal Assets is now live!
Docs
Log-inSign-up

fal-ai/nemotron-diffusion-vlm

Nemotron-Labs-Diffusion-VLM-8B is the vision-language extension of the Nemotron-Labs-Diffusion family.
Inference
Commercial use
Schema
LLMs
PlaygroundAPIExamples

Table of contents

About

Generate

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install --save @fal-ai/client

Setup your API Key#

Set FAL_KEY as an environment variable in your runtime.

export FAL_KEY="YOUR_API_KEY"

Submit a request#

The client API handles the API submit protocol. It will handle the request status updates and return the result when the request is completed.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/nemotron-diffusion-vlm", {
  input: {
    image_url: "https://storage.googleapis.com/falserverless/example_inputs/dog.png",
    prompt: "Describe the image in one short sentence."
  },
  logs: true,
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") {
      update.logs.map((log) => log.message).forEach(console.log);
    }
  },
});
console.log(result.data);
console.log(result.requestId);

2. Authentication#

The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.

API Key#

In case your app is running in an environment where you cannot set environment variables, you can set the API Key manually as a client configuration.
import { fal } from "@fal-ai/client";

fal.config({
  credentials: "YOUR_FAL_KEY"
});

3. Queue#

Submit a request#

The client API provides a convenient way to submit requests to the model.

import { fal } from "@fal-ai/client";

const { request_id } = await fal.queue.submit("fal-ai/nemotron-diffusion-vlm", {
  input: {
    image_url: "https://storage.googleapis.com/falserverless/example_inputs/dog.png",
    prompt: "Describe the image in one short sentence."
  },
  webhookUrl: "https://optional.webhook.url/for/results",
});

Fetch request status#

You can fetch the status of a request to check if it is completed or still in progress.

import { fal } from "@fal-ai/client";

const status = await fal.queue.status("fal-ai/nemotron-diffusion-vlm", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b",
  logs: true,
});

Get the result#

Once the request is completed, you can fetch the result. See the Output Schema for the expected result format.

import { fal } from "@fal-ai/client";

const result = await fal.queue.result("fal-ai/nemotron-diffusion-vlm", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b"
});
console.log(result.data);
console.log(result.requestId);

4. Files#

Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.

Data URI (base64)#

You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.

Hosted files (URL)#

You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.

Uploading files#

We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.

import { fal } from "@fal-ai/client";

const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);

Read more about file handling in our file upload guide.

5. Schema#

Input#

image_url string* required

URL of the image to be processed.

prompt string* required

Prompt to answer about the image.

max_tokens integer

Maximum number of tokens to generate. Default value: 512

num_inference_steps integer

Number of diffusion decoding steps. Defaults to 256, rounded up only when omitted and required by the upstream block schedule. Explicit values must be at least max_tokens / block_length and divisible by max_tokens / block_length. Default value: 256

inference_steps integer

Hidden alias for num_inference_steps. Default value: 256

block_length integer

Block length used by diffusion decoding. Default value: 32

threshold float

Confidence threshold used by diffusion decoding. Default value: 0.9

{
  "image_url": "https://storage.googleapis.com/falserverless/example_inputs/dog.png",
  "prompt": "Describe the image in one short sentence.",
  "max_tokens": 512,
  "num_inference_steps": 256,
  "inference_steps": 256,
  "block_length": 32,
  "threshold": 0.9
}

Output#

output string* required

Generated answer.

usage NemotronDiffusionVLMUsage* required

Token and diffusion decoding usage information.

timings NemotronDiffusionVLMTimings* required

Request timing breakdown in seconds.

{
  "output": "The image shows a dog sitting outdoors.",
  "usage": {},
  "timings": {}
}

Other types#

NemotronDiffusionVLMTimings#

total_time float* required

Total request time in seconds.

image_validation_time float* required

Time spent validating and loading the input image.

preprocessing_time float* required

Time spent expanding the chat template and preprocessing tensors.

gpu_transfer_time float* required

Time spent moving tensors to the GPU.

generation_time float* required

Time spent inside model generation.

decode_time float* required

Time spent decoding output token IDs to text.

NemotronDiffusionVLMUsage#

prompt_tokens integer* required

Number of prompt tokens sent to the model.

completion_tokens integer* required

Number of generated completion tokens.

num_function_evals integer* required

Number of model function evaluations used for generation.