Try H3 Max Director
Log-inSign-up
Contact SalesLog-inSign-up

fal-ai/flux-3-action/so101

FLUX 3 Action turns what the robot sees into what it does next. Give it the scene camera image, the wrist camera image, the current SO-101 joint state (shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll, gripper) and a plain-language instruction such as "Pick up the yellow cube and place it inside the black rectangle". It returns a chunk of 42 target joint positions at 30 Hz, that is 1.4 s of motion. In a control loop, execute the first 32 steps (about 1 s), then call again with fresh images and joint state.
Inference
Commercial use
Schema
LLMs
PlaygroundAPI

Table of contents

About

So101

1. Calling the API#

Install the client#

The client provides a convenient way to interact with the model API.

npm install --save @fal-ai/client

Setup your API Key#

Set FAL_KEY as an environment variable in your runtime.

export FAL_KEY="YOUR_API_KEY"

Submit a request#

The client API handles the API submit protocol. It will handle the request status updates and return the result when the request is completed.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/flux-3-action/so101", {
  input: {
    prompt: "place the box in the container",
    scene_image_url: "https://v3b.fal.media/files/b/0aaba0cf/rbESQ2h5nozh5K8Ms1FDm_scene.png",
    wrist_image_url: "https://v3b.fal.media/files/b/0aaba0cf/qyaj49X3PZc70wJwz90f1_wrist.png",
    state: [-18.94, 134.52, 120.15, 28.3, -32.17, 27.02]
  },
  logs: true,
  onQueueUpdate: (update) => {
    if (update.status === "IN_PROGRESS") {
      update.logs.map((log) => log.message).forEach(console.log);
    }
  },
});
console.log(result.data);
console.log(result.requestId);

2. Authentication#

The API uses an API Key for authentication. It is recommended you set the FAL_KEY environment variable in your runtime when possible.

API Key#

In an environment where you cannot set environment variables, you can configure the API key manually on the client.
import { fal } from "@fal-ai/client";

fal.config({
  credentials: "YOUR_FAL_KEY"
});

3. Queue#

Submit a request#

The client API provides a convenient way to submit requests to the model.

import { fal } from "@fal-ai/client";

const { request_id } = await fal.queue.submit("fal-ai/flux-3-action/so101", {
  input: {
    prompt: "place the box in the container",
    scene_image_url: "https://v3b.fal.media/files/b/0aaba0cf/rbESQ2h5nozh5K8Ms1FDm_scene.png",
    wrist_image_url: "https://v3b.fal.media/files/b/0aaba0cf/qyaj49X3PZc70wJwz90f1_wrist.png",
    state: [-18.94, 134.52, 120.15, 28.3, -32.17, 27.02]
  },
  webhookUrl: "https://optional.webhook.url/for/results",
});

Fetch request status#

You can fetch the status of a request to check if it is completed or still in progress.

import { fal } from "@fal-ai/client";

const status = await fal.queue.status("fal-ai/flux-3-action/so101", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b",
  logs: true,
});

Get the result#

Once the request is completed, you can fetch the result. See the Output Schema for the expected result format.

import { fal } from "@fal-ai/client";

const result = await fal.queue.result("fal-ai/flux-3-action/so101", {
  requestId: "764cabcf-b745-4b3e-ae38-1200304cf45b"
});
console.log(result.data);
console.log(result.requestId);

4. Files#

Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.

Data URI (base64)#

You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.

Hosted files (URL)#

You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.

Uploading files#

We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.

import { fal } from "@fal-ai/client";

const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);

Read more about file handling in our file upload guide.

5. Schema#

Input#

prompt string* required

Natural-language task instruction for the robot.

scene_image_url string* required

Current frame from the fixed scene camera (tiled left). Resized to 256x256.

wrist_image_url string* required

Current frame from the wrist camera (tiled right). Resized to 256x256.

state list<float>* required

Measured state: shoulder pan, shoulder lift, elbow flex, wrist flex, wrist roll (degrees), gripper (percent).

state_history list<list<float>>

Measured states at the 7 control ticks (30 Hz) before state, oldest first. Requires command_history. Omit for a fresh episode.

command_history list<list<float>>

Absolute commands in effect at each of the 8 observation ticks, oldest first; the last row is the most recent command sent. Requires state_history.

history_scene_image_url string

Scene frame from the oldest history tick (7 ticks before now). Requires state_history. Defaults to the current frame.

history_wrist_image_url string

Wrist frame from the oldest history tick. Requires state_history.

num_inference_steps integer

Euler denoising steps. The checkpoint was validated at 4. Default value: 4

guidance_scale float

Classifier-free guidance on video and action; 1.0 disables CFG. Default value: 3

seed integer

Sampling seed. Omit for a random seed; the reference evaluation seed is 0 for DROID and 42 for SO-101.

{
  "prompt": "place the box in the container",
  "scene_image_url": "https://v3b.fal.media/files/b/0aaba0cf/rbESQ2h5nozh5K8Ms1FDm_scene.png",
  "wrist_image_url": "https://v3b.fal.media/files/b/0aaba0cf/qyaj49X3PZc70wJwz90f1_wrist.png",
  "state": [
    -18.94,
    134.52,
    120.15,
    28.3,
    -32.17,
    27.02
  ],
  "num_inference_steps": 4,
  "guidance_scale": 3
}

Output#

actions list<list<float>>* required

Predicted action chunk, one row per control step in time order. DROID: 32x8 absolute joint targets (rad) plus gripper closed fraction at 15 Hz. SO-101: 42x6 absolute commands (deg, gripper %) at 30 Hz. Not clipped: enforce joint, velocity and workspace limits on the robot.

action_labels list<string>* required

Name of each action column.

action_hz float* required

Control rate of the action rows.

execute_steps integer* required

Execute this many leading actions, then request a new chunk.

seed integer* required

Seed used for sampling.

timings Timings* required

Server-side timings in seconds.

{}

Other types#