Real-Time GPU Inference over WebSockets and WebRTC

Some generative apps can't wait for a queue: a canvas that redraws as you sketch, a camera filter, a world model you steer with the keyboard. Every input needs an answer straight away, and the next input is already on its way. Submitting each one as a separate HTTP request through a queue adds connection setup and queue wait to every frame.

fal Serverless realtime endpoints use a persistent WebSocket instead. A client connects once, and inputs go straight to a GPU runner without passing through the queue. The runner stays warm for as long as the connection is open. For live video and audio, WebRTC can carry the media while the WebSocket only handles signaling. This guide covers how to build each kind of endpoint, connect to it safely from a browser, and size it for real traffic.

Realtime, streaming or the queue?

fal offers three ways to get results back, and they suit different jobs:

Use the queue for jobs that take seconds to minutes. Use streaming to show progress from a single request, such as diffusion previews or LLM tokens. Use realtime when the client sends many inputs in a row and each answer matters straight away.

Build a realtime endpoint

Add a method with the @fal.realtime decorator to your fal.App. One app can have both a normal endpoint and a realtime endpoint:

import fal
from pydantic import BaseModel

class Input(BaseModel):
    prompt: str

class Output(BaseModel):
    output: str

class RealtimeApp(fal.App):
    keep_alive = 60

    @fal.endpoint("/")
    def generate(self, input: Input) -> Output:
        return Output(output=input.prompt)

    @fal.realtime("/realtime")
    def generate_rt(self, input: Input) -> Output:
        return Output(output=input.prompt)

From the realtime endpoints guide.

The path matters. fal's realtime clients connect to /realtime by default, so @fal.realtime("/realtime") is the path that works without extra client configuration. If you need a different path, the clients accept a path option.

@fal.realtime handles serialization with fal's binary msgpack protocol by default. Msgpack is more compact than JSON for image data, and the decorator works with the realtime methods in the fal client SDKs for Python, JavaScript and Swift. You can swap msgpack for your own encoding, and an endpoint can take and yield async iterators for streaming, two-way sessions.

When you need your own protocol: raw WebSockets

If you want full control of the socket itself, declare an ordinary endpoint with is_websocket=True. Your function then receives the raw WebSocket:

import fal
from fastapi import WebSocket

class RealtimeApp(fal.App):
    @fal.endpoint("/echo", is_websocket=True)
    async def generate_ws(self, websocket: WebSocket) -> None:
        await websocket.accept()
        msg = await websocket.receive_text()
        for idx in range(3):
            await websocket.send_text(msg + f"-{idx}")
        await websocket.close()

Clients connect to these endpoints with fal_client.ws_connect() and pass the endpoint path, rather than with realtime():

import fal_client

with fal_client.ws_connect("your-username/your-app-name", path="/echo") as ws:
    ws.send("Hello, world!")
    for _ in range(3):
        print(ws.recv())

Connect from a client

From Python, open a connection, send inputs and read results:

import fal_client

with fal_client.realtime("your-username/your-app-name") as connection:
    connection.send({"prompt": "Hello, world!"})
    result = connection.recv()
    print(result)

realtime_async() does the same with async with. In JavaScript, fal.realtime.connect() takes onResult and onError callbacks. It also takes a throttleInterval, which limits how often inputs are sent while a user is dragging or typing. It defaults to 128 milliseconds, and 0 turns throttling off.

To try realtime before you build your own endpoint, connect to a hosted model that has one, such as fal-ai/fast-lcm-diffusion. For its fastest results, the docs suggest 512×512 inputs as base64 data URLs and sync_mode: true, so images come back inline instead of as CDN links.

Keep your API key out of the browser

A browser can't hold your fal API key safely, and a WebSocket opened from a page would expose it. There are two ways around this:

• A proxy URL. Set proxyUrl in the client config to a route on your own server that adds the key. This is the simplest option.
• A token provider. Pass a tokenProvider function that fetches a short-lived token from your backend. Set tokenExpirationSeconds to the same lifetime your backend requests, and the client refreshes the token before it expires.

Whichever you choose, protect that route with your own user authentication. Otherwise, anyone who finds it can spend your fal credits.

Stream live video with WebRTC

For webcam feeds, live rendering or audio, sending every frame as a msgpack message isn't the best transport. WebRTC is built for low-latency media. In fal's examples, a bidirectional @fal.realtime("/realtime") endpoint carries the WebRTC signaling: it reads offers, answers and ICE candidates from an async iterator and yields its replies. The media then flows between the browser and your runner over WebRTC, either directly or through a TURN relay when no direct path is available. Both examples need TURN credentials to run.

@fal.realtime("/realtime", buffering=5)
async def webrtc(
    self, inputs: AsyncIterator[RealtimeInput]
) -> AsyncIterator[RealtimeOutput]:
    ...

The signaling endpoint from the YOLO webcam example.

Two fal examples show the full loop:

• Real-time video-to-video streams a webcam to the app, runs YOLO detection on each frame and streams the annotated video back over the same connection.
• Real-time world model streams a live, interactive world model to the browser.

If you'd rather not write the signaling, tracks and frame batching yourself, the experimental World Model Accelerator wraps them.

What works differently from the queue

A realtime connection goes straight to a runner, so the queue's features don't apply:

Errors come back on the connection, through onError in JavaScript. If a connection closes, the JavaScript client opens a new one on the next send().

Realtime endpoints can't be tested in the fal Playground at the moment, apart from apps built on the World Model Accelerator, which has its own playground. Watch their activity on the Logs page of the dashboard instead.

If you only want a persistent connection to a model that has no realtime endpoint, fal also offers HTTP over WebSockets. It carries a model's normal request and response format over wss://ws.fal.run/ and works with any endpoint.

Scaling and cold starts

All runner scaling parameters work the same for WebSocket apps as for queue-based apps: keep_alive, min_concurrency, max_concurrency, concurrency_buffer, scaling_delay, max_multiplexing and request_timeout.

Once a client is connected, the connection keeps its runner warm. The first connection, though, may have to wait for a cold start if no runner is free. For interactive apps, that first wait is what users notice, so keep capacity ready:

• min_concurrency keeps a fixed number of runners alive at all times.
• concurrency_buffer keeps spare runners warm beyond current demand, so new connections land on a runner that is already running.

Both cost money while they wait, because idle runners are billed per second like busy ones. max_concurrency caps the total. For hardware, GPU-L40 and GPU-RTXPRO6000 suit image generation and have hardware video encoders. GPU-H100 has the most memory bandwidth of the three, but 80 GB of memory and no hardware video encoder. For the most memory on one of these GPUs, GPU-RTXPRO6000 has 96 GB.

For a closer look at warm capacity and its cost, see what a serverless GPU cold start is and the serverless inference guide.

FAQ

How do I build a WebSocket endpoint for GPU inference?

Add a method decorated with @fal.realtime("/realtime") to your fal.App and deploy it. Use the /realtime path, because fal_client.realtime() connects to that path by default. Clients then send inputs and receive results over one persistent connection.

What is the difference between @fal.realtime and a raw WebSocket endpoint?

@fal.realtime handles serialization with fal's binary msgpack protocol and works with the fal realtime clients. @fal.endpoint("/path", is_websocket=True) gives your function the raw WebSocket so you can run your own protocol. Clients connect to raw endpoints with fal_client.ws_connect().

Can I stream webcam video to a GPU model in real time?

Yes, with WebRTC as the transport. A bidirectional @fal.realtime("/realtime") endpoint handles signaling, and the media flows between the browser and your runner over WebRTC, through a TURN relay when no direct path exists. fal's examples run YOLO detection on a live webcam feed and stream a world model this way.

How do I keep my fal API key out of the browser?

Point the client at a server-side proxy URL that adds your key, or pass a tokenProvider that fetches short-lived tokens from your backend. Protect the token route with your own authentication.

Do realtime endpoints have cold starts?

An open connection keeps its runner warm, but the first connection may wait for a cold start if no runner is available. Keep capacity ready with min_concurrency or concurrency_buffer. Idle runners are billed while they wait.

Do I need approval to deploy a realtime endpoint?

Yes. Deploying your own apps on fal Serverless needs access that the fal team approves for each account. Request it on the Serverless Get Started page. Creating a fal account is free, and calling hosted Model APIs, including their realtime endpoints, needs no approval.