Serverless Inference: How Autoscaling and Scale-to-Zero Work

Generative media traffic is rarely flat. A launch, a viral post or a customer's nightly batch can multiply demand in minutes, and a quiet weekend can bring it close to zero. A fixed GPU fleet is sized either for the peak, so it sits idle most of the time, or for the average, so it falls over at the peak.

Serverless inference on fal sizes the fleet for you. Your app runs on runners: GPU or CPU instances that run your code. fal adds runners as requests arrive and removes them as traffic falls, down to zero when there is nothing to do. You pay per second for the time each runner is alive. A handful of parameters sets the floor, the ceiling and the headroom. This guide explains what each one does, how they combine, and how to tune them without redeploying.

What you pay for

Every runner moves through a series of states. Billing is per second at your machine type's rate, but only in some of those states:

Requests that fail with a 5xx error are not charged. Multi-GPU runners are billed as GPU count × duration. The pricing docs cover each state.

Two things follow from this table. First, an idle runner costs the same per second as a busy one, so every scaling parameter is a trade between paying for idle time and paying in latency when a request has to wait for a new runner. Second, the time before setup() (waiting for hardware and pulling the image) is free, but users still wait through it. What is a serverless GPU cold start? covers that wait in depth.

The parameters

All of these are class attributes on your fal.App, and they apply to WebSocket apps as well as queue-based apps. See the scaling parameter reference for each one.

class MyApp(fal.App):
    machine_type = ["GPU-H100", "GPU-A100"]  # tried in order
    min_concurrency = 0      # scale to zero
    keep_alive = 300         # seconds an idle runner stays warm
    concurrency_buffer = 0   # above 0 keeps warm runners even with no traffic
    max_concurrency = 10     # upper limit for total runners

How they combine

• Floor. fal keeps whichever of min_concurrency and concurrency_buffer is higher alive, even with no traffic. In the docs' example, an app with min_concurrency=0 and concurrency_buffer=2 still keeps 2 runners alive.
• Headroom. The buffer is the larger of concurrency_buffer and concurrency_buffer_perc / 100 × current request volume. A fixed buffer therefore acts as a minimum, and the percentage takes over as traffic grows.
• Ceiling. In normal scaling, fal doesn't start more than max_concurrency runners, however long the queue gets. If you don't set it, the limit is 2, so set it for any app that needs to scale. Queued requests wait for a free runner, unless the caller set a start timeout or a maximum queue length. A rollout is the one exception to the limit (see below).
• Packing. With max_multiplexing=4, one runner serves up to four requests at once, so fal needs fewer runners for the same traffic. Raise it only when your handlers are async and the model has the GPU memory for concurrent inference.

Recipes for common traffic shapes

Spiky or occasional traffic: scale to zero. Keep the defaults: min_concurrency=0 and concurrency_buffer=0. Runners stop keep_alive seconds after their last request, so an idle app costs nothing. Raise keep_alive if requests tend to arrive a few minutes apart and each gap would otherwise mean a cold start.

Interactive products: a small warm buffer. For apps with slow startup, the docs recommend concurrency_buffer over a high min_concurrency. The buffer keeps spare runners ahead of current demand, so a burst lands on capacity that is already up. A fixed buffer still keeps that many runners alive, and billed, when traffic is zero. Add concurrency_buffer_perc when bursts scale with your traffic.

Guaranteed baseline: set a floor. If some traffic must never wait for a cold start, set min_concurrency. Those runners are billed around the clock, even with zero traffic.

Brief blips: delay scaling. scaling_delay makes a queued request wait that many seconds before fal starts a new runner. A short spike can then drain on existing runners instead of starting one that will sit idle.

Any shape: set the ceiling on purpose. Always set max_concurrency. Too low, and traffic queues behind the default of 2 runners. Too high, and an unexpected surge becomes an unexpected bill.

Tune a live app without redeploying

Watch the app's real traffic in App Analytics. It shows request volume, latency, runner utilization and startup duration. Then adjust the running app from the CLI:

fal apps scale myapp --min-concurrency 2

The same settings are editable from the app's page in the dashboard. Changes take effect straight away, but they don't all behave the same when you next deploy:

• Runtime-tunable parameters (keep_alive, min_concurrency, max_concurrency, concurrency_buffer, concurrency_buffer_perc, scaling_delay, request_timeout, regions) persist across deploys. Your next fal deploy keeps the CLI or dashboard values, not the ones in your code. Run fal deploy --reset-scale to go back to the code values.
• Code-specific parameters (max_multiplexing, startup_timeout, machine_type) depend on how your code behaves, so a CLI change is temporary. The next deploy resets them to what the code says.

fal deploy --check prints a summary of the scale settings before it deploys, flags code values that won't apply without --reset-scale, and asks you to confirm.

Changing machine types

machine_type takes a list of types, which fal tries in order when the first has no capacity. A longer list gives the scheduler more hardware to choose from. If you change the machine type with fal apps scale or in the dashboard, only new runners use the new type. Run fal apps rollout to replace existing runners. fal starts the replacements before draining the old runners, on top of max_concurrency, so for a short time the app can run close to double its usual runner count and you pay for both. A fal deploy with a new machine_type moves the whole app as part of the deploy.

Scaling down cleanly

Runners stop when keep_alive expires, when the autoscaler removes extra capacity, when a new revision replaces them, when you stop them from the CLI or dashboard, or when a host needs maintenance. The shutdown sequence is the same every time:

1. The runner receives SIGTERM, and fal calls your handle_exit() straight away.
2. The runner stops accepting new requests and finishes the ones in flight.
3. fal calls teardown().
4. When termination_grace_period_seconds runs out (5 seconds by default, 1 hour at most), the process is killed.

Finishing requests and teardown() share that one grace period, and the runner is billed while it lasts. If requests can run long, use handle_exit() to stop work early, so teardown() has time to save state. The grace period needs fal SDK 1.61.0 or later. Older versions stop runners straight away.

FAQ

Does fal Serverless scale to zero?

Yes. min_concurrency and concurrency_buffer default to 0. With no traffic, runners shut down after keep_alive (60 seconds by default), and the app scales to zero. A concurrency_buffer above 0 keeps that many runners warm even with no traffic. Raise min_concurrency when you need guaranteed baseline capacity. Those runners are billed even with zero traffic.

How do I avoid cold starts during a traffic spike?

Keep headroom with concurrency_buffer or concurrency_buffer_perc, which the docs recommend over a high min_concurrency for apps with slow startup. Also shorten startup itself, with weights on /data, smaller images and compiled caches.

How is serverless GPU billing calculated?

Billing is per second of runner lifetime, at your machine type's rate. SETUP, IDLE (including keep_alive), RUNNING, DRAINING and TERMINATING are billed. PENDING, DOCKER_PULL and TERMINATED are not. Multi-GPU runners are billed as GPU count × duration.

How do I cap my GPU spend?

Set max_concurrency, the upper limit on total runners. It prevents runaway costs during unexpected traffic spikes. If you don't set it, the limit is 2. Queued requests beyond that capacity wait for a free runner, unless the caller set a start timeout or a maximum queue length.

Can one GPU runner handle several requests at once?

Yes, with max_multiplexing, the maximum number of concurrent requests per runner. Raise it only when your handlers are async and the model has capacity for concurrent inference, such as enough GPU memory.

Do I need approval to use fal Serverless autoscaling?

Yes. Deploying your own apps on fal Serverless needs access that the fal team approves for each account. Request it on the Serverless Get Started page. Creating a fal account is free, and calling hosted Model APIs needs no approval.