How to Choose a Serverless GPU Platform for Production Inference (2026)

The short answer

Choose the platform that runs your model with the shortest measured time to first response at your peak hours, bills only for runner states you can see and control, and lets you set the scaling floor and ceiling yourself. Everything else is a tiebreaker.

Most teams get this wrong in one of two ways. They compare list prices per GPU-hour and ignore how idle time is billed. Or they trust a headline cold-start number. That number is usually measured on a small model, on a cluster with GPUs to spare. In production, the longer wait is often for a GPU to be free at all. A fast cold start does not help if no GPU is available when your traffic spikes.

So test with your own container and your own weights. Read the billing page line by line. Then check the SLA, the observability and the migration effort.

The criteria table

Use this table as a checklist. The middle column is the question to ask a vendor. The right column is how fal answers it today.

Criteria in more detail

Cold start

A cold start happens when a request arrives and no warm runner can take it. From the user's side it has two parts. First, the request waits for a GPU to be free. Then the platform starts a machine, pulls the image, loads the weights and runs your setup code.

Published cold-start numbers usually measure only the second part. For large image and video models, loading weights is often the longest step in it. But at peak hours, or on scarce GPUs such as the B200 and B300, the wait for hardware can be longer than the whole startup. A platform with a slower startup and GPUs reliably available can serve your users sooner than one with a fast startup and no free GPU.

So ask for two things: a startup number measured on a model the size of yours, and how long requests wait for hardware at peak. Then measure both yourself, at your own peak hours.

On fal, three things shorten cold starts: a floor of warm runners (min_concurrency), weights kept on the persistent /data volume instead of downloaded on each start, and FlashPack, which streams weights from disk straight to GPU memory. For the mechanics, read What Is a Serverless GPU Cold Start?

SLA

Ask for the uptime figure in writing, and ask what it covers: the API, the scheduler, or your endpoint end to end. Uptime is not capacity: an API can be up while your requests wait for a GPU. Ask about both.

Billing granularity

Per-second billing matters most when requests are short and traffic is uneven. But granularity is only half the story. The other half is which runner states are billed.

Check four things:

• Is setup billed?
• Is idle time billed, and for how long?
• Are image pulls billed?
• Are failed requests billed?

On fal, setup, idle (including keep_alive), running, draining and teardown are billed. Pending time, image pulls and terminated runners are not. Server errors (HTTP 500 and above) are not charged. Multi-GPU runners are billed as GPU count times duration.

GPU types and price

Match memory to your model first, then match speed to your latency target. A model that does not fit in one GPU's memory needs a multi-GPU runner, and every GPU in it is billed for the whole time the runner is alive:

Runner cost per hour = number of GPUs × per-GPU hourly rate

For example, a model that needs 120 GB of GPU memory does not fit on one H100 (80 GB). On two H100s it costs 2 × $4.50 = $9.00 per hour. On one H200 (141 GB) it costs $6.00 per hour, a third less, even though the H200 has the higher hourly rate. So price the smallest set of GPUs your model fits on, not the cheapest GPU.

fal's published serverless rates on October 19, 2026:

Check fal.ai/pricing for current rates. Newer GPUs cost more per hour but can finish each request faster, so compare cost per request, not cost per hour:

Cost per request = hourly rate ÷ 3,600 × billed seconds per request ÷ requests served at once per runner.

Multiplexing and idle time move this more than list price. For one GPU in depth, see NVIDIA B200 Price in 2026.

Scaling controls and scale-to-zero

You want explicit control over the floor, the ceiling and the buffer. On fal these are:

• min_concurrency: runners that stay up regardless of traffic. Default 0.
• max_concurrency: the upper limit on total runners.
• concurrency_buffer: extra warm runners beyond current demand, for bursts. Default 0.
• keep_alive: seconds an idle runner stays up after its last request. Default 60.
• max_multiplexing: concurrent requests per runner. Default 1.

Each warm runner costs money even with zero traffic. Set the floor on purpose.

Scale-to-zero means you pay nothing when nobody calls your endpoint. The cost is a cold start on the next request. Tune keep_alive to trade idle cost against cold starts: a longer value means fewer cold starts and more billed idle time.

Container support

The cheapest migration is the one where your container runs as-is. Check three paths:

• Dockerfile. Can the platform build it?
• Pre-built image. Can it pull from your private registry?
• A server from another platform. Is there a documented migration path?

fal supports all three. An existing Docker server can run unchanged: fal starts the container and sends requests to the port you expose. Or wrap it in a fal.App for full control over the API.

Observability

In production you need logs per request, latency broken down by stage, and alerts. You also need to export metrics into the tools your on-call team already uses.

fal shows logs, request traces, analytics and latency breakdowns in one dashboard. Alerts go to the app, to Slack or by email. Metrics are exposed on a Prometheus-compatible endpoint that Grafana or Datadog can scrape, and log drains forward logs to external services.

Migration effort

Count the engineer-days, not the lines of code. Ask whether you must rewrite your server or can wrap it. fal can wrap an existing Docker container or Python inference server, and the docs include migration guides for common setups.

Workload → what to look for

How to run a fair evaluation

1. Pick your heaviest production model, not a demo model.
2. Deploy the same container on each platform you are considering.
3. Measure cold start from zero, ten times, including at your peak hours and with a burst of concurrent requests. Record the median and the worst case.
4. Replay a day of real traffic. Record total billed time and total cost.
5. Kill a runner mid-request. Check what the logs, traces and alerts show.
6. Write down every change you had to make to your container.

Rule out any platform whose worst case in step 3 misses your latency target. Of the rest, pick the lowest cost from step 4 and the fewest changes in step 6.

If your GPUs would be busy most of the day, serverless may not be the cheapest option at all. To find the line, see Dedicated GPUs vs Serverless Inference: A Break-Even Guide.

FAQ

What is a serverless GPU platform?

It is a service that runs your model on GPUs that start when requests arrive and stop when traffic ends. You deploy a container or an app. The platform handles machines, scaling and routing. You pay for the time runners are alive.

Is per-second billing always cheaper than hourly billing?

No. Per-second billing is cheaper when your GPUs would otherwise sit idle. If a GPU is busy most of the time, a lower hourly rate on a dedicated machine can cost less. With fal's published prices, the line is about 55% utilization for an H100 and about 69% for a B200. The break-even guide has the worked numbers.

How do I avoid cold starts in production?

Keep a floor of warm runners with min_concurrency. Add a concurrency_buffer for bursts. Keep weights on persistent storage and keep your image small. A warm floor does not help if a burst needs more GPUs than are free. Warm runners are billed, so size the floor to your steady traffic.

How much does a serverless H100 cost on fal?

$4.50 per GPU-hour, billed per second (about $0.00125 per second). A B200 is $7.99 per GPU-hour. See fal.ai/pricing for current rates.

Can I bring my own Docker container?

On fal, yes. You can use a Dockerfile or a pre-built image from any registry, including a private one.

Can I move a model from another platform to fal?

Yes. You can run your existing Docker server unchanged, or wrap it in a fal.App. The migration guides walk through each path.

Which GPUs does fal offer for serverless inference?

fal's pricing page lists B300, GB200, B200, H200, H100 and RTX PRO 6000. The machine types in the docs also include A100 and L40.

What uptime does fal commit to?

fal publishes a 99.99% uptime SLA.

Is idle time billed?

On fal, yes. Idle time, including the keep_alive window, is billed at the machine type's rate. Pending time and image pulls are not billed.

Sources

1. fal Serverless
2. fal pricing
3. fal docs: Serverless pricing
4. fal docs: Scaling your application
5. fal docs: Serverless FAQ
6. fal docs: Machine types
7. fal docs: Migrating to fal
8. fal docs: Migrate an external Docker server
9. FlashPack on GitHub
10. fal Learn: What Is a Serverless GPU Cold Start?

All sources accessed October 19, 2026.