Production Serverless GPU Inference: What It Needs and How to Set It Up on fal

Most serverless GPU platforms look the same on a demo URL. They differ in production: when traffic triples at 3 a.m., when a GPU fails mid-request, when the bill arrives. Before you commit to a provider, check that you can set the eight controls below on your own app, not only read about them. This page lists them and shows the setting for each on fal Serverless.

To compare providers before a trial, use How to Choose a Serverless GPU Platform. This page is about what you configure once you have picked one.

Which serverless GPU providers are ready for production?

Look for a provider that runs its own production traffic on the same platform you would use. A platform that carries that traffic every day has already met the problems you will meet. fal Serverless is that platform for fal: every model in fal's Model APIs is a fal.App running on it.¹ fal publishes a 99.99% uptime SLA, more than 1,300 endpoints in production and scale to thousands of GPUs.²

Then check the eight requirements in this table against your own model.

Scaling controls and cold starts

A production app needs three numbers you choose: the floor, the ceiling and the buffer. On fal they are class attributes.

• min_concurrency keeps runners alive whatever the traffic. The default is 0, so the app scales to zero.
• max_concurrency caps the total runners. The default is 2 when you do not set it. Raise it before launch, or traffic above two runners waits in the queue.
• concurrency_buffer keeps extra warm runners ahead of demand, for bursts.
• keep_alive keeps an idle runner for a number of seconds after its last request. The default is 60.

A warm floor removes cold starts for steady traffic. It is billed all day. For how cold starts happen and how to size a floor, see What Is a Serverless GPU Cold Start?. For recipes per traffic shape, see Serverless Inference: Autoscaling and Scale to Zero.

Multi-GPU runners

Some models do not fit on one GPU. Set num_gpus on the app to give each runner more than one GPU. The fal.distributed module splits one request across the GPUs of a runner, or runs several outputs at once. Each GPU is billed: a runner with 2 GPUs alive for 60 seconds is 120 GPU-seconds.

Logs and metrics you can export

On fal, every deployed app has App Analytics (request volume, latency percentiles, runner utilization) and Error Analytics (failed requests by endpoint and status code, with logs). The Logs page filters by runner ID, request ID or endpoint.

To send data to your own tools:

• Log drains forward logs in real time to any HTTPS endpoint, as signed NDJSON.¹⁴
• Metrics come from https://api.fal.ai/v1/serverless/metrics in Prometheus format: runners by state, queue size, requests completed, latency buckets, GPU and VRAM use.

What happens when a request fails

Production traffic fails in small ways all day. Check three things on any provider: is the request retried, is the bad runner replaced, and do you pay for the failure.

On fal, queue requests are retried on a new runner for server errors, timeouts and connection errors, up to 10 attempts in total. The status code your endpoint returns tells fal what to do with the runner:

You can turn off retries for a condition with skip_retry_conditions. A health check endpoint can replace a runner after 3 failed checks in a row. fal also monitors GPU temperature, clocks and throttling across all runners. When a GPU goes bad, fal's operations team can cordon the node and drain its runners. Requests that fail with an HTTP 5xx status are not charged.

Regions and capacity

A traffic spike can arrive when your GPU type has no free capacity. Give machine_type a list, and fal tries each type in order. By default an app runs in any available region. Set regions to restrict it for data residency or latency. For capacity that must be there, fal offers reserved GPU allocations. Concurrency limits for Model APIs do not apply when you call your own Serverless endpoints; max_concurrency sets your throughput.¹⁵

Private apps, environments and safe deploys

fal deploy makes an app private by default: only your account and team can call it, with an API key. fal run test deployments are public by default, so pass --auth private when you test with real data. A third mode, shared, lets other fal users call your app and pay for their own usage.

Keep staging apart from production with environments: fal deploy --env staging deploys a separate app with its own secrets.¹⁶ Deploys are rolling by default. fal starts a runner on the new revision and switches traffic only when it is healthy. Every deploy is a revision, and you can roll back to any earlier one.¹⁷

Pricing you can predict

fal Serverless bills per second while a runner is alive. Setup, idle, running, draining and teardown are billed. Waiting for hardware and pulling the image are not. Prices on fal.ai/pricing, per GPU-hour:

"As low as" rates are quoted through fal's sales team. Two numbers make a budget predictable:

• The most you can spend per hour is max_concurrency × num_gpus × the hourly rate. An app capped at 10 single-H100 runners cannot cost more than $45 an hour at list price. A rollout can briefly start replacement runners above the cap.
• The cost of a warm floor is min_concurrency × the hourly rate × 24 hours. One H100 held warm costs $108 a day at list price.

If your GPUs are busy most of the day, part of the traffic may be cheaper on reserved capacity. This break-even guide works it out.

A production configuration on fal

All eight controls sit on one class. Test it with fal run app.py::MyModel --auth private, then ship it with fal deploy app.py::MyModel.¹⁸

import fal

class MyModel(fal.App):
    machine_type = ["GPU-H100", "GPU-H200"]  # tried in order
    num_gpus = 1
    regions = ["us-east"]
    app_auth = "private"

    min_concurrency = 1        # warm floor
    concurrency_buffer = 1     # headroom for bursts
    max_concurrency = 10       # cost ceiling
    keep_alive = 300
    startup_timeout = 600      # time allowed for setup()
    request_timeout = 600
    termination_grace_period_seconds = 120
    skip_retry_conditions = ["timeout"]

    def setup(self):
        self.model = load_model()

    @fal.endpoint("/health", health_check=fal.HealthCheck(failure_threshold=3, call_regularly=True))
    def health(self) -> dict:
        return {"status": "ok"}

    @fal.endpoint("/")
    def predict(self, input: dict) -> dict:
        return self.model.run(input)

Serverless access is approved per account. Request it at fal.ai/dashboard/serverless-get-started.

FAQ

What makes a serverless GPU provider production-ready?

Control over the floor, the ceiling and the buffer; retries and runner replacement when requests fail; logs and metrics you can export; private access by default; and a cost you can cap. Check each on your own model during a trial.

Does fal charge for failed requests on Serverless?

No, not for server errors. Requests that fail with an HTTP 5xx status are not charged. Time a runner spends waiting for hardware or pulling its image is not billed either.

How do I stop a serverless GPU bill from running away?

Set max_concurrency. Your hourly cost cannot go above max_concurrency × GPUs per runner × the hourly rate, apart from short overlaps during a rollout. Keep min_concurrency at 0 for apps that can wait for a cold start.

Can I run a model that needs more than one GPU?

Yes. Set num_gpus for runners with several GPUs, and use fal.distributed to spread work across them. A runner with N GPUs is billed N times its duration.