What Is a Serverless GPU Cold Start?

One team goes from twenty GPUs to sixty inside a few minutes when their downstream customers wake up. Another has a two to three minute scale-up time written into the SLAs they sign, and loses deals over it. A third was turning customers away rather than risk the spike. Demand arrives faster than capacity does, and that is what breaks first when a team scales GPU inference in production.

The gap between demand arriving and a GPU being ready to answer is the cold start. Every model in the public catalog is served by runners that have to start before they can serve, so we hit cold starts on our own infrastructure daily. Most of what follows is what solving that for ourselves taught us.

If you want to watch one happen rather than read about it, create an account and grab an API key; the quick start takes about two minutes.

Whether a cold start is actually a problem for you depends on what you are building, not on a headline number. If you are already deep into building on fal, the docs cover optimizing cold starts directly. Here is where it actually starts.

What counts as a cold start

A runner moves through PENDING, then DOCKER_PULL if its image is not already cached on the node, then SETUP where your setup() loads the model, then IDLE once the health check passes. The docs draw the boundary explicitly: the time from PENDING to IDLE is your cold start latency.

Where you start that clock changes the number a lot, so it is worth knowing which one you are being quoted. It is tempting to start it at SETUP, since that is the first moment your code is doing anything. It also hides the two things a user actually waits through: waiting for hardware, and pulling a multi-gigabyte image. We measure from PENDING on purpose, because a cold-start number that excludes waiting for a GPU is not measuring what your user experienced.

Two of those states are free. The pricing page sets it out state by state: PENDING and DOCKER_PULL are not billed, and billing starts at SETUP, continuing through IDLE, RUNNING, DRAINING and TERMINATING. fal absorbs the scheduling and the image pull.

Why cold starts vary so much

Almost always, a starting runner is waiting on bytes, and where those bytes come from is the single biggest variable in the whole process.

Files on /data, which is where model weights belong, go through three cache layers:

A runner checks local first, then the datacenter cache, then the object store, populating both caches on the way back. So "how long is a cold start" is really another question: which layer did you land on?

This is why cold starts improve on their own as an app takes traffic. The first runner pulls the image from the registry and the weights from the object store, and pays the worst case. The second runner on that node finds both in local cache. Runners on other nodes read weights from the distributed cache, and the scheduler prefers nodes that already have your image. Over time the caches spread and cold starts converge on the time it takes to run setup() with everything already local.

The corollary matters just as much when you are estimating. Caches are not permanent: they are evicted when nodes recycle or come under pressure, so the first cold start after a quiet period can be slow again. If your traffic is quiet for hours and then spikes, that is the case to plan for, not the steady-state one.

Does this actually matter for your workload?

Three rough shapes, and the answer is different for each.

Batch and async work. It does not matter much. If you are generating a thousand images overnight or transcoding a queue, a cold start is a one-time cost amortised across the whole run, and scale-to-zero between runs is straightforwardly cheaper. Take the default and move on.

Interactive products. It matters, but as a tail problem rather than an average one. Most requests will land on warm runners; the ones that do not are the ones your users complain about. The fix is a small warm floor rather than a faster cold start, and the rest of this article is mostly about sizing that floor.

Hard real-time. Live video, voice, anything with a per-frame budget in the tens of milliseconds. A cold start of any length on the request path is unacceptable, so the answer is a warm floor sized to peak concurrency, with scale-to-zero reserved for genuinely idle periods. Cold-start work decides how small that floor can be. It does not remove the need for one.

The useful question is not "how fast are cold starts" but "how often will one land on a user, and what does it cost me when it does".

That depends on a second thing, separate from what kind of workload you run: how predictable your traffic is. A batch job can still arrive in an unpredictable rush, and an interactive product can run on a curve as steady as a clock. Workload type tells you how much a cold start costs you. Traffic pattern tells you which lever fixes it.

If your traffic is genuinely stable, keep_alive alone usually covers it: runners stay warm through the normal gaps between requests, and you are not paying for capacity you are not using. Add min_concurrency when that same stable curve still cannot tolerate the occasional cold start keep_alive lets through, since it holds a floor of runners on all the time rather than just between requests. Neither helps with a genuine surprise, though, which is what concurrency_buffer is for: it holds spare runners ahead of your current demand rather than your peak, so a burst lands on capacity that already exists instead of triggering a scramble to find it. This is the shape behind the highest-frequency question we hear on this topic, and the honest answer is that you are not expected to predict the spike. You are expected to tell fal how much runway to keep ahead of you.

Most real traffic is some mix of the last two, which is why min_concurrency and concurrency_buffer are usually set together rather than as alternatives.

Do you have to predict traffic and pre-warm capacity yourself?

No, and this is the part that most often surprises people arriving from self-managed GPU autoscaling, where the warm-up job and the scaling loop are yours to build and babysit.

Which brings us back to the team going from twenty GPUs to sixty. They do not need to forecast the spike. They need capacity that already exists when it arrives, and on fal that is configuration rather than code you write and operate:

• keep_alive, default 60 seconds. How long an idle runner survives after its last request. The cheapest lever, and usually the first one to reach for.
• min_concurrency, default 0. Runners alive at all times regardless of traffic. Your 24/7 warm floor, billed 24/7 to match.
• concurrency_buffer, default 0. Spare runners held beyond current demand, which is the one that answers bursts. Takes precedence over min_concurrency when higher.
• concurrency_buffer_perc, default 0. The same buffer as a percentage of current volume, so it grows with traffic instead of being a fixed number you have to revisit.
• scaling_delay, default 0. A pause before scaling up on a queued request, so brief spikes do not spawn runners you will not need.
• max_multiplexing, default 1. Concurrent requests per runner. Raising it means fewer runners for the same load, and fewer cold starts as a result.

The floor covers your baseline, the buffer absorbs the spike. For the twenty-to-sixty case, concurrency_buffer is doing the work: the burst lands on runners that already exist rather than on a scheduler that has to find forty GPUs first.

You can change any of them on a running app without redeploying, which matters more than it sounds. It means the setting is a knob you tune against real traffic, not a guess you bake into a release:

fal apps scale my-demo-app --min-concurrency 1 --concurrency-buffer 2

These also persist across deployments by default, so tuning survives a code deploy. Use fal deploy --reset-scale when you want the values in your code instead.

The trade is worth stating plainly, because warm capacity is not free: runners cost GPU-hours whether or not they are working, and idle time is the single biggest cost lever on the platform. A warm floor sized for peak traffic around the clock is the expensive way to solve this. A modest floor plus a buffer is usually the right shape.

All of that is tuning the shared pool: runners that scale with your traffic and, under load, could in principle be waiting on the same GPU type as someone else's burst. For teams that need guaranteed capacity rather than scaled capacity, fal also offers reserved GPU allocations, a pool of GPUs isolated to your account rather than shared across fal's customers. The docs state the effect directly: reserved capacity eliminates pending and setup time entirely, for a guaranteed pool of GPUs, giving predictable performance and economics. The shared pool trades that away for autoscaling and pay-per-use, at the cost of occasional contention. That contention has a concrete mechanism if you list multiple machine types as a fallback: machine types are tried in order, and if your preferred GPU has no available capacity, fal moves to the next one automatically. Different machine types bill at different per-second rates, so contention can mean landing on a pricier GPU for that runner rather than a change to the rate itself. If your workload cannot tolerate that variance, or needs unit economics fixed in advance, a reservation removes the question rather than asking you to size a buffer against it.

How small does a cold start get?

Small enough that the warm floor above can be modest, which is the practical thing you want to know.

The bytes are the bulk of it, and once they are on the node the constraint moves to how fast a loader can move them into GPU memory. That is why we wrote and open-sourced FlashPack, which streams weights from disk straight to GPU at up to 25 Gbps without GDS. At roughly 3.1 GB/s, a 20 GB model's weights land in about six to seven seconds, and a 5 GB model in under two. It is open source, so you can check that rather than take our word for it.

Beyond the weights, compiled kernel caches let one runner compile and the rest load, container image structure decides what a cache miss costs, and parallel file loading helps when your loader reads shards one at a time. Those are implementation concerns rather than evaluation ones, and the docs cover each properly.

When it isn't a cold start at all

Two things look like slow cold starts and are neither, and both come up early enough to be worth recognising.

Requests submitted through the queue sit in IN_QUEUE with a queue_position when every runner is busy, and stay there while fal scales up. Nothing fails. The request is waiting out somebody's cold start, so queue time nobody can account for is very often cold-start time nobody measured.

The other is CRASH_BACKOFF. If setup() crashes or times out, the runner is terminated, and to avoid a tight crash loop fal applies an incremental backoff to subsequent starts: each one delayed by a further 30 seconds, capped at 10 minutes. Runners sitting there are not starting slowly, they are waiting out a penalty from an earlier failure. One successful start clears it. Read your setup() logs before touching any scaling settings. Scheduling failures, where hardware is not available yet, use a flat 20-second wait instead.

Watch it happen on your own app

Reading about cold starts is a poor substitute for watching one. Here is the image generator from our getting-started guide, unchanged, because SDXL is big enough to have a cold start worth looking at:

import fal
from pydantic import BaseModel, Field
from fal.toolkit import Image

class Input(BaseModel):
    prompt: str = Field(
        description="The prompt to generate an image from",
        examples=["A beautiful image of a cat"],
    )

class Output(BaseModel):
    image: Image

class MyApp(fal.App):
    keep_alive = 300
    app_name = "my-demo-app"
    machine_type = "GPU-H100"
    requirements = [
        "hf-transfer==0.1.9",
        "diffusers[torch]==0.32.2",
        "transformers[sentencepiece]==4.51.0",
        "accelerate==1.6.0",
    ]

    def setup(self):
        # Enable HF Transfer for faster downloads
        import os

        os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"

        import torch
        from diffusers import StableDiffusionXLPipeline

        # Huggingface models will be automatically downloaded to
        # the persistent storage of your account (/data)
        self.pipe = StableDiffusionXLPipeline.from_pretrained(
            "stabilityai/stable-diffusion-xl-base-1.0",
            torch_dtype=torch.float16,
            variant="fp16",
            use_safetensors=True,
        ).to("cuda")

        # Warmup the model before the first request
        self.warmup()

    def warmup(self):
        self.pipe("A beautiful image of a cat")

    @fal.endpoint("/")
    def run(self, request: Input) -> Output:
        result = self.pipe(request.prompt)
        image = Image.from_pil(result.images[0])
        return Output(image=image)

Three lines in there are this whole article in miniature. keep_alive = 300 holds an idle runner for five minutes, so a second call inside that window skips the cold start completely. The comment about Hugging Face weights landing on /data is the cache hierarchy at work. And self.warmup() runs the first inference before the runner reports IDLE, moving that cost into fal's startup window rather than onto your first user, at the price of a longer billed SETUP.

Validate before deploying. fal run boots the app on a temporary worker and runs setup() exactly as production will, so a missing dependency surfaces here rather than as the CRASH_BACKOFF loop above:

fal run image_generator.py::MyApp
fal deploy image_generator.py::MyApp

Now watch it. Runners move through the exact states in the diagram above, and you can see them do it:

fal apps runners my-demo-app --state pending setup idle

It prints running counts for Runners Pending and Runners Setting Up, then a table with a State column and an Expires In column, which is the keep_alive countdown on each idle runner. Call the endpoint after a quiet period and watch a runner appear as pending, move to setting up, and land in idle before your response comes back. Call it again while Expires In is still counting down and there is nothing to watch, because the request went to the runner that was already there.

The CLI gives you the snapshot. For the distribution over time, the dashboard's Analytics page breaks the same thing down for you automatically:

Total startup time here is queue wait plus cold boot time, and the dashboard splits them so you are not left guessing which one to fix. In this example, queue wait sits at 0.00s at the 90th percentile, meaning nothing is waiting on capacity, while cold boot time at p90 is 18.97 seconds and 33% of requests are triggering one, defined here as any boot over one second. That split is the diagnosis: a high queue-wait number means you need more capacity (max_concurrency, concurrency_buffer), while a high cold-boot number, as it is here, means the fix is what this article has been about, either shortening setup() or keeping more runners warm with keep_alive and min_concurrency.

The goal on both charts is the same in either case: push cold boot rate toward zero and cold boot time toward the FlashPack floor, until a cold start stops being something your dashboard needs to show you at all.

Sign up free if you do not have an account yet.

Frequently asked questions

How fast are serverless GPU cold starts, and are they low enough for real-time inference?

It depends on image size, model size, setup() complexity, cache state, and hardware availability. FlashPack's 25 Gbps ceiling puts a 20 GB model's weight load at roughly six to seven seconds. For hard real-time, the answer is not a fast cold start but a warm floor sized to peak concurrency, so no request pays one.

Do I have to predict traffic and pre-warm capacity myself?

No. concurrency_buffer holds spare runners ahead of demand, min_concurrency holds a 24/7 floor, and keep_alive keeps runners alive between requests. All three are settings you change on a running app, not scripts you build and operate.

Is PENDING part of the cold start?

Yes. fal measures cold start as total wall-clock from PENDING to ready, including waiting for hardware and any image pull. A narrower definition starting at SETUP gives a smaller number but hides part of the wait your user actually felt.

Am I billed during a cold start?

Not for PENDING or DOCKER_PULL. Billing starts at SETUP and continues through IDLE, RUNNING, DRAINING, and TERMINATING. TERMINATED is not billed, and 5xx errors are not charged.

Why is my second cold start so much faster than my first?

Caching. The first runner pulls your image from the registry and weights from the object store. After that, both are in the node's local cache at 10-15 GB/s, or the datacenter cache at 6-8 GB/s. Caches also expire, so the first start after a quiet period can be slow again.

Why are my requests queuing?

Usually because every runner is busy and a new one is starting. Queued requests show IN_QUEUE with a queue_position and are dispatched as soon as a runner frees up.

My runners are in CRASH_BACKOFF. Is that a cold start problem?

No, it is a startup failure. setup() crashed or timed out, and starts are being delayed by 30 seconds more each time, up to 10 minutes. Check your setup() logs. A single successful start resets it.