How to Switch Serverless GPU Providers Without Downtime

Most teams choose their first GPU provider calmly, on price and docs. They choose the second one during an incident. The queue is backing up, users see errors, and nobody at the provider answers. That is the worst moment to evaluate anything. You compare a demo against your worst day, and you tend to move to whoever replies fastest.

The reasons teams look elsewhere tend to rhyme: capacity that isn't there when traffic spikes, runners that misbehave under load, outages, and help that is hard to reach when it matters. Much of this can be tested before you commit. This guide turns those concerns into tests, then covers how to move with as little disruption as possible.

Start from your last incident

Before you look at any provider, write down what failed: timestamps, queue wait, error rate by status code, and how long the provider took to respond. That record is your test plan. A new provider must survive a replay of your worst week, on your GPU type and in your regions.

1. Capacity: will a GPU be there at your peak?

Capacity failures look like requests sitting in a queue while the provider's status page is green. Test for them directly.

• Burst from cold. Scale to zero, then ramp to two to three times your worst observed peak of concurrent requests within a minute. Record queue wait separately from execution time, at p50, p95 and p99.
• Test at your own peak hour. An off-peak test on a common GPU tells you nothing about a weekday afternoon on the newest hardware.
• Ask what happens when your GPU type runs out. Does the platform fall back to another type or region, or do requests wait?
• Find the account limits. Many platforms cap concurrent runners per account. That cap, not the hardware, is often what stops a burst.
• Get capacity commitments in writing. "We have plenty" is not a commitment. Ask how many GPUs of which type are reserved for you, at what price, and how fast that can grow. To see whether reserved capacity pays off for your traffic, find the break-even point.

2. Stability: do runners stay healthy under load?

Instability is not an outage. The platform is up, but runners crash, hang, restart or get moved, and a small share of requests fail all day.

• Soak test. Run production-like traffic for two to three days. Count non-2xx responses, timeouts and runner restarts per thousand requests.
• Split errors by cause. Separate your own exceptions from platform causes: out-of-memory kills, host failures, network resets. Only the second group tells you about the provider.
• Deploy mid-test. Push a new version during the soak. Count the requests that fail while old runners are replaced.
• Ask about in-flight requests. When a host fails or is drained for maintenance, is the request retried, re-queued or lost? Does your code get a shutdown signal and time to finish?

3. Outages: how often, how long, how honest

• Read a year of status history. Count incidents that affected inference, and note how long each lasted. Check whether the status page moved before or after customers complained.
• Read the incident write-ups. A useful one names the cause, the blast radius and the fix. A provider with no write-ups has given you an answer.
• Get the SLA in writing. Ask what it covers: the control plane, the API, or your endpoint end to end. Ask what the credit is when it is missed.
• Check the blast radius. Can you pin your app to a region, or spread it across several, so one failure does not take you down?

4. Support: who answers when it breaks?

• Open a real ticket during the trial, ideally outside business hours. Time the first response and the time to a useful answer, not an acknowledgement.
• Ask what channel you get: a shared chat channel, a named engineer, an on-call path for severity-one issues, or an email queue.
• Ask whether support can see your runners and logs without you pasting screenshots.

5. What failure costs you

Outages cost twice: in lost users and on the invoice. Check whether failed requests, time waiting for hardware, image pulls and automatic retries are billed. Then price last month's failures under each provider's rules.

How to migrate without downtime

The safe migration never has a moment when only the new provider can serve traffic.

1. Make the container portable. One image, configuration through environment variables, weights loaded from storage you control. Keep provider SDK calls out of the request path, behind a thin adapter.
2. Deploy privately on the new provider. Run your existing smoke tests against it before any real traffic arrives.
3. Put your own router in front. Route each request by weight to one provider, and fail over to the other on timeouts and 5xx responses. Give retries a budget, so a failing provider does not double the load on the healthy one.
4. Shadow first, then ramp. Copy live requests to the new provider and discard its responses. For generative models, compare with fixed seeds, or check that outputs are valid rather than identical.
5. Keep the old provider warm. Leave a small floor of runners there until the new one has carried several real peaks.

Cold starts look worse in the first days, before warm runners build up. Judge them after the ramp. See What Is a Serverless GPU Cold Start?

Questions to ask any GPU provider

1. How many GPUs of my type will you reserve, in writing?
2. What happens to a request when my GPU type has no free capacity?
3. Which account limits apply to my endpoints, and who raises them?
4. Is a request retried or lost when its runner fails?
5. Are failed requests billed?
6. What does the SLA cover, and where are your incident write-ups?
7. Who answers a severity-one ticket at 3 a.m.?
8. Can I run my existing container unchanged?

For the broader criteria behind a first choice, see How to Choose a Serverless GPU Platform.

Why teams move to fal, test by test

fal built its Serverless platform to run its own inference first: every fal Model API runs on it, and it powers more than 1,300 production endpoints across thousands of GPUs.¹ The platform your endpoint lands on carries fal's own production traffic every day. Here is how it answers each test above, so you can check every point during a trial.

Capacity when traffic spikes

• Fallback across GPU types. List several machine types in order, and fal uses the next one when the first has no free capacity.²
• Every region by default. An app can run in any available region unless you restrict it for data residency or latency.³
• Warm runners ahead of demand. min_concurrency keeps a floor of warm runners. concurrency_buffer or concurrency_buffer_perc keeps spare runners ready before the next burst arrives.⁴
• Capacity in writing. Enterprise plans add dedicated serverless infrastructure, agreed with the fal team rather than left to chance. H100s list at $4.50 per hour and go as low as $2.49 per hour.⁵⁶

Runners that stay healthy under load

• Automatic retries on the queue. When a runner returns a 503 or 504, times out or drops the connection, fal re-queues the request and tries again, up to 10 attempts in total. You control which conditions retry.⁷
• GPU health is watched for you. fal monitors GPU metrics across all runners. When a GPU goes bad, the operations team cordons the node, drains its runners and resets the GPU.⁸
• Deploys that wait for health. Deploys are rolling by default: fal starts a runner on the new revision and switches traffic only after it finishes setup and is healthy. If setup fails, the old revision keeps serving.³
• Instant rollbacks. Every deploy creates a revision, and you can roll back to any earlier one immediately.⁹
• Time to finish in-flight work. When a runner must stop, including for host maintenance, termination_grace_period_seconds gives it up to an hour to finish requests.⁴

Outages in the open

• Public status history. status.fal.ai shows uptime per service and incident history, with alerts by email, Slack, Teams, webhook or RSS.¹⁰
• A 99.99% uptime track record. fal Serverless handles millions of inference calls a day at 99.99% uptime, and the status page publishes each service's history so you can check it yourself. Enterprise plans add contractual SLA guarantees.⁶¹¹⁰
• Room to spread risk. Because apps can run in any region, one region's trouble does not have to take an endpoint down.³

Support when it matters

• 24/7 priority on Enterprise plans, together with dedicated serverless infrastructure and SLA guarantees.⁶

Failures that do not inflate the bill

• Billing follows the runner's life, not its wait. Time waiting for hardware and image pulls is not billed, and requests that fail with a 5xx error are not charged.¹¹

A migration that starts with your existing container

fal's Direct Server Mode runs an existing Docker server with its code unchanged and forwards traffic to the port you expose. Images can come from a private registry, including Docker Hub, Google Artifact Registry, Amazon ECR and Azure Container Registry.¹² The configuration lives in pyproject.toml:¹³

[tool.fal.apps.my-server]
auth = "private"
machine_type = ["GPU-H100", "GPU-H200"]
exposed_port = 8000
keep_alive = 300
min_concurrency = 1
max_concurrency = 20
concurrency_buffer = 2
termination_grace_period_seconds = 300

[tool.fal.apps.my-server.image]
image = "my-org/my-server:latest"
cmd = ["your-server", "--host", "0.0.0.0", "--port", "8000"]

Two settings to change before cutover. max_concurrency defaults to 2, so raise it to cover your peak. The grace period defaults to 5 seconds, so set it to cover your longest request.⁴

fal run my-server --auth private starts a temporary private deployment for your smoke tests, and fal deploy my-server ships it. Your existing routes stay the same under your endpoint ID.¹²

Serverless access is approved per account. Request it on the Serverless get-started page, or talk to sales about reserved capacity.

FAQ

What is the best serverless GPU alternative?

The one that passes your own burst, soak and support tests at your peak hour, on your GPU type, with capacity commitments in writing. A provider that serves someone else's workload well can still struggle with yours, so judge each one on your own traffic.

How do I move inference to another GPU provider without downtime?

Deploy the same container on the new provider, shadow live traffic, then shift a growing share through your router. At no stage should only the new provider be able to serve.

References