How to Serve a Cog Model or Custom Container as a Scalable API

Packaging a model as a container was the hard part, and Cog already did it. What teams usually want next is control: over how many GPUs stay warm, over the GPU type, over what you pay when traffic is quiet. The good news is that you do not have to repackage the model. A Cog model is a container image with its own HTTP server, so any platform that runs containers can serve it.¹ This page covers what to look for in that platform, what changes when you move, and how scaling and cost work on fal Serverless.

For the exact commands, follow the docs guide "Migrate a Cog Model or Custom Container to fal", linked from Migrating to fal.

What to look for in a platform for a custom model API

Four things decide whether a move is worth it:

1. It runs your image as it is. You should not have to rewrite the predictor to get it live. A rewrite can come later, if it earns its keep.
2. You set the scaling. A floor of warm runners for steady traffic, a ceiling to cap cost, and scale to zero for idle models.
3. You pay for runner time you can see. Per-second billing, with a clear list of the runner states that are billed.
4. Callers get a queue. Long generations need an async API with status polling and webhooks, not only a blocking HTTP call.

fal Serverless does all four. It runs on the same infrastructure as fal's Model APIs, and a deployed app is called with the same clients and queue as any hosted model.²

Three ways to move a Cog model to fal

The times include the first image build and a test deploy. Large weights add download time.

Most teams should start with path 1. It proves that the model runs, scales and costs what you expect, before you spend a day on a rewrite. Move to path 3 when callers need file URLs instead of the data URIs a Cog server returns, or when you want a typed schema and the Playground form.

The same three paths work for a Dockerfile that has nothing to do with Cog. Any HTTP server in a container can run in Direct Server Mode: fal starts the container and forwards requests to the port you expose.³

What running the image looks like

For path 1, push the image to a registry fal can pull from, then describe the app in pyproject.toml. A minimal entry looks like this:

[tool.fal.apps.my-cog-model]
auth = "private"
machine_type = "GPU-H100"
exposed_port = 5000
keep_alive = 300

[tool.fal.apps.my-cog-model.image]
image = "docker.io/my-org/my-cog-model:v1"

[tool.fal.apps.my-cog-model.health_check]
path = "/health-check"
start_period_seconds = 600

Test it on a temporary runner with fal run my-cog-model --auth private, then ship it with fal deploy my-cog-model. Pass --auth private to fal run: test deployments are public by default. The image can come from a private registry; you give fal the registry credentials as a fal secret.

Callers send the same Cog request body to the /predictions path under your endpoint ID, through fal's queue:

curl -X POST "https://queue.fal.run/<your-username>/my-cog-model/predictions" \
  -H "Authorization: Key $FAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"input": {"prompt": "a sunset over mountains"}}'

The queue returns a status_url and a response_url to poll. You do not need Docker on your machine to deploy. fal builds images remotely and skips the build when nothing has changed.⁴

What changes after the move

How scaling works for a migrated model

Scaling is yours to set, on the image-only path as much as on a fal.App. Three settings matter most:⁵

• min_concurrency: runners kept alive at all times. The default is 0, so a model with no traffic scales to zero.
• max_concurrency: the most runners at once. It is 2 when you do not set it, so raise it before real traffic arrives.
• keep_alive: seconds an idle runner waits for the next request. The default is 60. The example above sets 300.

A Cog model that loads large weights at start-up has a long cold start. Keep weights on /data, the persistent volume shared by every runner in your account, instead of baking them into the image. The image stays small and only the first runner pays the download. For the trade-offs of a warm floor, see What Is a Serverless GPU Cold Start?.

What it costs

Per-prediction thinking does not carry over. On fal Serverless you pay per second while a runner is alive, at the rate for its machine type.⁶ Setup, idle (including keep_alive), running, draining and teardown are billed. Waiting for hardware and pulling your image are not. Requests that fail with an HTTP 5xx status are not charged.

fal's published rates for the GPUs most Cog models fit on:⁷

"As low as" rates are quoted through fal's sales team.

To estimate a month, count runner-seconds, not predictions. Take a model that needs 4 seconds of H100 time per prediction and serves 20,000 predictions a day, one at a time per runner:

• Busy time: 20,000 × 4 s = 80,000 s a day.
• Busy cost at list price: 80,000 × $0.00125 = $100 a day, or $0.005 a prediction.
• Billed time is higher than busy time, because setup and keep_alive are billed too. App Analytics shows your real startup, execution and idle time per app.⁸

If one H100 would be busy most of the day, compare against reserved capacity with the break-even guide.

Before you send production traffic

• Raise max_concurrency to cover your peak.
• Set start_period_seconds on the health check long enough for the server to load its weights.
• Keep the app private and give callers a fal API key (Authorization: Key $FAL_KEY).
• Open the app in the Apps dashboard and check its Logs and Requests pages after a test run.

Serverless access is approved per account. Request it at fal.ai/dashboard/serverless-get-started. For other ways to bring a model, see How to Choose a Serverless GPU Platform.

FAQ

Can I run my Cog image without changing it?

Yes. Add one app entry to pyproject.toml with the image name and exposed_port = 5000. fal starts the container and forwards requests to the Cog server, so the /predictions API stays the same.

Do I have to rewrite my predictor as a fal.App?

No. It is optional. Convert it when you want typed Pydantic inputs, fal CDN URLs for output files and the full Playground. setup() stays the same and predict() becomes an @fal.endpoint method.

How is a migrated model billed?

Per second while a runner is alive, at the machine type's rate, not per prediction. Pending time and image pulls are not billed, and 5xx failures are not charged.

Can I bring a Dockerfile that is not a Cog image?

Yes. Run any HTTP server in Direct Server Mode with exposed_port, or reference the Dockerfile from a fal.App with ContainerImage.from_dockerfile().⁹