How to Deploy ML Models on GPUs Without Kubernetes

Putting a model behind an API usually means a lot of infrastructure. You need a container image with the right CUDA version, a GPU node pool, an autoscaler, a load balancer and a queue. You also need health checks, rollouts and dashboards. None of it is the model, and all of it needs someone to keep it running.

fal Serverless replaces that stack with a Python class. You declare the hardware and dependencies, load the model in setup(), and expose it with @fal.endpoint. fal builds the image remotely, so you don't need Docker installed. It then runs the app on GPU runners that scale with traffic and bills per second of runner time. If you already have a Dockerfile or a running HTTP server, those deploy too, with the same scaling and observability. This guide shows how to choose between the three paths, what fal run and fal deploy do, and what to configure before production traffic arrives.

Three ways to deploy

Pick the path that matches what you already have:

All three get the same autoscaling, analytics, logs and runner management. The rest of this guide starts with the native path and then covers the other two.

A model in one class

import fal
from pydantic import BaseModel

class Input(BaseModel):
    prompt: str

class Output(BaseModel):
    result: str

class MyModel(fal.App):
    machine_type = "GPU-H100"
    requirements = ["torch", "transformers"]

    def setup(self):
        from transformers import pipeline
        self.pipe = pipeline("text-generation", model="gpt2", device="cuda")

    @fal.endpoint("/")
    def generate(self, input: Input) -> Output:
        result = self.pipe(input.prompt, max_length=50)[0]["generated_text"]
        return Output(result=result)

From the introduction to fal Serverless.

Each part has a job:

• machine_type chooses the hardware. CPU types run from XS to XL. GPU types run from GPU-A100 (40 GB) through GPU-L40, GPU-H100, GPU-RTXPRO6000 and GPU-H200 to GPU-B200 (192 GB). Set num_gpus for multi-GPU runners.
• requirements lists pip packages. fal builds them into a container for you. Pin exact versions for production, so that every build is the same.
• setup() runs once when a runner starts. Load the model here, not in the endpoint, so every request on that runner reuses it.
• @fal.endpoint exposes a method as an HTTP route. The Pydantic input and output models become an OpenAPI schema, which drives request validation and the Playground form for your app.

Large weights belong on /data, the persistent volume every runner in your account mounts. fal sets HF_HOME there, so Hugging Face downloads persist across runners without extra code.

Validate with fal run, ship with fal deploy

fal run path/to/myapp.py::MyModel
fal deploy path/to/myapp.py::MyModel

fal run starts the app on a temporary runner and gives you a throwaway URL. It runs setup() and your endpoints the same way production will. Missing dependencies, import errors and model-loading failures therefore show up here, not as a crash loop after you deploy. A fal run deployment is public by default, so anyone with the URL can call it while it runs.

fal deploy creates the permanent deployment. Your app gets a stable endpoint ID, your-username/app-name. The name comes from the class name (MyModel becomes my-model) unless you set app_name or pass --app-name. It is private by default: callers need an API key from you or your team. Call it with a fal client, or over HTTP at https://queue.fal.run/your-username/app-name.

To see what a deploy will change before it happens, add --check. It shows the target app and environment, the auth mode, the rollout strategy and the configuration changes, then asks you to confirm.

What happens during a deploy

fal uploads your code and builds the container image in the cloud. It computes a hash of the Dockerfile, the dependencies, the build args and the files in the build context. If an identical image already exists, the build is skipped. Otherwise, unchanged Docker layers are reused and only the changed layers are rebuilt. Runners pull the finished image when they start, so a smaller image means a faster cold start.

If the app is already running, the default rolling strategy starts a runner on the new revision first. It waits for setup() to finish and the runner to pass its health check, and only then switches traffic. If setup() fails, the deploy stops and traffic stays on the old revision. The recreate strategy (--strategy recreate) switches straight away, and the next request pays a cold start. Every deploy creates a revision, so you can roll back to an earlier one.

Bring your own container

When pip packages aren't enough, for example when you need system libraries, a particular CUDA base image or non-Python tools, define the image yourself:

import fal
from fal.container import ContainerImage

class MyModel(fal.App):
    image = ContainerImage.from_dockerfile_str("""
        FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
        RUN apt-get update && apt-get install -y ffmpeg curl
        RUN pip install diffusers transformers
        RUN pip install fal
    """)
    machine_type = "GPU-A100"
    # setup() and endpoints as in the native example

ContainerImage.from_dockerfile() reads a Dockerfile from disk instead. A custom container has two requirements:

1. curl must be installed. fal uses it while a runner starts.
2. pydantic, protobuf and boto3 must be compatible with the fal SDK. The simplest way is to install the fal package as the last step of your Dockerfile.

Build secrets use BuildKit secret mounts, so a private package index token never ends up in an image layer. Base images can come from private registries on Docker Hub, Google Artifact Registry, Amazon ECR or Azure Container Registry. When you deploy a file reference such as app.py::MyModel, the file that defines your fal.App class isn't baked into the image. It is serialized and sent at deploy time, so editing app code doesn't trigger a container rebuild.

Already have an HTTP server?

If your model already runs behind FastAPI, Flask or any other HTTP server, you don't have to rewrite it. Direct Server Mode forwards traffic straight to your server's port:

[tool.fal.apps.my-server]
auth = "private"
machine_type = "GPU-A100"
exposed_port = 8000

[tool.fal.apps.my-server.image]
dockerfile = "Dockerfile"

The server must listen on 0.0.0.0 on that port. Your existing routes, middleware and response formats keep working. A server that serves /generate is called at your-username/my-server/generate. To unlock the Playground and full analytics, serve your OpenAPI spec at /openapi.json. One more detail: fal run ignores the auth value in pyproject.toml and defaults to public, so pass --auth private when you test a server that shouldn't be reachable. See Migrate an external Docker server for a full walkthrough.

Before production traffic

A few settings are worth deciding on purpose:

• Scaling. min_concurrency defaults to 0, so the app scales to zero when idle. keep_alive (60 seconds by default) sets how long an idle runner waits for the next request. max_concurrency caps how many runners can run at once, and is 2 if you don't set it. After the first deploy, a redeploy keeps the previous scaling values: changed values in code only apply with fal deploy --reset-scale, or change them live with fal apps scale. See the serverless inference guide for how they work together.
• Startup time. If setup() downloads a large model, raise startup_timeout so the first boot can finish.
• Fallback hardware. machine_type can be a list, such as ["GPU-H100", "GPU-A100"]. fal tries the types in order when the first has no capacity.
• Regions. By default an app can run in any available region. Set regions to restrict it, for data residency or latency.
• Environments. Create an environment with fal environments create staging, then deploy with --env staging. That creates a separate app with its own configuration and secrets.
• Shutdown. When a runner stops, fal calls handle_exit() straight away and teardown() after in-flight requests finish. Finishing those requests and running teardown() share termination_grace_period_seconds, which is 5 seconds by default and at most 1 hour.

Once the app is live, App Analytics shows request volume, latency percentiles, runner utilization and startup duration. Error Analytics breaks failing requests down by endpoint and status code, with logs and stack traces for each one. You can also export metrics through a Prometheus-compatible API or forward logs with log drains.

FAQ

How do I deploy a PyTorch or Hugging Face model as an API?

Write a fal.App that lists your pip requirements, loads the model in setup() and exposes it with @fal.endpoint. Validate it with fal run. Then fal deploy publishes it as your-username/app-name, which you can call with the fal client SDKs or at queue.fal.run.

Do I need Docker or Kubernetes to deploy on fal?

No. fal builds your app into a container image remotely, so you don't need Docker installed locally. fal also handles provisioning, scaling, networking and observability. You can still bring a Dockerfile when you want full control of the image.

Can I bring my own Docker image?

Yes. ContainerImage.from_dockerfile_str() or ContainerImage.from_dockerfile() builds from your Dockerfile, and base images can come from private registries. The image needs curl and fal-compatible versions of pydantic, protobuf and boto3. Installing fal as the last step covers both.

I already have a FastAPI server. Do I have to rewrite it?

No. Direct Server Mode forwards traffic to your server's port: set exposed_port in pyproject.toml and bind the server to 0.0.0.0. Existing routes, middleware and response formats keep working, with the same scaling parameters as a fal.App.

What happens to my model when there is no traffic?

min_concurrency and concurrency_buffer default to 0. With no traffic, runners shut down after keep_alive (60 seconds by default), and the app scales to zero. A concurrency_buffer above 0 keeps that many runners warm even with no traffic.

Do I need approval to deploy my own model?

Yes. Deploying your own apps on fal Serverless needs access that the fal team approves for each account. Request it on the Serverless Get Started page. Creating a fal account is free, and calling hosted Model APIs needs no approval.