Skip to main content
Use Tailscale to connect your runners to your private network. Nothing is exposed to the internet. You can:
  • Reach private services from your app, for example an internal API or a model store.
  • Connect to a runner with SSH, for example to run commands, copy files with scp, or forward a port.
Tailscale is a private network service. Each runner joins your private network, called a tailnet, when it starts. Tailscale removes the runner from the tailnet after the runner stops.
For a quick shell on a runner, use fal runners shell. You do not need Tailscale for that.

Prerequisites

  1. In the Tailscale admin console, create an auth key. Make it reusable and ephemeral. Add a tag, for example tag:fal-runner.
  2. Add an access control policy that lets tag:fal-runner reach only the services that your app needs.
  3. Store the auth key as a fal secret:
Use a reusable key because each runner joins the tailnet with the same key. Use an ephemeral key so that Tailscale removes stopped runners automatically.
An auth key expires after 90 days or less. After the key expires, new runners cannot join the tailnet, and setup() fails. Create a new key and update the TS_AUTHKEY secret before the old key expires.

Example App

By default, runners do not have the CAP_NET_ADMIN capability, and they cannot create a TUN device. For this reason, the app runs tailscaled in userspace networking mode. In this mode, the app sends traffic to the tailnet through a local SOCKS5 or HTTP proxy.
This app installs Tailscale in a custom container image. The setup() method starts tailscaled, joins the tailnet, and waits for the connection. The endpoint then calls a private service through the SOCKS5 proxy.
Replace my-internal-api with the MagicDNS name or Tailscale IP address of your service.

Send Traffic Through the Tailnet

In userspace networking mode, only traffic that goes through the proxy reaches the tailnet. Other traffic uses the normal runner network. Use one of these methods:
  • SOCKS5 proxy on localhost:1055. Use this proxy for clients that support SOCKS5.
  • HTTP proxy on localhost:1056. Use this proxy for HTTP clients.
Set the proxy on each client that must reach the tailnet. Do not set it for other traffic.
Use the socks5h:// scheme, not socks5://. With socks5://, some clients such as requests look up the host name on the runner. The runner cannot resolve MagicDNS names, so the request fails.

Connect to a Runner with SSH

You can use the tailnet to open an SSH session on a runner. The session runs inside the runner, so you see the same files and processes as your app. In userspace networking mode, tailscaled sends incoming tailnet connections to services on localhost. You do not need to open a public port.

Use Tailscale SSH

Tailscale SSH uses your tailnet identity, so you do not manage SSH keys. Add --ssh to the tailscale up command in setup():
Then add an SSH rule to your tailnet policy. This rule lets members of your tailnet connect to runners as root:
From a device on your tailnet, connect with the hostname of the runner. Find the hostname on the Machines page of the admin console, or run tailscale status.

Use OpenSSH

If you want to use SSH keys, run sshd in the runner. Install openssh-server in your container image. Then start sshd in setup() before tailscale up:
Store your public key as a fal secret named SSH_PUBLIC_KEY, and add it to the secrets list of the app. sshd listens only on localhost. Only devices on your tailnet can connect to it.
If the SSH client also runs tailscaled in userspace networking mode, for example on another runner, add -o ProxyCommand="tailscale --socket=/tmp/tailscaled.sock nc %h %p" to the ssh command.

Troubleshooting

  • setup() fails with KeyError: 'TS_AUTHKEY'. Make sure that the TS_AUTHKEY secret exists in the app environment.
  • tailscale up fails with invalid key. The auth key expired, was revoked, or was copied incorrectly. Create a new key and set the secret again.
  • The runner refuses the SSH connection. Make sure that your tailnet policy has an SSH rule that lets you connect to tag:fal-runner.
  • The runner joins but cannot reach the service. Make sure that your access control policy lets tag:fal-runner reach the service and port.
  • Connections are slow. If a direct connection is not possible, Tailscale sends traffic through a DERP relay. Run tailscale --socket=/tmp/tailscaled.sock status in setup() and log the output to see the connection type.
  • Old nodes stay in the admin console. Use an ephemeral auth key. Tailscale removes ephemeral nodes some time after they go offline.