The Challenge

Realtime voice leaves no room for slow infrastructure. A translation or spoken reply has to arrive while the speaker is still talking, so every new conversation needs a warm GPU. But traffic rises and falls with business hours and meetings, and a cold H100 takes minutes to load a model this size: a fleet sized for the peak wastes money, and one sized for the average makes users wait. Kotoba also ships model improvements every one to two weeks to its own cloud and AWS, so a platform that needed its own build of each model would slow every release. And offering Koto to developers beyond enterprise contracts would mean building keys, billing, and a playground around every model.

The Solution

Koto runs on fal as four apps on H100 GPUs. Streaming apps hold a WebSocket open and return partial transcripts, translated audio, and synthesized speech as they are generated; batch transcription is a standard HTTP request. Two things make this work: Kotoba’s Docker image is deployed in fal’s runtime, and fal scales it with demand.

One Docker image, a thin wrapper

Koto ships as one GPU image that runs unchanged on Kotoba's own cloud, on AWS, and on fal. There it runs in Direct Server Mode: a small launcher starts the image's own server, and fal routes HTTP and WebSocket traffic straight to it.

One environment variable selects the platform at startup. Set to fal, it enables a gateway layer of about 550 lines that reads the platform's request headers, fetches input audio from its storage, and reports billable units. Request handlers and model code are platform agnostic, so one image carries each model fix to fal and to enterprise deployments.

Each model is one deploy-profile entry: model, serving settings, and app name. One CI workflow builds the image, deploys it to a private staging environment, and promotes the same image to production once benchmarks pass. Adding a model is a new profile entry. Every deploy is a revision, so a rollback is one command that points the app back at the previous revision, with no rebuild.

Private apps for enterprise customers. A customer endpoint is one more profile entry, deployed by the same workflow: the same image with the customer's own model version, such as a domain fine-tune, and its own scale limits. The app stays private or is shared unlisted, the customer calls it with its own key at a customer-specific price, and it can scale to zero between sessions.

Autoscaling for realtime traffic

Each Koto app runs on H100 GPUs in three US regions. fal keeps at least one GPU warm per model, adds GPUs as concurrent sessions grow, up to ten per model, and holds 25% spare capacity warm so a new conversation lands on a ready GPU instead of a cold boot. Idle GPUs shut down after five minutes.

Each GPU serves many conversations at once: Koto's serving engine decodes all active streams in one batched step, and fal routes up to 80 streaming speech-to-text sessions, or 16 speech or translation sessions, to each GPU.

Faster cold boots with FlashPack. A GPU spun up from burst traffic must load the model before it can take a conversation. Kotoba converts its largest recognition and translation models to fal's FlashPack format at publish time and places the weights on a volume ahead of time, so a new GPU never downloads the model after a deployment. With FlashPack and cached compiled kernels, cold boots of these models decreased 75%.

Scaling is a configuration and doesn’t cost a release. fal's autoscaler works from a few numbers per app: minimum warm GPUs, maximum GPUs, spare capacity held ahead of demand, idle timeout, and regions. Kotoba sets starting values in its deploy profile, and anyone on the team can change them on the live app from the dashboard or with one cli command. Changes apply immediately and settings persist, so Kotoba can raise the warm minimum before a known busy period and lower it afterwards so idle GPUs stop costing money.

Built for the marketplace

A Playground for streaming models. fal's Playground sends one request and shows one response, which does not fit a WebSocket model. So each streaming app also exposes a one-shot HTTP route that drives its own WebSocket over loopback, letting developers try streaming Koto in the browser.

Billing by audio, even on WebSockets. HTTP responses carry their billable units in a header. For WebSocket sessions, the app reports units to the billing webhook when the session ends, so callers pay for the audio they send, not for idle connection time.

“We build one Docker image, and fal runs it unchanged. When traffic grows, fal adds H100s; when it drops, they go away. We spend our time on Koto, not on a GPU fleet.”

Machine Learning Engineer, Kotoba Technologies

Serving Koto and selling it are the same step: the deployment that runs Koto is also its storefront, so Kotoba deploys a model once and fal both serves it and lists it for developers. Batch and streaming speech-to-text, text-to-speech, and speech-to-speech translation are live as Partner models on the fal marketplace, each with a playground and an API callable with an existing key. Kotoba sets its own prices in the units its customers think in: per second of audio for transcription and translation, and per 1,000 bytes of text for speech. fal handles keys, billing, and discovery, so Kotoba gets distribution without building a developer platform, and each new Koto version reaches developers as soon as it is deployed.

Four Koto endpoints are live on fal today:

• Speech-to-text: batch transcription and realtime streaming
• Text-to-speech: streaming speech
• Speech translation: realtime speech-to-speech
  

“Developers find Koto on fal, try it in the Playground, and call it with the key they already have. That puts Japanese voice AI in front of builders we would never reach through enterprise sales alone.”

Machine Learning Engineer, Kotoba Technologies

Results

Koto went from its first full staging test on fal to production in five days, with no fal-specific fork of its serving code. All four models have since shipped new versions every one to two weeks, each as the same image Kotoba runs on its other platforms, and an enterprise customer's private app takes one profile entry. Capacity follows demand: fal keeps a warm GPU ready for the next conversation, adds GPUs as sessions grow, and releases them as traffic falls, and the team tunes those limits in seconds from the dashboard or CLI, without a release. Kotoba pays for the GPUs its traffic needs instead of a fleet sized for the peak, and built no API keys, billing system, or developer portal for these APIs.