kotoba-technologies/kotoba-sts/v1/realtime_voice
Loading API documentation
Endpoint: wss://fal.run/Kotoba-Technologies/kotoba-sts/v1/realtime_voice
Playground: upload a clip at https://fal.ai/models/kotoba-technologies/kotoba-sts/v1/speech-to-speech to get the translated text and speech.
Overview: Stream microphone audio in and receive both the translated text and the synthesized translated speech back on the same connection, with latency low enough for a live conversation. Source and target languages are set when the session opens; one session translates in one direction.
Use cases
Technical specifications
language; columns are the translated output, set in target_language):| Input \ Output | Japanese (ja) | English (en) | Korean (ko) | Mandarin Chinese (zh) |
|---|---|---|---|---|
Japanese (ja) | — | ● | ● | ▲ |
English (en) | ● | — | ● | ● |
Korean (ko) | ● | ● | — | ▲ |
Mandarin Chinese (zh) | ▲ | ● | ▲ | — |
Example input
{"type": "voice_session.update",
"session": {"input_audio_format": "pcm16", "input_audio_sample_rate": 24000,
"input_audio_transcription": {"language": "ja", "target_language": "en"},
"turn_detection": false}}
{"type": "input_audio_buffer.append", "audio": "<base64 pcm16>"}
{"type": "input_audio_buffer.commit"}Example output
{"type": "voice_session.updated", "session": {...}}
{"type": "conversation.item.input_audio_transcription.text.delta", "delta": "Good morning."}
{"type": "conversation.item.input_audio_transcription.audio.delta", "delta": "<base64 pcm16 24 kHz>"}
{"type": "input_audio_buffer.committed"}Example (Python SDK)
import kotoba
STS = "wss://fal.run/Kotoba-Technologies/kotoba-sts/v1/realtime_voice"
client = kotoba.KotobaClient()
# File in, translated audio out
result = client.s2st.translate("meeting_ja.wav", src="ja", tgt="en", url=STS)
result.to_wav("meeting_en.wav")Live microphone: send audio and play results concurrently, so translated speech starts while the microphone is still open.
import asyncio
import kotoba
STS = "wss://fal.run/Kotoba-Technologies/kotoba-sts/v1/realtime_voice"
async def main():
client = kotoba.AsyncKotobaClient()
async with client.s2st.stream(src="ja", tgt="en", url=STS) as session:
async def feed():
async for chunk in mic_chunks(): # PCM16 chunks from the microphone
await session.send_audio(chunk)
await session.commit()
feeder = asyncio.create_task(feed())
async for event in session: # runs while audio is still being sent
if event.type == "partial_transcript":
print(event.text, end="", flush=True)
elif event.type == "audio_chunk":
await play(event.audio)
elif event.type == "done":
break
feeder.cancel()
asyncio.run(main())Pricing: $0.035 per minute of session (unit: second of session, $0.000583 per second).
The kotoba-sdk package (PyPI, Python 3.10+, version 0.2.3 or newer) covers all four apps.
It sends the fal key with fal's Authorization: Key scheme automatically for fal.run URLs,
so use the fal key as the API key and point the per-route URLs at the apps.
pip install kotoba-sdk
export KOTOBA_API_KEY=<fal key>
export KOTOBA_ASR_REST_URL=https://fal.run/Kotoba-Technologies/kotoba-stt/v1
export KOTOBA_ASR_URL=wss://fal.run/Kotoba-Technologies/kotoba-streaming-stt/v1/realtime
export KOTOBA_TTS_URL=wss://fal.run/Kotoba-Technologies/kotoba-tts/v2/tts/ws
export KOTOBA_S2ST_URL=wss://fal.run/Kotoba-Technologies/kotoba-sts/v1/realtime_voiceAn app that has been idle may boot a GPU runner on the first request, which can take a few
minutes. Allow for that in the request timeout or retry the first request; the SDK's default
timeouts are 30 seconds for HTTP requests (KotobaClient(timeout=...)) and 15 seconds for the
WebSocket handshake.
Any WebSocket client works with this endpoint. The examples use ws for Node.js and websockets for Python.
npm install wsSet FAL_KEY as an environment variable in your runtime.
export FAL_KEY="YOUR_API_KEY"This endpoint speaks its own message protocol over a WebSocket. Open the connection with any WebSocket client and authenticate the upgrade with the Authorization: Key header. Browsers must not hold the API key: connect through your own server, which adds the header. The message protocol itself is documented in the model's About section.
import WebSocket from "ws";
// Keep the API key on your server; a browser connects through your
// own backend, which adds the header. Use the "ws" package: Node's
// global WebSocket ignores the headers option and the upgrade gets 401.
const ws = new WebSocket("wss://fal.run/kotoba-technologies/kotoba-sts/v1/realtime_voice", {
headers: { Authorization: "Key " + process.env.FAL_KEY },
});
ws.on("open", () => {
// Send the app's first message here.
});
ws.on("message", (data, isBinary) => {
if (isBinary) return; // binary frame, e.g. audio
console.log(data.toString()); // the app's messages, shown as text
});
ws.on("error", console.error);The API uses an API Key for authentication. Read it from the FAL_KEY environment variable on your server and send it as the Authorization: Key <FAL_KEY> header on the WebSocket upgrade, as the real-time snippet above does.
When running code on the client-side (e.g. in a browser, mobile app or GUI applications), make sure to not expose your FAL_KEY. Instead, use a server-side proxy to make requests to the API. For more information, check out our server-side integration guide.
Some attributes in the API accept file URLs as input. Whenever that's the case you can pass your own URL or a Base64 data URI.
You can pass a Base64 data URI as a file input. The API will handle the file decoding for you. Keep in mind that for large files, this alternative although convenient can impact the request performance.
You can also pass your own URLs as long as they are publicly accessible. Be aware that some hosts might block cross-site requests, rate-limit, or consider the request as a bot.
We provide a convenient file storage that allows you to upload files and use them in your requests. You can upload files using the client API and use the returned URL in your requests.
import { fal } from "@fal-ai/client";
const file = new File(["Hello, World!"], "hello.txt", { type: "text/plain" });
const url = await fal.storage.upload(file);The client will auto-upload the file for you if you pass a binary object (e.g. File, Data).
Read more about file handling in our file upload guide.