@profullstack/live-translate

Live speech translation for streams and meetings. A listener picks a language and hears the speaker translated and re-voiced a few seconds behind, with an optional transcript. Extracted from nixamp's paid "translated audio" and reused by PairUX meetings, where every participant can hear every other participant in a language of their own.

• Hear: ElevenLabs Scribe v2 (word timestamps, speaker labels, language detection or a named language).
• Translate: pluggable TextTranslator (translate(texts, from, to)): a local model, an LLM, an API. Pairs are translated directly (French to German never goes through English unless your translator needs to).
• Speak: ElevenLabs Flash v2.5, streamed as 16 kHz PCM.
• Sell: prepaid CoinPay passes (Day $1, Month $5 by default) metered at 5x provider cost, plus a free daily allowance (10 sessions/account/day), on any Postgres Queryable.

Since 0.3.0 a pass is bought through a CoinPay hosted checkout session: one page where the buyer picks any coin the business takes, or PayPal. The sale is filed under the site's hostname (property), the buyer's account id and email.

const passes = new TranslationPasses({ db, key: coinpayKeyFrom(), site: "https://example.com", app: "example" });
// POST /checkout: email is required for a hosted session; accountId defaults to `by`.
const { url } = await passes.checkout(user.id, "day", "", requestKey, { email: user.email, accountId: user.id });
// Webhook: verify the raw body, then hand over the whole event.
if (!verifyCoinpaySignature(raw, req.headers["x-coinpay-signature"], secret)) return 401;
await passes.settle(readCoinpayWebhook(JSON.parse(raw)));

settle never trusts the body: it finds the order by checkout_session_id (or the older payment id) and asks CoinPay for the session before crediting anything, so replays credit nothing twice. A checkout without an email but with a coin still opens the older one-coin invoice, and orders opened that way keep settling. Options: property (default: site's hostname), sessions: false to keep the one-coin invoice only.

The ElevenLabs key never leaves the server. Browsers get short grants or an event stream.

// server
import { LiveVoice, RoomTranslators, validChoices } from "@profullstack/live-translate";
const rooms = new RoomTranslators({ voice: new LiveVoice({ limits: { serverAudioPerMinute: 6000 } }), translator, billing });

// POST /listen (event stream): this listener's per-participant choices
const room = rooms.open(meetingId, hostAccountId);           // everything is billed to the payer
const leave = await room.listen({ id, choices: validChoices(body.choices, me), send, close });

// POST /feed?participant=&client= (2 s of 16 kHz s16le)
return room.feed(participant, client, pcm);                  // { feeder, wanted }

// browser
import { RoomFeeder, RoomListener } from "@profullstack/live-translate/browser";
const feeder = new RoomFeeder({ path: "/api/…/feed", context: () => audioContext });
feeder.add(participantId, remoteAudioTrack);                 // uploads only while wanted and elected

const listener = new RoomListener({ path: "/api/…/listen", line: ({ participant, line }) => show(participant, line.text),
  speaking: who => duck(who) });
await listener.set([{ participant: "pierre", from: "", to: "en", audio: true, text: true },
                    { participant: "hans",   from: "de", to: "en", audio: false, text: true }]);

Each participant's track is heard once per source-language choice, each phrase translated and voiced once per target language, then fanned out. A listener who already speaks the speaker's language gets the original words as text and keeps the original voice. One client per participant uploads that participant's audio (a lease, re-elected when it goes quiet), so the speaker's own client needs nothing.

SharedTranslations.join(source, language, by, send, close) runs one pipeline per (source, language) and bills every listening account; parse narrows which sources are accepted. See nixamp's /api/v1/speech/shared.

new LiveVoice({ limits }) sets provider-protection budgets (concurrent hearings/voices, Scribe seconds and Flash characters per minute per caller and per server). A listened-to meeting speaker costs about 180 s of Scribe a minute (2 s steps over a 6 s window).

MIT