# Embeddings, audio & realtime (/docs/gateway/embeddings-audio)

Vectors, speech-to-text, text-to-speech, realtime voice over WebSocket and the ephemeral tokens that make browser clients safe.

Beyond chat, the gateway serves four OpenAI-compatible media surfaces. They take the same key, the same rate limits and write the same usage records — but none of them starts a billable task, so they never count against a task cap.

## Embeddings [#embeddings]

`POST /v1/embeddings`. Default model `codai-embed` resolves to **voyage-code-3**, chosen for code retrieval. Input is a string or a non-empty array of strings; optional `dimensions` and `user`. Only `encoding_format: "float"` is accepted.

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl https://ai.codai.ro/v1/embeddings \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-embed", "input": ["hello world", "vector me up"] }'
```

</Tab>
<Tab value="TypeScript">

```ts
const { embeddings } = await codai.embeddings({ input: ['hello world', 'vector me up'] });
console.log(embeddings[0].length); // vector width
```

</Tab>
<Tab value="Python">

```python
vectors = client.embeddings(["hello world", "vector me up"])
print(len(vectors[0]))
```

</Tab>
</Tabs>

```json
{
  "object": "list",
  "model": "voyage-code-3",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.0123, -0.0456, …] },
    { "object": "embedding", "index": 1, "embedding": [0.0789, 0.0012, …] }
  ],
  "usage": { "prompt_tokens": 7, "total_tokens": 7 }
}
```

If the requested embedding model is unavailable the gateway may serve a compatible alternative and discloses it in `x-codai-embed-fallback: <requested>-><served>`. Check the header when vector spaces must stay identical across writes and reads.

## Image generation [#image-generation]

`POST /v1/images/generations`, OpenAI-compatible, served by the **Google Gemini image models** on Vertex AI. Body: `prompt` (required, ≤ 8 000 characters), `model`, `n` (1–4, default 1), `size`, `aspect_ratio`, `image_size`, `reference_images`, `negative_prompt`. By default images come back inline as base64 with their `mime_type` (`response_format: "b64_json"`). With `response_format: "url"` each image is stored as an [asset](./assets.mdx) and `data` holds asset objects with a signed URL valid for 1 hour instead (see [Getting a URL instead of bytes](#getting-a-url-instead-of-bytes)). Needs a real API key; ephemeral tokens are rejected.

| `model` | Backend | Region | `image_size` | Max input images | ≈ Price per 1K image |
| --- | --- | --- | --- | --- | --- |
| `codai-image` (default) | gemini-3.1-flash-image | `eu` | `512`, `1K`, `2K`, `4K` | 14 | $0.074 |
| `codai-image-fast` | gemini-3.1-flash-lite-image | `global` | `1K` | 4 | $0.034 |
| `codai-image-pro` | gemini-3-pro-image | `global` | `1K`, `2K`, `4K` | 14 | $0.134 |

The raw backend ids are accepted as `model` too. `gemini-2.5-flash-image` (shut down by Google on 2026-10-02), `imagen-4` and `imagen-4-fast` are retired upstream and are served as `codai-image`, so old clients keep working. The region that processed the request is in the `x-codai-data-region` header — `codai-image` runs on the Vertex `eu` multi-region, so prompts and images stay in the EU.

- **`aspect_ratio`** — `1:1`, `2:3`, `3:2`, `3:4`, `4:3`, `4:5`, `5:4`, `9:16`, `16:9`, `21:9`, `9:21`, `1:4`, `4:1`, `1:8`, `8:1`. Overrides `size`.
- **`size`** (OpenAI style) maps to an aspect ratio: `1024x1024` → 1:1, `1536x1024` → 3:2, `1024x1536` → 2:3, `1792x1024` → 16:9, `1024x1792` → 9:16.
- **`image_size`** — resolution, default `1K`. A size the model does not support is a 400.
- **`reference_images`** — `[{ "b64_json": "…", "mime_type": "image/png" }]` (PNG, JPEG, WebP, HEIC, HEIF; ≤ 7 MB each). They are sent to the model before the prompt, which is how you keep a logo, palette, product or character consistent across generations.

```bash
curl https://ai.codai.ro/v1/images/generations \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-image", "prompt": "flat minimalist logo of a paper plane", "n": 1, "aspect_ratio": "1:1", "image_size": "1K" }' \
  | jq -r '.data[0].b64_json' | base64 -d > plane.png
```

```json
{ "created": 1790000000, "data": [{ "b64_json": "iVBORw0KGgo…", "mime_type": "image/png" }] }
```

### Editing images [#editing-images]

`POST /v1/images/edits` changes one or more existing images from an instruction — same models, limits and billing. Send JSON with `image` (one `{b64_json, mime_type}` or an array of up to 14), or an OpenAI-style `multipart/form-data` form with the file in `image` (or repeated `image[]`). `mask` is not supported (400): describe the region to change in the prompt.

```bash
curl https://ai.codai.ro/v1/images/edits \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -F model=codai-image \
  -F prompt="replace the background with a soft sunset gradient, keep the logo unchanged" \
  -F image=@plane.png \
  | jq -r '.data[0].b64_json' | base64 -d > plane-sunset.png
```

SDKs: `client.images.generate(...)` and `client.images.edit(...)` (TypeScript and Python, JSON body).

### Provenance [#provenance]

Images from the EU endpoint carry **C2PA content credentials** (a `caBX` chunk in the PNG) and a **SynthID** watermark. Store and serve the bytes exactly as returned — re-encoding, resizing or stripping metadata removes the C2PA manifest.

### Billing [#billing]

Cost is computed from the token usage the model reports, at the resolved model's Vertex rates: output image tokens dominate (a 1K image on `codai-image` ≈ $0.074; 2K ≈ $0.11; 4K ≈ $0.17), and each input or reference image adds a fraction of a cent. The charge is in `x-codai-cost-micro-usd`, counts toward your spend caps and is debited from the prepaid EUR wallet (see [Limits & pricing](./limits-pricing.mdx)). You pay only for images actually returned: when the safety filter blocks every image the response is `400` with code `content_filtered` and nothing is charged. Allow up to 120 s per request.

## Music & video (codai Studio) [#music--video-codai-studio]

Two Vertex AI generative media lanes. Both need a real API key, work best with **English prompts** (Veo and Lyria 2 reject other languages, and Lyria refuses prompts that name artists) and are processed in the **US** — EU data-residency accounts get a `400`. Prices below are upstream list prices in USD; the upstream cost is in `x-codai-cost-micro-usd` and the prepaid EUR wallet is debited at the codai media rate (converted to EUR with the media markup), exactly like images.

| Endpoint | Backend | Output | Price |
| --- | --- | --- | --- |
| `POST /v1/audio/music`, `codai-music` (default) | Lyria 3 Clip | 30 s MP3 with vocals + lyrics (or instrumental) | **$0.04 per call** |
| `POST /v1/audio/music`, `codai-music-pro` | Lyria 3 Pro | full song up to 184 s, MP3, vocals, lyrics, `bpm`, `duration_seconds` | **$0.08 per call** |
| `POST /v1/audio/music`, `codai-music-legacy` | Lyria 2 | 30 s instrumental WAV, 48 kHz | **$0.06 per clip** |
| `POST /v1/videos` + `GET /v1/videos/{id}` | Veo 3.1 Fast | MP4, 4/6/8 s, 720p or 1080p, optional audio | **$0.10 per second** |

### Music [#music]

`POST /v1/audio/music` is synchronous: a `codai-music` clip takes about 12 s, a `codai-music-pro` song 30–60 s (up to 4 minutes for long songs) — set a client timeout of at least 300 s. Body:

- `prompt` (required) — genre, mood, instruments, structure; no artist names.
- `model` — `codai-music` (default), `codai-music-pro` or `codai-music-legacy`.
- `lyrics` (≤ 3 000 characters) — words to sing exactly; `instrumental: true` — no vocals; `bpm` (40–220); `duration_seconds` (10–184, `codai-music-pro` only; `codai-music` is always about 30 s); `language` — lyrics language hint such as `ro`.
- `negative_prompt`, `seed`, `n` (1–4) — **`codai-music-legacy` only**; sending them to a Lyria 3 model returns `400`.
- `response_format` — `b64_json` (default) or `url`.

Each `data` item carries `mime_type` (`audio/mpeg` for Lyria 3, `audio/wav` for legacy), `duration_seconds`, and for Lyria 3 `lyrics` (as sung, one `[start:end]` line each), `caption` (the model's description of the track), `lyrics_match` and `warnings`. When the safety filter blocks the prompt the response is `400` `content_filtered` and nothing is charged.

<Callout type="warn">
**Romanian (and other non-English) lyrics are best effort.** Lyria 3 sings English most reliably. When you send `lyrics`, the gateway transcribes the result and returns `lyrics_match` (0–1, 1 = the sung words match yours); below 0.6 it adds the warning `lyrics_mismatch` — regenerate or simplify the lyrics. If the check cannot run you get `lyrics_match: null` and `lyrics_check_unavailable`. The call is charged either way.
</Callout>

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl --max-time 300 https://ai.codai.ro/v1/audio/music \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codai-music-pro",
    "prompt": "upbeat Romanian pop, female vocals, acoustic guitar and light synths",
    "lyrics": "Vara asta e a noastră\nSoarele ne cheamă afară",
    "language": "ro",
    "bpm": 118,
    "duration_seconds": 60
  }' | tee song.json | jq -r '.data[0].b64_json' | base64 -d > song.mp3
jq '.data[0] | {duration_seconds, lyrics_match, warnings}' song.json
```

</Tab>
<Tab value="TypeScript">

```ts
const r = await codai.music.generate({
  model: 'codai-music-pro',
  prompt: 'upbeat Romanian pop, female vocals, acoustic guitar and light synths',
  lyrics: 'Vara asta e a noastră\nSoarele ne cheamă afară',
  language: 'ro',
  bpm: 118,
  duration_seconds: 60,
});
await writeFile('song.mp3', r.clips[0]!);
console.log(r.tracks[0]?.lyrics_match, r.warnings); // e.g. 0.82, []
```

</Tab>
<Tab value="Python">

```python
r = client.music.generate({
    "model": "codai-music-pro",
    "prompt": "upbeat Romanian pop, female vocals, acoustic guitar and light synths",
    "lyrics": "Vara asta e a noastră\nSoarele ne cheamă afară",
    "language": "ro",
    "bpm": 118,
    "duration_seconds": 60,
})
open("song.mp3", "wb").write(r.clips[0])
print(r.tracks[0]["lyrics_match"], r.warnings)
```

</Tab>
</Tabs>

```json
{
  "created": 1790000000,
  "model": "codai-music-pro",
  "data": [{
    "b64_json": "SUQzB…",
    "mime_type": "audio/mpeg",
    "duration_seconds": 61.4,
    "lyrics": "[0:6] Vara asta e a noastră\n[6:12] Soarele ne cheamă afară",
    "caption": "Upbeat pop with acoustic guitar… bpm: 118, duration_secs: 61",
    "lyrics_match": 0.82,
    "warnings": []
  }]
}
```

The previous instrumental model stays available as `codai-music-legacy`:

```bash
curl --max-time 180 https://ai.codai.ro/v1/audio/music \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-music-legacy", "prompt": "calm lo-fi piano with soft vinyl crackle, 80 bpm", "negative_prompt": "vocals", "n": 1 }' \
  | jq -r '.data[0].b64_json' | base64 -d > track.wav
```

### Video [#video]

`POST /v1/videos` starts a job and answers `202` with `{ id, object: "video.job", status: "running", model, seconds, estimated_cost_micro_usd }`. Body: `prompt` (required), `negative_prompt`, `seconds` (`4`, `6` or `8`, default 8), `aspect_ratio` (`16:9` default, or `9:16`), `resolution` (`720p` default, or `1080p`), `audio` (default `true`), `image` (`{ b64_json, mime_type }` first frame for image-to-video; forces 8 s), `seed`.

Poll `GET /v1/videos/{id}` every few seconds (a job usually takes 1–3 minutes). `status` is `running`, `succeeded` (with `data[0].b64_json`, `video/mp4`), `failed` or `filtered` (safety filter — nothing is charged). The video is **billed exactly once, when the job succeeds** — even if your polls land on different gateway instances; further polls of the same job return the video again without charging. The finished MP4 is always stored as an [asset](./assets.mdx), so it also shows up in `GET /v1/assets?kind=video`. Add `?response_format=url` to the poll to get the asset (signed URL) instead of base64 — recommended for 1080p videos, which are large. The job id is signed and bound to your account: other users get `404`.

```bash
ID=$(curl -s https://ai.codai.ro/v1/videos \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "a paper plane gliding over a sunny city, cinematic", "seconds": 4 }' | jq -r .id)

until [ "$(curl -s https://ai.codai.ro/v1/videos/$ID -H "Authorization: Bearer $CODAI_API_KEY" | tee job.json | jq -r .status)" != "running" ]; do sleep 5; done
jq -r '.data[0].b64_json' job.json | base64 -d > plane.mp4
```

SDKs: `client.music.generate({ prompt })`, `client.videos.create(...)` / `.retrieve(id)`, and `client.videos.generate({ prompt })`, which creates the job and polls until it succeeds (throws on `failed`, `filtered` or timeout). Python: `client.music.generate({...})`, `client.videos.generate({...}, poll_interval=5.0, timeout=300.0)`.

### Getting a URL instead of bytes [#getting-a-url-instead-of-bytes]

`response_format: "url"` works on `POST /v1/images/generations`, `POST /v1/images/edits`, `POST /v1/audio/music` (body field) and `GET /v1/videos/{id}` (query parameter). The result is stored as an [asset](./assets.mdx) for 90 days and each `data` item is an asset object instead of `{ b64_json, … }`:

```json
{
  "created": 1790000000,
  "model": "codai-music-legacy",
  "data": [{
    "asset_id": "7d0f2c1e-…",
    "kind": "music",
    "status": "ready",
    "model": "codai-music-legacy",
    "url": "https://storage.googleapis.com/…&X-Goog-Signature=…",
    "mime_type": "audio/wav",
    "bytes": 5760044,
    "duration_seconds": 30,
    "width": null,
    "height": null,
    "created_at": "2026-10-10T08:00:00.000Z",
    "expires_at": "2026-10-10T09:00:00.000Z"
  }]
}
```

The `url` is a signed URL valid for **1 hour** that needs no API key, so it can go straight into an `<img>`, `<audio>` or `<video>` element. When it expires, call `GET /v1/assets/{asset_id}` for a fresh one. Price and billing are identical to `b64_json`. A gateway without asset storage answers `503` with code `assets_unavailable`; the default `b64_json` keeps working there.

SDKs: pass `response_format: 'url'` in the body of `images.generate` / `images.edit` / `music.generate` and read `result.assets`; for video pass `{ responseFormat: 'url' }` (Python `response_format="url"`) to `videos.retrieve` / `videos.generate` and read `result.asset`.

## Speech-to-text [#speech-to-text]

`POST /v1/audio/transcriptions`, multipart form, **25 MB** maximum. Fields: `file` (required), `model` (default `codai-transcribe`), and the optional `language`, `prompt`, `response_format`, `temperature` — forwarded as-is.

| `model` | Backend |
| --- | --- |
| `codai-transcribe`, `codai-stt`, `whisper-1`, `whisper` | Whisper |
| `gpt-4o-mini-transcribe` | gpt-4o-mini-transcribe |

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl https://ai.codai.ro/v1/audio/transcriptions \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -F file=@meeting.webm \
  -F model=codai-transcribe \
  -F language=ro
```

</Tab>
<Tab value="TypeScript">

```ts
import { readFile } from 'node:fs/promises';

const bytes = await readFile('meeting.webm');
const text = await codai.audio.transcribe({ file: bytes, filename: 'meeting.webm' });
```

</Tab>
<Tab value="Python">

```python
with open("meeting.webm", "rb") as f:
    text = client.transcribe(f.read(), filename="meeting.webm")
```

</Tab>
</Tabs>

```json
{ "text": "Let's start with the deploy checklist." }
```

Billing is flat per audio minute, estimated from the upload size (about 1 MB per minute of compressed voice, minimum one minute) and recorded as `cost_micro_usd` on the usage row.

## Text-to-speech [#text-to-speech]

`POST /v1/audio/speech`. JSON body: `input` (required, ≤ 4 096 characters), `model` (default `codai-tts`), `voice`, `response_format` (default `mp3`; `wav`, `opus`, `pcm` also work), `speed`, `instructions` (≤ 1 000 characters, expressive models only), `language` (BCP-47, default `ro-RO`, used by the codai-speech models), `timestamps` and `store`. By default the response body is the audio bytes.

| `model` | Backend | Default voice | Price (upstream list) | Notes |
| --- | --- | --- | --- | --- |
| `codai-speech` | Google Chirp 3 HD | `Aoede` | $30 / 1M characters | **Native Romanian**, 30 voices, fast (about 1 s for a 16 s line). |
| `codai-speech-expressive` | Gemini 2.5 Flash TTS | `Kore` | $0.50 / 1M input tokens + $10 / 1M audio tokens | Native Romanian, same voices, steered by `instructions` (*"speak warmly and slowly, like a radio host"*). About 6 s for a 16 s line. |
| `codai-tts`, `tts-1`, `tts` | Standard TTS | `alloy` | flat per 1 000 characters | Cheap, neutral. |
| `codai-tts-expressive`, `gpt-4o-mini-tts` | gpt-4o-mini-tts | `marin` | flat per 1 000 characters | Takes `instructions` such as *"calm, warm, short Romanian, slightly amused"*. Voices: alloy, ash, ballad, coral, echo, fable, onyx, nova, sage, shimmer, verse, marin, cedar. |

**Romanian voices** for `codai-speech` / `codai-speech-expressive` — pass the short name as `voice` (it becomes `<language>-Chirp3-HD-<Name>`), or a full Google voice name: Achernar, Achird, Algenib, Algieba, Alnilam, Aoede, Autonoe, Callirrhoe, Charon, Despina, Enceladus, Erinome, Fenrir, Gacrux, Iapetus, Kore, Laomedeia, Leda, Orus, Puck, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zephyr, Zubenelgenubi. The same voices speak other languages when you set `language` (for example `en-US`).

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl https://ai.codai.ro/v1/audio/speech \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-speech", "input": "Bună ziua! Bine ai venit la codai.", "voice": "Charon" }' \
  --output salut.mp3
```

</Tab>
<Tab value="TypeScript">

```ts
import { writeFile } from 'node:fs/promises';

const audio = await codai.audio.speech({
  model: 'codai-speech',
  input: 'Bună ziua! Bine ai venit la codai.',
  voice: 'Charon',
});
await writeFile('salut.mp3', Buffer.from(audio));
```

</Tab>
<Tab value="Python">

```python
audio = client.audio.speech({"model": "codai-speech", "input": "Bună ziua! Bine ai venit la codai.", "voice": "Charon"})
with open("salut.mp3", "wb") as f:
    f.write(audio)
```

</Tab>
</Tabs>

### Word timestamps and stored speech [#word-timestamps-and-stored-speech]

Set `timestamps: true` and/or `store: true` (any speech model) and the response becomes JSON instead of audio bytes:

- `words` — `[{ word, start, end }]` in seconds, computed by a transcription pass after synthesis (only with `timestamps: true`). If that pass fails the request still succeeds with `words: null` and the header `x-codai-timestamps: unavailable`.
- `b64_json` — the audio, when `store` is not set; or `asset` — with `store: true` the audio is saved as a studio [asset](./assets.mdx) of kind `speech` (with the `words` in its metadata) that you can reference as `asset:<asset_id>` in a [render](./renders.mdx), for example to burn in captions.
- `object: "audio.speech"`, `model`, `voice`, `language`, `mime_type`, `duration_seconds`, `cost_micro_usd`.

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl https://ai.codai.ro/v1/audio/speech \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-speech", "input": "Bună ziua!", "language": "ro-RO", "timestamps": true, "store": true }'
```

</Tab>
<Tab value="TypeScript">

```ts
const r = await codai.audio.speechJson({
  model: 'codai-speech',
  input: 'Bună ziua!',
  language: 'ro-RO',
  timestamps: true,
  store: true,
});
console.log(r.words, r.asset?.asset_id); // [{ word: 'Bună', start: 0.12, end: 0.48 }, …]
```

</Tab>
<Tab value="Python">

```python
r = client.audio.speech_json({
    "model": "codai-speech",
    "input": "Bună ziua!",
    "language": "ro-RO",
    "timestamps": True,
    "store": True,
})
print(r.words, r.asset["asset_id"])
```

</Tab>
</Tabs>

```json
{
  "object": "audio.speech",
  "model": "codai-speech",
  "voice": "Aoede",
  "language": "ro-RO",
  "mime_type": "audio/mpeg",
  "duration_seconds": 1.1,
  "asset": { "asset_id": "3f2b8c1e-…", "kind": "speech", "status": "ready", "url": "https://storage.googleapis.com/…", "…": "…" },
  "words": [
    { "word": "Bună", "start": 0.12, "end": 0.48 },
    { "word": "ziua!", "start": 0.52, "end": 1.02 }
  ],
  "cost_micro_usd": "300"
}
```

Billing: `codai-tts*` is flat per 1 000 input characters; `codai-speech*` follow the upstream list prices above. Prices are converted to EUR with the codai media markup and debited from the wallet; `timestamps: true` adds the transcription at the speech-to-text rate. The upstream cost is in `cost_micro_usd` / `x-codai-cost-micro-usd`. Like the other non-EU lanes, the codai-speech models are not available to EU data-residency accounts.

### Streaming speech with visemes (Romanian) [#streaming-speech-with-visemes-romanian]

`POST /v1/audio/speech/stream` streams **Azure AI Speech** neural voices as Server-Sent Events: audio chunks interleaved with **viseme** and **word** timing on the same audio clock, so an avatar can lip-sync while the audio plays. Model `codai-tts-ro`. Body: `input` (≤ 4 096 characters, XML-escaped into SSML, so markup is read literally), `voice` (`alina` → ro-RO-AlinaNeural, default; `emil` → ro-RO-EmilNeural; or any Azure neural voice name such as `en-US-AvaNeural`), `response_format` (`pcm` = PCM16 mono 24 kHz, default; or `mp3`), `speed` (0.5–2.0).

| SSE event | `data` |
| --- | --- |
| `start` | `{ "type": "start", "model", "voice", "format", "content_type", "sample_rate" }` |
| `audio` | `{ "type": "audio", "delta": "<base64 chunk>" }` |
| `viseme` | `{ "type": "viseme", "id": 0-21, "offsetMs": 125 }` — Azure viseme id |
| `word` | `{ "type": "word", "text": "Salut", "offsetMs": 50, "durationMs": 300 }` |
| `done` | `{ "type": "done", "audio_ms", "audioBytes", "visemes", "words" }` |
| `error` | `{ "type": "error", "error": { "code", "message" } }` |

`offsetMs` is measured from the first audio sample, so schedule each viseme at `playbackStart + offsetMs`.

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl -N https://ai.codai.ro/v1/audio/speech/stream \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "input": "Bună seara! Mulțumesc pentru trandafir!", "voice": "alina" }'
```

</Tab>
<Tab value="TypeScript">

```ts
for await (const ev of codai.audio.speechStream({ input: 'Bună seara!', voice: 'alina' })) {
  if (ev.type === 'audio') player.enqueue(ev.audio); // PCM16 mono 24 kHz
  if (ev.type === 'viseme') avatar.scheduleViseme(ev.id, ev.offsetMs);
}
```

</Tab>
<Tab value="Python">

```python
for ev in client.audio.speech_stream({"input": "Bună seara!", "voice": "emil"}):
    if ev["type"] == "audio":
        pcm.write(ev["audio_bytes"])
```

</Tab>
</Tabs>

Azure Speech runs in **westeurope**, so EU data-residency accounts may use this lane. Billing: **$16 per 1M input characters**, charged once audio was produced; it never starts a task. When the gateway has no Azure Speech key the route answers `503` with code `tts_unavailable`.

## Ephemeral tokens [#ephemeral-tokens]

Browsers and webviews must never hold your long-lived key. Instead, mint a **short-lived scoped token** server-side with `POST /v1/tokens` and hand that to the client. Scopes: `realtime`, `audio`, `embeddings`, and `live` — one token that reaches both `/v1/realtime` and `/v1/audio/*`, for a live co-host that listens and speaks. Chat is deliberately not mintable. TTL 60–3 600 s, default 600.

<Tabs items={['curl', 'TypeScript', 'Python']}>
<Tab value="curl">

```bash
curl https://ai.codai.ro/v1/tokens \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "scope": "realtime", "ttl_seconds": 600 }'
```

</Tab>
<Tab value="TypeScript">

```ts
const { token, expiresAt } = await codai.mintToken('realtime', 600);
// send `token` to the browser
```

</Tab>
<Tab value="Python">

```python
minted = client.mint_token("realtime", ttl_seconds=600)
token = minted["token"]
```

</Tab>
</Tabs>

```json
{ "token": "codai_eph_v1.…", "expires_at": "2026-09-23T12:10:00.000Z", "scope": "realtime" }
```

The token is used exactly like an API key — `Authorization: Bearer codai_eph_v1.…` — but only on its scope's surface. It cannot mint further tokens (`403 forbidden`). Revoking the minting key or downgrading its tier invalidates outstanding tokens within the principal-cache window.

## Realtime voice [#realtime-voice]

`WSS /v1/realtime?model=<alias>` — bidirectional audio over a single WebSocket, relayed verbatim to the upstream protocol.

| `model` | Protocol | Backend |
| --- | --- | --- |
| `codai-realtime` (default for OpenAI-style clients) | OpenAI Realtime | gpt-realtime-2.1 |
| `codai-transcribe-live` | OpenAI Realtime, `intent=transcription` | Streaming STT — partials and turn events, no model reply |
| `codai-voice` | Gemini Live `BidiGenerateContent` | gemini-live-2.5-flash-native-audio |

If `model` is omitted, the gateway assumes `codai-speech`.

### Live transcription parameters [#live-transcription-parameters]

`codai-transcribe-live` takes two optional query parameters:

- **`language`** — `ro`, `en`, `ro-RO`, … The hint is written into the transcription session (`session.audio.input.transcription.language`) unless your own `session.update` already sets one.
- **`session_id`** — your conversation id, recorded on the usage row (the same field `x-codai-session-id` fills on HTTP routes).

With **`language=ro`** and Deepgram configured on the gateway, the session runs on **Deepgram Nova-3** (interim results, punctuation, smart formatting). The wire protocol does not change: you still send `input_audio_buffer.append` (base64 PCM16 mono 24 kHz), `input_audio_buffer.commit` and `input_audio_buffer.clear`, and you receive `conversation.item.input_audio_transcription.delta` and `.completed`. Deltas are append-only; when Deepgram revises earlier words the deltas pause and `.completed.transcript` carries the final text — treat it as authoritative. Every `commit` is answered by exactly one `.completed`. If Deepgram is not configured or fails before the socket opens, the session falls back to the Azure lane with the audio you already sent. The usage row's `upstream_model` tells you which lane served (`deepgram:nova-3` or `gpt-live-transcribe`), and `ttfb_ms` is the time to the first transcript delta.

```ts title="live-captions.ts"
import { Codai } from 'codai-sdk';

// token minted server-side with scope 'live'
const codai = new Codai({ apiKey: token });
const live = codai.realtime.connect({ model: 'codai-transcribe-live', language: 'ro', sessionId: 'stream-42' });
mic.onFrame((pcm24k: Int16Array) => live.appendAudio(pcm24k));
for await (const ev of live) {
  if (ev.type === 'conversation.item.input_audio_transcription.delta') captions.append(ev.delta);
  if (ev.type === 'conversation.item.input_audio_transcription.completed') captions.commit(ev.transcript);
}
```

### Authenticating the upgrade [#authenticating-the-upgrade]

<Callout type="warn">
Raw `codai_…` keys are **rejected** in the query string (close code `4401`). They would leak into request logs, proxies and browser history. Browsers must use an ephemeral `realtime` or `live` token.
</Callout>

- **Browser:** `wss://ai.codai.ro/v1/realtime?model=codai-realtime&key=codai_eph_v1.…`
- **Server:** send `Authorization: Bearer <key or token>` on the upgrade request — `?key=` is not needed.

```ts title="browser.ts"
// token fetched from your own backend, which called POST /v1/tokens
const ws = new WebSocket(
  `wss://ai.codai.ro/v1/realtime?model=codai-realtime&key=${encodeURIComponent(token)}`,
);
ws.onopen = () => {
  ws.send(JSON.stringify({ type: 'session.update', session: { instructions: 'Answer briefly.' } }));
};
ws.onmessage = (ev) => {
  const event = JSON.parse(ev.data);
  if (event.type === 'response.audio.delta') {
    /* append base64 PCM to your player */
  }
};
```

Close codes you may see: `4401` invalid or disallowed credential, `4503` the requested lane is not configured on this gateway, `1011` internal error. Usage is recorded when the session closes, with its duration.