Embeddings, audio & realtime

Beyond chat, the gateway serves four OpenAI-compatible media surfaces. They take the same key, the same rate limits and write the same usage records — but none of them starts a billable task, so they never count against a task cap.

EmbeddingsCopy Anchor Link

POST /v1/embeddings. Default model codai-embed resolves to voyage-code-3, chosen for code retrieval. Input is a string or a non-empty array of strings; optional dimensions and user. Only encoding_format: "float" is accepted.

curl https://ai.codai.ro/v1/embeddings \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-embed", "input": ["hello world", "vector me up"] }'

{
  "object": "list",
  "model": "voyage-code-3",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.0123, -0.0456, …] },
    { "object": "embedding", "index": 1, "embedding": [0.0789, 0.0012, …] }
  ],
  "usage": { "prompt_tokens": 7, "total_tokens": 7 }
}

If the requested embedding model is unavailable the gateway may serve a compatible alternative and discloses it in x-codai-embed-fallback: <requested>-><served>. Check the header when vector spaces must stay identical across writes and reads.

Image generationCopy Anchor Link

POST /v1/images/generations, OpenAI-compatible, served by the Google Gemini image models on Vertex AI. Body: prompt (required, ≤ 8 000 characters), model, n (1–4, default 1), size, aspect_ratio, image_size, reference_images, negative_prompt. By default images come back inline as base64 with their mime_type (response_format: "b64_json"). With response_format: "url" each image is stored as an asset and data holds asset objects with a signed URL valid for 1 hour instead (see Getting a URL instead of bytes). Needs a real API key; ephemeral tokens are rejected.

The raw backend ids are accepted as model too. gemini-2.5-flash-image (shut down by Google on 2026-10-02), imagen-4 and imagen-4-fast are retired upstream and are served as codai-image, so old clients keep working. The region that processed the request is in the x-codai-data-region header — codai-image runs on the Vertex eu multi-region, so prompts and images stay in the EU.

• aspect_ratio — 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9, 9:21, 1:4, 4:1, 1:8, 8:1. Overrides size.
• size (OpenAI style) maps to an aspect ratio: 1024x1024 → 1:1, 1536x1024 → 3:2, 1024x1536 → 2:3, 1792x1024 → 16:9, 1024x1792 → 9:16.
• image_size — resolution, default 1K. A size the model does not support is a 400.
• reference_images — [{ "b64_json": "…", "mime_type": "image/png" }] (PNG, JPEG, WebP, HEIC, HEIF; ≤ 7 MB each). They are sent to the model before the prompt, which is how you keep a logo, palette, product or character consistent across generations.

curl https://ai.codai.ro/v1/images/generations \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-image", "prompt": "flat minimalist logo of a paper plane", "n": 1, "aspect_ratio": "1:1", "image_size": "1K" }' \
  | jq -r '.data[0].b64_json' | base64 -d > plane.png

{ "created": 1790000000, "data": [{ "b64_json": "iVBORw0KGgo…", "mime_type": "image/png" }] }

Editing imagesCopy Anchor Link

POST /v1/images/edits changes one or more existing images from an instruction — same models, limits and billing. Send JSON with image (one {b64_json, mime_type} or an array of up to 14), or an OpenAI-style multipart/form-data form with the file in image (or repeated image[]). mask is not supported (400): describe the region to change in the prompt.

curl https://ai.codai.ro/v1/images/edits \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -F model=codai-image \
  -F prompt="replace the background with a soft sunset gradient, keep the logo unchanged" \
  -F image=@plane.png \
  | jq -r '.data[0].b64_json' | base64 -d > plane-sunset.png

SDKs: client.images.generate(...) and client.images.edit(...) (TypeScript and Python, JSON body).

ProvenanceCopy Anchor Link

Images from the EU endpoint carry C2PA content credentials (a caBX chunk in the PNG) and a SynthID watermark. Store and serve the bytes exactly as returned — re-encoding, resizing or stripping metadata removes the C2PA manifest.

BillingCopy Anchor Link

Cost is computed from the token usage the model reports, at the resolved model's Vertex rates: output image tokens dominate (a 1K image on codai-image ≈ $0.074; 2K ≈ $0.11; 4K ≈ $0.17), and each input or reference image adds a fraction of a cent. The charge is in x-codai-cost-micro-usd, counts toward your spend caps and is debited from the prepaid EUR wallet (see Limits & pricing). You pay only for images actually returned: when the safety filter blocks every image the response is 400 with code content_filtered and nothing is charged. Allow up to 120 s per request.

Music & video (codai Studio)Copy Anchor Link

Two Vertex AI generative media lanes. Both need a real API key, work best with English prompts (Veo and Lyria 2 reject other languages, and Lyria refuses prompts that name artists) and are processed in the US — EU data-residency accounts get a 400. Prices below are upstream list prices in USD; the upstream cost is in x-codai-cost-micro-usd and the prepaid EUR wallet is debited at the codai media rate (converted to EUR with the media markup), exactly like images.

MusicCopy Anchor Link

POST /v1/audio/music is synchronous: a codai-music clip takes about 12 s, a codai-music-pro song 30–60 s (up to 4 minutes for long songs) — set a client timeout of at least 300 s. Body:

• prompt (required) — genre, mood, instruments, structure; no artist names.
• model — codai-music (default), codai-music-pro or codai-music-legacy.
• lyrics (≤ 3 000 characters) — words to sing exactly; instrumental: true — no vocals; bpm (40–220); duration_seconds (10–184, codai-music-pro only; codai-music is always about 30 s); language — lyrics language hint such as ro.
• negative_prompt, seed, n (1–4) — codai-music-legacy only; sending them to a Lyria 3 model returns 400.
• response_format — b64_json (default) or url.

Each data item carries mime_type (audio/mpeg for Lyria 3, audio/wav for legacy), duration_seconds, and for Lyria 3 lyrics (as sung, one [start:end] line each), caption (the model's description of the track), lyrics_match and warnings. When the safety filter blocks the prompt the response is 400 content_filtered and nothing is charged.

curl --max-time 300 https://ai.codai.ro/v1/audio/music \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "codai-music-pro",
    "prompt": "upbeat Romanian pop, female vocals, acoustic guitar and light synths",
    "lyrics": "Vara asta e a noastră\nSoarele ne cheamă afară",
    "language": "ro",
    "bpm": 118,
    "duration_seconds": 60
  }' | tee song.json | jq -r '.data[0].b64_json' | base64 -d > song.mp3
jq '.data[0] | {duration_seconds, lyrics_match, warnings}' song.json

{
  "created": 1790000000,
  "model": "codai-music-pro",
  "data": [{
    "b64_json": "SUQzB…",
    "mime_type": "audio/mpeg",
    "duration_seconds": 61.4,
    "lyrics": "[0:6] Vara asta e a noastră\n[6:12] Soarele ne cheamă afară",
    "caption": "Upbeat pop with acoustic guitar… bpm: 118, duration_secs: 61",
    "lyrics_match": 0.82,
    "warnings": []
  }]
}

The previous instrumental model stays available as codai-music-legacy:

curl --max-time 180 https://ai.codai.ro/v1/audio/music \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-music-legacy", "prompt": "calm lo-fi piano with soft vinyl crackle, 80 bpm", "negative_prompt": "vocals", "n": 1 }' \
  | jq -r '.data[0].b64_json' | base64 -d > track.wav

VideoCopy Anchor Link

POST /v1/videos starts a job and answers 202 with { id, object: "video.job", status: "running", model, seconds, estimated_cost_micro_usd }. Body: prompt (required), negative_prompt, seconds (4, 6 or 8, default 8), aspect_ratio (16:9 default, or 9:16), resolution (720p default, or 1080p), audio (default true), image ({ b64_json, mime_type } first frame for image-to-video; forces 8 s), seed.

Poll GET /v1/videos/{id} every few seconds (a job usually takes 1–3 minutes). status is running, succeeded (with data[0].b64_json, video/mp4), failed or filtered (safety filter — nothing is charged). The video is billed exactly once, when the job succeeds — even if your polls land on different gateway instances; further polls of the same job return the video again without charging. The finished MP4 is always stored as an asset, so it also shows up in GET /v1/assets?kind=video. Add ?response_format=url to the poll to get the asset (signed URL) instead of base64 — recommended for 1080p videos, which are large. The job id is signed and bound to your account: other users get 404.

ID=$(curl -s https://ai.codai.ro/v1/videos \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "prompt": "a paper plane gliding over a sunny city, cinematic", "seconds": 4 }' | jq -r .id)

until [ "$(curl -s https://ai.codai.ro/v1/videos/$ID -H "Authorization: Bearer $CODAI_API_KEY" | tee job.json | jq -r .status)" != "running" ]; do sleep 5; done
jq -r '.data[0].b64_json' job.json | base64 -d > plane.mp4

SDKs: client.music.generate({ prompt }), client.videos.create(...) / .retrieve(id), and client.videos.generate({ prompt }), which creates the job and polls until it succeeds (throws on failed, filtered or timeout). Python: client.music.generate({...}), client.videos.generate({...}, poll_interval=5.0, timeout=300.0).

Getting a URL instead of bytesCopy Anchor Link

response_format: "url" works on POST /v1/images/generations, POST /v1/images/edits, POST /v1/audio/music (body field) and GET /v1/videos/{id} (query parameter). The result is stored as an asset for 90 days and each data item is an asset object instead of { b64_json, … }:

{
  "created": 1790000000,
  "model": "codai-music-legacy",
  "data": [{
    "asset_id": "7d0f2c1e-…",
    "kind": "music",
    "status": "ready",
    "model": "codai-music-legacy",
    "url": "https://storage.googleapis.com/…&X-Goog-Signature=…",
    "mime_type": "audio/wav",
    "bytes": 5760044,
    "duration_seconds": 30,
    "width": null,
    "height": null,
    "created_at": "2026-10-10T08:00:00.000Z",
    "expires_at": "2026-10-10T09:00:00.000Z"
  }]
}

The url is a signed URL valid for 1 hour that needs no API key, so it can go straight into an <img>, <audio> or <video> element. When it expires, call GET /v1/assets/{asset_id} for a fresh one. Price and billing are identical to b64_json. A gateway without asset storage answers 503 with code assets_unavailable; the default b64_json keeps working there.

SDKs: pass response_format: 'url' in the body of images.generate / images.edit / music.generate and read result.assets; for video pass { responseFormat: 'url' } (Python response_format="url") to videos.retrieve / videos.generate and read result.asset.

Speech-to-textCopy Anchor Link

POST /v1/audio/transcriptions, multipart form, 25 MB maximum. Fields: file (required), model (default codai-transcribe), and the optional language, prompt, response_format, temperature — forwarded as-is.

curl https://ai.codai.ro/v1/audio/transcriptions \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -F file=@meeting.webm \
  -F model=codai-transcribe \
  -F language=ro

{ "text": "Let's start with the deploy checklist." }

Billing is flat per audio minute, estimated from the upload size (about 1 MB per minute of compressed voice, minimum one minute) and recorded as cost_micro_usd on the usage row.

Text-to-speechCopy Anchor Link

POST /v1/audio/speech. JSON body: input (required, ≤ 4 096 characters), model (default codai-tts), voice, response_format (default mp3; wav, opus, pcm also work), speed, instructions (≤ 1 000 characters, expressive models only), language (BCP-47, default ro-RO, used by the codai-speech models), timestamps and store. By default the response body is the audio bytes.

Romanian voices for codai-speech / codai-speech-expressive — pass the short name as voice (it becomes <language>-Chirp3-HD-<Name>), or a full Google voice name: Achernar, Achird, Algenib, Algieba, Alnilam, Aoede, Autonoe, Callirrhoe, Charon, Despina, Enceladus, Erinome, Fenrir, Gacrux, Iapetus, Kore, Laomedeia, Leda, Orus, Puck, Pulcherrima, Rasalgethi, Sadachbia, Sadaltager, Schedar, Sulafat, Umbriel, Vindemiatrix, Zephyr, Zubenelgenubi. The same voices speak other languages when you set language (for example en-US).

curl https://ai.codai.ro/v1/audio/speech \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-speech", "input": "Bună ziua! Bine ai venit la codai.", "voice": "Charon" }' \
  --output salut.mp3

Word timestamps and stored speechCopy Anchor Link

Set timestamps: true and/or store: true (any speech model) and the response becomes JSON instead of audio bytes:

• words — [{ word, start, end }] in seconds, computed by a transcription pass after synthesis (only with timestamps: true). If that pass fails the request still succeeds with words: null and the header x-codai-timestamps: unavailable.
• b64_json — the audio, when store is not set; or asset — with store: true the audio is saved as a studio asset of kind speech (with the words in its metadata) that you can reference as asset:<asset_id> in a render, for example to burn in captions.
• object: "audio.speech", model, voice, language, mime_type, duration_seconds, cost_micro_usd.

curl https://ai.codai.ro/v1/audio/speech \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai-speech", "input": "Bună ziua!", "language": "ro-RO", "timestamps": true, "store": true }'

{
  "object": "audio.speech",
  "model": "codai-speech",
  "voice": "Aoede",
  "language": "ro-RO",
  "mime_type": "audio/mpeg",
  "duration_seconds": 1.1,
  "asset": { "asset_id": "3f2b8c1e-…", "kind": "speech", "status": "ready", "url": "https://storage.googleapis.com/…", "…": "…" },
  "words": [
    { "word": "Bună", "start": 0.12, "end": 0.48 },
    { "word": "ziua!", "start": 0.52, "end": 1.02 }
  ],
  "cost_micro_usd": "300"
}

Billing: codai-tts* is flat per 1 000 input characters; codai-speech* follow the upstream list prices above. Prices are converted to EUR with the codai media markup and debited from the wallet; timestamps: true adds the transcription at the speech-to-text rate. The upstream cost is in cost_micro_usd / x-codai-cost-micro-usd. Like the other non-EU lanes, the codai-speech models are not available to EU data-residency accounts.

Streaming speech with visemes (Romanian)Copy Anchor Link

POST /v1/audio/speech/stream streams Azure AI Speech neural voices as Server-Sent Events: audio chunks interleaved with viseme and word timing on the same audio clock, so an avatar can lip-sync while the audio plays. Model codai-tts-ro. Body: input (≤ 4 096 characters, XML-escaped into SSML, so markup is read literally), voice (alina → ro-RO-AlinaNeural, default; emil → ro-RO-EmilNeural; or any Azure neural voice name such as en-US-AvaNeural), response_format (pcm = PCM16 mono 24 kHz, default; or mp3), speed (0.5–2.0).

offsetMs is measured from the first audio sample, so schedule each viseme at playbackStart + offsetMs.

curl -N https://ai.codai.ro/v1/audio/speech/stream \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "input": "Bună seara! Mulțumesc pentru trandafir!", "voice": "alina" }'

Azure Speech runs in westeurope, so EU data-residency accounts may use this lane. Billing: $16 per 1M input characters, charged once audio was produced; it never starts a task. When the gateway has no Azure Speech key the route answers 503 with code tts_unavailable.

Ephemeral tokensCopy Anchor Link

Browsers and webviews must never hold your long-lived key. Instead, mint a short-lived scoped token server-side with POST /v1/tokens and hand that to the client. Scopes: realtime, audio, embeddings, and live — one token that reaches both /v1/realtime and /v1/audio/*, for a live co-host that listens and speaks. Chat is deliberately not mintable. TTL 60–3 600 s, default 600.

curl https://ai.codai.ro/v1/tokens \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "scope": "realtime", "ttl_seconds": 600 }'

{ "token": "codai_eph_v1.…", "expires_at": "2026-09-23T12:10:00.000Z", "scope": "realtime" }

The token is used exactly like an API key — Authorization: Bearer codai_eph_v1.… — but only on its scope's surface. It cannot mint further tokens (403 forbidden). Revoking the minting key or downgrading its tier invalidates outstanding tokens within the principal-cache window.

Realtime voiceCopy Anchor Link

WSS /v1/realtime?model=<alias> — bidirectional audio over a single WebSocket, relayed verbatim to the upstream protocol.

If model is omitted, the gateway assumes codai-speech.

Live transcription parametersCopy Anchor Link

codai-transcribe-live takes two optional query parameters:

• language — ro, en, ro-RO, … The hint is written into the transcription session (session.audio.input.transcription.language) unless your own session.update already sets one.
• session_id — your conversation id, recorded on the usage row (the same field x-codai-session-id fills on HTTP routes).

With language=ro and Deepgram configured on the gateway, the session runs on Deepgram Nova-3 (interim results, punctuation, smart formatting). The wire protocol does not change: you still send input_audio_buffer.append (base64 PCM16 mono 24 kHz), input_audio_buffer.commit and input_audio_buffer.clear, and you receive conversation.item.input_audio_transcription.delta and .completed. Deltas are append-only; when Deepgram revises earlier words the deltas pause and .completed.transcript carries the final text — treat it as authoritative. Every commit is answered by exactly one .completed. If Deepgram is not configured or fails before the socket opens, the session falls back to the Azure lane with the audio you already sent. The usage row's upstream_model tells you which lane served (deepgram:nova-3 or gpt-live-transcribe), and ttfb_ms is the time to the first transcript delta.

import { Codai } from 'codai-sdk';

// token minted server-side with scope 'live'
const codai = new Codai({ apiKey: token });
const live = codai.realtime.connect({ model: 'codai-transcribe-live', language: 'ro', sessionId: 'stream-42' });
mic.onFrame((pcm24k: Int16Array) => live.appendAudio(pcm24k));
for await (const ev of live) {
  if (ev.type === 'conversation.item.input_audio_transcription.delta') captions.append(ev.delta);
  if (ev.type === 'conversation.item.input_audio_transcription.completed') captions.commit(ev.transcript);
}

Authenticating the upgradeCopy Anchor Link

• Browser: wss://ai.codai.ro/v1/realtime?model=codai-realtime&key=codai_eph_v1.…
• Server: send Authorization: Bearer <key or token> on the upgrade request — ?key= is not needed.

// token fetched from your own backend, which called POST /v1/tokens
const ws = new WebSocket(
  `wss://ai.codai.ro/v1/realtime?model=codai-realtime&key=${encodeURIComponent(token)}`,
);
ws.onopen = () => {
  ws.send(JSON.stringify({ type: 'session.update', session: { instructions: 'Answer briefly.' } }));
};
ws.onmessage = (ev) => {
  const event = JSON.parse(ev.data);
  if (event.type === 'response.audio.delta') {
    /* append base64 PCM to your player */
  }
};

Close codes you may see: 4401 invalid or disallowed credential, 4503 the requested lane is not configured on this gateway, 1011 internal error. Usage is recorded when the session closes, with its duration.