Lyrics and live jam

Three tools sit next to audio creatives: a lyrics writer that checks meter and rhyme, lyrics alignment that times the lines to a vocal clip, and a live jam that streams generated music you steer in real time.

Writing lyricsCopy Anchor Link

curl https://ai.codai.ro/v1/studio/lyrics \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: song-7" \
  -d '{
    "brief": "a summer night by the sea, two friends who meet again",
    "language": "ro",
    "genre": "pop",
    "structure": ["verse", "chorus", "verse", "chorus"],
    "syllables_per_line": 8,
    "rhyme_scheme": "AABB",
    "lines_per_section": 4
  }'

The writer runs one bounded codai turn with a forced tool, then checks every line with codai's own counters: syllables (Romanian rules handle diphthongs, triphthongs and hiatus; English counts vowel groups after dropping a silent -e / -es / -ed) and rhyme endings (the last word from its stressed vowel in Romanian, from its last vowel group in English, diacritics folded). When more than 30 % of the lines miss — a syllable count outside the target ±1 or a line of an unrhymed pair — one repair turn names the problems and the better of the two answers is returned (repaired: true when the repair won).

{
  "object": "studio.lyrics",
  "title": "Noapte de vară",
  "language": "ro",
  "rhyme_scheme": "AABB",
  "syllables_per_line": 8,
  "sections": [
    {
      "kind": "verse",
      "lines": [
        { "text": "Marea cântă lin sub lună", "syllables": 8, "rhyme": "una", "letter": "A", "off": false },
        { "text": "Ne-am regăsit iar împreună", "syllables": 8, "rhyme": "una", "letter": "A", "off": false }
      ]
    }
  ],
  "text": "[verse]\nMarea cântă lin sub lună\nNe-am regăsit iar împreună",
  "report": { "lines": 16, "off_lines": 1, "off_ratio": 0.063, "pairs": 8, "rhymed": 7, "repair_ratio": 0.125 },
  "repaired": false,
  "cost_micro_usd": "4210"
}

text is the whole song with one [kind] header per section — pass it as lyrics_text to the alignment call or as the lyrics param of music.generate_ace.

Billing. The turns are token-metered under the public model codai (usage model codai::lyrics, prompt caching on). A small hold of 20 000 µUSD is placed on your wallet first (402 insufficient_funds when it does not fit) and settled with the real token cost. When the model cannot produce lyrics at all the answer is 422 no_lyrics, and only the turns that ran are billed.

Aligning lyrics to a clip (karaoke)Copy Anchor Link

Alignment times written lyrics against one clip of an audio (or video) creative and writes them to doc.audio.lyrics with a single setLyrics edit:

curl https://ai.codai.ro/v1/studio/lyrics/align \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "creative_id": "8a1c…",
    "clip_id": "vox1",
    "lyrics_text": "[verse]\nMarea cântă lin sub lună\nNe-am regăsit iar împreună",
    "language": "ro"
  }'

The gateway takes the word timestamps of the clip's asset — from the cache that captions and earlier alignments leave on the asset (free, cached: true), or from a fresh Whisper transcription of the clip (at most 25 MB) — maps them onto the timeline through the clip's trim, position and stretch, and matches lyric words to transcript words with a diacritics- and typo-tolerant alignment. Lines whose words were not heard are interpolated between their neighbours. The answer:

{
  "object": "studio.lyrics_alignment",
  "creative_id": "8a1c…",
  "clip_id": "vox1",
  "applied": true,
  "seq": 12,
  "doc_hash": "…",
  "lines": [{ "startMs": 12040, "endMs": 14880, "text": "Marea cântă lin sub lună", "sectionId": "verse1" }],
  "lrc": "[ti:Noapte de vară]\n[00:12.04]Marea cântă lin sub lună\n…",
  "match_ratio": 0.94,
  "cached": false,
  "cost_micro_usd": "18000"
}

match_ratio is the share of lyric words matched to a heard word; below about 0.6 the timing is mostly interpolated — check the clip and the language. The LRC is also exported by the audio render (lrc: true).

Billing. Cached words are free. A fresh transcription bills codai::lyrics-align at the speech-to-text rate — 6 000 µUSD per started minute of the clip — under a wallet hold. Alignment needs a transcription service outside the EU and is not available for EU data-residency accounts (403 residency_unavailable); when transcription is not configured the answer is 503 transcription_unavailable.

Live jam (WebSocket)Copy Anchor Link

wss://ai.codai.ro/v1/studio/jam opens a live music session on Lyria RealTime: you send weighted text prompts and a few musical controls, the gateway streams back continuous generated audio. OpenAPI cannot describe WebSockets, so the protocol is documented here.

ConnectingCopy Anchor Link

Mint ephemeral tokens with POST /v1/tokens (scope realtime or live). ?session_id= is recorded on the usage row.

Client → gateway (JSON text frames)Copy Anchor Link

Messages sent before the upstream is ready are queued. Anything else — invalid JSON, an unknown type, a binary frame — closes the session with 4400.

Gateway → clientCopy Anchor Link

• Binary frames: raw PCM16 little-endian, interleaved stereo, 48 000 Hz. Audio is dropped (not buffered) while your socket is more than 4 MB behind.
• { "type": "status", "state": "connecting" | "ready" | "playing" | "paused" | "stopped", "remaining_ms", "sample_rate"?, "channels"? } — ready carries sample_rate: 48000 and channels: 2.
• { "type": "filtered", "text", "reason" } — the model's safety filter ignored a prompt.
• { "type": "error", "code", "message" } — sent before a refusal or an upstream close.

Close codesCopy Anchor Link

PricingCopy Anchor Link

$0.03 per started minute of the open upstream session (usage model codai::jam, upstream lyria-realtime-exp). The 10-minute price ($0.30) is held when the session opens and settled with the minutes actually used when it closes; a session that never reaches the upstream bills nothing. Lyria RealTime is an experimental model, so this price may change. Prompts and audio are never logged.

SDKsCopy Anchor Link

const song = await codai.lyrics.write({ brief: 'o noapte de vară', language: 'ro', rhyme_scheme: 'AABB' });
const aligned = await codai.lyrics.align({ creative_id: id, clip_id: 'vox1', lyrics_text: song.text });

const jam = codai.jam.connect({ token }); // browser: ephemeral token
jam.on('audio', (ev) => player.push(new Int16Array(ev.data)));
jam.on('close', (ev) => console.log('closed', ev.code));
jam.prompts([{ text: 'warm lo-fi piano', weight: 1 }]);
jam.config({ bpm: 90, density: 0.4 });
jam.play();

song = client.lyrics.write("o noapte de vară", "ro", rhyme_scheme="AABB", syllables_per_line=8)
aligned = client.lyrics.align(creative_id, "vox1", song["text"])

TypeScript: lyrics.write / align, jam.connect (JamSession: prompts, config, play, pause, stop, reset, close, on; JAM_CLOSE_CODES). Python: lyrics.write / align; the Python SDK has no WebSocket helper — use any WebSocket client with the protocol above.