Streaming

Set stream: true on any of the three formats and the gateway answers with Content-Type: text/event-stream. Frames are separated by a blank line; each carries a data: line with a JSON payload. Keep-alive comments may appear — skip lines that do not start with data: (or event: for Anthropic).

OpenAI Chat Completions

curl -N https://ai.codai.ro/v1/chat/completions \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "stream": true, "messages": [{ "role": "user", "content": "Count to three." }] }'

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"content":", two, three."},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"prompt_tokens_details":{"cached_tokens":0}}}

data: [DONE]

Usage in the final chunk

The gateway always emits a terminal chunk with empty choices and a usage object before [DONE], whether or not you send stream_options: { include_usage: true }. Sending it is harmless and keeps your code portable to upstream OpenAI. usage.prompt_tokens_details.cached_tokens is the number of prompt tokens served from the prompt cache.

Tool calls in a stream

Tool calls arrive piecewise: id and function.name first, then function.arguments appended across many deltas, keyed by index. Concatenate arguments per index until finish_reason: "tool_calls". The TypeScript SDK does this for you and exposes the assembled calls on stream.final.

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_ab12","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Cluj\"}"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}

Anthropic Messages

curl -N https://ai.codai.ro/v1/messages \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "max_tokens": 256, "stream": true, "messages": [{ "role": "user", "content": "Count to three." }] }'

event: message_start
data: {"type":"message_start","message":{"id":"msg_01Xf…","type":"message","role":"assistant","model":"codai","content":[],"stop_reason":null,"usage":{"input_tokens":12,"output_tokens":1,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"One, two, three."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":7}}

event: message_stop
data: {"type":"message_stop"}

Input tokens are final in message_start; message_delta.usage.output_tokens is cumulative — the last value wins. Tool use streams as content_block_start with type: "tool_use" followed by input_json_delta deltas, exactly as upstream.

OpenAI Responses

curl -N https://ai.codai.ro/v1/responses \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "stream": true, "input": "Count to three." }'

event: response.created
data: {"type":"response.created","response":{"id":"resp_9f1e…","object":"response","status":"in_progress"},"sequence_number":0}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_resp_9f1e…","delta":"One","sequence_number":1}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_resp_9f1e…","delta":", two, three.","sequence_number":2}

event: response.output_text.done
data: {"type":"response.output_text.done","item_id":"msg_resp_9f1e…","text":"One, two, three.","sequence_number":3}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_9f1e…","object":"response","status":"completed","output":[{"type":"message","id":"msg_resp_9f1e…","role":"assistant","status":"completed","content":[{"type":"output_text","text":"One, two, three.","annotations":[]}]}],"output_text":"One, two, three.","usage":{"input_tokens":12,"input_tokens_details":{"cached_tokens":0},"output_tokens":7,"total_tokens":19}},"sequence_number":4}

Function calls are flushed at the end as response.output_item.done events with type: "function_call" items, then included in response.completed.output. Usage lives only on response.completed.

Headers on a streamed response

Headers are sent before the first byte, so anything that depends on the final token count cannot be exact yet:

Header	On a stream
`x-codai-routed-to`, `x-codai-provider`	Known — the target was chosen before dispatch.
`x-codai-trace-id`	Known — echoes your `X-Request-Id` or a minted id.
`x-codai-cost-micro-usd`	Present only when token counts were known up front.
`x-codai-cost-estimate: pending`	Sent instead on native streams; the exact figure is in `GET /v1/receipt` and in the hub.
`x-codai-cache-read`, `x-codai-cache-write`	Set when cache figures were known before the first byte.

Client tips

Use curl -N (or your HTTP client's no-buffering mode) — proxies that buffer will hold the whole reply.
Treat data: [DONE] (Chat Completions), message_stop (Messages) or response.completed (Responses) as the end. Do not rely on the socket closing.
If a stream stalls after the first byte, the gateway closes it with a clean stop frame and records the turn as stalled; retry with the same request — prompt caching makes the retry cheap.
For browsers, the /v1/* CORS policy exposes the x-codai-* and x-ratelimit-* headers, so fetch can read them.