# Streaming (/docs/gateway/streaming)

Server-sent events for each wire format, and where token usage lands.

Set `stream: true` on any of the three formats and the gateway answers with `Content-Type: text/event-stream`. Frames are separated by a blank line; each carries a `data:` line with a JSON payload. Keep-alive comments may appear — skip lines that do not start with `data:` (or `event:` for Anthropic).

## OpenAI Chat Completions [#openai-chat-completions]

```bash
curl -N https://ai.codai.ro/v1/chat/completions \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "stream": true, "messages": [{ "role": "user", "content": "Count to three." }] }'
```

```text
data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{"content":", two, three."},"finish_reason":null}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl_7d3c…","object":"chat.completion.chunk","created":1790000000,"model":"codai","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":7,"total_tokens":19,"prompt_tokens_details":{"cached_tokens":0}}}

data: [DONE]
```

### Usage in the final chunk [#usage-in-the-final-chunk]

The gateway **always** emits a terminal chunk with empty `choices` and a `usage` object before `[DONE]`, whether or not you send `stream_options: { include_usage: true }`. Sending it is harmless and keeps your code portable to upstream OpenAI. `usage.prompt_tokens_details.cached_tokens` is the number of prompt tokens served from the prompt cache.

### Tool calls in a stream [#tool-calls-in-a-stream]

Tool calls arrive piecewise: `id` and `function.name` first, then `function.arguments` appended across many deltas, keyed by `index`. Concatenate `arguments` per index until `finish_reason: "tool_calls"`. The [TypeScript SDK](../sdks/typescript.mdx) does this for you and exposes the assembled calls on `stream.final`.

```text
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_ab12","type":"function","function":{"name":"get_weather","arguments":""}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"city\":"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"Cluj\"}"}}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}
```

## Anthropic Messages [#anthropic-messages]

```bash
curl -N https://ai.codai.ro/v1/messages \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "max_tokens": 256, "stream": true, "messages": [{ "role": "user", "content": "Count to three." }] }'
```

```text
event: message_start
data: {"type":"message_start","message":{"id":"msg_01Xf…","type":"message","role":"assistant","model":"codai","content":[],"stop_reason":null,"usage":{"input_tokens":12,"output_tokens":1,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"One, two, three."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":7}}

event: message_stop
data: {"type":"message_stop"}
```

Input tokens are final in `message_start`; `message_delta.usage.output_tokens` is cumulative — the last value wins. Tool use streams as `content_block_start` with `type: "tool_use"` followed by `input_json_delta` deltas, exactly as upstream.

## OpenAI Responses [#openai-responses]

```bash
curl -N https://ai.codai.ro/v1/responses \
  -H "Authorization: Bearer $CODAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "model": "codai", "stream": true, "input": "Count to three." }'
```

```text
event: response.created
data: {"type":"response.created","response":{"id":"resp_9f1e…","object":"response","status":"in_progress"},"sequence_number":0}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_resp_9f1e…","delta":"One","sequence_number":1}

event: response.output_text.delta
data: {"type":"response.output_text.delta","item_id":"msg_resp_9f1e…","delta":", two, three.","sequence_number":2}

event: response.output_text.done
data: {"type":"response.output_text.done","item_id":"msg_resp_9f1e…","text":"One, two, three.","sequence_number":3}

event: response.completed
data: {"type":"response.completed","response":{"id":"resp_9f1e…","object":"response","status":"completed","output":[{"type":"message","id":"msg_resp_9f1e…","role":"assistant","status":"completed","content":[{"type":"output_text","text":"One, two, three.","annotations":[]}]}],"output_text":"One, two, three.","usage":{"input_tokens":12,"input_tokens_details":{"cached_tokens":0},"output_tokens":7,"total_tokens":19}},"sequence_number":4}
```

Function calls are flushed at the end as `response.output_item.done` events with `type: "function_call"` items, then included in `response.completed.output`. Usage lives only on `response.completed`.

## Headers on a streamed response [#headers-on-a-streamed-response]

Headers are sent before the first byte, so anything that depends on the final token count cannot be exact yet:

| Header | On a stream |
| --- | --- |
| `x-codai-routed-to`, `x-codai-provider` | Known — the target was chosen before dispatch. |
| `x-codai-trace-id` | Known — echoes your `X-Request-Id` or a minted id. |
| `x-codai-ai-generated`, `x-codai-generator` | Always sent — `true` and `codai/<upstream model>` (AI-generated marker, EU AI Act art. 50(2)). |
| `x-codai-cost-micro-usd` | Present only when token counts were known up front. |
| `x-codai-cost-estimate: pending` | Sent instead on native streams; the exact figure is in `GET /v1/receipt` and in the hub. |
| `x-codai-cache-read`, `x-codai-cache-write` | Set when cache figures were known before the first byte. |

## Client tips [#client-tips]

- Use `curl -N` (or your HTTP client's no-buffering mode) — proxies that buffer will hold the whole reply.
- Treat `data: [DONE]` (Chat Completions), `message_stop` (Messages) or `response.completed` (Responses) as the end. Do not rely on the socket closing.
- If a stream stalls after the first byte, the gateway closes it with a clean stop frame and records the turn as stalled; retry with the same request — prompt caching makes the retry cheap.
- For browsers, the `/v1/*` CORS policy exposes the `x-codai-*` and `x-ratelimit-*` headers, so `fetch` can read them.