Streaming

With "stream": true in the body, the gateway delivers the response as server-sent events (SSE). The events have the same format as at the provider. An SDK that streams with the OpenAI or Anthropic API therefore works unchanged.

Endpoints with streaming

On the endpoints without streaming, the gateway answers "stream": true with status code 400 and the code streaming_not_supported_for_endpoint. For POST /v1/audio/speech this also applies to stream_format, for POST /v1/audio/transcriptions to the form field stream. Noirdoc Shield supports non-streaming inspection on /v1/embeddings and /v1/rerank as usual; those endpoints reject only "stream": true.

curl -N https://api.noirdoc.de/v1/chat/completions \
  -H "Authorization: Bearer $NOIRDOC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8-27b",
    "stream": true,
    "messages": [{
      "role": "user",
      "content": "Write a short cover letter."
    }]
  }'

stream = client.chat.completions.create(
    model="qwen3.8-27b",
    messages=[
        {
            "role": "user",
            "content": "Write a short cover letter.",
        }
    ],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        text = chunk.choices[0].delta.content
        print(text, end="", flush=True)

const stream = await client.chat.completions.create({
  model: "qwen3.8-27b",
  messages: [
    {
      role: "user",
      content: "Write a short cover letter.",
    },
  ],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

with client.messages.stream(
    model="<model-id>",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Write a short cover letter.",
        }
    ],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

const stream = client.messages.stream({
  model: "<model-id>",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: "Write a short cover letter.",
    },
  ],
});
stream.on("text", (text) => process.stdout.write(text));
await stream.finalMessage();

Streaming with Noirdoc Shield

Noirdoc Shield inspects only the inbound request before forwarding and never re-identifies responses. As a result, the gateway forwards every SSE event immediately chunk by chunk with zero sentence-buffering latency – including tool calls. Time to first token and inter-chunk latency match an uninspected stream.

If a protection filter configured with block matches, Shield rejects the request before opening the stream with a standard HTTP 403 JSON error response (shield_violation).

Usage data for Chat Completions

For billing, the gateway needs the token counts at the end of the stream. For POST /v1/chat/completions, it therefore sets stream_options.include_usage itself. If you have not set the option, the gateway removes the additional usage event from the stream again. Your application receives the same stream as without the gateway.

Errors and failover

Failover means that the gateway falls back to the next provider for the same model. For streams, this happens as long as no byte of the response has gone to your application yet, that is for network errors, 401, 402, 429 or status codes from 500 upwards before the stream begins. More on this under Providers & routing.

• If the request fails before the response begins, the gateway answers without SSE, with a status code and error object as for requests without streaming, for example 429 resource_exhausted or provider_overloaded with Retry-After. The provider’s message is kept only for provider_rejected_request, see Error codes.
• If the request fails after the gateway has sent status code 200, the error is in the stream. The code is the same one the request would have received as an HTTP error, such as provider_timeout, provider_overloaded, resource_exhausted or provider_error. Because no further header is possible, the wait time is in the error object as retry_after in seconds, where it applies. For /v1/chat/completions this is a line data: {"error": {"type": "...", "code": "provider_timeout", "message": "...", "param": null, "retry_after": 10}} without a chat.completion.chunk envelope and without finish_reason, followed by data: [DONE]; the OpenAI SDKs raise an APIError on it. For /v1/responses it is an error event followed by data: [DONE]. The OpenAI Python SDK raises no exception for this event, because its content is flat like OpenAI’s error event. It yields an event with type: "error" that your loop has to check. For /v1/messages it is an error event followed by message_stop.
• Once the stream has started, the provider stays fixed. The gateway no longer switches. If the stream then breaks off or the provider reports an error, the gateway sends an error with the code stream_interrupted in the same format and closes the stream.

If the gateway is still waiting for the provider during a restart (deployment), it sends the error gateway_restarting with retry_after 5 in the stream. Wait for the time in retry_after and send the request again.

Keep-alive

If the provider does not answer with the response headers within 15 seconds, the gateway sends status code 200 and then an SSE comment line : ping every 15 seconds until the provider answers. This applies to every request with "stream": true on /v1/chat/completions, /v1/responses and /v1/messages. In total, the gateway waits at most 600 seconds across all failover attempts. Requests without streaming keep the limit of 120 seconds.

SSE parsers such as the OpenAI and Anthropic SDKs or eventsource-parser ignore comment lines. If you read the stream yourself line by line, skip lines that begin with :.

A failover is still possible after the first comment lines. A comment line does not count as a byte of the response.

The X-Noirdoc-Provider header names the provider that delivers the stream if it answered within 15 seconds. If the gateway sent status code 200 before that, the header is missing. The provider is then in a comment line : provider=<slug> directly before the first data event.