Documentation

Guides for deploying models, connecting the OpenAI-compatible API, and configuring routing, protection, and retrieval.

Inference API

Stream responses

Receive model output incrementally through the OpenAI Server-Sent Events format.

Enable streaming

Set stream to true. OpenAI-compatible SDKs expose the result as an iterator and handle the final sentinel automatically.

stream.py

1stream = client.chat.completions.create(
2    model="your-serving-name",
3    messages=[{"role":"user","content":"Hello"}],
4    stream=True,
5)

LLM Firewall runs before the stream starts. Supported tool calls are executed or forwarded using the same rules as a non-streaming request.