Streaming LLM responses in Python with the OpenAI SDK

The short version

A full response can take several seconds, and a screen that shows nothing for that long looks frozen. Streaming sends tokens as the model produces them, so the first words appear almost at once (a short time to first token) even when the full answer takes as long as before. FastMetal is OpenAI-compatible, so you turn it on with stream=True and read the deltas.

How streaming works

A normal request returns once the server has the whole response. A streaming request returns a sequence of chunks over Server-Sent Events (SSE): one data: line per chunk, then data: [DONE] at the end. The OpenAI SDK parses the SSE for you and hands you each chunk in order.

A minimal Python loop

With stream=True, create returns an iterator instead of a response object. Loop over it and read choices[0].delta.content from each chunk.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.fastmetal.ai/v1",
    api_key="sk-...",  # your FastMetal API key
)

stream = client.chat.completions.create(
    model="anthropic-claude-haiku-4-5",
    messages=[{"role": "user", "content": "Write a short story"}],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Three details matter:

• Each chunk carries the new text in choices[0].delta.content.
• The first chunk (role only) and the last one have content set to None, so guard with if delta:.
• flush=True writes each piece to the terminal at once instead of waiting for the buffer to fill.

Cancellation and dropped connections

The sample works on a good connection. In production, streams get cut off by network drops and timeouts, and users press stop. Handle both.

try:
    stream = client.chat.completions.create(
        model="anthropic-claude-haiku-4-5",
        messages=[{"role": "user", "content": "Write a short story"}],
        stream=True,
    )
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
except KeyboardInterrupt:
    stream.close()  # the user stopped: close the connection
except Exception as e:
    print(f"\n[stream interrupted] {e}")

• A dropped connection raises an exception partway through the loop. Keep the text you already received, and log the error so it is visible.
• To stop generation on purpose, call stream.close(). That closes the connection and the model stops sending tokens.
• Billing works the same as for a normal request: per token, input and output. Streaming changes when you see the text, and it costs the same per token.

Relay the stream to a browser chat UI

For a browser chat, call FastMetal from your server and pass the tokens on to the page as SSE. The API key stays on the server. With FastAPI, use StreamingResponse:

import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import OpenAI

app = FastAPI()
client = OpenAI(base_url="https://api.fastmetal.ai/v1", api_key="sk-...")

@app.get("/chat")
def chat(q: str):
    def event_stream():
        stream = client.chat.completions.create(
            model="anthropic-claude-haiku-4-5",
            messages=[{"role": "user", "content": q}],
            stream=True,
        )
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                yield f"data: {json.dumps({'t': delta})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(event_stream(), media_type="text/event-stream")

Each delta is wrapped in JSON (data: {"t": "..."}). A raw delta that contains a newline would break the SSE framing, and JSON encoding escapes it. In the browser, open an EventSource on /chat?q=..., parse each onmessage payload and append t to the page, and close the source when [DONE] arrives.

FAQ

Does streaming cost more or less than a normal request?

Neither. Both are billed per input and output token at the model's rate, which is on the model's page and on pricing. Streaming changes only how the response is delivered.

Which models can stream?

The OpenAI-compatible endpoint accepts stream=True for chat completions. Find the model ids in the model catalog.

Can I stream with the Node SDK too?

Yes. Pass stream: true and iterate with for await (const chunk of stream). The chunk shape is the same. The OpenAI SDK setup guide shows the Node client configuration.

Next steps

Get stream=True working in a local script first, then move on to the server relay. The model catalog lists the ids, and the docs cover the endpoint and its parameters.