0%

Using the OpenAI Client · practice

Stream Through the Client

Chapter 3 took five functions to stream a reply: open the stream, pick out the data: lines, decode each chunk, take the text out, and print it while keeping the whole reply. The client does the first four. The rule you care most about, never presenting part of a reply as the whole, is still yours to keep, and this lesson keeps it in two steps.

Step 1: the happy path

Add stream=True to create and the call returns a stream instead of a reply. Add stream_options={"include_usage": True} as well, as Chapter 3 did, so the stream also reports the token counts:

stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    ...

The loop gets one chunk object per data: line. The client has already read the lines, skipped the : keep-alive comment, decoded the JSON and stopped at [DONE], so none of that appears in your code. For Say hello in a short sentence., the chunks hold the same values Chapter 3 decoded by hand, now as attributes:

Chunkchunk.choicesdelta.contentfinish_reason
firstone choice""None
one per pieceone choice"Hello there, it ", then "is nice to ", then "meet you."None
finishone choiceNone"stop"
usage[]no choice to readno choice to read

chunk.choices[0].delta.content is the piece of text. It is "" in the first chunk and None in the finish chunk, and a live model can send None in between, so print and keep a piece only when it is a non-empty string; if content: tests exactly that. The last chunk carries the token counts in chunk.usage and has an empty choices list, so skip a chunk whose choices is empty before you read choices[0]. Chapter 9 reads that usage chunk to keep a budget.

Build the first version of stream_chat(messages, *, client):

  1. Call create with model=MODEL, messages=messages, stream=True and stream_options={"include_usage": True}, and nothing else.

  2. Loop over the stream. Skip a chunk when chunk.choices is empty.

  3. Read content = chunk.choices[0].delta.content. If it is a non-empty string, print it with end="" and flush=True and append it to a list.

  4. After the loop, print one newline and return the pieces joined with "".join(pieces).

That is Chapter 3’s stream_reply without the line handling. Run python client_calls.py. The long reply streams in as before, and then the demonstration sends the same request through a client that asks the scripted model to cut the stream:

A streamed reply:
A list is an ordered collection that can hold any mix of values. You create one with square brackets, such as scores = [90, 72, 85]. Each item has a position, starting at 0, so scores[0] is 90 and scores[-1] is the last item. Slicing takes a range: scores[0:2] gives a new list with the first two items. Lists are mutable, which means you can change them after creating them: scores.append(60) adds an item to the end, scores.remove(72) removes the first matching value, and scores[1] = 75 replaces an item in place. Because a list is changed in place, two names that refer to the same list both see the change. Use list.copy() when you need an independent list. len(scores) tells you how many items it holds.
stream_chat returned 709 characters.
A stream that breaks halfway:
A list is an ordered collection that can hold 
openai raised APIConnectionError: Connection error.
Its cause, by name: RemoteProtocolError
Run stopped: stream_chat does not turn a broken stream into StreamBroken yet.

Step 2: the break

The broken stream printed three pieces, and then the loop raised openai.APIConnectionError. That is the client’s error for a connection that failed, and here it means the stream was cut. It plays the part requests.exceptions.ChunkedEncodingError played in Chapter 3, and the partial text is exactly the same.

The demonstration also printed the error’s cause by name, RemoteProtocolError. That error belongs to the library the client uses underneath to send requests, and the client wraps it in its own error for you. Read the name for interest, and never catch it: the library underneath is the client’s business and can change between versions, while openai.APIConnectionError is the part the client promises.

Change stream_chat so it keeps Chapter 3’s rule:

  1. A cut connection. Put the for loop inside a try block, with except openai.APIConnectionError: that prints one newline and raises StreamBroken("".join(pieces)).

  2. A stream that never finished. Set finished = False before the loop. For every chunk with a choice, if chunk.choices[0].finish_reason is not None, set finished = True. After the loop, print the newline as before, and if finished is still False, raise StreamBroken("".join(pieces)).

The second rule is Chapter 3’s “ended without [DONE]” in the client’s terms. The client stops at [DONE] without showing it to you, so the finish chunk is the last signal you can see that the model finished. Any finish_reason counts, including "length": the stream finished, and it is a different problem, which Chapter 9 handles, when the model ran out of tokens. The scripted model cannot send a stream that ends cleanly with no finish chunk, so the demonstration never shows this case; only the checks do.

Catch only openai.APIConnectionError. A typo in your own code inside the loop is not a broken connection, and must reach you as what it is.

A stream that goes silent for longer than the client’s timeout raises openai.APITimeoutError, which is a kind of APIConnectionError, so stream_chat reports it as StreamBroken with the text so far. In Chapter 3 that case escaped without the partial text; the client makes it one case.

Keep the create call itself outside the try block. If create raises, for example openai.InternalServerError because the service returned 500, nothing has been printed yet, so there is no partial reply to protect. That error reaches the caller unchanged, and Chapter 9 retries exactly those failures.

Who sends the scenario header

Chapter 3 passed the header X-Scripted-Scenario: broken-stream to open_stream yourself. Here the demonstration uses the supplied scenario_client("broken-stream"):

def scenario_client(scenario):
    return make_client().with_options(default_headers={"X-Scripted-Scenario": scenario})

with_options returns a copy of the client with some settings changed. default_headers are headers the copy sends on every call it makes, so every request through that client asks the scripted model for the scenario, and stream_chat needs no header parameter. Lesson 5 shows the other way: extra_headers= on one call adds a header to that call only.

Run python client_calls.py again. Now the broken stream is reported the way Chapter 3 reported it:

A stream that breaks halfway:
A list is an ordered collection that can hold 
StreamBroken. The partial text was 'A list is an ordered collection that can hold '
A quiz through the parse helper:
Run stopped at the next function to build: quiz_messages is built in Lesson 4.

Your loop reads chunk.choices[0].delta.content on every chunk and never checks chunk.choices. Streams that end normally fail at the very end with IndexError. Why?

The create call in stream_chat raises openai.InternalServerError before any chunk arrives. What should stream_chat do?

A broken stream raised openai.APIConnectionError, whose cause is printed as RemoteProtocolError. Which error should stream_chat catch?

Task

In client_calls.py, replace the unfinished body of stream_chat(messages, *, client).

  • Call client.chat.completions.create(model=MODEL, messages=messages, stream=True, stream_options={"include_usage": True}), outside any try block.

  • Loop over the chunks. Skip a chunk whose choices is empty. Print each non-empty chunk.choices[0].delta.content with end="" and flush=True, and keep it.

  • Remember whether any chunk’s finish_reason was not None.

  • If openai.APIConnectionError is raised while you loop, print one newline and raise StreamBroken("".join(pieces)).

  • After the loop, print one newline. If no chunk carried a finish_reason, raise StreamBroken("".join(pieces)); otherwise return the joined text.

Catch no other exception. Run python client_calls.py after each step to see the stream and the broken stream, then check your work.