0%

Streaming · capstone

Capstone project: Project: Stream Answers in the Study Chat

In Chapter 2 you built chat_turn, a study chat function that keeps the study system message first, trims the history, and waits for the whole reply. This project builds the same function again in streaming.py, with one difference: the reply streams to the screen as the model writes it.

history = []
reply, history = chat_turn(history, "Explain Python lists in detail.")

The reply appears piece by piece while chat_turn runs. When it returns, reply holds the whole text and history holds the conversation so far, exactly as in Chapter 2.

What stays and what changes

Look at Chapter 2’s chat_turn in conversation.py. It does four things: prepares the messages with prepare_messages, sends them, reads the reply, and returns the reply with a new history built by add_turn. Only the middle two change:

StepChapter 2This project
Prepareprepare_messages(history, question)the same
Sendsend_body(...)open_stream(...) from Lesson 3
Readanswer_text(response)stream_reply(response) from Lessons 6 and 7
Return(reply, add_turn(messages[:-1], question, reply))the same

prepare_messages and add_turn are imported at the top of streaming.py from Chapter 2’s finished code, so you use them as they are.

Build chat_turn

chat_turn(history, question, *, session=None) in streaming.py:

  1. messages = prepare_messages(history, question).

  2. Open a stream for {"model": MODEL, "messages": messages} with open_stream, passing session=session along.

  3. reply = stream_reply(response). This prints the reply as it arrives.

  4. Return (reply, add_turn(messages[:-1], question, reply)).

As in Chapter 2, messages[:-1] drops the pending question, and add_turn puts it back together with the reply as one whole turn. The caller’s list is never changed.

When the stream breaks

chat_turn catches nothing. If the stream breaks, stream_reply raises StreamBroken, and the error goes straight through chat_turn to its caller. That is the point: chat_turn never reaches step 4, so no history is returned, and the half reply is never added to one. The caller’s own history is exactly as it was, so it can try the same question again with nothing to undo. The partial text is still available to the caller as error.partial_text, if it wants to show what arrived.

Catching StreamBroken inside chat_turn and adding error.partial_text to the history would bring back exactly the problem Lesson 7 solved: half an answer stored as if the model had said it.

Run the chat

Run python streaming.py. The demonstration starts from an empty history and asks two questions. Your chat_turn prints each reply as it streams in:

The streaming study chat:
You: What does len() return?
len() returns the number of items in a sequence or collection, such as the characters in a string or the elements in a list.
You: Explain Python lists in detail.
A list is an ordered collection that can hold any mix of values. You create one with square brackets, such as scores = [90, 72, 85]. Each item has a position, starting at 0, so scores[0] is 90 and scores[-1] is the last item. Slicing takes a range: scores[0:2] gives a new list with the first two items. Lists are mutable, which means you can change them after creating them: scores.append(60) adds an item to the end, scores.remove(72) removes the first matching value, and scores[1] = 75 replaces an item in place. Because a list is changed in place, two names that refer to the same list both see the change. Use list.copy() when you need an independent list. len(scores) tells you how many items it holds.
The history now holds 5 messages.

The history holds the study system message and two whole turns. The demonstration printed only the You: lines and the count: the replies came from stream_reply inside your chat_turn, as they arrived.

The demonstration shows the happy path. Before you check your work, think through the cases it does not show: a history that has grown past MAX_MESSAGES, a history that starts with a different system message, and a stream that breaks halfway. prepare_messages handles the first two as it did in Chapter 2, and Lesson 7’s stream_reply handles the third, so chat_turn needs nothing extra for any of them. If you get stuck, ask Monty for a hint about one case without asking for the whole function.

Where this goes next

Your study chat now streams. Chapter 4 goes back to waiting for whole replies on purpose, because it asks the model for JSON that your program reads into Python objects, and half an object is no use to anyone. Chapter 5 then moves both kinds of request onto the openai client, which does what open_stream and stream_reply do for you, so you will know what it is doing underneath.

During chat_turn(history, "Explain decorators."), the stream breaks halfway. What does the caller get?

Who prints the reply while it streams, and how many times does it appear on screen?

Task

In streaming.py, replace the unfinished body of chat_turn(history, question, *, session=None).

  • messages = prepare_messages(history, question).

  • Open a stream for {"model": MODEL, "messages": messages} with open_stream(..., session=session).

  • reply = stream_reply(response), which prints the reply as it arrives.

  • Return (reply, add_turn(messages[:-1], question, reply)).

chat_turn prints nothing itself, catches no exceptions and never changes the caller’s list. When the stream breaks, the StreamBroken from stream_reply reaches the caller and no history is returned. Run python streaming.py to see the study chat stream, then check your work.