Chapter 3
Streaming
Why Wait for the Whole Answer?
Your study chat from Chapter 2 sends a question and then waits. Nothing appears until the whole reply has arrived, and for a long answer from a real model that can take many seconds. Chat apps you have used do something else: the answer appears a few words at a time, while the model is still writing it. That is called streaming, and this chapter builds it.
Your project for this chapter
Open streaming.py in the project editor. It starts with these imports:
from conversation import add_turn, prepare_messages
from errors import StreamBroken
from llm_basics import answer_text, send_body
conversation.py is Chapter 2’s finished code and llm_basics.py is Chapter 1’s, both supplied next to it. You do not need to change either. As before, checking your work uses its own copies of them, so an edit there cannot change a grade, but it can stop the demonstration from working. errors.py holds one exception, StreamBroken, that you use in Lesson 7.
Below the constants are five unfinished functions, one or two per lesson, and a finished function called preview_reply that the demonstration uses in this lesson. python streaming.py runs the file, and the demonstration at the bottom calls each of your functions as soon as it exists.
Watch the difference
In the Shell, run python streaming.py now, before you change anything. The demonstration asks Explain Python lists in detail. twice: once the way Chapter 2 does, and once as a stream. Watch the screen while it runs, not just the result:
Without streaming:
A list is an ordered collection that can hold any mix of values. You create one with square brackets, such as scores = [90, 72, 85]. Each item has a position, starting at 0, so scores[0] is 90 and scores[-1] is the last item. Slicing takes a range: scores[0:2] gives a new list with the first two items. Lists are mutable, which means you can change them after creating them: scores.append(60) adds an item to the end, scores.remove(72) removes the first matching value, and scores[1] = 75 replaces an item in place. Because a list is changed in place, two names that refer to the same list both see the change. Use list.copy() when you need an independent list. len(scores) tells you how many items it holds.
(The whole reply took 1.7 seconds.)
With streaming:
A list is an ordered collection that can hold any mix of values. You create one with square brackets, such as scores = [90, 72, 85]. Each item has a position, starting at 0, so scores[0] is 90 and scores[-1] is the last item. Slicing takes a range: scores[0:2] gives a new list with the first two items. Lists are mutable, which means you can change them after creating them: scores.append(60) adds an item to the end, scores.remove(72) removes the first matching value, and scores[1] = 75 replaces an item in place. Because a list is changed in place, two names that refer to the same list both see the change. Use list.copy() when you need an independent list. len(scores) tells you how many items it holds.
(The whole reply took 1.9 seconds.)
The lines of a stream, as iter_lines() gives them:
Run stopped at the next function to build: open_stream is built in Lesson 3.
On the page the two replies look identical. On your screen they did not. The first time, nothing happened for almost two seconds, and then the whole paragraph appeared at once. The second time, the first words appeared straight away and the rest followed a few words at a time. The last line is the usual stop line: the demonstration stops at the first function you have not built yet.
The scripted model streams this reply as 43 pieces of up to three words, about 40 milliseconds apart. When you do not ask for a stream, it waits just as long and then sends everything, so that you can compare the two. SCRIPTED_MODEL.md lists both rules. The times in brackets can differ by a few tenths of a second from run to run.
What streaming changes, and what it does not
A model writes its reply one token at a time, as Chapter 1 described. Without streaming, the service collects every token and sends the reply when the last one is written. With streaming, it sends the tokens on as they are written, in small groups.
That changes one thing: how soon the reader sees something. With streaming, the first words appear as soon as the model writes them. A reader can start reading, and can see that the program is working rather than stuck.
It does not change the rest:
The total time stays the same. The model writes the reply at the same speed either way. Both runs above took about two seconds from start to finish.
The cost stays the same. You pay for the tokens sent and the tokens written, as in Chapter 1, and a streamed reply has exactly the same tokens.
The answer stays the same. Streaming changes how the reply travels, not what the model writes.
For a short reply the difference hardly matters. For a long answer from a live model, which can take ten seconds or more, it is the difference between a program that seems to respond and one that seems to hang.
Streaming is not always what you want. When your program needs the whole reply before it can do anything with it, such as reading JSON out of it in Chapter 4, showing it piece by piece gains nothing. Chapter 4 goes back to waiting for whole replies for exactly that reason.
The plan for this chapter
The next lesson looks at what a stream actually contains. Lessons 3 to 7 then build a stream reader one step at a time: open a stream, decide which of its lines matter, take the text out of each piece, print the pieces as they arrive while keeping the whole reply, and refuse to treat a stream that breaks halfway as a complete answer. The project makes the Chapter 2 study chat stream its replies.
A program asks a live model for a long explanation. What does streaming the reply improve?
A reply costs $0.0004 when you wait for it whole. What does the same reply cost when you stream it?
Which program gains least from streaming the reply?