Streaming · practice
Open a Stream and Read Its Bytes
The stream in Lesson 2 was text on a page. Now you open it from your own code and print its lines as they arrive. That takes one function, open_stream, and one new idea about what those lines are.
Two settings, both called stream
A streamed request needs two changes from the requests you sent in Chapter 1, and both use the word “stream”:
In the body,
"stream": Trueasks the service to send the reply as server-sent events instead of one JSON document. Next to it,"stream_options": {"include_usage": True}asks for the usage chunk at the end, as Lesson 2 explained.In the call,
stream=Trueis an argument torequests.postitself, next tojson=andtimeout=. It changes whenpostreturns. Without it,requestsreads the entire response before giving it to you, so you would wait for the last piece even though the service sent the first one long before. Withstream=True,postreturns as soon as the status line and the headers arrive, and the body is read later, piece by piece, as your code asks for it.
You need both. The body setting makes the service send pieces; the call setting lets your program read them as they come.
Build open_stream
open_stream(body, *, headers=None, session=None) takes a chat body like the ones send_body sends and turns it into a streamed request:
Copy
body, and add"stream": Trueand"stream_options": {"include_usage": True}to the copy. Copy first, as you did with histories in Chapter 2: the caller’s dictionary must not change.body.copy()gives a new dictionary with the same keys and values.POST the copy to
f"{BASE_URL}/chat/completions"withjson=, and passheaders=headers,stream=Trueandtimeout=STREAM_TIMEOUT. Use the session pattern from Chapter 1:http = requests if session is None else session.Call
response.raise_for_status(), so an error status fails here rather than later.Return the response itself, without reading it. The caller reads the lines.
headers is None unless a caller passes some, and requests treats headers=None the same as no headers. Lesson 7 uses it to send a header that makes the scripted model break a stream on purpose.
The timeout for a stream
Chapter 1’s TIMEOUT = (3.05, 10) allows up to 10 seconds of waiting for data. streaming.py has its own constant, STREAM_TIMEOUT = (3.05, 30), and the second number means something more precise than “the whole reply”. It is the longest requests waits each time it is waiting for more bytes. The bound works the same way for every response. A normal response sends nothing until it is finished, so that one wait lasts about as long as the whole reply. A stream sends something every few milliseconds, so no single wait comes close, and a reply that keeps arriving can take minutes without ever timing out. The timeout fires only when the service goes silent for 30 seconds at a stretch. Comment lines count as bytes too, which is one reason services send them while a model is thinking. A live reasoning model can still think for longer than that before its first piece, so a read timeout on a live call is a normal outcome, as MODEL_SERVICE.md says.
Read the lines
Once open_stream works, run python streaming.py. After the two replies from Lesson 1, the demonstration opens a stream for Say hello in a short sentence. and loops over the response’s lines:
response = open_stream(hello)
for line in response.iter_lines():
print(line)
response.iter_lines() gives the body one line at a time, each as soon as it has fully arrived. That works only because the request used stream=True. Here is what it prints:
The lines of a stream, as iter_lines() gives them:
b': keep-alive'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": null}]}'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"content": "Hello there, it "}, "finish_reason": null}]}'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"content": "is nice to "}, "finish_reason": null}]}'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"content": "meet you."}, "finish_reason": null}]}'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}'
b''
b'data: {"id": "chatcmpl-scripted-60cb3303cf3e4ae68892c51f", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [], "usage": {"prompt_tokens": 8, "completion_tokens": 9, "total_tokens": 17}}'
b''
b'data: [DONE]'
b''
These are the lines from Lesson 2, blank lines included, but each one is wrapped in b'...'.
Bytes, not text
The b in front of the quotes means each line is a bytes value, not a string. A network connection carries bytes: numbers from 0 to 255. requests hands them to you as they came, because only you know how they should be read as text. Most characters you have used take one byte each, which is why the lines look almost like strings. Others take more:
line = b'data: {"content": "caf\xc3\xa9"}'
print(line)
print(line.decode("utf-8"))
print(type(line))
print(type(line.decode("utf-8")))
The é in café is two bytes, \xc3\xa9, in the encoding called UTF-8, which the service uses. .decode("utf-8") turns the bytes into a str, the same way you named encoding="utf-8" when you opened text files in Python in Practice. String methods such as .startswith work on the decoded string, and that is what the next lesson needs.
Do not use str(line) for this. It gives you the text b'data: ...', with the b and the quotes as part of the string, instead of decoding it.
The demonstration opens the stream a second time, because a stream’s lines can be read only once, and prints each line decoded:
response = open_stream(hello)
for line in response.iter_lines():
print(line.decode("utf-8"))
The same lines, decoded:
: keep-alive
data: {"id": "chatcmpl-scripted-ed4cc227957d410a99ce7adb", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": null}]}
data: {"id": "chatcmpl-scripted-ed4cc227957d410a99ce7adb", "object": "chat.completion.chunk", "created": 1794141681, "model": "pl-scripted", "choices": [{"index": 0, "delta": {"content": "Hello there, it "}, "finish_reason": null}]}
and so on to data: [DONE], followed by Run stopped at the next function to build: parse_line is built in Lesson 4. Your id and created values will differ.
iter_lines can also decode for you: response.iter_lines(decode_unicode=True) gives strings. This course decodes each line itself, so you can see the step.
open_stream puts "stream": True in the body and also passes stream=True to post. What does stream=True in the call do?
response.iter_lines() gives b'data: [DONE]'. How do you get the string data: [DONE]?
A streamed reply keeps arriving for 90 seconds, a piece every 50 milliseconds. The request used STREAM_TIMEOUT = (3.05, 30). What happens?
Task
In streaming.py, replace the unfinished body of open_stream(body, *, headers=None, session=None).
Copy
bodyand add"stream": Trueand"stream_options": {"include_usage": True}to the copy. The caller’s dictionary must not change.POST the copy to
f"{BASE_URL}/chat/completions"withjson=,headers=headers,stream=Trueandtimeout=STREAM_TIMEOUT. Use the session when one is passed, andrequestsotherwise.Call
raise_for_status()on the response, then return the response itself without reading its lines.
Run python streaming.py to see the stream’s lines as bytes and then decoded, then check your work.