0%

Streaming · practice

Pull the Text Out of Each Chunk

parse_line hands you the JSON text of each chunk. json.loads turns that text into a dictionary, as it did for whole documents in Python in Practice. This lesson takes the piece of reply out of that dictionary, and returns an empty string when there is none.

The shapes of a chunk

Here are the chunks of the Say hello in a short sentence. stream again, decoded and shortened to the parts that matter, plus one shape that only live models send:

role_chunk = {"choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}]}
piece_chunk = {"choices": [{"index": 0, "delta": {"content": "Hello there, it "}, "finish_reason": None}]}
finish_chunk = {"choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}
usage_chunk = {"choices": [], "usage": {"prompt_tokens": 8, "completion_tokens": 9, "total_tokens": 17}}
live_null_chunk = {"choices": [{"index": 0, "delta": {"content": None}, "finish_reason": None}]}
print(piece_chunk["choices"][0]["delta"]["content"])

Only piece_chunk carries text. The others are all ways of carrying nothing:

ChunkWhat delta holdsThe text
the first chunk{"role": "assistant", "content": ""}""
a piece{"content": "Hello there, it "}"Hello there, it "
the finish chunk{}, no content key at allnone
the usage chunkthere is no delta, because choices is emptynone
a live null delta{"content": None}none

chunk["choices"][0]["delta"]["content"] works for the first two rows and fails on the other three. The finish chunk raises KeyError, because its delta has no content. The usage chunk raises IndexError, because there is no choices[0]. The live chunk gives None, which is not text.

Reading what may not be there

Check the empty list first. An empty list is false in an if, so if not chunk["choices"]: catches the usage chunk before you index into it.

For the delta, use .get("content"), the dictionary method you have used before: it returns the value when the key is there, and None when it is not. That handles the finish chunk. It still gives None for the live chunk, so finish with or "":

for delta in [{"content": "Hi "}, {"content": ""}, {}, {"content": None}]:
    print("[" + (delta.get("content") or "") + "]")

a or b gives a when a is true and b otherwise. A non-empty string is true. None and the empty string are both false, so both become "". Together, the empty-list check and .get("content") or "" give a string for every shape in the table, never None and never an error.

.get("content", "") is not enough on its own. The default is used only when the key is missing. For {"content": None} the key is there, so you get None back.

Build chunk_text

chunk_text(chunk) takes one decoded chunk dictionary and returns "" when chunk["choices"] is empty, and otherwise the delta’s content, or "" when there is none. Return the content exactly as it is, spaces included: the spaces at the ends of the pieces are what join them into sentences.

Run python streaming.py. The demonstration reads the Say hello in a short sentence. stream again. For each line, it skips what parse_line returns None for, stops at "[DONE]", decodes the JSON and prints what chunk_text returns, in square brackets, so an empty string still shows:

The text of each chunk, in brackets:
[]
[Hello there, it ]
[is nice to ]
[meet you.]
[]
[]
A long reply, streamed by stream_reply:
Run stopped at the next function to build: stream_reply is built in Lesson 6.

Six chunks: the first chunk with its empty content, the three pieces, the finish chunk and the usage chunk. The comment, the blank lines and [DONE] never reached chunk_text, because parse_line sorted them out first. Look at the space before each closing bracket: it is part of the piece.

A live model sends a chunk whose delta is {"content": None}. What does chunk["choices"][0]["delta"].get("content", "") return?

What should chunk_text return for the usage chunk, {"choices": [], "usage": {...}}, and why?

Task

In streaming.py, replace the unfinished body of chunk_text(chunk). It receives one chunk that json.loads has already turned into a dictionary.

  • When chunk["choices"] is empty, return "".

  • Otherwise return the content of chunk["choices"][0]["delta"], exactly as it is, or "" when the delta has no content key or its content is None.

chunk_text always returns a string and never raises for these shapes. Run python streaming.py to see the text of each chunk in brackets, then check your work.