Decode a Chunked Body

Lesson 2 read a chunked body by eye. This lesson teaches wire.py to read one: read_chunked(reader, *, limit=MAX_BODY) takes the reader dict that read_head has just used and returns the body, leaving the reader ready for whatever comes next on the connection. Start by watching it fail:

python pl_http.py chunked

Here your make_server and serve run a small server written for this course instead of your reading list. pl_http sends GET /chunked, and that server answers with the chunked body from Lesson 2. pl_http reads the reply’s head itself, puts the bytes that came after it into a reader’s buffer, and hands the reader to your read_chunked. Before you write it, the report says:

--- your read_chunked() returned ---
raised NotImplementedError: not written yet: Lesson 3

Two readers you have

Two functions in wire.py do the reading for you.

read_exact(reader, n), from Chapter 1, returns exactly n bytes, from the buffer first and then from recv, and keeps any extra bytes in the buffer. It raises ConnectionError when the peer closes before n bytes have arrived.

read_crlf_line(reader, *, limit=MAX_LINE) is new in this chapter and written for you. It does for one line what read_head does for a head: it returns the bytes before the next CRLF, without the CRLF, and leaves the rest in the buffer. A line longer than MAX_LINE, 1024 bytes, raises ValueError, so a size line cannot grow without end, and a close in the middle of a line raises ConnectionError.

With those two, every chunk is three reads:

the size line   read_crlf_line(reader)
the data        read_exact(reader, size)
the CRLF        read_exact(reader, 2), which must be b"\r\n"

Repeat until the size is 0. The two bytes after the data are not optional: a chunk whose data is not followed by CRLF is malformed, and reading on as if it were fine would put the next size line in the wrong place. Raise ValueError when they are anything else.

The size line arrives as bytes. Decode it with "iso-8859-1", as parse_head decodes a head, then cut it at the first ;, because anything after that is an extension you ignore, and strip the spaces around what is left.

Hex, strictly

The size is hexadecimal, and int(text, 16) reads hexadecimal. It also reads more than that:

print(int("1a", 16), int("0x1a", 16))

Both are 26, but only 1a is a chunk size. int with base 16 also accepts 1_a and +1a, and it turns -1a into -26. A chunk size is one or more hex digits and nothing else, so check every character against HEX_DIGITS, the string "0123456789abcdefABCDEF" in wire.py, before you call int, and raise ValueError for any other character or for an empty size.

Lesson 1 showed what happens when two programs on one connection disagree about where a body ends: the bytes of one message are read as another. A size that one reader accepts and another refuses is exactly such a disagreement. http.client itself is lenient here: it cuts the size line at ; and passes the rest to int(line, 16). Your reader will be stricter.

The end

A size of 0 means the data is over, and no data or CRLF follows that line. What follows are trailer lines, header lines sent after the body, and then one empty line. Read lines with read_crlf_line until one is empty, drop them, and return the body. Count the bytes of those lines, as read_head limits a head, and raise ValueError once they pass MAX_HEAD. What a trailer field means is beyond this course; your reader only has to get past them.

Whatever comes after the empty line belongs to the next response on the same connection. Leave it in reader["buffer"]. That is why the reader is a dict that outlives one call: the next read_head starts from those bytes.

Limits

The notice board in Networks with Python refused a frame whose announced length was over its limit, before reading a byte of it. The same rule holds here, chunk by chunk: before reading a chunk’s data, check whether the body so far plus this chunk’s size would pass limit, and raise ValueError if it would. Otherwise a size line of FFFFFFFF would have your client wait for 4 GB. limit defaults to MAX_BODY, 1000000 bytes.

Your task

Write read_chunked(reader, *, limit=MAX_BODY) in wire.py: size lines, data, CRLFs, the zero chunk, trailer lines, and the empty line, with the checks above. About twenty lines do it, and a small helper that turns a size line into a number is welcome.

You can try it in the Shell tab with the scripted socket from fakes.py, whose recv hands out the bytes you give it:

python - <<'EOF'
import wire
from fakes import scripted_socket

reader = wire.make_reader(scripted_socket([b"4\r\nWiki\r\n5;x=1\r\npedia\r\n0\r\n\r\nNEXT"]))
print(wire.read_chunked(reader))
print(reader["buffer"])
EOF

It prints b'Wikipedia' and then b'NEXT', the start of a next response, still in the buffer. You can also add tests of your own to tests/test_wire.py and run them with python -m pytest -q; they are yours, and Check my work does not run them. Then run the helper again:

python pl_http.py chunked

--- your read_chunked() returned ---
body: b'{"books": ["Kindred", "Caf\xc3\xa9"]}'  (31 bytes)
left in the buffer: b''  (0 bytes)

Seven bytes and twenty-four, and nothing left over. A decoder that reads 18 as eighteen takes six bytes too few and then finds no CRLF where it expects one.

Then press Check my work. Lesson 4 puts this decoder into fetch, next to a reader for bodies that end at the close.