Three Ways a Body Ends

In Lesson 1 a refused body stayed on the connection, and the server read it as the next request. A client has the same problem from the other side. On a kept-open connection the next reply starts right after the last byte of this one, so the client has to stop reading this body at exactly that byte before it can find the next status line. You solved this once already: in Networks with Python, Chapter 8, every frame began with its length, and the reader took exactly that many bytes and no more. HTTP has three ways to say where a body ends, and one rule for which of them applies.

The order of the rule

A client reading a reply goes down this list and stops at the first line that fits (RFC 9112, section 6.3):

1. No body at all. A reply to HEAD, and any reply with a 1xx, 204, or 304 status, ends at the blank line after its headers, whatever those headers say. A 304 may even carry the content-length of the 200 it stands for; that number describes the stored body, not bytes on this connection.

2. transfer-encoding: chunked. The body arrives in chunks, each with its own size; a chunk of size zero, followed by an empty line, ends it.

3. content-length. The body is exactly that many bytes.

4. Neither. The body is every byte until the server closes the connection.

A reply can carry both transfer-encoding and content-length. The RFC settles it in favor of transfer-encoding, and adds that such a message “ought to be handled as an error”: a client and a proxy that picked different headers would cut the same bytes into different messages. Your client will refuse it.

The fetch in your wire.py handles two of these cases: no body after HEAD, and content-length. It refuses everything else. Lesson 4 teaches it the rest.

Chunks

Chunked framing writes the body as a series of chunks. Each chunk is a line with the chunk’s size in hexadecimal, then exactly that many bytes of data, then CRLF. A size may be followed by ; and an extension, which a reader is free to ignore. A chunk of size 0 says the data is over. After it come trailer lines, header lines sent after the body, usually none, and then one empty line that ends the message.

Here is the chunked body that a small server inside pl_http.py sends in Lessons 3 and 4, laid out one line per row:

7                         size: 7 bytes
{"books                   7 bytes of data
18;note=x                 size: hex 18 is 24 bytes; ;note=x is an extension
": ["Kindred", "Café"]}   24 bytes of data (é is two bytes in UTF-8)
0                         the zero chunk: no more data
X-Checksum: abc           a trailer line
                          the empty line: the message ends here

Every row ends with CRLF. Run the same bytes, split at each CRLF:

The two empty b'' lines at the end come from the final CRLF CRLF. Splitting at CRLF works here only because no chunk’s data contains CRLF. Data may contain any bytes, so a real decoder reads a size line, then exactly that many bytes, and never looks for the end of the data by searching.

The sizes are hexadecimal, and int reads them when you give it base 16:

print(int("18", 16), int("18"))

Why would a server choose chunks over a length? A length has to be known before the first byte of the body is sent. A server that produces the body as it goes, from a slow query or while compressing a large file, does not know the total yet. With chunks it can start sending at once and still mark the end, so the connection stays usable.

The close

The last item of the rule needs nothing from the server but a close. It is valid, and HTTP/1.0 servers often relied on it. Its price is the connection: the close is the only marker, so nothing can follow the body on that connection. When pl_http meets such a reply, it says (the reply has no content-length, so pl_http read until the server closed), and on a kept-open connection that read lasts until the server’s 2-second idle close. So a server that keeps connections open must give every body a content-length or chunked framing, which is why your Lesson 1 server needed no change to its replies.

Requests are framed by chunks or by content-length too, with one difference: a request with neither header has no body at all (RFC 9112, section 6.3). A request body cannot end at the close, because the client still needs the connection to read the reply. That is why your Chapter 5 server could insist on content-length for POST and PUT and treat any other request without one as having no body.

Two related techniques stay outside this course: sending several requests before reading any reply, called pipelining, and sending chunked bodies yourself. Your client will decode chunks and never send them. Lesson 3 writes that decoder.