🏆Capstone project: Project: Read Any Body

Your server keeps connections open, and your read_chunked decodes chunks, but your fetch still reads only two kinds of body: none after HEAD, and exactly content-length bytes. Anything else makes it raise ValueError. This project finishes the client: fetch reads any body by the rule of Lesson 2, and a new function, fetch_many, sends several requests on one connection, the way your server now expects them.

What the client must do

SERVICE_CONTRACT.md lists the rules under “The client in wire.py”. The two new ones:

• fetch reads a body by content-length, by chunks, or to the close, and reads none after HEAD or with a 1xx, 204, or 304 status. It refuses transfer-encoding together with content-length, and any transfer coding other than chunked.

• fetch_many sends one request at a time on one connection, asks to close on the last, and raises ConnectionError if the server ends the connection early.

Milestones

First, read_until_close(reader, *, limit=MAX_BODY). It returns the bytes already in the buffer and every byte after them, until recv returns b"", and leaves the buffer empty. A reply that ends at the close still gets a limit: raise ValueError once the bytes pass limit, before asking for more. About six lines.

Then read_body(reader, method, headers, *, status=200). Rewrite it by the rule of Lesson 2, in its order:

1. No body for HEAD, for a status below 200, or for 204 or 304: return b"" without reading.

2. With transfer-encoding: if content-length is there too, raise ValueError. The header must hold exactly one value, chunked in any case and with spaces around it allowed; anything else, such as gzip, chunked, raises ValueError, because this client decodes no other coding. Then return what read_chunked returns.

3. With no content-length: return what read_until_close returns.

4. Otherwise, read by content-length as before.

The rule from Chapter 1 that refused a body without a length is gone. The tests that came with this chapter in tests/test_wire.py no longer expect it, and the docstring at the top of that file says so.

Then fetch. It passes the reply’s status: read_body(reader, method, response["headers"], status=response["status"]). Without it, a 304 that carries a content-length would leave fetch waiting for bytes that never come.

Run the helper:

python pl_http.py bodies

Your fetch asks the small server from Lesson 3 for /length, /chunked, and /close, one connection each. Before your changes, the chunked reply ends in raised ValueError: this client cannot measure a body sent with Transfer-Encoding yet. After them, each of the three reports ends in a body:

body: b'{}'  (2 bytes)
body: b'{"books": ["Kindred", "Caf\xc3\xa9"]}'  (31 bytes)
body: b'{"ends": "at the close"}'  (24 bytes)

Last, fetch_many(base_url, targets, *, connect=socket.create_connection, timeout=TIMEOUT). It sends GET base_url + target for each target, in order, on one connection, and returns a list of records like the one fetch returns:

• Open the connection once, with connect((host, port), timeout=timeout) as sock:, with the host and port of base_url (port 80 when it has none), and make one reader for the whole connection. Bytes of the next reply can arrive in the same recv as the end of this one, and they wait in that reader’s buffer.

• Build each request with build_request("GET", base_url + target, headers). The last one carries the header Connection: close, so the server closes once every reply has arrived; the others need no Connection header.

• Send a request, read its head and its body, and only then send the next. Sending them all before reading, called pipelining, is something this course leaves alone.

• If a reply says connection: close before the last target, the server will read nothing more on this connection. Raise ConnectionError at once, with a message that holds both numbers, how many targets were answered and how many there were, such as the server closed the connection after 1 of 3 requests.

About sixteen lines. Then run it against your own server:

python pl_http.py many

statuses: [200, 200, 200]
/me: b'{"user": null}'  (14 bytes)
serve() returned after 1 connection; the server is closed

pl_http prints the three requests as one client -> server line, because it shows everything a connection carried together. Your server’s log has three requests, and serve counted one connection.

Your tests

You can add tests to tests/test_wire.py, as in Chapter 6. fakes.scripted_socket stands for a silent server that keeps the connection open when its script contains None, and for a closed one when the script simply ends. A reply with no length is read until the close, so a test of such a reply ends its script without None:

def test_a_body_that_ends_at_the_close():
    sock = scripted_socket([b"HTTP/1.1 200 OK\r\n\r\nab", b"c"])
    response = wire.fetch("GET", ITEM_URL, connect=fake_connect(sock, []))
    assert response["body"] == b"abc"

This time Check my work runs your tests once, as python -m pytest -x would. None may fail, and none may open a real connection. Run them yourself first:

python -m pytest -q

What is proven, and how

Check my work joins your client to your server without a socket. The connect it hands fetch_many returns a stand-in socket: what your client sends goes through your own HTTP/1.1 handler class in this process, and what your server writes comes back from recv. Once every reply has been read and the connection is still open, recv raises TimeoutError, which catches a client that reads to the close and a server reply with no length. The bodies and many runs are there to watch, not graded.

Then press Check my work. The framing, keep-alive, and close handling that requests does for you on a connection, you have now written by hand. Lesson 5 puts these same bytes inside TLS.