Let http.client Do the Socket Work

wire.py can send any request and read any reply that announces its length, and you wrote every step of it. Python’s standard library has a module that does the same steps, http.client. This chapter hands the bytes to it and leaves the decisions to you: what a status means, which headers matter, and what to do when no response comes at all. This lesson writes the first version of get_json, which makes one GET through http.client and returns a record like the one fetch returns.

What http.client does for you

Each piece of fetch has a counterpart in http.client:

None of what you learned goes away. http.client writes the same request line and Host line, stops reading the head at the blank line, reads exactly Content-Length body bytes, and reads no body after HEAD. It also reads bodies that arrive without a Content-Length, which your read_body still refuses and Chapter 9 takes apart. You may have heard that requests builds on a library called urllib3; urllib3 in turn builds on http.client, so the module you meet here sits under the tool you used in Python in Practice.

One connection, one timeout

Calling http.client.HTTPConnection gives you a connection object:

import http.client

conn = http.client.HTTPConnection("practice-api", 8080, timeout=2.0)

Nothing has connected yet. The connection opens when the first request goes out. The timeout works like the one you passed to socket.create_connection: it bounds the connect and then each wait for bytes. Leave it out and the connection waits as long as the server likes.

Three calls make one exchange:

conn.request("GET", "/v1/items?page=1&page_size=2", headers={"X-Practice-Name": "http-with-python"})
response = conn.getresponse()
body = response.read()

request sends the request. getresponse waits for the head and returns a response object: response.status is the code as an int, response.reason the reason text, and response.read() returns the body as bytes. If the head does not arrive within the timeout, getresponse raises TimeoutError, the same exception your sockets raised.

See what it sends

set_debuglevel(1) makes a connection print what it sends and receives. Run this in the terminal; it is the exchange above with the debug output switched on:

python -c "
import http.client
conn = http.client.HTTPConnection('practice-api', 8080, timeout=2.0)
conn.set_debuglevel(1)
conn.request('GET', '/v1/items?page=1&page_size=2', headers={'X-Practice-Name': 'http-with-python'})
response = conn.getresponse()
print(response.status, response.reason, response.read())
conn.close()
"

The first line of output is the request, as bytes:

send: b'GET /v1/items?page=1&page_size=2 HTTP/1.1\r\nHost: practice-api:8080\r\nAccept-Encoding: identity\r\nX-Practice-Name: http-with-python\r\n\r\n'

You should see Host: practice-api:8080 and Accept-Encoding: identity in the send: line, and a reply: line reading HTTP/1.1 200 OK followed by header: lines that include Server, Date, and Connection. If the reply line reads HTTP/1.0 and there is no Server line, the service answered directly; what http.client sent is the same either way.

http.client added two lines you did not write: the Host line your host_header wrote by hand, and Accept-Encoding: identity, which asks the server not to compress the body. Your header comes after them. The last line of output is 200 OK and the body bytes, starting b'{"has_next":true,"items":[.

A connection you can replace

fetch took connect=socket.create_connection so that a test could hand it a fake socket. get_json does the same with make_connection=http.client.HTTPConnection: a test passes a stand-in, and the real class is the default. You call make_connection(HOST, PORT, timeout=timeout) exactly as you would call HTTPConnection. HOST, PORT, and PRACTICE_HEADERS are already at the top of wire.py.

Open fakes.py: it has two new functions. scripted_response(status, reason, header_pairs, body) builds a response with the status, headers and body you choose. fake_make_connection(response, calls) returns a stand-in for HTTPConnection whose connection hands back that response and records each step in calls as a dict: "connect" with host, port and timeout, "request" with what was sent, and "close". Lesson 6 uses both for your own tests.

Close on every path

A connection holds a socket until you call conn.close(). A socket from socket.create_connection could be used in a with statement, but an HTTPConnection cannot. contextlib.closing fills that gap: it wraps any object that has a close method so that with can use it, and calls close() when the block ends:

with contextlib.closing(make_connection(HOST, PORT, timeout=timeout)) as conn:
    conn.request("GET", target, headers=PRACTICE_HEADERS)
    response = conn.getresponse()
    ...

The block ends the same way whether the code inside it finishes or raises. When getresponse raises TimeoutError, close() still runs, and the exception carries on to your caller. Read the body inside the block: closing the connection can close the response with it, and then read() returns b"" instead of the body.

The headers, as parse_head returned them

response.getheaders() returns a list of pairs, each a name and a value, in the order the server sent them and with the names cased as the server wrote them. Build the same dict parse_head built: loop over the pairs, take the name with pair[0] and lowercase it, and append pair[1] to the list for that name. A header that repeats keeps every value, and a lookup never depends on how the server cased the name.

Your task

1. In wire.py, write get_json(target, *, make_connection, timeout). Inside with contextlib.closing(make_connection(HOST, PORT, timeout=timeout)) as conn:, send GET for target with headers=PRACTICE_HEADERS, get the response, build the header dict, and read the body. Return a dict with "status", "reason", "headers", and "body".

2. Try it on practice-api:

python -c "import wire; print(wire.get_json('/v1/items?page=1&page_size=2'))"

You should see 'status': 200, a 'content-type' of ['application/json; charset=utf-8'], and the same body bytes as the debug run. What you see here is for you; nothing from the terminal is graded.

Then press Check my work. get_json now returns whatever status arrived. The next lesson asks what a client should do about it.