Let urllib Follow for You

follow decides everything a redirect leaves open: where the location points, which method and body go there, and when to stop. Python’s standard library has a client that makes the same decisions for you, urllib.request. This lesson does what Chapter 3 did with http.client: it takes code you wrote by hand and hands the job to a library, so you can see exactly which of your decisions it makes, and where it decides differently.

An opener

urllib.request sends requests through an opener, an object that holds the steps a request goes through: making the connection, following redirects, turning error statuses into exceptions. urllib.request.build_opener() returns one with the usual steps in place, and opener.open(url, timeout=...) sends a GET and follows any redirects before it returns:

import urllib.request

opener = urllib.request.build_opener()
with opener.open("http://books.test/v0/books/2", timeout=2.0) as response:
    print(response.status, response.geturl())
    body = response.read()

The response works in a with statement, which closes it when the block ends; read the body inside the block. response.status is the final status, and response.geturl() is the final URL, after every redirect. You may know urllib.request.urlopen(url): it does the same through an opener the module builds once and keeps. Passing your own opener instead is the seam you have used since Chapter 1. open_url takes opener=None and builds one only when none is given, the way session=None worked in Python in Practice, so a test can hand it a stand-in.

What urllib follows

Here are your rules from Lesson 4 beside urllib’s:

On HEAD after a 303, urllib keeps a HEAD a HEAD, which RFC 9110 allows; your follower sends GET and reads a body the caller did not ask for. For a GET the two agree: both resolve a relative location, and both keep the query. urllib’s limits come from its own counters: it gives up after ten redirects, or when the same URL comes back too often. The difference is in writes. urllib follows a request with a body only where history lets it turn into a GET. Wherever HTTP says to send the same method and body again, which is exactly what 307 and 308 exist for, urllib refuses and raises an error instead. That is a safe choice for a general library, but a client that has to send a PUT through a 308, as old reading list clients do, cannot use urllib’s follower for it. Your follow can.

Errors are replies

opener.open raises urllib.error.HTTPError for any 4xx or 5xx reply. Its name says error, but the reply arrived: the exception carries everything get_json kept in Chapter 3. exc.code is the status, exc.geturl() the URL that answered, and exc.read() the body, which you read and then release with exc.close(). So open_url catches HTTPError and returns the same kind of record as for a success. A 404 is something the caller decides about, as get_json taught you.

Catch exactly that class, written through the module as urllib.error.HTTPError. It is tempting to catch something wider, and it is wrong. HTTPError is a kind of urllib.error.URLError, which is a kind of OSError. When the client cannot connect at all, open raises URLError, with the original error, such as ConnectionRefusedError, in its .reason. When the connection opens and no reply comes in time, open raises a bare TimeoutError, which is also an OSError. Neither is a reply, and neither has a status. An except OSError or except Exception would turn both into records as if a server had answered, the same mistake as reporting a timeout as a 504 in Chapter 3. open_url lets both through.

Your task

Write open_url(url, *, opener=None, timeout=TIMEOUT) in wire.py:

1. When opener is None, set it to urllib.request.build_opener().

2. with opener.open(url, timeout=timeout) as response: return a dict with "status" (response.status), "url" (response.geturl()) and "body" (response.read()).

3. except urllib.error.HTTPError as exc: read the body with exc.read(), call exc.close(), and return the same three keys from exc.code, exc.geturl() and the body.

wire.py already has import urllib.error and import urllib.request. Keep them that way, and write the names through the module: a from urllib.error import HTTPError line would stop Check my work from handing your code a stand-in error.

Then run the helper:

python pl_http.py urllib

It calls your open_url twice with an opener whose connections go to your own server on this machine, so no name such as books.test is looked up. /v0/books/2 shows two requests, 308 Permanent Redirect and then 200 OK, and your record with url: 'http://books.test/books/2'. /books/99 returns your record with status: 404 and the error body. Then the helper sends its own urllib POST /v0/books, and the last block reads:

--- pl_http's own urllib POST /v0/books ---
raised HTTPError: HTTP Error 308: Permanent Redirect

That is urllib refusing to resend a body through a 308. Then press Check my work. The next lesson sends writes through your own follower instead, and proves that it and your server agree.