Follow Redirects with a Hop Limit

Your server now answers GET /v0/books/2?x=1 with 308 Permanent Redirect and location: /books/2?x=1, and http.client hands that reply back and stops. Nothing in HTTP sends the second request for you. Whether to follow, where exactly to go, which method and body to send there, and when to give up are all the client’s decisions. This lesson writes a client that makes them: follow, in wire.py, built from three small functions.

One request

request_once(method, url, *, body, headers, make_connection, timeout) sends one request and returns one record. You have written it before: it is get_json from Chapter 3, Lesson 1, with any method, any URL and an optional body instead of a GET to practice-api.

• url_parts(url) from Lesson 2 gives the host, the port and the target.

• with contextlib.closing(make_connection(host, port, timeout=timeout)) as conn: makes the connection object and closes it on every path, also when getresponse() raises TimeoutError.

• conn.request(method, target, body=body, headers=headers) sends the request. body is None for a request with no body.

• Inside the block, build the header dict from response.getheaders() with lowercase names and lists of values, exactly as get_json does, and read the body.

The record has "status", "reason", "headers", "body", and one new key, "url": the URL this request went to. A follower needs it, because after a redirect the URL that answered is no longer the one it started with.

The method after a redirect

Lesson 1 sorted the five redirect statuses by what they do to the method. next_method(status, method) turns that table into code and returns the method for the next request:

When the method changes, the body goes too: a GET carries none. So does the request’s content-type header, which described a body that is no longer sent. When the method stays, the body and every header go to the new address unchanged. That is the whole point of 307 and 308.

Stop

A server can send a client in a circle: /a redirects to /b, and /b back to /a. A follower that trusts every location sends requests until something breaks. So follow counts. It takes max_hops, the most redirects it will follow, 5 by default (MAX_HOPS). A hop is one redirect followed. When max_hops redirects have already been followed and the reply is yet another redirect, follow raises ValueError instead of sending one more request. With max_hops=3 against the circle, it sends four requests and raises after the fourth:

moves = {"/a": "/b", "/b": "/a"}
path = "/a"
requests = 1
while path in moves and requests < 4:
    path = moves[path]
    requests = requests + 1
print(path, requests)

Only a reply that is a redirect counts. REDIRECTS, at the top of wire.py, lists the five statuses follow follows. A reply with any other status is final, and so is a 3xx without a location header, which gives the client nowhere to go. 304 Not Modified is a 3xx too, but it is not in the list: it is an answer, not a pointer.

The history key

follow returns the final record with one more key, "history": a list with one dict per redirect followed, each holding the redirect’s "status" and the "url" that sent it. An empty list means there was no redirect. requests keeps the same information, as a list of the redirect responses, in response.history, which Chapter 10 shows.

Here is the loop that puts it all together:

1. Send the request with request_once.

2. If the status is not in REDIRECTS, or there is no location, stop: this record is the answer.

3. If len(history) already equals max_hops, raise ValueError.

4. Append {"status": ..., "url": ...} for this reply to history.

5. Work out the method with next_method. If it changed, set the body to None and leave the request’s content-type out of the headers.

6. Resolve the location against the current URL with resolve_location, make that the current URL, and go back to step 1.

A while True: loop that ends with break at step 2 fits this shape. Resolving against the current URL, not the first one, is what made the third URL in Lesson 2’s quiz /books/3.

Real clients do one more thing: they drop credentials such as an Authorization header, so a login meant for one server is never handed to another. RFC 9110 (section 15.4) asks for more than libraries usually do: it asks a client to drop Authorization and Cookie on any automatic redirect, while libraries such as requests drop them only when the host changes. Chapter 7 comes back to it.

Your task

In wire.py, write next_method, request_once and follow. follow turns headers=None into {} before the first request, closes every connection through request_once, and passes timeout to every one. Then run the helper:

python pl_http.py follow

It calls wire.follow("GET", "http://books.test/v0/books/2", make_connection=...), where the connection it hands over goes to your own server on this machine, so books.test is never looked up. Each request and the record your follow returned are printed:

request 1  GET /v0/books/2  ->  308 Permanent Redirect
request 2  GET /books/2  ->  200 OK
--- your client returned ---
status: 200
...
url: 'http://books.test/books/2'
history: [{'status': 308, 'url': 'http://books.test/v0/books/2'}]

The lines left out are the reason, the headers and the body of book 2.

Then press Check my work. Your client and your server now agree on one redirect. The next lesson hands the same job to a library and compares its rules with yours.