🏆Capstone project: Project: Build Any Request

You now have every piece of a request: a method chosen for what it promises, a URL taken apart with urlsplit, a target encoded with make_target, a Host line from host_header, and header lines from format_headers. What is missing is one function that puts them together for any request at all, and a fetch that uses it. That is this project: build_request(method, url, headers=None, body=b""), and a fetch that takes a full URL. It is also what lets Chapter 4 point wire.py at a server you write yourself, on an address that is not practice-api’s.

A request can carry a body

Chapter 1 read bodies in responses, measured by Content-Length. A request can carry a body the same way: the head, the empty line, then exactly as many bytes as its Content-Length line announces. A PUT that sets book 7 to some JSON sends that JSON as its body. The server reads the head, finds the length, and reads exactly that many bytes, as your read_body does on the client side.

Content-Length counts bytes, not characters. The JSON text {"title":"naïve"} is 17 characters, and its UTF-8 encoding is 18 bytes, because ï takes two. build_request receives the body as bytes already, so len(body) is the right count; a client that counted the characters of the text would announce one byte too few, and the server would stop reading one byte early.

When to send the line follows from what the method means:

• When the body is not empty, send Content-Length, whatever the method.

• For POST and PUT, send it even when the body is empty, as Content-Length: 0. These methods carry content, so a server reading one expects to be told the size, and 0 tells it there is none.

• For GET, HEAD, and DELETE with an empty body, send no Content-Length at all. These methods do not expect a body, and HTTP asks a client not to announce one that is not there.

The rules for build_request

build_request returns the whole request as bytes. Read the rules all the way through before you start:

1. Split the URL with urlsplit. If its scheme is not http, raise ValueError: wire.py speaks plain HTTP only, and an https URL needs the TLS layer Chapter 9 describes.

2. The target is parts.path, or "/" when the path is empty, as in http://books.test; then "?" and parts.query when the query is not empty. The fragment is dropped. Copy the path and query exactly as the URL has them: they should already be encoded, by make_target or by whoever wrote the URL, and encoding them a second time would turn every % into %25.

3. The head starts with request_line(method, target), which still refuses a target holding a space, \r, or \n.

4. Then host_header(parts.hostname, parts.port). A URL that names no port gives parts.port as None, and the Host line has no port. parts.hostname is the host in lowercase, which is fine for names and IPv4 addresses, the only hosts this course uses.

5. Then the caller’s headers, through format_headers. build_request writes Host and Content-Length itself, so if the caller passes either of them, in any casing, raise ValueError: two Host lines or two lengths would leave the server guessing which one to believe, the same ambiguity Chapter 1’s read_body refuses in a response.

6. Then Content-Length: <len(body)> when the rules above say so.

7. Then the empty line. Encode the head as ASCII, and add the body bytes after it, unchanged.

Header order between different names is up to you; putting Host first, as in Lesson 4, is what HTTP recommends.

fetch takes a URL

fetch keeps everything it does after connecting. What changes is where it connects and what it sends:

def fetch(method, url, *, headers=None, body=b"", connect=socket.create_connection, timeout=TIMEOUT):

It builds the request with build_request(method, url, headers, body) before it connects, so a refused URL never opens a connection. It connects to (parts.hostname, port), where port is parts.port, or 80 when that is None. Inside the with block, it sends the request and reads the reply as before.

main follows: rename the second argument in build_parser from target to url, give it the default BASE_URL + HOOK_TARGET, and pass args.url to fetch. Then python wire.py with no arguments fetches the same page as always, and python wire.py HEAD http://practice-api:8080/v1/items/1 sends a HEAD to that URL.

ADDRESS is no longer used by fetch; make_request still uses it, and you can leave both as they are.

Your tests

fetch takes a URL now, so the tests in tests/test_wire.py that call wire.fetch("GET", "/v1/items/1", ...) will fail until you give them one. Add a constant near the top, ITEM_URL = "http://practice-api:8080/v1/items/1", and pass it in place of the target. The address the fake connect records stays ("practice-api", 8080), because that is where the URL points.

Then add three tests, each a plain test_ function with no parameters:

• a request with a body outside ASCII, such as "café".encode("utf-8"), whose Content-Length line counts its bytes, and which ends with the empty line and the body;

• a URL with a port and a fragment, whose request line holds the path and query but no #, and whose Host line holds the port;

• an https URL, which build_request refuses with ValueError. Write it the way test_a_close_inside_the_body_is_an_error, already in the file, is written: call wire.build_request inside try, return from the test in the except ValueError: branch, and raise AssertionError after the try if no error came.

Call every function through the module, as wire.build_request(...) and wire.fetch(...), inside each test, and keep import wire without any from wire import line. When you press Check my work, it runs your tests against your own wire.py and then against versions with one broken rule each, and a name imported with from would keep your version and hide the broken one.

See it work

These runs reach practice-api. First, an item whose id is built with make_target:

python -c "import wire; url = wire.BASE_URL + wire.make_target(['v1', 'items', '1']); r = wire.fetch('GET', url, headers=wire.PRACTICE_HEADERS); print(r['status'], r['body'])"

200 b'{"category":"color","id":1,"name":"blue","value":10}\n'

Next, a query built with urlencode, from numbers, which encoding leaves as they are:

python -c "import wire; target = wire.make_target(['v1', 'items'], {'page': 1, 'page_size': 2}); print(target); print(wire.fetch('GET', wire.BASE_URL + target, headers=wire.PRACTICE_HEADERS)['status'])"

The target it prints is /v1/items?page=1&page_size=2, the hook target Chapter 1 typed by hand, and the status is 200.

Last, the same item again, with the 1 written as its percent-encoding, %31:

python -c "import wire; r = wire.fetch('GET', wire.BASE_URL + '/v1/items/%31', headers=wire.PRACTICE_HEADERS); print(r['status'], r['body'])"

You should see 404 with "route not found", where the first command found the item as /v1/items/1: practice-api matches the path byte for byte and decodes nothing in it, so %31 is a different path even though it decodes to 1. Encoding is a promise between the code that encodes and the code that decodes; this is why make_target encodes only what needs encoding. A digit never needs it, so make_target sends 1, and the server finds the item. If this reply is the item instead, practice-api decoded the segment before matching it; the bytes you sent were the same either way.

None of these replies is checked. They show your client working against a real server; your tests are the evidence.

Run your tests:

python -m pytest -q

With build_request, the new fetch, and the updated tests in place, pytest reports 9 passed: the six tests you started with and your three. When it does, press Check my work.

If a test fails and you cannot see why, ask Monty to read the failing assertion beside the rule it checks in this lesson and say which of the two is wrong, without fixing it for you.

wire.py can now build and send any request to any plain-HTTP server, and every line of it is yours. Chapter 3 hands the socket, the request line, and the framing to Python’s own http.client and leaves the decisions to you.