Send a Request by Hand

The last lesson split bytes that were already in hand. This one produces them: you write the request in wire.py, send it to practice-api through a connect function, and read the reply’s head from the socket as it arrives, piece by piece, until the empty line.

This lesson runs on Python Land’s servers. The Project tab holds the chapter’s files: wire.py, where you work, fakes.py, which Lessons 5 and 7 use, tests/test_wire.py, which Lesson 7 uses, and README.md, the routes and rules for this chapter. The Shell tab is a terminal in the same folder.

Open wire.py. Its constants include ADDRESS, the server’s name and port; HOST_LINE; HOOK_TARGET, the target of the request from Lesson 2, which this chapter calls the hook request; and the limits MAX_HEAD and MAX_BODY. make_reader and read_exact are the receive buffer from Networks with Python, already written. The functions after them are yours to write, a few per lesson.

The request line

A request line has three parts separated by single spaces: the method, which says what to do (GET asks for a resource); the request target, which names the resource on the server (a path, and possibly a query after ?); and the version, HTTP/1.1. CRLF ends the line. An f-string builds one from its parts:

A server reads the request line by cutting it at the spaces and at the CRLF, so the target must not hold any of them. /v1/items/a b would give the server four words where it expects three, and a target holding \r\n would end the line early and start a header line the caller never meant to send. So request_line refuses such a target with ValueError, as it refuses a target that does not start with /. Chapter 2 shows how to put any character safely inside a target.

The Host line

The second line is the Host line, which names the server the request is for. HTTP/1.1 requires it in every request, and Chapter 2 explains why. In this chapter it never changes, so wire.py supplies it as HOST_LINE:

HOST_LINE = "Host: practice-api:8080\r\n"

make_request(method, target) joins the request line, HOST_LINE, and one more "\r\n" for the empty line, and encodes the text with "ascii", because a socket sends bytes. For the hook request:

The request ends with \r\n\r\n, which tells the server the head is over; this request has no body. Every line ends in CRLF because HTTP/1.1 requires it. Some servers also accept a bare \n, but a tolerant server is no reason to send less than the protocol asks for.

Reading the head as it arrives

The reply’s head ends at the first \r\n\r\n. One recv returns whatever has arrived so far, maybe the whole reply, maybe part of the first line, so reading the head is a loop around the reader’s buffer:

1. Look for \r\n\r\n in reader["buffer"] with find, which returns its position, or -1 when it is not there.

2. While it is not there, call reader["sock"].recv(4096) and add the piece to the end of the buffer, then look again.

3. When it is there, return the bytes before it, and store everything after the empty line back in reader["buffer"].

Check the buffer before the first recv: if the bytes are already there, there is nothing to wait for. And keep what follows the head. It is the start of the body, which Lesson 5 reads, and a reply often arrives with head and body in one piece.

There are two ways the loop can end without a head, and each gets its own exception:

• recv returns b"": the server closed the connection before the head ended. Raise ConnectionError, as read_exact does.

• The buffer grows past limit bytes without a \r\n\r\n: a head that never ends would otherwise fill the program’s memory. Raise ValueError, as your read_line did in Networks with Python for a line that never ended. limit defaults to MAX_HEAD, 16384 bytes.

Sending through connect

fetch puts everything together. Its first two lines open the connection and send the request:

with connect(ADDRESS, timeout=timeout) as sock:
    sock.sendall(make_request(method, target))

connect is a parameter, so a test can pass a function that returns a fake socket, as your notice board client did in Networks with Python. The timeout given to socket.create_connection stays on the socket, so every later recv waits at most timeout seconds, 2 by default, and then raises TimeoutError. For now fetch returns the head as bytes; Lesson 5 turns it into a full response.

Your task

In wire.py, write these four functions, in this order:

1. request_line(method, target): return f"{method} {target} HTTP/1.1\r\n"; raise ValueError if the target does not start with /, or holds a space, \r, or \n.

2. make_request(method, target): return the request line, HOST_LINE, and "\r\n", encoded with "ascii".

3. read_head(reader, *, limit=MAX_HEAD): the loop above.

4. fetch(method, target, *, connect=socket.create_connection, timeout=TIMEOUT): connect to ADDRESS with timeout=timeout, send the request, make a reader, and return what read_head returns.

Leave the functions marked for Lessons 4 to 6 as they are.

Then send the hook request from the Shell tab. main prints whatever fetch returns, so you see the head as a bytes value:

python wire.py
python wire.py HEAD

The first sends GET with HOOK_TARGET, the second HEAD with the same target. Each prints the head as bytes: the status line first, every line break shown as \r\n, and no body. Both return at once, because read_head stops at the empty line. Nothing from your terminal is checked.

If you want to see what the Host line is for, send the request without it. This command writes the request line and the empty line, and nothing in between:

python -c "import socket, wire; sock = socket.create_connection(wire.ADDRESS, timeout=2.0); sock.sendall((wire.request_line('GET', wire.HOOK_TARGET) + '\r\n').encode('ascii')); print(wire.read_head(wire.make_reader(sock)))"

Leave out the Host line and you should get 400 Bad Request with a short HTML page instead of JSON: the web server in front of practice-api refuses the request before the service sees it. If you get the usual 200 instead, you reached the service directly, which tolerates the omission; HTTP/1.1 still requires the line. The command prints only the head, so look at the status line.

If the loop in read_head puzzles you, ask Monty:

Why should my read_head look for \r\n\r\n in the buffer before calling recv, and not only after?

When python wire.py shows the head, press Check my work. The next lesson turns those bytes into a status code and a dictionary of headers.