A Message Made of Lines

Here is the first request this course sends: it asks practice-api for the first page of its items, two items per page. In Python it is one bytes value:

request = b"GET /v1/items?page=1&page_size=2 HTTP/1.1\r\nHost: practice-api:8080\r\n\r\n"
print(request)

That is the whole request. No hidden structure sits behind it: these are the bytes a client writes to the socket, and the server reads them in the same order. The rest of this chapter writes them, sends them, and reads what comes back.

Lines that end in CR LF

The request is made of lines, and each line ends in two bytes: \r, a carriage return, then \n, a newline. Together they are called CRLF. The notice board in Networks with Python ended each line with \n alone. HTTP made a different choice, and as with the notice board, both sides follow the agreement, so a reader can find each line by looking for \r\n.

Read the request line by line:

1. GET /v1/items?page=1&page_size=2 HTTP/1.1 is the request line: what to do, to which resource, in which version of HTTP.

2. Host: practice-api:8080 is a header line: a name, a colon, and a value. This one names the server the request is meant for.

3. Then comes a line with nothing on it. The empty line ends the head of the request.

A response follows the same plan. Its first line is a status line, such as HTTP/1.1 200 OK; header lines follow; an empty line ends the head; and then the body follows, the JSON you decoded in Python in Practice.

request                               response
--------------------------------      --------------------------------
request line                  CR LF   status line                   CR LF
header line                   CR LF   header line                   CR LF
header line                   CR LF   header line                   CR LF
(empty line)                  CR LF   (empty line)                  CR LF
body (empty here)                     body: the JSON

Both messages have the same three parts. First a start line: the request line or the status line. Then a block of header lines. Then one empty line, which in bytes is just \r\n, so the head ends with \r\n\r\n: the end of the last header line followed by the empty line. Everything after that is the body. The request above has an empty body; a request that sends data, as later chapters do, puts it there.

Splitting bytes with split

Bytes and strings have a split method. data.split(separator) cuts data at every occurrence of separator and returns the pieces as a list, without the separators. Cut the request at every \r\n:

There are four pieces, and the last two are empty. The request ends with \r\n\r\n, two separators in a row: the first \r\n ends the Host line, and between it and the second one there is nothing, so split returns an empty piece there. After the last separator there is nothing either, which gives the second empty piece. The empty line that ends the head shows up as those empty pieces.

split works the same way on text, and the list it returns is an ordinary list, the same one you would get by writing it out:

words = "a,b,c".split(",")
print(words)
print(words == ["a", "b", "c"])

On bytes the separator must be bytes, b"\r\n"; on text it must be text.

print(len(b"HEAD / HTTP/1.1\r\nHost: x\r\n\r\n".split(b"\r\n")))

Splitting a response in two

A response needs a different cut first: the head from the body, at the first \r\n\r\n. Below is an example reply in practice-api’s shape, built from what this request is expected to get back on the site. The body is the one practice-api sends for this page; the date is made up.

split(b"\r\n\r\n", 1) returns a list of two items: parts[0] is the head, without the empty line, and parts[1] is the body. Splitting the head at \r\n then gives one item per line, status line first. The last number printed is 182, the body’s length in bytes, which is exactly what the Content-Length header announced. Lesson 5 uses that header to read the body.

The Server and Date lines come from the small web server that stands in front of practice-api; if a reply you receive has neither, it came from the service directly, and nothing in this course depends on them.

Why the second argument matters

The 1 in split(b"\r\n\r\n", 1) is maxsplit: cut at most once, at the first match, and keep everything after it as one piece. Without it, split cuts at every \r\n\r\n it finds. The head ends at the first one, but the body is data, and data can contain anything, including those four bytes. A body that held them would be cut into pieces, and parts[1] would be only the first of them. With maxsplit set to 1, the head comes from the first match and the body is whatever follows, intact.

Decoding the body

The body is bytes. JSON is text, so decode it first, with the encoding the Content-Type line names, then hand the text to json.loads:

This is the kind of dictionary requests handed you in Python in Practice. What requests.get did between receiving and decoding happens in the lines above, on bytes you can see.

Splitting works here because every byte was already in hand. A socket delivers bytes in pieces, as you saw in Networks with Python, so a client cannot split a reply it has only partly received. The next lesson sends the request for real and reads the reply’s head from a socket, piece by piece.