Percent-Encode What You Send

Suppose the reading list keeps one record per title, and a client wants the record for “Café & Co”. Pasting the title into the target by hand gives this request line:

A request line has exactly three words, separated by single spaces: method, target, version. This one has five, and a server reading it cannot tell where the target ends. Your request_line from Chapter 1 refuses a target with a space for exactly that reason, and make_request could not send é anyway, because it encodes the head as ASCII. The & is a quieter problem: in a query it separates one value from the next, so a title holding & would be cut in two.

The fix is to change how such characters are written, so that they reach the server as data and never as structure. That is percent-encoding, and in this lesson you write the two functions that build and read it in wire.py. This lesson’s terminal has no network: nothing here is sent anywhere, and every result is a string you can print.

A byte as three characters

Percent-encoding writes one byte as % followed by the byte’s value in two hex digits. A space is the byte 32, which is 20 in hex, so a space becomes %20. A slash becomes %2F, and an ampersand %26.

A character outside ASCII is more than one byte. As Networks with Python showed, UTF-8 writes é as two:

Each of those bytes is encoded on its own, so é becomes %C3%A9, and Café becomes Caf%C3%A9. The encoded text is plain ASCII, and every byte of the original survives.

ASCII letters, digits, and the four characters -, ., _, and ~ never need encoding; they mean the same thing anywhere in a URL. Everything else is encoded when it is data.

One path segment with quote

A path is made of segments separated by /: /v1/items/42 has the segments v1, items, and 42. When a value goes into one segment, every character in it that is not plain data must be encoded, including /.

urllib.parse has the function for it, quote. Called with only the text, it leaves one character alone on purpose:

By default, quote treats / as safe, because it expects a whole path, where slashes separate segments. For one segment that is wrong: /v1/items/AC/DC names a segment AC and then a segment DC, which is a different path. Passing safe="" says that no character is safe to leave as it is, so the slash is encoded as %2F, and the segment stays one segment.

Query values with urlencode

A query is a list of name=value pairs joined by &, the format Python in Practice sent with params=. Building it by hand runs into the same problem: a value holding & or = would start a new pair. urlencode takes a dict and encodes every name and value, turning a number into its text first.

One part of its encoding differs from quote: urlencode writes a space in a value as +, not %20. That comes from HTML forms: when a browser sends a form, it writes each space in a value as +, and a server reading a query as form data turns + back into a space. urlencode follows that convention, and so does the reading side, such as parse_qs. It holds for the query only. Everywhere else in a URL, + is an ordinary, legal character that means a plus sign. In a path segment, quote writes a space as %20 and leaves the meaning of + alone.

With that in mind, predict what urlencode makes of a title holding both an & and a space:

from urllib.parse import urlencode

print(urlencode({"title": "R&D notes", "page": 2}))

The way back with unquote

A server that receives AC%2FDC and wants the text back decodes it with unquote, which turns each %XX back into its byte and decodes the bytes as UTF-8. There is also unquote_plus, which first turns every + into a space. That is right for form data in a query and wrong for a path segment, where + is a plus sign:

The first line keeps the plus signs; the second turns the last one into a space.

Encoding is a promise between the code that encodes and the code that decodes. Your side of it, as the client, is to encode every value you put into a target, so that whatever the server does next, it receives the value whole. Which parts a server decodes, and when, is the server’s choice; Lesson 5 shows what practice-api does with an encoded path.

Your task

Two functions in wire.py are this lesson’s, make_target and decode_segment, and both still raise NotImplementedError. The imports at the top of the file already bring in quote, unquote, and urlencode.

Write make_target(segments, params=None). It gets a list of segment strings and, optionally, a dict of query values, and returns the target as a str:

1. put each segment through quote(segment, safe="") and collect the results in a list;

2. join them with "/" and put one "/" in front, so ["v1", "items"] gives /v1/items;

3. if params has something in it, add "?" and urlencode(params). With params left as None, or given as an empty dict, there is no query and no "?". The test if params: is false for both.

Write decode_segment(segment): return unquote(segment).

Then try both in the terminal:

python -c "import wire; print(wire.make_target(['v1', 'items', 'AC/DC live']))"
python -c "import wire; print(wire.make_target(['v1', 'items', 'Café']))"
python -c "import wire; print(wire.make_target(['v1', 'items'], {'q': 'R&D notes', 'page': 2}))"
python -c "import wire; print(wire.decode_segment('AC%2FDC%20live'))"

With both functions written, the four lines print:

/v1/items/AC%2FDC%20live
/v1/items/Caf%C3%A9
/v1/items?q=R%26D+notes&page=2
AC/DC live

Each target is one word with no space in it, so request_line would accept every one of them.

If a check fails, ask Monty to read the failing message with you and point at the line of make_target it is about, without writing the fix.

When those lines print as shown, press Check my work.

The targets are safe now, but every request still carries the Host line Chapter 1 supplied. The next lesson replaces it with one you build, and adds header lines of your own.