Three Ways to Say How Long

Seven bits hold the numbers 0 to 127, and a payload can be far longer than that. RFC 6455 solves this without making every header bigger. Lengths from 0 to 125 go straight into the 7-bit field, as in Lesson 3. The two values left over are not lengths at all but markers: 126 means “the real length follows in the next 2 bytes”, and 127 means “the real length follows in the next 8 bytes”. One small field chooses among three layouts.

Here are the headers at the boundaries, every byte before the payload. The first four are text frames and the rest binary, which changes only the first byte:

payload bytes   header
0               81 00
125             81 7d
126             81 7e 00 7e
127             81 7e 00 7f
256             82 7e 01 00
65535           82 7e ff ff
65536           82 7f 00 00 00 00 00 01 00 00

Read the 256-byte row: 7e is 126, the marker, and 01 00 is the length in two bytes, 1 times 256 plus 0. The 65536-byte row has the marker 127, 7f, and then eight bytes that hold 65536, which is 1 times 256 times 256. The RFC itself uses those two rows as its examples of the longer forms.

struct, again

Two bytes and eight bytes are jobs for struct, which you used in Networks with Python Chapter 8 with the format "!I", one unsigned 4-byte integer in network byte order. This lesson needs two other sizes. "!H" is one unsigned integer in 2 bytes, 0 to 65535, and "!Q" is one in 8 bytes. The ! still means network byte order, big-endian, most significant byte first, and the RFC requires exactly that order. struct.unpack still returns a tuple, so the number is item [0]:

import struct

print((bytes([0x81, 126]) + struct.pack("!H", 127)).hex(" "))

The shortest form, always

Nothing in the bytes stops a sender from writing the length 5 as the marker 126 and then 00 05. RFC 6455 rules it out: section 5.2 says the minimal number of bytes must be used to encode the length. So each length has exactly one header, and a reader refuses any other. These three are refused:

81 7e 00 7d                     a 16-bit length of 125, which fits in 7 bits
82 7f 00 00 00 00 00 00 ff ff   a 64-bit length of 65535, which fits in 16 bits
82 7f 80 00 00 00 00 00 00 00   a 64-bit length with its top bit set

The third rule is separate. The RFC requires the top bit of the 8-byte length to be 0, so the largest length a frame can announce is 2 to the power 63, minus 1. That is still far more than any server should accept, and a reader that believed it would wait for, or try to hold, a payload no program could handle. Lesson 5 adds a size limit for that reason.

The read_length function

Encoding is one function of one number. Decoding has a complication: after the 7-bit length, the reader may need 0, 2 or 8 more bytes, and it cannot know which until it has read byte 1. So read_length(length, read_exact) takes two arguments, the 7-bit value from parse_header_bytes and a function to read more bytes with.

read_exact(n) is the function you wrote in Networks with Python Chapter 6, lesson 3, as read_exact(reader, n): it returns exactly n bytes, however the connection delivers them, keeps any extra bytes in a buffer, and raises ConnectionError if the peer closes first. Here it arrives as a parameter, a function passed as a value, so read_length never needs to know where the bytes come from. A real connection, a test, and frames_demo.py can each hand it their own read_exact. For example, with a stand-in from the chapter’s fakes.py that holds the bytes 01 00:

read_exact = fakes.reader_over([bytes([1, 0])])
# reads 2 bytes and returns 256
ws_frames.read_length(126, read_exact)

For a length of 125 or less, read_length reads nothing and returns the length itself. For 126, it reads 2 bytes and unpacks them with "!H"; for 127, 8 bytes with "!Q". Then it checks the rules above before returning.

Your task

In ws_frames.py:

1. Write length_field(n). Return bytes([n]) for 0 to 125; the byte 126 followed by struct.pack("!H", n) for 126 to 65535; and the byte 127 followed by struct.pack("!Q", n) above that.

2. Write read_length(length, read_exact). Return length when it is 125 or less, without reading. After 126, read 2 bytes and raise ValueError if the number is 125 or less. After 127, read 8 bytes and raise ValueError if the top bit is set (the number is over 0x7FFFFFFFFFFFFFFF) or if the number is 65535 or less.

3. Change encode_frame: drop the 125-byte limit and build the length bytes with length_field(len(payload)). Everything else stays as Lesson 3 wrote it.

Press Run. The Lesson 4 section no longer prints a refusal. It shows the first ten bytes of a 65536-byte binary frame, the RFC’s header:

Lesson 4: the start of a 65536-byte frame
82 7f 00 00 00 00 00 01 00 00

Then press Submit. The check compares length_field with bytes built by struct at every boundary and at lengths you have not seen, checks the RFC’s 256-byte and 65536-byte headers from encode_frame along with Lesson 3’s small frames, and hands read_length a stand-in read_exact that records how many bytes were asked for. Asking for more than the form needs fails, and so does accepting any of the three refused headers.

The codec can now write and read every length. The next lesson puts the reading parts together into decode_frame, which reads one whole frame from a stream.