Come Back After a Drop

The server now survives a slow viewer and drops a dead one. This lesson looks at the other end of the connection: the viewer’s program, after its connection is gone. It may have been dropped by keepalive after its network went away for a while, as in Lesson 4. The server may have restarted, closing every connection with 1001. Or the network between them failed in one of the ways Networks with Python listed: a reset, a timeout, a route that disappeared. In each case a viewer that wants to keep showing scores has to connect again.

The obvious way is to try again at once, and again if that fails, as fast as the program can. That is harmful. When a server restarts, every one of its viewers loses its connection at the same moment, and if they all retry at once, the server comes back to a flood of handshakes that can knock it over again. A client that cannot reach the server also burns its own battery and network trying.

Double the wait, up to a cap, with jitter

The usual answer has three parts. Wait before each new attempt, and double the wait after each failure: half a second, then one, then two. Stop doubling at a cap, so a long outage does not leave a client waiting an hour once the server is back. And do not wait exactly that long: pick a random time between zero and the ceiling, which is called jitter. Two clients that lost the server at the same moment then pick different times and spread out. RFC 6455 itself recommends this, in section 7.2.3: after an abnormal closure a client SHOULD delay its first attempt by a random amount of time and then back off, doubling the delay up to a limit.

The program below computes one client’s schedule. A real client takes its jitter from a random source; here a fixed list of numbers between 0 and 1 stands in for it, so the program prints the same lines every time, the way Python in Practice injected time instead of reading the clock. Run it:

It prints six lines, from attempt 1: wait up to 0.5 s, this time 0.31 s to attempt 6: wait up to 8.0 s, this time 2.64 s. The ceiling doubles from 0.5 to 8.0 and then stays at the cap. The waits jump around underneath it: attempt 4 waits 1.6 s although its ceiling is 4.0. In a real client, each wait would be a sleep, followed by a new connect; a successful connection resets the attempt count to 1.

print(0.5 * 2 ** 4, round(8.0 * 0.97, 2))

Resync needs no history

Once the viewer is connected again, it has missed every score that happened while it was away. It does not need to ask for them. The server’s first message to every viewer is the join snapshot, the whole board, so one message brings a returning viewer up to date.

The viewer needs one rule to go with it: keep the version of the last snapshot it showed, and ignore any snapshot whose version is not newer. A snapshot with the same version would only show the same board again, and an older one would make the scores go backwards, which can happen when a client briefly holds two connections or reads a late message from the connection it was leaving. The program below applies the rule to four snapshots arriving after the viewer showed version 4:

It prints version 3: ignored, version 5: shown, version 5: ignored and version 6: shown.

Where this lives

The staleness rule is part of the scoreboard’s contract, and the capstone’s client applies it. A full reconnect loop, with its sleeps, its connect calls and its random source, is something to recognize and not something this course builds: once you know the three parts, you can read one in any client library’s documentation and check that it doubles, caps and jitters.

Lesson 6 proves scoreboard v2 with tests of your own and runs it with a slowed viewer and a dead one.