🏆Capstone project: Find Each Hole with a Test

For ten chapters, every lesson told you where the hole was. Chapter 3 pointed at the password table, Chapter 6 at the owner rule, Chapter 9 at one header. This chapter stops doing that.

The project in this chapter is the reading list as you left it after Chapter 10, with eleven weaknesses put back. Nothing marks them. The code reads like code that works, tests/test_attacks.py holds five imports and no test, and every route still answers. What you get instead of a lesson that points is one file: SECURITY.md. It describes each weakness the way someone outside would notice it, and it never names a function or a line.

Your work in this lesson is to turn each of those eleven descriptions into a test that fails. You close nothing yet.

Read the contract

Open SECURITY.md. It is frozen: you read it and never edit it. It has three parts that matter today.

Findings lists eleven symptoms, each with the name its test must have. Finding 7 reads “Another reader can delete my book with the form on the page.” That sentence does not say which route, which function, or which missing call. It says what a person would see.

What a browser sends is a table for findings 10 and 11. Those two are about requests that other pages cause. The contract also tells you that one rule of mini_browser.py was weakened, so you cannot trust the model until Lesson 3. The table says what a correct model decides, and you may build those requests by hand from it: helpers.form_request() with a Cookie and an Origin header, or helpers.make_request() with an Origin.

Must still work lists what a fix may not break. You need it today too, because a test must not demand something this list forbids.

A finding is a symptom. A test is the symptom made exact: this request, this state, this assertion. Eight of the eleven names are tests you wrote once before, in Chapters 2 to 9. The other three are new, and each follows the same shape: set up the state, perform the attack, assert that it was refused.

Two findings you cannot see in a response

Findings 1 and 2 leave no trace in any status, header, or body. A login for a name that does not exist gets the same 401 as a wrong password. A tag compared with == gives the same answer as one compared safely. The difference is in the work the server does.

You have tested exactly this before. helpers.count_scrypt() counts every call of hashlib.scrypt from the moment you call it, and records the cost n of each. helpers.count_compare_digest() does the same for hmac.compare_digest. Chapter 3’s test for an unknown user and Chapter 2’s test for verify() both rest on a count. Nothing is timed, here or anywhere in this course: a count is exact, and a stopwatch is not.

A test that proves something

A red test is only half of the proof. The same test must pass once the finding is closed, and that means it must pass against a correct server. Two mistakes break this, in opposite directions.

The first is a test that never finds the hole. Take finding 9. A natural first try asks for /covers/..%2Fsecret.txt and asserts a 404. Run it, and it passes against the code in this folder, because that code already refuses a name that contains two dots. The test is green and the hole is still there. A cover name can leave the folder in more than one way, and Chapter 7 named them.

The second mistake is a test that asks for too much. A test for finding 5 that edits a token and then signs it again with steps.TEST_KEYS fails against every server, correct ones included, because no attacker holds that key and a server must accept what its own key signed. Start every test with its precondition, as you have done since Chapter 1: first assert that the honest request works, then perform the attack.

Four findings need one more thought each. For finding 4, keep the clock under the absolute limit, so that only the idle limit can end the session. For finding 6, call signing.load_keys() with a dict you build, as in Chapter 5. Do not call reading_server.main() from a test: with the finding open it starts a server and waits for connections. For finding 10, send a trusted Origin, or the Origin check answers before the form token is ever looked at. For finding 7 and finding 10, you need a form token: it is in the session, at state["sessions"][session_id]["csrf"], and the session id is the part after = in the pair that steps.login_cookie() returns.

Your task

Write the eleven tests in tests/test_attacks.py, with the names SECURITY.md gives. Reach the code through its modules, as the comment at the top of the file says: reading_server.handle(...), signing.verify(...), passwords.check_login(...). Then run them:

python -m pytest tests/test_attacks.py -q

The result to aim for in this lesson is unusual:

11 failed

A plain python -m pytest -q shows 15 failed, because four tests of the model in tests/test_browser.py are red too. Those are for Lesson 3. Read each failure before you move on. A test that fails on its precondition, or with a KeyError in its own setup, is red for the wrong reason.

Then press Check my work. It does not look at the code in this folder at all. It keeps a correct copy of the service and eleven copies that each have one finding, and it runs your tests against them: every test must pass against the correct one, and each must fail against the copy with its finding. So the folder still holds the weak server when this lesson is done. The next lesson starts on it.