Testing LLM Code · capstone
Capstone project: Project: Tests for the Study Assistant
Lessons 2 to 5 each tested one function with one technique. The capstone in the next chapter puts the conversation, the retrieval tool and the agent loop behind one
The brief
Write tests/test_study_assistant.py, with tests for three functions:
trim_history(history, max_messages)inconversation.py, Chapter 2’s trimming: it returns a new list that keeps a system message at the start, drops the oldest whole turns (a user message and the reply to it) until the history fitsmax_messages, and never drops the newest question.search_notes(arguments)inretrieval.py, Chapter 8’s tool: forarguments["query"]it returns the bestKpassages of the shipped notes, each starting with its label in square brackets, such as[03-lists#4], separated by blank lines, orNo note matches that search.when none share a word with the query.The final
run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep)inreliable.py, as in Lessons 3 and 5. It stops with"answered","budget","stuck"or"max_steps", asksapprovebefore runningsave_flashcard, and records each step’stotal_tokensin the trace.
The suite must pass against the correct code and fail against each of these eleven defects:
trim_historydrops the system message along with old turns;trim_historydrops one message at a time instead of whole turns;trim_historytrims the caller’s list in place and returns it;search_notesreturns every matching chunk, not the bestK;search_notesleaves out the[label]at the start of each passage;search_notesreturns""instead ofNo note matches that search.;run_agentignoresmax_steps;run_agentletsBudgetExceededescape instead of returning stop reason"budget";run_agentnever stops a repeated call ("stuck");run_agentrunssave_flashcardwithout approval;run_agentrecords each step’s prompt tokens in the trace instead of its total.
Each defect is something a tired person could really write, and each would leave the study assistant looking fine on most questions. One test can catch more than one of them, and one defect may need a test of its own.
Things you will need
A history to trim. Write it out as a list of dictionaries in the test file: a system message, then turns, ending with a user message. Six messages trimmed to 5 is a good case, because an odd limit is where dropping one message at a time and dropping whole turns give different results. Work out by hand which messages should be left before you write the assertion.
Tests of search_notes read the shipped notes, through Path("notes"), relative to the project folder, like Lesson 4’s recordings. So assert on the structure of the result rather than on whole passages, which would break the day someone edits a note: split it on "\n\n", check that there are K passages, three with the shipped settings, and check how the first one starts. For the query independent copy of a list, the best passage is 03-lists#4. For capital of France, no note matches.
Your own tools for run_agent. As in Lessons 3 and 5, give the agent small tool functions of your own under real tool names: a read_note that returns a fixed string, and a save_flashcard that returns "Saved." without writing anything. A refused call never runs the tool; its result in the trace is Error: save_flashcard was not approved, so it did not run. Write tool arguments out as literal strings, such as '{"note_id": "n1"}'.
A budget that runs out at the right moment. call_model refuses a call when its estimate is more than what is left (Chapter 9 Lesson 6): the characters of every message whose content is a string, divided by 4, plus 800 for the reply. With the system message "S" and the question "Q?", the first step’s estimate is 3 / 4 + 800 = 800.75. If the first reply asks for one tool whose result is 10 characters long, such as "Ten chars.", the second request adds the assistant turn (content None, not counted) and the tool message, so its estimate is 13 / 4 + 800 = 803.25. After a first reply of 15 tokens, a limit of 810 leaves 795, so the second call is refused and the stop reason is "budget". A limit of 850 leaves 835, and the agent answers.
Run the suite
Run your suite in the
python -m pytest -q tests/test_study_assistant.py
The reference suite for this project has 13 tests. Yours can have more or fewer:
............. [100%]
13 passed in 0.23s
python -m pytest -q with no file runs this file and Lessons 2 to 5’s together.
Two ways to be fooled, as in Python in Practice’s project. A suite that is always red is not catching anything; it is just broken. And a red result because pytest could not collect the file, or could not import a module, has told you nothing about any defect. Check that the suite is green on the correct code before you believe a red one.
If you want to see a test catch a defect, you can make one yourself: in your copy of conversation.py, change turns = turns[2:] in trim_history to turns = turns[1:], run the suite, read which test failed and why, then change it back. Checking your work uses its own copies, so this cannot affect your grade.
How your suite is checked
As Python in Practice checked its project’s suite. tests/test_study_assistant.py runs with pytest, with a time limit, in a read-only copy of the project that holds the checker’s own copies of every course file, fakes.py, the shipped notes and the recordings. The suite must:
collect without errors, with at least one test;
pass twice on the correct code;
leave the project folder unchanged, so a test writes files only under
tmp_path;never call
llm-api;fail on each of the eleven defects, each installed by replacing one function in the checker’s copy.
A defect counts as caught only when a failing test called the function that was replaced, so a suite that fails for some other reason gets no credit. The feedback names the first defect no test caught, in the words of the list above, with a hint for the test that would catch it.
Why trim a history of six messages to 5, rather than to 4?
Your suite fails on the correct code because pytest could not import a module. What has it told you about the eleven defects?
Task
Write tests/test_study_assistant.py with tests for trim_history (conversation.py), search_notes (retrieval.py, reading the shipped notes) and run_agent (reliable.py). The suite must pass on the correct code and fail on each of the eleven defects in the lesson: three in trim_history, three in search_notes and five in run_agent.
Use a FakeClient and your own tool functions for run_agent, write tool arguments as literal strings, use paths relative to the project folder, and write files only under tmp_path. Run python -m pytest -q tests/test_study_assistant.py in the