0%

Testing LLM Code · capstone

Capstone project: Project: Tests for the Study Assistant

Lessons 2 to 5 each tested one function with one technique. The capstone in the next chapter puts the conversation, the retrieval tool and the agent loop behind one , and you will keep changing them. A suite that would notice them breaking is what makes those changes safe. This project writes it.

The brief

Write tests/test_study_assistant.py, with tests for three functions:

  • trim_history(history, max_messages) in conversation.py, Chapter 2’s trimming: it returns a new list that keeps a system message at the start, drops the oldest whole turns (a user message and the reply to it) until the history fits max_messages, and never drops the newest question.

  • search_notes(arguments) in retrieval.py, Chapter 8’s tool: for arguments["query"] it returns the best K passages of the shipped notes, each starting with its label in square brackets, such as [03-lists#4], separated by blank lines, or No note matches that search. when none share a word with the query.

  • The final run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep) in reliable.py, as in Lessons 3 and 5. It stops with "answered", "budget", "stuck" or "max_steps", asks approve before running save_flashcard, and records each step’s total_tokens in the trace.

The suite must pass against the correct code and fail against each of these eleven defects:

  1. trim_history drops the system message along with old turns;

  2. trim_history drops one message at a time instead of whole turns;

  3. trim_history trims the caller’s list in place and returns it;

  4. search_notes returns every matching chunk, not the best K;

  5. search_notes leaves out the [label] at the start of each passage;

  6. search_notes returns "" instead of No note matches that search.;

  7. run_agent ignores max_steps;

  8. run_agent lets BudgetExceeded escape instead of returning stop reason "budget";

  9. run_agent never stops a repeated call ("stuck");

  10. run_agent runs save_flashcard without approval;

  11. run_agent records each step’s prompt tokens in the trace instead of its total.

Each defect is something a tired person could really write, and each would leave the study assistant looking fine on most questions. One test can catch more than one of them, and one defect may need a test of its own.

Things you will need

A history to trim. Write it out as a list of dictionaries in the test file: a system message, then turns, ending with a user message. Six messages trimmed to 5 is a good case, because an odd limit is where dropping one message at a time and dropping whole turns give different results. Work out by hand which messages should be left before you write the assertion.

Tests of search_notes read the shipped notes, through Path("notes"), relative to the project folder, like Lesson 4’s recordings. So assert on the structure of the result rather than on whole passages, which would break the day someone edits a note: split it on "\n\n", check that there are K passages, three with the shipped settings, and check how the first one starts. For the query independent copy of a list, the best passage is 03-lists#4. For capital of France, no note matches.

Your own tools for run_agent. As in Lessons 3 and 5, give the agent small tool functions of your own under real tool names: a read_note that returns a fixed string, and a save_flashcard that returns "Saved." without writing anything. A refused call never runs the tool; its result in the trace is Error: save_flashcard was not approved, so it did not run. Write tool arguments out as literal strings, such as '{"note_id": "n1"}'.

A budget that runs out at the right moment. call_model refuses a call when its estimate is more than what is left (Chapter 9 Lesson 6): the characters of every message whose content is a string, divided by 4, plus 800 for the reply. With the system message "S" and the question "Q?", the first step’s estimate is 3 / 4 + 800 = 800.75. If the first reply asks for one tool whose result is 10 characters long, such as "Ten chars.", the second request adds the assistant turn (content None, not counted) and the tool message, so its estimate is 13 / 4 + 800 = 803.25. After a first reply of 15 tokens, a limit of 810 leaves 795, so the second call is refused and the stop reason is "budget". A limit of 850 leaves 835, and the agent answers.

Run the suite

Run your suite in the as you write it:

python -m pytest -q tests/test_study_assistant.py

The reference suite for this project has 13 tests. Yours can have more or fewer:

.............                                                            [100%]
13 passed in 0.23s

python -m pytest -q with no file runs this file and Lessons 2 to 5’s together.

Two ways to be fooled, as in Python in Practice’s project. A suite that is always red is not catching anything; it is just broken. And a red result because pytest could not collect the file, or could not import a module, has told you nothing about any defect. Check that the suite is green on the correct code before you believe a red one.

If you want to see a test catch a defect, you can make one yourself: in your copy of conversation.py, change turns = turns[2:] in trim_history to turns = turns[1:], run the suite, read which test failed and why, then change it back. Checking your work uses its own copies, so this cannot affect your grade.

How your suite is checked

As Python in Practice checked its project’s suite. tests/test_study_assistant.py runs with pytest, with a time limit, in a read-only copy of the project that holds the checker’s own copies of every course file, fakes.py, the shipped notes and the recordings. The suite must:

  • collect without errors, with at least one test;

  • pass twice on the correct code;

  • leave the project folder unchanged, so a test writes files only under tmp_path;

  • never call llm-api;

  • fail on each of the eleven defects, each installed by replacing one function in the checker’s copy.

A defect counts as caught only when a failing test called the function that was replaced, so a suite that fails for some other reason gets no credit. The feedback names the first defect no test caught, in the words of the list above, with a hint for the test that would catch it.

Why trim a history of six messages to 5, rather than to 4?

Your suite fails on the correct code because pytest could not import a module. What has it told you about the eleven defects?

Task

Write tests/test_study_assistant.py with tests for trim_history (conversation.py), search_notes (retrieval.py, reading the shipped notes) and run_agent (reliable.py). The suite must pass on the correct code and fail on each of the eleven defects in the lesson: three in trim_history, three in search_notes and five in run_agent.

Use a FakeClient and your own tool functions for run_agent, write tool arguments as literal strings, use paths relative to the project folder, and write files only under tmp_path. Run python -m pytest -q tests/test_study_assistant.py in the until it is green, then check your work.