Testing LLM Code · practice
Test Properties, Not Wording
Each test so far scripted one reply and checked one result. That is the right shape for “what does ask do with None?”, but it leaves a gap. The reply you scripted is one of endless replies a live model could send, and the next one will be worded differently. This lesson turns the question around: instead of “what did this reply produce?”, ask “what must be true whatever the reply was?”
A property
A property is something every acceptable result has, whatever its wording. Two that the study assistant promises:
answer_from_notesnever returns a citation it was not given. It may answer, or it may reject the reply with aValueError, but an answer it returns cites only labels of the passages it sent.run_agentstays within its limits. Its trace never has more steps thanmax_steps, and the budget it spent equals the tokens its trace records.
A property test checks one such sentence over many inputs. Python in Practice’s @pytest.mark.parametrize decorator does the repeating: it runs one test function once per row of a table, and reports each row as its own test.
This is not the “property-based testing” of libraries that invent random inputs for you. Here you choose the inputs, a table of replies including awkward ones, and the assertion is the property.
Citations come from the passages
The passages for the copy question are whatever best_chunks(question, note_chunks(load_notes()), 3) returns, and their labels come from Chapter 8’s chunk_label. The test builds that list of labels, then gives answer_from_notes each reply from a table:
@pytest.mark.parametrize(
"reply_text",
[
'{"answer": "Use scores.copy().", "citations": ["03-lists#4"]}',
'{"answer": "Use scores.copy().", "citations": ["03-lists#4", "99-nothing#1"]}',
'{"answer": "Use scores.copy().", "citations": []}',
"Use scores.copy().",
],
)
def test_an_answer_cites_only_passages_it_was_given(reply_text):
...
The rows are one valid reply, one that cites a passage that was never sent, one with no citations, and one that is not JSON at all. For each, the test calls answer_from_notes with a fake holding that reply, then asserts that every citation of the answer is one of the labels.
Rejecting a reply is also acceptable. For the second and the fourth rows, answer_from_notes raises a ValueError: Chapter 8’s own message for the unknown label, and pydantic’s ValidationError for the text that is not JSON, which is a kind of ValueError (Chapter 4). So the call goes inside try, and on except ValueError: the test does return, which ends that row as a pass. The property is not “the reply is accepted”; it is “nothing uncited gets through”.
Limits hold at every step limit
The second property drives run_agent with a model that never stops asking. Write out four tool-call replies, each asking for a different note, so no call repeats and the loop has no reason to stop as "stuck":
FOUR_CALLS = [
fakes.tool_call_reply([("c1", "read_note", '{"note_id": "n1"}')]),
fakes.tool_call_reply([("c2", "read_note", '{"note_id": "n2"}')]),
fakes.tool_call_reply([("c3", "read_note", '{"note_id": "n3"}')]),
fakes.tool_call_reply([("c4", "read_note", '{"note_id": "n4"}')]),
]
Then parametrize over a table of three rows, each a max_steps value and the replies:
@pytest.mark.parametrize(("max_steps", "replies"), [(1, FOUR_CALLS), (2, FOUR_CALLS), (3, FOUR_CALLS)])
def test_the_agent_stays_within_its_limits(max_steps, replies):
...
With two names, the first argument of parametrize is a tuple of names and each row is a tuple of values. In each row, run the agent with a FakeClient(replies), your own read_note tool as in Lesson 3, a fresh budget and sleep=waits.append, then assert two things: len(result["trace"]) <= max_steps, and budget["used"] equals the total of the tokens of every step in the trace, added up with a loop.
A loop that ignored max_steps would ask for a fifth reply, and the fake would fail the test with Your code called the model 5 times. A loop that counted some tokens twice would have a budget that disagrees with its trace.
Your turn
In tests/test_properties.py, write both tests: the citation property over the four replies, and the limits property over max_steps 1, 2 and 3. Then run them:
python -m pytest -q tests/test_properties.py
....... [100%]
7 passed in 0.23s
Two test functions, seven tests: pytest counts each row as one.
How your tests are checked
tests/test_properties.py runs against the correct code and three broken functions, and must fail on each:
answer_from_notesskips the citation check, so it only validates the reply as aCitedAnswer;run_agentignoresmax_steps;call_modeladds each reply’s usage to the budget twice.
None of these would show up in a test of one valid reply. Each breaks a property.
In the citation property test, answer_from_notes raises a ValueError for the reply that is not JSON, and the test does return. Why does that row count as a pass?
The limits test scripts four tool-call replies, each asking for a different note. Why different notes?
Task
In tests/test_properties.py, write two parametrized tests:
Over four reply texts for the copy question (one valid, one citing
03-lists#4and99-nothing#1, one with no citations, one that is not JSON): build the labels ofbest_chunks(question, note_chunks(load_notes()), 3)withchunk_label, callanswer_from_noteswith a fake holding the reply, and assert every citation is one of those labels. Insidetry, aValueErrormakes the testreturn.Over three rows,
max_steps1, 2 and 3, each with the same four written-out tool-call replies (n1ton4): runrun_agentwith your ownread_notetool and a fresh budget, and assertlen(result["trace"]) <= max_stepsand thatbudget["used"]equals the sum of the trace’stokens.
Run python -m pytest -q tests/test_properties.py, then check your work.