Chapter 10
Testing LLM Code
Why You Can't Assert the Exact Answer
The study assistant can answer from the notes, run an agent, make a quiz and chat, and since Chapter 9 every call fails well. How do you know it keeps working when you change it? In Python in Practice you wrote tests: call a function, compare the result with the value you expected, and let pytest tell you when the two disagree. That works for trim_history. It does not work in the same way for an answer a model wrote.
Your project for this chapter
Open evals.py in the project editor. It is this chapter’s entry file: in the Shell, python evals.py runs its demonstration. You build four of its functions: save_response in Lesson 4, and retrieval_hit, citation_hit and run_eval in Lesson 6. The rest of it is finished code that the demonstration uses.
Most of what you write in this chapter is tests. They go in the tests folder, one file per lesson: test_fake_client.py, test_agent.py, test_recorded.py, test_properties.py and, for the project, test_study_assistant.py. Each file starts with a docstring and the imports its lesson needs, and no tests. tests/recordings holds two saved answers from a live model, which you meet below.
reliable.py is now Chapter 9’s finished file, and the files it imports are supplied next to it as before. Your tests import them and check them. You do not need to change any of them. Checking your work uses its own copies, so an edit there cannot change a grade, but it can stop the demonstration or your tests from working.
Two more files are new. fakes.py is the fake client that has checked your work since Chapter 5; Lesson 2 shows you how to use it. requirements-dev.txt lists what a developer of this project needs: everything in requirements.txt, plus pytest==9.1.1, the same split Python in Practice used. The sandbox already has pytest.
One question, two live answers
In the Shell, run python evals.py now. The demonstration starts with show_variability(), which reads the two recordings and prints them. It calls no model. Both are answers from pl-live’s model, openai/gpt-oss-20b, recorded on 2026-11-08, to the same question with no notes attached. They are the two other samples Chapter 8 mentioned, shown in full this time.
Two live answers to: What example list does the Python Land lists note use, and what values are in it?
--- live-lists-note.json (openai/gpt-oss-20b, recorded on 2026-11-08)
In the “Lists” section on the Python Language reference (the “Python Land” page for the list type), the example that is repeatedly referenced is the list
```python
[1, 2, 3, 4, 5, 6, 7, 8, 9]
```
The note states that this list contains nine integer values, numbered consecutively from 1 to 9.
--- live-lists-note-2.json (openai/gpt-oss-20b, recorded on 2026-11-08)
In the “lists” chapter of the *Python Land* notes, the canonical example that is used whenever a list is introduced is the list
```python
['spam', 'ham', 'eggs']
```
That list contains three string elements:
| Index | Value |
|------|-------|
| 0 | `'spam'` |
| 1 | `'ham'` |
| 2 | `'eggs'` |
The chapter uses this list repeatedly to illustrate list creation, indexing, mutating, slicing, and other list operations. No other values are in that example list – it’s just those three strings.
--- What notes/03-lists.md says:
A list is an ordered collection of values. You write one with square brackets, such as `scores = [90, 72, 85]`. A list can hold values of any type, including a mix of types and other lists.
Your tests do not run here. Run them in the terminal: python -m pytest -q
Recording the copy question's reply:
Run stopped at the next function to build: save_response is built in Lesson 4.
Read the two answers against the last paragraph, which is the note itself. The first says the lists note uses [1, 2, 3, 4, 5, 6, 7, 8, 9]. The second, a few seconds later, says ['spam', 'ham', 'eggs'] and adds a table. The note’s list is scores = [90, 72, 85]. Two runs of the same request gave two different lists, and both are wrong.
A live model picks each word with some randomness, so the same request can come back worded differently, longer or shorter, or with different facts. Without the notes in the request, this one had nothing to go on and made something up, differently each time. Chapter 8’s grounded answers are much better, but even they are worded differently from one run to the next.
Why assert answer == expected fails
Suppose you wrote this test:
def test_the_lists_answer():
answer = ask("What example list does the Python Land lists note use, and what values are in it?", client=make_client())
assert answer == "The note uses scores = [90, 72, 85]."
Against a live model it fails on almost every run, even on a day when the answer is right, because the wording differs. Against pl-scripted it might pass every time, but only because someone wrote that exact sentence into the scripted model’s table. A passing run then proves that the table has that row. It says nothing about what a live model would answer, or about whether your code handles an answer it has never seen. The test also needs the service to be up, and with a live model it costs money each time it runs.
What a test can check instead
A test of model-calling code checks your program, not the model. Four kinds of check hold up:
The request. Your code decides the model name, the messages, the tools and the format. A test can check exactly what was sent.
What your code does with each kind of reply. Text, no text, a tool call, a cut-off reply, a 429, a 500: a fake client can hand your code any of them, in any order, without the service. Lessons 2 to 4 do this, Lesson 4 with a real reply saved to a file.
Properties every acceptable answer shares. You cannot say which words a good answer uses, but you can say what any acceptable result has: it validates as a
CitedAnswer, it cites only passages it was given, and the agent stops within its step limit and its budget. Lesson 5 checks such properties over many replies.Quality, as a pass rate. Whether answers are good is not a yes or no about one answer. Lesson 6 builds an eval: a fixed set of questions, each with what a good result must contain, scored as a count out of ten. Lesson 7 uses it to compare two system messages.
The first three run in a fraction of a second with no network and give the same result every time, so they belong in python -m pytest -q. The fourth calls the model and is a measurement, not a pass or fail.
The plan for this chapter
Lesson 2 tests ask with the supplied fake client, and Lesson 3 tests the whole agent loop by scripting a tool request and an answer. Lesson 4 saves a real response to a file and replays it in a test. Lesson 5 writes tests that hold for any acceptable answer. Lessons 6 and 7 build the eval and compare two prompts with it. The project, Lesson 8, is a test suite for trim_history, search_notes and run_agent that has to notice eleven ways the code could break.
Why does assert answer == "The note uses scores = [90, 72, 85]." fail against a live model, even on a run where the answer is correct?
Which of these checks holds for every acceptable answer that answer_from_notes returns for the copy question?
A test sends the copy question through make_client() to pl-scripted and passes. What does that prove?