0%

Testing LLM Code · practice

Save a Real Response and Replay It

The replies in your tests so far were typed by hand. That is quick, and it lets a test script any reply it needs, but a hand-typed reply can drift from what a real service sends: a field it never has, text no model would write, a tool call shaped slightly wrong. A recording fixes that. You save one real reply to a file, once, and every test after that replays it.

A response is a pydantic model

What client.chat.completions.create returns is not a dictionary. It is an object of the openai package’s class ChatCompletion, and that class is a pydantic model, like Chapter 4’s QuizSet. So it has the method you used the other way round in Chapter 4: QuizSet.model_validate_json(text) turned JSON text into an object, and response.model_dump_json() turns an object into JSON text. With indent=2 the text is spread over lines, easy to read in the editor.

Build save_response

In evals.py, write save_response(response, path):

  1. Open path for writing, with encoding="utf-8". path can be a string or a Path; open takes either.

  2. Write response.model_dump_json(indent=2) to it.

  3. Return nothing.

Mode "w" replaces whatever the file held, so saving again gives a fresh recording, not two responses run together.

Record one reply

Run python evals.py again. After the live answers from Lesson 1, the demonstration calls record_copy_answer(). It sends the copy question with its best three passages, exactly as Chapter 8 did, through the real client, passes the response to your save_response with the path tests/recordings/copy-question.json, and prints the file:

Your tests do not run here. Run them in the terminal: python -m pytest -q
Recording the copy question's reply:
Saved tests/recordings/copy-question.json:
{
  "id": "chatcmpl-scripted-fbe288a867ac4c1aba660a1c",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "content": "{\"answer\": \"Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.\", \"citations\": [\"03-lists#4\"]}",
        "refusal": null,
        "role": "assistant",
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": null
      }
    }
  ],
  "created": 1794172068,
  "model": "pl-scripted",
  "object": "chat.completion",
  "metadata": null,
  "moderation": null,
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "completion_tokens": 43,
    "prompt_tokens": 327,
    "total_tokens": 370,
    "completion_tokens_details": null,
    "prompt_tokens_details": null
  }
}
The eval with GROUNDED_SYSTEM:
Run stopped at the next function to build: run_eval is built in Lesson 6.

That is the whole response as the client saw it. Your file has another id and created time, because the service gives every response a new one. The course’s code reads four things from it, and only those: choices[0].message.content, here the CitedAnswer JSON as a string; choices[0].message.tool_calls, here null; choices[0].finish_reason; and usage, here 327 + 43 = 370 tokens. The rest, from logprobs to system_fingerprint, is there because the client’s class has those fields.

The demonstration then stops at Lesson 6.

Replay it in a test

fakes.from_recorded(path) reads such a file and builds a fake reply from it, keeping only the four things above. It accepts the client’s model_dump_json output, as you just saved it, and also a response body as the service sent it over HTTP. tests/recordings/live-lists-note.json, the first of Lesson 1’s live answers, is such a body: it came from OpenRouter, with extra fields such as provider, native_finish_reason and cost that this course never reads. It replays the same way.

reply = fakes.from_recorded("tests/recordings/copy-question.json")
client = fakes.FakeClient([reply])

The paths are relative to the project folder, as retrieval.py’s NOTES_DIR = Path("notes") is. That is the folder you run pytest from, and the folder checking your work runs your tests in, so the same paths work in both.

Your turn

In tests/test_recorded.py, write two tests:

  1. Replay tests/recordings/copy-question.json through answer_from_notes(question, load_notes(), client=client, k=3), with the copy question, How do I make an independent copy of a list?, and assert that the citations are ["03-lists#4"].

  2. Replay tests/recordings/live-lists-note.json through ask, and assert that it returns the recorded text. Do not type the text into the test: open the recording with json.load and read it from ["choices"][0]["message"]["content"], so the expected value is the one in the file.

If python evals.py could not reach llm-api, copy-question.json does not exist yet, and your first test fails with FileNotFoundError until a run of it succeeds. Checking your work uses its own copy of the file.

Run them:

python -m pytest -q tests/test_recorded.py
..                                                                       [100%]
2 passed in 0.22s

What to assert, and what not

Assert on what the course reads: the content, the citations, the tool calls, the usage. Never assert on id or created. They change with every recording, and the copy of copy-question.json that checking your work uses has its own.

Asserting on the recorded text is safe here, unlike Lesson 1’s assert answer == expected, because a recording never changes. The test does not ask the model anything. It checks how your program handles one real reply, so it can never say whether the model answers well. When you change a prompt, record again: an old recording answers a request your program no longer sends.

How your work is checked

save_response is checked with two real ChatCompletion objects, one with text and one with two tool calls. The file must hold the whole response as JSON, read back as UTF-8, fakes.from_recorded must rebuild the same content, tool calls, finish reason and usage from it, and nothing may be returned.

Then tests/test_recorded.py runs in a read-only copy of the project, against the correct code and two broken versions, and must fail on each:

  • answer_from_notes returns no citations;

  • ask returns "" instead of the reply’s text.

Why should a test of a recording never assert on its id or created?

Your test replays copy-question.json through answer_from_notes and passes. What has it shown?

Task

  1. In evals.py, write save_response(response, path): open path for writing with encoding="utf-8", write response.model_dump_json(indent=2), and return nothing. Run python evals.py in the Shell to record tests/recordings/copy-question.json.

  2. In tests/test_recorded.py, write two tests with fakes.from_recorded:

    • replay tests/recordings/copy-question.json through answer_from_notes for the copy question with load_notes() and k=3, and assert the citations are ["03-lists#4"];

    • replay tests/recordings/live-lists-note.json through ask, and assert it returns the recorded text, read from the file with json.load.

Never assert on id or created. Run python -m pytest -q tests/test_recorded.py, then check your work.