Testing LLM Code · practice
Use the Supplied Fake Client
Lesson 1 named four kinds of check. The first two need your code to get a reply without the model service: a reply you chose, so the test knows what the right result is, and one that arrives instantly, every time, with no network. That is what fakes.py is for.
The fake client
The graders that checked your work since Chapter 5 never called llm-api. They passed your functions a FakeClient instead of the real client, and fakes.py in your project is that same file. Your code cannot tell the difference, because it only ever calls client.chat.completions.create(...) and reads attributes of what comes back.
You use fakes.py; you do not need to read how it is built. This is its interface:
fakes.FakeClient(replies)
A client. Each call to client.chat.completions.create(...) returns the next
item of replies, in order, one per call. If the item is an error, that
call raises it.
client.requests
A list with one dictionary per call so far: "method" ("create"), then
every argument the call passed, such as "model", "messages" and "tools".
fakes.reply(content, *, finish_reason="stop", usage=(10, 5))
A text reply. content may be None. usage is (prompt_tokens,
completion_tokens), so this reply's total_tokens is 15.
fakes.tool_call_reply(calls, *, usage=(10, 5))
A reply that asks for tools. calls is a list of (id, name, arguments_text)
tuples, such as ("call_1", "read_note", '{"note_id": "03-lists"}').
fakes.stream(pieces)
A streamed reply, for code that passes stream=True.
fakes.from_recorded(path)
A reply rebuilt from a saved response. Lesson 4 uses it.
fakes.rate_limit_error(retry_after=1), fakes.server_error(status=500),
fakes.connection_error(), fakes.timeout_error(), fakes.bad_request_error()
Real openai errors. Put one in replies, and the call that takes it raises it.
A reply has only the attributes the course’s code reads: choices[0].message.content, choices[0].message.tool_calls, choices[0].finish_reason and usage. Code that reaches for anything else, such as response.id, fails with an AttributeError instead of passing by accident.
If your code calls the model more often than the list has replies, the extra call raises a plain AssertionError that names it: Your code called the model 2 times, but this check expected 1 call(s). A test that expected one call then fails, which is what you want.
A first test
Chapter 5’s ask(question, *, client) in client_calls.py sends one question and returns the reply’s text, or "" when the reply has no text. Here is a test of it, in the arrange, act, assert shape from Python in Practice:
import fakes
from client_calls import MODEL, ask
def test_ask_returns_the_reply_text():
client = fakes.FakeClient([fakes.reply("A list keeps items in order.")])
assert ask("What is a list?", client=client) == "A list keeps items in order."
assert client.requests[0]["model"] == MODEL
Arrange: a fake client with one scripted reply.
Act: call
askwith it, exactly as the program would call it with the real client.Assert: check what
askreturned, and check the request it sent, which the fake recorded inclient.requests[0].
The expected text is safe to write in the test, because the test wrote the reply. This is not a test of the model, which never ran; it is a test of what ask does with a reply. It never touches llm-api, so it runs in a few milliseconds and passes the same way with the service up or down.
The picture follows the same test from top to bottom. The text in the scripted reply comes back out of ask and into the assertion: the test wrote it into the fake, so it can expect it, and nothing on the way reaches llm-api.
![A test of ask with a fake client: the test scripts one reply, calls ask, gets that reply's text back, and checks it and the request the fake recorded in client.requests[0].](/media/pythonland/assets/b48a5acb1cced5e4-960.webp)
Your turn
Open tests/test_fake_client.py. Below its imports, write tests that check four things about ask:
It returns the reply’s text.
It returns
""when the reply’s content isNone. Scriptfakes.reply(None).The request it sends has
modelequal toMODEL, and itsmessagesare exactly[{"role": "user", "content": question}], with your question, and nothing else: no system message.Two calls on one fake with two scripted replies get the two texts back in order, and
client.requeststhen holds two requests.
Give each test a name that says what it checks, starting with test_. One assertion per behavior is enough; several assertions in one test are fine too.
Run your tests
python evals.py does not run your tests. Run them in the Shell, from the project folder:
python -m pytest -q tests/test_fake_client.py
With four tests that pass, pytest prints one dot per test and a summary. The time at the end varies:
.... [100%]
4 passed in 0.23s
python -m pytest -q with no file runs every test in the tests folder. Lessons 3 to 5 add files there, so that command grows with the chapter.
Never build the real client in a test. make_client() reaches for the service, and a test that needs the network is slow, fails when the service is down, and costs money on a live model. When your tests are checked, make_client() raises AssertionError: Tests must not call llm-api: pass a FakeClient., and any other request fails.
How your tests are checked
Checking your work runs tests/test_fake_client.py with pytest, in a read-only copy of the project that holds its own copies of every course file. Your tests must pass, twice, against the correct ask. Then they run against three broken versions of ask, one at a time, and must fail on each:
askreturnsNonefor a reply whose content isNone;asksends a system message before the question;asksends the model"pl-live"instead ofMODEL.
A test catches a broken version only when it calls ask and then fails. If one gets through, the feedback names it, so you know which test to add.
A test of a real model cannot assert its exact answer. Why may test_ask_returns_the_reply_text assert that ask returns exactly "A list keeps items in order."?
One broken ask sends a system message before the question. Which assertion catches it?
Task
In tests/test_fake_client.py, write tests for ask from client_calls.py that use fakes.FakeClient and check that:
askreturns the reply’s text;askreturns""when the reply’s content isNone(scriptfakes.reply(None));the request’s
modelisMODELand itsmessagesare exactly[{"role": "user", "content": question}];two calls on a fake with two replies return the two texts in order, and
client.requestsholds two requests.
Run them with python -m pytest -q tests/test_fake_client.py in the ask and fail on broken ones.