Testing LLM Code · practice
Test the Agent Loop with Scripted Replies
ask makes one call. The agent from Chapter 7 makes several, and what it sends on the second call depends on what the model asked for on the first. A test of the whole loop needs a fake that plays a conversation: first a reply that asks for a tool, then a reply with the answer. FakeClient already does that, because it hands out its replies in order.
The agent you are testing
Chapter 9 gave the agent its final form, in reliable.py:
run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep)
It returns a dictionary with answer, stop_reason, trace and messages. Each step of the trace is a dictionary with step, tokens, tool_calls and results. Every call goes through call_model, so it spends from budget and waits with sleep before a retry.
Everything run_agent would otherwise get from the program, the test supplies itself:
A client: a
FakeClientwith the two replies.A tool: a small function of the test’s own, under a real tool name, because
describe_toolslooks up each tool’s description by its name.def read_note(arguments): return "Lists keep order."is enough. It never opens a file, so the test does not depend on the notes.A budget: a fresh
{"limit": 20000, "used": 0}, so the test can check what the run spent.An approval function:
refuse_allfromagent.py. Nothing here needs approval, butrun_agentrequires one.A sleep:
waits = []and thensleep=waits.append. A list’sappendmethod is a function that takes one value, the same shape astime.sleep(seconds), so you can pass it as a value, as Chapter 7 passedapprove_all. If anything retried, the wait would land inwaitsinstead of pausing the test.
The scripted conversation
The two replies:
client = fakes.FakeClient(
[
fakes.tool_call_reply([("call_1", "read_note", '{"note_id": "03-lists"}')]),
fakes.reply("Done."),
]
)
On the first call the “model” asks for read_note with the arguments {"note_id": "03-lists"}. run_agent runs your read_note and sends the result back. On the second call it answers Done. Each fake reply uses the default usage=(10, 5), so it counts 15 tokens.
The arguments text is written out as a string, exactly as a model would send it. Write it with single quotes outside so the double quotes inside need no escaping.
Your turn
In tests/test_agent.py, write a test that runs run_agent with that client, your read_note in tools={"read_note": read_note}, any system message, approve=refuse_all, max_steps=6, a fresh budget and sleep=waits.append. Then assert that:
the stop reason is
"answered"and the answer is"Done.";the trace has two entries, and the first one’s
tool_calls[0]["name"]is"read_note"and itsresultsis["Lists keep order."];the second request,
client.requests[1], carries the tool round: itsmessagesare the system message, the question, the assistant turn that asked for the tool, and the tool message{"role": "tool", "tool_call_id": "call_1", "content": "Lists keep order."}, somessages[2]["role"]is"assistant"andmessages[3]is that tool message;budget["used"]is 30, two replies of 15 tokens;waitsis still empty, so nothing waited.
Run it in the
python -m pytest -q tests/test_agent.py
. [100%]
1 passed in 0.23s
One test, one dot. It ran the whole loop, two model calls and a tool, in a fraction of a second, with no network and no waiting.
Why one test checks so much
Each assertion catches a different way the loop could break while still looking fine from the outside. A loop that forgets to record results in the trace still answers. A loop that never sends the tool message back still gets Done. from this fake, because the fake does not read what it is sent; only the recorded request shows the gap. A loop that forgets to count tokens answers correctly and spends nothing, until a real bill arrives.
Checking your work runs tests/test_agent.py against the correct run_agent and against four broken ones, and your test must fail on each:
the tool results are left out of the trace;
the answer comes back as
"";the tool’s result is not sent back to the model;
the replies’ usage is not added to the budget.
A broken run_agent runs the tool but never sends its result back to the model. Which assertion catches it?
Why does the test pass sleep=waits.append to run_agent?
Task
In tests/test_agent.py, write a test that drives run_agent from reliable.py with fakes.FakeClient([fakes.tool_call_reply([("call_1", "read_note", '{"note_id": "03-lists"}')]), fakes.reply("Done.")]), a tool function of your own, def read_note(arguments): return "Lists keep order.", in tools={"read_note": read_note}, approve=refuse_all, max_steps=6, a fresh budget dictionary and sleep=waits.append with waits = []. Assert that:
the stop reason is
"answered"and the answer is"Done.";the trace has two entries; the first has the call’s name in
tool_callsand["Lists keep order."]as itsresults;in
client.requests[1]["messages"], item 2 is the assistant turn and item 3 is the tool message withtool_call_id"call_1"and your tool’s result;budget["used"]is 30;waitsis empty.
Run it with python -m pytest -q tests/test_agent.py, then check your work.