0%

Testing LLM Code · practice

Test the Agent Loop with Scripted Replies

ask makes one call. The agent from Chapter 7 makes several, and what it sends on the second call depends on what the model asked for on the first. A test of the whole loop needs a fake that plays a conversation: first a reply that asks for a tool, then a reply with the answer. FakeClient already does that, because it hands out its replies in order.

The agent you are testing

Chapter 9 gave the agent its final form, in reliable.py:

run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep)

It returns a dictionary with answer, stop_reason, trace and messages. Each step of the trace is a dictionary with step, tokens, tool_calls and results. Every call goes through call_model, so it spends from budget and waits with sleep before a retry.

Everything run_agent would otherwise get from the program, the test supplies itself:

  • A client: a FakeClient with the two replies.

  • A tool: a small function of the test’s own, under a real tool name, because describe_tools looks up each tool’s description by its name. def read_note(arguments): return "Lists keep order." is enough. It never opens a file, so the test does not depend on the notes.

  • A budget: a fresh {"limit": 20000, "used": 0}, so the test can check what the run spent.

  • An approval function: refuse_all from agent.py. Nothing here needs approval, but run_agent requires one.

  • A sleep: waits = [] and then sleep=waits.append. A list’s append method is a function that takes one value, the same shape as time.sleep(seconds), so you can pass it as a value, as Chapter 7 passed approve_all. If anything retried, the wait would land in waits instead of pausing the test.

The scripted conversation

The two replies:

client = fakes.FakeClient(
    [
        fakes.tool_call_reply([("call_1", "read_note", '{"note_id": "03-lists"}')]),
        fakes.reply("Done."),
    ]
)

On the first call the “model” asks for read_note with the arguments {"note_id": "03-lists"}. run_agent runs your read_note and sends the result back. On the second call it answers Done. Each fake reply uses the default usage=(10, 5), so it counts 15 tokens.

The arguments text is written out as a string, exactly as a model would send it. Write it with single quotes outside so the double quotes inside need no escaping.

Your turn

In tests/test_agent.py, write a test that runs run_agent with that client, your read_note in tools={"read_note": read_note}, any system message, approve=refuse_all, max_steps=6, a fresh budget and sleep=waits.append. Then assert that:

  1. the stop reason is "answered" and the answer is "Done.";

  2. the trace has two entries, and the first one’s tool_calls[0]["name"] is "read_note" and its results is ["Lists keep order."];

  3. the second request, client.requests[1], carries the tool round: its messages are the system message, the question, the assistant turn that asked for the tool, and the tool message {"role": "tool", "tool_call_id": "call_1", "content": "Lists keep order."}, so messages[2]["role"] is "assistant" and messages[3] is that tool message;

  4. budget["used"] is 30, two replies of 15 tokens;

  5. waits is still empty, so nothing waited.

Run it in the :

python -m pytest -q tests/test_agent.py
.                                                                        [100%]
1 passed in 0.23s

One test, one dot. It ran the whole loop, two model calls and a tool, in a fraction of a second, with no network and no waiting.

Why one test checks so much

Each assertion catches a different way the loop could break while still looking fine from the outside. A loop that forgets to record results in the trace still answers. A loop that never sends the tool message back still gets Done. from this fake, because the fake does not read what it is sent; only the recorded request shows the gap. A loop that forgets to count tokens answers correctly and spends nothing, until a real bill arrives.

Checking your work runs tests/test_agent.py against the correct run_agent and against four broken ones, and your test must fail on each:

  • the tool results are left out of the trace;

  • the answer comes back as "";

  • the tool’s result is not sent back to the model;

  • the replies’ usage is not added to the budget.

A broken run_agent runs the tool but never sends its result back to the model. Which assertion catches it?

Why does the test pass sleep=waits.append to run_agent?

Task

In tests/test_agent.py, write a test that drives run_agent from reliable.py with fakes.FakeClient([fakes.tool_call_reply([("call_1", "read_note", '{"note_id": "03-lists"}')]), fakes.reply("Done.")]), a tool function of your own, def read_note(arguments): return "Lists keep order.", in tools={"read_note": read_note}, approve=refuse_all, max_steps=6, a fresh budget dictionary and sleep=waits.append with waits = []. Assert that:

  1. the stop reason is "answered" and the answer is "Done.";

  2. the trace has two entries; the first has the call’s name in tool_calls and ["Lists keep order."] as its results;

  3. in client.requests[1]["messages"], item 2 is the assistant turn and item 3 is the tool message with tool_call_id "call_1" and your tool’s result;

  4. budget["used"] is 30;

  5. waits is empty.

Run it with python -m pytest -q tests/test_agent.py, then check your work.