Capstone · practice
Check ask's Sources Against What the Tools Returned
ask answers with the agent: the model chooses what to search for, reads the passages search_notes returns, and finishes with a JSON answer that names the labels of the passages it used. Before the program prints those labels after Sources:, it has to check them.
Chapter 8 had the same problem and a simple check. answer_from_notes chose the passages itself, put them in the prompt, and parse_cited_answer rejected any citation that was not one of their labels. In ask there is no such list up front. The model decided what to search, maybe more than once. The only record of what it was shown is the agent’s trace.
Where the labels are
Each entry of the trace that run_agent returns is a dictionary like this one, from the copy question’s first step:
{
"step": 1,
"tokens": 159,
"tool_calls": [{"id": "call_1_1", "name": "search_notes", "arguments": '{"query": "independent copy of a list"}'}],
"results": ["[03-lists#4] Because a list is changed in place, ...\n\n[01-variables#2] Assignment does not copy anything. ...\n\n[04-loops#7] Avoid adding or removing items ..."],
}
results holds the text each tool returned, in the same order as tool_calls. A search_notes result is Chapter 8’s format_sources of the best passages: each passage starts with its label in square brackets, and passages are separated by a blank line, "\n\n". So the labels the model may cite are the bracketed labels at the start of each block of each search_notes result, across every step of the run.
Three kinds of result have no labels, and the check has to leave them alone:
No note matches that search., whatsearch_notesreturns when no passage shares a word with the query;an error result, such as
Error: search_notes needs a query string, such as copying a list.;a
read_noteresult. A whole note might even start with[, but it is not a passage and has no label. The rule is that only a passage a search returned can be cited.
One more case: when the agent stops on a step that asked for tools, because of max_steps or a repeated call, that step’s calls never ran. Its entry has tool_calls but no results. Go through the results by position, with range(len(results)), and a call with no result is skipped without any special case.
trace_labels
trace_labels(trace) returns the labels in the order they came, repeats included:
Start with an empty list.
For each entry, take its
tool_callsand itsresults, and for eachindexinrange(len(results)), look at the call at the same index.If the call’s
nameis"search_notes", split its result on"\n\n". For each block that starts with[and contains], the label is the text between them:block[1:].split("]")[0].block[1:]drops the opening bracket, andsplit("]")[0]keeps what comes before the first closing one, so a later]inside the passage does not matter.Return the list.
A block such as No note matches that search. does not start with [, so it adds nothing.
check_answer
check_answer(text, labels) replaces Chapter 8’s parse_cited_answer for ask:
Validate the reply with
CitedAnswer.model_validate_json(text). If the text is not valid JSON, or not the right shape, this raises pydantic’sValidationError. Let it through: Lesson 3 catches it.For each citation, in order, if it is not in
labels, raiseValueError(f"The answer cites {citation}, which no search result in this run contained."). Only the first such citation is named.Otherwise return the
CitedAnswer.
An answer with an empty citations list passes: “the notes do not cover that” is a valid answer, and it cites nothing. A run that never searched has no labels, so any citation it makes is rejected.
The message says “in this run” on purpose. Lesson 3 sends it back to the model as feedback, so it should say exactly what was wrong.
Run it
When both functions are written, run python stages.py. Lesson 2’s block runs the agent on the copy question with ask’s tools and system message, prints the trace, the labels its one search returned, and what check_answer makes of the final reply:
Lesson 2: the agent's run for the copy question
Step 1 (159 tokens)
search_notes {"query": "independent copy of a list"} -> 763 characters
Step 2 (380 tokens)
answered
Labels its searches returned: ['03-lists#4', '01-variables#2', '04-loops#7']
check_answer gave: CitedAnswer(answer='Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.', citations=['03-lists#4'])
Lesson 3: ask_command
--- How do I make an independent copy of a list?
Run stopped at the next function to build: ask_command is built in Lesson 3.
The search returned three passages and the answer cites one of them, so it passes. Checking your work uses traces you have not seen: two searches in one step, searches across steps, a search that found nothing, a read_note result, an error, and a step whose calls never ran.
Why does trace_labels go through each entry with range(len(results)) rather than range(len(tool_calls))?
In one run, the only search returned No note matches that search., then the model called read_note for 03-lists and answered citing 03-lists#4. What does check_answer do?
Task
In study_assistant.py, write two functions.
trace_labels(trace): start with an empty list. For each entry of the trace, go through its tool_calls and results by position with range(len(results)). For a call whose name is "search_notes", split its result on "\n\n", and for each block that starts with [ and contains ], append block[1:].split("]")[0]. Return the list, in order, repeats included.
check_answer(text, labels): validate text with CitedAnswer.model_validate_json(text) and let a ValidationError through. For the first citation that is not in labels, raise ValueError(f"The answer cites {citation}, which no search result in this run contained."). Otherwise return the answer.
Run python stages.py to see the labels of the copy question’s run, then check your work.