0%

Capstone · practice

Build ask

You have every part of ask: Chapter 9’s run_agent runs the agent within a budget, call_model sends one request under the same rules, and Lesson 2’s trace_labels and check_answer check an answer against what the run’s searches returned. This lesson joins them into the function the ask command calls, with the one corrective retry Chapter 4 taught for invalid output.

You write three functions: two small ones, then ask_command.

feedback_message

When an answer is invalid, the model is told what was wrong, starting with INVALID_PREFIX from prompts.py, Your last reply was invalid:, as in Chapters 4 and 5. feedback_message(error) returns that user message:

  • for a pydantic ValidationError, the detail is Chapter 4’s short lines, "; ".join(error_lines(error)), such as (whole reply): Invalid JSON: expected ident at line 1 column 2. str(error) would also hold pydantic’s heading and a link to its documentation, which tell the model nothing useful;

  • for any other error, such as the ValueError from check_answer, the detail is str(error).

Use isinstance(error, ValidationError) to tell them apart, and return {"role": "user", "content": f"{INVALID_PREFIX} {detail}"}.

format_answer

format_answer(answer) turns a CitedAnswer into what ask prints: the answer’s text, a blank line, then Sources: and the citations joined with ", ". With no citations the line is Sources: none, so the person can see that nothing was cited, rather than wondering about an empty line. Return one string with no newline at the end; print adds one.

Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.

Sources: 03-lists#4

ask_command

ask_command(question, *, client, budget, sleep) returns that text, or raises. It never prints and never decides an exit code: Lesson 6 does that for every command in one place.

First run the agent with ask’s settings:

result = run_agent(
    question,
    client=client,
    tools=ASK_TOOLS,
    system=ASK_SYSTEM,
    approve=refuse_all,
    max_steps=ASK_MAX_STEPS,
    budget=budget,
    sleep=sleep,
)

ASK_TOOLS offers search_notes and read_note, and nothing that changes a file, so save_flashcard is never on offer. ASK_SYSTEM, in prompts.py, tells the model to search, to treat passage text as material and not instructions, and to reply with JSON that cites labels without brackets. refuse_all is there because run_agent needs an approve function; with these tools it is never asked.

Then look at result["stop_reason"]:

  • "budget": the run’s budget could not cover the next step. Raise BudgetExceeded(f"The budget ran out before agent step {len(result['trace']) + 1}."). The trace holds the steps that ran, so the one that did not is the next number.

  • "answered": go on.

  • anything else, "stuck" or "max_steps": raise ValueError(f"The agent stopped without an answer ({reason}).").

Then check the answer. Compute labels = trace_labels(result["trace"]), and inside try return format_answer(check_answer(result["answer"], labels)). When that works, ask_command is done.

One corrective retry

If check_answer raises ValueError, the answer cited a passage no search returned. A pydantic ValidationError is a kind of ValueError (Chapter 4), so the same except ValueError as error: also catches a reply that is not valid JSON. Either way the model gets one more chance:

messages = result["messages"] + [feedback_message(error)]
request = {"model": MODEL, "messages": messages, "tools": describe_tools(ASK_TOOLS, AGENT_DESCRIPTIONS)}
response = call_model(request, client=client, sleep=sleep, max_attempts=MAX_ATTEMPTS, budget=budget)

Build messages inside the except block; the name error only exists there. The rest can come after it.

result["messages"] is the whole run: the system message, the question, each tool request with its results, and the model’s last reply. So the model sees its own bad answer and the feedback, as in Chapter 4. The request offers the same two tools, because the conversation contains tool messages and the model should see the same offer it had. It goes through call_model with the same budget and sleep, so a rate limit is waited out, max_tokens is set, and the retry is refused before it is sent if the run’s budget cannot cover it. One budget covers the whole run; the retry does not get a fresh one.

Then check the reply:

  • If read_tool_calls(response) is not empty, the model asked to search again instead of answering. Raise ValueError("The corrected answer asked for tools instead of answering."). The run’s steps are spent, and its labels are fixed: a search now would mean running the agent again.

  • Otherwise take content = response.choices[0].message.content and return format_answer(check_answer(content or "", labels)). content or "" makes sure check_answer always gets a string; a reply with no text then fails validation like any other bad reply. If this answer is invalid too, its error reaches the caller: one retry is the rule.

BudgetExceeded from the retry, and every model error from run_agent or call_model, also reach the caller. ask_command catches only the ValueError from the first check.

Run it

When the three functions are written, run python stages.py. Lesson 3’s block asks the copy question and the France question through the course model, each with a fresh budget of 20,000, then shows a retry:

Lesson 3: ask_command
--- How do I make an independent copy of a list?
Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.

Sources: 03-lists#4
(tokens used: 539)
--- What is the capital of France?
The course notes don't cover that, so I can't answer it from them.

Sources: none
(tokens used: 329)
Lesson 3: the retry, scripted with FakeClient
Requests sent: 3
The last message of the third: Your last reply was invalid: The answer cites 05-functions#6, which no search result in this run contained.
The default list is created once and shared by every call.

Sources: 05-functions#5
Lesson 4: quiz_command
--- 03-lists, 2 questions
Run stopped at the next function to build: quiz_command is built in Lesson 4.

The copy question used two steps, 159 and 380 tokens. For the France question, the search returned No note matches that search. and the model said so with no citations, which prints Sources: none.

The retry is not shown with the course model, because the scripted model cannot be made to correct itself: it skips a message that starts with Your last reply was invalid: and answers the question again, with the same reply. A live model is where a retry happens for real. So stages.py scripts it with Chapter 10’s FakeClient: a search_notes call with {"query": "default argument list"}, which runs for real on the shipped notes and returns 05-functions#4, 05-functions#5 and 01-variables#2; then an answer citing 05-functions#5 and 05-functions#6; then a corrected answer citing only 05-functions#5. The third request ends with the feedback, and the corrected answer is what ask prints.

After the feedback, the model's corrected reply asks for search_notes again instead of answering. What does ask_command do?

Which budget does the corrective retry draw on?

Task

In study_assistant.py, write three functions.

feedback_message(error): return {"role": "user", "content": f"{INVALID_PREFIX} {detail}"}, where detail is "; ".join(error_lines(error)) for a ValidationError and str(error) for any other error.

format_answer(answer): return the answer’s text, a blank line, and Sources: followed by the citations joined with ", ", or none when there are none. No newline at the end.

ask_command(question, *, client, budget, sleep):

  1. result = run_agent(question, client=client, tools=ASK_TOOLS, system=ASK_SYSTEM, approve=refuse_all, max_steps=ASK_MAX_STEPS, budget=budget, sleep=sleep).

  2. For stop reason "budget", raise BudgetExceeded(f"The budget ran out before agent step {len(result['trace']) + 1}."). For any other stop reason except "answered", raise ValueError(f"The agent stopped without an answer ({reason}).").

  3. labels = trace_labels(result["trace"]). Inside try, return format_answer(check_answer(result["answer"], labels)).

  4. On ValueError, send {"model": MODEL, "messages": result["messages"] + [feedback_message(error)], "tools": describe_tools(ASK_TOOLS, AGENT_DESCRIPTIONS)} through call_model once, with max_attempts=MAX_ATTEMPTS and the same budget and sleep.

  5. If read_tool_calls of that response is not empty, raise ValueError("The corrected answer asked for tools instead of answering."). Otherwise return format_answer(check_answer(content or "", labels)), where content is the reply’s content, and let its errors through.

Run python stages.py to see ask answer two questions and a retry scripted with FakeClient, then check your work.