When Things Go Wrong · practice
Route Quiz and Agent Calls Through the Wrapper
call_model works, but nothing in the study assistant uses it yet. The quiz still goes through Chapter 5’s parse helper, and the agent’s loop still calls client.chat.completions.create itself, with a token count of its own. This lesson moves both onto call_model, so they get its retries, its time and length bounds, and one shared budget.
Part 1: the quiz
Chapter 5’s make_quiz asked the parse helper for a QuizSet. The helper did three things in one call: it sent a strict JSON schema as response_format, it checked the finish reason, and it validated the reply. The new make_quiz(note_text, *, client, budget, sleep) does the same three things through call_model:
request = {"model": MODEL, "messages": quiz_messages(note_text), "response_format": QUIZ_FORMAT}
response = call_model(request, client=client, sleep=sleep, max_attempts=MAX_ATTEMPTS, budget=budget)
QUIZ_FORMAT, supplied near the top of reliable.py, is exactly the response_format the parse helper sent for QuizSet in Chapter 5 Lesson 4, so a live model sees the same strict schema. call_model sends it through create with every other key. Then make_quiz validates the reply’s content with QuizSet.model_validate_json(content) and returns the QuizSet.
Moving off the helper gains three things:
A reply that fails validation still had its tokens counted, because
call_modelrecorded them beforemake_quizlooked at the text. The helper raised before your code saw any usage.A cut-off reply arrives as
ReplyCutOff, like every other call in the study assistant, instead of the helper’sLengthFinishReasonError.The retry can show the model its bad reply. On
ValidationError, build the retry request with Chapter 4’sretry_request(request, content, error), which copies the request and adds the bad reply as an assistant message and the feedback as a user message, and send it throughcall_modelonce more. If that reply is invalid too, let itsValidationErrorreach the caller: one corrective retry is the rule from Chapter 4.
One case is not retried. When the reply’s content is None, for a refusal or a reply with no text, raise ValueError("The model sent no quiz text.") straight away. That is Lesson 1’s “give up and report” group. Retrying would also send an assistant message with no content and no tool calls, which services reject. The second reply needs no such check: QuizSet.model_validate_json(None) raises ValidationError itself.
Part 2: the agent
run_agent in reliable.py is Chapter 7’s finished loop. It changes in four places, and the rest stays as it is:
The signature.
max_tokens_totalgoes, andbudgetandsleepcome in:run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep). This is the final signature, the one Chapters 10 and 11 call.Each step builds a request and sends it through
call_modelwithMAX_ATTEMPTS:request = {"model": MODEL, "messages": messages, "tools": descriptions}call_onceaddsmax_tokens, and a429in the middle of a run is now retried withsleepinstead of ending it.call_modelcan raiseBudgetExceededbefore a step is sent. Catch it, and only it, around the call, and return{"answer": "", "stop_reason": "budget", "trace": trace, "messages": messages}. The refused step never ran, so it is not in the trace, andmessagesis what would have been sent.Remove the old
usedcount.stop_reasongetsbudget["used"]andbudget["limit"]where it had the old total and limit, so a reply that brings the shared budget to its limit still stops the loop with"budget".
ReplyCutOff and every model error that call_model raises are not caught: they reach the caller. descriptions still comes from describe_tools(tools, AGENT_DESCRIPTIONS), and AGENT_DESCRIPTIONS in reliable.py now also describes Chapter 8’s search_notes, which the capstone’s agent uses.
Chapter 7 Lesson 4 promised this change. Its count belonged to one run and could only notice the limit after a reply. Now one budget covers every call: a quiz, an agent run and a chat can share one dictionary, and each step is refused before it is sent if it might not fit.
Run it
When both are written, run python chapter9_demo.py. The demonstration makes the quiz for notes/03-lists.md and the quiz for give me broken json with one budget, then runs the agent on What does the note on errors say about finally? with READ_TOOLS and system=TOOLS_SYSTEM, first with {"limit": 20000, "used": 0} and then with {"limit": 1200, "used": 0}:
A quiz through call_model:
1. For scores = [90, 72, 85], what is scores[-1]?
a) 90
b) 72
c) 85
2. What does scores.append(60) return?
a) The new list
b) None
c) 60
3. Which line makes an independent copy of scores?
a) backup = scores.copy()
b) backup = scores
c) backup = scores.append(0)
Answers: 1c, 2b, 3a
A quiz whose first reply is invalid:
1. What does len() return?
a) The number of items
b) The last item
c) The first item
Answers: 1a
Budget after both quizzes: {'limit': 20000, 'used': 808}
The agent with a budget of 20000:
stop_reason: answered, budget: {'limit': 20000, 'used': 591}
Step 1 (48 tokens)
list_notes {} -> 140 characters
Step 2 (89 tokens)
read_note {"note_id": "07-errors"} -> 1325 characters
Step 3 (454 tokens)
answered
The agent with a budget of 1200:
stop_reason: budget, budget: {'limit': 1200, 'used': 137}
Step 1 (48 tokens)
list_notes {} -> 140 characters
Step 2 (89 tokens)
read_note {"note_id": "07-errors"} -> 1325 characters
The study chat under rate-limit:1:
Run stopped at the next function to build: chat_turn is built in Lesson 9.
The second quiz took two calls, the invalid reply and the corrected one, and the budget counts all three quiz calls. With 20,000 tokens the agent answers after three steps, as in Chapter 7. With 1,200 it stops before step 3. The estimates were 844.25 for step 1 and 879.25 for step 2, both within the budget. After step 2, 137 tokens were used and 1,063 were left, and step 3’s request carries the whole errors note, so its estimate was 1,210.5. The step was refused before it was sent.
The quiz reply comes back with content None. What does make_quiz do?
With a budget of 1,200, the agent stopped with "budget" before step 3. What happened to step 3's request?
Task
In reliable.py, change two functions.
make_quiz(note_text, *, client, budget, sleep): send request = {"model": MODEL, "messages": quiz_messages(note_text), "response_format": QUIZ_FORMAT} through call_model with max_attempts=MAX_ATTEMPTS. If the reply’s content is None, raise ValueError("The model sent no quiz text."). Otherwise return QuizSet.model_validate_json(content). On ValidationError, send retry_request(request, content, error) through call_model once more and return its validated quiz; a second ValidationError reaches the caller.
run_agent: change the signature to run_agent(question, *, client, tools, system, approve, max_steps, budget, sleep). Send each step as {"model": MODEL, "messages": messages, "tools": descriptions} through call_model with max_attempts=MAX_ATTEMPTS. If that raises BudgetExceeded, return {"answer": "", "stop_reason": "budget", "trace": trace, "messages": messages}; catch nothing else. Pass budget["used"] and budget["limit"] to stop_reason instead of the old total and limit. Keep everything else from Chapter 7.
Run python chapter9_demo.py to see the quizzes and the agent under one budget, then check your work.