When Things Go Wrong · capstone
Capstone project: Project: A Call Wrapper That Fails Well
Lessons 2 to 6 built seven small functions, each with one job: call_once sends one bounded attempt, should_retry and wait_seconds decide whether and how long to wait, check_finished catches a cut-off reply, and estimate_tokens, check_budget and record_usage keep the budget. call_model calls six of them; check_budget calls estimate_tokens. In this project you put them together in call_model, the one function the rest of the study assistant will use to call the model.
What call_model promises
A function that other code depends on should say exactly what it can do. call_model has six outcomes, and its docstring lists them:
def call_model(request, *, client, sleep, max_attempts, budget):
"""Send request with retries, within budget, and return a complete response.
Returns the response when an attempt succeeds and the reply is complete.
Raises BudgetExceeded before any attempt that the budget cannot cover.
Raises ReplyCutOff when the reply stopped at max_tokens, after counting its tokens.
Raises an error that is not worth retrying at once.
Raises a retryable error at once when its wait would be longer than MAX_WAIT.
Raises the last error when all max_attempts attempts failed.
"""
A caller can rely on that list. Lesson 8’s quiz, for example, knows that a response from call_model is always complete, and that the budget has already been charged for it.
The order of the steps
For each attempt from 1 to max_attempts, which is at least 1:
check_budget(request, budget). The check runs before every attempt, not only the first. A wait between attempts can take seconds, and other code that shares the budget can spend it in the meantime.Send the attempt with
call_once(request, client=client), insidetry.On
openai.APIError:if
should_retry(error)is false, or this was the last attempt, raise the error;otherwise compute
wait = wait_seconds(error, attempt); ifwaitis longer thanMAX_WAIT, raise the error;otherwise call
sleep(wait)and go on to the next attempt.
On success,
record_usage(response, budget), thencheck_finished(response), then return the response.
The picture draws the same steps. Every attempt starts at check_budget, the only way back to it is through sleep(wait), and every other path ends the call.

A for loop over range(1, max_attempts + 1) gives attempt the values 1, 2, 3 for three attempts. After sleep, continue goes straight to the next attempt, so the success steps below the try run only when an attempt worked.
raise error raises the error you caught, by its name, so the caller gets the original RateLimitError or InternalServerError with its status and response. The except catches openai.APIError and nothing wider: a bug in your own code, such as a KeyError, is not a model failure and must not be retried.
Three details matter:
sleepis never called after the last attempt. Waiting before giving up wastes the wait.Usage is recorded once, for the reply that arrived. A failed attempt has no usage to record.
record_usagecomes beforecheck_finished. A cut-off reply was still paid for, so it is counted beforeReplyCutOffis raised.
sleep is a parameter, as Lesson 4 said. A real run passes time.sleep; the checks pass a function that writes each wait down and returns at once. The demonstration passes the supplied sleep_and_say from reliable.py, which prints each wait and then calls time.sleep.
Run it
When call_model is written, run python chapter9_demo.py. The demonstration calls reset_scripted_model() and then calls call_model four times with max_attempts=3. All but the third send What is a Python list?:
call_model under rate-limit:2:
(waiting 1.0 s)
(waiting 1.0 s)
A list is an ordered, changeable collection of values. You write one with square brackets, like [1, 2, 3], and you can add, remove and change items after creating it.
Budget: {'limit': 20000, 'used': 48}
call_model under server-error:5, three attempts:
(waiting 1 s)
(waiting 2 s)
InternalServerError: 500
call_model under length:
ReplyCutOff. The partial text was 'A list is an ordered collection that can'
Budget: {'limit': 20000, 'used': 18}
call_model with a budget of 500:
BudgetExceeded: Not enough budget for this call: 500 of 500 tokens are left.
A quiz through call_model:
Run stopped at the next function to build: make_quiz is built in Lesson 8.
Each case is one of the documented outcomes:
Under
rate-limit:2, two attempts failed with a429, each followed by the one-second wait itsRetry-Afterasked for, and the third succeeded. The budget counts only the reply: 48 tokens.Under
server-error:5, every attempt failed. The waits doubled, 1 then 2, and after the third attemptcall_modelraised the last error, without a third wait.Under
length, withExplain Python lists in detail., the reply was cut off. Its 18 tokens are in the budget, and thenReplyCutOffwas raised.With a budget of 500, the first estimate was already 805.5, so
BudgetExceededwas raised and no request was sent.
The waits add up to about five seconds of this demonstration.
In call_model, why does record_usage come before check_finished?
call_model gets max_attempts=3, and every attempt fails with a 500. How many times does it call sleep?
Task
In reliable.py, write call_model(request, *, client, sleep, max_attempts, budget). Give it a docstring that lists its outcomes. For each attempt from 1 to max_attempts:
Call
check_budget(request, budget).Send the attempt with
call_once(request, client=client)insidetry.On
openai.APIError: raise the error ifshould_retry(error)is false or this was the last attempt. Otherwise computewait_seconds(error, attempt), raise the error if the wait is longer thanMAX_WAIT, and otherwise callsleep(wait)and try again.On success, call
record_usage(response, budget), thencheck_finished(response), then return the response.
Never call sleep after the last attempt, and use the sleep you were given, not time.sleep. Raise the caught error itself with raise error. Run python chapter9_demo.py to see each outcome, then check your work.