Chapter 9
When Things Go Wrong
What Can Fail in a Model Call
Every function the study assistant has so far calls the model and hopes. If the service is busy, slow or unhappy with the request, the error travels up through your code and stops the program. That was the right choice while you were learning what a call looks like, and Chapter 5 turned off the client’s own retries so you would see every failure. This chapter decides what to do with each one.
A model call can fail in more ways than most function calls. The service can be overloaded, a reply can take too long, the request can be wrong, the reply can stop halfway, and a program that loops, like Chapter 7’s agent, can spend more tokens than anyone meant it to. Some of these go away if you wait and try again. Others never will, and trying again only costs time and money.
Your project for this chapter
Open reliable.py in the project editor. Below time, openai and pydantic’s ValidationError, it imports from the course files you built in earlier chapters. agent.py is Chapter 7’s finished agent, retrieval.py and answer_models.py are Chapter 8’s finished retrieval, and the files they import are supplied next to them as before. You do not need to change any of them. Checking your work uses its own copies, so an edit there cannot change a grade, but it can stop the demonstration from working.
errors.py now holds three supplied error classes: Chapter 3’s StreamBroken, and two new ones you meet in Lessons 5 and 6, ReplyCutOff and BudgetExceeded.
At the top of reliable.py are four numbers the chapter uses: DEFAULT_MAX_TOKENS = 800, BASE_WAIT = 1, MAX_WAIT = 60 and MAX_ATTEMPTS = 3. Below them are QUIZ_FORMAT, which Lesson 8 uses, then the functions. Two are finished code: stream_once, which you read in Lesson 9, and sleep_and_say, which the demonstration uses from Lesson 7. Ten are unfinished, and one, run_agent, starts as Chapter 7’s finished loop, which Lesson 8 changes.
In this chapter you run python chapter9_demo.py, not reliable.py. chapter9_demo.py is a supplied demonstration that imports your functions from reliable.py and calls each of them as soon as it exists, using finished helpers of its own: show_failures, retry_demo and budget_demo. You do not need to change it. Your work goes in reliable.py, and checking your work does not run the demonstration.
Seven ways to fail
In the Shell, run python chapter9_demo.py now, before you change anything. The demonstration calls show_failures(), which sends What is a Python list? seven times. Five times it uses a client from scenario_client, which makes the scripted model fail on purpose in a chosen way (MODEL_SERVICE.md lists every scenario). Once it asks for a model that does not exist, and once it sends a request with no messages at all. For each, it prints the error’s class name and, when the error came with an HTTP status, its status_code:
One question under each failure:
rate-limit: RateLimitError 429
server-error: InternalServerError 500
gateway-timeout:300: InternalServerError 504
slow:1500, timeout=0.5: APITimeoutError
length: finish_reason length
model pl-unknown: NotFoundError 404
no messages: BadRequestError 400
One bounded call:
Run stopped at the next function to build: call_once is built in Lesson 2.
Here is what each line means:
RateLimitError 429is “too many requests”. The service is fine but will not take more from you right now. Its reply carries aRetry-Afterheader that says how many seconds to wait, here1.InternalServerError 500means something went wrong on the service’s side. Your request may have been fine.InternalServerError 504is a gateway timeout: a server in front of the model gave up waiting for it. Theopenaiclient raisesInternalServerErrorfor every status from 500 up, so502and503arrive the same way.APITimeoutErrorcame from your own side. The client used for that line waits at most 0.5 seconds, andslow:1500takes 1.5 seconds to answer, so the client gave up. There is no status, because no reply arrived.finish_reason lengthis not an error at all. The call succeeded, but the reply stopped at its token limit, so it is incomplete. Nothing was raised, which is what makes it easy to miss.NotFoundError 404means the service has no model calledpl-unknown.BadRequestError 400means the service rejected the request itself, here becausemessageswas empty.
Two more failures do not appear in this output. A 403, PermissionDeniedError, is what pl-live answers when your account has no live access. And a connection can fail before any reply, which raises APIConnectionError; APITimeoutError is a kind of APIConnectionError.
Three groups
What matters is not the class name but what your program should do next. Every failure falls into one of three groups.
Retry after a wait
The request was fine and the problem is likely to pass: a 429, any 5xx, a timeout, or a connection that failed before any reply arrived. Waiting a moment and sending the same request again often works.
Fix the request
The same request will fail the same way every time: a 400, a 404, or a cut-off reply. A cut-off reply needs a higher max_tokens or a shorter request. Retrying a 400 or a 404 only fails again, and costs time and a request against your rate limit. Retrying a cut-off reply is worse: the model writes it again, and you pay for those tokens again.
Give up and report
Some failures are final for this program, so the right move is to stop and say what happened:
a
403: no live access, which waiting will not change;a refusal, where the model declined to answer, which is its answer and not a failure of the call;
output that is still invalid after the one corrective retry of Chapter 4;
a
429whoseRetry-Afterasks you to come back much later thanMAX_WAIT, which is 60 seconds.
That last one needs a closer look, because the same status can belong to two groups. pl-live has two limits. When you pass the per-minute limit, its Retry-After is the number of seconds until the next minute starts, between 1 and 60, so waiting is worth it and the program retries. When you pass the daily limit, its Retry-After is the number of seconds until 00:00 UTC, which can be many hours. No program should sleep that long in the middle of a call, so it gives up and reports.
A stream that breaks halfway raises APIConnectionError too. It looks like the first group, but if part of the reply is already on the screen, a retry would print a second, different answer after the first. Lesson 9 explains why a stream is retried only before any text is shown.
The plan for this chapter
Lesson 2 bounds every call, in time and in the length of the reply. Lesson 3 decides which errors are worth another attempt, and Lesson 4 decides how long to wait before it. Lesson 5 catches a cut-off reply, and Lesson 6 gives the program a token budget of its own that refuses a call before it is sent. The project, Lesson 7, puts these pieces into one function, call_model. Lessons 8 and 9 route the quiz, the agent and the streaming chat through it, so the whole study assistant follows one set of rules.
A program meets four failures: a 429 with Retry-After: 2, a 400 for a misspelled field, a 503, and a 429 with Retry-After: 30000. Which ones should it retry after a wait?
Why is retrying a 400 Bad Request pointless?
A streamed reply has printed two sentences when the connection drops and the client raises APIConnectionError. Why does the course not simply retry it?