Using the OpenAI Client
What the Client Hides
Lesson 1 built every client with max_retries=0 and promised an explanation. Here it is, as an experiment you can run: the same failing request through a client that gives up at once and through one that is allowed to try again.
A service that fails twice, then works
The scripted model can fail on purpose in a counted way. With the request header X-Scripted-Scenario: server-error:2, the first two requests that carry the header get status 500, and every request after that gets the normal reply. The scripted model keeps that count for your sandbox, so the third request succeeds whichever client sends it.
A count needs a starting point. The supplied reset_scripted_model() sends POST /v1/scripted/reset, which sets the count back to zero. Without it, running python client_calls.py a second time would find the count already past two, and the failures would not happen again.
The demonstration sends the header on single calls with extra_headers:
failing = {"X-Scripted-Scenario": "server-error:2"}
messages = [{"role": "user", "content": "What is a Python list?"}]
reset_scripted_model()
client.chat.completions.create(model=MODEL, messages=messages, extra_headers=failing)
reset_scripted_model()
retrying = make_client().with_options(max_retries=2)
response = retrying.chat.completions.create(model=MODEL, messages=messages, extra_headers=failing)
extra_headers= adds headers to that one call only, unlike the default_headers of Lesson 3’s scenario_client, which a client sends on every call. The first call goes through your make_client() client, with max_retries=0. The second goes through a copy of it made with with_options(max_retries=2), which is the client’s default setting. Each call is timed with time.monotonic(), as in Chapter 3.
Run it
Once you have finished Lesson 4, run python client_calls.py. This part of the output is the experiment:
What the client hides:
With max_retries=0: InternalServerError after 0.0 seconds
With max_retries=2: an answer after 1.7 seconds:
A list is an ordered, changeable collection of values. You write one with square brackets, like [1, 2, 3], and you can add, remove and change items after creating it.
The study chat on the client:
You: What does len() return?
Run stopped at the next function to build: chat_turn is built in Lesson 6.
The times can differ by a few tenths of a second on your run, because the client varies its waits a little.
What happened on the hidden attempts
With max_retries=0, the first request got its 500 and the client raised openai.InternalServerError at once. Your code saw the failure.
With max_retries=2, the same thing happened to the first request, and your code saw nothing. The client waited a little under half a second and sent the request again. The second attempt also got a 500. The client waited a little under a second and sent it a third time, and that one succeeded. Only then did create return, with the answer. Three requests reached the service; your code made one call and got one reply, and the only trace of the other two is the time it took.
The picture puts the two runs one above the other. Everything inside the bracket happened within a single call to create.

The client retries on its own when a request fails in a way that might go away: the connection fails or times out, or the service answers with a status such as 429 (too many requests) or 500 and above. It waits a little longer before each new attempt. With the default setting, one call in your code can become three requests.
Why this course turns it off
Hidden retries are convenient, and they are the right choice for many programs. For a program that calls a language model, they hide things you need to see.
Cost. Every request that reaches a model is paid for, by the tokens it reads and writes. A retry after a timeout, for example, can pay for a whole reply that never arrived. If the client retries on its own, your code cannot count those requests.
Budgets and time. Chapter 9 gives the study assistant a budget for tokens and time. A budget only works if the code that keeps it sees every request and every wait. More than a second of waiting inside one call is invisible to it.
Tests. A check that scripts one failure expects one call. With hidden retries, the same call can succeed, fail or take seconds, depending on a setting the test never sees.
Deciding what is worth retrying. A 500 might go away; a request the service rejects as malformed never will, and a refusal is the model’s answer, not a failure. Chapter 9 sorts failures into those that are worth another try and those that are not, and retries only the first kind, with waits your code chooses and can count.
So every client in this course is built with max_retries=0, and failures reach your code. Chapter 9 adds the retries back, where you can see them.
With max_retries=2 and the server-error:2 scenario, create returned an answer after about 1.7 seconds. What happened in that time?
Chapter 9 adds a budget that counts every request and every wait. Why does that need clients built with max_retries=0?
The demonstration calls reset_scripted_model() before each of the two calls that use server-error:2. What would change without those calls?