When Things Go Wrong · practice
Bound Every Call
Lesson 1’s demonstration showed two ways a call can run on without a useful end: a reply that takes too long, and a reply that goes on longer than you want to pay for. A program that calls a model should put a bound on both before it sends anything. This lesson writes call_once, a single attempt with both bounds, which every later function in the chapter uses.
A bound on time
Chapter 5 gave every client one timeout number, CLIENT_TIMEOUT = 10.0 in client_calls.py. The client uses it for each wait: opening the connection, sending the request, and every read from the server. When a wait runs longer, the client gives up and raises openai.APITimeoutError, which is a kind of openai.APIConnectionError.
That bound is already on every client the course builds, so call_once does not set it again. A caller who needs a different bound for one job makes a copy of the client with with_options, as this lesson’s demonstration does:
impatient = scenario_client("slow:1500").with_options(timeout=0.5)
slow:1500 makes the scripted model wait 1.5 seconds before it answers. With a timeout of 0.5 seconds, the client stops waiting after about half a second.
A bound on length
max_tokens caps how many tokens the reply may have (Chapter 2 Lesson 4). Without it, a provider decides how long a reply may run, and you pay for every token of it. From now on every request this chapter sends carries max_tokens: the request’s own value if it has one, or DEFAULT_MAX_TOKENS, which is 800.
The scripted model accepts max_tokens and ignores it, so you will not see a difference in its replies. pl-live lowers any value above 1,000 to 1,000, and it spends part of that limit on reasoning before it writes the visible answer. 800 leaves room for a quiz of several questions plus that reasoning.
A reply that reaches the limit stops there, with finish_reason "length". Lesson 5 deals with that.
A request as a dictionary
From this lesson on, a request is one dictionary that describes a call. It always has model and messages, and it can have tools, response_format and max_tokens:
question = {"model": MODEL, "messages": [{"role": "user", "content": "What is a Python list?"}]}
A dictionary is easy to pass around, to retry unchanged, and to check before it is sent, which Lesson 6 does. call_once(request, *, client) turns it into one create call:
client.chat.completions.create(
model=request["model"],
messages=request["messages"],
tools=request.get("tools", openai.omit),
response_format=request.get("response_format", openai.omit),
max_tokens=request.get("max_tokens", DEFAULT_MAX_TOKENS),
)
openai.omit is a special value from the openai package that means “leave this key out of the request”. A request for a plain answer has no tools key, so request.get("tools", openai.omit) gives openai.omit, and the body the client sends has no tools at all. Passing None instead would send "tools": null, which is not the same as leaving it out: the scripted model accepts it, but other services can reject it.
Each key is passed by name, so a reader of call_once sees exactly what a request can send. call_once returns the response that create returned, and catches nothing: an error from create reaches the caller, and Lesson 7 decides what to do with it.
Run it
When call_once is written, run python chapter9_demo.py:
One bounded call:
A list is an ordered, changeable collection of values. You write one with square brackets, like [1, 2, 3], and you can add, remove and change items after creating it.
The same call through a client that waits at most 0.5 seconds for a slow reply:
APITimeoutError: Request timed out.
Worth another attempt?
Run stopped at the next function to build: should_retry is built in Lesson 3.
The first call is an ordinary answer. The second shows the time bound: the demonstration catches openai.APITimeoutError and prints its text, Request timed out., after about half a second, a second before the scripted model would have answered.
MODEL_SERVICE.md says how to see the body that was really sent: last_request() in client_calls.py returns it. To see it yourself, open the python and type:
>>> from reliable import call_once
>>> from client_calls import MODEL, make_client, last_request
>>> response = call_once({"model": MODEL, "messages": [{"role": "user", "content": "What is a Python list?"}]}, client=make_client())
>>> "tools" in last_request()
False
A request has no tools key. Why does call_once pass request.get("tools", openai.omit) rather than request.get("tools")?
One job should wait at most half a second for a reply. Where does that bound go?
Task
In reliable.py, write call_once(request, *, client). Make exactly one client.chat.completions.create call with these keyword arguments, each read from the request:
model=request["model"]andmessages=request["messages"]tools=request.get("tools", openai.omit)andresponse_format=request.get("response_format", openai.omit), so a request without those keys sends neithermax_tokens=request.get("max_tokens", DEFAULT_MAX_TOKENS), using the constant by name
Return the response. Do not change the request dictionary, and catch no errors: they must reach the caller. Run python chapter9_demo.py to see a bounded call and a timeout, then check your work.