0%

When Things Go Wrong · practice

Bound Every Call

Lesson 1’s demonstration showed two ways a call can run on without a useful end: a reply that takes too long, and a reply that goes on longer than you want to pay for. A program that calls a model should put a bound on both before it sends anything. This lesson writes call_once, a single attempt with both bounds, which every later function in the chapter uses.

A bound on time

Chapter 5 gave every client one timeout number, CLIENT_TIMEOUT = 10.0 in client_calls.py. The client uses it for each wait: opening the connection, sending the request, and every read from the server. When a wait runs longer, the client gives up and raises openai.APITimeoutError, which is a kind of openai.APIConnectionError.

That bound is already on every client the course builds, so call_once does not set it again. A caller who needs a different bound for one job makes a copy of the client with with_options, as this lesson’s demonstration does:

impatient = scenario_client("slow:1500").with_options(timeout=0.5)

slow:1500 makes the scripted model wait 1.5 seconds before it answers. With a timeout of 0.5 seconds, the client stops waiting after about half a second.

A bound on length

max_tokens caps how many tokens the reply may have (Chapter 2 Lesson 4). Without it, a provider decides how long a reply may run, and you pay for every token of it. From now on every request this chapter sends carries max_tokens: the request’s own value if it has one, or DEFAULT_MAX_TOKENS, which is 800.

The scripted model accepts max_tokens and ignores it, so you will not see a difference in its replies. pl-live lowers any value above 1,000 to 1,000, and it spends part of that limit on reasoning before it writes the visible answer. 800 leaves room for a quiz of several questions plus that reasoning.

A reply that reaches the limit stops there, with finish_reason "length". Lesson 5 deals with that.

A request as a dictionary

From this lesson on, a request is one dictionary that describes a call. It always has model and messages, and it can have tools, response_format and max_tokens:

question = {"model": MODEL, "messages": [{"role": "user", "content": "What is a Python list?"}]}

A dictionary is easy to pass around, to retry unchanged, and to check before it is sent, which Lesson 6 does. call_once(request, *, client) turns it into one create call:

client.chat.completions.create(
    model=request["model"],
    messages=request["messages"],
    tools=request.get("tools", openai.omit),
    response_format=request.get("response_format", openai.omit),
    max_tokens=request.get("max_tokens", DEFAULT_MAX_TOKENS),
)

openai.omit is a special value from the openai package that means “leave this key out of the request”. A request for a plain answer has no tools key, so request.get("tools", openai.omit) gives openai.omit, and the body the client sends has no tools at all. Passing None instead would send "tools": null, which is not the same as leaving it out: the scripted model accepts it, but other services can reject it.

Each key is passed by name, so a reader of call_once sees exactly what a request can send. call_once returns the response that create returned, and catches nothing: an error from create reaches the caller, and Lesson 7 decides what to do with it.

Run it

When call_once is written, run python chapter9_demo.py:

One bounded call:
  A list is an ordered, changeable collection of values. You write one with square brackets, like [1, 2, 3], and you can add, remove and change items after creating it.
The same call through a client that waits at most 0.5 seconds for a slow reply:
  APITimeoutError: Request timed out.
Worth another attempt?
Run stopped at the next function to build: should_retry is built in Lesson 3.

The first call is an ordinary answer. The second shows the time bound: the demonstration catches openai.APITimeoutError and prints its text, Request timed out., after about half a second, a second before the scripted model would have answered.

MODEL_SERVICE.md says how to see the body that was really sent: last_request() in client_calls.py returns it. To see it yourself, open the , start python and type:

>>> from reliable import call_once
>>> from client_calls import MODEL, make_client, last_request
>>> response = call_once({"model": MODEL, "messages": [{"role": "user", "content": "What is a Python list?"}]}, client=make_client())
>>> "tools" in last_request()
False

A request has no tools key. Why does call_once pass request.get("tools", openai.omit) rather than request.get("tools")?

One job should wait at most half a second for a reply. Where does that bound go?

Task

In reliable.py, write call_once(request, *, client). Make exactly one client.chat.completions.create call with these keyword arguments, each read from the request:

  • model=request["model"] and messages=request["messages"]

  • tools=request.get("tools", openai.omit) and response_format=request.get("response_format", openai.omit), so a request without those keys sends neither

  • max_tokens=request.get("max_tokens", DEFAULT_MAX_TOKENS), using the constant by name

Return the response. Do not change the request dictionary, and catch no errors: they must reach the caller. Run python chapter9_demo.py to see a bounded call and a timeout, then check your work.