0%

When Things Go Wrong · capstone

Capstone project: Project: A Call Wrapper That Fails Well

Lessons 2 to 6 built seven small functions, each with one job: call_once sends one bounded attempt, should_retry and wait_seconds decide whether and how long to wait, check_finished catches a cut-off reply, and estimate_tokens, check_budget and record_usage keep the budget. call_model calls six of them; check_budget calls estimate_tokens. In this project you put them together in call_model, the one function the rest of the study assistant will use to call the model.

What call_model promises

A function that other code depends on should say exactly what it can do. call_model has six outcomes, and its docstring lists them:

def call_model(request, *, client, sleep, max_attempts, budget):
    """Send request with retries, within budget, and return a complete response.

    Returns the response when an attempt succeeds and the reply is complete.
    Raises BudgetExceeded before any attempt that the budget cannot cover.
    Raises ReplyCutOff when the reply stopped at max_tokens, after counting its tokens.
    Raises an error that is not worth retrying at once.
    Raises a retryable error at once when its wait would be longer than MAX_WAIT.
    Raises the last error when all max_attempts attempts failed.
    """

A caller can rely on that list. Lesson 8’s quiz, for example, knows that a response from call_model is always complete, and that the budget has already been charged for it.

The order of the steps

For each attempt from 1 to max_attempts, which is at least 1:

  1. check_budget(request, budget). The check runs before every attempt, not only the first. A wait between attempts can take seconds, and other code that shares the budget can spend it in the meantime.

  2. Send the attempt with call_once(request, client=client), inside try.

  3. On openai.APIError:

    • if should_retry(error) is false, or this was the last attempt, raise the error;

    • otherwise compute wait = wait_seconds(error, attempt); if wait is longer than MAX_WAIT, raise the error;

    • otherwise call sleep(wait) and go on to the next attempt.

  4. On success, record_usage(response, budget), then check_finished(response), then return the response.

The picture draws the same steps. Every attempt starts at check_budget, the only way back to it is through sleep(wait), and every other path ends the call.

Each call_model attempt checks the budget, sends one bounded attempt, and then returns, raises, or waits with wait_seconds and sleep before the next attempt.

A for loop over range(1, max_attempts + 1) gives attempt the values 1, 2, 3 for three attempts. After sleep, continue goes straight to the next attempt, so the success steps below the try run only when an attempt worked.

raise error raises the error you caught, by its name, so the caller gets the original RateLimitError or InternalServerError with its status and response. The except catches openai.APIError and nothing wider: a bug in your own code, such as a KeyError, is not a model failure and must not be retried.

Three details matter:

  • sleep is never called after the last attempt. Waiting before giving up wastes the wait.

  • Usage is recorded once, for the reply that arrived. A failed attempt has no usage to record.

  • record_usage comes before check_finished. A cut-off reply was still paid for, so it is counted before ReplyCutOff is raised.

sleep is a parameter, as Lesson 4 said. A real run passes time.sleep; the checks pass a function that writes each wait down and returns at once. The demonstration passes the supplied sleep_and_say from reliable.py, which prints each wait and then calls time.sleep.

Run it

When call_model is written, run python chapter9_demo.py. The demonstration calls reset_scripted_model() and then calls call_model four times with max_attempts=3. All but the third send What is a Python list?:

call_model under rate-limit:2:
  (waiting 1.0 s)
  (waiting 1.0 s)
  A list is an ordered, changeable collection of values. You write one with square brackets, like [1, 2, 3], and you can add, remove and change items after creating it.
  Budget: {'limit': 20000, 'used': 48}
call_model under server-error:5, three attempts:
  (waiting 1 s)
  (waiting 2 s)
  InternalServerError: 500
call_model under length:
  ReplyCutOff. The partial text was 'A list is an ordered collection that can'
  Budget: {'limit': 20000, 'used': 18}
call_model with a budget of 500:
  BudgetExceeded: Not enough budget for this call: 500 of 500 tokens are left.
A quiz through call_model:
Run stopped at the next function to build: make_quiz is built in Lesson 8.

Each case is one of the documented outcomes:

  • Under rate-limit:2, two attempts failed with a 429, each followed by the one-second wait its Retry-After asked for, and the third succeeded. The budget counts only the reply: 48 tokens.

  • Under server-error:5, every attempt failed. The waits doubled, 1 then 2, and after the third attempt call_model raised the last error, without a third wait.

  • Under length, with Explain Python lists in detail., the reply was cut off. Its 18 tokens are in the budget, and then ReplyCutOff was raised.

  • With a budget of 500, the first estimate was already 805.5, so BudgetExceeded was raised and no request was sent.

The waits add up to about five seconds of this demonstration.

In call_model, why does record_usage come before check_finished?

call_model gets max_attempts=3, and every attempt fails with a 500. How many times does it call sleep?

Task

In reliable.py, write call_model(request, *, client, sleep, max_attempts, budget). Give it a docstring that lists its outcomes. For each attempt from 1 to max_attempts:

  1. Call check_budget(request, budget).

  2. Send the attempt with call_once(request, client=client) inside try.

  3. On openai.APIError: raise the error if should_retry(error) is false or this was the last attempt. Otherwise compute wait_seconds(error, attempt), raise the error if the wait is longer than MAX_WAIT, and otherwise call sleep(wait) and try again.

  4. On success, call record_usage(response, budget), then check_finished(response), then return the response.

Never call sleep after the last attempt, and use the sleep you were given, not time.sleep. Raise the caught error itself with raise error. Run python chapter9_demo.py to see each outcome, then check your work.