0%

When Things Go Wrong · practice

Retry a Stream Only Before It Prints

One kind of call is still outside call_model: the streaming chat. call_model returns a whole response, but chat_turn prints the reply while it arrives, piece by piece. That makes retrying harder. If two sentences are already on the screen when the stream breaks, a retry would start a new reply, and a model rarely writes the same reply twice. The person would see half of one answer followed by all of another.

The rule

A stream is retried only when nothing has been printed yet. That leaves two cases:

  • create itself raised. A streamed request that fails with a 429 or a 500 fails before any chunk arrives, so a retryable error from create is retried just as call_model retries it.

  • The stream broke before its first piece of text, and StreamBroken arrived with partial_text "". Nothing was shown, so a retry is safe.

A StreamBroken whose partial_text is not empty is raised, never retried, and the caller decides what to tell the person.

The supplied stream_once

stream_once(request, *, client, budget) in reliable.py is finished code that does one streamed attempt. Read it now. It is Chapter 5’s stream_chat with three changes:

  1. It sends max_tokens, request.get("max_tokens", DEFAULT_MAX_TOKENS), like call_once, and it raises ReplyCutOff if the finish reason is "length".

  2. It adds the usage chunk to the budget. A streamed reply only reports its tokens because the request asks for them with stream_options={"include_usage": True}; the last chunk then carries usage, and stream_once adds its total_tokens to budget["used"]. A stream that breaks never reaches that chunk, so its tokens go uncounted here, although a provider would still bill them.

  3. When a stream breaks before its first piece, it prints nothing, not even a newline. So after a retry the screen shows only the reply that worked.

Errors from create reach the caller unchanged, and a broken stream becomes StreamBroken with the text so far, as in Chapter 5.

chat_turn

chat_turn(history, question, *, client, budget, sleep) builds the messages as Chapter 5 did and makes up to MAX_ATTEMPTS attempts:

messages = prepare_messages(history, question)
request = {"model": MODEL, "messages": messages}

For each attempt from 1 to MAX_ATTEMPTS:

  1. check_budget(request, budget).

  2. Inside try, reply = stream_once(request, client=client, budget=budget).

  3. On StreamBroken: if its partial_text is not "", or this was the last attempt, raise it.

  4. On openai.APIError: if should_retry(error) is false, or this was the last attempt, raise it.

  5. Otherwise wait as call_model does: compute wait_seconds(error, attempt), raise the error if the wait is longer than MAX_WAIT, else sleep(wait) and try again.

  6. When an attempt succeeds, return (reply, add_turn(messages[:-1], question, reply)), the reply and the new history, as Chapter 5’s chat_turn did.

Steps 3 and 4 each end with step 5, so write the wait inside both except blocks. It cannot go once after them: Python removes the name error when its except block ends.

List except StreamBroken and except openai.APIError as two separate clauses. StreamBroken is not an openai error, so neither clause catches the other’s errors. A StreamBroken has no response, so wait_seconds doubles BASE_WAIT for it. ReplyCutOff is not caught: a cut-off stream already printed its text.

chat_turn returns two values in a tuple. The demonstration reads them as result[0] and result[1].

Run it

When chat_turn is written, run python chapter9_demo.py. The demonstration calls reset_scripted_model(), then asks Explain Python lists in detail. through scenario_client("rate-limit:1") and then through scenario_client("broken-stream"):

The study chat under rate-limit:1:
  (waiting 1.0 s)
A list is an ordered collection that can hold any mix of values. You create one with square brackets, such as scores = [90, 72, 85]. Each item has a position, starting at 0, so scores[0] is 90 and scores[-1] is the last item. Slicing takes a range: scores[0:2] gives a new list with the first two items. Lists are mutable, which means you can change them after creating them: scores.append(60) adds an item to the end, scores.remove(72) removes the first matching value, and scores[1] = 75 replaces an item in place. Because a list is changed in place, two names that refer to the same list both see the change. Use list.copy() when you need an independent list. len(scores) tells you how many items it holds.
The history now holds 3 messages. Budget: {'limit': 20000, 'used': 229}
The study chat under broken-stream:
A list is an ordered collection that can hold 
StreamBroken. The partial text was 'A list is an ordered collection that can hold '

Under rate-limit:1, create raised a RateLimitError before any chunk arrived, so chat_turn waited the one second from Retry-After and tried again, and the second stream printed the whole reply once. The budget counts that reply, 229 tokens with the study system message, because stream_once asked for the usage chunk. The history holds the system message, the question and the answer.

Under broken-stream, the first pieces were already on the screen when the connection closed, so chat_turn raised StreamBroken without a second attempt.

Every model call in the study assistant now goes through these rules: call_model for the quiz, the agent and any single call, and chat_turn for the streaming chat, all with one budget.

When does chat_turn try a stream again?

Why does chat_turn have except StreamBroken and except openai.APIError as two separate clauses?

Task

In reliable.py, write chat_turn(history, question, *, client, budget, sleep). Build messages = prepare_messages(history, question) and request = {"model": MODEL, "messages": messages}. For each attempt from 1 to MAX_ATTEMPTS:

  1. Call check_budget(request, budget).

  2. Inside try, call stream_once(request, client=client, budget=budget).

  3. On StreamBroken, raise it if its partial_text is not "" or this was the last attempt.

  4. On openai.APIError, raise it if should_retry(error) is false or this was the last attempt.

  5. In each of those two clauses, if you did not raise, compute wait_seconds(error, attempt), raise the error if the wait is longer than MAX_WAIT, and otherwise call sleep(wait) and try again.

When an attempt succeeds, return (reply, add_turn(messages[:-1], question, reply)). Let ReplyCutOff reach the caller. Do not change the supplied stream_once. Run python chapter9_demo.py to see a retried stream and a broken one, then check your work.