When Things Go Wrong · practice
Stop at Your Own Limit First
Chapter 7’s agent counted its tokens and stopped once it had used too many. That count belonged to one run of the agent, and it could only notice the problem after a reply had arrived. The study assistant also makes quizzes and chats, and a provider bills all of them. If the only limit is the provider’s bill, or a live model’s daily limit, the program finds out it spent too much when it is too late.
This lesson gives the program a limit of its own, checked before each call is sent.
One budget, shared
The budget is one dictionary:
budget = {"limit": 20000, "used": 0}
Every function that calls the model is given the same dictionary and changes it in place, so a quiz, an agent run and a chat all spend from one pot. Before a call, the program checks that the call could fit in what is left. After it, the program adds what the call really used.
An estimate before the call
Before a call is sent, nobody knows how many tokens it will use. The program can estimate a ceiling:
The request’s text. A common rule of thumb is about four characters per token, so the program adds up the characters of every message whose
contentis a string and divides by 4. A message that asked for tools hascontentNoneand is skipped.The reply. It can be as long as
max_tokens, so the estimate addsrequest.get("max_tokens", DEFAULT_MAX_TOKENS), the same valuecall_oncesends.
estimate_tokens(request) returns that sum. It can have a fraction: What is a Python list? has 22 characters, so its estimate is 22 / 4 + 800, which is 805.5. A fraction is fine for a comparison. Divide with / and do not round.
The scripted model counts prompt_tokens with the same rule but rounds up, so your estimate of the text is never more than one token below its count. Real providers count with their own tokenizer, and they also count the tool descriptions, which this estimate leaves out. It is a guard, not a bill.
Check, then record
check_budget(request, budget) works out what is left, left = budget["limit"] - budget["used"], and raises the supplied BudgetExceeded when the estimate is more than left:
raise BudgetExceeded(f"Not enough budget for this call: {left} of {budget['limit']} tokens are left.")
An estimate equal to left still fits. check_budget only checks; it does not change the budget.
record_usage(response, budget) adds response.usage.total_tokens to budget["used"]. It changes the dictionary it was given and returns nothing, so every caller that holds the same dictionary sees the new total.
The reserve for max_tokens dominates the estimate. What is a Python list? used 48 tokens in all, but it needs 805.5 tokens of room, because the reply could have been 800 tokens long. That is deliberate: the check must hold even for the longest reply the request allows.
Run it
When the three functions are written, run python chapter9_demo.py. The demonstration’s budget_demo sends What is a Python list? again and again with your check_budget, call_once and record_usage, starting from {"limit": 1000, "used": 0}, until the budget refuses:
The same question until the budget refuses it:
Call 1: estimate 805.5, budget now {'limit': 1000, 'used': 48}
Call 2: estimate 805.5, budget now {'limit': 1000, 'used': 96}
Call 3: estimate 805.5, budget now {'limit': 1000, 'used': 144}
Call 4: estimate 805.5, budget now {'limit': 1000, 'used': 192}
Call 5: estimate 805.5, budget now {'limit': 1000, 'used': 240}
Call 6: BudgetExceeded: Not enough budget for this call: 760 of 1000 tokens are left.
call_model under rate-limit:2:
Run stopped at the next function to build: call_model is built in Lesson 7.
Each call really used 48 tokens. The sixth was refused although 760 tokens were left, because 805.5 is more than 760, and no request was sent for it. The scripted model’s counts are an approximation of what a real model would report.
The budget is {"limit": 1000, "used": 240}. The next request is What is a Python list?, 22 characters, with no max_tokens. What does check_budget do?
Why does record_usage change budget["used"] in the dictionary it was given instead of returning a new dictionary?
Task
In reliable.py, write three functions:
estimate_tokens(request): add up, with a loop, the characters of every message inrequest["messages"]whosecontentis a string. Return that number divided by 4 with/, plusrequest.get("max_tokens", DEFAULT_MAX_TOKENS).check_budget(request, budget): withleft = budget["limit"] - budget["used"], raiseBudgetExceeded(f"Not enough budget for this call: {left} of {budget['limit']} tokens are left.")whenestimate_tokens(request)is more thanleft. Do not change the budget.record_usage(response, budget): addresponse.usage.total_tokenstobudget["used"]in the dictionary you were given, and return nothing.
Run python chapter9_demo.py to see the budget refuse a call, then check your work.