Taking It Home · practice
Keep a Log of Every Run
When the
One JSON object per line
The study assistant’s log is a JSON lines file, study-assistant-log.jsonl: one JSON object per line, one line per run, each added at the end of the file. Here is a line:
{"started": 1794170907.41, "command": "ask", "model": "pl-scripted", "exit_code": 0, "tokens": 539, "seconds": 0.46}
Three things make this shape a good one:
Appending never rewrites earlier lines, so a run that crashes halfway can lose at most its own line, never last week’s.
Each line is complete JSON on its own, so a program reads the file one line at a time with
json.loads, as you read files line by line in Python in Practice.Almost any tool can read it, from a spreadsheet import to a few lines of Python.
Python’s standard library has a logging module, the usual tool for larger programs that write many kinds of messages. This program needs one record per run, so it writes its own lines.
What goes in a record, and what stays out
A record holds six keys: "started", when the run started (seconds since 1970, from time.time()); "command", the command’s name; "model"; "exit_code"; "tokens", the tokens the run used; and "seconds", how long it took.
It holds no question, no answer and no key. A log is read by more people than the person who typed the question: whoever runs the program, whoever debugs it, whoever the file is sent to. Questions and answers can be private, and a key in a log is a leaked key. Logging what people asked is a decision about their privacy, made on purpose and told to them, never a default.
log_run
log_run(path, record) appends one record to the log:
Open
pathin append mode,"a", withencoding="utf-8". Append mode creates the file if it is missing and writes after whatever it already holds."w"would empty the file first.Write
json.dumps(record)followed by a newline. With noindent,json.dumpswrites the whole object on one line.
It returns nothing.
main logs the run
main gains two keyword parameters, log_file=None and clock=time.time. log_file is where to write the record, and None means no log, so tests, checks and Chapter 11’s own calls write nothing. clock is the function that tells the time. Passing time.time by default, and letting a caller pass another, is the same idea as passing sleep since Chapter 9: a check can pass a fake clock that returns 100.0 and then 102.5, and know the run took exactly 2.5 seconds.
In main:
Read
started = clock()right after parsing the arguments.Keep Lesson 2’s missing-key check, but instead of returning at once, set
code = 1and skip the command. A missing key is the most likely failure at home, so it belongs in the log too, with 0 tokens.Otherwise build the budget and get
codefromrun_command, as before.When
log_fileis notNone, calllog_runwith this record, then returncode:
{
"started": started,
"command": args.command,
"model": MODEL,
"exit_code": code,
"tokens": budget["used"],
"seconds": clock() - started,
}
For the missing-key run there is no budget, so tokens is 0 there. One way is a tokens = 0 before the check, set to budget["used"] after run_command.
A usage error is not logged. argparse ends the program with exit code 2 inside parse_args, before main has a command to write down.
Finally, make a run from the terminal write to the real log. Change the guard at the bottom to:
if __name__ == "__main__":
raise SystemExit(main(log_file=LOG_FILE))
Run it
Run python stages.py. The Lesson 4 block calls your main four times, the way the terminal would, with a log in a temporary folder so the demonstration never adds to your own study-assistant-log.jsonl, and prints the log and its summary:
Lesson 4: four runs, logged
$ python study_assistant.py ask "How do I make an independent copy of a list?"
Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.
Sources: 03-lists#4
exit code 0
$ python study_assistant.py quiz 03-lists --count 1
1. For scores = [90, 72, 85], what is scores[-1]?
a) 90
b) 72
c) 85
Answers: 1c
exit code 0
$ python study_assistant.py quiz 99-magic
Error: There is no note called 99-magic.
exit code 4
$ python study_assistant.py --budget 1000 ask "How do I make an independent copy of a list?"
Error: The budget ran out before agent step 2.
exit code 3
The log, one line per run:
{"started": 1794170907.412106, "command": "ask", "model": "pl-scripted", "exit_code": 0, "tokens": 539, "seconds": 0.46033787727355957}
{"started": 1794170907.873569, "command": "quiz", "model": "pl-scripted", "exit_code": 0, "tokens": 420, "seconds": 0.24816584587097168}
{"started": 1794170908.122124, "command": "quiz", "model": "pl-scripted", "exit_code": 4, "tokens": 0, "seconds": 0.0007469654083251953}
{"started": 1794170908.123203, "command": "ask", "model": "pl-scripted", "exit_code": 3, "tokens": 159, "seconds": 0.046597957611083984}
Its summary:
Runs: 4
By exit code: exit 0: 2, exit 3: 1, exit 4: 1
Tokens: 1118 in all, 280 per run
Slowest run: 0.5 seconds
Cost estimate: about $0.0002, counting every token at $0.14 per million
(an estimate: the budget's own token count is an estimate, and a provider may
route a call to an endpoint with another price)
Your started and seconds values differ, and so may the slowest run in the summary: they are the time of your run and how long it took. Everything else matches. The budget run used 159 tokens on agent step 1 and was refused before step 2, so its line records those 159 tokens and exit code 3.
Then the terminal: run python study_assistant.py ask "How do I make an independent copy of a list?" twice, and open study-assistant-log.jsonl in the editor. It has one line per run since you changed the guard: two, if these were the first.
What to watch
The last part of the demonstration summarizes sample-run-log.jsonl, a made-up week of 28 runs supplied for this lesson, the kind of log the program would write at home:
The summary of sample-run-log.jsonl, a made-up week of runs supplied for this lesson:
Runs: 28
By exit code: exit 0: 22, exit 1: 4, exit 3: 2
Tokens: 44823 in all, 1601 per run
Slowest run: 64.2 seconds
Cost estimate: about $0.0063, counting every token at $0.14 per million
(an estimate: the budget's own token count is an estimate, and a provider may
route a call to an endpoint with another price)
summarize_log is finished code: it reads the file one line at a time with json.loads, counts the exit codes in a dictionary and adds up the tokens. Read the summary as the person running the program would:
Four runs in 28 ended with exit code 1. A rising share of exit 1 means the provider or the model is failing. The log says how often, not why; the
Error:line each failed run printed says why, such as a rate limit or a model error.Two runs ended with exit code 3. Runs that reach
--budgetmean the budget is too tight for what people ask, or the prompts are growing. Both used under 1,800 tokens, so they were started with a small--budget, such as 2000. The log records the tokens a run used, not the budget it was given, so the line alone cannot tell a tight budget from a growing prompt.Tokens per run is the number to watch over time. Open the file: the
askruns grow from about 1,900 tokens on the first day to about 2,600 on the last. Tokens per run creeping up usually means longer histories or bigger search results going into every prompt.The slowest run took 64.2 seconds and still succeeded. That is a run that waited out a rate limit for about a minute, close to
MAX_WAIT. The slowest runs show where the timeout andMAX_WAITbite.The cost estimate is the number to compare with the credit limit on your key. Here a week cost well under a cent, so a limit of a few dollars leaves room for mistakes without allowing a disaster.
Why does log_run open the log with "a" rather than "w"?
To debug bad answers, someone suggests adding each run's question to its log record. What should you do?
Task
In study_assistant.py:
Write
log_run(path, record): openpathwith"a"andencoding="utf-8", writejson.dumps(record)and a newline, and return nothing.Give
maintwo more keyword parameters,log_file=Noneandclock=time.time. Readstarted = clock()after parsing. When the key is missing, keep Lesson 2’s error line but set the exit code to1instead of returning, and skip the command. Whenlog_fileis notNone, calllog_run(log_file, record)with the keys"started","command"(args.command),"model"(MODEL),"exit_code","tokens"(the budget’s used tokens, or0with no key) and"seconds"(clock() - started). Return the exit code.Change the guard to
raise SystemExit(main(log_file=LOG_FILE)).
Run python stages.py to see four logged runs and two summaries, then check your work.