0%

Taking It Home · practice

Keep a Log of Every Run

When the closes, every run of the study assistant is gone: what it was asked to do, whether it worked, how many tokens it used. That was fine on the course model. Once a program spends money for someone, three questions come up again and again: how much is it spending, how often does it fail, and is either of those changing? A log answers them, if every run writes one line in the same shape.

One JSON object per line

The study assistant’s log is a JSON lines file, study-assistant-log.jsonl: one JSON object per line, one line per run, each added at the end of the file. Here is a line:

{"started": 1794170907.41, "command": "ask", "model": "pl-scripted", "exit_code": 0, "tokens": 539, "seconds": 0.46}

Three things make this shape a good one:

  • Appending never rewrites earlier lines, so a run that crashes halfway can lose at most its own line, never last week’s.

  • Each line is complete JSON on its own, so a program reads the file one line at a time with json.loads, as you read files line by line in Python in Practice.

  • Almost any tool can read it, from a spreadsheet import to a few lines of Python.

Python’s standard library has a logging module, the usual tool for larger programs that write many kinds of messages. This program needs one record per run, so it writes its own lines.

What goes in a record, and what stays out

A record holds six keys: "started", when the run started (seconds since 1970, from time.time()); "command", the command’s name; "model"; "exit_code"; "tokens", the tokens the run used; and "seconds", how long it took.

It holds no question, no answer and no key. A log is read by more people than the person who typed the question: whoever runs the program, whoever debugs it, whoever the file is sent to. Questions and answers can be private, and a key in a log is a leaked key. Logging what people asked is a decision about their privacy, made on purpose and told to them, never a default.

log_run

log_run(path, record) appends one record to the log:

  1. Open path in append mode, "a", with encoding="utf-8". Append mode creates the file if it is missing and writes after whatever it already holds. "w" would empty the file first.

  2. Write json.dumps(record) followed by a newline. With no indent, json.dumps writes the whole object on one line.

It returns nothing.

main logs the run

main gains two keyword parameters, log_file=None and clock=time.time. log_file is where to write the record, and None means no log, so tests, checks and Chapter 11’s own calls write nothing. clock is the function that tells the time. Passing time.time by default, and letting a caller pass another, is the same idea as passing sleep since Chapter 9: a check can pass a fake clock that returns 100.0 and then 102.5, and know the run took exactly 2.5 seconds.

In main:

  • Read started = clock() right after parsing the arguments.

  • Keep Lesson 2’s missing-key check, but instead of returning at once, set code = 1 and skip the command. A missing key is the most likely failure at home, so it belongs in the log too, with 0 tokens.

  • Otherwise build the budget and get code from run_command, as before.

  • When log_file is not None, call log_run with this record, then return code:

{
    "started": started,
    "command": args.command,
    "model": MODEL,
    "exit_code": code,
    "tokens": budget["used"],
    "seconds": clock() - started,
}

For the missing-key run there is no budget, so tokens is 0 there. One way is a tokens = 0 before the check, set to budget["used"] after run_command.

A usage error is not logged. argparse ends the program with exit code 2 inside parse_args, before main has a command to write down.

Finally, make a run from the terminal write to the real log. Change the guard at the bottom to:

if __name__ == "__main__":
    raise SystemExit(main(log_file=LOG_FILE))

Run it

Run python stages.py. The Lesson 4 block calls your main four times, the way the terminal would, with a log in a temporary folder so the demonstration never adds to your own study-assistant-log.jsonl, and prints the log and its summary:

Lesson 4: four runs, logged
$ python study_assistant.py ask "How do I make an independent copy of a list?"
Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.

Sources: 03-lists#4
exit code 0
$ python study_assistant.py quiz 03-lists --count 1
1. For scores = [90, 72, 85], what is scores[-1]?
   a) 90
   b) 72
   c) 85

Answers: 1c
exit code 0
$ python study_assistant.py quiz 99-magic
Error: There is no note called 99-magic.
exit code 4
$ python study_assistant.py --budget 1000 ask "How do I make an independent copy of a list?"
Error: The budget ran out before agent step 2.
exit code 3
The log, one line per run:
{"started": 1794170907.412106, "command": "ask", "model": "pl-scripted", "exit_code": 0, "tokens": 539, "seconds": 0.46033787727355957}
{"started": 1794170907.873569, "command": "quiz", "model": "pl-scripted", "exit_code": 0, "tokens": 420, "seconds": 0.24816584587097168}
{"started": 1794170908.122124, "command": "quiz", "model": "pl-scripted", "exit_code": 4, "tokens": 0, "seconds": 0.0007469654083251953}
{"started": 1794170908.123203, "command": "ask", "model": "pl-scripted", "exit_code": 3, "tokens": 159, "seconds": 0.046597957611083984}
Its summary:
Runs: 4
By exit code: exit 0: 2, exit 3: 1, exit 4: 1
Tokens: 1118 in all, 280 per run
Slowest run: 0.5 seconds
Cost estimate: about $0.0002, counting every token at $0.14 per million
  (an estimate: the budget's own token count is an estimate, and a provider may
  route a call to an endpoint with another price)

Your started and seconds values differ, and so may the slowest run in the summary: they are the time of your run and how long it took. Everything else matches. The budget run used 159 tokens on agent step 1 and was refused before step 2, so its line records those 159 tokens and exit code 3.

Then the terminal: run python study_assistant.py ask "How do I make an independent copy of a list?" twice, and open study-assistant-log.jsonl in the editor. It has one line per run since you changed the guard: two, if these were the first.

What to watch

The last part of the demonstration summarizes sample-run-log.jsonl, a made-up week of 28 runs supplied for this lesson, the kind of log the program would write at home:

The summary of sample-run-log.jsonl, a made-up week of runs supplied for this lesson:
Runs: 28
By exit code: exit 0: 22, exit 1: 4, exit 3: 2
Tokens: 44823 in all, 1601 per run
Slowest run: 64.2 seconds
Cost estimate: about $0.0063, counting every token at $0.14 per million
  (an estimate: the budget's own token count is an estimate, and a provider may
  route a call to an endpoint with another price)

summarize_log is finished code: it reads the file one line at a time with json.loads, counts the exit codes in a dictionary and adds up the tokens. Read the summary as the person running the program would:

  • Four runs in 28 ended with exit code 1. A rising share of exit 1 means the provider or the model is failing. The log says how often, not why; the Error: line each failed run printed says why, such as a rate limit or a model error.

  • Two runs ended with exit code 3. Runs that reach --budget mean the budget is too tight for what people ask, or the prompts are growing. Both used under 1,800 tokens, so they were started with a small --budget, such as 2000. The log records the tokens a run used, not the budget it was given, so the line alone cannot tell a tight budget from a growing prompt.

  • Tokens per run is the number to watch over time. Open the file: the ask runs grow from about 1,900 tokens on the first day to about 2,600 on the last. Tokens per run creeping up usually means longer histories or bigger search results going into every prompt.

  • The slowest run took 64.2 seconds and still succeeded. That is a run that waited out a rate limit for about a minute, close to MAX_WAIT. The slowest runs show where the timeout and MAX_WAIT bite.

  • The cost estimate is the number to compare with the credit limit on your key. Here a week cost well under a cent, so a limit of a few dollars leaves room for mistakes without allowing a disaster.

Why does log_run open the log with "a" rather than "w"?

To debug bad answers, someone suggests adding each run's question to its log record. What should you do?

Task

In study_assistant.py:

  1. Write log_run(path, record): open path with "a" and encoding="utf-8", write json.dumps(record) and a newline, and return nothing.

  2. Give main two more keyword parameters, log_file=None and clock=time.time. Read started = clock() after parsing. When the key is missing, keep Lesson 2’s error line but set the exit code to 1 instead of returning, and skip the command. When log_file is not None, call log_run(log_file, record) with the keys "started", "command" (args.command), "model" (MODEL), "exit_code", "tokens" (the budget’s used tokens, or 0 with no key) and "seconds" (clock() - started). Return the exit code.

  3. Change the guard to raise SystemExit(main(log_file=LOG_FILE)).

Run python stages.py to see four logged runs and two summaries, then check your work.