0%

A Small Agent · practice

Ask Before Changing Anything

So far the agent’s tools only read. A wrong read_note call costs a few tokens and nothing else. This lesson gives the agent a tool that writes a file, and makes the program ask a person before it runs.

A tool that changes something

Near the top of agent.py is the third tool, save_flashcard, with its helper:

def card_name_ok(name):
    """Return True when name has 1 to 40 characters, all from CARD_NAME_CHARACTERS."""
    if len(name) < 1 or len(name) > 40:
        return False
    for character in name:
        if character not in CARD_NAME_CHARACTERS:
            return False
    return True


def save_flashcard(arguments):
    """Tool: save a card as flashcards/<name>.md. It writes a file, so it needs approval."""
    name = arguments.get("name")
    front = arguments.get("front")
    back = arguments.get("back")
    if not isinstance(name, str) or not card_name_ok(name):
        raise ValueError(
            "save_flashcard needs a name made of lowercase letters, digits and hyphens, such as list-copying."
        )
    if not isinstance(front, str) or not isinstance(back, str):
        raise ValueError("save_flashcard needs front and back strings.")
    FLASHCARDS_DIR.mkdir(exist_ok=True)
    with open(FLASHCARDS_DIR / f"{name}.md", "w", encoding="utf-8") as handle:
        handle.write(f"Q: {front}\nA: {back}\n")
    return f"Saved flashcards/{name}.md."

The model chooses all three arguments, including the file name. A name is joined onto a folder to make a path, and a name like ../notes/01-variables would climb out of flashcards and overwrite one of the course notes. So card_name_ok accepts only lowercase letters, digits and hyphens (CARD_NAME_CHARACTERS), and save_flashcard refuses any other name with a ValueError before it builds a path. A dot or a slash cannot pass. The ValueError reaches the model as an error result, through Chapter 6’s run_tool.

That check keeps every write inside flashcards, but it cannot decide whether the person wants a flashcard at all, and it does not stop a new card from replacing an earlier one with the same name. A model that misread the question could save one nobody asked for, or many. Reading a note changes nothing, so the agent may do it freely. Writing a file changes something, so a person decides.

Who decides

agent.py lists the tools that change something:

CHANGING_TOOLS = ["save_flashcard"]

Before any of them runs, the loop asks. How it asks is up to the caller: a program might print the call and wait for input(), a web app might show a button, and a test answers automatically. So the asking is a function the caller passes in, called approve. The loop calls approve(call) with the call’s dictionary, and runs the tool only when it returns True.

That is passing a function as an argument. You have done the other half of the idea before: TOOLS stored functions as values in a dictionary, and run_tool called the one it found. Here the function arrives as a parameter, and run_calls calls it without knowing what it does.

Two such functions are supplied for the Runs:

def approve_all(call):
    """Approve every call. A real program would ask the person, for example with input()."""
    return True


def refuse_all(call):
    """Refuse every call. A real program would ask the person, for example with input()."""
    return False

A refusal is not an error in the program. The model asked for something, and the answer was no. It goes back to the model as a tool result, so the model can tell the person what happened:

Error: save_flashcard was not approved, so it did not run.

Build run_calls

run_calls(calls, tools, approve) replaces tool_messages in the loop:

  1. Start with an empty list.

  2. For each call, in order: if its name is in CHANGING_TOOLS and approve(call) does not return True, the result is f"Error: {name} was not approved, so it did not run.". Otherwise the result is run_tool(call, tools).

  3. Append tool_result(call, result) for each call, and return the list.

Ask only for tools in CHANGING_TOOLS, and ask once per call. A refused call does not stop the others in the same reply: each call gets its own result, as in Chapter 6 Lesson 7.

Then give run_agent a keyword-only parameter approve, after system, and call run_calls(calls, tools, approve) where it called tool_messages(calls, tools). The results still go into the trace, so a refusal shows up there.

Run it

Run python agent.py. The demonstration sends Make a flashcard about copying a list. with AGENT_TOOLS, first with approve_all and then with refuse_all, and then Make a flashcard called ../notes/01-variables. with approve_all:

Make a flashcard about copying a list. With approve_all:
Step 1 (50 tokens)
  read_note {"note_id": "03-lists"} -> 1226 characters
Step 2 (391 tokens)
  save_flashcard {"name": "copying-a-list", "front": "How do you make an independent copy of a list?", "back": "Use scores.copy() or list(scores). Both build a new list."} -> 35 characters
Step 3 (384 tokens)
  answered
answer: I saved a flashcard called copying-a-list. It asks how to make an independent copy of a list. (03-lists)
Make a flashcard about copying a list. With refuse_all:
Step 1 (50 tokens)
  read_note {"note_id": "03-lists"} -> 1226 characters
Step 2 (391 tokens)
  save_flashcard {"name": "copying-a-list", "front": "How do you make an independent copy of a list?", "back": "Use scores.copy() or list(scores). Both build a new list."} -> 58 characters
Step 3 (414 tokens)
  answered
answer: I did not save the flashcard, because you did not approve it. Here it is, so you can write it down yourself: How do you make an independent copy of a list? Use scores.copy() or list(scores). (03-lists)
Make a flashcard called ../notes/01-variables. With approve_all:
Step 1 (75 tokens)
  save_flashcard {"name": "../notes/01-variables", "front": "What is a variable?", "back": "A name that refers to a value."} -> 103 characters
Step 2 (95 tokens)
  answered
answer: I couldn't save that flashcard. The tool answered with an error, which you can read in the trace.
flashcards/copying-a-list.md holds:
Q: How do you make an independent copy of a list?
A: Use scores.copy() or list(scores). Both build a new list.
Run stopped at the next function to build: repeated_call is built in Lesson 7.

Approved, the card was saved, and its 35-character result is Saved flashcards/copying-a-list.md.. Refused, the result was the 58-character refusal, and the model’s last reply says it did not save the card. The scripted model can answer that way because its row for this question has a reply for a request whose tool results contain was not approved, as SCRIPTED_MODEL.md says. A live model would read the refusal and write its own reply.

The third run was approved, and save_flashcard refused anyway: the 103-character result is its ValueError message about the name. Approval is not the only guard. A person can approve a call without reading the file name closely, and the tool still checks what it is given.

There is a new flashcards folder in your workspace now, with the saved card in it.

During a run, the model asks for read_note and then save_flashcard. How many times does run_calls call approve?

approve refuses a save_flashcard call. What does the model get back for it?

approve_all approves the call save_flashcard with the name ../notes/01-variables. What happens?

Task

In agent.py, replace the unfinished body of run_calls(calls, tools, approve), then use it in run_agent.

  • run_calls: for each call, in order, if its name is in CHANGING_TOOLS and approve(call) does not return True, the result is f"Error: {name} was not approved, so it did not run.". Otherwise the result is run_tool(call, tools). Return one tool_result(call, result) message per call.

  • Call approve only for tools in CHANGING_TOOLS, once per call, with the call’s dictionary.

  • Give run_agent a keyword-only parameter approve, after system, and use run_calls(calls, tools, approve) where it used tool_messages(calls, tools).

Run python agent.py to ask for a flashcard three ways, then check your work.