0%

Tools · practice

Check the Request Before You Run Anything

Lesson 5’s demonstration ended with your whole program stopped by one bad tool call. The model asked for a note called 99-magic, read_note raised ValueError, and run_tool let it through. Nothing was wrong with your program. The model made a mistake, and models do: they ask for tools that do not exist, write arguments that are not valid JSON, or leave out an argument.

The model chose the name and wrote the arguments, so your program checks both before anything runs. When a check fails, the program does not stop. It sends an error message back to the model as the tool’s result, and the model can read it and try something else, or tell the user what went wrong.

Four ways a request can be bad

BAD_CALLS near the bottom of tools.py holds one of each, written by hand in the same form read_tool_calls returns:

CallWhat is wrongWhat run_tool returns
delete_note with '{"note_id": "03-lists"}'no tool has that nameError: there is no tool called delete_note. The tools are: list_notes, read_note.
read_note with '{"note_id": 'the arguments are not valid JSONError: the arguments are not valid JSON.
read_note with '["03-lists"]'valid JSON, but a list, not an objectError: the arguments must be a JSON object.
read_note with '{}'the tool itself refusesError: read_note needs a note_id string, such as 03-lists.

The 99-magic call is the last kind too: read_note raises ValueError with its own message, and run_tool passes that message on.

The new run_tool

run_tool(call, tools) now goes through five steps, in this order:

  1. The name. If call["name"] is not a key of tools, return f"Error: there is no tool called {name}. The tools are: {names}.", where names is the tool names joined with ", ". ", ".join(tools) does that: looping over a dictionary, as join does, gives its keys, in the order they were added. Listing the real names gives the model what it needs to try again.

  2. The JSON. Decode call["arguments"] with json.loads inside a try. If it raises json.JSONDecodeError, return "Error: the arguments are not valid JSON.".

  3. The type. If the decoded value is not a dictionary (isinstance(arguments, dict)), return "Error: the arguments must be a JSON object.". Text such as '["03-lists"]', '"03-lists"' or 'null' is valid JSON, but every tool expects one dictionary.

  4. The tool. Call the tool in a second try. If it raises ValueError, return f"Error: {error}", which puts the tool’s own message after Error: .

  5. The result. Otherwise return what the tool returned.

Steps 1 to 3 run before the tool is called, so a bad name or bad JSON never reaches a tool at all.

Two try blocks, not one

It is tempting to wrap everything in one try and catch ValueError. That does not work, because json.JSONDecodeError is itself a kind of ValueError:

import json

try:
    json.loads('{"note_id": ')
except ValueError as error:
    print(type(error).__name__, "is a ValueError:", isinstance(error, ValueError))

With one try, a JSON error would land in the tool’s except ValueError and come back as Error: Expecting value: line 1 column 13 (char 12), a message about the decoder rather than the request. Listing except json.JSONDecodeError first does not fully fix one try. A tool can raise JSONDecodeError itself, for example while reading a JSON file, and that would be reported to the model as bad arguments. A try around only json.loads catches only the model’s mistake. Decoding in its own try, before the tool runs, keeps the two problems apart.

What still stops the program

run_tool catches JSONDecodeError around the decoding and ValueError around the tool, and nothing else. If a tool raises KeyError or TypeError, that is not the model’s mistake: it is a bug in the tool’s code, and you want to see it, as a traceback, while you fix it. Catching every exception would send your own bug to the model as if the model had done something wrong, and you might never notice it.

The supplied tools follow that rule. read_note raises ValueError for every problem a model can cause, with a message written for the model, and anything else it raised would be a bug.

Run it

Run python tools.py. The demonstration asks What does the note 99-magic say? again, sends your error result back to the model, then passes each call in BAD_CALLS to your run_tool:

A request for a note that does not exist:
[{'id': 'call_1_1', 'name': 'read_note', 'arguments': '{"note_id": "99-magic"}'}]
Result: Error: There is no note called 99-magic. Call list_notes to see the ids.
Answer: I couldn't find a note called 99-magic. The notes run from 01-variables to 12-testing.
The calls in BAD_CALLS:
delete_note {"note_id": "03-lists"} -> Error: there is no tool called delete_note. The tools are: list_notes, read_note.
read_note {"note_id":  -> Error: the arguments are not valid JSON.
read_note ["03-lists"] -> Error: the arguments must be a JSON object.
read_note {} -> Error: read_note needs a note_id string, such as 03-lists.
Two calls in one reply:
[{'id': 'call_1_1', 'name': 'read_note', 'arguments': '{"note_id": "03-lists"}'}, {'id': 'call_1_2', 'name': 'read_note', 'arguments': '{"note_id": "06-dictionaries"}'}]
Run stopped at the next function to build: tool_messages is built in Lesson 7.

The program kept going. The model got an error result, and its answer tells the user what happened instead of the program crashing. That answer is fixed, as always with the scripted model; a live model would read the error and might call list_notes to look for a better id, which Chapter 7’s loop makes possible.

A learner puts json.loads and the tool call in one try with except ValueError as error: return f"Error: {error}". What happens to a call whose arguments are '{"note_id": '?

A tool has a bug and raises KeyError. Why should run_tool let it through instead of returning Error: ... to the model?

Why does run_tool check isinstance(arguments, dict) after decoding?

Task

In tools.py, change run_tool(call, tools) so a bad request returns an error string instead of stopping the program. In this order:

  1. If call["name"] is not in tools, return f"Error: there is no tool called {name}. The tools are: {names}.", with the names of tools joined by ", " in the dictionary’s order.

  2. Decode call["arguments"] in its own try. On json.JSONDecodeError, return "Error: the arguments are not valid JSON.".

  3. If the decoded value is not a dictionary, return "Error: the arguments must be a JSON object.".

  4. Call the tool in a second try. On ValueError, return f"Error: {error}".

  5. Otherwise return the tool’s result.

Do not call the tool when the name or the arguments are bad, and catch no other exception: a KeyError or TypeError from a tool must reach the caller. Run python tools.py to see the bad calls answered with errors, then check your work.