0%

Answering from Your Notes · capstone

Capstone project: Project: Answer Questions from the Course Notes

Lessons 2 to 7 built every step of a grounded answer, and the demonstration has been calling them one by one. This project joins them into one function that answers a question from the notes, and turns the search into a tool that an agent can call:

answer = answer_from_notes("How do I make an independent copy of a list?", load_notes(), client=make_client(), k=3)
print(answer.answer)
print(answer.citations)
Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.
['03-lists#4']

The pieces you have

StepYour code
Split the notes into chunksnote_chunks(notes) (Lesson 2)
Score and keep the best kbest_chunks(question, chunks, k) (Lessons 3 and 4)
Build the requestgrounded_messages(question, chunks) (Lesson 5)
Check the replyparse_cited_answer(text, chunks) (Lesson 6)
Decide whether to askshould_answer(best) (Lesson 7)

Build answer_from_notes

answer_from_notes(question, notes, *, client, k):

  1. best = best_chunks(question, note_chunks(notes), k).

  2. If should_answer(best) is false, return CitedAnswer(answer=I_DONT_KNOW, citations=[]) straight away. No request is made, so nothing is spent.

  3. Otherwise send one request: client.chat.completions.create(model=MODEL, messages=grounded_messages(question, best), response_format={"type": "json_object"}).

  4. Return parse_cited_answer(response.choices[0].message.content, best). Pass best, the chunks you sent, and not every chunk of the notes: a citation is real only if its chunk was in this request.

Let every error reach the caller: a ValidationError for a reply in the wrong shape (a reply with no text included), a ValueError for a citation that was not sent, and any error from the client. The caller decides what to do about each. Use the client you are given; do not build one with make_client() inside the function.

The picture follows one question through answer_from_notes. It shows three ways out: the fixed I_DONT_KNOW answer without a request, a checked CitedAnswer, or a ValueError for a citation that was not sent. A reply in the wrong shape raises ValidationError at the same check.

answer_from_notes either says I don't know without a request, or sends the best chunks as labeled sources and accepts the answer only if every citation was sent.

Why create and not the parse helper

Chapter 5’s client.chat.completions.parse could validate the reply into a CitedAnswer by itself. This project uses create with json_object and validates the text with parse_cited_answer instead, for two reasons. The citation check needs the list of chunks that were sent, which the parse helper knows nothing about. And parse_cited_answer works on any reply text, wherever it came from: in Chapter 11, the agent’s final reply is checked the same way, and that reply comes from the agent’s loop, not from one parse call.

Build search_notes

Chapter 7’s agent reads whole notes with read_note. A search tool lets it ask for the passages that fit a question instead. search_notes(arguments) is a tool function in Chapter 6’s shape: one argument, the decoded arguments dictionary, and a string result.

  1. query = arguments.get("query"). If it is not a string (isinstance), or query.strip() is empty, raise ValueError("search_notes needs a query string, such as copying a list."). Chapter 6’s run_tool turns that into an error result for the model.

  2. best = best_chunks(query, note_chunks(load_notes()), K). Call load_notes() each time, so a note added to the folder is found, and use the module’s K.

  3. Return "No note matches that search." when best is empty, otherwise format_sources(best), the labeled sources.

The model reads the result like any tool result, and the labels in it are what it can cite.

Describe it

Replace SEARCH_NOTES_DESCRIPTION = {} with a tool description in the form of Chapter 6 Lesson 2: "type": "function", and a "function" with "name": "search_notes", a description that says what the tool returns, and parameters with one property, query, of type "string" with a description of its own, listed under "required", and "additionalProperties": False. The capstone in Chapter 11 offers it to the agent next to read_note.

Run it

Run python retrieval.py. The demonstration asks three questions through answer_from_notes, then calls search_notes directly:

Answers from the notes:
You: How do I make an independent copy of a list?
Assistant: Use scores.copy() or list(scores). Both build a new list, so a change made through one name no longer shows through the other.
Sources: 03-lists#4
You: Why should a default argument never be a list?
Refused: The answer cites 05-functions#6, which is not one of its sources.
You: What is the capital of France?
Assistant: I don't know. The notes don't cover that.
search_notes({"query": "copy a list"}) returns:
[01-variables#2] Assignment does not copy anything. After `a = [1, 2]` and `b = a`, both names refer to the same list, so a change made through `b` is visible through `a`. Numbers and strings cannot be changed in place, so for them this sharing never causes surprises. Lists and dictionaries can be changed, which is why sharing matters for them.

[03-lists#4] Because a list is changed in place, two names that refer to the same list both see every change. Use `scores.copy()` or `list(scores)` when you need an independent list. Adding two lists with `+` also builds a new list and leaves both originals unchanged.

[04-loops#7] Avoid adding or removing items in a list while a `for` loop runs over that same list. Loop over a copy instead, or build a new list.
The tool description is called search_notes.

The copy question is answered and cited. The default-argument reply cites a chunk it was not sent, so it is refused instead of shown. The France question gets the fixed answer, and no request was sent for it. search_notes returns the three labeled sources for copy a list, in the tie order from Lesson 4.

The scripted answers are fixed; a live model writes its own from the sources, which is when the citation check matters. Before you check your work, think about what the demonstration does not show: a reply that is not JSON, a reply with no text, a different k, a query that is a number or only spaces, and a search with no match. Each one is covered by a step above.

Where this goes next

The study assistant can answer from its notes and prove where each answer came from. Chapter 9 makes every model call survive failures, and Chapter 11 gives the agent search_notes and checks its final answer with the same citation rule.

Why does answer_from_notes pass best, not every chunk of the notes, to parse_cited_answer?

answer_from_notes("What is the capital of France?", ...) returns the fixed answer. How many requests did it send?

Task

In retrieval.py:

  1. Replace the unfinished body of answer_from_notes(question, notes, *, client, k). Set best = best_chunks(question, note_chunks(notes), k). If should_answer(best) is false, return CitedAnswer(answer=I_DONT_KNOW, citations=[]) without calling the model. Otherwise make one request, client.chat.completions.create(model=MODEL, messages=grounded_messages(question, best), response_format={"type": "json_object"}), and return parse_cited_answer of the reply’s content and best. Let every error reach the caller.

  2. Replace the unfinished body of search_notes(arguments). Read query = arguments.get("query"). If it is not a string, or is empty after .strip(), raise ValueError("search_notes needs a query string, such as copying a list."). Otherwise find best_chunks(query, note_chunks(load_notes()), K), and return "No note matches that search." when there are none, or format_sources of them.

  3. Replace SEARCH_NOTES_DESCRIPTION = {} with a tool description named search_notes, with a description and one required string parameter query that has a description of its own, and "additionalProperties": False.

Use the client you are given and the module’s K, I_DONT_KNOW and NOTES_DIR (through load_notes) by name. Run python retrieval.py to answer three questions from the notes, then check your work.