Capstone
Run Your Eval and Read the Result
The study assistant is finished, and it has two kinds of check. Chapter 10’s tests say whether the code does what it should. The eval says how often retrieval finds the right note and how often the answer cites it. This lesson runs the eval with the program’s settings, makes two changes a person might try to improve it, and reads what the numbers do and do not say.
Run it
Run python stages.py. After Lesson 6’s block, stages.py runs the eval three times, with Chapter 10’s run_eval and print_eval from evals.py:
Lesson 8: the eval with the capstone's settings
K = 3, MIN_SCORE = 2: retrieval 9 of 10, citations 7 of 10, 3148 tokens
missed: Why should a default argument never be a list?
missed: How does pytest find my tests?
missed: How do I filter items with a list comprehension?
k = 5: retrieval 10 of 10, citations 7 of 10, 4084 tokens
missed: Why should a default argument never be a list?
missed: How does pytest find my tests?
missed: How do I filter items with a list comprehension?
MIN_SCORE = 1: retrieval 9 of 10, citations 8 of 10, 3460 tokens
missed: Why should a default argument never be a list?
missed: How do I filter items with a list comprehension?
The first line is the program as it is: K is 3 and MIN_SCORE is 2 in retrieval.py, and the system message is GROUNDED_SYSTEM. It is Chapter 10’s result: retrieval found the expected note for 9 of the 10 questions, and 7 answers cited it.
The second run passes k=5 to run_eval, so each question gets the best five passages instead of three. The third sets retrieval.MIN_SCORE = 1 for one run and puts it back afterwards, so a question whose best passage shares only one word with it is answered instead of getting “I don’t know”. That assignment is in supplied code you only read, made for this experiment.
A bigger k
With five passages, retrieval found the expected note for all ten questions. The one it had missed, How do I filter items with a list comprehension?, needs the comprehensions note, whose passages scored 2 and came fourth and fifth: the question’s word comprehension does not match the note’s comprehensions, so other passages with two shared words ranked above them.
The citation score did not move. The answer to that question still says the sources do not answer it. Finding a passage and using it are different things, which is why the eval keeps two scores. On the scripted model, the answer is a fixed row, so nothing about the passages could have changed it. On a live model the extra passages might help, or might not: that would need several live runs to tell.
What k = 5 certainly did was cost more. Every grounded prompt now carries five passages instead of three, so the run used 4,084 tokens instead of 3,148, about 30% more, for every question, including the eight that were already answered well.
A lower threshold
With MIN_SCORE = 1, the pytest question is no longer turned away. Its best passage, 12-testing#1, shares one word with the question, so the program now asks the model, and the scripted model’s answer cites 12-testing#1. Citations rose from 7 to 8.
The cost is the reason the threshold exists. Chapter 8 set it at 2 because a passage that shares a single word with a question is often about something else, and a model given weak passages tends to answer from memory instead, which is exactly what grounding is meant to prevent. A lower threshold also sends a request, and its tokens, for every weak match. Here one question gained; with real users, many questions that should get “I don’t know” would get an answer instead.
What is a real measurement here
The retrieval score is a real measurement, even on the scripted model. It is computed by your own code, best_chunks over the twelve real notes, so k = 5 really does find the comprehensions note, and that would be true with any model.
The citation score is not, on the scripted model. Its answers are fixed rows, written to show what an eval can catch; the only real part is whether a question was sent at all, which is why MIN_SCORE = 1 moved it. To judge whether a change makes answers better, run the eval with a live model, several times, on a larger set of questions, and keep a change only if it wins clearly every time. Chapter 10 Lesson 7 showed how to run the eval against pl-live, and what it costs in daily requests.
The eval also has a limit of its own. It scores Chapter 8’s way of answering, where the program chooses the passages. ask’s agent writes its own search query, so the eval measures the retrieval ask uses, not every step of ask.
At home, K and MIN_SCORE are at the top of retrieval.py. K decides how many passages search_notes returns to ask’s agent and how many the eval sends. MIN_SCORE decides when Chapter 8’s answer_from_notes and the eval answer “I don’t know”; in ask, the model makes that call itself from what its searches returned. Change one, run the eval, and read both scores before keeping it.
With k = 5, retrieval went from 9 to 10 of 10 and citations stayed at 7. What does that say?
Lowering MIN_SCORE from 2 to 1 rescued the pytest question. What does it cost?
An eval of the study assistant runs on the scripted model and reports a retrieval score and a citation score. Which one is a real measurement?