Testing LLM Code · practice
Build a Small Eval Set
Your tests now say whether the code is right: it builds the right requests, handles every kind of reply, and keeps its promises for any answer. They cannot say whether the answers are good. Is K = 3 enough passages? Is MIN_SCORE = 2 too strict? Is the system message clear? Each of those is a change you might make, and one answer before and one after cannot tell you whether it helped.
An eval can. It is a fixed set of questions, each with what a good result must contain, run through the program and scored as a pass rate. Change the prompt, run it again, and compare the numbers.
The ten cases
EVAL_CASES in evals.py holds ten questions about the notes, each with the note a good answer comes from:
{"question": "How do I make an independent copy of a list?", "note_id": "03-lists"}
The cases are supplied, not written by you, because the scripted model can only answer questions it has rows for. Each question is about a different note, from strings to modules to testing.
Answering a case
answer_case(question, notes, *, client, k, system, budget), also supplied, answers one question. Read it in evals.py: it is Chapter 8’s answer_from_notes with two changes. The system message is the system argument instead of always GROUNDED_SYSTEM, so Lesson 7 can try another one. And the call goes through Chapter 9’s call_model, with MAX_ATTEMPTS, time.sleep and the budget it is given, so a live run waits out a rate limit and every reply’s tokens are counted. Like answer_from_notes, it abstains without asking the model when the best passage scores below MIN_SCORE, and it raises a ValueError when the reply is not valid or cites a passage it was not sent.
Two scores
A case can go wrong in two places, so the eval scores each one:
Retrieval: did the passages chosen for the question include one from the right note? If not, the model never saw the answer.
Citations: did the answer cite a passage from the right note? That needs retrieval to have found it, and then the model to have used it and cited it correctly.
Retrieval can find the note while the answer still fails, and an answer can fail while retrieval was fine. With one score you could not tell which part to fix.
Build the three functions
In evals.py, write:
retrieval_hit(case, chunks): returnTruewhen thenote_idof any chunk inchunksequalscase["note_id"], otherwiseFalse.citation_hit(case, answer): returnTruewhen any citation inanswer.citationsstarts withcase["note_id"]followed by#, otherwiseFalse. Check forcase["note_id"] + "#", not the bare id, so a case about03-listdoes not match03-lists#4.run_eval(cases, notes, *, client, k, system):start with
budget = {"limit": EVAL_BUDGET, "used": 0}andresult = {"cases": len(cases), "retrieval": 0, "citations": 0, "misses": [], "tokens": 0};for each case, take its question, find
best_chunks(question, note_chunks(notes), k)and callretrieval_hiton it;then, inside
try, callanswer_case(question, notes, client=client, k=k, system=system, budget=budget)andcitation_hiton the answer. Onexcept ValueError:the reply was rejected, which counts as a citation miss;add one to
result["retrieval"]and toresult["citations"]for each check that hit, and append the question toresult["misses"]when either one missed;after the loop, set
result["tokens"]tobudget["used"]and returnresult.
Catch only ValueError. A call that failed with a 500, or a run that hit its budget, is not a fact about answer quality, so those errors propagate and stop the eval.
Run the eval
Run python evals.py. After the recording from Lesson 4, the demonstration runs your run_eval over EVAL_CASES with GROUNDED_SYSTEM and K, and prints the result with the supplied print_eval:
The eval with GROUNDED_SYSTEM:
GROUNDED_SYSTEM: retrieval 9 of 10, citations 7 of 10, 3148 tokens
missed: Why should a default argument never be a list?
missed: How does pytest find my tests?
missed: How do I filter items with a list comprehension?
A comparison of two system messages follows; it is Lesson 7’s. Nine of ten for retrieval, seven of ten for citations, and three misses, each for a different reason:
Why should a default argument never be a list? Retrieval found the functions note, but the reply cites
05-functions#5and05-functions#6, and the second was not among the three passages sent (05-functions#5,01-variables#2and05-functions#4).parse_cited_answerrejects it, so it is a citation miss. This is the reply Chapter 8 rejected too.How does pytest find my tests? The best passage scores only 1, below
MIN_SCORE, soanswer_caseabstains and asks nothing. The testing note was among the three passages, at second place, so retrieval counts as a hit, but “I don’t know” cites nothing.How do I filter items with a list comprehension? Retrieval missed: the question says
comprehension, the note sayscomprehensions, and Chapter 8’s word matching treats them as different words. The model was sent passages from the lists and loops notes, said they do not answer the question, and cited nothing. Both scores miss.
The eval made 9 requests, since the pytest case asked nothing, and they used 3148 tokens in all.
The table puts the two scores side by side for all ten cases. Read down a column for its total, and across the three rows with a miss: each one failed in a different place.

Add a case of your own
You can add a case to EVAL_CASES and run the eval again. Retrieval is scored the same way for any question, so a new case shows you what the word matching finds. The scripted model, though, only has answers for these ten. For any other question it sends its echo, which is not JSON, so the reply is rejected and the case scores a citation miss. That is a fact about the scripted model, not about your question. Checking your work uses its own cases, so a case you add changes nothing there.
How your work is checked
The checks use unseen notes and cases and the fake client, with the supplied answer_case installed by the checker (with a sleep that only records). They check retrieval_hit and citation_hit on near misses and empty lists; and run_eval’s counts, its misses in case order, its tokens, that an abstaining case makes no request, that a rejected reply counts as a miss without stopping the eval, that a 500 on every attempt propagates, that each run starts a fresh budget from EVAL_BUDGET, and that k and system reach the requests.
The case How do I filter items with a list comprehension? misses on retrieval and on citations. What should you look at first?
Why does run_eval catch only ValueError around answer_case?
Task
In evals.py, write:
retrieval_hit(case, chunks):Truewhen any chunk’snote_idequalscase["note_id"].citation_hit(case, answer):Truewhen any citation inanswer.citationsstarts withcase["note_id"] + "#".run_eval(cases, notes, *, client, k, system): start frombudget = {"limit": EVAL_BUDGET, "used": 0}andresult = {"cases": len(cases), "retrieval": 0, "citations": 0, "misses": [], "tokens": 0}. For each case, scoreretrieval_hitonbest_chunks(question, note_chunks(notes), k); then, insidetry, callanswer_case(question, notes, client=client, k=k, system=system, budget=budget)and scorecitation_hit, where aValueErrorcounts as a miss. Count each hit, append the question tomisseswhen either check missed, setresult["tokens"] = budget["used"]at the end, and returnresult. Catch nothing butValueError.
Run python evals.py to see the eval’s scores, then check your work.