0%

Testing LLM Code · practice

Build a Small Eval Set

Your tests now say whether the code is right: it builds the right requests, handles every kind of reply, and keeps its promises for any answer. They cannot say whether the answers are good. Is K = 3 enough passages? Is MIN_SCORE = 2 too strict? Is the system message clear? Each of those is a change you might make, and one answer before and one after cannot tell you whether it helped.

An eval can. It is a fixed set of questions, each with what a good result must contain, run through the program and scored as a pass rate. Change the prompt, run it again, and compare the numbers.

The ten cases

EVAL_CASES in evals.py holds ten questions about the notes, each with the note a good answer comes from:

{"question": "How do I make an independent copy of a list?", "note_id": "03-lists"}

The cases are supplied, not written by you, because the scripted model can only answer questions it has rows for. Each question is about a different note, from strings to modules to testing.

Answering a case

answer_case(question, notes, *, client, k, system, budget), also supplied, answers one question. Read it in evals.py: it is Chapter 8’s answer_from_notes with two changes. The system message is the system argument instead of always GROUNDED_SYSTEM, so Lesson 7 can try another one. And the call goes through Chapter 9’s call_model, with MAX_ATTEMPTS, time.sleep and the budget it is given, so a live run waits out a rate limit and every reply’s tokens are counted. Like answer_from_notes, it abstains without asking the model when the best passage scores below MIN_SCORE, and it raises a ValueError when the reply is not valid or cites a passage it was not sent.

Two scores

A case can go wrong in two places, so the eval scores each one:

  • Retrieval: did the passages chosen for the question include one from the right note? If not, the model never saw the answer.

  • Citations: did the answer cite a passage from the right note? That needs retrieval to have found it, and then the model to have used it and cited it correctly.

Retrieval can find the note while the answer still fails, and an answer can fail while retrieval was fine. With one score you could not tell which part to fix.

Build the three functions

In evals.py, write:

  1. retrieval_hit(case, chunks): return True when the note_id of any chunk in chunks equals case["note_id"], otherwise False.

  2. citation_hit(case, answer): return True when any citation in answer.citations starts with case["note_id"] followed by #, otherwise False. Check for case["note_id"] + "#", not the bare id, so a case about 03-list does not match 03-lists#4.

  3. run_eval(cases, notes, *, client, k, system):

    • start with budget = {"limit": EVAL_BUDGET, "used": 0} and result = {"cases": len(cases), "retrieval": 0, "citations": 0, "misses": [], "tokens": 0};

    • for each case, take its question, find best_chunks(question, note_chunks(notes), k) and call retrieval_hit on it;

    • then, inside try, call answer_case(question, notes, client=client, k=k, system=system, budget=budget) and citation_hit on the answer. On except ValueError: the reply was rejected, which counts as a citation miss;

    • add one to result["retrieval"] and to result["citations"] for each check that hit, and append the question to result["misses"] when either one missed;

    • after the loop, set result["tokens"] to budget["used"] and return result.

Catch only ValueError. A call that failed with a 500, or a run that hit its budget, is not a fact about answer quality, so those errors propagate and stop the eval.

Run the eval

Run python evals.py. After the recording from Lesson 4, the demonstration runs your run_eval over EVAL_CASES with GROUNDED_SYSTEM and K, and prints the result with the supplied print_eval:

The eval with GROUNDED_SYSTEM:
GROUNDED_SYSTEM: retrieval 9 of 10, citations 7 of 10, 3148 tokens
  missed: Why should a default argument never be a list?
  missed: How does pytest find my tests?
  missed: How do I filter items with a list comprehension?

A comparison of two system messages follows; it is Lesson 7’s. Nine of ten for retrieval, seven of ten for citations, and three misses, each for a different reason:

  • Why should a default argument never be a list? Retrieval found the functions note, but the reply cites 05-functions#5 and 05-functions#6, and the second was not among the three passages sent (05-functions#5, 01-variables#2 and 05-functions#4). parse_cited_answer rejects it, so it is a citation miss. This is the reply Chapter 8 rejected too.

  • How does pytest find my tests? The best passage scores only 1, below MIN_SCORE, so answer_case abstains and asks nothing. The testing note was among the three passages, at second place, so retrieval counts as a hit, but “I don’t know” cites nothing.

  • How do I filter items with a list comprehension? Retrieval missed: the question says comprehension, the note says comprehensions, and Chapter 8’s word matching treats them as different words. The model was sent passages from the lists and loops notes, said they do not answer the question, and cited nothing. Both scores miss.

The eval made 9 requests, since the pytest case asked nothing, and they used 3148 tokens in all.

The table puts the two scores side by side for all ten cases. Read down a column for its total, and across the three rows with a miss: each one failed in a different place.

Of the ten eval cases, retrieval hits nine and citations hit seven; one answer cites a passage that was not sent, one abstains, and one misses both because the note says comprehensions.

Add a case of your own

You can add a case to EVAL_CASES and run the eval again. Retrieval is scored the same way for any question, so a new case shows you what the word matching finds. The scripted model, though, only has answers for these ten. For any other question it sends its echo, which is not JSON, so the reply is rejected and the case scores a citation miss. That is a fact about the scripted model, not about your question. Checking your work uses its own cases, so a case you add changes nothing there.

How your work is checked

The checks use unseen notes and cases and the fake client, with the supplied answer_case installed by the checker (with a sleep that only records). They check retrieval_hit and citation_hit on near misses and empty lists; and run_eval’s counts, its misses in case order, its tokens, that an abstaining case makes no request, that a rejected reply counts as a miss without stopping the eval, that a 500 on every attempt propagates, that each run starts a fresh budget from EVAL_BUDGET, and that k and system reach the requests.

The case How do I filter items with a list comprehension? misses on retrieval and on citations. What should you look at first?

Why does run_eval catch only ValueError around answer_case?

Task

In evals.py, write:

  1. retrieval_hit(case, chunks): True when any chunk’s note_id equals case["note_id"].

  2. citation_hit(case, answer): True when any citation in answer.citations starts with case["note_id"] + "#".

  3. run_eval(cases, notes, *, client, k, system): start from budget = {"limit": EVAL_BUDGET, "used": 0} and result = {"cases": len(cases), "retrieval": 0, "citations": 0, "misses": [], "tokens": 0}. For each case, score retrieval_hit on best_chunks(question, note_chunks(notes), k); then, inside try, call answer_case(question, notes, client=client, k=k, system=system, budget=budget) and score citation_hit, where a ValueError counts as a miss. Count each hit, append the question to misses when either check missed, set result["tokens"] = budget["used"] at the end, and return result. Catch nothing but ValueError.

Run python evals.py to see the eval’s scores, then check your work.