Testing LLM Code
Compare Two Prompts with the Eval
An eval gives one pair of numbers for one version of the program. Its use is comparison: run it, change one thing, run it again. This lesson changes the system message and reads what the numbers can and cannot say.
Two system messages
prompts.py has a second grounded system message, ALT_GROUNDED_SYSTEM, next to Chapter 8’s GROUNDED_SYSTEM:
ALT_GROUNDED_SYSTEM = (
"Answer the question. The user's message includes some passages from the "
"course notes that may help. Reply with JSON only: an object with answer "
"and citations, a list of passage labels."
)
Read it against GROUNDED_SYSTEM. It still asks for JSON with answer and citations, but it never says to answer only from the passages, never says that each passage starts with its label in square brackets, never says to cite only those labels, and never says what to do when the passages do not answer. The passages “may help”, which invites the model to answer from what it already knows and to cite whatever looks plausible.
Run the comparison
Run python evals.py. After Lesson 6’s eval, the demonstration calls the supplied compare_prompts(). It runs your run_eval over the same ten cases twice, once with each system message, and prints both:
Two system messages compared:
GROUNDED_SYSTEM: retrieval 9 of 10, citations 7 of 10, 3148 tokens
missed: Why should a default argument never be a list?
missed: How does pytest find my tests?
missed: How do I filter items with a list comprehension?
ALT_GROUNDED_SYSTEM: retrieval 9 of 10, citations 5 of 10, 2494 tokens
missed: Why should a default argument never be a list?
missed: What does a finally block do?
missed: How does range work in a for loop?
missed: How does pytest find my tests?
missed: How do I filter items with a list comprehension?
Retrieval is 9 of 10 both times, as it must be: retrieval happens before the model is asked, and the system message does not change which passages are chosen. Citations drop from 7 to 5. The two extra misses are the finally and range questions. Under ALT_GROUNDED_SYSTEM their replies cite 07-errors#5 and 04-loops#1, passages that were not among the ones sent, so the citation check rejects them. The alternative also used fewer tokens, 2494 against 3148, because its system message is shorter and is sent with every question.
So GROUNDED_SYSTEM is better? On this run, with this model, the numbers say so. Here is what they do not say.
What this proves on the scripted model
Nothing about the prompts. pl-scripted does not read a system message the way a model does. It answers from fixed rows, and someone wrote the finally and range rows to cite a wrong passage when the system message contains may help. The comparison shows how the method works: the same cases, one change, two pairs of numbers, and the misses that changed. It cannot show that one prompt is better, because the scripted model’s answers were chosen in advance.
What a difference on ten cases means
With ten cases, one case moves a score by ten percentage points. Seven against five is two cases. On a live model, two cases could change between two runs of the same prompt, because, as Lesson 1 showed, a live model can answer the same request differently each time. A small difference on a small set is as likely to be that variation as an effect of the change.
Before you trust a comparison on a live model:
run each prompt more than once, and look at how much its own scores move between runs;
trust only a difference larger than that movement;
read the misses that changed, not only the counts, because a miss can be the eval’s fault, such as a case whose question nothing in the notes answers;
use more cases when you can, so one case counts for less.
Try it live
This is optional. If your account has live access, you can run the comparison on pl-live. In the
python -c "import evals; evals.MODEL = 'pl-live'; evals.compare_prompts()"
answer_case reads the model name from evals.MODEL, so this one assignment sends every eval question to the live model. It uses 18 of your daily live requests, 9 per prompt (the pytest case asks nothing). That is more than the per-minute limit allows, so some calls get a 429 whose Retry-After is the number of seconds until the next minute starts, and call_model waits it out: the run takes one to two minutes and prints nothing until both evals have finished.
One live run per prompt gives you one pair of numbers each, and you see how much they vary only when you run them again on another day. Do not try several runs in one session; the daily limit will stop you first.
The usual notes about pl-live apply. A 403 means your account has no live access. A 429 that call_model gives up on means you reached the daily limit. A read timeout is a normal outcome on a live model, which can think longer than this course’s read bound; call_model retries it, and after three timeouts the run stops with the error. A reply that stops at max_tokens raises ReplyCutOff. run_eval catches only ValueError, so the run stops there. That is rare on these short answers. If it happens, run the comparison again another day.
On one live run, prompt A scores 7 of 10 for citations and prompt B scores 5 of 10. What does that tell you?
compare_prompts() on pl-scripted gives GROUNDED_SYSTEM 7 of 10 and ALT_GROUNDED_SYSTEM 5 of 10. Why does that prove nothing about the two prompts?
You want to know whether a new system message really is better on the live model. What should you do before trusting the comparison?