Answering from Your Notes · practice
Say "I Don't Know" When Nothing Matches
best_chunks returns whatever matches, however weakly. For “Tell me about tuples.” it returns one chunk that shares a single word with the question. Sending that to the model invites trouble: the sources do not answer the question, so a model that wants to be helpful may answer from memory instead, which is the problem Lesson 1 started with. And every request costs tokens. This lesson adds a rule for when the program should not ask at all.
A threshold
MIN_SCORE at the top of retrieval.py is 2. A question is worth sending only when its best chunk shares at least two words with it:
no chunk scores above 0, so
bestis empty: do not ask;the best chunk scores below
MIN_SCORE: do not ask;otherwise: ask.
best is already sorted, best first, so only best[0] needs checking. The other chunks can score less; they come along as extra sources.
When the rule says no, the project in the next lesson answers with the fixed text in I_DONT_KNOW, “I don’t know. The notes don’t cover that.”, without any request. An honest “I don’t know” is better than a confident answer that did not come from the notes, and it costs nothing.
Two is a small number chosen for these short notes and questions. Too high, and good questions go unanswered; too low, and weak matches get through. Chapter 10 measures choices like this on a set of test questions instead of guessing.
Build should_answer
should_answer(best) returns False when best is empty or when best[0]["score"] is below MIN_SCORE, and True otherwise. Check for an empty list before you read best[0], or an empty list raises IndexError. You can write it as an if with or, or as one expression with len(best) > 0 and .... Either way, return True or False, not the list itself: best and ... returns [] for an empty list.
Use MIN_SCORE by name, so changing the constant changes the rule.
Run it
Run python retrieval.py. The demonstration finds the best chunks for three questions and asks your rule about each:
Should the model be asked?
What is the capital of France? best score 0: False
Tell me about tuples. best score 1: False
How do I make an independent copy of a list? best score 3: True
Answers from the notes:
You: How do I make an independent copy of a list?
Run stopped at the next function to build: answer_from_notes is built in Lesson 8.
capital and France appear in no note, so best is empty. “Tell me about tuples.” matches one chunk, 06-dictionaries#4, on the one word tuples: that paragraph says a dictionary key can be a tuple. It says nothing about what a tuple is, and 1 is below 2. The copy question scores 3 and is sent.
Why skip the model when the best chunk scores 1?
best holds two chunks that score 5 and 1, and MIN_SCORE is 2. What does should_answer(best) return?
Task
In retrieval.py, replace the unfinished body of should_answer(best), where best is a list from best_chunks. Return False when best is empty or when best[0]["score"] is below MIN_SCORE. Otherwise return True.
Use the module’s MIN_SCORE by name; the checks change it. Run python retrieval.py to see the rule on three questions, then check your work.