0%

Answering from Your Notes · practice

Say "I Don't Know" When Nothing Matches

best_chunks returns whatever matches, however weakly. For “Tell me about tuples.” it returns one chunk that shares a single word with the question. Sending that to the model invites trouble: the sources do not answer the question, so a model that wants to be helpful may answer from memory instead, which is the problem Lesson 1 started with. And every request costs tokens. This lesson adds a rule for when the program should not ask at all.

A threshold

MIN_SCORE at the top of retrieval.py is 2. A question is worth sending only when its best chunk shares at least two words with it:

  • no chunk scores above 0, so best is empty: do not ask;

  • the best chunk scores below MIN_SCORE: do not ask;

  • otherwise: ask.

best is already sorted, best first, so only best[0] needs checking. The other chunks can score less; they come along as extra sources.

When the rule says no, the project in the next lesson answers with the fixed text in I_DONT_KNOW, “I don’t know. The notes don’t cover that.”, without any request. An honest “I don’t know” is better than a confident answer that did not come from the notes, and it costs nothing.

Two is a small number chosen for these short notes and questions. Too high, and good questions go unanswered; too low, and weak matches get through. Chapter 10 measures choices like this on a set of test questions instead of guessing.

Build should_answer

should_answer(best) returns False when best is empty or when best[0]["score"] is below MIN_SCORE, and True otherwise. Check for an empty list before you read best[0], or an empty list raises IndexError. You can write it as an if with or, or as one expression with len(best) > 0 and .... Either way, return True or False, not the list itself: best and ... returns [] for an empty list.

Use MIN_SCORE by name, so changing the constant changes the rule.

Run it

Run python retrieval.py. The demonstration finds the best chunks for three questions and asks your rule about each:

Should the model be asked?
  What is the capital of France? best score 0: False
  Tell me about tuples. best score 1: False
  How do I make an independent copy of a list? best score 3: True
Answers from the notes:
You: How do I make an independent copy of a list?
Run stopped at the next function to build: answer_from_notes is built in Lesson 8.

capital and France appear in no note, so best is empty. “Tell me about tuples.” matches one chunk, 06-dictionaries#4, on the one word tuples: that paragraph says a dictionary key can be a tuple. It says nothing about what a tuple is, and 1 is below 2. The copy question scores 3 and is sent.

Why skip the model when the best chunk scores 1?

best holds two chunks that score 5 and 1, and MIN_SCORE is 2. What does should_answer(best) return?

Task

In retrieval.py, replace the unfinished body of should_answer(best), where best is a list from best_chunks. Return False when best is empty or when best[0]["score"] is below MIN_SCORE. Otherwise return True.

Use the module’s MIN_SCORE by name; the checks change it. Run python retrieval.py to see the rule on three questions, then check your work.