0%

Answering from Your Notes · practice

Pick the Best Few Chunks

Every chunk now has a score. The program should send the model only the few chunks that score highest, and it should pick them the same way every time, even when two chunks score the same. This lesson does both with one sort.

How Python compares tuples

Python compares two tuples item by item. It compares the first items, and only if they are equal does it move on to the second items, then the third, and so on:

print((2, "b") < (3, "a"))
print((2, "b") < (2, "c"))
print((2, "03-lists", 9) < (2, "03-lists", 10))

The first comparison is decided by 2 < 3, so the letters never matter. The second has a tie on 2 and is decided by "b" < "c". The third has ties on 2 and on the note id, and is decided by the numbers, compared as numbers: 9 comes before 10. Strings compare character by character, so "03-lists#9" would come after "03-lists#10": the first difference is "9" against "1". That is one reason to sort on the note id and the number separately, not on a label.

So sorting a list of tuples sorts by the first item, and uses the later items only to break ties:

ranked = [(2, "04-loops", 7), (3, "03-lists", 4), (2, "01-variables", 2)]
ranked.sort()
print(ranked)

Highest scores first

.sort() puts small values first, but the best chunk has the highest score. Store the score with a minus sign, and the highest score becomes the smallest number: a score of 3 is stored as -3, which sorts before -2. Ties then go to the earlier note id, and then to the lower chunk number. That order is arbitrary, but it is fixed, so the same question always gets the same sources in the same order.

Build best_chunks

best_chunks(question, chunks, k):

  1. Start a new, empty list. For each chunk, compute score = score_chunk(question, chunk["text"]). If the score is above 0, append (-score, chunk["note_id"], chunk["number"], chunk["text"]). A chunk that shares no word with the question is never a source, however few chunks match.

  2. Sort the new list with .sort().

  3. Take the first k tuples with a slice, ranked[:k]. A slice past the end of a list just stops, so with fewer than k matches you get them all.

  4. For each tuple, build a dictionary by position: {"note_id": item[1], "number": item[2], "text": item[3], "score": -item[0]}. The second minus sign turns the stored score back into a positive number. Return the list of these dictionaries.

The text is the last item only so that it travels with the chunk into the result. A note id and a number never repeat together, so the sort never has to compare two texts.

Build new tuples and new dictionaries, and leave the list you were given alone. Sorting chunks itself would reorder the caller’s list, and adding a "score" key to its dictionaries would change them.

Run it

Run python retrieval.py. The demonstration asks for the best three chunks for two questions:

The best 3 chunks for: How do I make an independent copy of a list?
  03-lists#4 scores 3
  01-variables#2 scores 2
  04-loops#7 scores 2
The best 3 chunks for: copy a list
  01-variables#2 scores 2
  03-lists#4 scores 2
  04-loops#7 scores 2
Run stopped at the next function to build: grounded_messages is built in Lesson 5.

For the copy question, 03-lists#4 wins with 3. The two chunks that score 2 are ordered by note id: 01-variables before 04-loops. copy a list has only two words after the stop word a, and three chunks share both of them, a three-way tie at 2. The note id decides, so 01-variables#2 comes first, although 03-lists#4 is the paragraph that answers the question. That is a limit of counting words: it ranks, but it does not understand. Sending three chunks instead of one leaves the model to find the answer among them.

Why does best_chunks store -score in each tuple?

Two chunks both score 2: 04-loops#7 and 01-variables#2. Which comes first in the result, and why?

Task

In retrieval.py, replace the unfinished body of best_chunks(question, chunks, k).

  1. For each chunk whose score_chunk(question, chunk["text"]) is above 0, append the tuple (-score, note_id, number, text) to a new list.

  2. Sort that list with .sort().

  3. For each of the first k tuples, build {"note_id": ..., "number": ..., "text": ..., "score": ...} from its items by position, with the score as a positive number, and return the list of these dictionaries.

Do not change or reorder the list of chunks you are given. Run python retrieval.py to see the best three chunks for two questions, then check your work.