Answering from Your Notes · practice
Score a Chunk Against a Question
A program cannot ask a chunk whether it answers a question. It can count something. This lesson counts the words a chunk shares with the question, with one exact rule, so the same question and chunk always get the same score.
The rule
words(text) turns any text into a set of words:
Lowercase the text, so
Listandlistare the same word.Replace each character of
PUNCTUATIONwith a space.PUNCTUATIONis a constant at the top ofretrieval.py:.,:;!?"', the backtick, brackets of every kind, and<>#*=+-/. Loop over it, one character at a time, withtext = text.replace(character, " ").Split the result with
.split(), with no argument. It splits on any run of spaces, tabs and line breaks, and never gives an empty word.Keep each word that is not in
STOP_WORDS, and return the words you kept as a set.
STOP_WORDS holds words such as the, how, of, what and note that appear in almost every question and every paragraph. Matching on them would make every chunk look relevant. A set keeps each word once, however often the text repeats it.
score_chunk(question, text) is then the number of words of words(question) that are also in words(text). Loop over the question’s words and add 1 for each one that is in the chunk’s set.
Why punctuation becomes spaces
The notes are full of code, and code glues words together with punctuation. Replacing each punctuation character with a space pulls them apart:
text = "Use `scores.copy()` or `list(scores)`."
for character in ".()`":
text = text.replace(character, " ")
print(text.split())
The dot, the brackets and the backticks are gone, so copy and list are words of their own and can match a question that asks how to copy a list. Removing punctuation only from the ends of each word would leave scores.copy as one word, which no question contains.
The hyphen is in PUNCTUATION too, so the note id 03-lists becomes the two words 03 and lists. The underscore is not, so a name such as max_messages stays one word.
What the rule cannot do
The rule matches spelling, not meaning. tuples does not match tuple, and change does not match changes. A question that says “duplicate” will not find a paragraph that says “copy”. This is the price of a rule you can read in four lines, and Chapter 12 explains embeddings, which match on meaning instead.
Build words and score_chunk
Write both in retrieval.py, following the rule above. Use PUNCTUATION and STOP_WORDS by name: the checks give them other values to make sure your functions read them. score_chunk should build both word sets with your words function.
Run it
Run python retrieval.py. The demonstration prints the words of the copy question, sorted so the output is the same every time (a set has no fixed order), and the scores of two chunks:
The words of the question, sorted: ['copy', 'independent', 'list', 'make']
03-lists#4 scores 3
01-variables#2 scores 2
Run stopped at the next function to build: best_chunks is built in Lesson 4.
How, do, I, an, of and a are stop words, and the question mark is punctuation, so four words are left. 03-lists#4 contains copy (from scores.copy()), list and independent: 3. 01-variables#2 says “Assignment does not copy anything” and talks about the same list: 2. It shares words with the question without answering it, which is normal for word matching and is why the next lesson keeps a few chunks, not just one.
The question is "list" and a chunk says "list" three times. What does score_chunk return?
A chunk says "A tuple cannot change." Why does it score 0 for the question "Tell me about tuples."?
Task
In retrieval.py, replace the unfinished bodies of two functions.
words(text): lowercasetext; replace each character ofPUNCTUATIONwith a space, using.replacein a loop; split the result with.split()and no argument; return the words that are not inSTOP_WORDS, as a set.score_chunk(question, text): return how many words ofwords(question)are also inwords(text).
Use the module’s PUNCTUATION and STOP_WORDS by name; the checks change them. Run python retrieval.py to see the scores of two chunks, then check your work.