Answering from Your Notes · practice
Check That the Answer Cites Real Sources
The model has cited 03-lists#4. Should your program believe it? The model chose that label, and a model can write any text, including a label it was never sent. This lesson makes the program decide: it validates the reply’s shape with a pydantic model, then checks every citation against the labels it really sent.
Write CitedAnswer
Open answer_models.py. It holds the placeholder:
# Lesson 6: replace the next line with your CitedAnswer model.
CitedAnswer = None
Replace both lines with a pydantic model in the form you wrote QuizSet in Chapter 4: a class line naming BaseModel, and one hinted field per key of the JSON, with no default values, so both are required:
class CitedAnswer(BaseModel):
answer: str
citations: list[str]
retrieval.py imports CitedAnswer from this file, so save it before you run python retrieval.py.
Valid is not the same as true
CitedAnswer.model_validate_json(text) checks the shape, as QuizSet.model_validate_json did in Chapter 4: the text must be JSON, answer must be a string and citations a list of strings. Any other text raises ValidationError, which parse_cited_answer lets reach its caller.
A reply can pass that check and still cite 05-functions#6 when you sent 05-functions#5, 01-variables#2 and 05-functions#4. The shape is right; the citation is not real. A fixed model such as CitedAnswer cannot catch it, because which labels are real changes with every request. Your program knows exactly which chunks it sent, so it can.
Build parse_cited_answer
parse_cited_answer(text, chunks), where chunks are the chunks that were sent:
answer = CitedAnswer.model_validate_json(text). Notry: aValidationErrorshould reach the caller.Build the list of labels of
chunkswithchunk_label.For each citation in
answer.citations, in order: if it is not in that list,raise ValueError(f"The answer cites {citation}, which is not one of its sources."). The first unknown citation stops the check.Return
answer.
Compare whole labels. Checking whether the citation appears somewhere in the sources text would let 03-lists or lists pass, and a citation with brackets is not a label either: GROUNDED_SYSTEM asks for labels without them.
An empty citations list passes. It is how the model says the sources do not answer the question, as GROUNDED_SYSTEM tells it to. A program that refused it would punish the model for being honest.
Run it
Run python retrieval.py. The demonstration checks the copy question’s reply, then sends the default-argument question, “Why should a default argument never be a list?”, with its best three chunks:
parse_cited_answer accepted it. Citations: ['03-lists#4']
The default-argument question, sent with 05-functions#5, 01-variables#2, 05-functions#4:
{"answer": "The default is created once, when the function is defined, so every call that relies on it shares the same list. Use None as the default and create a new list inside the function.", "citations": ["05-functions#5", "05-functions#6"]}
parse_cited_answer refused it: The answer cites 05-functions#6, which is not one of its sources.
Should the model be asked?
Run stopped at the next function to build: should_answer is built in Lesson 7.
The scripted model plays a model that cites a paragraph it was not given. 05-functions#6 is a real paragraph of the functions note, but it was not one of the three sources, so the model cannot have read it in this request. parse_cited_answer refuses the whole reply, although its first citation and its text are fine: an answer that misreports its sources cannot be trusted as a whole. What to do next is the caller’s choice. The capstone in Chapter 11 tries once more with feedback, as Chapter 4 did for invalid quizzes; this chapter lets the error reach the caller.
The reply {"answer": "...", "citations": ["05-functions#6"]} passes CitedAnswer.model_validate_json. Why does parse_cited_answer still refuse it?
The reply is {"answer": "The sources do not say.", "citations": []}. What does parse_cited_answer do?
Task
In
answer_models.py, replaceCitedAnswer = Nonewith a pydantic modelclass CitedAnswer(BaseModel):with two fields and no defaults:answer: strandcitations: list[str].In
retrieval.py, replace the unfinished body ofparse_cited_answer(text, chunks). Validate the text withCitedAnswer.model_validate_json(text)and let anyValidationErrorreach the caller. Then check each citation, in order: if it is not thechunk_labelof one ofchunks, raiseValueError(f"The answer cites {citation}, which is not one of its sources."). Otherwise return theCitedAnswer.
An empty citations list is valid. Run python retrieval.py to see one reply accepted and one refused, then check your work.