Structured Output · capstone
Capstone project: Project: Turn a Note into Quiz Questions
This project puts the chapter together. Give it a course note, and it asks the model for a quiz, validates the reply, asks once more if the reply is invalid, and prints the quiz in a form a learner can take:
>>> quiz = make_quiz(open("notes/03-lists.md", encoding="utf-8").read())
>>> print(format_quiz(quiz))
1. For scores = [90, 72, 85], what is scores[-1]?
a) 90
b) 72
c) 85
2. What does scores.append(60) return?
a) The new list
b) None
c) 60
3. Which line makes an independent copy of scores?
a) backup = scores.copy()
b) backup = scores
c) backup = scores.append(0)
Answers: 1c, 2b, 3a
You build two functions: make_quiz, which gets a valid QuizSet, and format_quiz, which turns one into that text.
Build make_quiz
make_quiz(note_text, *, session=None) takes the session=None seam from Chapter 1, so checking your work can pass a fake session that returns prepared replies. Pass session=session to every send_body call.
body = quiz_request(note_text).text = answer_text(send_body(body, session=session)).Return
parse_quiz(text).If step 3 raises
ValidationError, buildretry_request(body, text, error), send it withsend_body, and returnparse_quizof that reply’s text.
Catch only ValidationError, and only around parse_quiz. A request that fails, such as a 500 from the service, raises from send_body before there is any reply to correct; asking again with feedback about an invalid reply would make no sense, so let that error reach the caller. If the retry’s reply is invalid as well, its ValidationError reaches the caller too, as Lesson 8 explained: one retry, then stop.
One way to keep the second ValidationError from being caught by the first except is to return from inside the try and send the retry after it:
try:
return parse_quiz(text)
except ValidationError as error:
retry = retry_request(body, text, error)
The line that sends retry comes after the try statement, so nothing catches what it raises.
Build format_quiz
format_quiz(quiz) takes a QuizSet and returns the quiz as one string, with no newline at the end:
Number the questions from 1 with a counter: set
number = 0before the loop overquiz.questions, and add 1 at the start of each pass.For each question, a line
f"{number}. {question.question}", then one line per choice,f" {letter}) {choice}", indented by three spaces. The letters come from"abcdefgh"by position: the first choice getsa, the secondb, and so on.A blank line between questions.
After the last question, a blank line, then
Answers:followed by one entry per question, the number and the letter of its correct choice, such as1c, joined with", ".
A convenient way to get the letters is to slice "abcdefgh" to the number of choices, letters = "abcdefgh"[: len(question.choices)]. Then letters[0] is a and letters[question.answer_index] is the correct letter. An answer_index past the last choice then raises IndexError, like correct_choice in Lesson 4. Leave that uncaught: a broken question should not be printed with a made-up answer.
Collecting each question’s lines in a list and joining them with "\n", then joining the questions with "\n\n", gives the blank lines without counting them.
Run it
Run python structured.py. The demonstration makes a quiz from notes/03-lists.md and prints it. Then it calls make_quiz("give me broken json"). note_message puts that text inside the note tags, and the scripted model answers this one prompt wrongly on purpose the first time, so your make_quiz has to retry before it gets a valid quiz:
The quiz for notes/03-lists.md:
1. For scores = [90, 72, 85], what is scores[-1]?
a) 90
b) 72
c) 85
2. What does scores.append(60) return?
a) The new list
b) None
c) 60
3. Which line makes an independent copy of scores?
a) backup = scores.copy()
b) backup = scores
c) backup = scores.append(0)
Answers: 1c, 2b, 3a
A quiz whose first reply is invalid:
1. What does len() return?
a) The number of items
b) The last item
c) The first item
Answers: 1a
The second quiz only appears if the retry worked; without it, the first reply’s ValidationError would end the run with a traceback. The demonstration cannot show you the other cases, so think them through before you check your work: a valid first reply needs exactly one request; a second invalid reply must raise, not be retried again; a failed request must not be retried at all. Checking your work tries each of them with a fake session, along with quizzes of one question and of four choices. If you get stuck, ask Monty about one case at a time.
Where this goes next
make_quiz and format_quiz turn any note into a checked, printable quiz with all the work done by hand: the request body, the JSON mode, the schema in the prompt, validation and the retry. Chapter 5 moves this onto the openai client, whose parse helper sends a strict schema built from QuizSet and validates the reply for you. Because you built each step yourself, you will know what it is doing.
In make_quiz, the first request fails with a 500 from the service. What should happen?
A question has four choices and answer_index 3. Which entry does it get on the answers line, if it is question 2?
Task
In structured.py, replace the unfinished bodies of make_quiz(note_text, *, session=None) and format_quiz(quiz).
make_quiz:
body = quiz_request(note_text), thentext = answer_text(send_body(body, session=session)).Return
parse_quiz(text). If it raisesValidationError, sendretry_request(body, text, error)once, with the same session, and returnparse_quizof that reply’s text.A second
ValidationError, and any error fromsend_body, reach the caller.
format_quiz returns one string, with no newline at the end:
for each question, numbered from 1:
f"{number}. {question.question}", wherequestionis one item ofquiz.questions, then one line per choice,f" {letter}) {choice}"(three spaces), with lettersa,b,cand so on taken by position from"abcdefgh";a blank line between questions;
then a blank line and
Answers:followed by the number and correct letter of each question, such as1c, joined with", ".
Run python structured.py to see two quizzes, the second after a retry, then check your work.