0%

Conversations and Instructions · practice

Keep the History Within Budget

A history that grows forever will eventually be too expensive to send, and then too long for the model to read at all. The fix is to send less: keep what matters and drop the oldest turns. This lesson builds trim_history(history, max_messages), which does that by a precise rule.

Real programs usually trim by counting tokens, since tokens are what the context window and the bill are measured in. This course counts messages instead, so you can check every result by counting on your fingers. The rule is the same idea either way.

What a history looks like

The rule depends on the shape of a history, so here it is precisely. A history is:

  • an optional system message first;

  • then whole turns, each a user message followed by an assistant message;

  • optionally ending with one user message that has no reply yet. Call that a pending question. It is what next_request adds when it builds a request.

In the examples below, S is the system message, U1 and A1 are the first turn’s user and assistant messages, and P is a pending question.

The rule

  1. A max_messages below 1 raises ValueError. A limit of zero or less is a mistake in the calling code, and saying so at once beats returning something odd.

  2. The history has the shape described above.

  3. The system message, if there is one, is always kept. It holds your instructions, and dropping it would change how the model behaves.

  4. The newest messages are protected: the pending question if there is one, otherwise the last user-and-assistant turn. Without them, there is nothing for the model to answer or nothing recent to answer from.

  5. While the list is longer than max_messages and there is an older turn in front of the protected messages, drop the oldest turn, both of its messages together.

  6. Always return a new list, even when nothing is dropped. If the limit cannot be met, the result is the system message plus the protected messages.

Step 5 drops whole turns because half a turn misleads the model. An answer without its question, or a question whose answer was dropped, makes the history say something that never happened.

Here is how the rule plays out:

Historymax_messagesResult
S U1 A19S U1 A1, as a new list
S U1 A1 U2 A2 U3 A3 U4 A49unchanged, as a new list, because exactly 9 fits
S U1 A1 U2 A2 U3 A3 U4 A4 U5 A59S U2 A2 U3 A3 U4 A4 U5 A5
S U1 A1 U2 A2 U3 A3 U4 A4 P9S U2 A2 U3 A3 U4 A4 P
U1 A1 U2 A2 U3 A34U2 A2 U3 A3
U1 A1 U2 A2 U3 A3 P4U3 A3 P
S U1 A1 U2 A2 P2S P
S U1 A11S U1 A1, because the limit cannot be met
S3S
(empty)3(empty)

Look at the fourth row. Ten messages is one too many, but the rule can only drop two at a time, so the result has 8 messages, not 9. With a system message and a pending question, the list always has an even length, so a limit of 9 gives at most 8 messages. That is expected, not a bug.

Here is that fourth row as a picture. Look at the two ends, which never move, and at the oldest turn, which leaves as a pair.

trim_history drops the oldest whole turn, U1 and A1, while the system message stays first and the pending question stays last, leaving 8 messages.

Build it

Here is one way to lay the function out. Each comment is one or a few lines of code:

def trim_history(history, max_messages):
    # Raise ValueError if max_messages is below 1.
    # If the first message is a system message, keep it aside; the rest are the turns.
    # Protect one message if the turns end with a user message, otherwise two.
    # While the total is too long and an older turn remains in front of the
    # protected messages, drop the oldest turn: the first two of the turns.
    # Return the system message (if any) followed by the remaining turns.

A slice drops the oldest turn and builds a new list at the same time, so the caller’s list is never changed:

turns = ["U1", "A1", "U2", "A2", "P"]
turns = turns[2:]
print(turns)

The total in the loop condition includes the system message, so count it when you compare with max_messages. And if history starts without a system message, every message in it belongs to the turns.

A history can be empty, or hold only the system message, which leaves no turns. Check that a list has items before you read its first or last message.

Check your function against the table above before you check your work. A few cases catch most mistakes: the fourth row, where the result is 8 and not 9; the second row, which is exactly at the limit and stays as it is; the row S U1 A1 | 1, where the limit cannot be met and nothing more can be dropped; and the last two rows, where the history already fits and one of them is empty.

See the trim

Run python conversation.py. The demonstration from Lesson 3 built a long conversation, five made-up turns after the study system message, and sent one more question. Now it sends the same messages a second time, trimmed with MAX_MESSAGES, which is 9:

After trimming:
[pl-scripted echo] No scripted reply matches this prompt, so here is what I received.
Messages: 8
System message: You are a patient Python tutor. Answer only questions about the course notes and Python. If a question is about something else, say so and suggest a Python topic instead.
Last user message: How many messages did you get?
A note sent with a question:
Run stopped at the next function to build: note_message is built in Lesson 7.

Twelve messages went in and eight reached the model: the system message, the last three turns and the pending question. The system message survived and the question is still last.

A history is S U1 A1 U2 A2 U3 A3 P, and max_messages is 6. What does trim_history return?

With a system message, a pending question and max_messages 9, a long history is trimmed to 8 messages. Why not 9?

Task

In conversation.py, replace the unfinished body of trim_history(history, max_messages) so it follows the rule in the lesson:

  1. A max_messages below 1 raises ValueError.

  2. A history is an optional system message, then whole user-and-assistant turns, optionally ending with one pending user message.

  3. The system message, if present, is always kept.

  4. The newest messages are protected: the pending question if there is one, otherwise the last user-and-assistant turn.

  5. While the list is longer than max_messages and an older turn exists in front of the protected messages, drop the oldest turn, both messages together.

  6. Return a new list every time, and never change the list you were given. If the limit cannot be met, return the system message plus the protected messages.

Run python conversation.py to see the message count before and after trimming, then check your work.