Conversations and Instructions
The Context Window and max_tokens
Your conversation now grows by two messages every turn, and all of it is sent on every call. That has two consequences you have to design for: a limit on how much a model can read, and a bill that grows with every turn.
The context window
A model can only work with a limited number of tokens in one call. That limit is its context window. Everything in the request counts toward it: the system message, every earlier user and assistant message, and the new question. On most models the reply the model writes counts toward the same limit.
Windows differ a lot between models, and they have grown over the years. They range from tens of thousands of tokens to a million or more, depending on the model, so look up the number for the model you use. Even the smaller end sounds like plenty for a chat. It stops being plenty when a conversation runs for a long time, or when each message carries a pasted document, as yours will in Chapter 8.
When a request does not fit, you do not get a reply that quietly ignores the oldest messages. What happens depends on the provider: usually the request fails with an error, and some services cut part of the conversation for you, without asking which part matters. Either way, your program is the right place to decide what to keep. Lesson 5 does that.
Every turn pays for the whole history again
In Chapter 1 you priced a call from its usage: input tokens at one price, output tokens at another. In a conversation, the input of every call includes the whole history so far, so every turn reads, and pays for, everything before it again. Some providers give a discount on input they have seen recently, called prompt caching, but you still send the whole history every time, and this course prices every input token at the full rate.
Here is a worked example with round numbers. Suppose the system message is 90 tokens, every question is 10 tokens, and every answer is 90 tokens. Turn 1 sends the system message and one question. Each later turn also sends every earlier question and answer, 100 more tokens per turn:
| Turn | Input tokens | Output tokens | Cost at pl-live prices | Cost at the larger model’s prices |
|---|---|---|---|---|
| 1 | 100 | 90 | $0.0000147 | $0.0011 |
| 2 | 200 | 90 | $0.0000177 | $0.0013 |
| 5 | 500 | 90 | $0.0000267 | $0.0019 |
| 10 | 1,000 | 90 | $0.0000417 | $0.0029 |
The costs come from Chapter 1’s price tables: $0.03 per million input tokens and $0.13 per million output tokens for pl-live, and $2.00 and $10.00 for the larger model. Turn 10 at pl-live prices is 1,000 × 0.03 / 1,000,000 + 90 × 0.13 / 1,000,000 dollars.
Turn 10 sends ten times as many input tokens as turn 1. At pl-live prices it costs about 2.8 times what turn 1 does, not ten times, because the answer, priced as output, stays the same size. The input part of the cost grows with every turn, and over the whole ten-turn conversation the calls send 5,500 input tokens, where ten separate calls, each with the system message and one question, would send 1,000.
A long conversation also gets slower. Before a model writes the first word of its reply, it has to read the whole request, so a longer request takes longer to start answering.
max_tokens limits the reply
You can put a cap on how long the reply may be with max_tokens in the request body:
body = {
"model": "pl-scripted",
"messages": [{"role": "user", "content": "Explain decorators."}],
"max_tokens": 200,
}
print(body["max_tokens"])
max_tokens limits only what the model writes. It does nothing about the history you send. When the model reaches the cap, it stops where it is and the response has finish_reason "length", which you met in Chapter 1 Lesson 5. The reply is then cut off mid-answer, and on a model that thinks before answering it can even be empty. So a cap protects your budget, but too low a cap gives you broken answers.
The live model honors max_tokens. The scripted model accepts it and ignores it, so your requests to pl-scripted keep working whether you send it or not. SCRIPTED_MODEL.md says so.
In a conversation that sends the whole history every time, what grows with each new turn?
You set "max_tokens": 50 on a request to the live model. What does that limit?
A study chat has been running for an hour. Why is each new reply slower and more expensive than the first one was?