An LLM Call Is an HTTP Request
What a Language Model Does
Before you send a model anything, it helps to know what it will do with what you send. The short version: a language model takes text in and gives text back. Everything a chat assistant seems to do, from answering questions to writing code, comes out of that one operation.
Text in, text out
You send the model some text, called the prompt. The model works out what text is likely to come next and sends that back, called the completion. Ask it What is a Python list? and the most likely continuation is an explanation of lists, so that is what you get.
In a plain call, the model does not look the answer up anywhere. It has no search engine and no database of facts to consult during your call. What it produces comes from patterns it learned from a very large amount of text before it was ever deployed. That is why a model can explain lists well and still state a wrong fact with complete confidence: both come out of the same process.
One token at a time
A model does not write a whole answer at once. It produces one small piece, adds it to the text so far, then produces the next piece based on everything before it, and repeats until it decides to stop or hits a length limit.
Those pieces are called tokens. A token is often a short word, part of a longer word, a space plus a word, or a punctuation mark. A tokenizer, the part that does the splitting, might cut the sentence Lists are changeable. like this:
Lists | are | change | able | .
That would be five tokens for three words. Every model has its own way of splitting text, so the same sentence can be a different number of tokens on another model. For English text, a token averages roughly four characters, and that rough rule is what the scripted course model uses to count.
Tokens matter to you as a programmer for two reasons. Providers charge per token, separately for the tokens you send and the tokens you get back. And every model has a limit on how many tokens a single call can hold. You price a call in Lesson 6 of this chapter and meet the limit in Chapter 2.
No memory between calls
Each call to a model is separate. If you tell a model your name in one request and ask for it in the next, it does not know it. Nothing from the first request is kept for the second.
A chat app seems to remember because the app sends the whole conversation again with every new message. When you build a conversation yourself in Chapter 2, your program will do the same.
The same question, different answers
Most models pick each next token with a little randomness, so asking the same question twice can give two different wordings, or occasionally two different answers. A setting called temperature controls how much: lower values make the model stick to its most likely choices, higher values let it wander.
That variation is a problem when you want to check a program. So the course’s scripted model, pl-scripted, never varies. It answers from a fixed table of questions, gives the same reply every time, and ignores temperature. The live model, pl-live, varies like any real model.
You ask a model a question about Python. Where does its answer come from?
In one request you send My name is Sam. In the next request you send only What is my name? What does the model know?
Which statement about tokens is true?
You send pl-live exactly the same request twice. What should you expect?