0%

Chapter 5 · practice

Using the OpenAI Client

Install the Client and Point It at the Course Model

For four chapters you have talked to the course model by hand. Every call built the address with f"{BASE_URL}/chat/completions", sent the body with json=, passed timeout=TIMEOUT, called raise_for_status(), decoded the reply with .json() and dug the answer out with a chain of keys. Chapter 3 added a stream reader on top. You can now write all of that yourself, which is exactly why this chapter hands it to a library.

The openai package is a client for services that speak the same API as llm-api. You tell it once where the service is, and then each call is one line. This chapter moves the study assistant onto it.

Your project for this chapter

Open client_calls.py in the project editor. It starts with these imports:

import openai
import requests
from openai import OpenAI
from pydantic import ValidationError

from conversation import add_turn, note_message, prepare_messages
from errors import StreamBroken
from prompts import INVALID_PREFIX, QUIZ_PROMPT, QUIZ_SYSTEM
from quiz_models import QuizSet
from structured import error_lines

structured.py and quiz_models.py are Chapter 4’s finished code, and conversation.py and llm_basics.py are Chapters 2 and 1, all supplied next to it. You do not need to change them. As before, checking your work uses its own copies, so an edit there cannot change a grade, but it can stop the demonstration from working. errors.py still holds StreamBroken, and prompts.py the fixed prompt texts.

import openai gives you the package’s error classes, such as openai.APIConnectionError. from openai import OpenAI brings in the client class itself, so you can write OpenAI(...).

Below the constants are seven unfinished functions, one or two per lesson, then three finished functions that the demonstration uses in later lessons. In the Shell, run python client_calls.py now. It stops straight away at the first function to build:

Run stopped at the next function to build: make_client is built in Lesson 1.

What the client does for you

A call through the client does the same work your send_body did:

  • it builds the URL from a base address and the path of the endpoint;

  • it sends the request body as JSON;

  • it checks the status code, and raises its own error for a failed request instead of requests.HTTPError;

  • it decodes the reply and gives it back as an object whose values you read with dots.

The package is already installed in this sandbox, at the version the course is written for. To use it on your own computer, install it into a virtual environment with pip install openai==3.37.0. requirements.txt in this project lists it with pydantic and requests.

Build make_client

make_client() returns a client for the course model:

def make_client():
    return OpenAI(base_url=BASE_URL, api_key=API_KEY, timeout=CLIENT_TIMEOUT, max_retries=0)

The three constants it uses are at the top of client_calls.py. Each argument replaces something you used to write by hand.

base_url=BASE_URL is the course model’s address, http://llm-api:8080/v1. The client adds the path of each endpoint itself, such as /chat/completions or /models, so you never build a URL again. It stores the address with a slash at the end.

api_key=API_KEY is a key the client sends with every request. The client refuses to start without one, unless the environment variable OPENAI_API_KEY holds one. Passing api_key= yourself means the client never picks up a real key from your computer by accident. llm-api checks no key, so API_KEY is the placeholder "sandbox". It is not a secret. A real provider needs a real key here, and Chapter 12 uses one.

timeout=CLIENT_TIMEOUT is 10.0, one number of seconds. In Chapter 1 you passed a pair, (3.05, 10): one bound for opening the connection, one for each wait for data. The client takes a single number and uses it for each wait: opening the connection, sending the request, and every read from the server. It is not a limit on the whole call. A long streamed reply that keeps sending pieces never reaches it, because each piece starts a new wait, just as with Chapter 3’s STREAM_TIMEOUT. A wait that does reach it raises openai.APITimeoutError.

max_retries=0 turns off something the client does by default: when a request fails in certain ways, it quietly sends it again, up to two more times. Lesson 5 shows what that looks like and why this course keeps it off until Chapter 9.

Two rules for the checks. Use the constants, not the values typed out again, so the settings live in one place. And build the client inside the function, so that every call to make_client() returns a new client and importing client_calls.py builds nothing.

Run it

Run python client_calls.py. The demonstration builds a client, prints three of its settings, and asks the service which models it has:

A client for http://llm-api:8080/v1/ with timeout=10.0 and max_retries=0
The models it can reach:
  pl-scripted
  pl-live
One question through the client:
Run stopped at the next function to build: ask is built in Lesson 2.

client.base_url, client.timeout and client.max_retries are attributes of the client, read with a dot. The model list came from this loop in the demonstration:

for model in client.models.list():
    print(f"  {model.id}")

client.models.list() sends GET /v1/models and gives back one object per model. Chapter 1’s list_models read the same reply as a dictionary, with model["id"]; here the client has already decoded it, so the id is the attribute model.id.

Building a client sends nothing. Only a call such as client.models.list() reaches the service, which is why checking your work can build a real client and read its settings without any network.

A streamed reply from a live model takes 25 seconds in total, and a new piece arrives every half second. The client was built with timeout=10.0. What happens?

Why does make_client pass api_key="sandbox" when llm-api checks no key?

Task

In client_calls.py, replace the unfinished body of make_client().

  • Return OpenAI(base_url=BASE_URL, api_key=API_KEY, timeout=CLIENT_TIMEOUT, max_retries=0).

  • Use the constants BASE_URL, API_KEY and CLIENT_TIMEOUT, not their values typed out again.

  • Build the client inside the function, so every call returns a new client.

Run python client_calls.py to see the client’s settings and the models it can reach, then check your work.