HTTPS Is HTTP Inside TLS

Every byte in this course has been readable on the wire. In Chapter 7 your client sent a password in an authorization header as base64, which anyone who sees it can reverse; your server handed out session ids in cookies; writes carried bearer tokens. That was safe only because every one of those bytes stayed inside one machine, on the loopback address. Sent across a real network, each of them could be read, and copied, by anyone on the path: the Wi-Fi, the internet provider, a proxy in between. HTTPS is how the web avoids that, and it changes less than you might expect.

The same bytes, wrapped

HTTPS is HTTP sent through TLS, the encryption layer you met in Networks with Python. TLS sits between the TCP connection and the HTTP bytes:

http://books.example/books          https://books.example/books

HTTP bytes                          the same HTTP bytes
                                    TLS: encrypts and checks them
TCP connection to port 80           TCP connection to port 443

An https URL means three things: connect to port 443 unless the URL names another, run a TLS handshake before anything else, and then send exactly the HTTP you have written all course, through the encrypted layer. The request line, the headers, content-length, chunks, keep-alive, and Connection: close are all unchanged. TLS does not know or care that the bytes are HTTP.

What TLS adds

TLS gives a connection three things:

• Encryption. Everything inside, the path and query, every header including cookie and authorization, and every body, travels as bytes only the two ends can read.

• Integrity. If anything on the path changes a byte, the receiving end detects it and the connection fails, instead of handing over altered data.

• Server identity. The server presents a certificate for its name, signed by an authority the client trusts, and the client checks that the name matches the host in the URL. Without that check, a client could set up a perfectly encrypted connection with an impostor.

Some things stay visible. An observer still sees the addresses and ports, when the connection opens and closes, and roughly how many bytes flow each way. Usually they also see the server’s name, because the client states it early in the handshake so that the server can pick the right certificate.

In Python

The standard library’s ssl module wraps a connected socket. These lines are for reading, not for running here:

import socket
import ssl

host = "books.example"
context = ssl.create_default_context()
with socket.create_connection((host, 443), timeout=5) as sock:
    with context.wrap_socket(sock, server_hostname=host) as tls:
        tls.sendall(b"GET /books HTTP/1.1\r\nHost: books.example\r\nConnection: close\r\n\r\n")
        print(tls.recv(4096))

ssl.create_default_context() checks the certificate and the host name by default. http.client.HTTPSConnection(host, timeout=5), the https sibling of the class you have used since Chapter 3, does this wrapping for you. server_hostname is the name to check, and the name the client states in the handshake. The object wrap_socket returns has sendall and recv, so read_head, read_chunked, and read_body would read from it exactly as they read from a plain socket.

With requests, an https URL is all it takes: requests.get("https://books.example/books", timeout=5) runs the handshake and checks the certificate, because its verify setting is on by default. Chapter 10 comes back to it. Turning verify off keeps the encryption but throws away the server’s identity.

Why not here

This site has no TLS endpoint. The practice service and your own server speak plain HTTP, and lesson sandboxes reach nothing else, so no lesson here can show a handshake or a certificate.

If you have Python on your own computer and an internet connection, you can look at a real one. This optional step is not part of the course, and nothing checks it. It prints the TLS version and the subject of the certificate of a site you choose. The block is for a Unix shell such as bash or zsh; on Windows, save the Python lines between the EOF markers in a file and run it with python.

python3 - <<'EOF'
import socket
import ssl

host = "www.python.org"  # or any https site you choose
context = ssl.create_default_context()
with socket.create_connection((host, 443), timeout=5) as sock:
    with context.wrap_socket(sock, server_hostname=host) as tls:
        print(tls.version())
        print(tls.getpeercert()["subject"])
EOF

HTTP/2 and HTTP/3

Many sites answer https with HTTP/2. It keeps methods, statuses, and headers, and what they mean; what it changes is how messages travel on the connection. Instead of text lines, it sends binary frames, and it interleaves many requests and replies on one connection at once, so no request waits for the one before it as yours did in fetch_many. It compresses headers, and it writes every header name in lowercase, as your response dicts have all along. HTTP/3 carries the same messages over QUIC, which runs on UDP instead of TCP and has TLS built in.

A client library chooses the version for you, and everything this course taught, from methods and statuses to caching, cookies, and authentication, means the same in all three. Chapter 10 picks up requests and httpx, now that you know what each of them does for you.