0%

Chapter 3

Load and Inspect a Table

Meet the DataFrame

The Chapter 1 project answered its question in about seventy lines, and most of them read rows one at a time. Chapter 2 showed that a whole column of numbers can be one , an array. A table is the next step: several columns side by side, one array per column, each with a name. , the library this chapter starts, calls that table a DataFrame.

A table in one call

Here is how you load the readings file with pandas. The first line imports it under the short name pd, which almost all pandas code uses, just as code uses np:

import pandas as pd

readings = pd.read_csv("data/readings.csv")
print(readings.shape)
print(list(readings.columns))

pd.read_csv opens the file, reads the header, splits every line at the commas, and returns one DataFrame holding all of it. There is no with block, no csv.DictReader and no . The path is the same kind of path you passed to open() in Python in Practice, and pd.read_csv also accepts a pathlib.Path.

On data/readings.csv, the two print calls show:

(1079, 6)
['station_id', 'date', 'temp_min_c', 'temp_max_c', 'rain_mm', 'battery_low']

readings.shape works like the .shape of a NumPy array: a pair of numbers, rows first, then columns. The file has 1,079 rows of readings and 6 columns. readings.columns holds the column names, in the order of the file’s header, and list(...) turns them into an ordinary Python for printing. list() is a built-in : give it a collection of values, and it returns a new list holding the same values in the same order. list("ST01"), for example, is ['S', 'T', '0', '1'], because a is a collection of characters.

The block above has no Run button. The Run buttons in a lesson run in your browser, which cannot see the project’s files, so any example that reads data/readings.csv is shown for reading only. To experiment in the browser, you build a small table in the code itself.

Build a small table

pd.DataFrame(...) builds a table from a . Each key becomes a column name, and each list becomes that column’s values, one per row. Press Run. The first time a block imports pandas, your browser downloads the library once, which can take a few seconds; later runs start straight away.

Try it

Printing the table shows it as a grid:

  station_id  rain_mm
0       ST01      0.0
1       ST02      6.0
2       ST01      5.8

The two keys became the two column names across the top. Each list became a column, and all the lists have the same length, three, because every row needs one value in each column. table.shape is (3, 2): three rows, two columns.

The numbers down the left side, 0, 1 and 2, are not part of your data. They are the table’s , a label for each row. pd.read_csv and pd.DataFrame both number the rows from 0 unless you ask for something else, so the readings table is labeled 0 to 1078. table.index holds those labels, and list(table.index) prints them as [0, 1, 2].

Change the rain_mm list to hold four values and run the block again. pandas refuses with a , because the columns would no longer have the same number of rows.

What does this print?

import pandas as pd

table = pd.DataFrame({
    "station_id": ["ST03", "ST03", "ST04", "ST04"],
    "temp_min_c": [-1.7, -4.4, -2.9, -3.7],
    "rain_mm": [0.0, 0.0, 0.0, 5.8],
})
print(table.shape)

Columns are arrays with names

A DataFrame is a set of columns that share one index. Each column works like a Chapter 2 array: all its values have one dtype, and arithmetic and summaries work on the whole column at once. A column of numbers is stored as a NumPy array underneath. What the table adds is a name on top of each column and a label beside each row.

Because each column has its own dtype, one table can mix kinds of values. In the example, station_id holds text and rain_mm holds decimal numbers. A single NumPy array could not hold both kinds and still do arithmetic, but a table of two arrays can.

So the rows of the readings table are the records you read one at a time in Chapter 1, and the columns are what Chapter 2 computed with. The next lesson takes one column out of the table and computes with it.

Chapter 1Chapter 2Chapter 3
a row as a dictionary from csv.DictReadera column as a NumPy arraya whole table as one DataFrame
a loop visits every rowone uses the whole columna column name picks the column

Which statement about this table is true?

table = pd.DataFrame({"station_id": ["ST01", "ST02"], "rain_mm": [0.0, 6.0]})

Nothing has been computed yet: the readings are in memory as one with a known shape. In the next lesson you open the chapter’s project file, load.py, and write your first functions that work with that object.