Analyze Data and Do Science

Here is a very small amount of business information.

What can we say with confidence? That March holds the largest number in this table. That is genuinely all.

Before using it for anything, I would want to know what the label means, whether the three months were counted the same way, and whether any records are missing. The arithmetic can be perfect while the answer is worthless.

That is the whole lesson in miniature. Python is very good at calculating, checking, exploring, and explaining. It cannot make an unclear question or unreliable information trustworthy.

The work is a chain, not a calculation

Data is recorded information we can examine: sales, temperatures, survey answers, scientific measurements, website activity, millions of other things.

Real data work moves through stages:

raw records
     ↓
clean and check
     ↓
explore
     ↓
analyze
     ↓
explain
     ↓
report, chart, or product

Raw records are the information as first collected, which means missing values, typing mistakes, duplicated rows, unclear labels, and three different spellings of the same city. To clean data is to correct what you can and document what you cannot. To explore it is to go looking for patterns and for things that look wrong.

A person makes a decision at every one of those arrows. Python’s job is to carry the decisions out.

Why Python covers the whole chain

It can pull information out of files, databases, websites, scientific instruments, and other systems. It can clean the records, run the calculation, draw the chart, produce the report, and hand the result to another program.

That range is the real strength. A team can stay in one language across the whole chain instead of stitching together a different tool per stage, and the cleaned data can flow onward into deeper analysis or into the back end of a website.

The specialist tools are packages: pandas for information arranged in labeled tables, NumPy for large groups of values and numerical work, SciPy for established scientific methods. You do not need to remember which does what. What matters is that serious, mature data tooling exists here and people have trusted it with real research for years.

Some of the heavy arithmetic does not run as ordinary Python at all. It happens inside the library, in code built for exactly that operation, while Python stays the place where a person describes and connects the work. That is the same arrangement we met in lesson 1.2, and it is coming back twice more.

For range, connection, and mature tools, data and scientific work are a strong fit.

The part Python cannot do

Go back to the three months. Python could find the largest value instantly, and the average, and draw you a chart. None of that tells you whether Orders recorded means orders placed, orders paid for, or orders shipped.

Suppose one month counted only online orders and another included the shop. The numbers would be stored perfectly, calculated perfectly, and the comparison would still be nonsense.

Suppose a system broke and lost a day of records. Python has no way of knowing unless a person or another check tells it.

Suppose the result is surprising. That might be a discovery. It might be a bug. Someone who understands the actual subject has to go and find out which.

This is why data work is programming skill and domain knowledge together, and why the second one is not optional. A medical researcher, a climate scientist, a financial analyst, a journalist, and a shop owner can run the same Python tools and still need entirely different knowledge to ask a sensible question and judge the answer. The tool supports a conclusion. A person remains responsible for it.

What about R?

R is another language built for learning from data and drawing charts. It supports a great many statistical methods and graphics.

Which one fits depends on the methods you need, the work that already exists, the people around you, and what your field teaches. A research group with fifteen years of trusted R has no reason to throw it away. Another group might prefer Python because it wants the analysis connected to a website or a larger product. Plenty of groups use both.

Go deeper

The official pandas documentation shows the range of one of Python’s most important data packages, and the R Project’s own introduction describes R in its own words. Both optional.