Data analysis with NumPy & pandas

Vectorised NumPy arrays, pandas DataFrames, filtering, groupby, pivot tables, cleaning data and merging.

Python is the most popular language for data work, thanks to two libraries: NumPy, which provides fast numeric arrays, and pandas, which builds labelled tables (DataFrames) on top of them — think "Excel and SQL, but programmable". Data analysts, data engineers, backend developers and ML engineers all use them. This lesson gives you a practical, working introduction.

Terminal

pip install numpy pandas

NumPy: fast arrays#

A NumPy array holds many values of the same type in one contiguous block of memory. Operations apply to every element at once (vectorisation), running in optimised C instead of Python loops:

Python

import numpy as np

prices = np.array([2499, 799, 12999, 3499])
print(prices * 1.18)              # add 18% tax to every element
print(prices.sum(), prices.mean(), prices.max())
print(prices > 3000)              # element-wise comparison
print(prices[prices > 3000])      # boolean indexing
print(prices.dtype, prices.shape)

Output

[ 2948.82   942.82 15338.82  4128.82]
19796 4949.0 12999
[False False  True  True]
[12999  3499]
int64 (4,)

Compare with plain Python, where you'd need loops or comprehensions for each of these. Vectorised code is shorter and typically 10–100× faster on large data:

Python

import time

import numpy as np

values = list(range(1_000_000))
arr = np.arange(1_000_000)

start = time.perf_counter()
py_total = sum(v * v for v in values)
py_time = time.perf_counter() - start

start = time.perf_counter()
np_total = int((arr * arr).sum())
np_time = time.perf_counter() - start

print(py_total == np_total, f"NumPy was {py_time / np_time:.0f}x faster")

Output

True NumPy was ...x faster

Multi-dimensional arrays

Python

import numpy as np

sales = np.array([
    [120, 135, 150],   # store A: Jan, Feb, Mar
    [ 90,  80, 110],   # store B
    [200, 210, 190],   # store C
])
print(sales.shape)
print(sales.sum(axis=1))     # total per store (across columns)
print(sales.mean(axis=0))    # average per month (down rows)
print(sales[0, 2], sales[:, 0])   # one cell; first column
print(np.round(sales / sales.sum() * 100, 1)[0])

Output

(3, 3)
[405 280 600]
[136.66666667 141.66666667 150.        ]
150 [120  90 200]
[ 9.3 10.5 11.7]

axis=0 works down the rows (per column); axis=1 works across the columns (per row). NumPy also offers random numbers (np.random.default_rng()), linear algebra (np.linalg) and statistics — it's the foundation of pandas, scikit-learn, PyTorch and most scientific Python.

pandas: DataFrames#

A DataFrame is a table with labelled columns and an index; each column is a Series. Let's create a small sales dataset as a CSV and load it:

Python

from pathlib import Path

import pandas as pd

Path("sales.csv").write_text("""date,city,product,units,unit_price
2026-07-03,Pune,Keyboard,3,2499
2026-07-03,Delhi,Mouse,10,799
2026-07-15,Pune,Monitor,1,12999
2026-08-02,Mumbai,Keyboard,5,2499
2026-08-09,Delhi,Webcam,2,3499
2026-08-21,Pune,Mouse,8,799
2026-09-05,Mumbai,Monitor,2,12999
2026-09-18,Delhi,Keyboard,4,2499
""", encoding="utf-8")

df = pd.read_csv("sales.csv", parse_dates=["date"])
print(df.head(3))
print(df.shape)
print(df.dtypes)

Output

        date   city   product  units  unit_price
0 2026-07-03   Pune  Keyboard      3        2499
1 2026-07-03  Delhi     Mouse     10         799
2 2026-07-15   Pune   Monitor      1       12999
(8, 5)
date          datetime64[us]
city                     str
product                  str
units                  int64
unit_price             int64
dtype: object

(pandas 3 stores text columns with a dedicated str dtype; on pandas 2.x you'll see object instead.) First steps with any new dataset: head(), shape, dtypes, info() (columns, types and missing values) and describe() (summary statistics).

Selecting and filtering#

Python

import pandas as pd

df = pd.read_csv("sales.csv", parse_dates=["date"])

print(df["city"].unique().tolist())                 # one column (a Series)
print(df[["product", "units"]].tail(2))             # several columns
print(df[df["units"] >= 5])                         # boolean filter
print(df[(df["city"] == "Pune") & (df["unit_price"] > 1000)][["date", "product"]])
print(df.loc[2, "product"], df.iloc[0, 1])          # by label / by position

Output

['Pune', 'Delhi', 'Mumbai']
    product  units
6   Monitor      2
7  Keyboard      4
        date    city   product  units  unit_price
1 2026-07-03   Delhi     Mouse     10         799
3 2026-08-02  Mumbai  Keyboard      5        2499
5 2026-08-21    Pune     Mouse      8         799
        date   product
0 2026-07-03  Keyboard
2 2026-07-15   Monitor
Monitor Pune

Combine conditions with & (and), | (or) and ~ (not), and wrap each condition in parentheses — Python's and/or don't work on Series. df.query("units >= 5 and city == 'Pune'") is a readable alternative.

Adding columns and aggregating#

Python

import pandas as pd

df = pd.read_csv("sales.csv", parse_dates=["date"])
df["revenue"] = df["units"] * df["unit_price"]          # vectorised new column
df["month"] = df["date"].dt.strftime("%Y-%m")

print(df["revenue"].sum())
print(df.groupby("city")["revenue"].sum().sort_values(ascending=False))
print(df.groupby("month").agg(orders=("revenue", "size"), revenue=("revenue", "sum")))

Output

90365
city
Mumbai    38493
Pune      26888
Delhi     24984
Name: revenue, dtype: int64
         orders  revenue
month
2026-07       3    28486
2026-08       3    25885
2026-09       2    35994

groupby follows the split–apply–combine pattern, just like SQL's GROUP BY. Named aggregations (orders=("revenue", "size")) give you tidy column names.

A pivot table reshapes data Excel-style:

Python

import pandas as pd

df = pd.read_csv("sales.csv")
df["revenue"] = df["units"] * df["unit_price"]
pivot = df.pivot_table(index="city", columns="product", values="revenue", aggfunc="sum", fill_value=0)
print(pivot)

Output

product  Keyboard  Monitor  Mouse  Webcam
city
Delhi        9996        0   7990    6998
Mumbai      12495    25998      0       0
Pune         7497    12999   6392       0

Cleaning messy data#

Real data has gaps, duplicates and inconsistent formatting. pandas has tools for each:

Python

import numpy as np
import pandas as pd

raw = pd.DataFrame({
    "name": [" Ada ", "grace", "Linus", "grace", None],
    "age": [36, np.nan, 54, np.nan, 41],
    "city": ["Pune", "Delhi", "delhi", "Delhi", "Mumbai"],
})

print(raw.isna().sum())
clean = (
    raw.dropna(subset=["name"])                                    # drop rows without a name
       .assign(name=lambda d: d["name"].str.strip().str.title(),   # string methods via .str
               city=lambda d: d["city"].str.title())
       .drop_duplicates()
)
clean["age"] = clean["age"].fillna(clean["age"].median())          # fill gaps
print(clean)

Output

name    1
age     2
city    0
dtype: int64
    name   age   city
0    Ada  36.0   Pune
1  Grace  45.0  Delhi
2  Linus  54.0  Delhi

The duplicate "grace" row disappeared only after normalising text — the order of cleaning steps matters.

Joining tables and exporting#

Python

import pandas as pd

orders = pd.DataFrame({"order_id": [1, 2, 3], "customer_id": [10, 20, 10], "amount": [2499, 799, 349]})
customers = pd.DataFrame({"customer_id": [10, 20], "name": ["Ada", "Grace"]})

merged = orders.merge(customers, on="customer_id", how="left")     # like SQL LEFT JOIN
print(merged)

summary = merged.groupby("name", as_index=False)["amount"].sum()
summary.to_csv("summary.csv", index=False)
print(open("summary.csv").read(), end="")

Output

   order_id  customer_id  amount   name
0         1           10    2499    Ada
1         2           20     799  Grace
2         3           10     349    Ada
name,amount
Ada,2848
Grace,799

pandas reads and writes many formats: read_csv/to_csv, read_excel/to_excel (needs openpyxl), read_json/to_json, read_parquet, and read_sql to load query results straight from a database connection (sqlite3 or SQLAlchemy).

Plotting (quick look)#

With matplotlib installed (pip install matplotlib), DataFrames can plot themselves:

Python

# in a Jupyter notebook or script with matplotlib installed
import pandas as pd

df = pd.read_csv("sales.csv")
df["revenue"] = df["units"] * df["unit_price"]
ax = df.groupby("city")["revenue"].sum().plot(kind="bar", title="Revenue by city")
ax.figure.savefig("revenue.png")

For exploratory work, most people use Jupyter notebooks (pip install jupyterlab, then jupyter lab), which show tables and charts inline.

Performance tips#

Vectorise: use column operations, not for loops or df.iterrows() — loops can be 100× slower.
Use df["col"].map(dict) or np.where(cond, a, b) instead of row-by-row apply where possible.
Choose efficient types: category for repeated strings, and load only the columns you need with usecols=.
For data bigger than memory, read in chunks (chunksize=) or look at Polars or DuckDB, newer libraries popular for large datasets.

Common mistakes#

Using and/or with Series — use &/| with parentheses.
Chained assignment like df[df["a"] > 1]["b"] = 9 — it modifies a temporary copy, so df is unchanged (pandas 3's copy-on-write makes this explicit with a ChainedAssignmentError warning). Write df.loc[df["a"] > 1, "b"] = 9 instead.
Forgetting index=False in to_csv, which adds an unnamed index column.
Looping over rows instead of vectorising.
Not parsing dates (parse_dates=), leaving them as plain strings.

What's next#

You can now store and analyse data. Next, put your Python on the web: building web apps and APIs with Flask and FastAPI.

NumPy: fast arrays#

Multi-dimensional arrays

pandas: DataFrames#

Selecting and filtering#

Adding columns and aggregating#

Cleaning messy data#

Joining tables and exporting#

Plotting (quick look)#

Performance tips#

Common mistakes#

What's next#

Quick quiz