NumPy, Pandas, and Data Manipulation for Model Building

Data Work Is the Daily Job

Most ML tutorials treat data preparation as an afterthought before the interesting model code begins. In practice, the inverse is true: data work - loading, validating, cleaning, and shaping - occupies most of the time in any ML project and is where most models either succeed or fail silently.

This module covers NumPy and Pandas at the depth needed for model-building workflows, with explicit attention to the mistakes that cause models to learn the wrong thing.

NumPy: Vectorized Computation

NumPy's core concept is the ndarray - an n-dimensional array of typed elements that supports vectorized operations far faster than Python loops.

pythonCopy
import numpy as np

# Creating arrays
a = np.array([1.0, 2.0, 3.0, 4.0])
B = np.array([[1, 2], [3, 4]])         # 2D array (matrix)
zeros = np.zeros((100, 10))            # 100×10 array of zeros
ones = np.ones((50,), dtype=np.int32)
linspace = np.linspace(0, 1, 100)     # 100 values from 0 to 1

# Shape and dtype
print(B.shape)    # (2, 2)
print(B.dtype)    # int64

Vectorized Operations - No Loops

pythonCopy
X = np.random.randn(10000, 50)   # 10K examples, 50 features each

# SLOW: Python loop
means = []
for row in X:
    means.append(row.mean())

# FAST: vectorized (100× faster or more)
means = X.mean(axis=1)     # Compute mean of each row

# Column-wise normalization (axis=0 operates along the sample dimension)
X_normalized = (X - X.mean(axis=0)) / X.std(axis=0)

Indexing and Slicing

pythonCopy
X = np.random.randn(100, 5)

X[0]          # First row (shape: (5,))
X[:, 2]       # Third column (shape: (100,))
X[10:20]      # Rows 10–19
X[X[:, 0] > 0]  # Boolean indexing: rows where column 0 is positive

# Fancy indexing
indices = np.array([0, 5, 10, 15])
X[indices]    # Rows at these positions

Broadcasting

NumPy operations between arrays of different shapes automatically expand the smaller array to match:

pythonCopy
X = np.random.randn(1000, 10)  # 1000 examples, 10 features
mean = X.mean(axis=0)           # Shape: (10,)  -  one mean per feature

# Broadcasting: mean is automatically applied to each row
X_centered = X - mean           # Shape: (1000, 10)  -  no loop needed

Broadcasting rules: NumPy aligns dimensions from the right. Dimensions of size 1 are stretched to match.

Reproducibility With Seeds

pythonCopy
rng = np.random.default_rng(42)     # Modern API: preferred over np.random.seed
X = rng.normal(0, 1, size=(1000, 10))
y = (X[:, 0] > 0).astype(int)

Using default_rng with a fixed seed produces identical results regardless of when the code runs - essential for reproducible experiments.

Pandas: Structured Data for ML

Pandas DataFrame is the standard for tabular data in Python ML workflows.

Loading and Inspecting

pythonCopy
import pandas as pd

df = pd.read_csv("data/raw/transactions.csv", parse_dates=["timestamp"])

print(df.shape)          # (rows, cols)
print(df.dtypes)         # Data type of each column
print(df.head())         # First 5 rows
print(df.describe())     # Summary statistics for numeric columns
print(df.isnull().sum()) # Missing value count per column

The first five minutes with a new dataset should always start here - before any preprocessing or modeling.

Selection and Filtering

pythonCopy
# Column selection
ages = df["age"]                           # Series (1D)
subset = df[["age", "income", "label"]]    # DataFrame (2D)

# Row filtering
high_value = df[df["revenue"] > 1000]
mobile_users = df[df["device"].isin(["mobile", "tablet"])]
recent = df[df["timestamp"] > "2024-01-01"]

# .loc (label-based) and .iloc (position-based)
df.loc[df["user_id"] == "u123", "label"]
df.iloc[0:100, 2:5]   # First 100 rows, columns 2-4

Aggregation and GroupBy

pythonCopy
# User-level features from event-level data
user_features = df.groupby("user_id").agg(
    total_spend=("amount", "sum"),
    avg_spend=("amount", "mean"),
    num_transactions=("amount", "count"),
    days_active=("date", lambda x: (x.max() - x.min()).days),
    last_event=("timestamp", "max"),
).reset_index()

GroupBy + aggregation is one of the most common feature engineering patterns: take event-level data and create user-level (or entity-level) features.

Missing Values

pythonCopy
# Understand the missingness pattern first
print(df.isnull().mean())   # Fraction missing per column

# Imputation (always on a copy, never in place on training data)
df_clean = df.copy()
df_clean["income"] = df_clean["income"].fillna(df_clean["income"].median())
df_clean["category"] = df_clean["category"].fillna("Unknown")

# Add missing indicator before imputing
df_clean["income_was_missing"] = df["income"].isna().astype(int)

Adding a missingness indicator before imputing preserves the signal that the value was absent - which is often predictive in its own right.

Merging DataFrames

pythonCopy
# Inner join: only rows in both
user_transactions = pd.merge(users, transactions, on="user_id", how="inner")

# Left join: all rows from left, matching rows from right (NaN if no match)
enriched = pd.merge(transactions, user_metadata, on="user_id", how="left")

# Merge on multiple keys
orders = pd.merge(orders, items, on=["order_id", "item_id"])

Always validate join cardinality: if you expect one-to-one and get one-to-many, something is wrong with your key assumptions.

pythonCopy
# Check for unexpected duplicates after a merge
assert len(result) == len(left_df), f"Unexpected row multiplication after merge"

Data Leakage: The Most Common Silent Bug

Data leakage occurs when your preprocessing uses information from the validation or test set - even indirectly. The result is a model that appears to perform well but will fail in production.

Pipeline Leakage (The Most Common Kind)

pythonCopy
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)

# WRONG: scaler sees validation set statistics
scaler = StandardScaler()
X_all_scaled = scaler.fit_transform(X)  # Leakage!
X_train_scaled = X_all_scaled[:len(X_train)]
X_val_scaled = X_all_scaled[len(X_train):]

# CORRECT: scaler fitted only on training data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit on train only
X_val_scaled = scaler.transform(X_val)          # Transform val using train stats

The sklearn.pipeline.Pipeline makes the correct approach the default - the preprocessor is only fitted on the training fold in cross-validation.

Temporal Leakage

For time-series data, features that use future information produce models that appear strong offline but fail when deployed.

pythonCopy
# WRONG: uses future spend to predict churn
df["lifetime_spend"] = df.groupby("user_id")["amount"].transform("sum")
# lifetime_spend at event time i includes spend from events i+1, i+2, ...

# CORRECT: use only spend up to and including the event time
df = df.sort_values("timestamp")
df["cumulative_spend"] = df.groupby("user_id")["amount"].cumsum()

The rule: for every feature you add, ask "at prediction time, would I actually have this value?"

Common Mistakes and Bad Instincts

Modifying the original DataFrame instead of a copy. Always df.copy() before modifying. In-place modifications can affect other parts of the pipeline in ways that are hard to trace.

Using df.apply(lambda x: ..., axis=1) for row-wise operations. apply with axis=1 loops over rows in Python - it is slow for large datasets. Vectorized column operations are typically 10–100× faster.

Not checking join cardinality. A merge that unexpectedly multiplies rows will inflate your training set with duplicates and corrupt your evaluation metrics without any warning.

Ignoring the missingness pattern. Dropping all rows with any missing value discards usable data and introduces selection bias. Imputing without an indicator column loses the signal that the value was absent.

Where to Go Next

Module 7 covers SQL - the other half of the data manipulation toolkit. SQL and Pandas are complementary: SQL is where data lives in most production systems; Pandas is where you work with it once you have loaded it into Python. Module 8 then puts the Python engineering habits from Module 1 together with the data skills from Modules 6 and 7 into a testable, maintainable project structure.