Reading and Exploring a Dataset in Python

What We Are Doing and Why

Data exploration - loading a dataset, understanding its structure, finding patterns and problems - is the first step in almost every data project. It is where you discover what the data can and cannot tell you. Done well, it prevents weeks of downstream work built on wrong assumptions.

This module walks through a complete exploration on a real public dataset. No prior Python experience is assumed beyond Module 11.

Loading Data With Pandas

Pandas is the standard Python library for working with tabular data. The core object is a DataFrame - think of it as a programmable spreadsheet.

pythonCopy
import pandas as pd

# Load a CSV file
df = pd.read_csv("sales_data.csv")

# Load from a URL (many public datasets are available this way)
url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
df = pd.read_csv(url)

df is the standard variable name for a DataFrame. You can name it anything, but using df makes your code easier for others to read.

The First Five Commands to Run on Any Dataset

1. Shape: How big is it?

pythonCopy
print(df.shape)
# Output: (891, 12)  -  891 rows, 12 columns

2. Head: What does it look like?

pythonCopy
df.head()        # First 5 rows
df.head(10)      # First 10 rows
df.tail()        # Last 5 rows

3. Info: What are the column types?

pythonCopy
df.info()
# Shows: column names, non-null counts, data types, memory usage

The "non-null count" column is your first view of missing values. A column with 714 non-null values in a 891-row dataset has 177 missing values.

4. Describe: What are the distributions?

pythonCopy
df.describe()

For numeric columns: shows count, mean, standard deviation, min, 25th percentile, median (50th percentile), 75th percentile, max. This one command reveals outliers, suspicious values, and distribution shape.

5. Value Counts: What categories exist?

pythonCopy
df["Survived"].value_counts()
# Output:
# 0    549  (did not survive)
# 1    342  (survived)

Use this for categorical and boolean columns to understand the distribution of categories and spot any unexpected values.

Checking for Missing Values

pythonCopy
# Count missing values per column
df.isnull().sum()

# What fraction of each column is missing?
df.isnull().mean()  # Multiply by 100 for percentage

A column that is 70% missing is usually not useful for analysis unless the missingness itself is the signal. A column that is 1% missing may still be usable with appropriate handling.

Filtering and Selecting

pythonCopy
# Select one column
ages = df["Age"]

# Select multiple columns
subset = df[["Age", "Fare", "Survived"]]

# Filter rows meeting a condition
survivors = df[df["Survived"] == 1]
first_class = df[df["Pclass"] == 1]
adults = df[df["Age"] >= 18]

# Combine conditions with & (and) and | (or)
rich_survivors = df[(df["Survived"] == 1) & (df["Pclass"] == 1)]

Aggregation: Answering Questions With Data

pythonCopy
# Average age of survivors vs. non-survivors
df.groupby("Survived")["Age"].mean()

# Survival rate by passenger class
df.groupby("Pclass")["Survived"].mean()

# Count by two categories at once
df.groupby(["Pclass", "Sex"])["Survived"].sum()

groupby is one of the most powerful Pandas operations. It splits the data by a category, applies a function (mean, sum, count, etc.) to each group, and returns the results.

Sorting

pythonCopy
# Sort by age, youngest first
df.sort_values("Age")

# Sort by age, oldest first
df.sort_values("Age", ascending=False)

# Sort by multiple columns
df.sort_values(["Pclass", "Fare"], ascending=[True, False])

A Complete Exploration Example

pythonCopy
import pandas as pd

# Load Titanic data
df = pd.read_csv("titanic.csv")

# 1. Basic dimensions
print(f"Rows: {df.shape[0]}, Columns: {df.shape[1]}")

# 2. Missing values
print("\nMissing values:")
print(df.isnull().sum()[df.isnull().sum() > 0])

# 3. Survival rate overall
print(f"\nOverall survival rate: {df['Survived'].mean():.1%}")

# 4. Survival rate by class
print("\nSurvival rate by class:")
print(df.groupby("Pclass")["Survived"].mean().round(2))

# 5. Survival rate by gender
print("\nSurvival rate by gender:")
print(df.groupby("Sex")["Survived"].mean().round(2))

# 6. Age distribution among survivors
print("\nAge of survivors (mean):", df[df["Survived"]==1]["Age"].mean().round(1))
print("Age of non-survivors (mean):", df[df["Survived"]==0]["Age"].mean().round(1))

This 20-line script produces a meaningful first look at a dataset. Adapt it to any dataset by changing the column names.

The Exploration Questions to Always Answer

After running the basic commands, write down answers to:

1. What does each row represent? (grain)
2. What columns have missing values, and might those be meaningful?
3. What is the distribution of the target variable (if there is one)?
4. Are there any surprising values in describe()?
5. Which groupings seem most informative?

These written answers become the foundation for any further analysis or modeling.

Where to Go Next

The next module explains what makes data useful for ML - the properties of a dataset that determine whether you can train a model on it. This is the conceptual bridge between data exploration and AI system design.

Common Mistakes

Using df.iterrows() for row-level operations. Iterating over a DataFrame row by row in pure Python is 10 to 100 times slower than using vectorized pandas or NumPy operations. If you find yourself writing a loop over rows, there is almost always a apply, map, or vectorized expression that does the same thing orders of magnitude faster.

Not specifying dtypes when reading data. When pandas infers dtypes, it may read an integer column as float or a categorical column as object, causing silent type mismatches later in the pipeline. Passing dtype= explicitly on read makes your assumptions about the data contract visible and catches data quality issues at load time rather than at training time.

Modifying a DataFrame slice in place with chained indexing. Writing df[condition][column] = value triggers SettingWithCopyWarning because pandas cannot guarantee whether you are modifying the original DataFrame or a temporary copy. Use .loc[condition, column] = value to make the assignment unambiguous and prevent hard-to-trace bugs.