2.1 DataFrames, Indexing, Slicing & Filtering

Why this week matters¶

In Week 2, we worked with Python’s built-in data structures. This week we move to the central structure used for tabular statistical computing in Python: the pandas DataFrame.

The main question is:

How do we tell the computer exactly which observations and variables belong in an analysis?

The approved course schedule identifies Week 3 as the first data-engineering module, covering pandas DataFrames, indexing, slicing, and filtering.

1. From Python objects to tabular data¶

Last week we could represent observations as a list of dictionaries:

participants = [
    {"id": 101, "age": 24, "group": "A", "score": 81},
    {"id": 102, "age": 31, "group": "B", "score": 94},
    {"id": 103, "age": 27, "group": "A", "score": 88}
]

Using pandas:

import pandas as pd

df = pd.DataFrame(participants)

we obtain a labeled table.

Python objects
      ↓
lists / dictionaries
      ↓
pandas Series
      ↓
pandas DataFrame
      ↓
statistical data manipulation

2. What is a DataFrame?¶

A pandas DataFrame is a two-dimensional tabular data structure with labeled rows and columns.

Conceptually:

             age   group   score
subject_id
101           24      A      81
102           31      B      94
103           27      A      88
104           19      B      72

A DataFrame contains:

observations (rows);
variables (columns);
an index (row labels);
column labels;
values;
data types.

This structure closely matches the rectangular datasets used throughout statistics and data science.

3. Creating a DataFrame¶

import pandas as pd

data = {
    "id": [101, 102, 103, 104],
    "age": [24, 31, 27, 19],
    "group": ["A", "B", "A", "B"],
    "score": [81, 94, 88, 72]
}

df = pd.DataFrame(data)

Useful attributes:

df.shape
df.columns
df.index
df.dtypes

Useful methods:

df.head()
df.tail()
df.info()
df.describe()

4. Inspect before you analyze¶

A professional workflow starts by asking:

What exactly is in this dataset?

Before analysis, determine:

How many observations are present?
How many variables?
Which variables are numeric?
Which are categorical or text?
Are names and data types what we expected?
Are values missing?

Do not begin modeling before understanding the structure of the data.

5. Series vs. DataFrame¶

A DataFrame contains columns represented as pandas Series objects.

df["age"]

returns a Series.

df[["age"]]

returns a DataFrame.

Compare:

type(df["age"])
type(df[["age"]])

This distinction becomes important because Series and DataFrames behave differently in some operations.

6. Selecting columns¶

Select one column:

df["age"]

Select multiple columns:

df[["age", "score"]]

Think of this as choosing the variables required for an analysis.

7. The DataFrame index¶

Every DataFrame has an index.

Initially:

df.index

may return a default integer index.

We can create a meaningful index:

df = df.set_index("id")

Now the row labels are participant IDs.

8. `.loc` and `.iloc`¶

Two essential pandas indexers are:

.loc

and:

.iloc

Remember:

loc  → labels
iloc → integer positions

9. Selecting by position with `.iloc`¶

df.iloc[0]

means:

Give me the first row.

df.iloc[0:3]

selects the first three rows.

Rows and columns can be selected together:

df.iloc[0:3, 0:2]

General form:

df.iloc[row_positions, column_positions]

10. Selecting by label with `.loc`¶

If id is the index:

df.loc[101]

means:

Give me the row labeled 101.

Several labels:

df.loc[[101, 103]]

Rows and columns:

df.loc[
    [101, 103],
    ["age", "score"]
]

General form:

df.loc[row_labels, column_labels]

11. `.loc` vs. `.iloc`¶

Suppose the row labels are:

Then:

df.loc[205]

selects the row labeled 205.

df.iloc[1]

selects the second row by position.

The commands may identify the same observation, but they express different instructions.

12. Slicing¶

Position-based slicing follows ordinary Python conventions:

df.iloc[1:4]

includes positions 1, 2, and 3, but excludes 4.

Label-based slices behave differently:

df.loc[101:103]

typically include both endpoint labels when they exist.

This difference is easy to overlook.

13. Selecting rows and columns together¶

A very useful general pattern is:

df.loc[rows, columns]

For example:

df.loc[:, ["age", "score"]]

means:

All observations, but only age and score.

The comma separates which observations from which variables.

14. Boolean conditions¶

Consider:

df["score"] > 80

This produces a Boolean Series such as:

id
101     True
102     True
103     True
104    False

We can then use it to select observations:

df.loc[df["score"] > 80]

15. Boolean filtering as statistical logic¶

Statistically, the expression:

df["score"] > 80

acts like an indicator:

I(score_i > 80)

For each observation:

True  → include
False → exclude

Thus:

D* = {i : score_i > 80}

Filtering is therefore an implementation of a statistical inclusion rule.

16. Filtering with one condition¶

Examples:

df.loc[df["age"] >= 30]

df.loc[df["group"] == "A"]

df.loc[df["score"] >= 85]

General form:

df.loc[condition]

17. Multiple conditions¶

Use & for AND:

df.loc[
    (df["group"] == "A") &
    (df["score"] >= 85)
]

Use | for OR:

df.loc[
    (df["group"] == "A") |
    (df["score"] >= 90)
]

Use ~ for NOT:

df.loc[
    ~(df["group"] == "A")
]

Use parentheses around each condition.

18. Why not `and` and `or`?¶

This does not work as intended:

(df["age"] >= 30) and (df["score"] >= 85)

Each comparison produces an entire Series of Boolean values.

For element-wise Boolean logic in pandas, use:

&
|
~

19. Filter rows and choose variables together¶

Suppose we want:

Participants scoring at least 85, showing only their group and score.

df.loc[
    df["score"] >= 85,
    ["group", "score"]
]

Think:

WHO?
    score >= 85

WHAT?
    group, score

20. Filtering categorical values with `.isin()`¶

To select multiple categories:

df.loc[
    df["group"].isin(["A", "C"])
]

This is often clearer than a long sequence of OR conditions.

21. Missing values¶

Real datasets contain missing values.

Inspect:

df.isna()

Count missing values:

df.isna().sum()

Select rows with missing scores:

df.loc[df["score"].isna()]

Select rows with observed scores:

df.loc[df["score"].notna()]

An important statistical principle is:

Missingness is data.

Missing-data decisions should be explicit and documented.

22. Assignment with `.loc`¶

.loc can also modify selected observations.

df["performance"] = "medium"

Then:

df.loc[
    df["score"] >= 90,
    "performance"
] = "high"

This clearly identifies both the rows to modify and the target column.

23. Avoid chained assignment¶

Avoid:

df[df["score"] > 80]["score"] = 100

Prefer:

df.loc[
    df["score"] > 80,
    "score"
] = 100

The .loc version is more explicit and avoids ambiguity around intermediate objects.

24. Filtering defines the analysis population¶

Consider:

analysis_df = df.loc[
    (df["age"] >= 18) &
    (df["group"] == "A") &
    (df["score"].notna())
]

This code defines:

Adults in group A with observed scores.

That is not merely data manipulation. It is part of the statistical analysis specification.

25. Separate criteria from selection¶

For more complicated analyses:

eligible = (
    (df["age"] >= 30) &
    (df["score"] >= 85)
)

Then:

analysis_df = df.loc[
    eligible,
    ["age", "group", "score"]
]

Naming the criterion can make the analysis easier to read, audit, and modify.

26. Always inspect the result¶

Code can execute successfully and still produce the wrong subset.

After filtering, inspect:

analysis_df.shape
analysis_df.head()
analysis_df.index
analysis_df.describe()

Ask:

Does this dataset contain exactly the observations I intended?

27. Common mistakes¶

Mistake 1 — Confusing `.loc` and `.iloc`¶

loc  → labels
iloc → positions

Mistake 2 — Forgetting parentheses¶

Wrong:

df["age"] >= 30 & df["score"] >= 85

Right:

(df["age"] >= 30) & (df["score"] >= 85)

Mistake 3 — Using `and` / `or`¶

Use &, |, and ~ with pandas Boolean Series.

Mistake 4 — Confusing Series and DataFrames¶

df["age"]      # Series
df[["age"]]    # DataFrame

Mistake 5 — Chained assignment¶

Prefer:

df.loc[condition, "column"] = value

28. Connection to the rest of STA 556¶

This week’s work forms the foundation for later data-engineering tasks:

DataFrame
   ↓
Indexing
   ↓
Filtering
   ↓
Joining
   ↓
Reshaping
   ↓
Missing-data strategies
   ↓
External data
   ↓
Statistical analysis

The underlying question remains:

Which observations and variables are we operating on?

29. Key ideas¶

By the end of Week 3, you should be able to explain:

What a pandas DataFrame represents.
The relationship between a DataFrame and Series.
Why data should be inspected before analysis.
How to select one or several columns.
The role of the DataFrame index.
The difference between .loc and .iloc.
How positional and label slicing differ.
How Boolean conditions work.
How to combine multiple conditions.
How .isin() supports categorical filtering.
How .isna() and .notna() identify missingness.
How to assign values safely with .loc.
Why chained assignment should be avoided.
Why filtering is part of the statistical analysis specification.

30. Recommended reading¶

pandas — 10 Minutes to pandas¶

A high-level introduction to DataFrame creation, viewing data, selection, and basic operations.

https://pandas.pydata.org/docs/user_guide/10min.html

pandas — Indexing and selecting data¶

The main pandas reference for .loc, .iloc, slicing, Boolean indexing, and assignment.

https://pandas.pydata.org/docs/user_guide/indexing.html

pandas — How do I select a subset of a DataFrame?¶

An accessible official tutorial focused on selecting columns, filtering rows, and using .loc and .iloc.

https://pandas.pydata.org/docs/getting_started/intro_tutorials/03_subset_data.html

Python for Data Analysis — Wes McKinney¶

The chapters on pandas and data manipulation are especially relevant for Weeks 3 and 4.

https://wesmckinney.com/book/

31. YouTube recommendations¶

1. Data School — “How do I filter rows of a pandas DataFrame by column value?”¶

A clear step-by-step explanation of pandas Boolean filtering. The video builds the filtering syntax from ordinary Python logic, making it particularly useful for understanding what a Boolean mask represents.

Recommended use: Watch before or after the Boolean-filtering portion of the tutorial.

Watch on YouTube

2. StrataScratch — “Pandas iloc, loc and Dataframes — Beginner Python Pandas Tutorial #1”¶

This tutorial introduces Series and DataFrames and demonstrates how .loc and .iloc select subsets of tabular data.

Recommended use: Watch alongside the sections on indexing, slicing, .loc, and .iloc.

Watch on YouTube

3. Sundeep Saradhi Kanthety — “Data Filtering (Using Conditions) in Pandas”¶

A practical walkthrough of conditional filtering in pandas, useful for reinforcing Boolean conditions and filtering syntax after students have attempted the exercises themselves.

Recommended use: Optional reinforcement after the hands-on filtering exercises.

Watch on YouTube

32. Week 3 takeaway¶

The central lesson is:

Indexing and filtering are how we translate statistical inclusion rules into executable code.

DataFrame
   ↓
Inspect
   ↓
Select variables
   ↓
Select observations
   ↓
.loc / .iloc
   ↓
Boolean logic
   ↓
Filter
   ↓
Validate
   ↓
Analysis population

Next week we will extend these ideas to joining, merging, concatenating, reshaping, and missing-data strategies.

Why this week matters¶

1. From Python objects to tabular data¶

2. What is a DataFrame?¶

3. Creating a DataFrame¶

4. Inspect before you analyze¶

5. Series vs. DataFrame¶

6. Selecting columns¶

7. The DataFrame index¶

8. .loc and .iloc¶

9. Selecting by position with .iloc¶

10. Selecting by label with .loc¶

11. .loc vs. .iloc¶

12. Slicing¶

13. Selecting rows and columns together¶

14. Boolean conditions¶

15. Boolean filtering as statistical logic¶

16. Filtering with one condition¶

17. Multiple conditions¶

18. Why not and and or?¶

19. Filter rows and choose variables together¶

20. Filtering categorical values with .isin()¶

21. Missing values¶

22. Assignment with .loc¶

23. Avoid chained assignment¶

24. Filtering defines the analysis population¶

25. Separate criteria from selection¶

26. Always inspect the result¶

27. Common mistakes¶

Mistake 1 — Confusing .loc and .iloc¶

Mistake 2 — Forgetting parentheses¶

Mistake 3 — Using and / or¶

Mistake 4 — Confusing Series and DataFrames¶

Mistake 5 — Chained assignment¶

28. Connection to the rest of STA 556¶

29. Key ideas¶

30. Recommended reading¶

pandas — 10 Minutes to pandas¶

pandas — Indexing and selecting data¶

pandas — How do I select a subset of a DataFrame?¶

Python for Data Analysis — Wes McKinney¶

31. YouTube recommendations¶

1. Data School — “How do I filter rows of a pandas DataFrame by column value?”¶

2. StrataScratch — “Pandas iloc, loc and Dataframes — Beginner Python Pandas Tutorial #1”¶

3. Sundeep Saradhi Kanthety — “Data Filtering (Using Conditions) in Pandas”¶

32. Week 3 takeaway¶

8. `.loc` and `.iloc`¶

9. Selecting by position with `.iloc`¶

10. Selecting by label with `.loc`¶

11. `.loc` vs. `.iloc`¶

18. Why not `and` and `or`?¶

20. Filtering categorical values with `.isin()`¶

22. Assignment with `.loc`¶

Mistake 1 — Confusing `.loc` and `.iloc`¶

Mistake 3 — Using `and` / `or`¶