2.1 Activity

DataFrames, Indexing, Slicing & Filtering¶

Tools: Python, pandas, VS Code, and/or Jupyter

Learning objectives¶

By the end of this tutorial, you should be able to:

• create and inspect pandas DataFrames;

• distinguish Series from DataFrames;

• select columns;

• select rows using .loc and .iloc;

• slice rows and columns;

• create Boolean conditions;

• filter with one or several conditions;

• use .isin(), .isna(), and .notna();

• modify selected observations with .loc;

• translate statistical inclusion criteria into reproducible code;

• validate a filtered dataset.

Part 0 — Set up¶

Use the project/repository you have been building during the earlier parts of this course.

Create:

notebooks/week03_dataframes.ipynb

And at the top create a code cell to import pandas:

import pandas as pd

Part 1 — Create a DataFrame¶

Create:

data = {
    "id": [101, 102, 103, 104, 105, 106],
    "age": [24, 31, 27, 19, 42, 35],
    "group": ["A", "B", "A", "B", "A", "B"],
    "score": [81, 94, 88, 72, 91, 85]
}

df = pd.DataFrame(data)

Display:

df

Then inspect:

df.head()
df.shape
df.columns
df.index
df.dtypes
df.info()
df.describe()

Questions¶

1. How many observations are there?

2. How many variables?

3. Which columns are numeric?

4. What is the default index?

5. What does df.shape return?

Part 2 — Series vs. DataFrame¶

Run:

age = df["age"]
type(age)

Now:

age_df = df[["age"]]
type(age_df)

Question¶

Explain the difference between:

df["age"]

and:

df[["age"]]

Part 3 — Select columns¶

Select:

df["score"]

Then:

df[["age", "score"]]

Challenge¶

Create analysis_df containing only:

• id

• group

• score

Part 4 — Select rows by position¶

Use:

df.iloc[0]

Then:

df.iloc[0:3]

Then:

df.iloc[[0, 2, 5]]

Now select rows 1–3 and columns 1–2:

df.iloc[1:4, 1:3]

Questions¶

1. What does the first argument to .iloc[] control?

2. What does the second argument control?

3. Is the stop value included in positional slicing?

Part 5 — Create a meaningful index¶

Set:

df = df.set_index("id")

Inspect:

df
df.index

Compare:

df.loc[101]

with:

df.iloc[0]

Reflection¶

They identify the same observation here. Why do the two commands mean different things?

Part 6 — Select rows by label¶

Try:

df.loc[101]

df.loc[[101, 103, 105]]

Then:

df.loc[
    [101, 103],
    ["age", "score"]
]

Challenge¶

Select IDs 102, 104, and 106 and display only:

age
group
score

Part 7 — .loc vs. .iloc¶

Create:

small = pd.DataFrame(
    {"value": [10, 20, 30, 40]},
    index=[101, 205, 310, 412]
)

Compare:

small.loc[205]

with:

small.iloc[1]

Complete:

.loc  uses ______________________
.iloc uses ______________________

Part 8 — Slicing¶

Try:

df.iloc[1:4]

Then:

df.loc[102:105]

Count the returned rows.

Question¶

Why does label-based slicing behave differently from positional slicing?

Part 9 — Select rows and columns together¶

Try:

df.loc[:, ["age", "score"]]

Then:

df.loc[
    102:105,
    ["age", "score"]
]

Challenge¶

Select IDs 101 through 104 and show only group and score.

Part 10 — Create a Boolean condition¶

Run:

condition = df["score"] > 80

Inspect:

condition
type(condition)

Now:

df.loc[condition]

Question¶

What does each True and False represent?

Part 11 — Simple filters¶

Find participants:

df.loc[df["score"] > 80]

Then:

df.loc[df["age"] >= 30]

Then:

df.loc[df["group"] == "A"]

Challenge¶

Create separate subsets for participants who are:

• younger than 30;

• in group B;

• scoring at least 85.

Part 12 — Multiple conditions¶

Use AND:

df.loc[
    (df["age"] < 30) &
    (df["score"] > 80)
]

Use OR:

df.loc[
    (df["group"] == "A") |
    (df["score"] > 90)
]

Use NOT:

df.loc[
    ~(df["group"] == "A")
]

Challenge¶

Find participants who are at least 30 years old and score at least 85.

Part 13 — Predict before running¶

Without executing the code, predict which IDs will appear:

df.loc[
    (df["group"] == "A") &
    (df["score"] >= 90)
]

Write down your prediction and then run the code.

Question¶

Did your result match your prediction?

Part 14 — Filter rows and choose variables¶

Try:

df.loc[
    df["score"] >= 85,
    ["age", "group", "score"]
]

Challenge¶

Return only group and score for participants aged at least 30.

Part 15 — .isin()¶

Create:

df2 = pd.DataFrame(
    {
        "id": range(201, 209),
        "group": ["A", "B", "C", "A", "C", "B", "A", "C"],
        "score": [81, 94, 88, 72, 91, 85, 79, 96]
    }
).set_index("id")

Select groups A and C:

df2.loc[
    df2["group"].isin(["A", "C"])
]

Challenge¶

Select observations in groups B or C and with score at least 85.

Part 16 — Missing values¶

Create:

missing_df = pd.DataFrame(
    {
        "id": [1, 2, 3, 4, 5],
        "age": [21, 32, None, 45, 28],
        "score": [81, None, 77, 92, 88]
    }
).set_index("id")

Inspect:

missing_df
missing_df.isna()
missing_df.isna().sum()

Find missing scores:

missing_df.loc[
    missing_df["score"].isna()
]

Find observed scores:

missing_df.loc[
    missing_df["score"].notna()
]

Reflection¶

Why is score = 0 different from score = missing in statistical analysis?

Part 17 — Assign values with .loc¶

Return to df.

df["performance"] = "medium"

Then:

df.loc[
    df["score"] >= 90,
    "performance"
] = "high"

Then:

df.loc[
    df["score"] < 80,
    "performance"
] = "low"

Challenge¶

Create an age_group variable:

young → age < 30
older → age >= 30

using .loc.

Part 18 — Avoid chained assignment¶

Consider:

df[df["score"] > 80]["score"] = 100

Prefer:

df.loc[
    df["score"] > 80,
    "score"
] = 100

Question¶

Why is the .loc version clearer? Connect your answer to Week 2’s discussion of references and mutation.

Part 19 — Define an analysis population¶

Suppose the study protocol says:

Include adults in group A who have a non-missing score.

Write:

analysis = df.loc[
    (df["age"] >= 18) &
    (df["group"] == "A") &
    (df["score"].notna())
]

Part 20 — Write criteria first¶

Suppose your analysis question is:

What are the scores among older high-performing participants?

Before writing code, define:

Age criterion:
Score criterion:
Missing-data criterion:
Variables required:

Then translate the rules into pandas:

eligible = (
    (df["age"] >= 30) &
    (df["score"] >= 85) &
    (df["score"].notna())
)

analysis_df = df.loc[
    eligible,
    ["age", "group", "score"]
]

Part 21 — Mini data-analysis challenge¶

Create:

participants = pd.DataFrame(
    {
        "id": range(1001, 1011),
        "age": [22, 34, 29, 41, 19, 37, 26, 45, 31, 28],
        "group": ["A", "B", "A", "C", "B", "A", "C", "B", "A", "C"],
        "score": [78, 91, 85, 88, 72, 95, 81, 67, 90, 84]
    }
).set_index("id")

Complete each task.

A. Participants aged 30 or older¶

older = ...

B. Participants in groups A or C¶

groups_ac = ...

C. Participants with scores at least 85¶

high_scores = ...

D. Participants aged at least 30 AND scoring at least 85¶

older_high = ...

E. Only group and score for high scorers¶

high_score_summary = ...

F. Group A participants scoring at least 80¶

group_a_good = ...

G. Participants who are NOT in group B¶

not_group_b = ...

Part 22 — Verify every answer¶

For each result, inspect:

result.shape
result.head()
result.index

Then manually compare it with the original dataset.

Ask:

Does every included row satisfy the condition?

and:

Did I exclude anyone who should be included?

Code that executes successfully is not necessarily correct.

Part 23 — Function challenge¶

Write a reusable function:

def select_high_scores(
    data: pd.DataFrame,
    minimum_score: float
) -> pd.DataFrame:
    ...

It should return observations whose scores are at least minimum_score.

Test:

select_high_scores(participants, 85)

and:

select_high_scores(participants, 90)

Extension¶

Modify the function so the caller can also specify which columns to return.

Part 24 — Reproducibility challenge¶

Imagine you wrote:

good_data = participants.loc[
    participants["score"] > 80
]

Six months later, someone asks:

Why did you choose 80?

Improve the workflow:

MINIMUM_SCORE = 80

good_data = participants.loc[
    participants["score"] > MINIMUM_SCORE
]

Discussion¶

Does naming the threshold improve reproducibility? What documentation is still missing?

Part 25 — Git checkpoint¶

Save your work.

git status

Then:

git add .

Commit:

git commit -m "Complete Week 3 pandas indexing and filtering"

Push:

git push

Continue the workflow established in Week 1:

Create
   ↓
Inspect
   ↓
Transform
   ↓
Validate
   ↓
Document
   ↓
Commit
   ↓
Share

Part 26 — Final reflection¶

Answer in Markdown.

1. DataFrames¶

What does a pandas DataFrame provide that a list of dictionaries does not?

2. Series¶

What is the difference between:

df["age"]

and:

df[["age"]]

?

3. .loc vs. .iloc¶

Explain the difference in one sentence.

4. Boolean filtering¶

What does:

df["score"] >= 85

produce before it is used to subset a DataFrame?

5. Analysis populations¶

Why is a pandas filter more than simply a programming operation?

6. Validation¶

Why should you inspect a filtered dataset even if no Python error occurred?

7. Reproducibility¶

How can explicit filtering criteria make statistical analyses easier to reproduce?

Completion checklist¶

• Created the Week 3 notebook

• Imported pandas

• Created a DataFrame

• Used head(), shape, dtypes, info(), and describe()

• Distinguished Series and DataFrames

• Selected individual columns

• Selected multiple columns

• Used .iloc

• Used .loc

• Created a meaningful index

• Compared positional and label selection

• Practiced slicing

• Created Boolean conditions

• Filtered using one condition

• Filtered using multiple conditions

• Used &, |, and ~

• Used .isin()

• Used .isna() and .notna()

• Assigned values with .loc

• Avoided chained assignment

• Defined an analysis population

• Completed the mini data-analysis challenge

• Wrote a reusable filtering function

• Validated filtered results

• Committed Week 3 work to Git

• Pushed Week 3 work to GitHub

What you should now understand¶

Raw tabular data
       ↓
DataFrame
       ↓
Inspect
       ↓
Select variables
       ↓
Define conditions
       ↓
Filter observations
       ↓
Validate
       ↓
Analysis-ready subset

Next week we will extend the wrangling workflow to joins, merges, concatenation, reshaping, and missing-data strategies.