Activity 2.1

This page is an ipython notebook, meaning you can download it and run the executable cells. It directly follows Activity 2.1. Reference answers appear after the complete Part they answer, so they do not interrupt setup code, comparison examples, or multi-cell questions. They show one reasonable solution or interpretation, but other correct approaches may exist.

DataFrames, Indexing, Slicing & Filtering¶

Tools: Python, pandas, VS Code, and/or Jupyter

Learning objectives¶

By the end of this tutorial, you should be able to:

• create and inspect pandas DataFrames;

• distinguish Series from DataFrames;

• select columns;

• select rows using .loc and .iloc;

• slice rows and columns;

• create Boolean conditions;

• filter with one or several conditions;

• use .isin(), .isna(), and .notna();

• modify selected observations with .loc;

• translate statistical inclusion criteria into reproducible code;

• validate a filtered dataset.

Part 0 — Set up¶

Use the project/repository you have been building during the earlier parts of this course.

Create:

And at the top create a code cell to import pandas:

Part 1 — Create a DataFrame¶

Create:

Display:

Then inspect:

Questions¶

1. How many observations are there?

2. How many variables?

3. Which columns are numeric?

4. What is the default index?

5. What does df.shape return?

Reference answer¶

1. There are 6 observations.

2. There are 4 variables.

3. The numeric columns are id, age, and score.

4. The default index is a RangeIndex running from 0 to 5.

5. df.shape returns:

(6, 4)

The first value is the number of rows and the second is the number of columns.

Part 2 — Series vs. DataFrame¶

Run:

Now:

Question¶

Explain the difference between:

and:

Reference answer¶

df["age"]

returns a pandas Series, which is one-dimensional.

df[["age"]]

returns a DataFrame containing one column, so it remains two-dimensional.

This difference can matter because some pandas operations expect a Series while others expect a DataFrame.

Part 3 — Select columns¶

Select:

Then:

Challenge¶

Create analysis_df containing only:

• id

• group

• score

Reference answer¶

analysis_df = df[["id", "group", "score"]]

Part 4 — Select rows by position¶

Use:

Then:

Then:

Now select rows 1–3 and columns 1–2:

Questions¶

1. What does the first argument to .iloc[] control?

2. What does the second argument control?

3. Is the stop value included in positional slicing?

Reference answer¶

1. The first argument to .iloc[] controls the rows selected by integer position.

2. The second argument controls the columns selected by integer position.

3. The stop value in a positional slice is not included.

For example:

df.iloc[1:4, 1:3]

selects row positions 1, 2, 3 and column positions 1, 2.

Part 5 — Create a meaningful index¶

Set:

Inspect:

Compare:

with:

Reflection¶

They identify the same observation here. Why do the two commands mean different things?

Reference answer¶

Although these identify the same observation here, they use different rules:

df.loc[101]

selects the row whose index label is 101.

df.iloc[0]

selects the first row by position.

They only agree here because label 101 happens to be on the first row.

Part 6 — Select rows by label¶

Try:

Then:

Challenge¶

Select IDs 102, 104, and 106 and display only:

Reference answer¶

df.loc[
    [102, 104, 106],
    ["age", "group", "score"]
]

Part 7 — .loc vs. .iloc¶

Create:

Compare:

with:

Complete:

Reference answer¶

.loc  uses index labels
.iloc uses integer positions

In the example:

small.loc[205]

uses the label 205, while:

small.iloc[1]

uses position 1, meaning the second row.

Part 8 — Slicing¶

Try:

Then:

Count the returned rows.

Question¶

Why does label-based slicing behave differently from positional slicing?

Reference answer¶

df.iloc[1:4]

returns 3 rows because positional slicing excludes the stop position.

df.loc[102:105]

returns 4 rows: labels 102, 103, 104, and 105.

Label-based slicing with .loc includes the ending label when that label exists in the index.

Part 9 — Select rows and columns together¶

Try:

Then:

Challenge¶

Select IDs 101 through 104 and show only group and score.

Reference answer¶

df.loc[
    101:104,
    ["group", "score"]
]

Because .loc uses labels, IDs 101, 102, 103, and 104 are included.

Part 10 — Create a Boolean condition¶

Run:

Inspect:

Now:

Question¶

What does each True and False represent?

Reference answer¶

Each True or False corresponds to one row of the DataFrame.

For:

condition = df["score"] > 80

True means that row has a score greater than 80, while False means it does not.

The Boolean Series can then be used as a row filter.

Part 11 — Simple filters¶

Find participants:

Then:

Then:

Challenge¶

Create separate subsets for participants who are:

• younger than 30;

• in group B;

• scoring at least 85.

Reference answer¶

Younger than 30:

younger = df.loc[df["age"] < 30]

IDs: 101, 103, 104

Group B:

group_b = df.loc[df["group"] == "B"]

IDs: 102, 104, 106

Score at least 85:

high_scores = df.loc[df["score"] >= 85]

IDs: 102, 103, 105, 106

Part 12 — Multiple conditions¶

Use AND:

Use OR:

Use NOT:

Challenge¶

Find participants who are at least 30 years old and score at least 85.

Reference answer¶

df.loc[
    (df["age"] >= 30)
    & (df["score"] >= 85)
]

This returns IDs:

102, 105, 106

Part 13 — Predict before running¶

Without executing the code, predict which IDs will appear:

Write down your prediction and then run the code.

Question¶

Did your result match your prediction?

Reference answer¶

The only matching ID is:

105

because ID 105 is in group A and has a score of 91, which is at least 90.

If your prediction matched this result, your Boolean conditions were interpreted correctly.

Part 14 — Filter rows and choose variables¶

Try:

Challenge¶

Return only group and score for participants aged at least 30.

Reference answer¶

df.loc[
    df["age"] >= 30,
    ["group", "score"]
]

This returns IDs 102, 105, and 106 with only the requested columns.

Part 15 — .isin()¶

Create:

Select groups A and C:

Challenge¶

Select observations in groups B or C and with score at least 85.

Reference answer¶

df2.loc[
    df2["group"].isin(["B", "C"])
    & (df2["score"] >= 85)
]

The matching IDs are:

202, 203, 205, 206, 208

Part 16 — Missing values¶

Create:

Inspect:

Find missing scores:

Find observed scores:

Reflection¶

Why is score = 0 different from score = missing in statistical analysis?

Reference answer¶

A score of 0 is an observed value. It means the score was measured and its value was zero.

A missing score means the value was not observed or is unavailable.

Treating missing values as zero would change the meaning of the data and could distort summaries such as means, distributions, and model estimates.

Part 17 — Assign values with .loc¶

Return to df.

Then:

Then:

Challenge¶

Create an age_group variable:

using .loc.

Reference answer¶

One clear solution is:

df["age_group"] = "older"

df.loc[
    df["age"] < 30,
    "age_group"
] = "young"

Alternatively, you could initialize "young" first and then assign "older" to ages 30 or above.

The important point is that .loc explicitly identifies both the rows being changed and the column being changed.

Part 18 — Avoid chained assignment¶

Consider:

Prefer:

Question¶

Why is the .loc version clearer? Connect your answer to Week 2’s discussion of references and mutation.

Reference answer¶

The .loc version is clearer because it performs the row selection and assignment in one explicit operation:

df.loc[
    df["score"] > 80,
    "score"
] = 100

By contrast:

df[df["score"] > 80]["score"] = 100

first creates an intermediate object and then tries to modify it. That intermediate object may be a copy rather than the original DataFrame.

This connects to the earlier discussion of references and mutation: it is important to know which object is actually being modified.

Part 19 — Define an analysis population¶

Suppose the study protocol says:

Include adults in group A who have a non-missing score.

Write:

Reference answer¶

The filter is:

analysis = df.loc[
    (df["age"] >= 18)
    & (df["group"] == "A")
    & (df["score"].notna())
]

For the current dataset, the included IDs are:

101, 103, 105

The key idea is that these conditions define the analysis population, not merely a convenient subset of rows.

Part 20 — Write criteria first¶

Suppose your analysis question is:

What are the scores among older high-performing participants?

Before writing code, define:

Then translate the rules into pandas:

Reference answer¶

A reasonable set of criteria is:

Age criterion: age >= 30
Score criterion: score >= 85
Missing-data criterion: score must be observed
Variables required: age, group, score

The supplied pandas code implements exactly those rules.

With the current df, the resulting IDs are:

102, 105, 106

Part 21 — Mini data-analysis challenge¶

Create:

Complete each task.

A. Participants aged 30 or older¶

B. Participants in groups A or C¶

C. Participants with scores at least 85¶

D. Participants aged at least 30 AND scoring at least 85¶

E. Only group and score for high scorers¶

F. Group A participants scoring at least 80¶

G. Participants who are NOT in group B¶

Reference answer¶

A. Participants aged 30 or older¶

older = participants.loc[
    participants["age"] >= 30
]

IDs: 1002, 1004, 1006, 1008, 1009

B. Participants in groups A or C¶

groups_ac = participants.loc[
    participants["group"].isin(["A", "C"])
]

IDs: 1001, 1003, 1004, 1006, 1007, 1009, 1010

C. Participants with scores at least 85¶

high_scores = participants.loc[
    participants["score"] >= 85
]

IDs: 1002, 1003, 1004, 1006, 1009

D. Participants aged at least 30 AND scoring at least 85¶

older_high = participants.loc[
    (participants["age"] >= 30)
    & (participants["score"] >= 85)
]

IDs: 1002, 1004, 1006, 1009

E. Only group and score for high scorers¶

high_score_summary = participants.loc[
    participants["score"] >= 85,
    ["group", "score"]
]

F. Group A participants scoring at least 80¶

group_a_good = participants.loc[
    (participants["group"] == "A")
    & (participants["score"] >= 80)
]

IDs: 1003, 1006, 1009

G. Participants who are NOT in group B¶

not_group_b = participants.loc[
    participants["group"] != "B"
]

IDs: 1001, 1003, 1004, 1006, 1007, 1009, 1010

Part 22 — Verify every answer¶

For each result, inspect:

Then manually compare it with the original dataset.

Ask:

Does every included row satisfy the condition?

and:

Did I exclude anyone who should be included?

Code that executes successfully is not necessarily correct.

Reference answer¶

Successful execution does not prove that a filter is logically correct.

For each result, check both directions:

• Every included row should satisfy the stated criterion.

• Every row that satisfies the criterion should have been included.

Inspecting .shape, .head(), and .index helps verify that the result matches the intended analysis population.

Part 23 — Function challenge¶

Write a reusable function:

It should return observations whose scores are at least minimum_score.

Test:

and:

Extension¶

Modify the function so the caller can also specify which columns to return.

Reference answer¶

A reusable function is:

def select_high_scores(
    data: pd.DataFrame,
    minimum_score: float
) -> pd.DataFrame:
    return data.loc[
        data["score"] >= minimum_score
    ]

For the extension:

def select_high_scores(
    data: pd.DataFrame,
    minimum_score: float,
    columns=None
) -> pd.DataFrame:
    mask = data["score"] >= minimum_score

    if columns is None:
        return data.loc[mask]

    return data.loc[
        mask,
        columns
    ]

Example:

select_high_scores(
    participants,
    85,
    ["group", "score"]
)

Part 24 — Reproducibility challenge¶

Imagine you wrote:

Six months later, someone asks:

Why did you choose 80?

Improve the workflow:

Discussion¶

Does naming the threshold improve reproducibility? What documentation is still missing?

Reference answer¶

Naming the threshold improves reproducibility because the value is explicit and can be changed in one place:

MINIMUM_SCORE = 80

However, the name alone does not explain why 80 was chosen.

Useful documentation might include:

• the scientific or practical rationale for the threshold;

• whether the rule should be > 80 or >= 80;

• the source of the rule (protocol, policy, prior literature, etc.);

• when or why the threshold was changed.

Reproducibility requires both executable code and enough context to understand the analytical choices.

Part 25 — Git checkpoint¶

Save your work.

Then:

Commit:

Push:

Continue the workflow established in Week 1:

Part 26 — Final reflection¶

Answer in Markdown.

1. DataFrames¶

What does a pandas DataFrame provide that a list of dictionaries does not?

2. Series¶

What is the difference between:

and:

?

3. .loc vs. .iloc¶

Explain the difference in one sentence.

4. Boolean filtering¶

What does:

produce before it is used to subset a DataFrame?

5. Analysis populations¶

Why is a pandas filter more than simply a programming operation?

6. Validation¶

Why should you inspect a filtered dataset even if no Python error occurred?

7. Reproducibility¶

How can explicit filtering criteria make statistical analyses easier to reproduce?

Completion checklist¶

• Created the Week 3 notebook

• Imported pandas

• Created a DataFrame

• Used head(), shape, dtypes, info(), and describe()

• Distinguished Series and DataFrames

• Selected individual columns

• Selected multiple columns

• Used .iloc

• Used .loc

• Created a meaningful index

• Compared positional and label selection

• Practiced slicing

• Created Boolean conditions

• Filtered using one condition

• Filtered using multiple conditions

• Used &, |, and ~

• Used .isin()

• Used .isna() and .notna()

• Assigned values with .loc

• Avoided chained assignment

• Defined an analysis population

• Completed the mini data-analysis challenge

• Wrote a reusable filtering function

• Validated filtered results

• Committed Week 3 work to Git

• Pushed Week 3 work to GitHub

What you should now understand¶

Next week we will extend the wrangling workflow to joins, merges, concatenation, reshaping, and missing-data strategies.

Reference answer¶

1. DataFrames¶

A DataFrame provides labeled rows and columns plus tabular operations such as filtering, grouping, missing-value handling, joins, dtype management, and vectorized computation. A list of dictionaries can represent similar records, but does not provide these operations directly.

2. Series¶

df["age"]

returns a one-dimensional Series.

df[["age"]]

returns a one-column DataFrame.

3. .loc vs. .iloc¶

.loc selects using labels, while .iloc selects using integer positions.

4. Boolean filtering¶

df["score"] >= 85

produces a Boolean Series containing one True or False value for each row.

5. Analysis populations¶

A filter is more than a programming operation because it defines which observations are included in the statistical analysis and therefore affects the scientific question being answered.

6. Validation¶

Code can run without errors while still implementing the wrong criterion. Inspecting the result helps detect logical errors.

7. Reproducibility¶

Explicit criteria make the inclusion/exclusion rules visible in code so another analyst can understand and recreate the same analysis population.