Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Activity 3.1

This page is an ipython notebook, meaning you can download it and run the executable cells. It directly follows Activity 3.1. Reference answers appear after the questions, challenges, predictions, and reflections. They show one reasonable solution or interpretation, but other correct approaches may exist.

Exploratory Data Analysis & Visualization

Estimated time: 2–3 hours
Tools: Python, pandas, matplotlib, seaborn, plotnine, Jupyter/VS Code

Learning objectives

By the end of this tutorial, you should be able to:

  • use plots as part of exploratory statistical analysis;

  • select plots based on variable type;

  • create histograms, boxplots, count plots, and scatterplots;

  • compare distributions across groups;

  • recognize the effect of histogram binning;

  • use seaborn for statistical visualization;

  • construct plots using Grammar of Graphics concepts;

  • distinguish mapped and fixed aesthetics;

  • use layers and facets;

  • improve labels, scales, and figure design;

  • save publication-quality figures;

  • identify data-quality problems visually.

Part 0 — Set up

Create:

notebooks/week06_visualization.ipynb

Ensure the project contains:

figures/

Import:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

If plotnine is installed:

from plotnine import (
    ggplot,
    aes,
    geom_point,
    geom_boxplot,
    geom_jitter,
    geom_smooth,
    facet_wrap,
    labs,
    theme_minimal
)

Part 1 — Create an analysis dataset

Use a reproducible random-number generator:

rng = np.random.default_rng(556)

Create:

n = 150

df = pd.DataFrame({
    "id": range(1, n + 1),
    "group": rng.choice(
        ["A", "B", "C"],
        size=n
    ),
    "age": rng.normal(
        loc=35,
        scale=10,
        size=n
    ),
    "baseline": rng.normal(
        loc=70,
        scale=12,
        size=n
    )
})

Add an outcome with structure:

group_effect = df["group"].map({
    "A": 0,
    "B": 5,
    "C": -4
})

df["score"] = (
    40
    + 0.7 * df["baseline"]
    + 0.25 * df["age"]
    + group_effect
    + rng.normal(0, 8, size=n)
)

Round age:

df["age"] = df["age"].round(1)

Inspect:

df.head()
df.shape
df.dtypes
df.describe()

Part 2 — Identify variable types

Classify each variable:

id:
group:
age:
baseline:
score:

Possible labels:

identifier
categorical
continuous
discrete numeric

Question

Why should variable type be considered before choosing a visualization?

Part 3 — One continuous variable

Start numerically:

df["score"].describe()

Then visualize:

plt.hist(
    df["score"]
)

plt.show()

Questions

  1. Where is the center?

  2. Is the distribution symmetric?

  3. Are there unusual values?

  4. What does the histogram reveal that describe() does not?

Reference answer

Choose the display to match the variable type and statistical question. A histogram or density plot suits one continuous variable; a bar chart suits categories; a scatterplot suits two continuous variables.

Part 4 — Investigate binning

Compare:

plt.hist(df["score"], bins=5)
plt.show()
plt.hist(df["score"], bins=15)
plt.show()
plt.hist(df["score"], bins=40)
plt.show()

Reflection

How does the visual impression change? Is there one objectively correct number of bins?

Reference answer

The histogram shows distributional shape—such as skewness, gaps, multimodality, and unusual values—that numerical summaries may hide. There is no single objectively correct bin count; different bin widths show the data at different resolutions.

Part 5 — Density plot

sns.kdeplot(
    data=df,
    x="score"
)

plt.show()

Compare the density with the histogram.

Questions

  1. What features are easier to see?

  2. What information has been smoothed?

  3. Why is this not the raw data?

Reference answer

A density curve makes broad shape easier to see, but it smooths away individual observations and exact counts. It is an estimate of distribution shape, not the raw data.

Part 6 — One categorical variable

df["group"].value_counts()

Then:

sns.countplot(
    data=df,
    x="group"
)

plt.show()

Calculate proportions:

df["group"].value_counts(
    normalize=True
)

Explain the difference between counts and proportions.

Reference answer

Counts report how many observations fall in each category; proportions divide by the total and are easier to compare when sample sizes differ.

Part 7 — Continuous variable by group

Start with a boxplot:

sns.boxplot(
    data=df,
    x="group",
    y="score"
)

plt.show()

Questions

  1. Which group has the highest median score?

  2. Which group has the greatest spread in the middle 50% of its scores?

  3. What information does the boxplot summarize well?

  4. What information does the boxplot hide?

Reference answer

  1. Group B has the highest median score.

  2. Group A has the greatest spread in the middle 50% of its scores, as indicated by the largest interquartile range (the height of the box).

  3. The boxplot is useful for comparing:

    • medians;

    • quartiles;

    • interquartile ranges;

    • overall spread;

    • potential outliers.

  4. The boxplot hides the individual observations and much of the detailed distributional shape. For example, it does not show exactly how many observations are in each group, whether values cluster within part of the box, or whether there are gaps or multiple modes.

For this simulated dataset, the group medians are approximately:

A:  97.0
B: 105.2
C:  92.0

So the ordering of typical scores is approximately:

B > A > C

Part 8 — Add raw observations

Now overlay the individual observations on the boxplot:

fig, ax = plt.subplots()

sns.boxplot(
    data=df,
    x="group",
    y="score",
    ax=ax
)

sns.stripplot(
    data=df,
    x="group",
    y="score",
    alpha=0.5,
    jitter=True,
    ax=ax
)

plt.show()

Questions

  1. What additional information becomes visible when the individual observations are shown?

  2. Does seeing the raw observations change your interpretation of which group has the highest typical score or the greatest variability?

  3. Why can showing the raw observations be especially important when group sample sizes are small or unequal?

Reference answer

  1. The overlaid points show the actual observations behind the boxplot. This makes it easier to see:

    • the approximate sample size in each group;

    • where observations are concentrated;

    • gaps or clusters;

    • individual extreme values;

    • whether the boxplot is summarizing many observations or only a few.

  2. For this dataset, the raw observations do not substantially change the main conclusions from the boxplot. Group B still has the highest typical score, while Group A appears to have the greatest spread in the middle 50% of its scores. The value of the strip plot here is that it lets us verify what the summary is based on.

  3. A boxplot can look quite similar for a group with 10 observations and a group with 100 observations. Showing the raw data makes sample size and data density visible, which is particularly important for small or unequal groups.

Part 9 — Compare boxplots, strip plots, and violin plots

Now create a violin plot:

sns.violinplot(
    data=df,
    x="group",
    y="score"
)

plt.show()

You have now examined the same grouped distributions in three different ways:

boxplot
boxplot + strip plot
violin plot

Questions

  1. Which display makes the median and quartiles easiest to compare?

  2. Which display makes the individual observations and sample sizes easiest to see?

  3. Which display makes the overall distributional shape easiest to see?

  4. What can a violin plot reveal that a boxplot may hide?

  5. Based on all three displays, briefly summarize how the score distributions differ among groups A, B, and C.

The goal is not to choose one universally “best” plot. Different displays emphasize different features of the same data.

Reference answer

  1. The boxplot makes the median, quartiles, and interquartile range easiest to compare directly.

  2. The boxplot + strip plot makes the individual observations and approximate sample sizes easiest to see.

  3. The violin plot makes the smoothed distributional shape easiest to see.

  4. A violin plot can reveal features such as:

    • skewness;

    • areas of high or low density;

    • multiple peaks or modes;

    • differences in distributional shape that may produce similar boxplot summaries.

    However, the violin is based on a smoothed density estimate, so it does not show the raw observations directly.

  5. A reasonable summary for this simulated dataset is:

    • Group B has the highest typical scores.

    • Group C has the lowest typical scores.

    • Group A is intermediate in center and has the largest interquartile spread.

    • The three distributions overlap substantially, so group membership does not perfectly separate scores.

The three displays answer complementary questions:

boxplot            → center and quartiles
boxplot + points   → summary plus actual observations
violin plot        → smoothed distributional shape

Part 10 — Two continuous variables

sns.scatterplot(
    data=df,
    x="baseline",
    y="score"
)

plt.show()

Then:

df[
    ["baseline", "score"]
].corr()

Questions

  1. Is the relationship approximately linear?

  2. Are there outliers?

  3. Does variance appear constant?

  4. What does the figure show that correlation does not?

Reference answer

A scatterplot lets you assess direction, form, strength, outliers, and changing variance in the relationship between two continuous variables.

Part 11 — Add a third variable

sns.scatterplot(
    data=df,
    x="baseline",
    y="score",
    hue="group"
)

plt.show()

What new statistical question can this figure address?

Reference answer

Color encodes a third variable, allowing the two-variable relationship to be compared across groups. It can reveal group-specific structure that an overall correlation hides.

Part 12 — Faceting

g = sns.relplot(
    data=df,
    x="baseline",
    y="score",
    col="group"
)

Compare faceting with color. When are separate panels easier to interpret?

Reference answer

Faceting separates groups into panels and is often easier than color when groups overlap strongly or precise within-group comparison matters.

Part 13 — Introduce a data-quality problem

bad_rows = pd.DataFrame({
    "id": [151, 152],
    "group": ["A", "B"],
    "age": [999, 31],
    "baseline": [72, 300],
    "score": [82, 90]
})

Combine:

dirty = pd.concat(
    [df, bad_rows],
    ignore_index=True
)

Visualize:

sns.histplot(
    data=dirty,
    x="age"
)

plt.show()
sns.scatterplot(
    data=dirty,
    x="baseline",
    y="score"
)

plt.show()

What problems become obvious visually?

Reference answer

Suspicious observations can appear as implausible values, isolated points, or patterns inconsistent with the rest of the data. Visualization is useful for discovering these problems before modeling.

Part 14 — Find problematic observations

dirty.loc[
    dirty["age"] > 120
]
dirty.loc[
    dirty["baseline"] > 150
]

How does visualization complement explicit validation rules?

Reference answer

Filter or sort to identify the observations causing the suspicious pattern, then determine whether they are valid extremes, data-entry errors, or coding problems.

Part 15 — Grammar of Graphics

Using plotnine:

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score"
        )
    )
    + geom_point()
)

Identify:

data:
x aesthetic:
y aesthetic:
geometry:

Part 16 — Map an aesthetic

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score",
            color="group"
        )
    )
    + geom_point()
)

What information does color now encode?

Part 17 — Mapping vs. setting

Compare:

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score",
            color="group"
        )
    )
    + geom_point()
)

with:

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score"
        )
    )
    + geom_point(
        color="black"
    )
)

Explain the difference between mapped and fixed color.

Reference answer

Mapped color is tied to a variable, so observations can receive different colors based on data values. Fixed color gives an entire layer the same appearance and does not encode data.

Part 18 — Add a layer

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score"
        )
    )
    + geom_point()
    + geom_smooth()
)

What information does the second geometry add?

Reference answer

A second geometry adds complementary information—for example, raw points can show observations while a summary or smooth shows the broader pattern.

Part 19 — Facet in plotnine

(
    ggplot(
        df,
        aes(
            x="baseline",
            y="score"
        )
    )
    + geom_point()
    + facet_wrap("group")
)

Compare this with mapping group to color.

Reference answer

Faceting reduces overlap by separating groups; color overlays groups in one panel. Faceting is often clearer when overlap is severe.

Part 20 — Grouped distribution with layers

(
    ggplot(
        df,
        aes(
            x="group",
            y="score"
        )
    )
    + geom_boxplot()
    + geom_jitter(
        width=0.15,
        alpha=0.5
    )
)

Which layer provides a summary? Which shows raw observations?

Reference answer

Combining a distribution summary with raw observations gives both the statistical summary and the underlying data.

Part 21 — Make a bad plot deliberately

Create a cluttered plot using too many visual encodings such as color, size, and shape.

Then answer:

  1. Is more encoded information always better?

  2. Which encodings are useful?

  3. Which would you remove?

Simplify the plot.

Part 22 — Titles and units

fig, ax = plt.subplots(
    figsize=(7, 5)
)

sns.scatterplot(
    data=df,
    x="age",
    y="score",
    hue="group",
    ax=ax
)

ax.set(
    title="Assessment score by participant age",
    xlabel="Age (years)",
    ylabel="Assessment score"
)

fig.tight_layout()

How does this differ from the default exploratory figure?

Part 23 — Axis experiment

Create group means:

group_means = (
    df.groupby(
        "group",
        as_index=False
    )["score"]
    .mean()
)

Plot them with a bar chart and experiment with y-axis limits.

Reflection

How can a truncated baseline exaggerate group differences?

Part 24 — Compare bar and point plots

sns.pointplot(
    data=df,
    x="group",
    y="score"
)

plt.show()

If the goal is comparing group means rather than quantities represented by bar lengths, which display is easier to interpret?

Reference answer

If the goal is comparing group means, a point plot is usually easier to interpret than bars because position directly represents the estimate without unnecessary bar area.

Part 25 — Overplotting

large = pd.DataFrame({
    "x": rng.normal(size=5000),
    "y": rng.normal(size=5000)
})

Plot:

sns.scatterplot(
    data=large,
    x="x",
    y="y"
)

plt.show()

Then:

sns.scatterplot(
    data=large,
    x="x",
    y="y",
    alpha=0.2
)

plt.show()

Why does transparency help?

Reference answer

Transparency helps because overlapping points accumulate visually; dense regions become apparent while individual observations remain visible.

Part 26 — Hexbin alternative

plt.hexbin(
    large["x"],
    large["y"],
    gridsize=30
)

plt.xlabel("x")
plt.ylabel("y")
plt.show()

How does this represent dense regions differently?

Reference answer

Hexbin plots aggregate nearby observations into spatial bins, making density patterns visible when ordinary scatterplots suffer from severe overplotting.

Part 27 — Visualize missingness

missing = df.copy()

missing.loc[
    missing.sample(
        15,
        random_state=556
    ).index,
    "score"
] = np.nan

Create an indicator:

missing["score_missing"] = (
    missing["score"].isna()
)

Plot:

sns.countplot(
    data=missing,
    x="group",
    hue="score_missing"
)

plt.show()

Does missingness appear equally distributed across groups?

Part 28 — Publication-quality figure

fig, ax = plt.subplots(
    figsize=(7, 5)
)

sns.scatterplot(
    data=df,
    x="baseline",
    y="score",
    hue="group",
    ax=ax
)

ax.set(
    title=(
        "Follow-up score increases "
        "with baseline score"
    ),
    xlabel="Baseline assessment score",
    ylabel="Follow-up assessment score"
)

ax.legend(
    title="Study group"
)

fig.tight_layout()

Inspect the figure as a reader rather than as its author.

Part 29 — Save the figure

figure_path = (
    "figures/"
    "week06_baseline_vs_score.png"
)

Save:

fig.savefig(
    figure_path,
    dpi=300,
    bbox_inches="tight"
)

Also save as PDF:

fig.savefig(
    "figures/"
    "week06_baseline_vs_score.pdf",
    bbox_inches="tight"
)

Part 30 — EDA vs. final figure

Create two versions of the same plot.

Version A — exploratory

Fast and minimally styled.

Version B — communication

Include:

  • descriptive title;

  • informative labels;

  • units where relevant;

  • clear legend;

  • suitable figure size;

  • intentional output format.

Why is it inefficient to fully polish every exploratory figure?

Reference answer

EDA figures are for discovering and diagnosing patterns; final figures are for communicating a focused result. Final figures should usually be simpler, clearly labeled, and tailored to the supported message.

Part 31 — Visualization design challenge

Choose a visualization for each question before writing code.

A

What is the distribution of participant age?

B

How many participants are in each study group?

C

How does score vary across groups?

D

Is baseline score associated with follow-up score?

E

Does that relationship differ by group?

F

Are there unusual age values?

G

How does the distribution of score differ among groups?

Explain each choice.

Part 32 — Mini EDA challenge

Using df, answer:

  1. Which group has the highest typical score?

  2. Which group has the greatest variability?

  3. Is age related to score?

  4. Is baseline related to score?

  5. Does the baseline-score relationship appear similar across groups?

  6. Are there observations deserving further investigation?

For each include:

statistical question
numerical summary
visualization
brief interpretation

Part 33 — Critique your own plot

Choose one figure.

Statistical

  • What question does it answer?

  • Are the correct variables represented?

  • Is anything important hidden?

Design

  • Is the title meaningful?

  • Are axes understandable?

  • Is color doing useful work?

  • Is the figure cluttered?

Reproducibility

  • Can it be regenerated from code?

  • Is it saved predictably?

  • Is the source data identifiable?

Revise the figure after the critique.

Reference answer

A useful critique asks whether the plot matches the variable types, uses readable scales and labels, avoids misleading encodings, shows enough context/raw data, and answers a clear statistical question.

Part 34 — Git checkpoint

git status

Stage appropriate files:

git add .

Commit:

git commit -m "Complete Week 6 exploratory visualization"

Push:

git push

Part 35 — Final reflection

1. EDA

What is the purpose of exploratory data analysis?

2. Variable types

Why should variable type influence plot choice?

3. Histograms

Why can changing the number of bins change interpretation?

4. Grammar of Graphics

What are the roles of data, aesthetics, and geometries?

5. Mapping vs. setting

What is the difference?

6. Raw data

Why can showing individual observations improve a summary plot?

7. Visualization and cleaning

How can plots reveal data-quality problems?

8. Communication

What separates an exploratory plot from a publication-quality figure?

9. Reproducibility

Why should final figures be created and saved from code rather than manually edited?

Completion checklist

  • Created Week 6 notebook

  • Generated a reproducible analysis dataset

  • Classified variables by type

  • Created histograms

  • Explored histogram binning

  • Created a density plot

  • Created a count plot

  • Created a boxplot

  • Added raw observations to a group comparison

  • Created a violin plot

  • Created a scatterplot

  • Compared scatterplots with correlation

  • Mapped group to a visual aesthetic

  • Used faceting

  • Used visualization to detect bad data

  • Created a plot using plotnine

  • Identified data/aesthetic/geometry components

  • Distinguished mapping from setting

  • Added layers

  • Created faceted Grammar of Graphics plots

  • Simplified an overcomplicated plot

  • Improved titles and labels

  • Investigated axis choices

  • Explored overplotting

  • Used transparency or hexbin aggregation

  • Visualized missingness

  • Created a publication-quality figure

  • Saved PNG and PDF output

  • Completed the mini EDA challenge

  • Critiqued and revised a visualization

  • Committed Week 6 work to Git

  • Pushed work to GitHub

What you should now understand

Statistical question
       ↓
Variable types
       ↓
Numerical summaries
       ↓
Visual encodings
       ↓
Explore
       ↓
Detect structure / problems
       ↓
Revise analysis
       ↓
Communicate clearly
       ↓
Save reproducibly

Next week we will move into Modular Programming & Functions, including pure functions, variable scope, and functional programming patterns.

Reference answer

Good EDA combines numerical summaries with plots, treats visualization as a diagnostic tool, checks data quality, and chooses encodings that match the statistical question.

9. Reference Activity Notebooks
Activity 2.3
9. Reference Activity Notebooks
Activity 4.1