7.2 Activity

Profiling, Bottlenecks, Parallelism & Numba¶

Tools: Python, NumPy, pandas, pytest, cProfile, optional line_profiler, optional Numba, Jupyter/VS Code

Learning objectives¶

By the end of this tutorial, you should be able to:

• benchmark Python code;

• distinguish noisy one-off timings from repeated benchmarks;

• identify algorithmic bottlenecks;

• use cProfile;

• optionally use line-level profiling;

• inspect memory consumption;

• compare loop and vectorized implementations;

• reason about vectorization vs. memory tradeoffs;

• benchmark chunked processing;

• understand parallel overhead;

• accelerate appropriate loops with Numba;

• separate JIT compilation time from steady-state timing;

• validate optimized code against tests.

Part 0 --- Set up¶

Create:

notebooks/week14_performance.ipynb

and:

src/week14_performance.py

Import:

from time import perf_counter

import numpy as np
import pandas as pd

Create:

rng = np.random.default_rng(
    556
)

Before optimizing anything, run your Week 13 test suite:

pytest

Codespaces performance note¶

All performance comparisons in this activity should be made within your own Codespace. Do not interpret differences between students’ absolute timings as differences in code quality. Cloud machine size and system load may differ.

For multiprocessing exercises, first inspect the CPU resources visible to your environment:

import os

os.cpu_count()

Parallel code is not expected to be faster for every workload or every Codespace. Measuring the overhead is part of the lesson.

Part 1 --- One-off timing¶

Create:

x = rng.normal(
    size=1_000_000
)

Time:

start = perf_counter()

result = np.sum(
    x ** 2
)

elapsed = (
    perf_counter()
    - start
)

elapsed

Repeat manually several times.

Question¶

Are all timings identical?

Why not?

Part 2 --- Use %timeit¶

Run:

%timeit np.sum(
    x ** 2
)

Compare with your one-off timings.

Reflection¶

Why is repeated benchmarking usually more informative?

Part 3 --- Python loop vs. NumPy¶

Write:

def sum_squares_python(
    x
):
    total = 0.0

    for value in x:
        total += (
            value
            * value
        )

    return total

Compare with:

def sum_squares_numpy(
    x
):
    return np.sum(
        x ** 2
    )

Verify:

np.isclose(
    sum_squares_python(x),
    sum_squares_numpy(x)
)

Then benchmark both.

Part 4 --- Correctness before speed¶

Temporarily introduce a bug:

total += value

instead of:

total += value * value

The broken version may still be fast.

Question¶

Why must equivalent output be verified before benchmarking implementations?

Restore the correct code.

Part 5 --- Scaling experiment¶

Benchmark both methods using:

n = 1,000
10,000
100,000
1,000,000

Store:

n
python_time
numpy_time
speedup

in a DataFrame.

Reflection¶

Does the relative advantage stay constant as n changes?

Part 6 --- Algorithmic bottleneck¶

Create:

values = list(
    range(
        100_000
    )
)

lookup_values = list(
    range(
        50_000,
        150_000
    )
)

Write:

def count_matches_list(
    values,
    lookup_values
):
    count = 0

    for value in values:
        if value in lookup_values:
            count += 1

    return count

Benchmark.

Part 7 --- Change the data structure¶

Rewrite:

def count_matches_set(
    values,
    lookup_values
):
    lookup = set(
        lookup_values
    )

    count = 0

    for value in values:
        if value in lookup:
            count += 1

    return count

Verify identical output.

Benchmark.

Question¶

Was the improvement caused by syntax or algorithm/data structure?

Part 8 --- Profile a workflow¶

Add to src/week14_performance.py:

def slow_transform(
    x
):
    output = []

    for value in x:
        output.append(
            np.sqrt(
                value ** 2
                + 1
            )
        )

    return np.array(
        output
    )


def summarize(
    x
):
    transformed = (
        slow_transform(
            x
        )
    )

    return {
        "mean": transformed.mean(),
        "sd": transformed.std(),
        "max": transformed.max()
    }


def main():
    rng = np.random.default_rng(
        556
    )

    x = rng.normal(
        size=500_000
    )

    print(
        summarize(x)
    )


if __name__ == "__main__":
    main()

Part 9 --- Use cProfile¶

From the terminal:

python -m cProfile \
    src/week14_performance.py

Inspect the output.

Identify:

which function has highest cumulative time?
which has highest direct time?
how many calls were made?

Part 10 --- Sort profile output¶

Use:

python -m cProfile \
    -s cumulative \
    src/week14_performance.py

Question¶

Why is sorting by cumulative time useful when investigating a call chain?

Part 11 --- Profile only the hotspot¶

Once slow_transform() is identified, focus your attention there.

Reflection¶

Why would optimizing summarize() first be a poor use of time if almost all runtime occurs inside slow_transform()?

Part 12 --- Optional line profiling¶

If installed:

%load_ext line_profiler

Import the function, then run:

%lprun \
    -f slow_transform \
    slow_transform(x)

Identify the line consuming most runtime.

If line_profiler is unavailable, use this as a conceptual exercise rather than installing blindly.

Part 13 --- Vectorize the hotspot¶

Rewrite:

def fast_transform(
    x
):
    return np.sqrt(
        x ** 2
        + 1
    )

Check:

slow = slow_transform(
    x[:1000]
)

fast = fast_transform(
    x[:1000]
)

np.allclose(
    slow,
    fast
)

Benchmark both on a large array.

Part 14 --- Reprofile¶

Replace the slow implementation in the workflow.

Run cProfile again.

Questions¶

1. What is now the bottleneck?

2. How did total runtime change?

3. Why must profiling be repeated after optimization?

Part 15 --- Inspect memory use¶

For:

x = rng.normal(
    size=10_000_000
)

Inspect:

x.nbytes

Convert to MB:

x.nbytes / (
    1024 ** 2
)

Question¶

How much memory do 10 million float64 values require?

Part 16 --- DataFrame memory¶

Create:

df = pd.DataFrame({
    "id": np.arange(
        1_000_000
    ),
    "group": rng.choice(
        [
            "A",
            "B",
            "C"
        ],
        size=1_000_000
    ),
    "score": rng.normal(
        size=1_000_000
    )
})

Inspect:

df.memory_usage(
    deep=True
)

and:

df.memory_usage(
    deep=True
).sum()

Part 17 --- Efficient dtypes¶

Convert:

df["group"] = (
    df["group"]
    .astype(
        "category"
    )
)

Recheck memory.

Reflection¶

Why can categorical data use less memory than repeated Python strings?

Part 18 --- Copying cost¶

Run:

df_copy = df.copy()

Estimate the additional memory footprint.

Question¶

When is copying worth the memory cost?

Connect this to Week 2’s mutation/reference material.

Part 19 --- Select only needed columns¶

Compare:

large_subset = df.copy()

with:

small_subset = df[
    [
        "group",
        "score"
    ]
].copy()

Compare memory usage.

Principle¶

Move and retain only the data needed for the task.

Part 20 --- Chunking concept¶

Create a temporary CSV from a manageable DataFrame.

Then read:

for chunk in pd.read_csv(
    path,
    chunksize=100_000
):
    ...

Calculate the sum/count required for an overall mean without loading the entire file at once.

Challenge¶

Maintain:

running_sum
running_count

then calculate:

running_sum / running_count

Part 21 --- Verify chunked output¶

Compare the chunked result with:

pd.read_csv(
    path
)["score"].mean()

Use:

np.isclose(...)

Question¶

What resource tradeoff does chunking make?

Part 22 --- Parallel-task concept¶

Suppose:

def simulation_scenario(
    seed
):
    rng = np.random.default_rng(
        seed
    )

    return rng.normal(
        size=1_000_000
    ).mean()

Each seed defines an independent task.

Question¶

Why is this a good candidate for parallel execution?

Part 23 --- Serial baseline¶

Create several independent seeds.

Run each scenario serially.

Time total execution.

Store results.

This is your baseline.

Part 24 --- Process-based parallelism¶

Use Python’s standard library:

from concurrent.futures import (
    ProcessPoolExecutor
)

Conceptually:

with ProcessPoolExecutor() as executor:
    results = list(
        executor.map(
            simulation_scenario,
            seeds
        )
    )

Benchmark.

Reflection¶

Did parallelism help?

If not, what overhead might explain the result?

Part 25 --- Vary task size¶

Repeat the serial/parallel comparison for:

small per-task workload
large per-task workload

Question¶

At what point does parallel overhead become less important?

Part 26 --- Randomness and parallel workers¶

Do not use:

np.random.default_rng(
    556
)

inside every worker with the same seed.

Instead, pass distinct reproducible seeds.

Question¶

What statistical error could identical random streams create?

Part 27 --- Install/check Numba¶

Check:

import numba

numba.__version__

If Numba is not installed, the remaining Numba sections may be treated as optional.

The goal is to understand when JIT compilation helps.

Part 28 --- Numba baseline loop¶

Use:

def sum_squares_loop(
    x
):
    total = 0.0

    for value in x:
        total += (
            value
            * value
        )

    return total

Benchmark.

Part 29 --- Compile with @njit¶

If Numba is available:

from numba import njit

Create:

@njit
def sum_squares_numba(
    x
):
    total = 0.0

    for value in x:
        total += (
            value
            * value
        )

    return total

Call once:

sum_squares_numba(
    x
)

Then benchmark subsequent calls.

Part 30 --- Compilation overhead¶

Time:

first Numba call

and:

second Numba call

separately.

Reflection¶

Why should the first call not be directly compared with steady-state NumPy execution?

Part 31 --- Compare Python, NumPy, and Numba¶

Benchmark:

pure Python loop
NumPy vectorized
Numba-compiled loop

Verify all results are approximately equal.

Create a table:

method
runtime
speedup vs Python

Part 32 --- When NumPy is already excellent¶

Compare a matrix multiplication:

A @ B

with any hand-written loop version.

Question¶

Why might Numba wrapping add little to an operation already implemented through optimized numerical libraries?

Part 33 --- Numba-friendly algorithm challenge¶

Create a numerical loop that is less convenient to vectorize.

Example: a simple recurrence.

def recurrence(
    x
):
    output = np.empty_like(
        x
    )

    output[0] = x[0]

    for i in range(
        1,
        len(x)
    ):
        output[i] = (
            0.9
            * output[i - 1]
            + x[i]
        )

    return output

Compile with Numba and compare.

Question¶

Why is this a better Numba candidate than simple element-wise squaring?

Part 34 --- Optional Numba parallel loop¶

If supported, experiment with:

from numba import (
    njit,
    prange
)

Use:

@njit(
    parallel=True
)
def ...

with an independent loop.

Compare serial-JIT and parallel-JIT performance.

Important¶

Verify correctness before interpreting speed.

Part 35 --- Performance regression check¶

After every optimization, rerun:

pytest

Question¶

Why is Week 13’s test suite essential when replacing a simple implementation with a faster one?

Part 36 --- Mini optimization project¶

Choose one previous course algorithm:

bootstrap
permutation test
Monte Carlo integration
gradient descent
distance calculation

Complete:

1. Run tests / create correctness checks.

2. Benchmark a representative workload.

3. Profile it.

4. Identify the hotspot.

5. Make exactly one major performance improvement.

6. Re-run correctness tests.

7. Rebenchmark.

8. Calculate speedup.

9. Discuss memory implications.

10. Explain whether the added complexity is worth it.

Part 37 --- Amdahl’s-law thought experiment¶

Suppose:

90 seconds

of a:

100-second

program occurs inside one function.

If you make that function 10 times faster:

90 sec → 9 sec

What is the new total runtime?

What is the total speedup?

Reflection¶

Why is hotspot identification so powerful?

Part 38 --- Performance audit¶

For your mini project, answer:

What is the representative input size?
What was the baseline runtime?
How was runtime measured?
What did profiling identify?
What optimization was applied?
Did memory usage change?
Did tests still pass?
What is the new runtime?
What is the speedup?
What new complexity was introduced?
Would you keep the optimized version?

Part 39 --- Git checkpoint¶

Run:

pytest

Then:

git status
git add .
git commit -m "Complete Week 14 profiling and performance exercises"
git push

Part 40 --- Final reflection¶

1. Profiling¶

Why should profiling occur before optimization?

2. Benchmarking¶

Why are repeated representative benchmarks preferable to one timing?

3. Complexity¶

Why can changing an algorithm or data structure outperform micro-optimization?

4. Vectorization¶

Why is vectorization often faster?

5. Memory¶

Why can a fast vectorized method still be inappropriate?

6. Parallelism¶

When can parallel computation be slower than serial computation?

7. RNG¶

Why do parallel simulations need independent random streams?

8. Numba¶

What does JIT compilation do?

9. Compilation overhead¶

Why separate first-call and steady-state Numba timing?

10. Software engineering¶

Why must performance optimization be followed by correctness testing?

Completion checklist¶

• Created Week 14 notebook

• Ran correctness tests before optimization

• Timed code using perf_counter

• Used %timeit

• Compared Python-loop and NumPy implementations

• Verified equivalent output before benchmarking

• Ran a scaling experiment

• Improved a membership algorithm using a better data structure

• Created a profileable Python script

• Used cProfile

• Interpreted cumulative vs. direct timing

• Reprofiled after optimization

• Used line profiling or reviewed the concept

• Inspected NumPy array memory

• Inspected pandas memory usage

• Reduced memory with an efficient dtype

• Investigated copying cost

• Selected only needed columns

• Processed data in chunks

• Validated chunked output

• Established a serial parallel-computing baseline

• Used or reviewed ProcessPoolExecutor

• Compared parallel workloads of different sizes

• Used distinct reproducible RNG seeds

• Checked Numba availability

• Compiled a numerical loop with @njit

• Separated compilation from execution timing

• Compared Python, NumPy, and Numba

• Identified a case where NumPy is already optimized

• Accelerated a recurrence-like loop with Numba

• Explored prange or reviewed parallel JIT concepts

• Reran tests after optimization

• Completed the mini optimization project

• Calculated overall speedup

• Audited the performance tradeoff

• Committed Week 14 work to Git

• Pushed work to GitHub

What you should now understand¶

Correct code
    ↓
Benchmark
    ↓
Profile
    ↓
Find hotspot
    ↓
Improve algorithm / vectorize / reduce memory / parallelize / JIT
    ↓
Retest
    ↓
Rebenchmark
    ↓
Keep only worthwhile complexity

This completes the main computational workflow arc of STA 556: from environments and data engineering through reproducibility, numerical statistics, testing, and performance-aware statistical software.