Activity 7.2

This page is an ipython notebook, meaning you can download it and run the executable cells. It directly follows Activity 7.2. Reference answers appear after the questions, challenges, predictions, and reflections. They show one reasonable solution or interpretation, but other correct approaches may exist.

Profiling, Bottlenecks, Parallelism & Numba¶

Tools: Python, NumPy, pandas, pytest, cProfile, optional line_profiler, optional Numba, Jupyter/VS Code

Learning objectives¶

By the end of this tutorial, you should be able to:

• benchmark Python code;

• distinguish noisy one-off timings from repeated benchmarks;

• identify algorithmic bottlenecks;

• use cProfile;

• optionally use line-level profiling;

• inspect memory consumption;

• compare loop and vectorized implementations;

• reason about vectorization vs. memory tradeoffs;

• benchmark chunked processing;

• understand parallel overhead;

• accelerate appropriate loops with Numba;

• separate JIT compilation time from steady-state timing;

• validate optimized code against tests.

Part 0 --- Set up¶

Create:

notebooks/week14_performance.ipynb

text src/week14_performance.py

python from time import perf_counter

import numpy as np import pandas as pd

python rng = np.random.default_rng( 556 )

bash pytest

python import os

os.cpu_count()

Part 1 --- One-off timing¶

Create:

python x = rng.normal( size=1_000_000 )

python start = perf_counter()

result = np.sum( x ** 2 )

elapsed = ( perf_counter() - start )

elapsed

Part 2 --- Use %timeit¶

Run:

python

x ** 2

)

Reflection¶

Why is repeated benchmarking usually more informative?

Reference answer¶

A single timing is noisy because of scheduling, caches, background work, and warm-up. %timeit repeats the operation and summarizes repeated measurements.

Part 3 --- Python loop vs. NumPy¶

Write:

python def sum_squares_python( x ): total = 0.0

for value in x:
    total += (
        value
        * value
    )

return total

python def sum_squares_numpy( x ): return np.sum( x ** 2 )

python np.isclose( sum_squares_python(x), sum_squares_numpy(x) )

Reference answer¶

Repeated benchmarking reduces the influence of one anomalous run. NumPy usually beats an equivalent Python loop because its inner work executes in optimized compiled code.

Part 4 --- Correctness before speed¶

Temporarily introduce a bug:

python total += value

python total += value * value

Part 5 --- Scaling experiment¶

Benchmark both methods using:

text n = 1,000 10,000 100,000 1,000,000

text n python_time numpy_time speedup

Reflection¶

Does the relative advantage stay constant as n changes?

Reference answer¶

Verify equivalent outputs before timing; otherwise the faster implementation may simply be computing something different.

Part 6 --- Algorithmic bottleneck¶

Create:

python values = list( range( 100_000 ) )

lookup_values = list( range( 50_000, 150_000 ) )

python def count_matches_list( values, lookup_values ): count = 0

for value in values:
    if value in lookup_values:
        count += 1

return count

Reference answer¶

Relative speedups can change with input size because fixed overhead and algorithmic scaling contribute differently as n grows.

Part 7 --- Change the data structure¶

Rewrite:

python def count_matches_set( values, lookup_values ): lookup = set( lookup_values )

count = 0

for value in values:
    if value in lookup:
        count += 1

return count

Part 8 --- Profile a workflow¶

Add to src/week14_performance.py:

python def slow_transform( x ): output = []

for value in x:
    output.append(
        np.sqrt(
            value ** 2
            + 1
        )
    )

return np.array(
    output
)

def summarize( x ): transformed = ( slow_transform( x ) )

return {
    "mean": transformed.mean(),
    "sd": transformed.std(),
    "max": transformed.max()
}

def main(): rng = np.random.default_rng( 556 )

x = rng.normal(
    size=500_000
)

print(
    summarize(x)
)

if name == “main”: main()

Reference answer¶

Profile the entire workflow to discover where time is actually spent, then optimize measured hotspots rather than guessed ones.

Part 9 --- Use cProfile¶

From the terminal:

bash python -m cProfile
src/week14_performance.py

text which function has highest cumulative time? which has highest direct time? how many calls were made?

Part 10 --- Sort profile output¶

Use:

bash python -m cProfile
-s cumulative
src/week14_performance.py

Part 11 --- Profile only the hotspot¶

Once slow_transform() is identified, focus your attention there.

Reflection¶

Why would optimizing summarize() first be a poor use of time if almost all runtime occurs inside slow_transform()?

Reference answer¶

Cumulative-time sorting identifies functions responsible for the most total work, including work in callees. Optimizing a tiny non-hotspot function gives little end-to-end benefit.

Part 12 --- Optional line profiling¶

If installed:

python

python

-f slow_transform \
slow_transform(x)

Part 13 --- Vectorize the hotspot¶

Rewrite:

python def fast_transform( x ): return np.sqrt( x ** 2 + 1 )

python slow = slow_transform( x[:1000] )

fast = fast_transform( x[:1000] )

np.allclose( slow, fast )

Part 14 --- Reprofile¶

Replace the slow implementation in the workflow.

Run cProfile again.

Questions¶

1. What is now the bottleneck?

2. How did total runtime change?

3. Why must profiling be repeated after optimization?

Reference answer¶

After optimization, profile again because the bottleneck can move. Compare new total runtime as well as the new hotspot.

Part 15 --- Inspect memory use¶

For:

python x = rng.normal( size=10_000_000 )

python x.nbytes

python x.nbytes / ( 1024 ** 2 )

Part 16 --- DataFrame memory¶

Create:

python df = pd.DataFrame({ “id”: np.arange( 1_000_000 ), “group”: rng.choice( [ “A”, “B”, “C” ], size=1_000_000 ), “score”: rng.normal( size=1_000_000 ) })

python df.memory_usage( deep=True )

python df.memory_usage( deep=True ).sum()

Reference answer¶

Ten million float64 values require about 80,000,000 bytes—roughly 80 MB (about 76.3 MiB)—before container/object overhead.

Part 17 --- Efficient dtypes¶

Convert:

python df[“group”] = ( df[“group”] .astype( “category” ) )

Part 18 --- Copying cost¶

Run:

python df_copy = df.copy()

Reference answer¶

Categorical data can store repeated labels as compact integer codes plus one copy of each category, using less memory than many repeated Python strings.

Part 19 --- Select only needed columns¶

Compare:

python large_subset = df.copy()

python small_subset = df[ [ “group”, “score” ] ].copy()

Reference answer¶

Selecting only necessary columns reduces memory, copying, serialization, and downstream computation.

Part 20 --- Chunking concept¶

Create a temporary CSV from a manageable DataFrame.

Then read:

python for chunk in pd.read_csv( path, chunksize=100_000 ): ...

text running_sum running_count

text running_sum / running_count

Reference answer¶

Chunking lowers peak memory by processing smaller pieces, at the cost of extra I/O/coordination and sometimes slower execution.

Part 21 --- Verify chunked output¶

Compare the chunked result with:

python pd.read_csv( path )[“score”].mean()

python np.isclose(...)

Reference answer¶

The chunked result should agree with the full-data result within numerical tolerance before interpreting performance.

Part 22 --- Parallel-task concept¶

Suppose:

python def simulation_scenario( seed ): rng = np.random.default_rng( seed )

return rng.normal(
    size=1_000_000
).mean()

Reference answer¶

Chunking trades lower peak memory for additional I/O/loop overhead and more complex aggregation logic.

Part 23 --- Serial baseline¶

Create several independent seeds.

Run each scenario serially.

Time total execution.

Store results.

This is your baseline.

Reference answer¶

Independent tasks with little shared state are good parallel candidates because workers need little synchronization.

Part 24 --- Process-based parallelism¶

Use Python’s standard library:

python from concurrent.futures import ( ProcessPoolExecutor )

python with ProcessPoolExecutor() as executor: results = list( executor.map( simulation_scenario, seeds ) )

Part 25 --- Vary task size¶

Repeat the serial/parallel comparison for:

text small per-task workload large per-task workload

Reference answer¶

Parallel execution can be slower for small tasks because startup, serialization, scheduling, and communication overhead exceed useful work.

Part 26 --- Randomness and parallel workers¶

Do not use:

python np.random.default_rng( 556 )

Reference answer¶

Parallel overhead becomes less important as each task contains more computation relative to setup/communication.

Part 27 --- Install/check Numba¶

Check:

python import numba

numba.version

Reference answer¶

Identical random streams across workers can duplicate simulated samples, understate Monte Carlo variability, and violate independence assumptions. Use independent streams.

Part 28 --- Numba baseline loop¶

Use:

python def sum_squares_loop( x ): total = 0.0

for value in x:
    total += (
        value
        * value
    )

return total

Part 29 --- Compile with @njit¶

If Numba is available:

python from numba import njit

python njit def sum_squares_numba( x ): total = 0.0

for value in x:
    total += (
        value
        * value
    )

return total

python sum_squares_numba( x )

Part 30 --- Compilation overhead¶

Time:

text first Numba call

text second Numba call

Part 31 --- Compare Python, NumPy, and Numba¶

Benchmark:

text pure Python loop NumPy vectorized Numba-compiled loop

text method runtime speedup vs Python

Reference answer¶

The first Numba call includes compilation, so compare steady-state performance only after compilation. Verify Python/NumPy/Numba outputs agree first.

Part 32 --- When NumPy is already excellent¶

Compare a matrix multiplication:

python A @ B

Reference answer¶

NumPy matrix multiplication already uses highly optimized compiled libraries, so wrapping it in Numba often adds little.

Part 33 --- Numba-friendly algorithm challenge¶

Create a numerical loop that is less convenient to vectorize.

Example: a simple recurrence.

python def recurrence( x ): output = np.empty_like( x )

output[0] = x[0]

for i in range(
    1,
    len(x)
):
    output[i] = (
        0.9
        * output[i - 1]
        + x[i]
    )

return output

Reference answer¶

Numba is most useful for numeric loops that are awkward to express as a single optimized NumPy operation; simple element-wise squaring is already excellent in NumPy.

Part 34 --- Optional Numba parallel loop¶

If supported, experiment with:

python from numba import ( njit, prange )

python njit( parallel=True ) def ...

Reference answer¶

Parallel Numba helps when iterations are independent and each does enough work to amortize parallel overhead.

Part 35 --- Performance regression check¶

After every optimization, rerun:

bash pytest

Reference answer¶

A performance regression test checks that code remains correct and does not become unexpectedly slower after future changes.

Part 36 --- Mini optimization project¶

Choose one previous course algorithm:

text bootstrap permutation test Monte Carlo integration gradient descent distance calculation

Reference answer¶

Tests are essential during optimization because implementation changes can easily alter statistical behavior even while improving speed.

Part 37 --- Amdahl’s-law thought experiment¶

Suppose:

text 90 seconds

text 100-second

text 90 sec → 9 sec

Reference answer¶

Amdahl’s law limits total speedup by the fraction of runtime left unoptimized. Making a small component infinitely fast cannot produce a large total improvement.

Part 38 --- Performance audit¶

For your mini project, answer:

text What is the representative input size? What was the baseline runtime? How was runtime measured? What did profiling identify? What optimization was applied? Did memory usage change? Did tests still pass? What is the new runtime? What is the speedup? What new complexity was introduced? Would you keep the optimized version?

Reference answer¶

A performance audit should record representative input, environment, baseline runtime/memory, hotspot, optimization, new runtime/memory, correctness checks, and complexity tradeoffs.

Part 39 --- Git checkpoint¶

Run:

bash pytest

bash git status git add . git commit -m “Complete Week 14 profiling and performance exercises” git push

Part 40 --- Final reflection¶

1. Profiling¶

Why should profiling occur before optimization?

2. Benchmarking¶

Why are repeated representative benchmarks preferable to one timing?

3. Complexity¶

Why can changing an algorithm or data structure outperform micro-optimization?

4. Vectorization¶

Why is vectorization often faster?

5. Memory¶

Why can a fast vectorized method still be inappropriate?

6. Parallelism¶

When can parallel computation be slower than serial computation?

7. RNG¶

Why do parallel simulations need independent random streams?

8. Numba¶

What does JIT compilation do?

9. Compilation overhead¶

Why separate first-call and steady-state Numba timing?

10. Software engineering¶

Why must performance optimization be followed by correctness testing?

Completion checklist¶

• Created Week 14 notebook

• Ran correctness tests before optimization

• Timed code using perf_counter

• Used %timeit

• Compared Python-loop and NumPy implementations

• Verified equivalent output before benchmarking

• Ran a scaling experiment

• Improved a membership algorithm using a better data structure

• Created a profileable Python script

• Used cProfile

• Interpreted cumulative vs. direct timing

• Reprofiled after optimization

• Used line profiling or reviewed the concept

• Inspected NumPy array memory

• Inspected pandas memory usage

• Reduced memory with an efficient dtype

• Investigated copying cost

• Selected only needed columns

• Processed data in chunks

• Validated chunked output

• Established a serial parallel-computing baseline

• Used or reviewed ProcessPoolExecutor

• Compared parallel workloads of different sizes

• Used distinct reproducible RNG seeds

• Checked Numba availability

• Compiled a numerical loop with @njit

• Separated compilation from execution timing

• Compared Python, NumPy, and Numba

• Identified a case where NumPy is already optimized

• Accelerated a recurrence-like loop with Numba

• Explored prange or reviewed parallel JIT concepts

• Reran tests after optimization

• Completed the mini optimization project

• Calculated overall speedup

• Audited the performance tradeoff

• Committed Week 14 work to Git

• Pushed work to GitHub

What you should now understand¶

text Correct code ↓ Benchmark ↓ Profile ↓ Find hotspot ↓ Improve algorithm / vectorize / reduce memory / parallelize / JIT ↓ Retest ↓ Rebenchmark ↓ Keep only worthwhile complexity

This completes the main computational workflow arc of STA 556: from
environments and data engineering through reproducibility, numerical
statistics, testing, and performance-aware statistical software.

Reference answer¶

Profile first, benchmark repeatedly on representative inputs, verify correctness, and optimize the real bottleneck. Algorithm/data-structure changes often beat micro-optimizations; vectorization is fast but can use substantial memory.