7.1 Activity

Debugging, Testing & Defensive Programming¶

Tools: Python, NumPy, pandas, pytest, Jupyter/VS Code, terminal

Learning objectives¶

By the end of this tutorial, you should be able to:

• distinguish syntax, runtime, and logic errors;

• read Python tracebacks;

• reduce bugs to minimal examples;

• validate assumptions using assertions and exceptions;

• write defensive statistical functions;

• create pytest unit tests;

• test expected exceptions;

• use approximate numerical comparisons;

• test deterministic and stochastic code;

• write fixtures and parametrized tests;

• add regression tests for discovered bugs;

• organize tests alongside reusable modules.

Part 0 --- Set up the Week 13 project¶

Create:

src/week13_utils.py
tests/test_week13_utils.py
notebooks/week13_testing.ipynb

Check:

pytest --version

pytest is included in the STA 556 Codespace. If pytest --version fails in the official course Codespace, report the environment issue rather than installing packages ad hoc.

Run:

pytest

You should initially see that no tests were collected.

Part 1 --- Three kinds of failure¶

Consider:

A¶

if x > 2
    print(x)

B¶

x = 1 / 0

C¶

def variance(x):
    return np.std(
        x
    )

when the function is supposed to return variance.

Classify each as:

syntax error
runtime error
logic error

Which is hardest to detect automatically?

Part 2 --- Read a traceback¶

Run:

def divide(
    x,
    y
):
    return x / y


def calculate():
    return divide(
        10,
        0
    )


calculate()

Read the traceback from the bottom upward.

Identify:

exception type
error message
line where failure occurred
function-call chain

Part 3 --- Create a minimal example¶

Suppose a large analysis fails during:

X @ beta

Reduce it to:

X = np.ones(
    (
        5,
        3
    )
)

beta = np.ones(
    4
)

X @ beta

Question¶

Why is this small example easier to debug?

Fix the shape problem.

Part 4 --- Debug with object inspection¶

Create:

x = np.array([
    1,
    2,
    3
])

X = x.reshape(
    -1,
    1
)

Inspect:

type(x)
x.shape
X.shape
x.dtype

Reflection¶

How might shape inspection have prevented a Week 9 broadcasting bug?

Part 5 --- Assertions¶

Create:

def standardize(
    x
):
    x = np.asarray(
        x,
        dtype=float
    )

    assert x.ndim == 1
    assert len(x) > 0

    return (
        x - x.mean()
    ) / x.std()

Test with:

standardize([
    1,
    2,
    3
])

Then try:

standardize(
    np.ones(
        (
            2,
            2
        )
    )
)

Part 6 --- Identify a missing invariant¶

Try:

standardize([
    5,
    5,
    5
])

What happens?

Add a check for zero standard deviation.

Question¶

What invariant was missing?

Part 7 --- Replace developer assertions with validation¶

Rewrite:

def standardize(
    x
):
    x = np.asarray(
        x,
        dtype=float
    )

    if x.ndim != 1:
        raise ValueError(
            "x must be one-dimensional."
        )

    if len(x) == 0:
        raise ValueError(
            "x must not be empty."
        )

    sd = x.std()

    if sd == 0:
        raise ValueError(
            "x must have positive variability."
        )

    return (
        x - x.mean()
    ) / sd

Why is this a better public function interface?

Part 8 --- Improve error messages¶

Modify each validation message to include the received value/shape where helpful.

For example:

raise ValueError(
    "x must be one-dimensional; "
    f"received shape {x.shape}."
)

Principle¶

Good errors make debugging faster.

Part 9 --- Exception handling¶

Create:

def read_numeric(
    text
):
    try:
        value = float(
            text
        )
    except ValueError:
        return None
    else:
        return value

Test:

read_numeric(
    "3.14"
)

and:

read_numeric(
    "hello"
)

Part 10 --- Catch specific exceptions¶

Compare:

except:

with:

except ValueError:

Question¶

Why is a bare except dangerous?

Part 11 --- Do not hide failures¶

Consider:

def safe_analysis(
    data
):
    try:
        return run_analysis(
            data
        )
    except Exception:
        return None

List three reasons this could be dangerous in a research pipeline.

Rewrite it to handle only an error you genuinely know how to recover from.

Part 12 --- Create a convergence exception¶

In src/week13_utils.py:

class ConvergenceError(
    RuntimeError
):
    pass

Then create:

def check_convergence(
    converged
):
    if not converged:
        raise ConvergenceError(
            "Numerical algorithm "
            "failed to converge."
        )

Test manually.

Part 13 --- First pytest test¶

Add to src/week13_utils.py:

def square(x):
    return x ** 2

In:

tests/test_week13_utils.py

write:

from src.week13_utils import (
    square
)


def test_square():
    assert square(3) == 9

Run:

pytest

Part 14 --- Make a test fail deliberately¶

Change:

assert square(3) == 10

Run:

pytest

Read pytest’s failure output.

Restore the correct expected value.

Question¶

How does the test output differ from a generic exception traceback?

Part 15 --- Arrange--Act--Assert¶

Rewrite:

def test_square():
    # Arrange
    x = 4

    # Act
    result = square(
        x
    )

    # Assert
    assert result == 16

Why might this structure help when tests become more complicated?

Part 16 --- Test standardize¶

Move standardize() into:

src/week13_utils.py

Write:

def test_standardize_mean():
    x = np.array([
        1.0,
        2.0,
        3.0
    ])

    z = standardize(
        x
    )

    assert np.isclose(
        z.mean(),
        0.0
    )

Add a second test for standard deviation.

Part 17 --- Test approximate equality with pytest¶

Import:

import pytest

Then:

def test_standardize_sd():
    x = np.array([
        1.0,
        2.0,
        3.0
    ])

    z = standardize(
        x
    )

    assert z.std() == pytest.approx(
        1.0
    )

Question¶

Why not use exact floating-point equality?

Part 18 --- Test invalid input¶

Write:

def test_standardize_constant():
    with pytest.raises(
        ValueError
    ):
        standardize([
            5,
            5,
            5
        ])

Run:

pytest

The error behavior is now part of the tested contract.

Part 19 --- Test the error message¶

Use:

with pytest.raises(
    ValueError,
    match="positive variability"
):
    standardize([
        5,
        5,
        5
    ])

Why might checking the message be useful?

When could it make a test too brittle?

Part 20 --- Parametrized tests¶

Write:

@pytest.mark.parametrize(
    "x, expected",
    [
        (0, 0),
        (1, 1),
        (2, 4),
        (-3, 9)
    ]
)
def test_square_many(
    x,
    expected
):
    assert (
        square(x)
        == expected
    )

Run the test suite.

Reflection¶

What repetition did parametrization remove?

Part 21 --- Create a fixture¶

Add:

@pytest.fixture
def sample_scores():
    return np.array([
        70.0,
        80.0,
        90.0
    ])

Use:

def test_score_mean(
    sample_scores
):
    assert (
        sample_scores.mean()
        == 80.0
    )

What role does the fixture play?

Part 22 --- Test a Week 11 statistic¶

Add:

def difference_in_means(
    x,
    y
):
    return (
        np.mean(x)
        - np.mean(y)
    )

Test with:

x = np.array([
    4.0,
    6.0
])

y = np.array([
    1.0,
    3.0
])

Expected difference:

3

Part 23 --- Add defensive validation¶

Modify:

def difference_in_means(
    x,
    y
):

so that empty inputs raise ValueError.

Write tests for both:

empty x
empty y

Part 24 --- Test random code deterministically¶

Write:

def simulate_normal(
    n,
    rng
):
    if n <= 0:
        raise ValueError(
            "n must be positive."
        )

    return rng.normal(
        size=n
    )

Test:

def test_simulate_normal_shape():
    rng = np.random.default_rng(
        556
    )

    result = simulate_normal(
        10,
        rng
    )

    assert result.shape == (
        10,
    )

Part 25 --- Test reproducibility¶

Write:

def test_simulate_normal_reproducible():
    rng1 = np.random.default_rng(
        556
    )

    rng2 = np.random.default_rng(
        556
    )

    x1 = simulate_normal(
        5,
        rng1
    )

    x2 = simulate_normal(
        5,
        rng2
    )

    assert np.array_equal(
        x1,
        x2
    )

Why is this stronger than checking that a random sample mean is exactly zero?

Part 26 --- Test probability bounds¶

Suppose a function returns a permutation p-value.

Write a property test:

assert (
    0 <= p_value <= 1
)

Challenge¶

List three other useful statistical invariants from Weeks 10--12.

Part 27 --- Test bisection from Week 12¶

Move your bisection() function into src/.

Write a test using:

def f(x):
    return x ** 2 - 2

Expected:

np.sqrt(2)

Use:

pytest.approx(
    np.sqrt(2),
    abs=1e-7
)

Part 28 --- Test invalid bisection brackets¶

Write:

def test_bisection_invalid_bracket():
    with pytest.raises(
        ValueError
    ):
        bisection(
            f,
            2,
            3
        )

This protects the fail-fast behavior you added in Week 12.

Part 29 --- Test optimization properties¶

For the simple loss:

def objective(x):
    return (
        x - 3
    ) ** 2

Test that gradient descent finishes near:

3

Also test:

final objective <= initial objective

What mathematical property is this checking?

Part 30 --- Deliberately introduce a bug¶

Change the gradient from:

2 * (x - 3)

to:

2 * (x + 3)

Run the test suite.

Which test fails?

Restore the correct gradient.

Reflection¶

How did the test protect against a logic error that Python itself would never flag?

Part 31 --- Regression test workflow¶

Introduce a realistic bug:

standardize(
    [5, 5, 5]
)

Suppose this once returned NaNs instead of failing.

Document the workflow:

bug discovered
      ↓
test written
      ↓
test fails
      ↓
validation added
      ↓
test passes

Keep the test.

Part 32 --- Test a DataFrame function¶

Create:

def require_columns(
    data,
    columns
):
    missing = [
        column
        for column in columns
        if column
        not in data.columns
    ]

    if missing:
        raise ValueError(
            "Missing required columns: "
            + ", ".join(
                missing
            )
        )

    return True

Write tests for:

• all columns present;

• one missing column;

• several missing columns.

Part 33 --- Minimal datasets¶

For testing, use tiny datasets.

Example:

df = pd.DataFrame({
    "id": [1, 2],
    "score": [80, 90]
})

Why might this be better than loading a 500 MB production dataset in a unit test?

Part 34 --- Test independence¶

Create two tests.

Make the first modify a global list.

Make the second depend on the modified list.

Run the suite in different orders if possible.

Reflection¶

Why is shared mutable state dangerous in tests?

Rewrite so each test creates its own data.

Part 35 --- Build a Week 13 test suite¶

Your final test file should include tests for:

square
standardize
difference_in_means
simulate_normal
bisection
gradient descent
required DataFrame columns

Include at least:

• one normal case;

• one edge case;

• one invalid-input case;

• one approximate numerical test;

• one stochastic/reproducibility test.

Part 36 --- Run subsets of tests¶

Run all:

pytest

Run one file:

pytest tests/test_week13_utils.py

Run one test by keyword:

pytest -k standardize

Use verbose output:

pytest -v

Part 37 --- Debug a failing test¶

Choose one passing test and temporarily break the corresponding function.

Then:

1. run only that test;

2. read the failure;

3. inspect inputs/intermediate values;

4. form a hypothesis;

5. fix one thing;

6. rerun;

7. run the full suite.

This is the core development cycle.

Part 38 --- Testing audit¶

For one statistical function, answer:

What is its contract?
What inputs are valid?
What inputs are invalid?
What normal case can be checked exactly?
What edge case is important?
What mathematical invariant should hold?
Does it involve floating-point tolerance?
Does it involve randomness?
What past bug should become a regression test?

Part 39 --- Project structure¶

Your repository should now look approximately like:

project/
├── src/
│   └── week13_utils.py
├── tests/
│   └── test_week13_utils.py
├── notebooks/
│   └── week13_testing.ipynb
└── ...

Explain why this is cleaner than keeping functions and tests inside one notebook.

Part 40 --- Git checkpoint¶

Run:

pytest

Only commit after the suite passes.

Then:

git status
git add .
git commit -m "Complete Week 13 debugging and testing exercises"
git push

Part 41 --- Final reflection¶

Answer in Markdown.

1. Bugs¶

What is the difference between syntax, runtime, and logic errors?

2. Debugging¶

Why is debugging similar to hypothesis testing?

3. Tracebacks¶

What information does a traceback provide?

4. Assertions¶

What is an invariant?

5. Exceptions¶

When should a function raise an exception rather than continue?

6. Defensive programming¶

What does “fail fast” mean?

7. Unit testing¶

What should a unit test verify?

8. Floating point¶

Why use approximate equality?

9. Randomness¶

How can random code be tested reproducibly?

10. Regression tests¶

Why should a fixed bug usually lead to a new permanent test?

Completion checklist¶

• Created Week 13 notebook

• Created src/week13_utils.py

• Created tests/test_week13_utils.py

• Distinguished syntax/runtime/logic errors

• Read a traceback

• Built a minimal failing example

• Inspected shape/type/value diagnostics

• Added assertions

• Replaced public assertions with explicit validation

• Improved error messages

• Used try/except

• Caught specific exceptions

• Avoided silent exception swallowing

• Created a custom convergence exception

• Wrote a first pytest test

• Deliberately observed a failing test

• Used Arrange--Act--Assert

• Tested standardize

• Used pytest.approx

• Tested expected exceptions

• Checked an exception message

• Wrote parametrized tests

• Used a fixture

• Tested difference in means

• Tested random simulation deterministically

• Tested reproducibility

• Tested statistical invariants

• Tested bisection

• Tested invalid brackets

• Tested gradient-descent properties

• Introduced and caught a logic bug

• Created a regression test

• Tested DataFrame requirements

• Removed shared mutable state between tests

• Built a multi-function test suite

• Ran all and selected tests

• Debugged a failing test systematically

• Audited a function’s test design

• Ran the full suite before committing

• Committed Week 13 work to Git

• Pushed work to GitHub

What you should now understand¶

Statistical function
      ↓
Contract
      ↓
Validation
      ↓
Normal + edge + failure cases
      ↓
Automated tests
      ↓
Bug discovered
      ↓
Regression test
      ↓
More reliable statistical software

Next week we move into profiling, bottleneck identification, vectorization, parallel computation, and Numba.