Statistics

paired_bootstrap and one_sample_bootstrap.

from modeldiffr import paired_bootstrap
from modeldiffr.stats import one_sample_bootstrap

Pure Python, no dependencies.

paired_bootstrap

paired_bootstrap(
    base: list[float],
    derivative: list[float],
    n_resamples: int = 2000,
    confidence: float = 0.95,
    seed: int = 0,
) -> dict

base

list[float], required

Per item scores for the base model.

derivative

list[float], required

Per item scores for the derivative, aligned with base.

n_resamples

int, default 2000

Bootstrap resamples.

confidence

float, default 0.95

Interval level.

seed

int, default 0

Seed for random.Random.

Raises ValueError when the lists differ in length or are empty.

Returns

• n int: Number of pairs.

• base_mean float: Mean of base.

• derivative_mean float: Mean of derivative.

• delta float: Mean of derivative - base.

• ci_low float: Lower interval bound.

• ci_high float: Upper interval bound.

• confidence float: Interval level.

• significant bool: ci_low > 0 or ci_high < 0.

from modeldiffr import paired_bootstrap

paired_bootstrap([1, 0, 1, 0], [1, 1, 1, 1], seed=0)

one_sample_bootstrap

one_sample_bootstrap(
    values: list[float],
    n_resamples: int = 2000,
    confidence: float = 0.95,
    seed: int = 0,
    min_effect: float = 0.0,
) -> dict

Bootstraps values against zero. Returns the same fields as paired_bootstrap, with base_mean equal to 0.0, plus:

• min_effect float: The threshold that was applied.

• significant bool: ci_low > min_effect. Only an increase above the threshold counts.

Used for output_divergence_kl with min_effect=1e-3.