🔑 Performance Separability

The rings.separability module tests whether performance distributions from different perturbations are statistically distinguishable. It pairs a comparator (the statistical test) with a functor (the orchestration: pairwise comparison, permutation testing, Bonferroni correction).

Comparators

class rings.separability.comparator.KSComparator

Comparator that uses the Kolmogorov-Smirnov test to compare two distributions.

This class implements a statistical test based on the Kolmogorov-Smirnov (KS) statistic to determine if two samples come from the same distribution. It uses permutation testing to empirically determine significance, making it suitable for small sample sizes where asymptotic approximations may not be reliable.

__call__(x, y, n_permutations=10_000, alpha=0.01, n_hypotheses=1, random_state=42, \*\*kwargs)

Compare two samples using Kolmogorov-Smirnov test with permutation testing.

Examples

>>> import numpy as np
>>> from rings.separability.comparator import KSComparator
>>>
>>> # Create two samples from different distributions
>>> x = np.random.normal(0, 1, 20)  # Standard normal
>>> y = np.random.normal(1, 1, 20)  # Shifted normal
>>>
>>> # Compare distributions with KS test
>>> comparator = KSComparator()
>>> result = comparator(x, y, n_permutations=1000, alpha=0.05)
>>>
>>> # Interpret results
>>> print(f"KS statistic: {result['score']}")
>>> print(f"p-value: {result['pvalue']}")
>>> print(f"Significant: {result['significant']}")

Notes

The Kolmogorov-Smirnov test measures the maximum difference between the empirical cumulative distribution functions of two samples. It is sensitive to differences in both location and shape of distributions.

class rings.separability.comparator.WilcoxonComparator

Comparator that uses the Wilcoxon signed-rank test to compare two distributions.

This class implements a statistical test based on the Wilcoxon signed-rank test for paired samples. It uses permutation testing to empirically determine significance, making it suitable for small sample sizes or when assumptions for parametric tests are not met.

__call__(x, y, n_permutations=10_000, alpha=0.01, n_hypotheses=1, random_state=42, \*\*kwargs)

Compare two samples using Wilcoxon signed-rank test with permutation testing.

Examples

>>> import numpy as np
>>> from rings.separability.comparator import WilcoxonComparator
>>>
>>> # Create paired samples (must be equal length)
>>> np.random.seed(42)
>>> x = np.random.normal(0, 1, 20)
>>> y = x + np.random.normal(0.5, 0.5, 20)  # Same as x with shift + noise
>>>
>>> # Compare distributions
>>> comparator = WilcoxonComparator()
>>> result = comparator(x, y, n_permutations=1000, alpha=0.05)
>>>
>>> # Interpret results
>>> print(f"Wilcoxon statistic: {result['score']}")
>>> print(f"p-value: {result['pvalue']}")
>>> print(f"Significant: {result['significant']}")

Notes

The Wilcoxon signed-rank test compares paired samples by ranking the absolute differences between pairs and summing the ranks of positive differences. Unlike the t-test, it does not require the differences to be normally distributed.

This implementation requires paired samples of equal length.

Functor

class rings.separability.functor.SeparabilityFunctor(comparator: Callable, n_jobs: int = 1, alpha: float = 0.01, **kwargs)

A functor for computing separability between multiple distributions.

This class computes all pairwise statistical comparisons between distributions, applies Bonferroni correction for multiple testing, and returns structured results. It’s designed to test whether different graph perturbations lead to statistically significant differences in model performance or metrics.

Parameters:

• comparator (callable) – A comparator class (e.g., KSComparator, WilcoxonComparator) that implements a __call__ method for comparing two distributions.

• n_jobs (int, default=1) – Number of jobs to run in parallel. If 1, no parallelism is used.

• alpha (float, default=0.01) – Family-wise significance level for hypothesis testing. After Bonferroni correction, a p-value less than alpha/num_tests is considered significant.

• **kwargs (dict) – Additional arguments passed to the comparator.

Examples

>>> import numpy as np
>>> from rings.separability.functor import SeparabilityFunctor
>>> from rings.separability.comparator import KSComparator
>>>
>>> # Create distributions to compare
>>> distributions = {
...     "Original": np.random.normal(0.8, 0.1, 30),  # High performance
...     "CompleteFeatures": np.random.normal(0.5, 0.1, 30),  # Low performance
...     "EmptyGraph": np.random.normal(0.5, 0.1, 30)  # Low performance
... }
>>>
>>> # Create functor and run analysis
>>> functor = SeparabilityFunctor(comparator=KSComparator(), alpha=0.05)
>>> results = functor.forward(distributions)
>>>
>>> # Results contain pairwise comparisons with significance tests
>>> print(results[['mode1', 'mode2', 'score', 'pvalue_adjusted', 'significant']])

__init__(comparator: Callable, n_jobs: int = 1, alpha: float = 0.01, **kwargs)

Initialize the SeparabilityFunctor.

Parameters:

• comparator (Callable) – An instance of a comparator class (e.g., KSComparator, WilcoxonComparator) that implements a compare() method.

• n_jobs (int, default=1) – Number of jobs to run in parallel. If 1, no parallelism is used.

• alpha (float, default=0.01) – Family-wise significance level for hypothesis testing.

• **kwargs (dict) – Additional arguments passed to the comparator.

forward(distributions: Dict[str, List[float] | ndarray], n_permutations: int = 10000, random_state: int | None = 42, as_dataframe: bool = True) → List[Dict[str, Any]] | DataFrame

Compute all pairwise separability tests between distributions.

This method performs statistical tests to determine whether distributions are significantly different from each other, applying Bonferroni correction for multiple comparisons.

Parameters:

• distributions (Dict[str, Union[List[float], np.ndarray]]) – Dictionary mapping mode names to performance distributions. Example: {“Original”: np.array([0.8, 0.7, …]), “EmptyGraph”: np.array([0.5, 0.6, …])}

• n_permutations (int, default=10_000) – Number of permutations for the statistical tests.

• random_state (Optional[int], default=42) – Random seed for reproducibility.

• as_dataframe (bool, default=True) – If True, return results as a pandas DataFrame, otherwise as a list of dictionaries.

Returns:

Results of all pairwise comparisons, either as a list of dictionaries or a pandas DataFrame. Each result contains the mode names, p-value, statistic, and significance indicator.

Return type:

Union[List[Dict[str, Any]], pd.DataFrame]

Examples

>>> functor = SeparabilityFunctor(comparator=KSComparator(), n_jobs=4, alpha=0.05)
>>> distributions = {
...     "Original": np.array([0.8, 0.7, 0.75, 0.82, 0.79]),
...     "EmptyGraph": np.array([0.5, 0.48, 0.52, 0.49, 0.51]),
...     "NoFeatures": np.array([0.45, 0.47, 0.5, 0.52, 0.48])
... }
>>> results = functor.forward(distributions)
>>> print(results)