c4fairness

Overview

c4fairness is a Python package and command-line tool for auditing machine learning models. It clusters the rows of a model's test set and reports how prediction-error disparities and sensitive-attribute composition vary across the discovered clusters — surfacing under-served subgroups without pre-specifying the protected group.

Aggregate accuracy hides localized harm: a model can look fair on average yet fail a specific intersection of groups. Unsupervised clustering of the test set exposes those pockets and quantifies each disparity with a significance test. Binary, multi-class, and regression tasks are all supported.

Install

pip install c4fairness              # from PyPI
pip install "c4fairness[web]"       # + the Gradio web UI
pip install "c4fairness[r]"         # + exact r×c Fisher via rpy2 (optional)

Python 3.10+. The import name is c4fairness; the CLI command is c4fairness (equivalently python -m c4fairness.main). Current release: v0.1.2.

Quick start

c4fairness --data_path compas_audit.csv \
    --regular_cols age,priors_count \
    --sensitive_cols sex,race,age --continuous_sensitive_cols age \
    --error_col errors --error_type binary \
    --algorithm kmeans --n_clusters 4 --seed 42

Each row of the input CSV is one test-set example: its features, the ground truth, and the model's prediction. The run clusters on age/priors_count plus the sensitive columns, then writes a per-cluster recap and a heatmap showing each cluster's error rate alongside its sex/race/age make-up. String columns are one-hot encoded automatically.

On the bundled COMPAS extract, one cluster holds 9.3% of the test set and carries a false-positive rate of 0.85 against 0.34 overall — a gap of +0.54 at p ≈ 0. Its sensitive columns say who is in it: 76% African-American, 91% male, median age 37. A single per-race average would flatten that.

What you get

Every run writes a detailed table — one row per cluster — and, in experiment mode, an overview table with one row per condition. Both are rendered as heatmaps: blue for cluster size, red for error, violet for sensitive composition, with p-value cells shaded darker the more significant they are.

Alongside the tables and heatmaps, each run saves projection scatter plots (PCA, t-SNE, MDS), per-attribute composition bars, and the CSVs behind every figure.

Experiment mode

A finding should not depend on what you happened to cluster on. --experiment reruns the audit for every combination of the feature groups — regular features (REG), sensitive attributes (SEN), and the error column itself (ERR) — and collects the results in a single overview heatmap, with per-condition p-values Benjamini-Hochberg corrected across sensitive features.

c4fairness --data_path compas_audit.csv --experiment \
    --regular_cols age,priors_count --sensitive_cols sex,race \
    --error_col errors --error_type binary \
    --algorithm kmeans --n_min 2 --n_max 6 --scoring chi2_error

Web demo

Hosted demo — coming soon. An interactive version of the app will be published on Hugging Face Spaces: upload a CSV, assign column roles, and read the heatmaps in the browser, with no installation at all.

In the meantime the same app runs locally:

pip install "c4fairness[web]"
c4fairness-web        # opens http://localhost:7860

The web UI wraps the full CLI. Upload a CSV or press Load example dataset to try it on COMPAS; assign column roles, describe how the model's mistakes should be read, pick an algorithm, and run. Results — heatmaps, an overview table, downloadable CSVs — appear in tabs, and the run log streams while the job is in flight. Single run audits exactly the configuration on the form; full sweep repeats it for every feature-group combination.

What's under the hood

• Algorithms — k-means, bisecting k-means, k-medoids, k-prototypes, DBSCAN, HDBSCAN.
• Distances — Euclidean, Manhattan, and Gower for mixed numeric/categorical data.
• Choosing k — automatic selection over a range, scored by silhouette, error separability, sensitive separability, or a weighted composite of the three.
• Error definitions — a raw 0/1 column, a confusion-matrix rate (FPR, FNR, precision), a signed regression residual, or a multi-class error typed per class, per confusion cell, or one-hot.
• Significance — Fisher, chi-square, Mann-Whitney, ANOVA, and exact r×c Fisher–Freeman–Halton where R is installed, with a scipy fallback when it is not.
• Preprocessing — string columns are one-hot encoded automatically and kept out of StandardScaler, so rare categories do not get inflated in the distance computation.

Datasets

The package has been evaluated on five public benchmarks spanning classification and regression. Each row of an input CSV is one test-set example, carrying the features, the ground truth, and a model's prediction.

The two extracts marked bundled ship with the repository and can be audited as-is. The rest are not redistributed: fetch them from the source and add your model's predictions as a column. Every original carries its own licence and citation requirements — check the source before redistributing.

Links

• Package — pypi.org/project/c4fairness
• Source and documentation — github.com/emma-ba/Clustering_4_Fairness
• Worked notebooks — docs/example_binary.ipynb, docs/example_regression.ipynb
• Hosted demo — coming soon