How to audit ML models for bias, with practical code examples and decision frameworks.
A model can have 99% accuracy and still be discriminatory. If it learned historical biases from training data (and it almost certainly did), those biases will be replicated — and potentially amplified — in production.
Fairness testing isn't just ethical; it's increasingly a legal requirement. The EU AI Act, the NYC Local Law 144 (for hiring algorithms), and the CFPB's stance on fair lending all require demonstrable bias auditing.
Definition: The probability of receiving a positive prediction should be the same regardless of group membership.
Formula (stdlib):
# For each group g: positive_rate(g) = count(ŷ=1 in g) / count(g)
# Ratio = min(rates) / max(rates)
# Pass when ratio ≥ 0.80 (the four-fifths rule)When to use: When you care about *equal representation* in outcomes. Appropriate for hiring, lending, and resource allocation where unequal impact requires justification.
When NOT to use: When base rates legitimately differ between groups. Medical diagnosis, for example — disease prevalence varies by demographics, so forcing equal positive rates would misdiagnose patients.
Code:
from src.fairness import demographic_parity_ratio
groups = ["male", "male", "female", "female", "female"]
y_pred = [1, 1, 1, 0, 0]
ratio = demographic_parity_ratio(groups, y_pred)
print(f"DP ratio: {ratio:.2f}")
# male rate: 2/2 = 1.0, female rate: 1/3 = 0.33
# ratio = 0.33 / 1.0 = 0.33 → FAILS four-fifths ruleDefinition: The true positive rate (TPR) and false positive rate (FPR) should be equal across groups.
Why both TPR and FPR?
Formula (stdlib):
# For each group g:
# TPR(g) = TP(g) / (TP(g) + FN(g))
# FPR(g) = FP(g) / (FP(g) + TN(g))
# Max TPR gap = max(TPR) - min(TPR)
# Max FPR gap = max(FPR) - min(FPR)
# Pass when both gaps ≤ 0.10Code:
from src.fairness import equalized_odds_gap
groups = ["A", "A", "A", "A", "B", "B", "B", "B"]
y_true = [1, 1, 0, 0, 1, 1, 0, 0]
y_pred = [1, 1, 0, 0, 1, 0, 0, 0] # Group B misses one positive
tpr_gap, fpr_gap = equalized_odds_gap(groups, y_true, y_pred)
print(f"TPR gap: {tpr_gap:.2f}, FPR gap: {fpr_gap:.2f}")
# Group A TPR: 1.0, Group B TPR: 0.5 → gap = 0.5 → FAILSDefinition: Identical to demographic parity ratio. Named differently because the legal concept ("disparate impact") has specific implications in US employment and lending law.
The four-fifths rule: if the selection rate for any protected group is less than 80% of the rate for the most-selected group, there is evidence of adverse impact.
from src.fairness import audit_fairness
report = audit_fairness(
groups=test_df["demographic_group"].tolist(),
y_true=test_df["label"].tolist(),
y_pred=predictions.tolist(),
dp_threshold=0.80,
eo_tpr_threshold=0.10,
eo_fpr_threshold=0.10,
)
print(report.summary())
# Output:
# Fairness audit: PASSED
# [PASS] demographic_parity_ratio: 0.8523 (threshold: 0.80)
# [PASS] equalized_odds_tpr_gap: 0.0412 (threshold: 0.10)
# [PASS] equalized_odds_fpr_gap: 0.0187 (threshold: 0.10)
# [PASS] disparate_impact_ratio: 0.8523 (threshold: 0.80)When the audit *fails*, you need to understand *which* group is affected and *why*:
from src.fairness import compute_subgroup_metrics
metrics = compute_subgroup_metrics(groups, y_true, y_pred)
for group, scores in metrics.items():
print(f"{group}: acc={scores['accuracy']:.3f}, "
f"prec={scores['precision']:.3f}, rec={scores['recall']:.3f}, "
f"f1={scores['f1']:.3f}, n={scores['count']:.0f}")This table often reveals the root cause:
Test on all attributes that could create legal or ethical risk:
| Domain | Typical protected attributes |
|---|---|
| Hiring | Gender, race/ethnicity, age, disability status |
| Lending | Race, gender, national origin, marital status |
| Healthcare | Race, gender, age, socioeconomic proxies |
| Insurance | Gender, age, zip code (as race proxy) |
| Criminal justice | Race, gender, age |
Proxy features: Be aware that even if you don't include protected attributes directly, the model can learn to approximate them from correlated features (zip code → race, name → gender). Test fairness on the *actual protected attribute*, not just the features in the model.
Groups can be intersectional. A model might be fair for "women" and "Black applicants" individually but biased against "Black women" specifically. Test intersections:
# Create intersectional groups
groups = [f"{row['gender']}_{row['race']}" for _, row in df.iterrows()]
report = audit_fairness(groups, y_true, y_pred)The trade-off: more intersections = smaller group sizes = noisier estimates. You need at least 50–100 samples per intersectional group for reliable results.
Decision framework:
1. Is the disparity due to training data imbalance?
→ Collect more data for underrepresented groups
→ Apply class-reweighting or oversampling
2. Is the disparity due to proxy features?
→ Identify and remove or decorrelate proxies
→ Use adversarial debiasing techniques
3. Is the disparity due to legitimate base-rate differences?
→ Document the justification
→ Switch from demographic parity to equalized odds
→ Consider calibration-based fairness
4. Is the disparity small but persistent?
→ Apply post-processing threshold adjustment per group
→ Trade off a small amount of overall accuracy for fairness
5. None of the above work?
→ The model should not be deployed for this use caseAdd fairness as a required gate in your deployment pipeline:
# In your CI test file
def test_fairness_audit():
"""Block deployment if fairness checks fail."""
report = audit_fairness(
groups=test_groups,
y_true=test_labels,
y_pred=model_predictions,
)
assert report.passed, (
f"Fairness audit failed:\n{report.summary()}\n\n"
f"Failures: {[f.name for f in report.failures]}\n"
f"See guides/fairness-testing.md for remediation steps."
)Set the gate as non-blocking initially while you calibrate thresholds, then make it blocking once you've established baselines. Failing a fairness gate should be treated with the same severity as a security vulnerability.
Testing machine-learning systems is fundamentally different from testing traditional software. A function that returns the "wrong" answer might be perfectly acceptable if its error rate is below a threshold, while a function that returns the "right" answer might be a disaster if it's biased against a protected group.
This guide lays out a practical, layered strategy.
Traditional software has the test pyramid (unit → integration → E2E). ML systems need a different shape:
┌──────────────┐
│ A/B Tests │ (Production)
┌┴──────────────┴┐
│ Regression │ (Pre-deploy gate)
┌┴────────────────┴┐
│ Fairness Audit │ (Pre-deploy gate)
┌┴──────────────────┴┐
│ Performance Gates │ (Post-training)
┌┴────────────────────┴┐
│ Behavioral Tests │ (Post-training)
┌┴──────────────────────┴┐
│ Data Validation │ (Pre-training)
┌┴────────────────────────┴┐
│ Unit Tests (transforms) │ (Development)
└──────────────────────────┘Each layer catches different failure modes. Skip a layer and you leave a class of bugs undetected.
What to test: Feature engineering functions, preprocessing pipelines, data loaders.
Why this layer matters: Bugs in feature transforms are the single most common source of silent model degradation. A feature that worked in training but silently returns NaN in production will tank accuracy without any error message.
| Check | Example |
|---|---|
| Output shape | assert output.shape == (n, expected_features) |
| No unexpected NaN | assert not output.isna().any().any() |
| Output dtype | assert output["age"].dtype == np.float64 |
| Idempotency | assert transform(transform(x)) == transform(x) |
| Known-output pairs | assert transform(known_input) == known_output |
predict() calls preprocess() → model.forward(), test preprocess() independently.Run these checks on every dataset *before* it reaches the model. Use the validators module.
Define the expected column names, types, and constraints in configs/validation_rules.yaml:
from src.validators import SchemaValidator, ColumnRule
rules = [
ColumnRule("age", dtype="integer", nullable=False, min_value=0, max_value=120),
ColumnRule("income", dtype="float", nullable=True, min_value=0),
ColumnRule("segment", dtype="categorical",
allowed_values={"enterprise", "mid_market", "smb"}),
]
result = SchemaValidator(rules).validate(df)
assert result.passed, f"Schema validation failed: {result.errors}"Compare your current training/serving data against a reference snapshot using PSI:
from src.validators import check_distribution_drift
result = check_distribution_drift(
reference=training_data["income"].values,
current=serving_batch["income"].values,
feature_name="income",
psi_threshold=0.25,
)
if not result.passed:
alert_on_call_team(result.errors)Decision tree for PSI results:
PSI < 0.10 → No action needed
0.10 ≤ PSI < 0.25 → Investigate: check data source, look for seasonal patterns
PSI ≥ 0.25 → Block deployment, retrain on fresh dataUse the behavioral module. These test *properties* of the model rather than specific outputs.
"Changing X should NOT change the prediction."
from src.behavioral import invariance_test
# Adding trailing whitespace shouldn't change text classification.
results = invariance_test(
model_fn=classifier.predict,
inputs=test_texts,
perturbation_fn=lambda text: text + " ",
test_name="trailing_whitespace_invariance",
)
assert all(r.passed for r in results)"Changing X should move the prediction in direction Y."
from src.behavioral import directional_test
# Increasing credit score should never decrease approval probability.
results = directional_test(
model_fn=model.predict_proba,
inputs=test_applications,
perturbation_fn=lambda app: {**app, "credit_score": app["credit_score"] + 50},
expected_direction="increase",
)"Transforming the input by T should transform the output by R."
Common metamorphic relations:
Use the performance module. Define thresholds in configs/thresholds.yaml.
Start with your current production model's metrics as the baseline. Then:
1. Set the gate threshold 2–5% below the current value (to allow natural variance).
2. Set the warn_delta to 2% so you get early warning of drift.
3. For safety-critical models (medical, financial), set tighter thresholds.
Never gate on a single metric. At minimum:
| Model type | Primary | Secondary | Guardrail |
|---|---|---|---|
| Binary classifier | F1 | ROC-AUC | Recall ≥ threshold |
| Regression | RMSE | R² | MAPE |
| Ranking | NDCG@k | MAP | MRR |
Use the fairness module. See guides/fairness-testing.md for a deep dive.
At minimum, every model that makes decisions about people must pass:
1. Demographic parity ratio ≥ 0.80 (four-fifths rule)
2. Equalized odds TPR gap ≤ 0.10
3. Equalized odds FPR gap ≤ 0.10
Use the regression module. Compare the candidate model against the current production model on the same test set.
from src.regression import RegressionDetector
from src.performance import accuracy_score
detector = RegressionDetector(
metric_fns={"accuracy": accuracy_score},
higher_is_better={"accuracy"},
alpha=0.05,
)
report = detector.compare(y_true, old_preds, new_preds)
assert not report.has_regression, report.summary()A model can improve on aggregate while regressing on a critical subgroup. Always test slices:
slices = {
"high_value_customers": df["ltv"] > 10000,
"new_users": df["tenure_days"] < 30,
"mobile_traffic": df["platform"] == "mobile",
}
report = detector.compare(y_true, old_preds, new_preds, slices=slices)# .github/workflows/ml-tests.yml (pseudo-config)
steps:
- name: Unit tests
run: pytest tests/test_transforms.py
- name: Data validation
run: pytest tests/test_validators.py
- name: Train model
run: python train.py
- name: Performance gates
run: pytest tests/test_performance.py
- name: Behavioral tests
run: pytest tests/test_behavioral.py
- name: Fairness audit
run: pytest tests/test_fairness.py
- name: Regression check
run: pytest tests/test_regression.pyEach step is a gate. If any step fails, the pipeline stops and the model is not deployed.
1. Testing on training data. Always use a held-out test set that the model has never seen during training.
2. Not versioning test data. Pin the test set version alongside the model version so results are reproducible.
3. Threshold rot. Revisit thresholds quarterly. As models improve, yesterday's "good enough" becomes tomorrow's regression.
4. Only testing happy paths. Include adversarial examples, edge cases, and out-of-distribution inputs in your behavioral tests.
5. Ignoring calibration. A model can have high accuracy but poorly calibrated probabilities. If you use probabilities downstream (for ranking, thresholding, or risk scoring), test calibration explicitly.