ML bias is not a vague ethical concern -- it is a measurable, technical problem with measurable solutions. Models can and do perform worse for some demographic groups than others, and this disparity can cause real harm when those models are used in hiring, lending, healthcare, or criminal justice. This guide walks through the practical process of finding and addressing bias in ML systems.
Types of Bias in ML Systems
Understanding where bias enters the pipeline is the first step to addressing it.
Representation bias occurs when your training data does not represent all groups equally. If your facial recognition training set is 80% lighter-skinned individuals, the model will likely perform worse on darker-skinned individuals -- not because of any explicit design choice, but because there was less data to learn from. The model has seen fewer examples of some groups and has had less opportunity to develop accurate representations.
Measurement bias occurs when labels or features are collected unequally across groups. A classic example: using arrest records as a proxy for criminal behavior. Arrest rates vary by neighborhood and policing intensity, not just by underlying behavior. A model trained on arrest records learns not just behavior patterns but also patterns of differential policing.
Aggregation bias occurs when a single model is built for a population with distinct subgroups that have meaningfully different patterns. If the relationship between features and outcomes differs across groups, a single model will fail to capture those group-specific patterns. The solution is either separate models per group or group-aware modeling.
Feedback loop bias is often overlooked: a model's predictions influence future training data. A recommendation system that shows certain content to certain groups will create training data reflecting those recommendations, not the underlying preferences. This can entrench and amplify initial disparities over time.
Protected Attributes and Why They Are Tricky
Protected attributes are characteristics that should not be the basis for differential treatment: race, gender, age (for people over 40 in the US), religion, national origin, disability status, and others depending on jurisdiction and context.
The naive fix -- remove the protected attribute from your model's features -- does not work. Other features in your data are often correlated with protected attributes (zip code correlates with race, job title correlates with gender). Models will find these proxies and effectively use the protected attribute indirectly. This is called "redlining by proxy."
The technically correct approach requires explicitly measuring and constraining the model's predictions across demographic groups, not just removing the protected feature.
Fairness Metrics: The Three You Need to Know
There is no single definition of "fair." Different fairness metrics capture different intuitions, and choosing the right metric depends on your use case and values.
Demographic parity (statistical parity): The model should produce positive predictions at the same rate for all groups. If 30% of group A receives a loan, 30% of group B should also receive a loan. This metric does not account for whether the underlying base rates of the outcome differ between groups.
Equalized odds: The model should have equal true positive rate AND equal false positive rate across groups. A hiring model is fair by equalized odds if it correctly identifies equally-qualified candidates at the same rate for all groups AND incorrectly rejects qualified candidates at the same rate for all groups. This is generally considered a more rigorous fairness criterion than demographic parity.
Individual fairness: Similar individuals should receive similar predictions. If two people are identical on all relevant features, they should receive the same outcome regardless of their group membership. This metric is intuitive but hard to operationalize because it requires defining a similarity metric across individuals.
The impossibility result: Chouldechova (2017) and Kleinberg et al. (2016) proved mathematically that demographic parity, equalized odds, and calibration cannot all be satisfied simultaneously when base rates differ between groups. You have to choose which fairness criteria matter most for your application.
The Audit Process
Here is a concrete, repeatable process for auditing a model for bias:
Step 1: Define your protected groups. Which demographic groups are you concerned about? This depends on your application domain and jurisdiction. Document this explicitly.
Step 2: Collect group membership data. You cannot measure disparities you cannot see. You need group labels for your test set. This can be from self-reported surveys, proxy inference (imperfect but sometimes necessary), or external data linking.
Step 3: Stratify your test set. Split your evaluation data by group. Ensure each subgroup has enough samples for statistically reliable measurement (rule of thumb: at least 100 samples per group).
Step 4: Measure error rates per group. For classification: measure accuracy, false positive rate, and false negative rate for each group. For regression: measure mean error and error distribution per group. For ranking: measure position of relevant items per group.
Step 5: Test for statistical significance. A difference in error rates between groups might be noise. Use bootstrap confidence intervals or permutation tests to determine whether observed disparities are real.
Step 6: Document and report. Even if you do not change the model, document what disparities exist. This is increasingly required by regulation and is necessary for ongoing monitoring.
Tools for Bias Auditing
Fairlearn (Microsoft, open source): The most comprehensive open source toolkit. Provides fairness metrics, constraint-based training (optimize for performance while constraining disparity), and a dashboard for visualizing disparities across groups. Works with any scikit-learn-compatible model.
from fairlearn.metrics import MetricFrame, selection_rate, false_positive_rate
metric_frame = MetricFrame(
metrics={"selection_rate": selection_rate, "fpr": false_positive_rate},
y_true=y_test,
y_pred=y_pred,
sensitive_features=group_labels
)
print(metric_frame.by_group)
AI Fairness 360 (IBM, open source): A broader collection of fairness metrics and bias mitigation algorithms. Supports pre-processing (modify training data), in-processing (constrained optimization), and post-processing (adjust predictions) approaches.
Aequitas (University of Chicago): A bias audit toolkit specifically designed for public policy applications. Clear documentation on which metrics to use for which policy contexts.
What to Do When Bias Is Found
Finding disparate performance is step one. Addressing it requires choosing from several intervention points:
Collect more representative data. The most reliable long-term fix. If your model performs worse for group X, collect more high-quality labeled examples from group X. This is often the highest-leverage intervention but also the most expensive.
Reweigh training examples. Assign higher weights to underrepresented groups during training. Most ML frameworks support sample weights. Simple, fast, and often effective. The risk: over-weighting noisy examples from small groups.
Constrained optimization. Add a fairness constraint to your training objective. Instead of minimizing loss alone, minimize loss subject to the constraint that disparity across groups is below a threshold. Fairlearn provides this out of the box.
Post-processing threshold adjustment. Set different decision thresholds for different groups to equalize a chosen fairness metric. This can be done without retraining. The downside: it is a band-aid that does not fix the underlying model.
Keep Reading
- ML Model Evaluation Metrics Guide -- precision, recall, AUC, and the metrics that precede fairness analysis
- Cross-Validation Guide -- reliable evaluation is the foundation of reliable bias measurement
- The Complete Machine Learning Guide for Software Developers -- broader ML context
Pristren builds AI-powered software for teams. Zlyqor is our all-in-one workspace -- chat, projects, time tracking, AI meeting summaries, and invoicing -- in one tool. Try it free.
