What is Bias in AI? — AI Glossary

Bias in AI refers to systematic errors in AI model outputs that unfairly favor or disadvantage particular groups. AI bias typically originates in training data that reflects historical inequalities, but it can also arise from model design, objective functions, or deployment decisions. Biased AI can cause real harm — in hiring, lending, healthcare, criminal justice, and beyond.

AI does not invent bias from nothing. It learns from data produced by humans in a world with deep inequalities — and then often amplifies those inequalities at scale. When a hiring AI trained on historical promotions learns to penalize women’s resumes because women were historically promoted less, the AI hasn’t done something random or malicious. It has faithfully learned the patterns in its data. That is the problem.

Types of AI Bias

Bias enters AI systems through multiple pathways:

• Historical bias — training data reflects past discrimination. A credit model trained on 1990s lending data will reflect the discriminatory lending practices of that era.
• Representation bias — certain groups are underrepresented in training data. Facial recognition trained primarily on lighter-skinned faces performs worse on darker-skinned faces — a documented issue that has led to wrongful arrests.
• Measurement bias — the features used to represent a concept are measured differently across groups. Using arrest records as a proxy for criminality encodes policing disparities.
• Aggregation bias — training on aggregate data that doesn’t account for subgroup differences. A medical AI trained on general population data may perform poorly on demographic subgroups.
• Evaluation bias — testing a model on benchmarks that don’t represent the full deployment population.
• Deployment bias — a model built for one context is used in a different context where its assumptions don’t hold.

Real-World Consequences

AI bias has caused documented harm in several domains:

• Hiring — Amazon’s internal AI recruiting tool, trained on historical hiring data, learned to penalize resumes mentioning “women’s” and was scrapped
• Criminal justice — COMPAS, a recidivism prediction tool used in US courts, was found to produce false positives for Black defendants at nearly twice the rate as for white defendants
• Healthcare — an algorithm used to allocate healthcare resources was found to assign lower risk scores to Black patients with the same health conditions as white patients
• Facial recognition — MIT and NIST studies found commercial facial recognition systems had significantly higher error rates for darker-skinned women

Addressing AI Bias

Bias mitigation requires action at multiple stages:

• Data audits — examining training data for representation gaps and historical inequalities
• Fairness constraints — adding model objectives that enforce similar performance across demographic groups
• Diverse development teams — teams with diverse backgrounds are more likely to anticipate and catch bias issues
• Ongoing monitoring — measuring model performance disaggregated by demographic group after deployment
• Transparency and disclosure — publishing model cards and datasheets that document known limitations

There is no single definition of “fairness” — demographic parity, equal opportunity, and calibration are different mathematical formulations that are provably incompatible. This means addressing AI bias requires value choices, not just technical fixes, connecting to AI governance.

Common Misconceptions

Misconception: Removing demographic variables from the model eliminates bias. Models often infer demographic proxies from other correlated features — zip code, name, browsing behavior. Removing protected attributes does not prevent the model from using correlated proxies.

Misconception: AI is more objective than human decision-makers. AI encodes human biases in data at scale and applies them consistently — which can make the bias worse, not better, by removing the variability and discretion that human decision-makers sometimes exercise in favor of applicants.

---

Key Takeaways

• AI bias is systematic unfairness in model outputs, often amplifying historical inequalities.
• Bias enters through training data, model design, measurement choices, and deployment context.
• Real-world AI bias has caused harm in hiring, criminal justice, healthcare, and facial recognition.
• Mitigation requires data audits, fairness constraints, monitoring, and transparent disclosure.
• There is no single definition of fairness — addressing bias requires value choices, not just math.

Frequently Asked Questions

Is AI bias intentional?

Usually not — it is a byproduct of learning from biased data or optimizing for metrics that don’t capture fairness. But intent doesn’t reduce harm. Whether a discriminatory hiring decision came from a biased human or a biased algorithm, the person denied the job is equally affected.

What is algorithmic fairness?

Algorithmic fairness is a set of mathematical definitions for what it means for an algorithm to be fair — demographic parity (equal prediction rates across groups), equalized odds (equal true positive and false positive rates), and calibration (equal accuracy across groups). These definitions are provably incompatible, which is why fairness involves inherent value tradeoffs.

What is a model card?

A model card is a document accompanying an AI model that describes its intended use, performance characteristics, limitations, evaluation results disaggregated by demographic group, and known biases. Introduced by Google researchers in 2019, model cards are a transparency standard for responsible AI disclosure.

How does bias differ from statistical bias?

Statistical bias refers to systematic error in an estimator (like a sample mean that consistently overestimates the true mean). Social bias in AI refers to unfair treatment of individuals or groups. They are related — statistically biased performance metrics for certain groups can produce socially biased outcomes — but they are distinct concepts.

Are LLMs biased?

Yes. Studies have found that LLMs trained on internet text can exhibit gender, racial, and cultural biases — associating certain professions with genders, perpetuating stereotypes, or performing differently on text from different dialects. This is an active area of alignment and safety research.

---

Sources: Wikipedia — AI Bias · arXiv: Model Cards for Model Reporting · ProPublica: Machine Bias (COMPAS Investigation)

You May Also Like

• What Is Artificial Intelligence
• AI Glossary
• AI for Dummies
• How Does AI Work
• Essential AI Skills

---

Get free AI tips daily → Subscribe to Beginners in AI