# Algorithmic Bias: When Artificial Intelligence Reproduces Human Prejudice

Author: Norbert Biedrzycki
Published: 2020-10-15
Language: en
URL: https://norbertbiedrzycki.pl/en/algorithmic-bias-when-artificial-intelligence-reproduces-human/
Summary: In 2018 Reuters reported that Amazon had abandoned an experimental recruitment tool. Trained on ten years of CVs submitted to the company, most of them from men, it had learned to penalize…

In 2018 Reuters reported that Amazon had abandoned an experimental recruitment tool. Trained on ten years of CVs submitted to the company, most of them from men, it had learned to penalize applications that contained the word "women's", as in "women's chess club captain". Amazon's engineers tried to correct it and eventually concluded they could not guarantee it would not find other ways to discriminate.

Last year a study published in Science by Ziad Obermeyer and colleagues examined an algorithm widely used by US hospitals to decide which patients needed extra care. It used past healthcare spending as a proxy for health needs. Because less money had historically been spent on Black patients with the same conditions, the algorithm systematically underestimated how sick they were.

Neither system was designed to discriminate. Both did.

## The myth of the neutral machine

There is a persistent assumption that automated decisions are more objective than human ones. A person can be prejudiced. A calculation cannot.

The assumption is wrong for a simple reason. Machine learning does not learn the world. It learns the data about the world. If that data reflects past discrimination, unequal access or incomplete representation, the model will learn those patterns as if they were laws of nature. And because it applies them consistently and at scale, it can amplify what a single biased human would only occasionally do.

Bias enters at several points.

**The data.** Who is represented, who is missing, and what historical decisions are embedded in it.

**The proxy.** What the model is actually asked to predict. Healthcare spending is not health. Past hiring is not future performance.

**The deployment.** Where and on whom the system is used, which may be very different from where it was trained.

## Accuracy is not enough

For companies, this changes the definition of a good model. A system can be highly accurate on average and still systematically wrong for a particular group. Average accuracy is precisely the metric that hides the problem.

The organizations that handle this well do a few unglamorous things. They examine training data before building anything. They test outcomes across groups, not only overall. They monitor decisions after deployment, because data and populations drift. And they assign someone with authority to stop a model when the results look wrong.

None of that is technically difficult. What makes it difficult is that it slows deployment and requires business owners, not only data scientists, to take responsibility for outcomes.

## A question of responsibility

The harder question is legal and organizational. The developer built the model. The company deployed it. A third party may have supplied the data. When the outcome is discriminatory, each can plausibly point at the others. That ambiguity is convenient for everyone except the person affected, and it will not survive the first serious regulatory or legal test.

> *Leadership question: If an algorithm makes a discriminatory decision based on historical data, who is responsible: the developer, the company deploying it or the organization that provided the data?*
