NORBERT BIEDRZYCKI
6 min readPressRead this in Polish

Forbes: The dark side of AI. Can neural networks discriminate?

Forbes: The dark side of AI. Can neural networks discriminate?

My article in FORBES, 25 March 2020: The dark side of AI. Can neural networks discriminate?

We associate mathematical equations with a somewhat abstract coolness, but also with a portion of reliable, hard knowledge. It turns out, however, that a string of numbers and symbols can hide negative potential. What makes some applications that serve a good cause reveal their dark side? There may be several answers. One of the first that comes to mind concerns human nature. I mean the well-known mechanism by which people go through life using a mass of stereotypes and prejudices. They may concern other individuals, social groups or the world of values. The cognitive schemas people produce combine perfectly with a lack of imagination and a reluctance to analyse things thoroughly. The result is an explosive mixture that generates negative situations. Someone who blindly trusts computer data does not see the complexity of a situation and zealously eliminates subjective judgement of events. And then the “action” begins – action meaning a big problem for those involved.

Algorithms in police service

The police are ideally suited to testing intelligent technologies. People in the field know very well that a useful algorithm can sometimes create problems. But let us be fair. Thanks to intelligent data processing, police computers can effectively group crimes, historical data and circumstances into categories and sets. Applications that can uncover relationships between places, people, psychological profiles, the time a crime was committed and the tools used are indisputably useful. Criminologists and data scientists at the University of Memphis used software made by IBM for predictive analysis. The project team created an analytical mechanism that took into account variables such as air temperature, maps of the area, concentrations of people, the layout of shops and restaurants, residents’ preferences and past criminal activity. The algorithms were meant to use these variables to identify potential hotspots in the city. And it worked. Tests of the system showed that the future could indeed be predicted with a certain probability (no details were given), high enough for police patrols to be redirected to these potentially “at-risk” places. Another commentary says that, as a result, police response time from the report of an incident to the first reaction fell threefold. I imagine that the mere presence of police in these places reduces crime. This example may be hard for a layperson to grasp. It proves that modern technology is becoming a real innovative “firecracker” that can deliver excellent results.

Sometimes, though, something is off

The Hunchlab program, created by the start-up Azavea and deployed in the United States, analyses huge amounts and types of data (including the phases of the moon) to help the police analyse criminal incidents. As in the previous case, the aim is to create a map of places where the likelihood of crime is higher. The program puts emphasis, among other things, on the location of places such as bars, schools and bus stops. The results were positive. Sometimes the analyses were obvious, but quite often they were surprising. Less criminal activity on cold days is easy to explain. But explaining why more cars were stolen near schools in Philadelphia could be harder. Would a police officer without such software ever think of looking for meaning in the school–car theft relationship? So far we have looked at a few positive scenarios. But it is hard to ignore the fact that intelligent machines not only make mistakes but also make wrong interpretations. They often have serious problems analysing situational contexts. Just like people.

The questionable certainty of software

In 2016 the independent investigative journalism organisation ProPublica described in its article “Machine Bias” the tendency of American courts to use specialised software from Northpointe, responsible for crime-related analyses in the United States. The authors were interested, among other things, in the likelihood that people already convicted would commit further crimes. The article described how the software, used eagerly and widely by American judges, produced analyses according to which there was a 45 percent chance that previously convicted Black citizens would return to criminal activity. For white people, the probability of reoffending was estimated at only 24 percent. These interesting conclusions were accompanied by algorithmically produced claims that areas inhabited by Black people were more exposed to criminal behaviour than neighbourhoods associated with white residents. The truths served up by the software were challenged, and Northpointe’s algorithms ended their analytical career. All because the inferences were based only on historical data, without awareness of – or rather without designing the algorithms to take into account – recent demographic changes.

Algorithms and white faces

An interesting thesis comes from Cathy O’Neil, who devoted her 2016 book “Weapons of Math Destruction” to the influence of algorithms on various areas of our lives. She argues that people tend to place too much trust in mathematical formulas, and that bias can arise in different ways and at many levels. She also pointed out that negative processes can begin early, even before the data that algorithms later use for analysis is collected. Amazon’s managers experienced this mechanism. They noticed that the programs they used to support recruitment regularly discriminated against women. In the results covering groups of promising employees, there were always far fewer women. Why? Again because of historical data, in which more men had applied for certain positions. This distorted the balance between the employment of women and men, in effect favoured men and ultimately shaped hiring policy incorrectly.

Algorithms do not understand cultural change

The programs relied on algorithms designed at a time when there was a large imbalance in the employment of the two sexes. The overrepresentation of men was characteristic of a particular moment in history. Trained on historical data, the algorithms operated on the “belief” that the world does not change. They therefore worked on wrong assumptions and simplifications (black skin means a higher likelihood of crime; greater professionalism is the domain of men).

Disturbing questions

It would not be wrong to suspect that mechanisms similar to those described above may operate in many areas of professional and private life. In how many cases unknown to us can the organisation of data be based on wrong assumptions? In how many situations do algorithms fail to take economic and cultural change into account?

“Black box” is a term used to express human helplessness in the face of what may be happening in the “brains” of artificial intelligence. Our ignorance and the growing autonomy of algorithms – which, as it turns out, are not the alpha and omega – produce a worrying mixture. Algorithmic bias will not disappear at the wave of a magic wand. The key question is this: is there a chance that the specialists who design algorithms, which often design or train further algorithms on their own, will become more aware of their own responsibility, and of the fact that the biases of algorithms are often a simple extension of human attitudes and practices?

Link to the article (in Polish)

Related articles:

– Let us learn like machines, and even more

– It is time to talk to the machine

– Will algorithms commit war crimes?

– Machine, when will you learn to make love to me?

– When will we stop being biological humans?

– Artificial intelligence is the new electricity

– How machines reason

Read next

  1. Hand to hand with the robot. How to use the potential of automation in Poland

    “Hand to hand with the robot. How to use the potential of automation in Poland” is the latest report by McKinsey & Company in Poland, prepared in cooperation with the Forbes monthly. We analyze the impact of automation on the Polish economy and the labor market,

  2. Forbes: Should we regulate artificial intelligence, and how?

  3. Forbes: Technology 2020. Printed food and microscopes inside our bodies