# The Black Box Problem: Can We Govern What We Cannot Explain?

Author: Norbert Biedrzycki
Published: 2022-06-15
Language: en
URL: https://norbertbiedrzycki.pl/en/the-black-box-problem-can-we-govern-what-we-cannot-explain/
Summary: In May, the US Consumer Financial Protection Bureau issued a circular with a very clear message to lenders. When a creditor denies an application, the law requires it to give the applicant…

In May, the US Consumer Financial Protection Bureau issued a circular with a very clear message to lenders. When a creditor denies an application, the law requires it to give the applicant specific reasons. According to the CFPB, that obligation does not disappear because the decision was made by a complex algorithm that the creditor itself cannot fully explain. There is, in effect, no black box exception.

That sentence captures one of the central dilemmas of applied AI.

## Performance versus understanding

Modern machine learning, and deep learning in particular, achieves remarkable accuracy by finding patterns in thousands or millions of variables. The price of that performance is opacity. The model works, but neither its developers nor its users can give a simple account of why it produced a particular result.

In some applications that does not matter. Nobody needs an explanation of why a recommendation engine suggested a film. In others it matters a great deal: credit, insurance, hiring, medical diagnosis, criminal justice, welfare. In those domains, decisions affect people's lives and are subject to legal standards that assume reasons can be given.

Franz Kafka described a world in which a man is tried and condemned without ever learning the charge. The comparison is overused, but it explains why opacity provokes such a strong reaction. Being judged by a process you cannot understand is experienced as an injustice in itself, regardless of whether the outcome is correct.

## Three levels of explanation

It helps to separate who needs an explanation and why.

**The affected individual** needs to understand the main reasons for a decision and what could change it.

**The organization** needs to understand how the system behaves overall, where it fails and whether it is drifting.

**The regulator or court** needs to verify that the system complies with the law, which requires documentation, testing and evidence.

These are different requirements, and they can be met with different tools. Explainability techniques that approximate a model's reasoning for individual decisions, documentation of training data and design choices, monitoring of outcomes across groups. None of them makes a deep learning model transparent. Together they can make it governable.

## The real choice

Sometimes the honest answer is that a simpler, more interpretable model should be used, even at some cost in accuracy. In high stakes domains, a model that is two percent less accurate but can be explained and defended may be far more valuable than one that cannot.

For investors and boards, this is a material question. A business that depends on a model it cannot explain carries regulatory and legal risk that grows as rules tighten. The CFPB circular is an early signal of where enforcement is heading.

> *Leadership question: Is a highly accurate AI system acceptable if neither the customer nor the company can adequately explain its decisions?*
