Human in the Loop: Does AI Need Human Supervision to Remain Responsible?

In January, a television drama about the British Post Office scandal reached millions of viewers and turned a long running injustice into a national political issue. Between 1999 and 2015, more than 900 subpostmasters were prosecuted on the basis of data from the Horizon accounting system, which contained bugs. Courts, investigators and managers treated the computer’s numbers as more reliable than the people in front of them. Lives were destroyed. Horizon was not an AI system. But the scandal is the most powerful warning we have about what happens when human oversight exists on paper and fails in practice.
Oversight as a design principle
Human in the loop has become the default answer to AI risk. The EU AI Act requires high risk systems to be designed for effective human oversight. Corporate AI policies routinely require that a human reviews AI outputs before they affect customers or employees.
The principle is right. The problem is that a human in the loop is not automatically a safeguard. Research on automation bias, going back decades in aviation and medicine, shows that people tend to over trust automated recommendations, especially when the system is usually right, when they are busy and when disagreeing requires effort.
When oversight becomes theater
Oversight fails in predictable ways.
Volume. A reviewer asked to approve hundreds of AI decisions a day cannot evaluate them meaningfully. Approval becomes a click.
Competence. The reviewer does not understand how the system works or what its typical failure modes are.
Authority. The reviewer can technically override the system but faces pressure, targets or culture that discourage it.
Information. The reviewer sees the AI’s conclusion but not the evidence needed to challenge it.
Accountability asymmetry. Following the system is never punished. Overriding it and being wrong might be.
In each case the organization can say a human was involved. In none of them did the human provide real oversight. The Post Office had many humans in the loop.
Making oversight meaningful
Effective oversight is a design problem, not a policy statement.
Match oversight to risk. Not every AI output needs review. High stakes decisions need serious review. Low stakes ones need monitoring.
Give reviewers time and context. If the economics of a process depend on reviewers spending ten seconds per case, the oversight is not real.
Train for failure modes. Reviewers need to know where the system tends to be wrong.
Measure override rates. An override rate close to zero is not evidence that the system is perfect. It may be evidence that oversight is not happening.
Protect disagreement. People who challenge the system in good faith should be supported, even when they turn out to be wrong.
As generative AI moves into financial analysis, customer communication and HR processes, this question is becoming operational in almost every company. The risk is not that organizations will remove humans from the loop. It is that they will keep them there in a form that provides comfort without control.
Leadership question: How do we ensure that human oversight is genuine and meaningful rather than simply a formal approval step?


