AI and Algorithms

Any drawing contrasting a helpful computer use with a uniquely human activity. For example: 'Computers help with finding the way on a map. Only a person can give me a hug when I am sad.'

Computers can be helpful when they do things fast, like finding information or playing music. We should always be able to ask a real person when something is wrong, when we need help, or when we need someone to listen.

An algorithm is a set of step-by-step instructions for solving a problem or making a decision. Computers follow algorithms very quickly, on a huge scale. Every recipe is an algorithm; every route from one place to another is an algorithm. Algorithms are everywhere in modern life. One example I have used today is the recommendation algorithm on a video streaming app. When I finished watching one video, the app suggested another — this was not random. An algorithm looked at what I had watched before, what other people like me had watched, and what was new. It then picked videos most likely to keep me watching. This is helpful — I sometimes find things I enjoy — but it also means the app is shaping my choices, showing me some things and hiding others. Algorithms are useful, but it helps to know they are there and what they are doing.

Bias in AI systems is a serious problem because AI is increasingly used to make decisions that affect people's lives — hiring, loans, housing, healthcare, and criminal justice. When an AI system is biased, it treats some groups unfairly based on race, gender, or other characteristics. Bias happens because AI learns from data, and if the data reflects past discrimination, the AI reproduces it. For example, a major company built AI to sort job applications but had to stop using it when it was found to downgrade women's CVs — because it had learned from a hiring history dominated by men. Facial recognition systems have been found to work less well on darker skin tones, leading to wrongful arrests. To address this, AI systems should be tested for bias before use, use diverse and representative data, be designed with fairness in mind, and be subject to independent checking. When AI systems make important decisions about people, there should always be a way to appeal to a human, and laws should make companies responsible for the harm their AI causes.

The claim that AI should not make significant decisions without meaningful human oversight captures a widely shared principle in AI ethics. On careful examination, the claim is substantially correct and supported by strong evidence — though it requires some refinement. The key question is what 'meaningful human oversight' actually means, and whether current practices provide it.

The case for the claim is powerful. Multiple high-profile cases have shown AI systems making consequential errors or producing discriminatory outcomes. The Dutch childcare benefits scandal (2013-2020) used algorithmic profiling that wrongly accused over 30,000 families — predominantly with dual nationality — of welfare fraud. Families lost homes, children were taken into state care, and the government eventually resigned. Amazon's experimental hiring AI was abandoned in 2018 after systematically downgrading women's CVs. UK's 2020 A-Level algorithm had to be withdrawn after it unfairly penalised students from poorer schools. COMPAS, a US criminal justice risk assessment tool, was shown by ProPublica to produce racially biased outputs. In each case, AI affected crucial life outcomes — welfare, employment, education, liberty — with insufficient human checking.

These are not isolated failures. They reflect structural features of machine learning. AI systems learn from data reflecting past discrimination. They can be confident about wrong answers. They combine training data in ways their designers may not fully understand. They cannot easily explain their reasoning. And the speed and scale at which they operate means errors can affect thousands of people before being noticed.

The principle of human oversight responds to these realities. A human who actually reviews decisions can catch obvious errors. A human who understands the context can identify discriminatory patterns. A human who can explain a decision provides the accountability that AI alone cannot. Appeal rights — the ability to ask for human review — provide protection against specific harms.

However, the claim requires important qualification. 'Meaningful' human oversight is harder than it sounds. Research shows humans often rubber-stamp AI decisions — 'automation bias'. A human reviewer who sees hundreds of AI decisions a day cannot meaningfully check each one. Oversight must be genuine, informed, and empowered, not theatrical. Technical explainability matters — humans cannot oversee what they cannot understand. Adequate time and resources matter — oversight costs money that organisations may resist.

Furthermore, not all decisions are equivalent. Applying the principle too broadly may block useful AI applications where automation causes little harm. A streaming service recommending a film does not need human oversight for each recommendation. AI sorting spam emails does not need individual review. The principle should apply most strongly to high-stakes decisions — welfare, justice, employment, healthcare, liberty, safety — where errors cause serious harm.

The EU AI Act's risk-based approach reflects this nuance. Unacceptable risk systems (social scoring, manipulation) are banned entirely. High-risk systems (in employment, education, criminal justice, healthcare) face strict oversight requirements. Lower-risk systems face lighter rules. This framework tries to preserve benefits of AI while protecting against the most serious harms. Critics argue it goes too far; defenders argue it does not go far enough. The direction is defensible even where details are contested.

On specific design questions: meaningful oversight requires transparency (knowing when AI is used), explainability (understanding how decisions were reached), appeal rights (genuine human review), documentation (audit trails), accountability (someone specific answerable for harm), and genuine authority (oversight that can actually override AI when needed). Each is a design challenge; each matters.

In conclusion, the claim is correct for significant decisions affecting people's lives. The evidence of AI errors, bias, and opacity justifies it. But 'meaningful oversight' is demanding — it requires more than nominal human presence. AI systems can be powerful tools, but in the areas where their errors cause serious harm, they should support human decision-making rather than replace it. Building systems that genuinely meet this standard is the central governance challenge for AI in the coming years.

Algorithmic bias describes systematic unfairness in AI systems, producing outcomes that disadvantage particular groups — often those already facing discrimination. Bias arises from several connected mechanisms. First, training data often reflects past discrimination. An AI learning from historical hiring patterns in a male-dominated industry will favour male candidates. Second, representation gaps mean AI performs less well on underrepresented groups. Facial recognition trained mostly on lighter-skinned faces works poorly on darker-skinned faces, with documented error rate differences of up to 34%. Third, design choices — what counts as 'success', 'risk', or 'qualification' — embed assumptions that can reflect existing inequalities. Fourth, proxy variables (postal codes, names, speech patterns) can reintroduce protected characteristics even when explicitly excluded. Fifth, feedback loops mean AI decisions shape future data, reinforcing initial biases. The Dutch childcare benefits scandal, the COMPAS recidivism tool, and Amazon's hiring AI all illustrate how these mechanisms produced serious harms.

One important approach to addressing bias is algorithmic auditing — systematic testing of AI systems for fairness before and during deployment. Audits examine training data for representation gaps and historical bias. They test outputs across different demographic groups to identify disparate impacts. They examine design choices for embedded assumptions. They look for proxy variables that might reintroduce bias. Auditing can be internal (organisations checking their own systems) or external (independent reviewers with access to the system). The EU AI Act requires high-risk AI systems to undergo conformity assessments, and some jurisdictions now mandate algorithmic impact assessments.

Auditing has real limits. It requires technical expertise and access that is often restricted. Organisations may resist external oversight. Fairness itself is mathematically complex — different definitions of fairness can conflict. But audits have also successfully identified significant biases that would otherwise have caused harm. Combined with other tools — diverse design teams, appeal mechanisms, legal accountability — auditing helps translate the principle of fair AI into practice.