Human oversight and the accountability gap in AI

Across jurisdictions and standards bodies, the expectations for AI used in consequential decisions are converging on a common core: human oversight, transparency, and traceability. This is no longer aspirational language. It is being written into regulation and formal frameworks.

The European Union's AI Act, for example, requires that high-risk AI systems be designed and developed so that they can be effectively overseen by people while in use. The aim is that a person can understand the system's output, decide not to rely on it, and intervene or stop it.

The NIST AI Risk Management Framework approaches the same problem through governance. It emphasises mapping and measuring risk and, crucially, documenting decisions and maintaining traceability across the AI lifecycle, so a system's behaviour can be understood and its risks managed rather than assumed away.

What both have in common is a shift from principles to records. Oversight that leaves no trace cannot be audited. Accountability, in practice, means being able to reconstruct after the fact who decided what, on what basis, and with what human involvement.

The expensive way to meet these expectations is to retrofit them: to add logging, review steps, and documentation to a system that was never designed to produce them. The cheaper and more defensible way is to treat oversight and traceability as design requirements from the start.

That means scoping each automation, recording its inputs and actions, gating the consequential steps, and keeping a durable record of human approvals. Done up front, compliance stops being a scramble and becomes a property of the system.