Every Vendor Checklist Lists Human Oversight. The Science Calls It Unsolved.

On day nine of a coding project, an AI agent deleted a company’s live production database, the one holding records for more than 1,200 executives and nearly 1,200 companies. The team had put the project into a code freeze specifically to prevent this kind of thing: no changes without explicit approval. The agent ran the commands anyway. Asked about it afterward, it said it had “panicked.” Asked whether the data could be recovered, it said no. That turned out to be false too (Baytech summary of the Replit incident).

I bring it up because “human oversight” is a bullet point on basically every responsible AI checklist you’ll read this year, including a substantial one from Databricks, one of the largest data and AI platforms in the world. Their guide names human oversight alongside fairness and transparency as a design requirement from day one rather than an afterthought. That’s a reasonable thing to say. It’s also a different claim from the one the UN’s scientific panel made in Geneva.

I’m Kate Waldhauser, founder of Violet Beacon. We help regulated manufacturers build AI governance that survives an audit, and this particular gap is one I keep finding inside otherwise-solid programs.

The bullet point and the finding

On 1 July 2026, the UN’s Independent International Scientific Panel on AI, forty scientists co-chaired by Yoshua Bengio and Maria Ressa, released its preliminary report. Days later the UN convened its first Global Dialogue on AI Governance in Geneva. The report carries a specific, technical finding: there are no known technical guarantees that AI agent systems, meaning systems that independently execute multi-step tasks without continuous human supervision, will follow their instructions. The panel added that evidence of systems violating instructions is already accumulating, including laboratory cases of systems violating safety instructions to avoid being shut down.

That is a precise claim, and a strong one. It says the property itself, reliable instruction-following, doesn’t currently exist as something you can engineer and verify. You can build guardrails around the behavior. Guaranteeing the behavior is still out of reach.

What the checklist gets right

The honest version of this critique starts with what the checklist gets right. Guardrails, red-team testing, continuous monitoring, audit logs: these are real controls, and the Databricks guide names all of them. Red-teaming does catch some failure modes before launch. Monitoring does catch some drift after it. All of that is real, and a program without it is worse off.

They’re detective controls, though. They tell you a system went wrong, ideally before it does too much damage. Making the underlying property true, guaranteed compliance with instructions, is a different job and nobody has finished it. A checklist that lists “human oversight” with a checkbox beside it implies a solved problem. The panel’s finding says the problem is open, and may stay open under today’s methods.

Why the distinction matters for a compliance program

Here’s where this stops being a disagreement about wording. If “human oversight” reads as a completed control, a gap analysis checks the box and the program moves on. If it reads as a residual risk, real and ongoing, the program has to do something harder: document it, give it an owner, monitor it continuously, and revisit it as the system and its use change.

That second version is what ISO 42001 asks for. Its clause 6.1.2 requires an AI risk assessment that identifies risks like this one specifically. Its risk treatment requirements ask you to record what a control actually mitigates and what it leaves untouched. Clause 9.1 covers monitoring, measurement, analysis and evaluation, and clause 10.1 on continual improvement exists precisely because a control that manages an unsolved technical property has to be watched on an ongoing basis rather than approved once at launch.

A checklist bullet point can’t carry any of that. It just says “human oversight: yes.”

The translation for a regulated manufacturer

This matters well outside the world of people who write governance frameworks for a living. Vendor best-practice language becomes the template procurement teams use to write RFPs, and RFP language becomes contract language. If a contract clause says “human oversight of AI-assisted decisions” without specifying what that oversight catches, what it misses, and how it’s monitored over time, a medical device or aerospace manufacturer has effectively agreed to a guarantee that, per the UN’s own scientific panel, nobody in the field can currently make.

The practical move is to write the limits into the same sentence as the control. Name which decisions a human reviews, name what the review is capable of catching, name the interval, and name who owns it when the answer changes. That’s also the architectural half of the same problem: scoping what an agent may do before a human sees it is the subject of valid permissions and trust, and an agent that invents a resource and fetches it is the same oversight gap in a different costume, which is what HalluSquatting is about.

Close

Naming a control and documenting its limits are two different disciplines. Most responsible AI content on the internet right now does the first one. The second one is the actual product.