Your AI Governance Program Has No Plan for Loops

In early 2021, Zillow made one governance decision that would eventually cost it $500 million and a quarter of its workforce: it told the people paid to question its home-pricing algorithm to stop questioning it. Everything else in this post is what happens once that decision becomes normal practice instead of a scandal.

For the last two years, the unit of AI work has been the prompt: write a better instruction, get a better answer. That phase is ending. The industry is calling the next thing “loop engineering,” systems that observe, act, get feedback, and adjust on their own, without a person issuing every instruction. The examples so far are mostly technical, like coding agents that write and review each other’s work. But the shift underneath it is bigger than software. When a company lets an AI system run its sales targeting, its procurement, or its credit decisions in a loop, that stops being a technical choice. It becomes a governance structure.

What changed when AI moved from prompts to loops?

A prompt asks for one answer and disappears. A loop creates behavior and compounds it.

That distinction matters because a loop can compound small decisions over time. It learns what gets rewarded, what gets ignored, and what makes a dashboard look better, even when those signals are incomplete or misaligned with what people actually want.

In practice, the governance target has changed. A prompt can be reviewed as a point-in-time interaction. A loop has to be reviewed as an ongoing system, because a launch checklist can approve a starting state but can’t govern what the loop becomes later. For boards, operators, and compliance leads, that’s the shift worth naming: the question is no longer only “did we approve this use case.” It’s also “what is this system learning to optimize now.”

Why does every AI loop carry a values decision?

Whatever a loop optimizes for is a decision about what the company values, whether or not anyone wrote it down. Call it the loop’s politics. A customer service loop optimized for resolution speed might learn to close tickets faster while trust quietly erodes underneath. A pricing loop optimized for margin might produce outcomes that look efficient on a dashboard and land as discriminatory to the people on the other end of it. A loop nobody is watching closely enough is sufficient to produce results like these, with no malicious model required.

What does Zillow’s Project Ketchup show about losing real oversight?

That decision has a name: Project Ketchup, Zillow’s internal push to use its Zestimate algorithm as the actual cash offer on homes it was buying. Read the failure again, slowly: the people whose job was to catch the algorithm’s blind spots were explicitly instructed to stop doing that job. The people who could have caught the drift were told not to, a governance decision made on purpose to remove the one checkpoint that might have caught it before it compounded. The failure runs deeper than a bad model.

By the third quarter of 2021, Zillow reported a $421 million loss tied to the iBuying business (the unit shut down entirely soon after). The case is often told as a forecasting failure: the algorithm couldn’t predict home prices accurately enough in a volatile market. That’s true, and it’s also not the whole story, as a teaching case built around the closure makes clear. The algorithm failed with its safety mechanism deliberately switched off.

Why isn’t “human in the loop” a control by itself?

The phrase sounds reassuring, but it hides the details that matter most. A person rubber-stamping machine-speed decisions is liability with a user interface. Zillow shows the sharper version of that failure: the humans present were told their judgment no longer counted. A real control needs answers to specific questions: which person is responsible, what can they see, when do they step in, and what can they actually stop, reverse, or escalate.

What does continuous governance actually require?

Most AI governance is still built for a static object: a model gets assessed once, a use case gets approved once, a compliance binder gets filed once, and everyone moves on. A learning loop doesn’t hold still long enough for that to work. It changes through use, the same way a system trained on its own output keeps drifting from where it started.

The NIST AI Risk Management Framework is built around a cycle of govern, map, measure, and manage, not a single approval gate. ISO/IEC 42001’s continual-improvement clause, Section 10.1, requires the same thing: an AI management system that gets maintained and improved on an ongoing basis, not signed off once at launch. Both frameworks point at the same requirement: governance has to run alongside the loop, not just at launch. The loop itself needs a declared objective, a visible reward function, an audit trail, and a real stopping condition, built into how it runs rather than filed away as a policy nobody rereads after go-live.

Is loop engineering just old automation with a new name?

That’s a fair objection. Companies have run pricing bots and algorithmic trading loops for decades without a fresh governance framework built for the occasion. Maybe boards already have the tools they need, and this is vocabulary catching up to a practice that already exists.

That’s worth taking seriously. But Zillow ran Project Ketchup for the better part of a year before the loop’s blind spot became a $500 million write-down (and cost a quarter of the company’s workforce their jobs), and the fix that might have caught it earlier, letting the pricing experts keep questioning the algorithm, was a governance decision that got quietly removed, not a modeling bug that got introduced. The vocabulary is new. The stakes of getting the governance wrong aren’t.

What should this mean for a mid-market regulated manufacturer?

If you’re running AI in a regulated shop, medical device, defense, aerospace, or any supply chain with an OEM watching your quality record, the pathway from “our AI agent handles procurement” to “this loop is now making decisions that used to require a sign-off” is short, and it’s usually invisible until an audit or an incident forces the question.

Zillow’s failure came down to a checkpoint that got quietly switched off. The same mechanism shows up in AI agents that write to production data: a vendor-risk-scoring agent that starts auto-approving suppliers once a procurement lead stops double-checking its exceptions, or a quality-inspection model that gets trusted to auto-clear a batch after enough good runs, until the engineer who used to spot-check it stops getting asked. Industries change. The failure mode is constant: quietly revoking a person’s authority to say no. That’s exactly the gap ISO 42001’s continual-improvement clause, Section 10.1, is built to close, and it’s the same gap that’s increasingly showing up in vendor contracts before it shows up in statutes.

The real test is what this loop has learned to do since it was approved, and who’s still allowed to question it. Violet Beacon’s work on AI governance and services starts with that question, the one most launch checklists never get asked to answer.

The clients who avoid this failure aren’t the ones with the fanciest monitoring dashboard. They’re the ones who can still tell me, without hesitating, exactly who on their team is allowed to say “that doesn’t look right” out loud, and have it actually stop the process.

Key references

• NIST AI Risk Management Framework
• ISO/IEC 42001 overview
• Fast Company, “AI learning loops aren’t an engineering trick. They’re a governance issue,” Enrique Dans, July 7, 2026
• Gudigantala, N. & Mehrotra, V., “Teaching Case: When Strength Turns Into Weakness: Exploring the Role of AI in the Closure of Zillow Offers,” Journal of Information Systems Education, Vol. 35, Issue 1 (2024)