In November 2021, Zillow shut down its AI-powered home-buying arm, wrote off more than $500 million, and cut a quarter of its workforce.
The model wasn’t hacked, and no regulator forced the closure. It simply kept being wrong, and nobody with the authority to stop it did until the losses were too large to absorb.
Although in the early days of AI, this is model risk in 2026: not a compliance checkbox, but a live financial and reputational exposure that traditional model risk frameworks were never built to catch.
The 30-second take
Traditional model risk management is like inspecting a train before it leaves the depot. AI risk management is more like operating a railway signal box.
The train may have been tested and approved, but once it is running, conditions keep changing. Routes change, other trains enter the network, signals move, faults emerge and decisions made elsewhere can affect the journey.
AI is the same. Models drift, data changes, vendors update their systems and the way the business uses AI evolves.
That means validation before deployment is no longer enough. Someone needs to continuously monitor the network, understand what is changing, control where the system is allowed to go and intervene when risk increases.
AI governance needs to move from inspecting the train to managing the signal box — continuously monitoring, directing and controlling AI throughout its journey.
Why the old model risk playbook doesn’t fit
Conventional model risk management was built for things like credit scoring cards: a model gets validated, deployed, and periodically reviewed against a fixed set of assumptions.
AI systems break that pattern. They can retrain on new data automatically, shift behaviour without a code change, and rely on third-party foundation models that update on someone else’s schedule.
A validation stamp from six months ago tells you almost nothing about what the model is doing today.
When the model breaks: Zillow’s $500 million lesson
Zillow Offers used a pricing algorithm, the Zestimate, to make automated cash offers on homes. Through 2021, as COVID-era market conditions shifted rapidly, the model’s price predictions drifted out of step with reality, but the business kept buying. Internal accounts of the collapse point to a deeper governance failure: management had fixed a target of purchasing 5,000 homes a month, and when the algorithm’s offers were too conservative to hit that volume, pricing experts were pushed to override the model’s own signals rather than the reverse. The technology problem, model drift, was compounded by a business decision to keep feeding it. Zillow closed the unit, absorbed Q3 losses of $304 million, and laid off roughly 2,000 staff.
The lesson for leaders is “don’t just blindly trust AI pricing models.”
Clear authority is required to challenge, and even pause, the model outputs to ensure they continue to make sense.
NIST’s answer: govern before you build
The US National Institute of Standards and Technology’s (NIST) AI Risk Management Framework puts accountability at the centre of its “Govern” function, which requires organisations to embed risk culture, clear ownership, and cross-functional sign-off across the entire AI lifecycle, not just at launch.
Engineering, legal, risk, and the business unit deploying the model are expected to share responsibility continuously.
Through 2026, NIST has been extending the framework with additional profiles, including guidance for agentic AI systems, reflecting how fast the ground is still moving.
The framework is voluntary and American, but the underlying principle, that governance has to be a standing function rather than a gate, applies wherever the model is deployed.
Australian boards are moving faster than their governance
Research from the Australian Institute of Company Directors (AICD) and the University of Technology Sydney’s Human Technology Institute found 90 per cent of surveyed organisations are now using or planning to use AI, up from 64 per cent in 2022.
The same research found 69 per cent of director respondents had personally used AI for board work in the previous six months. Adoption, in other words, has outrun governance at the board level itself, not just inside the business units deploying customer-facing models.
AICD and HTI have since published director-focused guidance to close that gap, but the survey data is a warning: the accountability question Zillow faced at the model layer is now showing up in boardrooms too.
Questions to ask your organisation
- Who owns the decision to override or pause an AI model, and how quickly can they act on it?
- Is model drift tracked after deployment, or only validated once before go-live?
- If your AI vendor changes the underlying model, would you find out before your customers notice a difference?
- Are business targets ever allowed to override a model’s own risk signals, and who has to approve that trade-off?
- Has your board had AI risk training beyond a single introductory briefing?
- Would your organisation know within 24 hours if an AI system started producing consistently degraded or biased outputs?
Where to start
None of this requires a full framework rebuild overnight, but it does require knowing where your organisation actually stands today.
The AI Signal BoxTM is our approach to embracing risk, and governing AI in a positive and proactive manner.
Innovation of Risk has a free readiness snapshot built for exactly this gap, a fast way to see how your AI model governance holds up before you have your own drift story to tell.