I was in a workshop recently when a client said something that has stayed with me. She told me her organization had a real appetite for AI right now and that they were ready to move. I asked where they wanted to plug it in. She wasn't sure. I asked about their AI initiatives team. She said they were still working on it.
That organization was already mid-implementation on an AI tool in another part of the business.
This is not an isolated case. It is a pattern. And the more I see it, the more I think the most pressing question in AI adoption right now is not "what can this do?" but "what happens when you move forward without answering that first?"
The Board Said Yes. That Is Not a Strategy.
There is a version of AI enthusiasm that looks like momentum but functions more like pressure. A board identifies cost savings or competitive positioning, approves an initiative, and the organization begins moving forward before the foundational questions have been answered. What is this tool for, specifically? Who owns the outcomes? What are the boundaries of where it operates and where it does not? What does a failure look like, and who is accountable for it?
When those questions stay unanswered, implementation still happens. It just happens without the structural support it needs to work well.
Researchers studying this pattern have given it a name: "ungoverned incompetence." It describes what occurs when leadership makes strategic AI decisions without the digital context required to make them well. The consequences are not abstract. Organizations in this situation tend to have lower revenue growth and weaker market performance than competitors who approached AI with clearer governance structures. The enthusiasm is real. The outcomes are not what anyone intended.
What Gets Lost When Humans Are Replaced Too Quickly
One of the most significant and least discussed costs of ungoverned AI adoption is what happens to institutional knowledge.
When organizations reduce headcount in response to AI capabilities, they are often making a short-term calculation against a long-term asset. The people who are let go carry context that does not live in any system: knowledge of client history, of what was tried and failed, of why certain decisions were made the way they were, of the nuances that make a relationship work. That knowledge is not easily documented, and it is not something an AI system accumulates through use.
The cycle that follows is becoming familiar. Organizations lay off significant portions of their workforce expecting AI to absorb the capacity. Complaints rise. Customer trust erodes. Operational losses increase. A study of U.S. bank holding companies found that a one standard deviation increase in AI investment, without corresponding risk controls, was associated with a 24% increase in quarterly operational losses. At that point, organizations begin hiring again, but with reduced budgets. They can no longer afford the senior practitioners they let go. They hire junior staff who lack the institutional context that was just eliminated. The problem compounds.
This is not a technology failure. It is a governance failure that the technology made visible.
Speed Can Produce Catastrophic Outcomes
Automated systems operate at a pace that human intervention cannot always match. When something goes wrong inside a well-governed system, there are checkpoints. Someone is watching. There are defined thresholds and escalation paths. When something goes wrong inside an ungoverned system, a small error can escalate before anyone realizes it is happening.
The 2012 Knight Capital incident is the frequently cited example: a single algorithmic error caused hundreds of millions of dollars in losses in under an hour. The technology was not the problem. The absence of adequate controls around the technology was.
Most organizations implementing AI today are not financial trading firms operating at millisecond speeds. But the principle holds. Systems that operate without human accountability structures, without defined roles for oversight, without documented limits on what the tool should and should not do, carry a risk profile that the initial business case almost never accounts for.
What Governance Actually Requires
Governance is not a bureaucratic exercise. It is the operational discipline that makes AI implementation sustainable rather than brittle.
At a minimum, it requires knowing what AI tools are in use across the organization, including the ones that came in through vendors or were quietly adopted by individual teams. It requires ranking those tools by impact and regulatory sensitivity, not all AI carries the same risk, but all of it carries some. It requires defining who is responsible for each system's outcomes and what that accountability looks like in practice. And it requires integrating those definitions into how the organization actually operates, not just into a policy document that sits in a shared drive.
None of this is complicated in principle. It is just work that gets skipped when the pressure to move fast overrides the discipline to move well.
The Human Readiness Problem Is Just as Real
Even well-designed AI tools underperform when the people who need to use them are not prepared to use them. Research consistently identifies what is called a "human readiness gap" as one of the primary barriers to successful AI implementation. Staff who lack sufficient AI literacy do not just underuse the tool. They resist it, work around it, or misapply it in ways that introduce new problems while solving none of the original ones.
This gap is not a training problem that gets solved with a single onboarding session. It reflects something deeper about how the organization has or has not built AI fluency across its workforce. Addressing it requires time, investment, and the kind of honest assessment of where people actually are, not where leadership assumes they are.
The Question Worth Sitting With
Appetite for AI is not a strategy. It is a starting point. The organizations that will use AI well are the ones that treated the governance work as seriously as the implementation work, that asked the hard questions before timelines hardened, and that kept humans in the loop not as a formality but as a structural requirement.
The ones that skipped that work are already starting to see what it costs. Some are trying to undo decisions that were made too quickly with too little context. Some are rebuilding trust with customers who experienced the consequences of systems that had no one accountable for their outputs.
Having an appetite for AI is not the risk. Moving forward without defining what you are actually doing is.
