The AI Governance Paradox
MIT Sloan Management Review ·
Getty Images
Operations that run under extreme risk — such as nuclear power plants, aircraft carriers, and air traffic control centers — have long understood that safety depends not just on good technology but on how people exercise judgment under uncertainty. The scholarly field of high-reliability organizing (HRO) has identified a clear normative framework to distinguish organizations that manage catastrophic risk well from those that succumb to it: Reliable organizations protect front-line judgment , distribute decision authority to those closest to the problem, and treat emerging anomalies as signals rather than noise.
These principles are hard-won lessons from disasters. At Three Mile Island in 1979, for instance, operators of the U.S. nuclear power plant confronted more than 100 simultaneous alarms with no prioritization mechanism , producing cognitive paralysis rather than an informed response when the reactor began to melt down. Before the space shuttle Columbia ’s deadly final flight in 2003, NASA engineers who suspected that a wing’s insulating foam had sustained damage during launch had escalation channels available but refrained from using them due to the organizational costs of challenging senior leaders. NASA’s accident investigation report said that “management techniques unknowingly imposed barriers that kept at bay both engineering concerns and dissenting views, and ultimately helped create ‘blind spots’ that prevented them from seeing the danger the foam strike posed.”
Today’s AI-enabled threat-detection systems — the adaptive machine-learning tools that surface fraudulent transactions, network intrusions, compliance violations, early signs of patient deterioration in nursing care, and the like — promise superhuman pattern recognition at scale. These threat-detection technologies, however, remain prone to bias, drift, and hallucination, and pose their own risks. A false positive can flood an organization with noise and lead to wasted resources. A missed threat can be catastrophic, as the cyber hacks a few years ago by Salt Typhoon against U.S. telecom networks and the massive 2024 data breach of Change Healthcare have shown.
Emerging AI governance frameworks, designed in part to protect organizations from AI technology falling short of its intended task, are paralleling underlying principles of the original HRO playbook (whether they realize it or not). Regulatory and academic debate is converging on governance practices that operationalize HRO principles for AI contexts, with edicts to, for example, keep a human in the loop where AI is implemented.
But my research on accountability systems suggests that these governance structures may harbor a hidden problem that both HRO theories and emerging AI governance frameworks do not adequately address. The paradox is that instead of protecting vigilance, expertise, and sensitivity to emerging failure, the processes being implemented may actually be subtly eroding the very behavioral foundations on which reliable threat detection depends.
The accountability structures that underpin decision-making and HRO are not value-neutral conduits for oversight. In research with colleagues, I have shown that they instead send unspoken relational signals — about who is trusted, what kind of reasoning is valued, and when independent thinking is or is not welcome — in ways that can reshape how employees exercise judgment.
These are accountability signals : implicit messages that governance mechanisms transmit to employees. The danger is that these messages can encourage employees to step back from what their instinct tells them to do. Sometimes these signals produce mindless conformity , where employees default to procedurally defensible decisions rather than exercising independent judgment. At other times, they encourage reckless deviation , where employees stray from compliance procedures to protect their own interests.
The technologists, risk officers, and other leaders designing AI governance today all aim to follow the HRO-like playbook. Because AI technologies and governance are still maturing, there is not yet longitudinal evidence to document the effects of these design choices systematically. But latent behavioral mechanisms like relational signaling are well established in accountability scholarship, and such concepts suggest that as high-stakes organizations rush to implement AI technologies, the most common responsible AI governance principles may be setting companies up for less-secure environments.
Generally speaking, common best practices for AI governance are emerging across regulatory standards and academic discourse. In an ongoing collaboration with legal and policy scholars , we mapped 14 major governance frameworks — across financial services, health care, cybersecurity, and cross-sector AI governance — and found recurring features that center on preserving human oversight. Among the most prominent are designating a human in the loop (to preserve front-line oversight); implementing escalation and override mechanisms (to protect deference to expertise); ensuring documentation and auditability (to maintain sensitivity to operations); and scaling oversight proportionally to risk (to sustain preoccupation with failure where stakes are highest).
On their face, these emerging practices make sense. They are explicitly designed to correct for both “bad models” and “bad users” who may succumb to automation bias on the one hand and complacency on the other. Each practice is well intentioned.
But each one also embeds an implied accountability signal that can undermine the very reliability it was designed to protect. Here, I detail the signals inherent in four of the most recurring practices across AI governance frameworks.
1. Human-in-the-loop mandates send an implicit accountability signal that “your job is to review, not to decide.”
Responsible AI governance commonly assigns formal accountability to a designated individual. Front-line analysts are positioned as reviewers and approvers of AI-generated alerts, with responsibility codified in workflows and checklists. The intent is to preserve human oversight and prevent blame from being displaced onto the system.
In practice, this principle reshapes how analysts understand their role. They become auditors of machine output rather than originators of risk assessment. The distinction matters: Auditing involves evaluating whether an existing conclusion (either the presence or absence of a signal) warrants challenge, whereas judgment requires integrating context, uncertainty, and tacit knowledge to form an independent assessment.
When AI outputs arrive first, the system’s assessment becomes the cognitive anchor. Instead of asking, “What is happening here?” analysts ask, “Is there a reason to contest what the system flagged?” Judgment becomes comparative and exception-focused rather than anticipatory. Equally concerning, when the system does not flag a case, analysts may treat the absence of an alert as evidence that no threat exists, reducing the likelihood that they will proactively investigate potential risks the system might have missed.
These effects are particularly pronounced among less experienced analysts , who are more likely to lack the capabilities to distinguish signal from noise. Without well-developed intuition for contextual nuance, novices are more likely to defer to algorithmic outputs, treating them as authoritative rather than provisional.
Reliability depends on front-line actors who actively imagine failure, question assumptions, and search for weak signals beyond what formal systems surface. As independent sensemaking recedes, expertise is redefined as procedural compliance — undermining a core behavioral foundation of reliable threat detection.
2. Escalation and override protocols send an implicit accountability signal that “judgment belongs above you, not within you.”
Responsible AI governance emphasizes the ability to challenge or override AI recommendations. Organizations operationalize this principle through formal escalation protocols and often require supervisory or committee approval before deviation. The intent is to prevent blind automation and provide procedural safeguards.
In practice, escalation mechanisms are embedded within organizational power hierarchies, particularly in asymmetries between subordinates and superiors. As a result, these protocols often signal that exercising independent judgment at the front line is not fully authorized. Deviating from an AI recommendation becomes an exception that must be justified upward, rather than a routine exercise of professional discretion.
The result Is that some front-line employees refrain from escalation altogether, a dynamic we have seen manifest in clinical and cybersecurity settings. Nurses using Epic-based sepsis-detection tools are typically permitted to override alerts only by escalating their concerns to supervising physicians. Many nurses report refraining from doing so, not because they agree with the system’s recommendations but because escalation itself is burdensome within hierarchical structures, requiring them to interrupt or challenge higher-status actors. Others report disengaging from the alerts entirely and deferring responsibility upward. Security analysts also report frequent disengagement from making the escalation decision: A 2023 survey of 2,000 security analysts found that security operations center teams received an average of 4,484 alerts daily, with 67% going uninvestigated due to alert fatigue and false positives.
In either case, override protocols transform front-line discretion into a request for permission, and judgment migrates upstream. But high reliability depends on deference to the expertise located closest to the problem . When overrides require hierarchical approval, judgment and disputed decisions are deferred to supervisors or committees that often are operationally distant from the situation at hand. As a result, adjudication concentrates among actors less embedded in evolving conditions. They are often least positioned to recognize failure and identify opportunities for process improvement.
3. Documentation and auditability requirements send an implicit accountability signal that “decisions must be explainable to others, not just appropriate in the moment.”
The third principle of governance emphasizes traceability and defensibility. AI-mediated decisions must be logged, explained, and reviewable to support accountability, learning, and transparency for internal and external oversight. This principle is unimpeachable, in theory.
Again, though, there is an important caveat: Documentation requirements subtly shift judgment from real-time sensemaking to post hoc justification. When decisions must be rationalized and reviewed by third parties, actors end up prioritizing what will appear reasonable after the fact over what feels appropriate in the moment. Rather than reasoning forward from the unfolding particulars of a situation, they reason backward from what will be defensible. The burden is not hypothetical: In anti-money-laundering compliance, an estimated 90% to 95% of transaction-monitoring alerts are false positives , yet each must be individually investigated and documented to satisfy audit requirements.
Privileging defensibility over operational fit disproportionately affects recognition-based judgment, a hallmark of expertise in time-pressured, high-stakes environments. Research on naturalistic decision-making shows that humans often rely on rapid, context-sensitive pattern recognition grounded in experience. These judgments are not irrational or impressionistic; they are the product of extensive tacit learning. But because recognition-based judgments can be difficult to articulate ex post and translate into standardized documentation formats, they are easily marginalized in audit-oriented regimes. Over time, front-line users may simply avoid judgments that cannot be easily documented, weakening real-time sensitivity to evolving conditions.
4. Proportionality oversight and risk tiering send an implicit accountability signal that “you should know before the event which decisions will later be judged high stakes.”
Responsible AI governance emphasizes scaling oversight with risk. Higher-stake cases should trigger additional review, while routine decisions are streamlined to preserve efficiency. The intent is to allocate scarce human attention where it matters most.
Tiered oversight regimes place an ex ante burden of foresight on users operating under uncertainty. They must classify decisions as routine or exceptional before outcomes unfold, even though the consequences of a decision often become apparent only retrospectively. Routine cases can escalate rapidly; exceptional cases can resolve benignly.
In practice, users adapt by narrowing their attention to what the system flags as risky, treating “low-stakes” cases as sites of speed rather than vigilance. Uncertainty is simplified into thresholds, eroding vigilance in “routine” work.
High reliability depends on resisting these oversimplified interpretations of risk and maintaining openness to ambiguity in routine work. Proportional oversight can dull vigilance precisely where early-warning signals are most likely to emerge. Rather than continuously monitoring for weak signals or emerging anomalies, users can learn that careful judgment is expected only when a case crosses a predefined threshold.
Responsible AI governance, if set up with principles that send the wrong signals to the humans who oversee AI, can inadvertently erode high reliability. Solutions should focus on recalibrating accountability signals, not abandoning oversight. (Deploying AI systems with no governance at all makes no sense.) The disasters that prompted the development of HRO-inspired frameworks were real, and the instinct to ensure human oversight was correct.
Thus, the challenge is not to add more controls but to design operating principles that restore rather than displace judgment within professional roles. Each of these governance practices can be recalibrated to send a different signal.
Change “Review the system’s output” to “Form your own view, then compare.” Human-in-the-loop mandates position employees as auditors of AI conclusions. To counteract this, organizations can require users to articulate an initial assessment before consulting AI outputs. This preserves sensemaking as a generative act and reduces the risk of false negatives by encouraging analysts to identify potential threats the system may have missed. The mechanism need not be burdensome; a brief, structured judgment (even a single sentence documenting the analyst’s initial read) fundamentally changes the cognitive starting point. The analyst begins from their own reading of the situation, not the machine’s. This approach mirrors a well-established debiasing principle established in intelligence analysis, of forming independent judgments before exposure to potentially anchoring information.
Change “Escalate to override” to “Consult to decide.” Escalation protocols channel disagreement upward through hierarchies, transforming front-line discretion into a request for permission. The alternative is lateral: peer-based consultation and review that reduce the relational costs of raising concerns while preserving deference to local expertise. When a nurse can consult a colleague rather than escalate to a supervising physician, or when a cybersecurity analyst can present a concern to a peer for a second opinion before it becomes a formal override, judgment stays close to the problem. The critical shift is structural: Disagreement with the AI system should trigger a conversation, not an approval chain.
Change “Justify your reasoning” to “Name your basis.” Documentation requirements sideline recognition-based expertise because standard formats privilege linear, rule-based justification. The problem is not documentation itself but documentation formats that treat only explicit reasoning as legitimate.
Organizations can redesign case templates to include a “confidence and basis” field that distinguishes among different grounds for a decision: “I followed the established rule,” “This matches a pattern I’ve seen before,” or “Something appears anomalous but I cannot yet articulate why.” This legitimizes tacit expertise within the audit trail rather than excluding it. Anti-money-laundering compliance units, for instance, could allow analysts to flag cases as “pattern-informed dismissals” rather than requiring a fully articulated justification for every alert closed, thereby preserving traceability while respecting how expert judgment operates in practice.
Change “Reserve judgment for flagged cases” to “Question agreement, not just deviation.” Proportional oversight concentrates vigilance on cases the system classifies as high risk, dulling people’s attention to everything else. The counterweight is for conformity to be scrutinized as carefully as deviation is. In most governance regimes, agreement with AI systems is treated as unremarkable — deviation is what triggers review. But unquestioned conformity can be as consequential as an override.
Periodically asking professionals to briefly reflect on why they agree with a routine classification reframes alignment as an active judgment rather than the absence of one. It also functions as a calibration mechanism: If analysts cannot articulate why they concur, the organization gains an early signal that vigilance is narrowing. Security operations center teams could implement this through random “agreement audits” on a small percentage of cases classified as routine, ensuring that low-risk designations receive periodic scrutiny without creating unsustainable review burdens.
AI governance leaders typically worry about “bad models” or “bad users.” These emphases are incomplete. The midlayer of accountability design choices can subtly reshape how employees interpret the value of their judgment and the terms of their authority, through implicit relational signaling — potentially eroding the very vigilance these structures are meant to protect.
This AI governance paradox warrants attention now, while governance architectures are still being designed and can still be recalibrated.
For leaders deploying AI in threat-detection environments, the imperative is clear: Audit your accountability structures with the same rigor you apply to your algorithms. The signals your governance sends to the people operating the system may matter as much as the system you ultimately procure.