UK FCA exec wants to require disclosure of ‘significant AI failures and near-misses’
CSO Online ·
When UK Financial Conduct Authority (FCA) Chief Data Officer Jessica Rusu gave a speech on Monday advocating that UK companies be required to report “significant AI failures and near-misses,” she added to the drumbeat of demands for some kind of regulatory control over enterprise AI deployments. But her speech also highlighted regulator frustration over futile attempts to slow down the seemingly unstoppable agentic and genAI rollouts, as reports of agent misbehavior and the lack of meaningful consequences continue to emerge with alarming frequency. Rusu , speaking to a Bloomberg UK Policy Series audience, said that firms need to invest more in cyber and operational resilience. “Our rules already require firms to report major incidents, and that must extend to significant AI failures and near-misses. As the AI Minister has recognized, near-miss reporting for agentic AI is becoming a national security priority. It is a regulatory priority too,” Rusu said . “It also means thinking differently about assurance. A system that is safe at deployment may behave differently at scale, under stress, or as the models beneath it change. Agentic systems therefore require continuous, conditional assurance.” She added, “the FCA will work with industry, fellow regulators, and frontier labs to develop a trusted framework for agentic finance, including clear approaches to agent identity, delegated authority, accountability and execution.” One of the questions that came out of the speech is what precisely was meant by “significant AI failures and near-misses.” Questions to FCA to clarify their terminology were not answered by publication time. AI failure vs near miss But Pieter Arntz , a malware intelligence researcher at security firm Malwarebytes, offered his own take on what those terms likely mean. “A significant AI failure is one with serious consequences, not simply an incorrect answer. In financial services, that could mean an AI system authorizing substantial fraudulent payments, exposing sensitive customer information, denying customers access to essential financial services, or triggering transactions that threaten market integrity,” Arntz said. “The test should be the scale and severity of the impact, rather than whether the failure looks technically dramatic.” On the other hand, he added, “An AI near-miss is an event that could realistically have caused serious harm, but was caught before that harm occurred. For example, an AI agent might attempt an unauthorized transfer that a separate approval control blocks, or a prompt-injection attack might induce it to disclose sensitive information, only for an outbound security control to stop the disclosure.” However, Arntz warned of the risk to Rusu’s approach, should the final penalties be underwhelming. “There is a risk of this becoming a paper tiger if compliance amounts to filing a report and absorbing a predictable fine,” he said. Warnings from multiple directions UK-based Gartner VP analyst Nader Henein said he is skeptical about the significance of the FCA’s proposal, in part because so many UK enterprises already have to comply with EU regulations, which have started asking companies to make such disclosures. “Financial watchdogs including the Bank of England (BoE), the UAE Central Bank, and the Monetary Authority of Singapore have all issued warnings,” he said. “A few months ago, the European central banks, together with the European Systemic Risk Board, escalated the cyber risk assessment level associated with frontier models to ‘severe,’ warning that simultaneous cyber incidents could impact the financial sector. They acknowledged that AI-driven cyber incidents now threaten core financial sector stability.” Henein said that he is especially skeptical that meaningful changes will occur in the financial sector. “Financial institutions tend to heed the warnings of the regulator that decides whether they can continue to operate, but without clear laws, regulatory oversight, and the obligatory fines, these disjointed statements are not likely to make a difference,” he said. Could ‘incentivize the wrong behavior’ Noah Kenney , principal consultant at Digital 520, added that one risk of this increased visibility requirement is that it could inadvertently incentivize the wrong behavior. “A bank with good monitoring may catch hundreds of attempts by AI agents to act outside their authority,” he said. “A bank with poor monitoring may report zero, because it never saw them. On paper, the safer bank looks worse. A model failing a red-team test is the test working, not a regulatory incident.” However, Frank Dickson , principal analyst at Dickson Research, said that the near-miss approach has the strongest potential to change the conversation. “I love the near-miss idea. It is the most important thing in the speech. It is the part financial firms are least ready for,” he said. But, he stressed, “[Rusu] also warns about reliance on a single model. An outage at one AI supplier that takes down a customer service channel counts, too.” Another concern about reliance on a specific model is the ever-present risk that an AI vendor could change the model without telling all of its customers. “A model can pass testing in March and misbehave in May without anyone at the firm changing a line of code, because the provider changed the model underneath,” Dickson said. “A model version change should get the same scrutiny as any other production change.” This article originally appeared on CIO.com .