What Could Go Wrong When Agents Handle Your Sensitive Data? A New Taxonomy Answers.
MLCommons ·
Agents create privacy risks chatbots do not. The MLCommons Privacy Working Group's new taxonomy maps five areas of agentic data protection risk - and v0.1 is open for feedback. The post What Could Go Wrong When Agents Handle Your Sensitive Data? A New Taxonomy Answers. appeared first on MLCommons .
MLCommons Privacy and Confidentiality Working Group: Andrew Gruen, Kristie Chon Flynn, and Vinh Nguyen
The MLCommons AI Risk & Reliability (AIRR) working group starts from a simple idea: define the behavior you want from an AI system, then measure how reliably the system delivers it. Deployers need that measurement to manage both risk and cost. By turning that visibility into higher standards across the industry, we can build the trust required to grow the industry and protect society. In April, we published the AI Reliability Map , which lays out the rules a system should follow (functionality, data protections, product safety, frontier safety, psychosocial limits) and the circumstances we test them under (normal use and under attack).
The Reliability Map highlights the role that data protection practices play in reliability. As the Privacy Working Group has gotten started, we have chosen to focus explicitly on how to evaluate the reliability of an agent handling personal data. But before anyone can measure whether an agent handles personal data reliably, we have to agree on which privacy risks are specific to agents.
The scope of data protection risks posed by a chatbot working on its own is – for the most part – limited to the individual user interacting with that chatbot. For an agent, however, the potential scope of data protection risks is vast. To put some structure around the risks that are unique to an agent and how it engages in privacy-related activity, particularly data minimization, we followed the first step in our community-based benchmark development process: we built a taxonomy .
Today we are releasing the v0.1 of the Agent Privacy Risk Taxonomy.
The taxonomy focuses on five areas of data protection risk in agentic systems, drawing on the expertise across industry, academia, and civil society. We assessed known privacy and security incidents and existing risk management frameworks. The five areas are:
We encourage our industry partners to integrate this taxonomy into their AI development and deployment, from pretraining all the way to deployment monitoring. Our work will help reduce privacy violations and risks, improve privacy and security protections, reduce potential liabilities, and reduce harm to individuals and organizations.
v0.1 does not attempt to rank the risks. Capability and context matter a lot here: a bank’s customer service agent and a personal assistant with access to your inbox face different risks in a different order. So the next piece of work is working with deployers to figure out which of these risks matter most in which kinds of deployment. And, ideally, which risks are most universal. Our goal will be to produce measurements that can be used in the broadest set of use cases.
We plan to convene our diverse team in multiple engagements with large-scale deployers, in consultation with AIRR teammates, to prioritize the most critical risks. We will then identify indicators to detect those specific risks, pilot key benchmarks to measure them, and best practices for mitigating them. At the same time, we will continue to iterate on the agent privacy risk taxonomy. We plan to complete this effort by Q1 2027.
From there, we build. Some of these risks are behaviors we can test before deployment, which is where the Reliability Map starts. Does the agent pull more data than the task needs? Does it pass a deletion request downstream? Other risks, particularly in accountability and governance, are about the organization around the agent, so a benchmark will not capture them. Sorting out which is which is part of the job. The goal is measurement tools that tell deployers how their agents perform overall and more specifically in the areas that matter most in their context.
This is a v0.1, and we expect it to change. We want feedback from industry practitioners, academic researchers, civil society organizations, and regulators.
The post What Could Go Wrong When Agents Handle Your Sensitive Data? A New Taxonomy Answers. appeared first on MLCommons .