Our mission, updated: standardize AI measurement and enable informed decisions
MLCommons ·
Peter Mattson, President, on MLCommons' updated mission - and the changes ahead The post Our mission, updated: standardize AI measurement and enable informed decisions appeared first on MLCommons .
Over the last eight years, we have seen tremendous growth and change within the AI industry, and here at MLCommons. None of that growth would have been possible without you all, our members.
Today, we are updating and clarifying the MLCommons mission to reflect what has always been our core focus:
Standardize AI measurement and enable informed decisions.
Measurement is fundamental to responsibly unlocking the tremendous potential of AI. These systems are the ultimate black boxes: they use intricate patterns across billions of numbers to generate sophisticated behavior. We have only vague, unsatisfying explanations of how AI systems work, and they are becoming more complex every day. Not only that, but training and operating AI systems is expensive and power-hungry. Better measurements enable us to empirically characterize the behavior of AI systems and their operational costs.
If a characteristic of AI hardware, software, or data matters to the value delivered – be it speed, power, reliability, or quality – we aim to measure it. In doing so, we support informed decisions regarding the AI systems available today, enable cost-benefit tradeoffs, provide a north star for innovation, and support risk management such as assurance and insurance. Our work touches and benefits everyone involved in the AI industry – academic researchers, hardware and software vendors, deployers, end users, and even policymakers.
We do this because it de-risks and grows the AI market.
We specifically aim to develop and operate standardized measurements in the form of industrial-quality benchmarks, which combine detailed protocols with artifacts such as test harnesses, held-out test data sets, and evaluation models. Given the complexity and (in many cases) subjectivity of AI testing, comparable measurements cannot be achieved by multiple implementations built according to a paper standard. Further, many of the artifacts must be technically and legally protected to avoid overfitting, and the measurements must evolve to keep pace with the technology – all of which requires a trusted, sustainably operating organization.
We exist to meet these independent measurement needs and, in doing so, provide an essential service that supports the broader work of standards organizations such as ISO, Appia, CoSAI, and ADA.
Our work is to define, build, operate, certify, explain, evolve, and connect our benchmarks. Each of these activities is vital to our mission:
In the coming days and months, we will make organizational changes to better support this vital mission. These changes will include:
We welcome your ideas and feedback as we undertake these changes. We will continue to evolve as the field progresses, and new areas such as AI data science advance. MLCommons is your organization, and we aim to deliver maximum value to you, our members.
What we do would not be possible without your involvement and support. Thank you. We look forward to partnering with you to measure AI and inform the growth of this amazing industry and its thoughtful adoption by society.
Kind regards,
Peter
The post Our mission, updated: standardize AI measurement and enable informed decisions appeared first on MLCommons .