Don’t assume composability in AI safety. Deployments need their own testing.
MLCommons ·
Teams deploy whole systems but cite safety scores from base models. Fine-tunes drift safety in unpredictable directions, and many benchmarks are too noisy to detect the change. This post explains why safety scoring is not composable, and what MLCommons is doing to make it comparable. The post Don’t assume composability in AI safety. Deployments need their own testing. appeared first on MLCommons .
Practitioners know that a deployed system behaves differently from its base model, and that its behavior keeps changing as they change the system prompt, retrieval layers, fine-tuning, and so on. But safety evaluation often assumes the opposite, with teams citing the base model’s safety score from the system card for whatever they’ve done on top of the model. Many in the field end up holding both beliefs at once: expecting behavior to shift when they build on a model, yet basing purchasing and deployment decisions on safety scores as if it stays the same.
System composability is just a fancy term for something all of us grapple with regularly: whether or not a configuration in one context holds up in another. For example, a good home baker knows that “30 minutes at 350°F” in a recipe, or on the box of cake mix, is guidance, not a guarantee. In a different oven or pan (or depending on the humidity in the baking tent that day) the same cake might be done at twenty-five minutes or still too wet at forty, so bakers check it with a toothpick anyway.
AI researchers use the term composability to describe how well the evaluation results for an upstream system carry over to the systems built on it. In classical software this assumption generally held true. In AI safety, with proper testing (more on that below!), it often has not.
Recent research supports practical observations of AI safety not being composable. CDT’s AI Governance Lab and MIT’s Algorithmic Alignment Group compared publicly available medical and legal fine-tunes against their base models. Some fine-tuned models scored as safer than their base models and some as less safe; no single fine-tuning choice predicted which direction a model would move. The amount of modification didn’t predict the amount of change in safety either.
While certain changes can make safety worse relative to the base model (Cf. the admirably clearly titled paper “ Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ”), the opposite sometimes holds. To point to a fairly obvious example, adding runtime guardrails to an open-weight model generally improves safety scores. But how, and by how much, is often anyone’s guess unless you’re properly testing.
Unfortunately, as Solomon Messing showed in a recent paper , many widely used benchmarks are too noisy to be reliably useful in comparisons between pre- and post-deployed systems, shifts over time, or even comparing one system to another. These comparisons only work if the results separate real differences between systems from noise introduced by the evaluation itself, such as which test prompts were selected, how they were worded, and which evaluator graded the responses. If you can’t normalize a test, you can’t be sure whether a change in score means a real change in a system’s safety or simply measurement noise. (Disclosures: Messing was employed by MLCommons when he wrote this paper, and I’m acknowledged in the paper.)
MLCommons’ safety benchmarks are built to be comparable.
Because teams deploy whole systems, safety testing has to cover the whole system. Our new Candela benchmarking-as-a-service product brings AILuminate measurement to the production-deployed assembly enterprises need to test — the base model together with its system prompt, safeguards, retrieval, tools, and filters. It’s a private test, right in your own tenant.
Changes we just announced to our evaluator, such as using classifiers instead of LLMs-as-judges, and introducing Item Response Theory (IRT) methodology for benchmark content development, allow us to reliably generate defensible margins of error. Our next releases of AILuminate Safety and Jailbreak in November will include margins of error that account for variance across the evaluation pipeline, and we will be transparent about exactly which sources of variance the margins cover. To our knowledge, no other widely used AI safety benchmark reports margins of error this way. Our knowledge here is fairly broad, because our AIRR working groups draw participants from more than 80 organizations, including AI developers, consultancies, universities, and civil-society groups, and many of those participants build and run benchmarks of their own.
With margins of error and comparable evaluations, a deployer can more reliably tell whether a change to its system caused a safety score to change.
Today, the defensible position is based on observation: that evaluation results don’t carry over from a base model to a deployed system given certain changes to an AI system deployment. But when measurement uncertainty is known, researchers can test which modifications preserve a result, which modifications break it, and whether either outcome can be predicted in advance.
The answers would help even teams that already test their own systems. If some kinds of change are known to preserve a certain safety score, they could make better informed choices about which changes to deploy untested, which to give a cheaper smoke test, and which to fully scrutinize with an annotated report. If a base model’s score predicts how systems built on it will score, teams could use upstream scores to decide which model to build on and know how much change they can introduce to their deployment before those upstream scores cease to apply. Such reliable, comparable scoring would support regulators in drawing better lines between a developer’s obligations and a deployer’s, and ultimately would reduce the measurement and compliance burden for organizations deploying AI systems and their assurers.
Teams deploying AI can test their systems with Candela today. Researchers who want to work on composability, or on measurement error in AI evaluation more broadly, can join the AIRR Analytics Working Group. To get involved, set up your MLCommons Profile . We look forward to seeing you in our Working Group meetings.
The post Don’t assume composability in AI safety. Deployments need their own testing. appeared first on MLCommons .