Trustworthy AI needs more than a better model. It needs a better system.
Capgemini ·
As AI moves into the physical world and more consequential decisions, trust will depend on what surrounds the model as much as what happens inside it. The post Trustworthy AI needs more than a better model. It needs a better system. appeared first on Capgemini .
The debate around artificial intelligence has been dominated by capability. Models have become larger, faster and more powerful, taking on tasks that would have seemed beyond the reach of software only a few years ago.
But capability is only part of the equation. As AI moves from experimentation into real-world deployment, organizations face a harder question: how do you rely on systems whose outputs are inherently probabilistic when the consequences matter?
That challenge runs through a recent New Era of Engineering podcast conversation between Daniela Rus, Director of the MIT Computer Science and Artificial Intelligence Laboratory, and Mark Roberts, Head of the Capgemini Global AI Futures Lab and CTO of Applied Sciences at Capgemini Engineering. Their discussion points to an important shift: from asking whether a model can perform a task to asking whether the whole system can be trusted.
That matters when AI influences a medical decision, controls a vehicle or robot, or supports a critical business judgment. Competence is essential. But so are predictability and honesty: does the system know what it knows, what it does not know and how confident it should be?
This is where the idea of “AI humility” becomes important. An AI system that admits uncertainty may appear less capable than one that always produces an answer. In practice, it can be more trustworthy. If a model recognizes the limits of its competence, a decision can be escalated, a fallback invoked or a task routed elsewhere.
The technology to do this is already emerging. Themis AI, for example, is developing ways to identify uncertainty and errors in AI models and understand what data could improve them. Yet the commercial focus remains heavily weighted toward model performance and capability. The tools for understanding uncertainty may be advancing faster than the appetite to use them.
That same shift in thinking applies to safety. Large AI models are probabilistic, so proving that every possible output will always be correct is unrealistic. Assurance therefore has to extend beyond the model itself.
Research between MIT and Capgemini is exploring one approach: placing mathematically specified safety layers around learned AI components. The model provides performance and adaptability, while the surrounding layer monitors its proposed actions and keeps the system within defined boundaries.
The distinction is important. Many AI “guardrails” still sit within the model itself, effectively relying on the system generating an output to police its own behavior. External assurance provides a different level of control.
Commercial aviation offers a useful parallel. People routinely board aircraft heavily dependent on software because decades of engineering, testing, certification, incident analysis and redundancy have created confidence in the wider system. Passengers do not inspect the autopilot or personally verify the pilots. They trust an infrastructure of regulators, engineering processes and accountable organizations to do that for them.
AI has yet to develop an equivalent system of trust at the same level of maturity.
For organizations, this means moving beyond a narrow focus on model performance. They need to define what acceptable behavior looks like for each application: the accuracy required, the limits that matter, when human intervention is needed and what happens when confidence falls below an agreed threshold.
Those decisions also need to be revisited. AI systems change, the data they encounter changes and the environments in which they operate change. Trust cannot be treated as a launch-stage approval. It has to become an ongoing discipline of measurement, monitoring and accountability.
That becomes even more important as AI grows more distributed. The future will not consist solely of increasingly powerful models in large data centers. Smaller, more specialized models will operate on devices, vehicles and robots alongside larger cloud-based systems. Managing them at scale will require observability, version control, access control and clear accountability.
Smaller models may also help solve the trust problem itself. In one MIT experiment discussed in the podcast, a liquid neural network inspired by the C. elegans worm used just 19 neurons and a few thousand parameters to steer a car within a lane. A conventional approach required around half a million parameters. The smaller system was not just efficient. Its decision-making was easier to examine.
This points toward a broader change in how AI progress is measured. Bigger will remain valuable where scale is needed. But in many applications, the better system may be the one whose behavior can be understood, whose uncertainty can be measured and whose operation can be bounded.
For organizations, the question is therefore no longer simply whether AI can perform a task. It is whether the system around it makes that capability dependable enough to use.
AI has already demonstrated what it can do. The next challenge is engineering the confidence to use it at scale.
Hear Daniela Rus and Mark Roberts explore the ideas and engineering approaches that could make trusted AI possible at scale: Listen to the full New Era of Engineering podcast
The post Trustworthy AI needs more than a better model. It needs a better system. appeared first on Capgemini .