Framework links trust measures for large language and multimodal AI
A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems
Artificial Intelligence
Summary
It can be hard to know if smart AI systems like chatbots and image readers are trustworthy just by their scores. The authors created a framework that looks at different ways to check trust across various types of AI. They group these checks into things like skill, safety, fairness, and transparency. The framework also helps make sense of the results so people building and watching AI can understand where the systems work well or fail. Next, it needs to be tested in real-world uses.
What this means in practice
- •For ai system developers: Assess and improve safety and fairness of language and multimodal AI before deployment using a unified trustworthiness framework.
- •For ai compliance teams: Integrate technical evaluation results with regulatory standards and governance requirements to facilitate oversight of AI products.
Tested on one dataset.
Authors
Shaina Raza, Ahmed Y. Radwan, Imran Liaquat, Kathryn Hume
Abstract
Benchmark scores alone provide an incomplete basis for assessing the trustworthiness of modern artificial intelligence systems. Large language models (LLMs), agentic systems, and multimodal models (MLLMs) require different forms of assessment, yet their evaluation evidence must remain interpretable for development and oversight. We propose a unified framework that connects output-level, trajectory-level, and cross-modal assessment through eight trustworthiness dimensions: capability, robustness, safety, fairness, transparency, governance, oversight, and efficiency. The framework preserves system-specific metrics while mapping native measurements to common performance bands, accompanied by uncertainty estimates and traceable evidence. A meta-evaluation layer examines the validity, reliability, and reproducibility of the evaluation itself. Multidimensional profiles expose strengths and weaknesses, while safety-critical overrides prevent aggregate scores from masking critical failures. Mappings to governance frameworks, international standards, and European Union regulatory requirements connect technical assessment with oversight needs. The framework provides a structured basis for assessing both system performance and the credibility of the evidence supporting it, with empirical validation across deployment contexts remaining an essential next step.