Metrics assess visual analytics chat agents without reference answers
Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
Human-Computer InteractionArtificial Intelligence
Summary
It is hard to check how well computer programs answer questions with both pictures and text because there are many correct answers and no easy way to compare them. The authors created a new set of tools called Lexara-RF that checks these programs’ answers by looking at the question, the data, and the answer itself, without needing examples of right answers. These tools use rules about good picture design and clear communication to score answers on consistency and correctness. When tested, their scoring matched human judgments as well as methods that rely on example answers and did better than simpler comparison methods.
What this means in practice
- •For software developers: Integrate reference-free metrics to automatically check chatbots generating visual data insights for consistency and design quality.
- •For product managers: Assess conversational AI tools for data visualization without costly creation of benchmark answer sets.
Authors
Srishti Palani, Vidya Setlur
Abstract
Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.