FigAct turns scientific figures into interactive visual explanations
FigAct: Turning Scientific Figures into Active Canvases for Explanation
Artificial Intelligence
Summary
Scientific figures usually show information visually, but AI models often explain them by turning the images back into text, which makes it hard to connect the explanation to the picture. The authors created FigAct, a system that makes these figures interactive by highlighting parts and adding short spoken notes, helping viewers focus on the important details as if a person is presenting. FigAct uses a smart search method to find figure elements efficiently and was trained to explain accurately and clearly. Tests showed FigAct helps make explanations easier to follow than just text alone.
What this means in practice
- •For scientific communication teams: Enhance presentations by transforming static scientific figures into guided visual explanations to improve audience engagement and understanding.
- •For technical documentation writers: Create interactive visual explanations for complex technical diagrams to help users follow instructions more intuitively.
Authors
Shishi Xiao, Zichao Wang, Alexa Siu, David H. Laidlaw, Jennifer Healey
Abstract
Scientific figures are designed to communicate information visually, yet MLLMs typically explain them by translating their visual content back into text. This requires readers to manually map the resulting explanations back to the figure. Inspired by how people present visual information, we introduce FigAct, a framework that transforms static scientific figures into question-conditioned visual presentations by acting directly on their existing graphical elements. Like a human presenter, FigAct generates a sequence of short narrations, grounds each narration in the corresponding visual evidence, and applies visual actions to guide the viewer's attention. We develop a hierarchical search strategy for efficient element localization, reducing token usage by approximately 40$\times$. We further train FigAct-8B using three task-specific rewards for grounding accuracy, search efficiency, and rendering quality. We further build a human-verified benchmark from figures in real-world scientific papers to evaluate the ability of MLLMs to generate grounded visual explanations. Our results demonstrate the effectiveness of FigAct and show that treating scientific figures as presentation canvases makes explanations clearer and easier to follow.