Vision and text transformer improves score prediction for visual questions
Vision And Text Transformer For Predicting Answerability On Visual Question Answering
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Sometimes, a question about an image cannot be answered because the question or image lacks clarity. The authors designed a system called VT-Transformer that looks at the picture and question together to give a score indicating how answerable the question is. Instead of just saying yes or no, their system predicts a range of how answerable the question might be. They tested this on a dataset called VizWiz 2020 and found their approach works better than other methods.
What this means in practice
- •For mobile app developers: Improve apps that answer user questions from photos by predicting if a question is likely unanswerable before processing.
- •For customer support teams: Detect when users’ visual queries are unclear or unanswerable to prioritize requests needing clarification.
Tested on one dataset.
Authors
Tung Le, Huy Tien Nguyen, Le Minh Nguyen
Abstract
Answerability on Visual Question Answering is a novel and attractive task to predict answerable scores between images and questions in multi-modal data. Existing works often utilize a binary mapping from visual question answering systems into Answerability. It does not reflect the essence of this problem. Together with our consideration of Answerability in a regression task, we propose VT-Transformer, which exploits visual and textual features through Transformer architecture. Experimental results on VizWiz 2020 dataset show the effectiveness and robustness of VT-Transformer for Answerability on Visual Question Answering when comparing with competitive baselines.