Multi layer transformers improve vietnamese visual question answering performance
Fusing Visual and Textual Representations via Multi-layer Fusing Transformers for Vietnamese Visual Question Answering
Computer Vision and Pattern Recognition
Summary
Visual Question Answering (VQA) helps computers answer questions about images in natural language. The authors focused on VQA in Vietnamese, where there has been little research. They created a model called Multi-layer Fusing Transformer that combines image and text information from different processing stages to better understand both. Their experiments show this approach improves performance on a Vietnamese VQA dataset compared to other methods.
What this means in practice
- •For mobile app developers: Build apps for Vietnamese speakers that answer image-based questions more accurately using layered visual and text data fusion.
- •For healthcare data teams: Integrate Vietnamese VQA systems to assist in interpreting medical images with accompanying text queries.
Tested on one dataset.
Authors
Cong Phu Nguyen, Huy Tien Nguyen, Tung Le
Abstract
In recent decades, artificial intelligence has made significant progress in understanding and interacting with images. One of the important applications of this technology is Visual Question Answering (VQA), a research field that requires computers to understand and answer questions about images in a natural manner. Despite extensive research and development in VQA for English, there have been very few similar efforts made for other languages, especially Vietnamese. This gap presents a significant challenge and opportunity for the advancement of VQA technology in the Vietnamese language context. By bridging this gap, the field of Vietnamese VQA not only enriches the diversity of research in artificial intelligence but also enables practical applications in various domains, such as education, healthcare, and entertainment, catering to Vietnamese-speaking populations worldwide. Thus, the exploration and development of Vietnamese VQA systems hold immense potential for advancing both research and practical applications in the intersection of computer vision and natural language processing. In this paper, we propose a Multi-layer Fusing Transformer model utilizing a cross attention module to combine multiple modality features of images and texts from different layers in an aggregated representation. Our architecture allows us extract information from low level to high level. Through detailed experiments and ablation studies, our model achieves promising results against the competitive baselines in ViVQA dataset for Vietnamese language.