AI summaryⓘ
The authors identify a problem where existing segmentation models using multimodal large language models (MLLMs) struggle to recognize new or recently emerged things because they lack updated knowledge. They introduce a new task called NEST to focus on these novel and emerging entities and create a benchmark dataset from news to test this. To improve performance, they develop ROSE, a framework that helps MLLMs using real-time internet retrieval, enhanced text prompts, and images from the web to better understand new entities. They also include a component to decide when to search the web to save time. Their tests show ROSE greatly improves segmentation accuracy compared to previous methods.
Multimodal Large Language ModelsSegmentationNovel EntitiesEmerging EntitiesBenchmark DatasetInternet RetrievalPrompt EnhancementVision and LanguageReal-time InformationgIoU
Authors
Song Tang, Guangquan Jie, Henghui Ding, Yu-Gang Jiang
Abstract
Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.