Fieldguiderag improves species recognition using photos in knowledge graphs
FieldGuideRAG: Image-Grounded GraphRAG for Fine-Grained Recognition with Vision-Language Models
Computer Vision and Pattern RecognitionInformation Retrieval
Summary
Identifying very similar species can be tricky because their names or descriptions don’t show what makes them different visually. The authors created FieldGuideRAG, a system that uses pictures stored in a network of species info, and compares a new species photo to these stored photos with a vision-language model. This method helps the system pick the right species better than just using names or text, and it can also answer questions about the species’ habits or range. It even lets the system know when a species isn’t in the database.
What this means in practice
- •For wildlife conservation teams: Improve accuracy of species identification in field data by comparing photos directly on knowledge graph nodes rather than relying on names alone.
- •For mobile app developers: Create apps for nature enthusiasts that identify species from photos and offer detailed ecological facts by verifying images against a graph of real photographs.$Commercial implications: Enables commercial nature identification apps that provide fine-grained recognition and ecological details, outperforming text-based retrieval.
Authors
Jiacheng Ding, Yanliang Qi, Cong Guo, Peihua Mai
Abstract
We present FieldGuideRAG, a training-free GraphRAG method in which a vision-language model, or VLM, recognizes fine-grained species by comparing the query with real photographs stored on the nodes of a knowledge graph. Multimodal GraphRAG systems describe each entity in text, and text conveys little of the visual detail that separates look-alike species. Our graph stores one photograph on every species node and links each species to its relatives, its geographic range and its ecological facts. A frozen VLM verifies the query against the photograph of each candidate node. FieldGuideRAG fuses the verification scores with the range edge and answers questions about the organism from the fact edges of the verified species. On CUB-200, a Wikidata graph and four iNaturalist domains, the node photograph improves verification over the species name on every graph, while text descriptions add little. Verification of the candidate pool exceeds a SigLIP retriever by up to 16 points, the range edge adds 7 to 19 points on iNaturalist, and answering from fact edges is correct on up to 89.5% of 28,820 ecological questions. The verification scores also detect queries whose species is missing from the graph. Graphs, prompts and code will be released.