Multimodal verification improves food portion estimation from images
Improving Image-Based Nutrition Estimation Through Multimodal Food-Item Verification and Recovery
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Estimating how much food is on your plate from just one photo can be tricky, especially if some foods are missed or not clearly identified. The authors developed a method that uses large language models combining text and images to double-check which foods are visible and whether the photo areas are good enough to guess portion sizes. They then fix any missing or uncertain parts with a second check, making the final nutrition estimates more accurate. This approach doesn’t need extra training for the specific task, yet it improves the accuracy of identifying and measuring food portions compared to some existing methods.
What this means in practice
- •For mobile health app developers: Integrate enhanced food identification and portion estimation from photos to improve dietary tracking features.$Commercial implications: Enables development of consumer apps offering more accurate nutrition estimates from user food photos.
- •For dietary assessment teams: Use the multimodal verification framework to reduce errors in single-image food intake surveys during nutrition studies.
Authors
Jingbo Yue, Bruce Coburn, Jinge Ma, Jui-Feng Chi, Fengqing Zhu
Abstract
Single-image nutrition estimation can fail silently when visible foods are missed. Even when a food is correctly identified, its proposed region may not support portion estimation. We propose a framework that uses multimodal large language models (MLLMs) to inventory visible foods and separately verify food identity and whether each proposed 2D region supports portion estimation. One whole-image review uses these verification results to identify unresolved gaps and omitted foods, triggering at most one targeted recovery pass. Recovered regions are re-verified without access to the recovery prompt, then reconciled into a final item set for nutrition estimation. The framework requires no task-specific fine-tuning. Matched evaluation on common valid-output samples shows that item-level grounding improves mass accuracy across all tested settings and energy accuracy relative to an adapted retrieval baseline, with item-identity precision and recall also improving, while post-recovery visual coverage is assessed separately at inference time without ground-truth annotations.