Geometry-Enhanced Portion Estimation for Multimodal LLMs
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The authors study how large multimodal language models (MLLMs) can identify foods in photos but struggle to estimate portion sizes. They create a small extra network that works with a frozen image model and uses information like food names and bounding boxes to better guess portions without needing depth sensors or retraining the big model. Tested on real-world data, their method reduces portion estimation errors by about one-third compared to using the MLLMs alone and beats existing image-only methods. This shows a practical way to improve dietary assessments from photos without making big changes to the original AI models.
Authors
Lin Liao, Peng Li
Abstract
Image-based dietary assessment promises to replace costly, bias-prone manual recalls, but portion estimation remains a major blocker. Multimodal LLMs (MLLMs) recognize a wide range of foods zero-shot in uncontrolled photos, yet they are weak at portion estimation -- a gap we measure across the current frontier (Gemini, GPT, and Claude flagships alike). We present a method that enhances a frozen, commercial MLLM with an accurate portion head: a small geometry-enhanced network on a frozen DINOv2 backbone with a structured softmax-ownership volume, consuming the MLLM's per-food name, bounding box, and density range -- no depth sensor, no MLLM fine-tuning. Evaluated fully open-vocabulary on three real-world benchmarks, the head cuts per-food portion error by 33-41% relative to the MLLM alone, outperforms every flagship MLLM's direct estimates, and surpasses each benchmark's originally published image-only model at its own reported metric.