Vision language models better identify agriculture problems with rubric guidance
Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap
Computer Vision and Pattern Recognition
Summary
Vision-language models struggle to identify agricultural issues like diseases and pests without extra help. The authors found that these models actually recognize important visual features but don’t always connect them to the right agricultural knowledge. By giving the models a clear set of guidelines to check their guesses, the team improved their accuracy significantly, showing these models know more than they initially appear to. However, the confidence score the system uses doesn’t reliably tell when the model is correct.
What this means in practice
- •For agriculture technology developers: Improve AI tools for detecting crop diseases and pests by integrating rubric-guided reasoning in vision-language systems to increase identification accuracy.
- •For environmental monitoring teams: Use rubric-grounded verification methods to enhance automated recognition of plant health and species in ecological assessments.
Authors
Earl Ranario, Jared Smith, Lars Lundqvist, Urmil Jatin Chandarana
Abstract
Vision-language models (VLMs) show promise for agricultural classification, but zero-shot performance on disease, pest, damage, quality, and species identification remains poor, and it is unclear whether this reflects weak visual features or a failure to connect them to domain knowledge. We build a benchmark of 116 datasets, 834 classes, and 8,324 images spanning these tasks to isolate where the gap arises. Linear probing shows VLM vision encoders already encode agricultural features nearly as separable as a self-supervised DINOv3 baseline, ruling out weak visual representations as the primary bottleneck. Conditioning each model on an oracle reference description (an upper bound on its parametric knowledge) closes most of the gap left by an unaided lower bound, showing VLMs already know more about agriculture than they show. To close this gap without an oracle description at inference time, we structure test-time reasoning around a fixed, per-task diagnostic rubric: the model generates $K$ candidate responses and a Probabilistic Pivot Tournament (PPT) verifier, scored pairwise against the rubric, selects the best one. This nearly doubles judged F1 over the lower bound and matches or exceeds the upper bound on several tasks, notably pushing Gemma 4 E4B-it's disease F1 to 0.71, above its own upper bound of 0.60. However, the verifier's letter-scale confidence score has the opposite of its intended effect: filtering to its most confident predictions does not improve accuracy and correlates negatively with correctness across every model and pool size tested, so the score cannot serve as a measure of predictive uncertainty, and most of the observed gain likely comes from rubric-grounded generation rather than pairwise verification.