Image-based screening improves South African food-package warnings
Region-Guided Food-Package Evidence Extraction and Abstention-Aware Draft R3337 Screening with Selective VLM Review
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Checking food packages for warning labels in South Africa requires carefully reading nutrition facts, ingredients, and product details from photos. The authors designed a system that first finds relevant parts of the package, extracts the information using text recognition, and checks if warnings apply using clear rules. If some information is missing or unclear, the system either asks for human review or uses an advanced AI model to take another look. Their approach improved how many packages could get automatic decisions while keeping accuracy high.
What this means in practice
- •For food safety teams: Automate review of food-package photos for compliance with South Africa's draft R3337 warning criteria using a combination of detection, OCR, and selective AI review.
- •For regulatory compliance officers: Provide partial automation of complex food-package warning assessments, reducing manual workload while maintaining high accuracy and handling uncertain cases via human or AI review.
Authors
Mayimunah Nagayi, Alice Scaria Khan, Tamryn Frank, Rina Swart, Clement Nyirenda
Abstract
Applying South Africa's draft R3337 warning criteria to food-package photographs requires nutrition, ingredient and product-form evidence. This study presents a region-guided evidence extraction and abstention-aware screening workflow for draft R3337. A three-class detector localises nutrition information panels, ingredient lists and product-identity regions, after which PaddleOCR and structured extraction recover nutrient values, ingredient evidence and product-form information. Recovered evidence is aggregated and assessed using deterministic draft R3337 rules. When evidence is incomplete, uncertain or conflicting, the workflow abstains through Review or Insufficient evidence rather than forcing an automatic outcome. For qualifying unresolved cases, a selective vision-language model (VLM) review stage using Qwen3-VL 8B Instruct reassesses unresolved evidence. Accepted evidence is merged and the same deterministic rules are reapplied. The workflow was applied to 442 products represented by 3,929 images. Evaluation used a manual validation set of 65 products. Before selective VLM review, 31 of 60 decidable products received automatic outcomes and 30 were correct, giving 51.7% coverage and 96.8% accuracy among automatic outcomes (selective accuracy). Selective VLM review resolved eight additional products, increasing coverage to 65.0%. Of 39 automatic outcomes, 38 were correct, giving 97.4% selective accuracy. The single false warning was retained from the rules-only workflow, while no unsafe warning omissions occurred after selective review. Evidence-level evaluation nevertheless identified incorrect VLM-recovered nutrition, ingredient and product-form evidence. These findings show that selective VLM review can recover unresolved evidence, while deterministic screening and explicit abstention remain necessary for reliable image-based draft R3337 screening.