Multimodal continual learning improves leaf counting accuracy

MCLC-NET: Multimodal Continual Learning for Leaf Counting

Computer Vision and Pattern RecognitionMachine Learning

Summary

Counting leaves on plants helps farmers track growth and predict harvests, but it can be tricky when leaves overlap or lighting changes. The authors created a new method called MCLC-NET that uses different types of images—including normal color photos, depth, and thermal pictures—to get better leaf counts. Their method learns from data collected over time and keeps important information from earlier data to improve its accuracy. They also introduced a new dataset with these multiple image types collected in real farm environments. Tests showed their method worked better than previous ones across different ways of organizing the data.

What this means in practice

  • For agriculture technology companies: Develop systems to accurately count leaves on crops using multiple image types collected over time to monitor plant growth under varying conditions.$Commercial implications: Enables precision agriculture tools that improve crop monitoring and yield estimation by integrating multimodal continual learning for leaf counting.
  • For farm management teams: Use improved leaf counting methods that handle evolving data streams to better track plant development and inform management decisions.

Authors

Ruchi Bhatt, Pratibha Kumari, Shreya Bansal, Vedant Agnihotri, Dwarikanath Mahapatra, Mukesh Saini

Abstract

Leaf counting is an important task in plant phenotyping for monitoring plant growth and estimating crop yield. Most existing methods rely on RGB images, but their performance is often affected by occlusion, lighting variations, and other real-world challenges. Additional modalities, such as depth and thermal images, can provide useful complementary information. However, multimodal leaf counting remains underexplored. Also, many existing methods assume that all training data are available simultaneously, which is impractical in real agricultural settings, where data is collected over time from multiple sources. To address these challenges, we propose MCLC-NET, a multimodal continual learning framework for leaf counting. It learns tasks sequentially using a memory-based strategy with a memory buffer to retain important samples from previous tasks. We also introduce MMLC, a real-world multimodal leaf-counting dataset designed for a domain incremental scenario (DIS) in CL. It contains RGB, depth, and thermal images collected across different crop types under varying environmental conditions, arranged in three orderings: crop-wise, time-wise, and mixed. Experimental results, averaged over three random seeds, demonstrate that MCLC-NET consistently outperforms existing methods across all three task orderings, achieving the lowest AMSE of 0.675$\pm$0.027, 0.542$\pm$0.069, and 0.745$\pm$0.057, respectively.