HMMF Net improves fish feeding intensity assessment on edge devices
HMMF-Net: A Lightweight Hierarchical Multi-Frame Multimodal Fusion Network for Accurate Fish Feeding Intensity Assessment
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Measuring how much fish eat is important for saving food and growing fish better. The authors created a new computer model called HMMF-Net that looks at both pictures and sounds of fish feeding to make better guesses. This model is small and fast enough to run on simple devices, unlike older methods. Their tests show it is more accurate than using just pictures and works well on real hardware.
What this means in practice
- •For aquaculture managers: Monitor fish feeding intensity in real time on low-cost devices to optimize feed use and reduce waste.
- •For embedded system developers: Deploy efficient multimodal models that combine audio and video for real-time monitoring on edge devices with limited computing power.
Authors
Phi-Hung Hoang, Van-Manh Tran, Nam-Thuan Trinh, Thi-Thu-Hong Phan
Abstract
Automated fish feeding intensity assessment (FFIA) is important for precision aquaculture and feed cost optimization. However, unimodal methods are vulnerable to acoustic noise and visual disturbances, while existing multimodal frameworks often rely on computationally intensive 3D convolutions or Transformers, limiting their use on edge devices. To address these limitations, we propose HMMF-Net, a lightweight multimodal architecture that combines early multi-frame visual encoding and multi-axis acoustic modeling through a hierarchical bidirectional messenger cross-modal fusion (BMCF) mechanism. The visual branch captures temporal variations in feeding behavior by stacking temporally distant frames as a multi-channel input to an adapted 2D EfficientNetB0 backbone. The acoustic branch employs adaptive multi-axis residual units (AMARU) to extract temporal, frequency, and local time-frequency representations from log-mel spectrograms. BMCF progressively exchanges information between the two modalities at intermediate representation levels through compact messenger tokens, reducing the computational overhead of dense cross-modal attention. Evaluated on the AV-FFIA dataset using five-fold cross-validation, HMMF-Net achieves 96.77% accuracy and 96.88% F1-score on the held-out test folds, with 3.987M parameters and 1.002 GFLOPs. Compared with the best-performing visual configuration (95.49% accuracy), HMMF-Net reduces the classification error by 28.38%. Deployment on an NVIDIA Jetson Orin Nano using TensorRT with FP32 precision maintains comparable accuracy, with an inference time of 11.54 ms per 2-second sample and a model size of 18.81 MB. These results demonstrate that HMMF-Net combines effective multimodal representation learning with computational efficiency, enabling accurate and practical FFIA on resource-constrained edge platforms.