A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture

Ekrem Misimi,Håkon Måløy,Agnar Aamodt

doi:10.1016/j.compag.2019.105087

Ekrem Misimi, Håkon Måløy + Show 1 more

Open Access

https://doi.org/10.1016/j.compag.2019.105087

Copy DOI

Abstract

Recent developments have shown that Deep Learning approaches are well suited for Human Action Recognition. On the other hand, the application of deep learning for action or behaviour recognition in other domains such as animal or livestock is comparatively limited. Action recognition in fish is a particularly challenging task due to specific research challenges such as the lack of distinct poses in fish behavior and the capture of spatio-temporal changes. Action recognition of salmon is valuable in relation to managing and optimizing many aquaculture operations today such as feeding, as one of the most costly operations in aquaculture. Inspired by these application domains and research challenges we introduce a deep video classification network for action recognition of salmon from underwater videos. We propose a Dual-Stream Recurrent Network (DSRN) to automatically capture the spatio-temporal behavior of salmon during swimming. The DSRN combines the spatial and motion-temporal information through the use of a spatial network, a 3D-convolutional motion network and a LSTM recurrent classification network. The DSRN shows an accuracy that is suitable for industrial use in prediction of salmon behavior with a prediction accuracy of 80%, validated on the task of predicting Feeding and NonFeeding behavior in salmon at a real fish farm during production. Our results show that the DSRN architecture has high potential in feeding action recognition for salmon in aquaculture and for applications domains lacking distinct poses and with dynamic spatio-temporal changes.

Highlights

Feeding is an important part of the salmon breeding process
We proposed the Dual-Stream Recurrent Network (DSRN) architecture for spatial temporal salmon action recognition for applications with a lack of distinct poses and dynamic spatio-temporal changes during motion
The underwater video salmon data set is in many ways more challenging than the data sets used for human action recognition due to challenging light conditions, turbidity, the lack of discriminative poses, dynamic and undiscriminating motion patterns

Summary

Introduction

Feeding is an important part of the salmon breeding process. The feed is approximated to account for over half of the total fish farming costs in Norway (Fiskeridirektoratet, 2016). Traditional feeding processes are labor-intensive processes requiring manual observation and maneuvering of cameras within breeding cages. By using fish behavior rather than the amount of feed falling to the bottom of the cage as an indicator for when to stop the feeding process, the system can stop the feeding process at a more appropriate time This can result in a reduced environmental impact and reduced feed costs as less feed is wasted. Deep learning is the use of deep neural networks that use multiple processing layers to learn representations of data with multiple levels of abstraction (LeCun et al, 2015). These methods have shown remarkable improvements in speech recognition, image recognition and machine translation. The aquaculture industry has seen the benefits from such methods through the use of deep learning and Support Vector Machines (SVMs) for segmentation of blood defects in cod fillet (Misimi et al, 2017)

Objectives

Methods

Results

Discussion

Conclusion