Long video-based action segmentation for earthmoving excavators using improved Temporal Convolutional Network models

Abstract

Earthwork excavator, as an all-terrain and high-efficiency earthwork excavation equipment, has been widely used in earthwork sites. It is very necessary to analyze the work of earthmoving excavator by means of machine vision. In this paper, the action segmentation method based on long video was applied to the analysis and recogniton of the excavator’s action, and compared with other two current best action segmentation models using the real construction site video. Firstly, the sequence features of the excavator’s work video obtained at the cnstruction site was extracted through 3D convolution method, and then two differernt networks with the extracted sequence features were trained and tested. The experimental results showed that the average frame accuracy of MS-TCN model and ASRF model in excavator action segmentation were 82.6490% and 86.1042% respectively. However, for the recognition task under different working environment, the performance of the two models is quite different. The experimental results manifest that the motion segmentation model based on long video reached good results in excavator motion recognition in earthmoving operation. And it is helpful to analyze the long video working behavior sequence of excavator. This research contributes to the identification of critical elements that explains serial action and to the development of a new application scenario for vision-based behavior segmentation network. Additionally, the results of this study were helpful to automatically analyze the working efficiency and monitor the productivity of earthmoving excavators. Using this kind of data-driven decision can improve the work efficiency of earthmoving excavator and promote the project progress.