Temporal image analytics for abnormal construction activity identification

Abstract

Abnormal activities on construction jobsites may compromise productivity and pose threat to workers' safety. This paper proposes the analysis of consecutive image sequences for automatic identification of irregular operations and their visualization. The data analytics is composed of four steps: object detection, object tracking, action recognition, and operational analysis. The Faster Region-proposal Convolutional Neural Network (Faster R-CNN) is adapted with transfer learning for detection of workers and pieces of construction equipment on the jobsite, while the Simple Online and Realtime Tracking (SORT) approach is applied for object tracking. A hybrid model integrating CNN and Long Short Term Memory (LSTM) is employed for action recognition. An alternative form of the Crew-balance Chart (CBC), called line chart in which anomalies are pre-screened, is utilized for recognized actions. Validation was carried out with earthmoving operations. The trained Faster R-CNN reached a 73% Average Precision (AP), and the SORT algorithm modified by this work successfully reduced identity switches. Irregular operations in the testing videos were identified, and truck exchanges were filtered. In addition, an activity log was produced with basic information along with starting and ending times of the identified irregular operations. With the line chart and the log provided by the proposed framework, field managers can efficiently identify potential abnormal activities, providing the opportunity for further investigations and adjustments accordingly.