Hybrid kinematic–visual sensing approach for activity recognition of construction equipment

Abstract

Activity recognition of construction equipment is vital for operational productivity and safety analysis. For automated equipment monitoring, many researchers have developed kinematic or visual sensing approaches and found that the two approaches have their own technical advantages and disadvantages in classifying different types of equipment activities. However, since previous methods adopted only one of kinematic or visual sensing, there is a limitation to fully benefit from both approaches, causing difficulty in monitoring construction equipment precisely. Additionally, despite the great potential of data fusion, the hybrid effects of kinematic–visual sensing are still unclear. To fill such knowledge gaps, this study developed a hybrid kinematic–visual sensing approach and investigated its impacts on the recognition of equipment activities. Specifically, a smartphone was installed inside the equipment's cabin, and kinematic and visual data were collected from its built-in sensors, gyroscopes, accelerometers, and cameras. Total 60-min data were collected, and the data were further split into training (40-min) and testing data (20-min). The data were then used to experiment three different models: kinematic, visual, and hybrid models. In the experiments, the average F-score of the hybrid model was 77.4%, whereas those of kinematic and visual models were 61.7% and 72.4%, respectively. These results indicated that the hybrid sensing could improve the recognition performance and monitor construction equipment better than relying only on sole type of data sources. The findings can contribute to more reliable activity recognition and operation analysis of construction equipment, and provide meaningful insights for future research.