Temporal Shift and Spatial Attention-Based Two-Stream Network for Traffic Risk Assessment

Abstract

On-board vision based traffic risk assessment is a challenging task for intelligent driving systems, which has some special challenges including spatial-temporal feature extraction, different judgements of risks, real-time requirement, lacking data, etc. To overcome these difficulties, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Temporal Shift and Spatial Attention based Two-stream Network</i> (TSSAT-Net) for on-board vision based traffic risk assessment. Firstly, we build new judgement measures that integrate actual driving experience and scenario complexity, and release an on-board vision based traffic risk assessment dataset. Secondly, a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">weighted Temporal Shift Module</i> (weighted-TSM) based two-stream network is proposed; unlike previous methods that rely on complex and time-consuming 3D CNN or LSTM calculation, the proposed two-stream network can effectively extract spatial-temporal features using a weighted temporal shift mechanism with just 2D CNN calculation requirement. Thirdly, a spatial and channel attention mechanism is proposed to make the TSSAT-Net more focus on the features closely related to traffic risks, avoiding redundant information in complex traffic scenarios. Experiments based on the released comprehensive dataset show that our method achieves the state-of-the-art classification accuracy in real-time.