Multi-class US traffic signs 3D recognition and localization via image-based point cloud model using color candidate extraction and texture-based recognition

Abstract

Continuous condition monitoring and inspection of traffic signs are essential to ensure that safety and performance criteria are met. The use of 3D point cloud modeling by the construction industry has been significantly increased in recent years especially for recording the as-is conditions of facilities. The high-precision and dense 3D point clouds generated by photogrammetry can facilitate the process of asset condition assessment. This paper presents an automated computer-vision based method that detects, classifies, and localizes traffic signs via street-level image-based 3D point cloud models. The proposed pipeline integrates 3D object detection algorithm. An improved Structure-from-Motion (SfM) procedure is developed to create a 3D point cloud of roadway assets from the street level imagery. In order to assist with accurate 3D recognition and localization by color and texture features extraction, an automated process of point cloud cleaning and noise removal is proposed. Using camera pose information from SfM, the points within the bounding box of detected traffic signs are then projected into the cleaned point cloud by using the triangulation method (linear and non-linear) and the 3D points corresponding to the traffic sign in question are labeled and visualized in 3D. The proposed framework is validated using real-life data, which represent the most common types of traffic signs. The robustness of the proposed pipeline is evaluated by analyzing the accuracy in detection of traffic signs as well as the accuracy in localization in 3D point cloud model. The results promise to better and more accurate visualize the location of the traffic signs with respect to other roadway assets in 3D environment.