3D Object Detection Algorithm Research Articles

Anti-Noise 3D Object Detection of Multimodal Feature Attention Fusion Based on PV-RCNN.

3D object detection methods based on camera and LiDAR fusion are susceptible to environmental noise. Due to the mismatch of physical characteristics of the two sensors, the feature vectors encoded by the feature layer are in different feature spaces. This leads to the problem of feature information deviation, which has an impact on detection performance. To address this problem, a point-guided feature abstract method is presented to fuse the camera and LiDAR at first. The extracted image features and point cloud features are aggregated to keypoints for enhancing information redundancy. Second, the proposed multimodal feature attention (MFA) mechanism is used to achieve adaptive fusion of point cloud features and image features with information from multiple feature spaces. Finally, a projection-based farthest point sampling (P-FPS) is proposed to downsample the raw point cloud, which can project more keypoints onto the close object and improve the sampling rate of the point-guided image features. The 3D bounding boxes of the object is obtained by the region of interest (ROI) pooling layer and the fully connected layer. The proposed 3D object detection algorithm is evaluated on three different datasets, and the proposed algorithm achieved better detection performance and robustness when the image and point cloud data contain rain noise. The test results on a physical test platform further validate the effectiveness of the algorithm.

A Transformer Based Complex-YOLOv4-Trans for 3D Point Cloud Object Detection on Embedded Device

3D object detection is widely used in the field of autonomous driving. The embedded devices have few resources and low power consumption, so the existing 3D object detection algorithms are difficult to complete the detection task in the required time under some conditions. In this paper, the Complex-YOLOv4-Trans model is proposed to use point cloud data for 3D object detection. Firstly, a transformer encoder block is introduced to improve the structure of Backbone, which makes full use of its context information to improve detection accuracy. Secondly, the general convolution in Backbone is replaced with a depth-wise separable convolution, which effectively reduces the computational load of the model. Finally, Complete-IoU (CIoU) is used as the loss function, and then 8-bit model quantization is performed to speed up the inference. We select the KITTI dataset and deploy the model on NVIDIA Jetson Xavier NX by using TensorRT acceleration tools to test the methods. The mAP of Bird’s Eye View for Car is 71.79%, the mAP for 3D Detection is 15.82%, and the FPS on the NX device is 42 frames. Experimental results show that our Complex-YOLOv4-Trans can perform real-time 3D object detection tasks on low-energy embedded devices.

User Experience Adaptation of Complex Game Interface for User Behaviour Modeling Using RNN

In a video game review, the main focus is the narratives, characters, graphics, and mechanics in the gameplay. Some recent research mentions the user interface only when it comes into light as a creative platform for simple interactive narratives from a technical point of view; this narrative is mainly a software tool that requires traditionally modernized inputs from the user. The user needs to interact with the navigational controls or menus in order to start a basic game play. A complex game interface as stimulus is generally considered as having a feeling of immersion that allows for visual tracking of user behavioural patterns and use it to predict the next strategy of the user using robust computational models. A number of users have limited sensory perception in a gameplay and hence rely on complex game stimulus and an adaptive model is paramount when considering behavioural expectations that place the user in a digital environment with more expressive perceptions. We developed a custom based eye tracking and 3D object detection algorithm which was utilised by recruiting users to interact with visual 3D objects and trace their eye movement behaviour to generated data. We then applied the use of recurrent neural network (RNN) for direct tracing of user behavioural activities in a sequential manner to predict their behaviour for interface adaptation. Result indicates that redundant user attributes are flexible and flawless for identifying predicted response of the user in a controlled environment. This would lead to prototypical representation of user behavioural analytics as an embedded platform in the confined digital environment. One of the limitations of the project is its inability to basically specify the 3D gaze point at the inner boundaries of the visual field. Data visualisation is strictly based on combined object flow detection. The originality of the work is its ability to redefine fixation point to a rendered cascaded 3D gaze point and space-defined saccade which is indicated by the distance between one gaze points to the other. The 3D gaze point would be well suited for fixation generalisation on 3D as well as on 2D digital oriented environment.

A Survey for 3D Object Detection Algorithms from Images

Image-based 3D object detection is one of the important and difficult problems in autonomous driving and robotics, and aims to find and represent the location, dimension and orientation of the object of interest. It generates three dimensional (3D) bounding boxes with only 2D images obtained from cameras, so there is no need for devices that provide accurate depth information such as LiDAR or Radar. Image-based methods can be divided into three main categories: monocular, stereo, and multi-view 3D object detection. In this paper, we investigate the recent state-of-the-art models of the above three categories. In the multi-view 3D object detection, which appeared together with the release of the new benchmark datasets, NuScenes and Waymo, we discuss the differences from the existing monocular and stereo methods. Also, we analyze their performance and discuss the advantages and disadvantages of them. Finally, we conclude the remaining challenges and a future direction in this field.

FCNet: Stereo 3D Object Detection with Feature Correlation Networks

Deep-learning techniques have significantly improved object detection performance, especially with binocular images in 3D scenarios. To supervise the depth information in stereo 3D object detection, reconstructing the 3D dense depth of LiDAR point clouds causes higher computational costs and lower inference speed. After exploring the intrinsic relationship between the implicit depth information and semantic texture features of the binocular images, we propose an efficient and accurate 3D object detection algorithm, FCNet, in stereo images. First, we construct a multi-scale cost–volume containing implicit depth information using the normalized dot-product by generating multi-scale feature maps from the input stereo images. Secondly, the variant attention model enhances its global and local description, and the sparse region monitors the depth loss deep regression. Thirdly, for balancing the channel information preservation of the re-fused left–right feature maps and computational burden, a reweighting strategy is employed to enhance the feature correlation in merging the last-layer features of binocular images. Extensive experiment results on the challenging KITTI benchmark demonstrate that the proposed algorithm achieves better performance, including a lower computational cost and higher inference speed in 3D object detection.

3D Object Detection from Point Cloud Based on Deep Learning

In order to study the modern 3D object detection algorithm based on deep learning, this paper studies the point-based 3D object detection algorithm, that is, a 3D object detection algorithm that uses multilayer perceptron to extract point features. This paper proposes a method based on point RCNN. A three-stage 3D object detection algorithm improves the accuracy of the algorithm by fusing image information. The algorithm in this paper integrates the information and image information of the three stages well, which improves the information utilization of the whole algorithm. Compared with the traditional 3D target detection algorithm, the structure of the algorithm in this paper is more compact, which effectively improves the utilization of information.

LiDAR Filtering in 3D Object Detection Based on Improved RANSAC

At present, the LiDAR ground filtering technology is very mature. There are fewer applications in 3D-object detection due to the limitations of filtering accuracy and efficiency. If the ground can be removed quickly and accurately, the 3D-object detection algorithm can detect objects more accurately and quickly. In order to meet the application requirements of 3D-object detection, inspired by Universal-RANSAC, we analyze the detailed steps of RANSAC and propose a precise and efficient RANSAC-based ground filtering method. The principle of GroupSAC is analyzed, and the sampled points are grouped by attributes to make it easier to sample the correct point. Based on this principle, we devise a method for limiting sampled points that is applicable to point clouds. We describe preemptive RANSAC in detail. Its breadth-first strategy is adopted to obtain the optimal plane without complex iterations. We use the International Society for Photogrammetry and Remote Sensing (ISPRS) datasets and the KITTI dataset for testing. Experiments show that our method has higher filtering accuracy and efficiency compared with the currently widely used methods. We explore the application of ground filtering methods in 3D-object detection, and the experimental results show that our method can improve the object detection accuracy without affecting the efficiency.

A survey of 3D object detection algorithms for intelligent vehicles development.

With the rapid development of Artificial Intelligent algorithms on Computer Vision, 2D object detection has greatly succeeded and been applied in various industrial products. In the past several years, the accuracy of 2D object detection has been dramatically improved, even beyond the human eyes detection ability. However, there is still a limitation of 2D object detection for the applications of Intelligent Driving. A safe and reliable self-driving car needs to detect a 3D model of the around objects so that an intelligent driving car has a perception ability to real driving situations. This paper systematically surveys the development of 3D object detection methods applied to intelligent driving technology. This paper also analyzes the shortcomings of the existing 3D detection algorithms and the future development directions of 3D detection algorithms on intelligent driving.

3D Object Detection with SLS-Fusion Network in Foggy Weather Conditions.

The role of sensors such as cameras or LiDAR (Light Detection and Ranging) is crucial for the environmental awareness of self-driving cars. However, the data collected from these sensors are subject to distortions in extreme weather conditions such as fog, rain, and snow. This issue could lead to many safety problems while operating a self-driving vehicle. The purpose of this study is to analyze the effects of fog on the detection of objects in driving scenes and then to propose methods for improvement. Collecting and processing data in adverse weather conditions is often more difficult than data in good weather conditions. Hence, a synthetic dataset that can simulate bad weather conditions is a good choice to validate a method, as it is simpler and more economical, before working with a real dataset. In this paper, we apply fog synthesis on the public KITTI dataset to generate the Multifog KITTI dataset for both images and point clouds. In terms of processing tasks, we test our previous 3D object detector based on LiDAR and camera, named the Spare LiDAR Stereo Fusion Network (SLS-Fusion), to see how it is affected by foggy weather conditions. We propose to train using both the original dataset and the augmented dataset to improve performance in foggy weather conditions while keeping good performance under normal conditions. We conducted experiments on the KITTI and the proposed Multifog KITTI datasets which show that, before any improvement, performance is reduced by 42.67% in 3D object detection for Moderate objects in foggy weather conditions. By using a specific strategy of training, the results significantly improved by 26.72% and keep performing quite well on the original dataset with a drop only of 8.23%. In summary, fog often causes the failure of 3D detection on driving scenes. By additional training with the augmented dataset, we significantly improve the performance of the proposed 3D object detection algorithm for self-driving cars in foggy weather conditions.

스테레오 영상처리 기반의 3차원 객체 검출을 위한 3차원 포인트 필터링 알고리즘

Unlike LIDAR-based 3D object detection algorithms, stereo-based 3D object detection algorithms have low performance. The main reason for this problem is the low accuracy of the 3D information predicted from stereo image processing. Inaccurate 3D information distorts the 3D shape of an object. Therefore, this study aims to improve the performance of the stereo-based 3D object detection algorithms by proposing a filtering algorithm that can filter out inaccurate 3D information. The filtering algorithm uses a 3D offset, which is the distance between a 3D point on an object and the center of gravity of the object. Since the object size is limited, the 3D offset length of the points constituting the object is always smaller than the object size itself. However, there is a case where the 3D offset length of inaccurate 3D information is larger than the object size. The proposed filtering algorithm filters out inaccurate 3D information to address this issue. This study experimentally proves that the filtering algorithm is effective in enhancing the stereo-based 3D object detection performance.

GridNet‐3D: A Novel Real‐Time 3D Object Detection Algorithm Based on Point Cloud

3D object detection based on point cloud has an important application prospect in automatic driving technology. Aiming at the low precision of 3D object detection based on point cloud and the poor real-time performance caused by large numbers of 3D convolutions, a novel end-to-end real-time object detection algorithm named GridNet-3D is proposed. In the work, 2D gridmapping is used to preprocess the original point clouds. Then a novel structure grid encoding layer is adopted to encode point cloud features and is gotten grid feature maps in bird's eye view which is connected to region proposal network module to generate detections. Despite only using point clouds, the results on the KITTI 3D detection benchmark show that our algorithm has higher detection precision and better real-time performance on the detection of cars, pedestrians and cyclists, which has high practical value.

Outdoor Mobile Mapping and AI-Based 3D Object Detection with Low-Cost RGB-D Cameras: The Use Case of On-Street Parking Statistics

A successful application of low-cost 3D cameras in combination with artificial intelligence (AI)-based 3D object detection algorithms to outdoor mobile mapping would offer great potential for numerous mapping, asset inventory, and change detection tasks in the context of smart cities. This paper presents a mobile mapping system mounted on an electric tricycle and a procedure for creating on-street parking statistics, which allow government agencies and policy makers to verify and adjust parking policies in different city districts. Our method combines georeferenced red-green-blue-depth (RGB-D) imagery from two low-cost 3D cameras with state-of-the-art 3D object detection algorithms for extracting and mapping parked vehicles. Our investigations demonstrate the suitability of the latest generation of low-cost 3D cameras for real-world outdoor applications with respect to supported ranges, depth measurement accuracy, and robustness under varying lighting conditions. In an evaluation of suitable algorithms for detecting vehicles in the noisy and often incomplete 3D point clouds from RGB-D cameras, the 3D object detection network PointRCNN, which extends region-based convolutional neural networks (R-CNNs) to 3D point clouds, clearly outperformed all other candidates. The results of a mapping mission with 313 parking spaces show that our method is capable of reliably detecting parked cars with a precision of 100% and a recall of 97%. It can be applied to unslotted and slotted parking and different parking types including parallel, perpendicular, and angle parking.

Temporal-Channel Transformer for 3D Lidar-Based Video Object Detection for Autonomous Driving

The strong demand of autonomous driving in the industry has led to vigorous interest in 3D object detection and resulted in many excellent 3D object detection algorithms. However, the vast majority of algorithms only model single-frame data, ignoring the temporal clue in video sequence. In this work, we propose a new transformer, called Temporal-Channel Transformer (TCTR), to model the temporal-channel domain and spatial-wise relationships for video object detecting from Lidar data. As the special design of this transformer, the information encoded in the encoder is different from that in the decoder. The encoder encodes temporal-channel information of multiple frames while the decoder decodes the spatial-wise information for the current frame in a voxel-wise manner. Specifically, the temporal-channel encoder of the transformer is designed to encode the information of different channels and frames by utilizing the correlation among features from different channels and frames. On the other hand, the spatial decoder of the transformer decodes the information for each location of the current frame. Before conducting the object detection with detection head, a gate mechanism is further deployed for re-calibrating the features of current frame, which filters out the object-irrelevant information by repetitively refining the representation of target frame along with the up-sampling process. Experimental results reveal that TCTR achieves the state-of-the-art performance in grid voxel-based 3D object detection on the nuScenes benchmark.

A Coarse-to-Fine Approach for Rock Bolt Detection From 3D Point Clouds

Rock bolts have been widely used to enhance the structural stability of underground infrastructures. Careful tracking of rock bolt positions is highly significant since it assists with operational success of ground support and has applications to predictive maintenance practices. This paper presents a practical algorithm, CFBolt , to detect rock bolts from a 3D laser scanned point cloud. Considering that rock bolts are relatively tiny objects, CFBolt follows a two-step coarse-to-fine strategy. It first computes a single-scale proportion of variance (POV) for each point as the local point descriptor and filters out near 95% not-bolt points with a simple but effective classifier, Linear Discriminant Analysis (LDA), which allows for the pruned point cloud to be then used as a compatible input to a deep neural network, designed and trained to precisely detect rock bolts from the pruned point cloud. CFBolt was tested for detecting rock bolts from LiDAR scan data collected from Sydney’s civil tunnelling project site. The entire dataset contains more than 160 million points. The obtained scores of Intersection over Union (IoU) and precision for individual bolt points were 89.33% and 92.04%, respectively. For rock bolt objects, the precision and recall were 98.34% and 98.73%, respectively. The detection quality of CFBolt is superior to the state-of-the-art 3D object detection algorithms and the newest rock bolt detection algorithm, demonstrating the robustness and effectiveness of CFBolt .

A Survey on Monocular 3D Object Detection Algorithms Based on Deep Learning

An accurate and effective perception of environment is important for autonomous vehicle and robot. The perception system needs to obtain the 3D information of objects, which includes objects’ space location and pose. Camera is widely equipped on autonomous vehicle because of its price advantage. However, the monocular camera cannot provide depth information which is necessary for 3D object detection. Many algorithms based on monocular 3D object detection have been developed in recent years. Deep learning is popular for perception system which transforms image data from camera into semantic information. This paper presents an overview of monocular 3D object detection algorithms based on deep Learning and summarize the contributions and limitations of these algorithms. We also compare the performance of different algorithms on different datasets.

An End-to-End Deep Learning Network for 3D Object Detection From RGB-D Data Based on Hough Voting

Existing outdoor three-dimensional (3D) object detection algorithms mainly use a single type of sensor, for example, only using a monocular camera or radar point cloud. However, camera sensors are affected by light and lose depth information. When scanning a distant object or an occluded object, the data collected by the short-range radar point cloud sensor are very sparse, which affects the detection algorithm. To address the above challenges, we design a deep learning network that can combine the texture information of two-dimensional (2D) data and the geometric information of 3D data for object detection. To solve the problem of a single sensor, we use a reverse mapping layer and an aggregation layer to combine the texture information of RGB data with the geometric information of point cloud data and design a maximum pooling layer to deal with the input of multi-view cameras. In addition, to solve the defects of the 3D object detection algorithm based on the region proposal network (RPN) method, we use the Hough voting algorithm implemented by a deep neural network to suggest objects. Experimental results show that our algorithm has a 1.06% decrease in average precision (AP) compared to PointRCNN in easy car object detection, but our algorithm requires 37.7% less time to calculate than PointRCNN under the same hardware environment. Moreover, our algorithm improves the AP by 1.14% compared to PointRCNN in hard car object detection.

Recent Advances in 3D Object Detection in the Era of Deep Neural Networks: A Survey.

With the rapid development of deep learning technology and other powerful tools, 3D object detection has made great progress and become one of the fastest growing field in computer vision. Many automated applications such as robotic navigation, autonomous driving, and virtual or augmented reality system require estimation of accurate 3D object location and detection. Under this requirement, many methods have been proposed to improve the performance of 3D object localization and detection. Despite recent efforts, 3D object detection is still a very challenging task due to occlusion, viewpoint variations, scale changes, and limited information in 3D scenes. In this paper, we present a comprehensive review of recent state-of-the-art approaches in 3D object detection technology. We start with some basic concepts, then describe some of the available datasets that are designed to facilitate the performance evaluation of 3D object detection algorithms. Next, we will review the state-of-the-art technologies in this area, highlighting their contributions, importance, and limitations as a guide for future research. Finally, we provide a quantitative comparison of the results of the state-of-the-art methods on the popular public datasets.

SARPNET: Shape attention regional proposal network for liDAR-based 3D object detection

Real-time 3D object detection is a fundamental technique in numerous applications, such as autonomous driving, unmanned aerial vehicles (UAV) and robot vision. However, current LiDAR-based 3D object detection algorithms allocate inadequate attention to the inhomogeneity of LiDAR point clouds and the shape encoding capability of regional proposal schemes. This paper introduces a novel 3D object detection network called the Shape Attention Regional Proposal Network (SARPNET), which deploys a new low-level feature encoder to remedy the sparsity and inhomogeneity of LiDAR point clouds with an even sample method, and embodies a shape attention mechanism that learns the statistic 3D shape priors of objects and uses them to spatially enhance semantic embeddings. Experimental results show that the proposed one-stage method outperforms state-of-the-art one-stage and even two-stage methods on the KITTI 3D object detection benchmark. It achieved a BEV AP of (87.26%, 62.80%), 3D AP of (75.64%, 60.43%), and orientation AP of (88.86%, 71.01%) for the detection of cars and cyclists, respectively. Besides, the method is the third winner in the nuScenes 3D Detection challenge of CVPR2019 Workshop on Autonomous Driving (WAD).

Multi-View Fusion-Based 3D Object Detection for Robot Indoor Scene Perception.

To autonomously move and operate objects in cluttered indoor environments, a service robot requires the ability of 3D scene perception. Though 3D object detection can provide an object-level environmental description to fill this gap, a robot always encounters incomplete object observation, recurring detections of the same object, error in detection, or intersection between objects when conducting detection continuously in a cluttered room. To solve these problems, we propose a two-stage 3D object detection algorithm which is to fuse multiple views of 3D object point clouds in the first stage and to eliminate unreasonable and intersection detections in the second stage. For each view, the robot performs a 2D object semantic segmentation and obtains 3D object point clouds. Then, an unsupervised segmentation method called Locally Convex Connected Patches (LCCP) is utilized to segment the object accurately from the background. Subsequently, the Manhattan Frame estimation is implemented to calculate the main orientation of the object and subsequently, the 3D object bounding box can be obtained. To deal with the detected objects in multiple views, we construct an object database and propose an object fusion criterion to maintain it automatically. Thus, the same object observed in multi-view is fused together and a more accurate bounding box can be calculated. Finally, we propose an object filtering approach based on prior knowledge to remove incorrect and intersecting objects in the object dataset. Experiments are carried out on both SceneNN dataset and a real indoor environment to verify the stability and accuracy of 3D semantic segmentation and bounding box detection of the object with multi-view fusion.

3D Object Detection Algorithm for Panoramic Images With Multi-Scale Convolutional Neural Network

This paper addresses the challenge of 3D object detection from a single panoramic image under severe deformation. The advent of the two-stage approach has impelled significant progress in 3D object detection. However, most available methods only can localize region proposals by a single-scale architecture network, which are sensitive to deformation and distortion. To address this issue, we propose a multi-scale convolutional neural network (MSCNN) to estimate the 3D pose of an object. To be specific, the proposed MSCNN consists of three steps for effectively detecting the distorted object on the panoramic images. The MSCNN contains the CycleGAN network that converts rectilinear images into panoramas, a fused framework that improves both accuracy and speed for object detection, and an adversarial spatial transformer network (ASTN) that extracts the deformation features of the object on panoramic images. Additionally, we recover the 3D pose of the object using a coordinate projection and a 3D bounding box. Extensive experiments demonstrate that the proposed method can achieve a 3D detection accuracy of 38.7% in high-resolution panoramic images, which is higher than the current state-of-the-art algorithm of 5.2%. Moreover, the speed of detection is only about 0.6 seconds per image, which is six times faster than Faster R-CNN (COCO). The code will be available at https://github.com/Yanhui-He.