Camera Fusion Research Articles

End-to-End Multimodal Sensor Dataset Collection Framework for Autonomous Vehicles.

Autonomous driving vehicles rely on sensors for the robust perception of their surroundings. Such vehicles are equipped with multiple perceptive sensors with a high level of redundancy to ensure safety and reliability in any driving condition. However, multi-sensor, such as camera, LiDAR, and radar systems raise requirements related to sensor calibration and synchronization, which are the fundamental blocks of any autonomous system. On the other hand, sensor fusion and integration have become important aspects of autonomous driving research and directly determine the efficiency and accuracy of advanced functions such as object detection and path planning. Classical model-based estimation and data-driven models are two mainstream approaches to achieving such integration. Most recent research is shifting to the latter, showing high robustness in real-world applications but requiring large quantities of data to be collected, synchronized, and properly categorized. However, there are two major research gaps in existing works: (i) they lack fusion (and synchronization) of multi-sensors, camera, LiDAR and radar; and (ii) generic scalable, and user-friendly end-to-end implementation. To generalize the implementation of the multi-sensor perceptive system, we introduce an end-to-end generic sensor dataset collection framework that includes both hardware deploying solutions and sensor fusion algorithms. The framework prototype integrates a diverse set of sensors, such as camera, LiDAR, and radar. Furthermore, we present a universal toolbox to calibrate and synchronize three types of sensors based on their characteristics. The framework also includes the fusion algorithms, which utilize the merits of three sensors, namely, camera, LiDAR, and radar, and fuse their sensory information in a manner that is helpful for object detection and tracking research. The generality of this framework makes it applicable in any robotic or autonomous applications and suitable for quick and large-scale practical deployment.

Multi-modality 3D object detection in autonomous driving: A review

Autonomous driving perception has made significant strides in recent years, but accurately sensing the environment using a single sensor remains a daunting task. This review offers a comprehensive overview of the current research on LiDAR and camera fusion for 3D object detection in multi-modality domains. The review first identifies the perception task, open public detection dataset, and data representation related to 3D object detection. It then presents an in-depth survey of coarse-grained and fine-grained fusion approaches, reporting their respective performances on the KITTI and nuScenes datasets. The review identifies general trends in multi-modality 3D object detection and provides insights and promising research directions based on these observations. Additionally, the review summarizes the current challenges of fusion strategies for perception problems in autonomous driving. Based on a critical review of existing literature, this paper identifies and discusses key research directions in the field of fusion-based 3D object detection approach for perception problems in autonomous driving, which is instructive for future work.

Three-dimensional structural displacement estimation by fusing monocular camera and accelerometer using adaptive multi-rate Kalman filter

Structural displacements play an important role in the health monitoring of civil structures; however, the accurate measurement of structural displacements remains a difficult task. Previous efforts have combined a monocular camera and an accelerometer to estimate structural displacement, but only in-plane displacements could be estimated in this way. In this study, the fusion of a monocular camera and an accelerometer was further extended for out-of-plane or three-dimensional displacement estimation. A computer vision algorithm and an adaptive multi-rate Kalman filter were integrated to efficiently estimate high-sampled displacements from low-sampled vision images and high-sampled acceleration measurements. All parameters associated with the computer vision algorithm were automatically calibrated without using any user-defined thresholds. Experimental validation was performed on two building structures and a 10-m-long bridge structure, and the proposed method accurately estimated the displacement for all three structures with a root mean square error of less than 1 mm.

Obstacle Detection Method Based on RSU and Vehicle Camera Fusion

Road obstacle detection is an important component of intelligent assisted driving technology. Existing obstacle detection methods ignore the important direction of generalized obstacle detection. This paper proposes an obstacle detection method based on the fusion of roadside units and vehicle mounted cameras and illustrates the feasibility of a combined monocular camera inertial measurement unit (IMU) and roadside unit (RSU) detection method. A generalized obstacle detection method based on vision IMU is combined with a roadside unit obstacle detection method based on a background difference method to achieve generalized obstacle classification while reducing the spatial complexity of the detection area. In the generalized obstacle recognition stage, a VIDAR (Vision-IMU based identification and ranging) -based generalized obstacle recognition method is proposed. The problem of the low accuracy of obstacle information acquisition in the driving environment where generalized obstacles exist is solved. For generalized obstacles that cannot be detected by the roadside unit, VIDAR obstacle detection is performed on the target generalized obstacles through the vehicle terminal camera, and the detection result information is transmitted to the roadside device terminal through the UDP (User Data Protocol) protocol to achieve obstacle recognition and pseudo-obstacle removal, thereby reducing the error recognition rate of generalized obstacles. In this paper, pseudo-obstacles, obstacles with a certain height less than the maximum passing height of the vehicle, and obstacles with a height greater than the maximum passing height of the vehicle are defined as generalized obstacles. Pseudo-obstacles refer to non-height objects that appear to be “patches” on the imaging interface obtained by visual sensors and obstacles with a height less than the maximum passing height of the vehicle. VIDAR is a vision-IMU-based detection and ranging method. IMU is used to obtain the distance and pose of the camera movement, and through the inverse perspective transformation, it can calculate the height of the object in the image. The VIDAR-based obstacle detection method, the roadside unit-based obstacle detection method, YOLOv5 (You Only Look Once version 5), and the method proposed in this paper were applied to outdoor comparison experiments. The results show that the accuracy of the method is improved by 2.3%, 17.4%, and 1.8%, respectively, compared with the other four methods. Compared with the roadside unit obstacle detection method, the speed of obstacle detection is improved by 1.1%. The experimental results show that the method can expand the detection range of road vehicles based on the vehicle obstacle detection method and can quickly and effectively eliminate false obstacle information on the road.

Real time object detection using LiDAR and camera fusion for autonomous driving

Autonomous driving has been widely applied in commercial and industrial applications, along with the upgrade of environmental awareness systems. Tasks such as path planning, trajectory tracking, and obstacle avoidance are strongly dependent on the ability to perform real-time object detection and position regression. Among the most commonly used sensors, camera provides dense semantic information but lacks accurate distance information to the target, while LiDAR provides accurate depth information but with sparse resolution. In this paper, a LiDAR-camera-based fusion algorithm is proposed to improve the above-mentioned trade-off problems by constructing a Siamese network for object detection. Raw point clouds are converted to camera planes to obtain a 2D depth image. By designing a cross feature fusion block to connect the depth and RGB processing branches, the feature-layer fusion strategy is applied to integrate multi-modality data. The proposed fusion algorithm is evaluated on the KITTI dataset. Experimental results demonstrate that our algorithm has superior performance and real-time efficiency. Remarkably, it outperforms other state-of-the-art algorithms at the most important moderate level and achieves excellent performance at the easy and hard levels.

Fusion of an RGB camera and LiDAR sensor through a Graph CNN for 3D object detection

Despite the recent development of RGB camera and LiDAR sensor fusion technology using deep learning, fusion without loss of information is a very difficult problem because the structural data characteristics of the two sensors are different. To solve this problem, we use a graph convolutional neural network (Graph CNN) to fuse RGB and LiDAR sensors. The proposed method creates a fusion feature by supplementing the geometric information of each feature in the process of fusing the features of two different sensors. Based on the experimental data, the proposed method has higher accuracy in detecting distant objects and complex situations than the existing method.

A Novel Stereo Camera Fusion Scheme for Generating and Tracking Real-Time 3-D Patient-Specific Head/Face Kinematics and Facial Muscle Movements

Recovery and rehabilitation of facial mimics need enhanced decision support with multimodal bio-feedbacks from 3D real-time biomechanical head animation. Kinect V2.0 can detect and track 3-D high-definition face features (FFs) but the end of production can lead to difficult deployment of the developed solutions. Deep Neural Network (DNN)-based methods were employed, but the detected features were in 2-D or not accurate in 3-D. So, we developed a novel stereo-fusion scheme for enhancing the accuracy of 3-D features and generating biomechanical heads. Four stereo cameras were employed for detecting 2-D FFs based on DNN-based models. Stereo-triangulated 3-D FFs were fused using the Kalman filter. A head, skull, and muscle network were generated from the fused FFs. We validated the method with 1,000 virtual subjects and 5 Computed Tomography-based subjects. The in-silico trial errors (Mean±SD) were 2.27±0.29mm, 3.15±0.23mm, 1.72±0.13mm, and 3.08±0.39mm for the facial head, facial skull, muscle insertion point, and muscle attachment point regions. The experimental errors were 1.8384±0.1451mm, 2.6937±0.0575mm, 1.8271±0.1242mm, and 3.1428±0.2407mm. The errors were compatible with those using the Kinect V2.0 sensor and smaller than those using monovision-based 3-D feature detectors. This study has four contributions: (1) a stereo-fusion scheme for reconstructing 3-D FFs from 2-D FFs, (2) an enhancement accuracy for 3D DNN-based FF detection, (3) a biomechanical head generation from stereo-fusion cameras, and (4) a full validation procedure for 3-D FF detection. The method will be validated with facial palsy patients. Soft-tissue deformation will be integrated with mixed reality technology toward a next-generation of face decision-support system.

Heterogeneous Fusion of Camera and mmWave Radar Sensor of Optimizing Convolutional Neural Networks for Parking Meter System.

In this article, a novel heterogeneous fusion of convolutional neural networks that combined an RGB camera and an active mmWave radar sensor for the smart parking meter is proposed. In general, the parking fee collector on the street outdoor surroundings by traffic flows, shadows, and reflections makes it an exceedingly tough task to identify street parking regions. The proposed heterogeneous fusion convolutional neural networks combine an active radar sensor and image input with specific geometric area, allowing them to detect the parking region against different tough conditions such as rain, fog, dust, snow, glare, and traffic flow. They use convolutional neural networks to acquire output results along with the individual training and fusion of RGB camera and mmWave radar data. To achieve real-time performance, the proposed algorithm has been implemented on a GPU-accelerated embedded platform Jetson Nano with a heterogeneous hardware acceleration methodology. The experimental results exhibit that the accuracy of the heterogeneous fusion method can reach up to 99.33% on average.

CCA-Based Fusion of Camera and Radar Features for Target Classification Under Adverse Weather Conditions

CCA-Based Fusion of Camera and Radar Features for Target Classification Under Adverse Weather Conditions

Radar–Camera Fusion Vehicle Detection Based on Feature Weighting and Visual Enhancement

Radar–Camera Fusion Vehicle Detection Based on Feature Weighting and Visual Enhancement

Image Processing Integration into Global Navigation Satellite System Reflectometry to Remotely Sense Freshwater Surface

Global Navigation Satellite System Reflectometry (GNSS-R) is a technique that detects signals reflected off the surface and analyzes the reflected signals to obtain surface characteristics. This technique was used to differentiate between surface ice and water condition of Lake Michigan in Chicago. To confirm GNSS-R results validity, a separate reference is needed to compare with. A sensor fusion of camera and a lidar was used to reconstruct the lake surface. This reconstructed surface was as a reference to compare with GNSS-R results. Since the current camera frames rate being used to reconstruct the lake surface using the sensor fusion of camera and lidar is two frames per minute, this research explores the effect of increasing the camera frames rate on the reconstructed surface.

Detecting Darting Out Pedestrians With Occlusion Aware Sensor Fusion of Radar and Stereo Camera

Early and accurate detection of crossing pedestrians is crucial in automated driving in order to perform timely emergency manoeuvres. However, this is a difficult task in urban scenarios where pedestrians are often occluded (not visible) behind objects, e.g., other parked vehicles. We propose an occlusion aware fusion of stereo camera and radar sensors to address scenarios with crossing pedestrians behind such parked vehicles. Our proposed method adapts both the expected rate and properties of detections in different areas according to the visibility of the sensors. In our experiments on a real-world dataset, we show that the proposed occlusion aware fusion of radar and stereo camera detects the crossing pedestrians on average 0.26 seconds earlier than using the camera alone, and 0.15 seconds earlier than fusing the sensors without occlusion information. Our dataset containing 501 relevant recordings of pedestrians behind vehicles will be publicly available on our website for non-commercial, scientific use.

IMU and Smartphone Camera Fusion for Knee Adduction and Knee Flexion Moment Estimation During Walking

Wearable sensing and computer vision could move biomechanics from specialized laboratories to natural environments, but better algorithms are needed to extract meaningful outcomes from these emerging modalities. In this article, we present new models for estimating biomechanical outcomes—the knee adduction moment (KAM) and knee flexion moment (KFM)—from fusion of smartphone cameras and wearable inertial measurement units (IMUs) among young healthy nonobese males. A deep learning model was developed to extract features, fuse multimodal data, and estimate KAM and KFM. Walking data from 17 subjects were recorded with eight IMUs and two smartphone cameras. The model that used IMU-camera fusion was significantly more accurate than those using IMUs or cameras alone. The root-mean-square errors of the fusion model were 0.49 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%\;\mathbf {BW}\cdot \mathbf {BH}$</tex-math></inline-formula> for KAM and 0.66 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%\;\mathbf {BW}\cdot \mathbf {BH}$</tex-math></inline-formula> for KFM estimation, which are lower than clinically significant thresholds. With larger and more diverse data, this model could enable assessment of knee moments in clinics and homes.

Fully convolutional neural networks for LIDAR–camera fusion for pedestrian detection in autonomous vehicle

Fully convolutional neural networks for LIDAR–camera fusion for pedestrian detection in autonomous vehicle

Retracted: Cable tunnel unmanned aerial vehicle location and navigation method based on ultra wideband and depth camera fusion

AbstractWith the increasing demand for electric energy, more and more cities are constructing cable tunnels. In order to prevent the cable in the tunnel from being damaged, regular inspection of the cable tunnel is very important. However, due to the large scale of cable tunnel construction, the traditional manual inspection is difficult to meet the maintenance requirements. With the development of intelligent technology, the intelligent patrol mode of UAV has gradually replaced the traditional manual patrol mode and become the main patrol mode of cable tunnel. Therefore, the positioning and navigation method of cable tunnel UAV is studied in this paper. By using UWB localization algorithm and depth camera fusion, the positioning and navigation methods of UAVs are determined and compared with the positioning algorithm based on optical flow. Experimental comparison results show that the maximum and minimum yaw angles of UAV positioning based on the proposed algorithm are 2.41° and 0.16°, while the maximum and minimum yaw angles of UAV positioning based on the comparison algorithm are 3.11° and 1.51°. It can be seen that the integration of UWB and depth camera is of great significance for the research of positioning and navigation methods of cable tunnel UAVs.

High-Precision External Parameter Calibration Method for Camera and Lidar Based on a Calibration Device

The fusion of camera and lidar plays an important role in the field of robotic perception. The accurate external parameter calibration is a necessary prerequisite for sensor fusion. Herein, an auxiliary calibration device with distinctive geometric features was designed to address the problems of low accuracy and poor robustness associated with external parameter calibrations of camera and lidar. Moreover, a coarse-to-fine two-stage calibration method was proposed for the external parameters of the camera and lidar. The first stage of the method is the extraction of multiple groups of two-dimensional (2D) and three-dimensional (3D) lines corresponding to the edge of the calibration device from the image and lidar point cloud that yields a unique initial estimation of the external parameters. In the second stage, the 2D–3D center point of the sphere of the calibration device was detected, and the initial external parameters were further optimized using a nonlinear optimization method. The proposed method provides two different features that add stability against noise to the calibration system. Both simulated and actual experiments show that the method can yield high-precision external parameters without an initial value. Compared to state-of-the-art methods, our method has advantages in terms of accuracy. the calibration system has a certain degree of noise resistance and stability under different laser noise and vertical resolution.

Adaptive Active Fusion of Camera and Single-Point LiDAR for Depth Estimation

Depth sensing is an important problem in many applications such as autonomous driving, robotics, and automation. This paper presents an adaptive active fusion method for scene depth estimation by using an RGB camera and a single-point LiDAR. An active scanning mechanism is proposed to guide laser scanning based on critical visual and saliency features, and the convolutional spatial propagation network (CSPN) is designed to generate and refine full depth map from the sparse depth scans. The active scanning mechanism generates a depth mask by using log-spectrum saliency detection, Canny edge detection, and uniform sampling, which indicate critical regions that require a high resolution of laser scanning. To reconstruct a full depth map, the designed CSPN network extracts affinity matrices from the sparse depth scans, while reserving global spatial information in the RGB images. The performance of proposed method was evaluated and compared with the state-of-the-art methods on the NYUv2 dataset, and the experiment demonstrated its outperformance in reconstruction accuracy and robustness to measurement noise. The proposed method was also evaluated in real-world scenarios.

Event Camera and LiDAR based Human Tracking for Adverse Lighting Conditions in Subterranean Environments

In this article, we propose a novel LiDAR and event camera fusion modality for subterranean (SubT) environments for fast and precise object and human detection in a wide variety of adverse lighting conditions, such as low or no light, high-contrast zones and in the presence of blinding light sources. In the proposed approach, information from the event camera and LiDAR are fused to localize a human or an object-of-interest in a robot's local frame. The local detection is then transformed into the inertial frame and used to set references for a Nonlinear Model Predictive Controller (NMPC) for reactive tracking of humans or objects in SubT environments. The proposed novel fusion uses intensity filtering and K-means clustering on the LiDAR point cloud and frequency filtering and connectivity clustering on the events induced in an event camera by the returning LiDAR beams. The centroids of the clusters in the event camera and LiDAR streams are then paired to localize reflective markers present on safety vests and signs in SubT environments. The efficacy of the proposed scheme has been experimentally validated in a real SubT environment (a mine) with a Pioneer 3AT mobile robot. The experimental results show real-time performance for human detection and the NMPC-based controller allows for reactive tracking of a human or object of interest, even in complete darkness.

Gmapping Mapping Based on Lidar and RGB-D Camera Fusion

Gmapping Mapping Based on Lidar and RGB-D Camera Fusion

Multi-Objective Association Detection of Farmland Obstacles Based on Information Fusion of Millimeter Wave Radar and Camera

In order to remedy the defects of single sensor in robustness, accuracy, and redundancy of target detection, this paper proposed a method for detecting obstacles in farmland based on the information fusion of a millimeter wave (mmWave) radar and a camera. Combining the advantages of the mmWave radar in range and speed measurement and the camera in type identification and lateral localization, a decision-level fusion algorithm was designed for the mmWave radar and camera information, and the global nearest neighbor method was used for data association. Then, the effective target sequences of the mmWave radar and the camera with successful data association were weighted to output, and the output included more accurate target orientation, longitudinal speed, and category. For the unassociated sequences, they were tracked as new targets by using the extended Kalman filter algorithm and were processed and output during the effective life cycle. Lastly, an experimental platform based on a tractor was built to verify the effectiveness of the proposed association detection method. The obstacle detection test was conducted under the ROS environment after solving the external parameters of the mmWave radar and the internal and external parameters of the camera. The test results show that the correct detection rate of obstacles reaches 86.18%, which is higher than that of a single camera with 62.47%. Furthermore, through the contrast experiment of the sensor fusion algorithms, the detection accuracy of the decision level fusion algorithm was 95.19%, which was higher than 4.38% and 6.63% compared with feature level and data level fusion, respectively.