Karlsruhe Institute Of Technology And Toyota Technological Institute Research Articles

Multi-Classifier Feature Fusion-Based Road Detection for Connected Autonomous Vehicles

Connected autonomous vehicles (CAVs) currently promise cooperation between vehicles, providing abundant and real-time information through wireless communication technologies. In this paper, a two-level fusion of classifiers (TLFC) approach is proposed by using deep learning classifiers to perform accurate road detection (RD). The proposed TLFC-RD approach improves the classification by considering four key strategies such as cross fold operation at input and pre-processing using superpixel generation, adequate features, multi-classifier feature fusion and a deep learning classifier. Specifically, the road is classified as drivable and non-drivable areas by designing the TLFC using the deep learning classifiers, and the detected information using the TLFC-RD is exchanged between the autonomous vehicles for the ease of driving on the road. The TLFC-RD is analyzed in terms of its accuracy, sensitivity or recall, specificity, precision, F1-measure and max F measure. The TLFC- RD method is also evaluated compared to three existing methods: U-Net with the Domain Adaptation Model (DAM), Two-Scale Fully Convolutional Network (TFCN) and a cooperative machine learning approach (i.e., TAAUWN). Experimental results show that the accuracy of the TLFC-RD method for the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset is 99.12% higher than its competitors.

Stereo visual odometry with velocity constraint for ground vehicle applications

AbstractThis paper proposes a novel method of error mitigation for stereo visual odometry (VO) applied in land vehicles. A non-holonomic constraint (NHC), which imposes physical constraint to the rightward velocity of a land vehicle, is implemented as an observation in an extended Kalman filter (EKF) to reduce the drift of stereo VO. The EKF state vector includes position errors in an Earth-centred, Earth-fixed (ECEF) frame, velocity errors in the camera frame, angular rate errors and attitude errors. All the related equations are described and presented in detail. In this approach, no additional sensors are used but NHC, namely velocity constraint in the right direction , is applied as an external measurement to improve the accuracy. Tests are conducted with the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) datasets. Results show that the relative horizontal positioning error improved from 0⋅63% to 0⋅22% on average with the application of the velocity constraints. The maximum and root mean square of the horizontal error with velocity constraints are both reduced to less than half of the error with stand-alone stereo VO.

Study of a Multi-Beam LiDAR Perception Assessment Model for Real-Time Autonomous Driving

Light detection and ranging (LiDAR) provides a 3-D understanding of environment and plays an important role in autonomous driving. To study the influence of 3-D data quality on the environment perception and provide a theoretical basis for optimizing system design, a multi-beam LiDAR perception assessment model has been established to reveal the relationship between data quality and multi-parameters, including system and motion parameters. A novel ground segmentation algorithm was proposed with a combination of the grid elevation and the neighbor relationship, which was used to validate how the data quality influences the results of environment perception. By the way of down-sampling based on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset, the experimental results showed that the proposed ground segmentation with combination of grid-elevation and neighbor-relationship (GSCGN) method was superior than other general ground segmentation methods in terms of accuracy and efficiency. It should be noted that the mean vertical angular resolution (MVAR), laser repetition frequency, and beam numbers were the dominant influencing parameters on the point density and the accuracy of ground segmentation. Based on the experimental results, the lower limits of system parameters were determined as 16-beam and 4-kHz repetition frequency, with the acceptable recall of 92.2% for ground and 93.5% for object, the accuracy of 92.9% and the runtime of 0.036 s, which can not only provide a reliable environment perception effect, but also reducing the computational burden to satisfy the real-time autonomous driving. This study offers a meaningful investigation to guide LiDAR system design with balancing the contradiction between the optimized system design and the high-degree environment perception.

Vehicle Pedestrian Detection Method Based on Spatial Pyramid Pooling and Attention Mechanism

Object detection for vehicles and pedestrians is extremely difficult to achieve in autopilot applications for the Internet of vehicles, and it is a task that requires the ability to locate and identify smaller targets even in complex environments. This paper proposes a single-stage object detection network (YOLOv3-promote) for the detection of vehicles and pedestrians in complex environments in cities, which improves on the traditional You Only Look Once version 3 (YOLOv3). First, spatial pyramid pooling is used to fuse local and global features in an image to better enrich the expression ability of the feature map and to more effectively detect targets with large size differences in the image; second, an attention mechanism is added to the feature map to weight each channel, thereby enhancing key features and removing redundant features, which allows for strengthening the ability of the feature network to discriminate between target objects and backgrounds; lastly, the anchor box derived from the K-means clustering algorithm is fitted to the final prediction box to complete the positioning and identification of target vehicles and pedestrians. The experimental results show that the proposed method achieved 91.4 mAP (mean average precision), 83.2 F1 score, and 43.7 frames per second (FPS) on the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) dataset, and the detection performance was superior to the conventional YOLOv3 algorithm in terms of both accuracy and speed.

Multiple-Joint Pedestrian Tracking Using Periodic Models.

Estimating accurate positions of multiple pedestrians is a critical task in robotics and autonomous cars. We propose a tracker based on typical human motion patterns to track multiple pedestrians. This paper assumes that the legs’ reflection and extension angles are approximately changing periodically during human motion. A Fourier series is fitted in order to describe the moving, such as describing the position and velocity of the hip, knee, and ankle. Our tracker receives the position of the ankle, knee, and hip as measurements. As a proof of concept, we compare our tracker with state-of-the-art methods. The proposed models have been validated by experimental data, the Human Gait Database (HuGaDB), and the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) tracking benchmark. The results indicate that our tracker is able to estimate the reflection and extension angles with a precision of 90.97%. Moreover, the comparison shows that the tracking precision increases up to 1.3% with the proposed tracker when compared to a constant velocity based tracker.

Spatial-Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking.

Multi-object tracking (MOT) plays a crucial role in various platforms. Occlusion and insertion among targets, complex backgrounds and higher real-time requirements increase the difficulty of MOT problems. Most state-of-the-art MOT approaches adopt the tracking-by-detection strategy, which relies on compute-intensive sliding windows or anchoring schemes to detect matching targets or candidates in each frame. In this work, we introduce a more efficient and effective spatial–temporal attention scheme to track multiple objects in various scenarios. Using a semantic-feature-based spatial attention mechanism and a novel Motion Model, we address the insertion and location of candidates. Some online-learned target-specific convolutional neural networks (CNNs) were used to estimate target occlusion and classify by adapting the appearance model. A temporal attention mechanism was adopted to update the online module by balancing current and history frames. Extensive experiments were performed on Karlsruhe Institute of Technologyand Toyota Technological Institute (KITTI) benchmarks and an Armored Target Tracking Dataset (ATTD) built for ground-armored targets. Experimental results show that the proposed method achieved outstanding tracking performance and met the actual application requirements.

Analysis of the Influence of Foggy Weather Environment on the Detection Effect of Machine Vision Obstacles.

This study is to analyze the influence of visibility in a foggy weather environment on the accuracy of machine vision obstacle detection in assisted driving. We present a foggy day imaging model and analyze the image characteristics, then we set up the faster region convolutional neural network (Faster R-CNN) as the basic network for target detection in the simulation experiment and use Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) data for network detection and classification training. PreScan software is used to build weather and traffic scenes based on a foggy imaging model, and we study object detection of machine vision in four types of weather condition—clear (no fog), light fog, medium fog, and heavy fog—by simulation experiment. The experimental results show that the detection recall is 91.55%, 85.21%, 72.54~64.79%, and 57.75% respectively in no fog, light fog, medium fog, and heavy fog environments. Then we used real scenes in medium fog and heavy fog environment to verify the simulation experiment. Through this study, we can determine the influence of bad weather on the detection results of machine vision, and hence we can improve the safety of assisted driving through further research.

Unsupervised Learning for Depth, Ego-Motion, and Optical Flow Estimation Using Coupled Consistency Conditions

Herein, we propose an unsupervised learning architecture under coupled consistency conditions to estimate the depth, ego-motion, and optical flow. Previously invented learning techniques in computer vision adopted a large amount of the ground truth dataset for network training. A ground truth dataset, including depth and optical flow collected from the real world, requires tremendous effort in pre-processing due to the exposure to noise artifacts. In this paper, we propose a framework that trains networks while using a different type of data with combined losses that are derived from a coupled consistency structure. The core concept is composed of two parts. First, we compare the optical flows, which are estimated from both the depth plus ego-motion and flow estimation network. Subsequently, to prevent the effects of the artifacts of the occluded regions in the estimated optical flow, we compute flow local consistency along the forward–backward directions. Second, synthesis consistency enables the exploration of the geometric correlation between the spatial and temporal domains in a stereo video. We perform extensive experiments on the depth, ego-motion, and optical flow estimation on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset. We verify that the flow local consistency loss improves the optical flow accuracy in terms of the occluded regions. Furthermore, we also show that the view-synthesis-based photometric loss enhances the depth and ego-motion accuracy via scene projection. The experimental results exhibit the competitive performance of the estimated depth and the optical flow; moreover, the induced ego-motion is comparable to that obtained from other unsupervised methods.

Fusing Appearance and Prior Cues for Road Detection

Road detection is a crucial research topic in computer vision, especially in the framework of autonomous driving and driver assistance. Moreover, it is an invaluable step for other tasks such as collision warning, vehicle detection, and pedestrian detection. Nevertheless, road detection remains challenging due to the presence of continuously changing backgrounds, varying illumination (shadows and highlights), variability of road appearance (size, shape, and color), and differently shaped objects (lane markings, vehicles, and pedestrians). In this paper, we propose an algorithm fusing appearance and prior cues for road detection. Firstly, input images are preprocessed by simple linear iterative clustering (SLIC), morphological processing, and illuminant invariant transformation to get superpixels and remove lane markings, shadows, and highlights. Then, we design a novel seed superpixels selection method and model appearance cues using the Gaussian mixture model with the selected seed superpixels. Next, we propose to construct a road geometric prior model offline, which can provide statistical descriptions and relevant information to infer the location of the road surface. Finally, a Bayesian framework is used to fuse appearance and prior cues. Experiments are carried out on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) road benchmark where the proposed algorithm shows compelling performance and achieves state-of-the-art results among the model-based methods.

Development of Stereo Visual Odometry Based on Photogrammetric Feature Optimization

One of the important image processing technologies is visual odometry (VO) technology. VO estimates platform motion through a sequence of images. VO is of interest in the virtual reality (VR) industry as well as the automobile industry because the construction cost is low. In this study, we developed stereo visual odometry (SVO) based on photogrammetric geometric interpretation. The proposed method performed feature optimization and pose estimation through photogrammetric bundle adjustment. After corresponding the point extraction step, the feature optimization was carried out with photogrammetry-based and vision-based optimization. Then, absolute orientation was performed for pose estimation through bundle adjustment. We used ten sequences provided by the Karlsruhe institute of technology and Toyota technological institute (KITTI) community. Through a two-step optimization process, we confirmed that the outliers, which were not removed by conventional outlier filters, were removed. We also were able to confirm the applicability of photogrammetric techniques to stereo visual odometry technology.

An Automatic Key-Frame Selection Method for Monocular Visual Odometry of Ground Vehicle

In response to the rapid change of attitude in complex situations such as sharp turns and camera jitter, visual odometry/visual simultaneous localization and mapping (VO/VSLAM) often requires a high camera frame rate. However, the high frame rate of the camera poses a great challenge to real-time VO/VSLAM, especially when the vehicle travels along a relatively flat trajectory which requires higher real-time performance. Therefore, in this paper, we propose an automatic method for key-frame selection based on the motion state of the vehicle, which can reduce data redundancy and improve the real-time performance and robustness of VO/VSLAM. First, a pyramid-layered Kanade-Lucas-Tomasi (KLT) algorithm is used to track the feature points, and the five-point method and RANSAC are used to calculate the essential matrix. Second, two-step decomposition is used to calculate the change of inter-frame attitude, and then, key-frames are automatically selected based on the motion state of the vehicle predicted by attitude change within a short time interval. To evaluate the method, we conduct extensive experiments on real data and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset based on monocular visual odometry. Then, we made qualitative and quantitative evaluations from the comparison with the reference trajectory, relative error (RE), root mean square error (RMSE), and absolute trajectory error (ATE). The results indicate that our method can improve the real-time performance while ensuring accuracy, with the data redundancy reduced by about 40%-60%. In addition, the performance of our method is further verified by comparison with the current representative ORB-SLAM.

DEVELOPMENT OF A SINGLE-VIEW ODOMETER BASED ON PHOTOGRAMMETRIC BUNDLE ADJUSTMENT

Abstract. Recently, a vehicle is equipped with various sensors, which aim smart and autonomous functions. Single-view odometer estimates its pose using a monoscopic camera mounted on a vehicle. It was generally studied in the field of computer vision. On the other hands, photogrammetry focuses to produce precise three-dimensional position information using bundle adjustment methods. Therefore, this paper proposes to apply photogrammetric approach to single view odometer. Firstly, it performs real-time corresponding point extraction. Next, it estimates the pose using relative orientation based on coplanarity conditions. Then, scale calibration is performed to convert the estimated translation in the model space to the translation in the real space. Finally, absolute orientation is performed using more than three images. In this step, we also extract the appropriate model points through verification procedure. For experiments, we used the data provided by KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) community. This technique took 0.12 seconds of processing time per frame. The rotation estimation error was about 0.005 degree per meter and the translation estimation error was about 6.8 %. The results of this study have shown the applicability of photogrammetry to visual odometry technology.

Robust Stereo Visual Odometry Using Improved RANSAC-Based Methods for Mobile Robot Localization.

In this paper, we present a novel approach for stereo visual odometry with robust motion estimation that is faster and more accurate than standard RANSAC (Random Sample Consensus). Our method makes improvements in RANSAC in three aspects: first, the hypotheses are preferentially generated by sampling the input feature points on the order of ages and similarities of the features; second, the evaluation of hypotheses is performed based on the SPRT (Sequential Probability Ratio Test) that makes bad hypotheses discarded very fast without verifying all the data points; third, we aggregate the three best hypotheses to get the final estimation instead of only selecting the best hypothesis. The first two aspects improve the speed of RANSAC by generating good hypotheses and discarding bad hypotheses in advance, respectively. The last aspect improves the accuracy of motion estimation. Our method was evaluated in the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) and the New Tsukuba dataset. Experimental results show that the proposed method achieves better results for both speed and accuracy than RANSAC.

Plane-based scan registration with moving vehicles exclusion

Moving vehicles have a considerable negative effect on the accuracy of scan registration and lidar odometry. To remove the negative effect, we propose an extended 2D virtual scan to obtain all moving objects in the sensing range of lidar by a scan differencing operation between two consecutive scans. The dynamic objects’ poses are estimated with our proposed likelihood-field-based vehicle measurement model and the motion evidence is utilized to classify the objects as moving vehicles or not. In this way, the moving/dynamic vehicles are detected and the points hitting them are removed. The remaining points are then taken as an input into the alignment.In the registration, we adjust the raw distorted points by modeling the lidar motion as the constant angular and linear velocities within a scan interval, and then exploit the probabilistic framework to model the local plane structure of the matched feature points instead of the original point-to-point mode. The transform is achieved by the combination of coarse motion estimation and fine batch adjustment. The algorithm has been validated by a large set of qualitative tests on our collected point clouds and quantitative comparisons with the excellent methods on the public Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) odometry datasets.

Visual-LiDAR Odometry Aided by Reduced IMU

This paper proposes a method for combining stereo visual odometry, Light Detection And Ranging (LiDAR) odometry and reduced Inertial Measurement Unit (IMU) including two horizontal accelerometers and one vertical gyro. The proposed method starts with stereo visual odometry to estimate six Degree of Freedom (DoF) ego motion to register the point clouds from previous epoch to the current epoch. Then, Generalized Iterative Closest Point (GICP) algorithm refines the motion estimation. Afterwards, forward velocity and Azimuth obtained by visual-LiDAR odometer are integrated with reduced IMU outputs in an Extended Kalman Filter (EKF) to provide final navigation solution. In this paper, datasets from KITTI (Karlsruhe Institute of Technology and Toyota technological Institute) were used to compare stereo visual odometry, integrated stereo visual odometry and reduced IMU, stereo visual-LiDAR odometry and integrated stereo visual-LiDAR odometry and reduced IMU. Integrated stereo visual-LiDAR odometry and reduced IMU outperforms other methods in urban areas with buildings around. Moreover, this method outperforms simulated Reduced Inertial Sensor System (RISS), which uses simulated wheel odometer and reduced IMU. KITTI datasets do not include wheel odometry data. Integrated RTK (Real Time Kinematic) GPS (Global Positioning System) and IMU was replaced by wheel odometer to simulate the response of RISS method. Visual Odometry (VO)-LiDAR is not only more accurate than wheel odometer, but it also provides azimuth aiding to vertical gyro resulting in a more reliable and accurate system. To develop low-cost systems, it would be a good option to use two cameras plus reduced IMU. The cost of such a system will be reduced than using full tactical MEMS (Micro-Electro-Mechanical Sensor) based IMUs because two cameras are cheaper than full tactical MEMS based IMUs. The results indicate that integrated stereo visual-LiDAR odometry and reduced IMU can achieve accuracy at the level of state of art.