Toyota Technological Institute Research Articles

Kinematic and Dynamic Vehicle Model-Assisted Global Positioning Method for Autonomous Vehicles with Low-Cost GPS/Camera/In-Vehicle Sensors.

Real-time, precise and low-cost vehicular positioning systems associated with global continuous coordinates are needed for path planning and motion control in autonomous vehicles. However, existing positioning systems do not perform well in urban canyons, tunnels and indoor parking lots. To address this issue, this paper proposes a multi-sensor positioning system that combines a global positioning system (GPS), a camera and in-vehicle sensors assisted by kinematic and dynamic vehicle models. First, the system eliminates image blurring and removes false feature correspondences to ensure the local accuracy and stability of the visual simultaneous localisation and mapping (SLAM) algorithm. Next, the global GPS coordinates are transferred to a local coordinate system that is consistent with the visual SLAM process, and the GPS and visual SLAM tracks are calibrated with the improved weighted iterative closest point and least absolute deviation methods. Finally, an inverse coordinate system conversion is conducted to obtain the position in the global coordinate system. To improve the positioning accuracy, information from the in-vehicle sensors is fused with the interacting multiple-model extended Kalman filter based on kinematic and dynamic vehicle models. The developed algorithm was verified via intensive simulations and evaluated through experiments using KITTI benchmarks (A project of Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago) and data captured using our autonomous vehicle platform. The results show that the proposed positioning system improves the accuracy and reliability of positioning in environments in which the Global Navigation Satellite System is not available. The developed system is suitable for the positioning and navigation of autonomous vehicles.

Towards depth estimation in a single aerial image

ABSTRACT Nowadays, depth estimation from a single image is a task that has been successfully addressed by Convolutional Neural Network (CNN) architectures. In this regard, several authors have taken advantage of depth datasets publicly available to the scientific community to train their CNN-based methods. From a project of Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago has emerged KITTI (acronym derived from the institutions' names) as one of the most popular public datasets providing depth estimates associated to RGB (Red, Green, Blue) images. Regarding the depth data in KITTI and typically in many other datasets, these include monocular or stereo RGB images associated with depth images obtained via laser, stereo cameras or a combination of both. These images and depth data have been collected by driving around outdoor urban environments with cameras looking forward to the horizon. In contrast, in this work, we are interested in CNN-based depth estimation in a single aerial image for which depth datasets are not available. In addition, popular CNN architectures for depth estimation in a single-image struggle to estimate depth in aerial scenes due to the fact that the camera angle and object appearance in aerial imagery are significantly different. Nevertheless, we propose to harvest the depth information available in KITTI in order to tackle the problem of depth estimation in a single aerial image. To this end, our approach is a two-step methodology based on patch processing that is later used as input for a set of proposed CNN architectures. Our results indicate that this approach is promising, and those datasets such as KITTI may indeed be exploited in other domains, especially where the data acquisition may be expensive or difficult to be carried out such as for aerial scenes.

Road detection using cycle-consistent adversarial networks

Today, road detection is still a challenging task in intelligent driving. With the continuous improvement of computer vision, many methods of deep learning are used for road detection because they can achieve image features at a deeper level and discover road areas from raw RGB data. However, the method to detect the road areas accurately needs to be improved. We present a method that can extract the road area features and complete road detection tasks. Our method mainly includes the following points: (1) to introduce the cycle-consistent adversarial network to extract the road area features in a picture and complete image to image conversion and (2) to complete road detection by adding a new model and to improve the accuracy of detection. The results of our method are evaluated by uploading to the Karlsruhe Institute of Technology and Toyota Technological Institute road detection benchmark and named it as “road detection cycle-consistent adversarial networks.” Our method achieves an overall max F -measure of 88.63% and precision of 91.35%. In addition to high precision, our method also has a good robustness. Meanwhile, the accuracy for narrow road areas needs to be optimized in the future.

Unsupervised Learning for Depth, Ego-Motion, and Optical Flow Estimation Using Coupled Consistency Conditions

Herein, we propose an unsupervised learning architecture under coupled consistency conditions to estimate the depth, ego-motion, and optical flow. Previously invented learning techniques in computer vision adopted a large amount of the ground truth dataset for network training. A ground truth dataset, including depth and optical flow collected from the real world, requires tremendous effort in pre-processing due to the exposure to noise artifacts. In this paper, we propose a framework that trains networks while using a different type of data with combined losses that are derived from a coupled consistency structure. The core concept is composed of two parts. First, we compare the optical flows, which are estimated from both the depth plus ego-motion and flow estimation network. Subsequently, to prevent the effects of the artifacts of the occluded regions in the estimated optical flow, we compute flow local consistency along the forward–backward directions. Second, synthesis consistency enables the exploration of the geometric correlation between the spatial and temporal domains in a stereo video. We perform extensive experiments on the depth, ego-motion, and optical flow estimation on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset. We verify that the flow local consistency loss improves the optical flow accuracy in terms of the occluded regions. Furthermore, we also show that the view-synthesis-based photometric loss enhances the depth and ego-motion accuracy via scene projection. The experimental results exhibit the competitive performance of the estimated depth and the optical flow; moreover, the induced ego-motion is comparable to that obtained from other unsupervised methods.

Fusing Appearance and Prior Cues for Road Detection

Road detection is a crucial research topic in computer vision, especially in the framework of autonomous driving and driver assistance. Moreover, it is an invaluable step for other tasks such as collision warning, vehicle detection, and pedestrian detection. Nevertheless, road detection remains challenging due to the presence of continuously changing backgrounds, varying illumination (shadows and highlights), variability of road appearance (size, shape, and color), and differently shaped objects (lane markings, vehicles, and pedestrians). In this paper, we propose an algorithm fusing appearance and prior cues for road detection. Firstly, input images are preprocessed by simple linear iterative clustering (SLIC), morphological processing, and illuminant invariant transformation to get superpixels and remove lane markings, shadows, and highlights. Then, we design a novel seed superpixels selection method and model appearance cues using the Gaussian mixture model with the selected seed superpixels. Next, we propose to construct a road geometric prior model offline, which can provide statistical descriptions and relevant information to infer the location of the road surface. Finally, a Bayesian framework is used to fuse appearance and prior cues. Experiments are carried out on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) road benchmark where the proposed algorithm shows compelling performance and achieves state-of-the-art results among the model-based methods.

A 3D Object Detection Based on Multi-Modality Sensors of USV

Unmanned Surface Vehicles (USVs) are commonly equipped with multi-modality sensors. Fully utilized sensors could improve object detection of USVs. This could further contribute to better autonomous navigation. The purpose of this paper is to solve the problems of 3D object detection of USVs in complicated marine environment. We propose a 3D object detection Depth Neural Network based on multi-modality data of USVs. This model includes a modified Proposal Generation Network and Deep Fusion Detection Network. The Proposal Generation Network improves feature extraction. Meanwhile, the Deep Fusion Detection Network enhances the fusion performance and can achieve more accurate results of object detection. The model was tested on both the KITTI 3D object detection dataset (A project of Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago) and a self-collected offshore dataset. The model shows excellent performance in a small memory condition. The results further prove that the method based on deep learning can give good accuracy in conditions of complicated surface in marine environment.

FRED-Net: Fully residual encoder–decoder network for accurate iris segmentation

Iris recognition is now developed enough to recognize a person from a distance. The process of iris segmentation plays a vital role in maintaining the accuracy of the iris-based recognition systems by limiting the errors at the current stage. However, its performance is affected by non-ideal situations created by environmental light noise and user non-cooperation. The existing local feature-based segmentation methods are unable to find the true iris boundary in these non-ideal situations, and the error created at the segmentation stage traverses to all the subsequent stages, which results in reduced accuracy and reliability. In addition, it is necessary to segment the true iris boundary without the extra cost of denoising as preprocessing. To overcome these challenging issues during iris segmentation, a deep learning-based fully residual encoder–decoder network (FRED-Net) is proposed to determine the true iris region with the flow of high-frequency information from the preceding layers via residual skip connection.The main four impacts and significances of this study are as follows. First, FRED-Net is an end-to-end semantic segmentation network that does not use conventional image processing schemes, and does not have a preprocessing overhead. It is a standalone network in which eyelid, eyelash, and glint detections are not required to obtain the true iris boundary. Second, the proposed FRED-Net is the final resultant structure of a step-by-step development, and in each step, a new complete variant network is created for semantic segmentation considering the detailed descriptions of the networks. Third, FRED-Net uses the residual connectivity between convolutional layers by the residual shortcut for both encoder and decoder, which enables a high-frequency component to flow through the network and achieve higher accuracy with few layers. Fourth, the performance of the proposed FRED-Net is tested with five different iris datasets under visible and NIR light environments, and two general road scene segmentation datasets. To achieve fair comparisons with other studies, our trained FRED-Net models, along with the algorithms, are made publicly available through our website (Dongguk FRED-Net Model with Algorithm. accessed on 16 May 2018).The experiments include two datasets: Noisy Iris Challenge Evaluation – Part II (NICE-II) selected from the UBIRIS.v2 database and Mobile Iris Challenge Evaluation (MICHE-I), for the visible light environment and three datasets: Institute of Automation, Chinese Academy of Sciences (CASIA) v4.0 interval, v4.0 distance, and IIT Delhi v1.0, for the near-infrared (NIR) light environment. Moreover, to evaluate the performance of the proposed network in general segmentation, experiments with two famous road scene segmentation datasets: Cambridge-driving Labeled Video Database (CamVid) and Karlsruhe Institute of Technology and Toyota Technological Institute at Chicago (KITTI), are included. The experimental results showed the optimum performance of the proposed FRED-Net on the above-mentioned seven datasets of iris and general road scene segmentation.

Development of Stereo Visual Odometry Based on Photogrammetric Feature Optimization

One of the important image processing technologies is visual odometry (VO) technology. VO estimates platform motion through a sequence of images. VO is of interest in the virtual reality (VR) industry as well as the automobile industry because the construction cost is low. In this study, we developed stereo visual odometry (SVO) based on photogrammetric geometric interpretation. The proposed method performed feature optimization and pose estimation through photogrammetric bundle adjustment. After corresponding the point extraction step, the feature optimization was carried out with photogrammetry-based and vision-based optimization. Then, absolute orientation was performed for pose estimation through bundle adjustment. We used ten sequences provided by the Karlsruhe institute of technology and Toyota technological institute (KITTI) community. Through a two-step optimization process, we confirmed that the outliers, which were not removed by conventional outlier filters, were removed. We also were able to confirm the applicability of photogrammetric techniques to stereo visual odometry technology.

An Automatic Key-Frame Selection Method for Monocular Visual Odometry of Ground Vehicle

In response to the rapid change of attitude in complex situations such as sharp turns and camera jitter, visual odometry/visual simultaneous localization and mapping (VO/VSLAM) often requires a high camera frame rate. However, the high frame rate of the camera poses a great challenge to real-time VO/VSLAM, especially when the vehicle travels along a relatively flat trajectory which requires higher real-time performance. Therefore, in this paper, we propose an automatic method for key-frame selection based on the motion state of the vehicle, which can reduce data redundancy and improve the real-time performance and robustness of VO/VSLAM. First, a pyramid-layered Kanade-Lucas-Tomasi (KLT) algorithm is used to track the feature points, and the five-point method and RANSAC are used to calculate the essential matrix. Second, two-step decomposition is used to calculate the change of inter-frame attitude, and then, key-frames are automatically selected based on the motion state of the vehicle predicted by attitude change within a short time interval. To evaluate the method, we conduct extensive experiments on real data and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset based on monocular visual odometry. Then, we made qualitative and quantitative evaluations from the comparison with the reference trajectory, relative error (RE), root mean square error (RMSE), and absolute trajectory error (ATE). The results indicate that our method can improve the real-time performance while ensuring accuracy, with the data redundancy reduced by about 40%-60%. In addition, the performance of our method is further verified by comparison with the current representative ORB-SLAM.

DEVELOPMENT OF A SINGLE-VIEW ODOMETER BASED ON PHOTOGRAMMETRIC BUNDLE ADJUSTMENT

Abstract. Recently, a vehicle is equipped with various sensors, which aim smart and autonomous functions. Single-view odometer estimates its pose using a monoscopic camera mounted on a vehicle. It was generally studied in the field of computer vision. On the other hands, photogrammetry focuses to produce precise three-dimensional position information using bundle adjustment methods. Therefore, this paper proposes to apply photogrammetric approach to single view odometer. Firstly, it performs real-time corresponding point extraction. Next, it estimates the pose using relative orientation based on coplanarity conditions. Then, scale calibration is performed to convert the estimated translation in the model space to the translation in the real space. Finally, absolute orientation is performed using more than three images. In this step, we also extract the appropriate model points through verification procedure. For experiments, we used the data provided by KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) community. This technique took 0.12 seconds of processing time per frame. The rotation estimation error was about 0.005 degree per meter and the translation estimation error was about 6.8 %. The results of this study have shown the applicability of photogrammetry to visual odometry technology.

Robust Stereo Visual Odometry Using Improved RANSAC-Based Methods for Mobile Robot Localization.

In this paper, we present a novel approach for stereo visual odometry with robust motion estimation that is faster and more accurate than standard RANSAC (Random Sample Consensus). Our method makes improvements in RANSAC in three aspects: first, the hypotheses are preferentially generated by sampling the input feature points on the order of ages and similarities of the features; second, the evaluation of hypotheses is performed based on the SPRT (Sequential Probability Ratio Test) that makes bad hypotheses discarded very fast without verifying all the data points; third, we aggregate the three best hypotheses to get the final estimation instead of only selecting the best hypothesis. The first two aspects improve the speed of RANSAC by generating good hypotheses and discarding bad hypotheses in advance, respectively. The last aspect improves the accuracy of motion estimation. Our method was evaluated in the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) and the New Tsukuba dataset. Experimental results show that the proposed method achieves better results for both speed and accuracy than RANSAC.

Accurate Image-Guided Stereo Matching With Efficient Matching Cost and Disparity Refinement

Stereo matching is a challenging problem, and high-accuracy stereo matching is still required in various computer vision applications, e.g., 3-D scanning, autonomous navigation, and 3-D reconstruction. Therefore, we present a novel image-guided stereo matching algorithm, which employs the efficient combined matching cost and multistep disparity refinement, to improve the accuracy of existing local stereo matching algorithms. Different from all the other methods, we introduce a guidance image for the whole algorithm. This filter-based guidance image is generated by extracting the enhanced information from the raw stereo image. The combined matching cost consists of the novel double-RGB gradient, the improved lightweight census transform, and the image color. This cost measurement is robust against image noise and textureless regions in computing the matching cost. Furthermore, a new systemic multistep refinement process, which includes outlier classification, four-direction propagation, leftmost propagation, and an exponential step filter, is proposed to remove the outliers in the raw disparity map. Experiments on the Middlebury benchmark demonstrate our algorithm's superior performance that it ranks first among the 158 submitted algorithms. Moreover, the proposed method is also robust on the 30 Middlebury data sets and the real-world Karlsruhe Institute of Technology and Toyota Technological Institute benchmark.

Plane-based scan registration with moving vehicles exclusion

Moving vehicles have a considerable negative effect on the accuracy of scan registration and lidar odometry. To remove the negative effect, we propose an extended 2D virtual scan to obtain all moving objects in the sensing range of lidar by a scan differencing operation between two consecutive scans. The dynamic objects’ poses are estimated with our proposed likelihood-field-based vehicle measurement model and the motion evidence is utilized to classify the objects as moving vehicles or not. In this way, the moving/dynamic vehicles are detected and the points hitting them are removed. The remaining points are then taken as an input into the alignment.In the registration, we adjust the raw distorted points by modeling the lidar motion as the constant angular and linear velocities within a scan interval, and then exploit the probabilistic framework to model the local plane structure of the matched feature points instead of the original point-to-point mode. The transform is achieved by the combination of coarse motion estimation and fine batch adjustment. The algorithm has been validated by a large set of qualitative tests on our collected point clouds and quantitative comparisons with the excellent methods on the public Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) odometry datasets.

Noise robust image matching using adjacent evaluation census transform and wavelet edge joint bilateral filter in stereo vision

Automation application systems based on stereo vision require robust image matching methods to achieve available depth image information. This paper presents a novel noise robust stereo matching using adjacent evaluation census transform and wavelet edge joint bilateral filter. The adjacent evaluation census is firstly proposed to improve the robustness against noise of the census transform. Meanwhile, two different and complementary types of metrics are extracted (the adjacent evaluation census mean and the adjacent evaluation census weighted difference). Moreover, the weighted template is composed of four different directions. Then, to improve the robustness of cost aggregation and disparity optimization, the random walk is integrated into the proposed stereo matching method. Additionally, a disparity map post-processing method named wavelet edge joint bilateral filter is employed to eliminate error regions. An obtained wavelet-based edge image is considered as an important weighted coefficient to guide the post-processing. Experimental results demonstrate that the proposed method presents the best performance of the robustness against noise on the Middlebury dataset. Even in the toughest situation with additive Gaussian noise, our method can still achieve the moderate disparity map. In addition, the wider applicability of the proposed method is demonstrated on the KITTI (i.e., Karlsruhe Institute of Technology (KIT) and Toyota Technological Institute at Chicago (TTI-C)) dataset and some typical real-world sequences.

Visual-LiDAR Odometry Aided by Reduced IMU

This paper proposes a method for combining stereo visual odometry, Light Detection And Ranging (LiDAR) odometry and reduced Inertial Measurement Unit (IMU) including two horizontal accelerometers and one vertical gyro. The proposed method starts with stereo visual odometry to estimate six Degree of Freedom (DoF) ego motion to register the point clouds from previous epoch to the current epoch. Then, Generalized Iterative Closest Point (GICP) algorithm refines the motion estimation. Afterwards, forward velocity and Azimuth obtained by visual-LiDAR odometer are integrated with reduced IMU outputs in an Extended Kalman Filter (EKF) to provide final navigation solution. In this paper, datasets from KITTI (Karlsruhe Institute of Technology and Toyota technological Institute) were used to compare stereo visual odometry, integrated stereo visual odometry and reduced IMU, stereo visual-LiDAR odometry and integrated stereo visual-LiDAR odometry and reduced IMU. Integrated stereo visual-LiDAR odometry and reduced IMU outperforms other methods in urban areas with buildings around. Moreover, this method outperforms simulated Reduced Inertial Sensor System (RISS), which uses simulated wheel odometer and reduced IMU. KITTI datasets do not include wheel odometry data. Integrated RTK (Real Time Kinematic) GPS (Global Positioning System) and IMU was replaced by wheel odometer to simulate the response of RISS method. Visual Odometry (VO)-LiDAR is not only more accurate than wheel odometer, but it also provides azimuth aiding to vertical gyro resulting in a more reliable and accurate system. To develop low-cost systems, it would be a good option to use two cameras plus reduced IMU. The cost of such a system will be reduced than using full tactical MEMS (Micro-Electro-Mechanical Sensor) based IMUs because two cameras are cheaper than full tactical MEMS based IMUs. The results indicate that integrated stereo visual-LiDAR odometry and reduced IMU can achieve accuracy at the level of state of art.

2506 ペルソナの感性ワード選択による感性の定量化と製品設計への適用(OS11-2 感性と設計II,OS11 感性と設計)

Due to rise of our standards of living, people come to request not a quantitative satisfaction but a qualitative satisfaction from the product in recent years. In order to solve this problem, we consider the method that is to understand customers' needs and the sensibility by making the persona, and using neural network. The persona can be understood as an image of customer, and this is used by some companies to grasp customers' needs. The neural network is used to decide to feature of products. Its learning method is a backpropagation. In this paper, a product object is a cellular phone, and we take surveys about cellular phone on students of Toyota technological institute. Based on these surveys, we make a persona, and make a decision about features of a cellular phone. We confirm the effectiveness of this method based on this example.

9732090 Effects of EGR on direct injection gasoline combustion Shizuo Sasaki, Daisaku Sawada, Takanori Ueda (Toyota Motor Corporation), Hiroyuki Sami (Toyota Technological Institute)

9732090 Effects of EGR on direct injection gasoline combustion Shizuo Sasaki, Daisaku Sawada, Takanori Ueda (Toyota Motor Corporation), Hiroyuki Sami (Toyota Technological Institute)

Re-Education of Engineers in Toyota Technological Institute

Re-Education of Engineers in Toyota Technological Institute