Autonomous Vehicle Applications Research Articles

Improving the planarity and sharpness of monocularly estimated depth images using the Phong reflection model

Depth estimation based on a single RGB image (a.k.a. monocular depth estimation) is a challenging task, with applications in robotics, autonomous vehicles, and other areas. With the advance of deep learning, several monocular depth estimation approaches have emerged with remarkable results. However, it is still possible to observe deficiencies in the depth maps generated by current techniques. We present a deep neural network that takes the result produced by some existing monocular depth estimation approach and enhances it by adding the details that the depth map requires to be sharp. We train the proposed Depth Enhancer Neural Network (DENN) using a new loss function that compares the input color image of the scene to the color image produced by rendering it using the Phong reflection model. Our experiments show a clear visual improvement in the sharpness of the depth image produced by the DENN, leading to edge enhancement and regularity of planar surfaces without compromising non-planar objects. On average, the standard metrics show a 5% reduction in error when considering the quantitative results. This reduction is dependent on the initial monocular depth estimation technique used by DENN to train the model.

Reoptimisation strategies for dynamic vehicle routing problems with proximity-dependent nodes

Autonomous vehicles create new opportunities as well as new challenges to dynamic vehicle routing. The introduction of autonomous vehicles as information-collecting agents results in scenarios, where dynamic nodes are found by proximity. This paper presents a novel dynamic vehicle-routing problem variant with proximity-dependent nodes. Here, we introduced a novel variable, detectability, which determines whether a proximal dynamic node will be detected, based on the sight radius of the vehicle. The problem considered is motivated by autonomous weed-spraying vehicles in large agricultural operations. This work is generalisable to many other autonomous vehicle applications. The first step to crafting a solution approach for the problem is to decide when reoptimisation should be triggered. Two reoptimisation trigger strategies are considered—exogenous and endogenous. Computational experiments compared the strategies for both the classical dynamic vehicle routing problem as well as the introduced variant. Experiments used extensive standardised vehicle-routing problem benchmarks with varying degrees of dynamism and geographical node distributions. The results showed that for both the classical problem and the novel variant, an endogenous trigger strategy is better in most cases, while an exogenous trigger strategy is only suitable when both detectability and dynamism are low. Furthermore, the optimal level of detectability was shown to be dependent on the combination of trigger, degree of dynamism, and geographical node distribution, meaning practitioners may determine the required detectability based on the attributes of their specific problem.

An ROI Optimization Method Based on Dynamic Estimation Adjustment Model

An important research direction in the field of traffic light recognition of autonomous systems is to accurately obtain the region of interest (ROI) of the image through the multi-sensor assisted method. Dynamic evaluation of the performance of the multi-sensor (GNSS, IMU, and odometer) fusion positioning system to obtain the optimum size of the ROI is essential for further improvement of recognition accuracy. In this paper, we propose a dynamic estimation adjustment (DEA) model construction method to optimize the ROI. First, according to the residual variance of the integrated navigation system and the vehicle velocity, we divide the innovation into an approximate Gaussian fitting region (AGFR) and a Gaussian convergence region (GCR) and estimate them using variational Bayesian gated recurrent unit (VBGRU) networks and a Gaussian mixture model (GMM), respectively, to obtain the GNSS measurement uncertainty. Then, the relationship between the GNSS measurement uncertainty and the multi-sensor aided ROI acquisition error is deduced and analyzed in detail. Further, we build a dynamic estimation adjustment model to convert the innovation of the multi-sensor integrated navigation system into the optimal ROI size of the traffic lights online. Finally, we use the YOLOv4 model to detect and recognize the traffic lights in the ROI. Based on laboratory simulation and real road tests, we verify the performance of the DEA model. The experimental results show that the proposed algorithm is more suitable for the application of autonomous vehicles in complex urban road scenarios than the existing achievements.

Autonomous Underwater Environment Perceiving and Modeling: An Experimental Campaign With FeelHippo AUV for Forward Looking Sonar-Based Automatic Target Recognition and Data Association

Seabed inspection is one of the most sought-after applications for autonomous underwater vehicles (AUVs). Acoustical sensors, such as side-scan sonars and forward-looking sonars (FLSs), are commonly favored over optical cameras to carry out such a task. Indeed, sonars are not influenced by illumination conditions and can provide high-range data. However, due to the lack of features and low resolution, acoustical images are often hard to interpret with conventional automatic techniques, forcing human operators to analyze thousands of collected images to identify the so-called objects of potential interest (OPIs). In this article, we report the development of an automatic target recognition (ATR) methodology to identify and localize OPIs in FLS imagery. Such detections have been then exploited to realize a virtual world model with the probabilistic multiple hypothesis anchoring data association and model tracking algorithm. Distinct models of convolutional neural networks have been trained with a data set acquired in May 2019 at the Naval Support and Experimentation Centre (Centro di Supporto e Sperimentazione Navale—CSSN) basin in La Spezia, Italy. The ATR strategy has been successfully validated offline with the data gathered in October 2019 in the same site where the seabed targets were replaced and relocated. As regards the world modeling technique, it has been preliminarily tested on a simulated scenario built upon unmanned underwater vehicle Simulator. Finally, both the ATR and world modeling systems were on-field tested in October 2020 at the CSSN basin in a multivehicle architecture by employing an acoustical channel between FeelHippo AUV and an autonomous moving buoy.

Intrusion Attacks on Deep Learning Frameworks Employed in Self-Driving Vehicles

Deep convolutional networks have proven practical for autonomous vehicle applications as deep CNN technology has advanced. There has been a growing vogue for using end-to-end computational methods for the mechanization of vehicular activities. Preliminary studies, though, have demonstrated that deep learning network classifiers are sensitive to adversarial approaches. But, the impact of adversarial strategies on regression problems is not sufficiently known. We propose two white-box direct security breaches targeting progressive self-driving vehicles in this research. A prediction model is used in the navigation mechanism, which receives a picture as feed and returns a steering angle. By altering the input image, we may influence the actions of the automated driving unit. Two different attacks may be launched in practice on CPUs with no need for GPUs. The effectiveness of the threats is demonstrated by trials carried out in Udacity.

Wide-area vehicle trajectory data based on advanced tracking and trajectory splicing technologies: Potentials in transportation research

Trajectory data of road users play a crucial role in transportation engineering and traffic safety research. Previous vehicle trajectory data, including naturalistic driving data, trajectory data captured by drones and traffic cameras, have greatly promoted the research of traffic safety, traffic simulation, and other transportation engineering fields. However, existing technologies in collecting trajectory data are limited their detection range. With the development of the sensing and tracking technologies, it is possible to track the trajectory information of road users at the wide-area level, i.e., the real-time trajectory data of all road users on the entire road. Recently, we published our data sharing platform, Tongji Road Trajectory Sharing (TJRD TS, available online at https://tjrdts.com) Platform, providing our open wide-area vehicle trajectory data for researchers, academics, and road traffic professionals. Advanced vehicle tracking and trajectory splicing technologies, such as millimeter wave radar and edge computing technology, were employed to obtain such wide-area vehicle trajectory data. In this short communication article, we brainstorm ideas of using wide-area vehicle trajectory data in traffic safety research, traffic efficiency research, and connected and autonomous vehicle (CAV) application. We hope this short communication and our published data sharing platform can serve as the steppingstone for those who are interested in road transportation engineering and road safety research using such wide-area vehicle trajectory data.

MMCAN: Multi-Modal Cross-Attention Network for Free-Space Detection with Uncalibrated Hyperspectral Sensors

Free-space detection plays a pivotal role in autonomous vehicle applications, and its state-of-the-art algorithms are typically based on semantic segmentation of road areas. Recently, hyperspectral images have proven useful supplementary information in multi-modal segmentation for providing more texture details to the RGB representations, thus performing well in road segmentation tasks. Existing multi-modal segmentation methods assume that all the inputs are well-aligned, and then the problem is converted to fuse feature maps from different modalities. However, there exist cases where sensors cannot be well-calibrated. In this paper, we propose a novel network named multi-modal cross-attention network (MMCAN) for multi-modal free-space detection with uncalibrated hyperspectral sensors. We first introduce a cross-modality transformer using hyperspectral data to enhance RGB features, then aggregate these representations alternatively via multiple stages. This transformer promotes the spread and fusion of information between modalities that cannot be aligned at the pixel level. Furthermore, we propose a triplet gate fusion strategy, which can increase the proportion of RGB in the multiple spectral fusion processes while maintaining the specificity of each modality. The experimental results on a multi-spectral dataset demonstrate that our MMCAN model has achieved state-of-the-art performance. The method can be directly used on the pictures taken in the field without complex preprocessing. Our future goal is to adapt the algorithm to multi-object segmentation and generalize it to other multi-modal combinations.

A Survey on Model-Based Control and Guidance Principles for Autonomous Marine Vehicles

With the increasing number of applications for both surface and underwater autonomous vehicles, a great amount of control methods and guidance principles has been developed over the years. This work proposes a review of the most common of these methods. It is mainly focused on model-based nonlinear control methods and guidance principles. Notably, this work details examples and variations of model-based linearizing controllers, applications of line of sight guidance, sliding mode controllers and several other less common control methods for both fully-actuated and underactuated vehicles. Additionally, this work proposes an alternative definition of underactuation with respect to the task allowing for a better understanding of the consequences of underactuation on control. Comparison of fully-actuated and underactuated cases shows how control laws can be used to solve the problems of underactuation and what mechanisms can be used to compensate for the lack of actuation on a degree of freedom. The reviewed methods are compared and discussed with respect to their capabilities, limitations and suitability for typical tasks.

Continual Object Detection: A review of definitions, strategies, and challenges

The field of Continual Learning investigates the ability to learn consecutive tasks without losing performance on those previously learned. The efforts of researchers have been mainly focused on incremental classification tasks. Yet, we believe that continual object detection deserves even more attention due to its vast range of applications in robotics and autonomous vehicles. This scenario is also more complex than conventional classification, given the occurrence of instances of classes that are unknown at the time but can appear in subsequent tasks as a new class to be learned, resulting in missing annotations and conflicts with the background label. In this review, we analyze the current strategies proposed to tackle the problem of class-incremental object detection. Our main contributions are: (1) a short and systematic review of the methods that propose solutions to traditional incremental object detection scenarios; (2) A comprehensive evaluation of the existing approaches using a new metric to quantify the stability and plasticity of each technique in a standard way; (3) an overview of the current trends within continual object detection and a discussion of possible future research directions.

Deep Dense Network-Based Curriculum Reinforcement Learning for High-Speed Overtaking

Reinforcement learning methods have promising applications in autonomous vehicles, such as lane keeping, adaptive cruise control, and overtaking. However, long-time sequential decision making in complicated scenarios, such as continuous high-speed overtaking, remains challenge. One such challenge is that current neural networks in reinforcement learning employ a shallow neural network, which has limitations in feature extractions. To address this, we propose a deep actor–critic network framework using multilayer dense-connection networks to obtain efficient extracted features. To be specific, we employ a deep dense architecture in the framework of soft actor–critic, which has multiple hidden layers serving as feature extractions. To tackle the vanishing or exploding gradient problem, we reuse features and design shortcut connections, as well as feature-concatenating operations between hidden layers. Meanwhile, when learning multitask high-speed overtaking as one whole progression, the agent has to learn driving, lane changing, and avoiding other vehicles at the same time. Therefore, another challenge is that sparse rewards worsen the speed of convergence. We design a task-level sequencing curriculum reinforcement learning method to tackle the reward sparseness. It breaks down a complex task into two successive subtasks: the first aims to drive as fast as possible in a single-vehicle model; and the second aims to overtake other vehicles without any collisions as soon as possible according to the knowledge gained from the first task in a multivehicle model. Finally, we evaluate our method using a racing car simulator with different tracks. In a comparison using the standard actor–critic algorithm in three different multivehicle scenarios, the best median results show that our deep actor–critic network framework reduces overtaking distance and overtaking time by up to 4.9% and 8.7%, respectively.

Exact Set-valued Estimation using Constrained Convex Generators for uncertain Linear Systems

Set-valued state estimation when in the presence of uncertainties in the model have been addressed in the literature essentially following three main approaches: i) interval arithmetic of the uncertain dynamics with the estimates; ii) factorizing the uncertainty into matrices with unity rank; and, iii) performing the convex hull for the vertices of the uncertainty space. Approach i) and ii) introduce a lot of conservatism because both disregard the relationship of the parameters with the entries of the dynamics matrix. On the other hand, approach iii) has a large growth on the number of variables required to represent the set or is approximated losing its main advantage in comparison with i) and ii). In this paper, with the application of autonomous vehicles in GPS-denied areas that resort to beacon signals for localization, we develop an exact (meaning no added conservatism) and optimal (smallest growth in the number of variables) closed-form definition for the convex hull of Convex Constrained Generators (CCGs). This results in a more efficient method to represent the minimum volume convex set corresponding to the state estimation. Given that reductions methods are still lacking in the literature for CCGs, we employ an approximation using ray-shooting that is comparable in terms of accuracy with methods for Constrained Zonotopes as the ones implemented in CORA. Simulations illustrate the greater accuracy of CCGs with the proposed convex hull operation in comparison to Constrained Zonotopes.

Recipient-Dependent Last-Mile Delivery Routing with Autonomous Vehicle Applications

Recipient-Dependent Last-Mile Delivery Routing with Autonomous Vehicle Applications

Safety-Critical Trajectory Generation and Tracking Control of Autonomous Underwater Vehicles

Safety-critical control is crucial but difficult in the applications of autonomous underwater vehicles (AUVs). This article proposes a novel hierarchical safety-critical framework for the control of AUVs consisting of waypoint-based optimal trajectory generation and tracking. The objective is to generate a smooth and feasible trajectory to conform to the dynamics of the AUV and then control the AUV to operate in a safe region. To this end, the offline trajectory generator and the online controller are designed. In the offline step, an improved minimum snap polynomial trajectory is generated as the tracking reference. Specifically, the trajectory is optimized by explicitly considering the practical constraints of the AUV's actuator, which alleviates the complexity of the controller design and computational burden compared with the real-time nonlinear optimization. In the online step, the time- and state-dependent high-order control barrier functions are incorporated into optimization through quadratic programming (QP) that modifies the backstepping-based nominal controller in a minimally invasive way. As a supplement and adjustment to the offline planning, the online step ensures that the system states are maintained in the safety set, thus ensuring obstacle avoidance. The online computational efficient QP structure guarantees convenience and scalability in practical implementation. Both the online static and dynamic obstacle avoidance simulations demonstrate the adherence to the safety constraints. Experimental results validate the effectiveness of the proposed method.

Multistep Ahead Prediction of Vehicle Lateral Dynamics Based on Echo State Model

Lateral dynamics are critical for high-level autonomous driving, and multistep forecasting can significantly improve vehicle safety. However, the dynamics prediction method is rarely announced due to the difficulty of accurate prediction. In contrast to conventional time-series prediction, this article proposes a novel method for predicting a dynamical system with implicit parameters. Specifically, we construct an echo state model (ESM) incorporating a rational data organization and an efficient neural network structure. We observe the fading information property (FIP) of dynamic vehicle parameters and use temporal data to construct equivalent states for implicit parameters. Finally, a gated recurrent unit (GRU) neural network is designed and trained to accomplish the complex mapping of a vehicle dynamical system. Its performance is compared to that of several benchmark networks. The proposed method is capable of performing four-step-ahead prediction with high accuracy in both dry and slippery road conditions, implying the increased potential for some safe and comfortable driving applications of autonomous vehicles.

Motion Optimization and Control of Single and Multiple Autonomous Aerial, Land, and Marine Robots

Fast-paced developments in the fields of aerial, land, and marine robotics are steadily paving the way for a wide spectrum of scientific and commercial applications of autonomous vehicles with far-reaching societal implications [...].

A Novel Compact High Gain Wide-Band Log Periodic Dipole Array Antenna for Wireless Communication Systems

This paper implies a prototype for a broad-band log periodic dipole antenna (LPDA) with dimensions of 20 × 40 × 0.508 mm3 equipped with directors. The LPDA is based on Roger’s RT5880 with εr = 2.2. The 4-arms with alternative stubs constitute the LPDA, which is evenly spaced on both lines. At the substrate’s back, the 50Ω main feeder line is placed on partial ground. The directors are also incorporated and studied in a regular matrix at a predetermined distance to improve gain, further reduce side lobes, and widen the frequency band. The performance of the antenna covers a wide spectrum of millimeter waves ranging from 26 up to 44 GHz. The realized gain for the antenna is 8.97, 11.96, 13.96, and 14.29 dB at 28 GHz, 35 GHz, 38 GHz, and 43 GHz, respectively. The total gain is 14.29 dB as a peak gain. The antenna was designed and tested for 5G communication applications in autonomous vehicles, and the design cost for the antenna is low in comparison to dielectric lenses. The model exhibits symmetrical radiation patterns for the antenna range. The results of the simulations and the testing of the implemented antenna elements coincide quite well. The proposed design outperforms earlier work in this field.

Communication-Constrained Routing and Traffic Control: A Framework for Infrastructure-Assisted Autonomous Vehicles

With the increasing demand for advanced autonomous driving, the available communication resources may become constrained over different geographic areas. In addition, due to dynamic channel variations and imperfect cell deployments, guaranteeing the required communication resources for data hungry and delay-sensitive applications in autonomous vehicles (AVs), along their entire trips, becomes challenging. To address these issues, the paper investigates the feasibility of a hybrid system-optimum and user-equilibrium AV traffic framework subject to communication constraints, as well as its performance gain. Within such a framework, the paper introduces the problems of communication-constrained routing (CCR) and traffic control (CCTC) in the context of infrastructure-assisted autonomous driving and presents respective solutions. For CCR, an efficient two-layered routing scheme is proposed which can provide optimal trip duration. Simulation results show that the routing scheme achieves a good balance between longer duration of communication coverage and acceptable source-to-destination travel time. For CCTC, it is shown that there exists an optimal AV speed on each road segment, as well as an optimal inter-AV distance and an optimal number of AVs in each cell, to maximize the road-network AV throughput within a single cell. Moreover, spectrum allocation is used to achieve Pareto-optimal road-network throughput across cells, and a new key performance index (KPI) is defined to evaluate the traffic control capability of cellular systems. Simulation results validate the improvement of AV throughput via the proposed CCTC solution.

Low-Cost Real-Time PPP GNSS Aided INS for CAV Applications

Many connected and autonomous vehicle (CAV) applications benefit from navigation technologies that reliably achieve lane-level accuracy. Global organizations have recently begun to provide real-time common-mode error corrections for Global Navigation Satellite Systems (GNSS) that enable this level of positioning accuracy using Precise Point Positioning (PPP). Incorporating an Inertial Measurement Unit (IMU) with a GNSS receiver can achieve this positioning performance reliably on moving platforms while providing a full state estimate, with high bandwidth at a high sampling rate. For commercial automotive applications cost is critical; therefore, this article focuses on commercial grade IMU’s and single-frequency GNSS code and Doppler measurements. This article presents and demonstrates a tightly-coupled PPP GNSS aided Inertial Navigation System (INS) using only publicly available real-time data sources. The article discusses the role of the INS in automotive applications and how the quality of the IMU affects the ability to achieve that role. The experimental results demonstrate positioning accuracy that surpasses the Society of Automotive Engineering (SAE) J2945 specification (horizontal error ≤1.5 m and vertical error ≤ 3 m 68%).

Causal Temporal–Spatial Pedestrian Trajectory Prediction With Goal Point Estimation and Contextual Interaction

Forecasting pedestrian trajectories in complex dynamic environments is highly critical for the application of autonomous vehicles and robots. Accordingly, this paper proposes a novel pedestrian trajectory prediction model called CTSGI, which, utilizes self-attention mechanism to construct an interactive graph between pedestrians and their neighbors based on their spatial relationship, to model the crowd’s interaction. At the same time, it uses self-attention to extract the temporal dependence for a single pedestrian. In order to effectively model the interaction between the pedestrian and the context, such as where a pedestrian can walk or approach, the semantic segmentation of background image is utilized. CTSGI estimates the goal points of pedestrian and their neighbors to assist in predicting the future trajectory. In addition, the causal structure model is used to analyze the confounding factors existing in the encoding stage, and Do-calculus is introduced for eliminating the confounding impact to improve the prediction performance. Moreover, extensive experiments are conducted for the proposed model on ETH and UCY datasets, and clearly, the experimental results reveal that the model reported herein outperforms the comparative state-of-the-art methods by 27.59% in Average Displacement Error (ADE) and 8.33% in Final Displacement Error (FDE). Furthermore, visualization of attention indicates that our model can capture the interaction between some specific pedestrian and their neighbors better.

Legal and Ethical Concerns of Adopting Digital Solution for Smart Metropolises

The advent of technologies has created a need for a system that can provide better management and application of smart technologies in smart cities. The paper attempts to provide a review of legal and ethical considerations in adopting digital solutions for smart cities. There are challenges in terms of legal and ethical issues regarding the application of autonomous and near-autonomous vehicles, robotics technology, three-dimensional (3D) printing, mobile technology, and the Internet of Things that act as a primary or secondary source of new regulatory and legal challenges. The paper addresses the concerns through an extensive literature review and suggests a roadmap for better adoption of smart technologies in smart cities. The legal and ethical implications are observed in the examples of the digital payment, waste management, and contractual systems. This study focuses on the legal and ethical issues arising in the application of smart technologies in smart cities. The methodology focuses on the analysis of the literature review to derive some critical points to explain the impact of legal and ethical issues on smart city applications. The legal and ethical concerns are studied in terms of use of data, application of technology, proper functioning of technology, misuse of the data, commercialization of the data, and unwanted calls.