Advanced Driver Assistance Systems Applications Research Articles

Sustainable edge computing: Challenges and future directions

AbstractThe advent of edge computing holds immense promise for advancing the digitization of society, ushering in critical applications that elevate the overall quality of life. Yet, the practical implementation of the edge paradigm proves more challenging than anticipated, encountering disruptions primarily due to the constraints of applying conventional cloud‐based strategies at the network's periphery. Increasingly influenced by sustainability commitments, industry regulations currently view edge computing as a potential threat, primarily due to the energy inefficiency of solutions situated in close proximity to data generation sources and the rising density of computing. This paper presents a proactive strategy to transform the perceived threat into an opportunity, steering the sustainable evolution of future edge infrastructures to make them both environmentally and economically competitive for accelerated adoption. The vision outlined addresses key challenges associated with edge deployment and operation, emphasizing energy efficiency, fault‐tolerant automation, and collaborative orchestration. The proposed approach integrates two‐phase immersion cooling, formal modeling, machine learning, and federated management to effectively harness heterogeneity, propelling the sustainability of edge computing. To substantiate the efficacy of this approach, the paper details initial efforts towards establishing the sustainability of an edge infrastructure designed for an Advanced Driver Assistance Systems application.

On-chip hyperspectral image segmentation with fully convolutional networks for scene understanding in autonomous driving

Most of current computer vision-based advanced driver assistance systems (ADAS) perform detection and tracking of objects quite successfully under regular conditions. However, under adverse weather and changing lighting conditions, and in complex situations with many overlapping objects, these systems are not completely reliable. The spectral reflectance of the different objects in a driving scene beyond the visible spectrum can offer additional information to increase the reliability of these systems, especially under challenging driving conditions. Furthermore, this information may be significant enough to develop vision systems that allow for a better understanding and interpretation of the whole driving scene. In this work we explore the use of snapshot, video-rate hyperspectral imaging (HSI) cameras in ADAS on the assumption that the near infrared (NIR) spectral reflectance of different materials can help to better segment the objects in real driving scenarios. To do this, we have used the HSI-Drive 1.1 dataset to perform various experiments on spectral classification algorithms. However, the information retrieval of hyperspectral recordings in natural outdoor scenarios is challenging, mainly because of deficient color constancy and other inherent shortcomings of current snapshot HSI technology, which poses some limitations to the development of pure spectral classifiers. In consequence, in this work we analyze to what extent the spatial features codified by standard, tiny fully convolutional network (FCN) models can improve the performance of HSI segmentation systems for ADAS applications. In order to be realistic from an engineering viewpoint, this research is focused on the development of a feasible HSI segmentation system for ADAS, which implies considering implementation constraints and latency specifications throughout the algorithmic development process. For this reason, it is of particular importance to include the study of the raw image preprocessing stage into the data processing pipeline. Accordingly, this paper describes the development and deployment of a complete machine learning-based HSI segmentation system for ADAS, including the characterization of its performance on different embedded computing platforms, including a single board computer, an embedded GPU SoC and a programmable system on chip (PSoC) with embedded FPGA. We verify the superiority of the FPGA-PSoC over the GPU-SoC in terms of energy consumption and, particularly, processing latency, and demonstrate that it is feasible to achieve segmentation speeds within the range of ADAS industry specifications using standard development tools.

A Declarative Application Framework for Evaluating Advanced V2X-Based ADAS Solutions

Application of Vehicular Ad Hoc Networks (VANETs) aims to help in the solution of some problems that have arisen in road transportation systems via short-range, low-latency mobile communication. The application of V2X (Vehicle-to-Everything) communication technologies to the next generation of Advanced Driver Assistance Systems (ADAS) is essential to the extension of the operational design domain (ODD) of the systems to provide safe, secure, and efficient automated driving solutions. Due to the safety-critical nature of the problem, the large-scale testing of V2X enabled ADAS solutions to evaluate and measure the anticipated quality and functionality of the experimental system is of great significance. This article proposes a novel ADAS application prototyping framework, using declarative programming, built on top of the popular Artery/OMNeT++ simulator. The framework is capable of simulating V2X-enabled ADAS applications using accurate network simulation and realistic simulated traffic on real-world maps. The solution features XML descriptions for application specification. The sensor model of Artery is used to provide information to applications. By using the simulator, one can conclude the performance of the applications and discover locations, circumstances and design patterns, where design limits should apply.

Implementation of Deep Learning Algorithm on a Custom Dataset for Advanced Driver Assistance Systems Applications

Road hazards such as jaywalking pedestrians, stray animals, unmarked speed bumps, vehicles, and road damage can pose a significant threat in poor visibility conditions. Vehicles are fitted with safety technologies like advanced driver assistance systems (ADAS) and AW (automatic warning) systems to tackle these issues. However, these safety systems are complex and expensive, and these proprietary systems are exclusive to high-end models. The majority of the existing vehicles on the road lacks these systems. The YOLO model (You Only Look Once Architecture) was chosen owing to its lightweight architecture and low inference latency. Since YOLO is an open-source architecture, it can enhance interoperability and feasibility of aftermarket/retrofit ADAS devices, which helps in reducing road fatalities. An ADAS which implements a YOLO-based object detection algorithm to detect and mark obstacles (pedestrians, vehicles, animals, speed breakers, and road damage) using a visual bounding box was proposed. The performance of YOLOv3 and YOLOv5 has been evaluated on the Traffic in the Tamil Nadu Roads dataset. The YOLOv3 model has performed exceptionally well with an F1-Score of 76.3% and an mAP (mean average precision) of 0.755, whereas the YOLOv5 has achieved an F1-Score of 73.7% and an mAP of 0.7263.

Configuring ADAS Platforms for Automotive Applications Using Metaheuristics.

Modern Advanced Driver-Assistance Systems (ADAS) combine critical real-time and non-critical best-effort tasks and messages onto an integrated multi-core multi-SoC hardware platform. The real-time safety-critical software tasks have complex interdependencies in the form of end-to-end latency chains featuring, e.g., sensing, processing/sensor fusion, and actuating. The underlying real-time operating systems running on top of the multi-core platform use static cyclic scheduling for the software tasks, while the communication backbone is either realized through PCIe or Time-Sensitive Networking (TSN). In this paper, we address the problem of configuring ADAS platforms for automotive applications, which means deciding the mapping of tasks to processing cores and the scheduling of tasks and messages. Time-critical messages are transmitted in a scheduled manner via the timed-gate mechanism described in IEEE 802.1Qbv according to the pre-computed Gate Control List (GCL) schedule. We study the computation of the assignment of tasks to the available platform CPUs/cores, the static schedule tables for the real-time tasks, as well as the GCLs, such that task and message deadlines, as well as end-to-end task chain latencies, are satisfied. This is an intractable combinatorial optimization problem. As the ADAS platforms and applications become increasingly complex, such problems cannot be optimally solved and require problem-specific heuristics or metaheuristics to determine good quality feasible solutions in a reasonable time. We propose two metaheuristic solutions, a Genetic Algorithm (GA) and one based on Simulated Annealing (SA), both creating static schedule tables for tasks by simulating Earliest Deadline First (EDF) dispatching with different task deadlines and offsets. Furthermore, we use a List Scheduling-based heuristic to create the GCLs in platforms featuring a TSN backbone. We evaluate the proposed solution with real-world and synthetic test cases scaled to fit the future requirements of ADAS systems. The results show that our heuristic strategy can find correct solutions that meet the complex timing and dependency constraints at a higher rate than the related work approaches, i.e., the jitter constraints are satisfied in over 6 times more cases, and the task chain constraints are satisfied in 41% more cases on average. Our method scales well with the growing trend of ADAS platforms.

Construction of Collision-Type Prediction Models Based on Pre-crash Data for Advanced Driver Assistance Systems

ABSTRACT: Collision-type prediction models based on pre-crash information are important because there is a relationship between collision type, avoidance operations, and occupant injuries. Thus, they can be applied to autonomous driving systems (ADS) or advanced driver assistance systems (ADAS) to prevent serious accidents or minimize damage during collisions. In this study, we investigated the application of collision-type prediction models based on several machine learning methods and compared their performance to determine the best model based on their f1 scores. The results revealed that the light gradient boosting machine (LGBM) model had a high f1 score that exceeded 0.92, which implied that it could potentially be used for ADS and ADAS applications. Furthermore, a brief analysis was performed on the ranking of various factors, which provided useful insight into the significance of several pre-crash factors and their distributions.

Enhancing Deep-Learning Object Detection Performance Based on Fusion of Infrared and Visible Images in Advanced Driver Assistance Systems

Image recognition technology plays an important role in advanced driver assistance systems (ADAS). The objective of this study is to explore the feasibility of using heterogeneous image fusion to improve the object detection performance of the ADAS. Among the many possible combinations of image types, the fusion of infrared (IR) and visible (VIS) images has great potential because of their complementary characteristics. Most studies on image fusion assume that the images involved align themselves perfectly, which is unrealistic. We address this alignment issue in this study, review various methods of image alignment and fusion, and propose an image-fusion approach that combines alignment and fusion methods for the ADAS application. Finally, we used deep learning networks to detect pedestrian and vehicle objects before and after the image fusion. The experimental results show that the fusion of IR and VIS images can improve the object detection performance of deep-learning networks. Compared with previous studies on fusion, the proposed approach ranks top if the detection accuracy improvement and execution speed are considered as a whole. This study also found that, to use image fusion to improve the object detection accuracy of deep learning networks, it is better to use fused images directly instead of unfused VIS images as the training samples.

Worst-case scenarios identification approach for the evaluation of advanced driver assistance systems in intelligent/autonomous vehicles under multiple conditions

To demonstrate the expected performance of an advanced driver assistance system (ADAS) in an intelligent or a highly automated vehicle test approaches should include a combination of simulations, track tests, and road tests. The main objective of our work was to propose a new evaluation approach conducted by an external entity where the vehicle is treated as a black box. This approach allowed for the identification of a set of worst-case scenarios for a given ADAS application and combined the three test approaches mentioned earlier. Our proposed evaluation approach is broken down into three parts: (1) Scenarios synthesis, sampling strategy, and simulations, (2) Risk assessment and classification, and (3) Validation. In this article, we present our proposed approach for the validation portion. The validation portion can be further broken down into three parts. The first step included a description of the studied autonomous emergency braking (AEB), physical track tests, and of the different machines and ensemble learning techniques employed to create the predictive model. The second part utilized the Field Operational Tests database (SPMD), to implement the new sampling strategy based on the original, modified, and multivariate Metropolis-Hastings algorithm. The third part focused on collecting the prediction results, then assess the risk of each test, to classify them using a non-supervised technique (k-Means clustering). This allowed us to build a set of worst-case scenarios to make a final selection. Finally, two web applications were developed and deployed.

A Deep-Learning Model with Task-Specific Bounding Box Regressors and Conditional Back-Propagation for Moving Object Detection in ADAS Applications.

This paper proposes a deep-learning model with task-specific bounding box regressors (TSBBRs) and conditional back-propagation mechanisms for detection of objects in motion for advanced driver assistance system (ADAS) applications. The proposed model separates the object detection networks for objects of different sizes and applies the proposed algorithm to achieve better detection results for both larger and tinier objects. For larger objects, a neural network with a larger visual receptive field is used to acquire information from larger areas. For the detection of tinier objects, the network of a smaller receptive field utilizes fine grain features. A conditional back-propagation mechanism yields different types of TSBBRs to perform data-driven learning for the set criterion and learn the representation of different object sizes without degrading each other. The design of dual-path object bounding box regressors can simultaneously detect objects in various kinds of dissimilar scales and aspect ratios. Only a single inference of neural network is needed for each frame to support the detection of multiple types of object, such as bicycles, motorbikes, cars, buses, trucks, and pedestrians, and to locate their exact positions. The proposed model was developed and implemented on different NVIDIA devices such as 1080 Ti, DRIVE-PX2 and Jetson TX-2 with the respective processing performance of 67 frames per second (fps), 19.4 fps, and 8.9 fps for the video input of 448 × 448 resolution, respectively. The proposed model can detect objects as small as 13 × 13 pixels and achieves 86.54% accuracy on a publicly available Pascal Visual Object Class (VOC) car database and 82.4% mean average precision (mAP) on a large collection of common road real scenes database (iVS database).

Tracking and climbing behavior recognition of heavy-duty trucks on roadways

The tracking and behavior recognition of heavy-duty trucks on roadways are keys for the development of automated heavy-duty trucks and an advanced driver assistance system. The spatiotemporal information of trucks from trajectory tracking and motions learnt from behavior analysis can be employed to predict possible driving risks and generate safe motion to avoid roadway accidents. This article presents a unified tracking and behavior recognition algorithm that can model the mobility of heavy-duty trucks on long inclined roadways. Random noise within the sampled elevation data is addressed by time-based segmentation to extract time-continuous samples at geographical locations. A Kalman filter is first used to distinguish error offsets from random noise and to estimate the distribution of truck elevations for different time intervals. A Markov chain Monte Carlo model is then applied to classify truck behaviors based on the change in elevation between two geographical locations. A heavy-duty truck mobility (HVMove) model is constructed based on the map information to apply the roadway geometry to the tracking and behavior recognition algorithm. We develop an extended Metropolis–Hastings algorithm to tune the parameters of the HVMove model. The proposed model is verified and evaluated through extensive experiments based on a real-world trajectory dataset covering sections of an expressway and national and provincial highways. From the experimental results, we conclude that the HVMove model provides sufficient accuracy and efficiency for automated heavy-duty trucks and advanced driver assistance system applications. In addition, HVMove can generate maps with the elevation information marked automatically.

Hardware implementation of real-time pedestrian detection system

Histogram of oriented gradients (HOG) descriptor and support vector machine (SVM) classifier have been considered as a very efficient and robust tools used in the development of pedestrian detection systems. However, the enormous calculations make it difficult to meet real-time requirement specifically for advanced driver assistance system applications. The latter requires higher frame rate and resolution. In the present work, efficient pipelined architecture founded on cell-based scanning and simultaneous linear SVM classification is highlighted. The proposed real-time pedestrian detection architecture performs without external memories, and only 3 × 3 pixels are processed using two line buffers. Indeed, intermediate results will contribute directly to build the cell histograms. Consequently, it allows to reduce the required memory and to speed up the HOG feature extraction procedure. The implementation results on a heterogeneous platform show that the proposed architecture achieves an efficient pedestrian detection for full-HD video recordings (1080 × 1920 pixels) at 60 fps with reduced hardware resource consumption. Using the same hardware resources at 444 MHz, the proposed system turns out to handle a HD video at 180 fps.

Resource-Constrained Machine Learning for ADAS: A Systematic Review

The advent of machine learning (ML) methods for the industry has opened new possibilities in the automotive domain, especially for Advanced Driver Assistance Systems (ADAS). These methods mainly focus on specific problems ranging from traffic sign and light recognition to pedestrian detection. In most cases, the computational resources and power budget found in ADAS systems are constrained while most machine learning methods are computationally intensive. The usual solution consists in adapting the ML models to comply with the memory and real-time (RT) requirements for inference. Some models are easily adapted to resource-constrained hardware, such as Support Vector Machines, while others, like Neural Networks, need more complex processes to fit into the desired hardware. The ADAS hardware (HW platforms) are diverse, from complex MPSoC CPUs down to classical MCUs, DPSs and application-specific FPGAs and ASICs or specific GPU platforms (such as the NVIDIA families Tegra or Jetson). Therefore, there is a tradeoff between the complexity of the ML model implemented and the selected platform that impacts the performance metrics: function results, energy consumption and speed (latency and throughput). In this paper, a survey in the form of systematic review is conducted to analyze the scope of the published research works that embed ML models into resource-constrained implementations for ADAS applications and what are the achievements regarding the ML performance, energy and speed trade-off.

Performance Optimisation of Parallelized ADAS Applications in FPGA-GPU Heterogeneous Systems: A Case Study With Lane Detection

The explosive growth of massive data captured by various sensors on modern vehicles has impelled the deployment of Commercial Off-The-Shelf (COTS) accelerators for the research and development of Advanced Driver Assistance Systems (ADAS). Although the advent of cross-platform programming framework such as Open Computing Language (OpenCL) facilitates the programmability of ADAS applications on heterogeneous devices, the performance portability is still vulnerable and subject to different hardware implementations by the heterogeneous manufacturers. With this issue in mind, in this article we propose a detailed procedure that helps guide the performance optimisation of parallelized ADAS applications in an FPGA-GPU combined heterogeneous system. Taking two different lane detection applications as case studies, we provide one intra-accelerator and two interaccelerator optimisation methods, as well as both FPGA-specific and application-oriented optimisation strategies, to boost the program runtime performance. Experiment results on a heterogeneous platform with COTS FPGA and GPU components reveal that the optimal designs generated from the procedure can improve the runtime performance of the two applications by an average of 109.21% and 83.48% over the native parallel implementations, respectively.

A 20.5 TOPS Multicore SoC With DNN Accelerator and Image Signal Processor for Automotive Applications

Advanced driver-assistance systems (ADASs) that provide machine support to avoid critical accidents are already in wide use in vehicles in today's market. SoCs for such systems have several requirements: 1) high computational performance to run several advanced algorithms at low latency; 2) low power consumption to permit running under the extreme heat and power conditions of real-world vehicles; and 3) high reliability to reduce the risk of serious accidents caused by faults. This article presents a novel SoC for ADAS applications, which resolves these difficult challenges. To achieve high performance with low power consumption, the SoC adopts the heterogeneous architecture, with ten processors, four DSPs, and eight types of accelerators, including the DNN accelerator and the image signal processor (ISP). To achieve higher reliability, the SoC implements several safety mechanisms, including system partitioning to prevent fault propagation, diagnostic features to detect faults, and a dedicated controller to operate runtime built-in self-test (BIST) of the ISP's diagnostic features during the vertical blanking interval (VBLANK). The SoC is implemented in a 16-nm process, and its size is 94.52 mm <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . Peak performance of 20.5 TOPS and low power consumption of 9.78 W are achieved.

Towards Next-Generation Vehicles Featuring the Vehicle Intelligence

Safe driving and minimizing the number of casualties are the main motivations of researchers and car companies for decades. They also care very much on saving fuel consumption and high comfort-level trips. With the help of advanced driver assistance systems (ADAS) applications, safer, more comfortable, and greener trips are very likely at the present time. However, today humankind is very close to make a very old dream, namely, driverless vehicles, to come true. In this paper, we address the concept of next-generation vehicles, their requirements, challenges, advantages, and problems. Regarding Society of Automotive Engineers (SAE), levels (1–5), we first define the crucial contexts for next-generation vehicles. We then discuss existing ADAS, their abilities, and available platforms which they run on from past to present. Next, we introduce a novel vehicle intelligence (VI) architecture consisting of ADAS modules and VI services which would pave the way for fully autonomous vehicles regarding not only driving issue, but also human-centric new demands such as entertainment and comfort level of the journey. The proposed conceptual design is built on sensors, vehicle ad hoc networks (VANETs), and big data. Afterward, we describe how current ADAS applications would transform on the way toward SAE Level 5 cars. We finally discuss the open issues for next-generation vehicles.

Fast Angle Adaption of a MEMS-based LiDAR System

The amount of Advanced Driver-Assistance Systems (ADAS) in middle-class cars are increasing steadily. Next generation ADAS will take control in fully automated vehicles. Light Detection And Ranging (LiDAR) will be one of the key enablers for highly automated or even autonomous vehicles. The 1D Micro-Electro-Mechanical System (MEMS) Micro-Scanning LiDAR is able to detect obstacles in a predefined Field-of-View (FoV). Future ADAS applications will require to dynamically change FoVs as fast as possible (e.g. quickly switching between long-range narrow FoV and short-range wide FoV).In this paper we introduce a novel system architecture that enables fast angle adaption of a 1D MEMS Micro-Scanning LiDAR System. The system architecture was implemented in an FPGA prototype to prove its feasibility and to evaluate its performance.

Online stereo camera calibration for automotive vision based on HW-accelerated A-KAZE-feature extraction

Abstract Nowadays ongoing integration of camera based advanced driver assistance systems (ADAS) in vehicles demands increasingly complex digital image processing in order to interpret the surrounding situations. To ensure a timely reaction for obstacles lying ahead, far reaching depth information of the observed scene is necessary. Using stereo camera systems, this is achievable by enlarging the camera base line. With rapid driving, the exact alignment of the stereo images is no longer ensured due to vibrations of the vehicle. Based on detection, extraction and matching of Accelerated-KAZE image features (A-KAZE), the geometric distortions are compensable by estimating the external camera parameters for image rectification. The indispensable frame rate for applications in vehicles and the limited power budget in combination with the SW-flexibility demanded for future ADAS applications requires the usage of optimized hardware architectures. Thus, an online camera calibration based on an HW-accelerated A-KAZE extraction is introduced in this work. The suitability of A-KAZE features for an online camera calibration is proven. Furthermore, the Tensilica Vision P5-processor is evaluated regarding its suitability for real-time A-KAZE feature extraction. This processor provides a comprehensive instruction-set extension for high-performance digital image processing. While preserving the initial A-KAZE accuracy, a feature descriptor length reduction of factor × 3.8 is attained compared to the initial descriptor size and an estimated frame rate of 20 fps is achieved for A-KAZE feature extraction on the Tensilica Vision P5-processor.

Effects of smartphone based advanced driver assistance system on distracted driving behavior: A simulator study

Nowadays smartphone distractions are frequently encountered while driving and are causing an increasing number of crashes. A method to prevent them is related to the use of advanced driver assistance systems (ADAS) which ensures that the driver's attention will return to the road when he/she is tempted to use a smartphone. An experimental study based on a driving simulator was conducted in order to investigate the influence of an in-vehicle smartphone based ADAS on the drivers' behavior when they are distracted by social networking applications. A number of 17 participants performed three scenarios consisting in driving with and without distraction factors related to various Facebook tasks with and without ADAS notifications. The statistical analysis shows that the use of the ADAS application help in reducing driving infractions on average by 43.43% compared to the situation in which the driver does not use it.

Topological Semantic Mapping and Localization in Urban Road Scenarios

Autonomous vehicle self-localization must be robust to environment changes, such as dynamic objects, variable illumination, and atmospheric conditions. Topological maps provide a concise representation of the world by only keeping information about relevant places, being robust to environment changes. On the other hand, semantic maps correspond to a high level representation of the environment that includes labels associated with relevant objects and places. Hence, the use of a topological map based on semantic information represents a robust and efficient solution for large-scale outdoor scenes for autonomous vehicles and Advanced Driver Assistance Systems (ADAS). In this work, a novel topological semantic mapping and localization methodology for large-scale outdoor scenarios for autonomous driving and ADAS applications is presented. The methodology uses: (i) a deep neural network for obtaining semantic observations of the environment, (ii) a Topological Semantic Map (TSM) for storing selected semantic observations, and (iii) a topological localization algorithm which uses a Particle Filter for obtaining the vehicle’s pose in the TSM. The proposed methodology was tested on a real driving scenario, where a True Estimate Rate of the vehicle’s pose of 96.9% and a Mean Position Accuracy of 7.7[m] were obtained. These results are much better than the ones obtained by other two methods used for comparative purposes. Experiments also show that the method is able to obtain the pose of the vehicle when its initial pose is unknown.

A novel global methodology to analyze the embeddability of real-time image processing algorithms

Advanced driver assistance systems applications increasingly use cameras and image processing algorithms. To embed and achieve real-time execution of these algorithms, semiconductor companies propose heterogeneous systems-on-chip (SoCs). Embedding algorithms on this type of hardware is not trivial: One needs to determine how to partition the computational load on the different processing units. In addition, it is not easy to predict whether a given algorithm can be executed on a given heterogeneous SoC while meeting real-time constraints. We propose a novel global methodology to assist with embedding image processing algorithms on heterogeneous SoC while meeting real-time constraints (using a soft real-time analysis). Our approach proposes several heuristics predicting delays and execution times and is based on a set of multi-level test vectors which extract key features of heterogeneous architectures.