Learning from Failures: LLM-Guided Safety-Critical Scenario Recommendation for Self-Evolving Autonomous Driving

Ensuring robust and generalizable autonomous driving requires not only broad scenario coverage but also efficient repair of failure cases, particularly those related to challenging and safety-critical scenarios. However, existing scenario generation and selection methods often lack adaptivity and semantic relevance, limiting their impact on performance improvement. In this paper, we propose SERA, an LLM-powered framework that enables autonomous driving systems to self-evolve by repairing failure cases through targeted scenario recommendation. By analyzing performance logs, SERA identifies failure patterns and dynamically retrieves semantically aligned scenarios from a structured bank. An LLM-based reflection mechanism further refines these recommendations to maximize relevance and diversity. The selected scenarios are used for few-shot fine-tuning, enabling targeted adaptation with minimal data. Experiments on the benchmark show that SERA consistently improves key metrics across multiple autonomous driving baselines, demonstrating its effectiveness and generalizability under safety-critical conditions.

The field of Autonomous Driving (AD) has witnessed significant progress in recent years. Among the various challenges faced, the safety evaluation of autonomous vehicles (AVs) stands out as a critical concern. Traditional evaluation methods are both costly and inefficient, often requiring extensive driving mileage in order to encounter rare safety-critical scenarios, which are distributed on the long tail of the complex real-world driving landscape. In this paper, we propose a unified approach, Diffusion-Based Safety-Critical Scenario Generation (DiffScene), to generate high-quality safety-critical scenarios which are both realistic and safety-critical for efficient AV evaluation. In particular, we propose a diffusion-based generation framework, leveraging the power of approximating the distribution of low-density spaces for diffusion models. We design several adversarial optimization objectives to guide the diffusion generation under predefined adversarial budgets. These objectives, such as safety-based objective, functionality-based objective, and constraint-based objective, ensure the generation of safety-critical scenarios while adhering to specific constraints. Extensive experimentation has been conducted to validate the efficacy of our approach. Compared with 6 SOTA baselines, DiffScene generates scenarios that are (1) more safety-critical under 3 metrics, (2) more realistic under 5 distance functions, and (3) more transferable to different AV algorithms. In addition, we demonstrate that training AV algorithms with scenarios generated by DiffScene leads to significantly higher performance in terms of the safety-critical metrics compared to baselines. These findings highlight the potential of DiffScene in addressing the challenges of AV safety evaluation, paving the way for safer AV development.

The field of autonomous automation is of interest to researchers, and much has been accomplished in this area, of which this paper presents a detailed chronology. This paper can help one understand the trends in autonomous vehicle technology for the past, present, and future. We see a drastic change in autonomous vehicle technology since 1920s, when the first radio controlled vehicles were designed. In the subsequent decades, we see fairly autonomous electric cars powered by embedded circuits in the roads. By 1960s, autonomous cars having similar electronic guide systems came into picture. 1980s saw vision guided autonomous vehicles, which was a major milestone in technology and till date we use similar or modified forms of vision and radio guided technologies. Various semi-autonomous features introduced in modern cars such as lane keeping, automatic braking and adaptive cruise control are based on such systems. Extensive network guided systems in conjunction with vision guided features is the future of autonomous vehicles. It is predicted that most companies will launch fully autonomous vehicles by the advent of next decade. The future of autonomous vehicles is an ambitious era of safe and comfortable transportation.

Artificial Intelligence (AI) methods need to be evaluated thoroughly to ensure reliable behavior. In applications like autonomous driving, a complex environment with an uncountable number of different situations and conditions needs to be handled by a method whose behavior needs to be predictable. To accomplish this, simulations can be used as a first step. However, the physical world behaves differently, as the example of autonomous driving shows. There, erroneous behavior has been found in test drives that was not noticed in simulations. Errors were caused by conditions or situations that were not covered by the simulations (e.g., specific lighting conditions or other vehicle's behavior). However, the problem with real world testing of autonomous driving features is that critical conditions or situations occur very rarely—while the test effort is high. A solution can be the combination of physical world tests and simulations—and miniature vehicles as an intermediate step between both. With model cars (in a sufficiently complex model environment) advantages of both can be combined: (1) low test effort and a repeatable variation of conditions/situations as an advantage like in simulations and (2) (limited) physical world testing with unspecified and potentially unknown properties as an advantage like in real-world tests. Additionally, such physical tests can be carried out in less stable cases like already in the early stages of AI method testing and/or in approaches using online learning. Now, we propose to use a) miniature vehicles at a small scale of 1:87 and b) use sensors and computational power only on the vehicle itself. By this limitation, a further consequence is expected: Here, autonomy methods need to be optimized drastically or even redesigned from scratch. The resulting methods are supposed to be less complex—and, thus, again less error-prone. We call this approach “Miniature Autonomy” and apply it to the road, water, and aerial vehicles. In this article, we briefly describe a small test area we built (3 sqm.), a large test area used alternatively (1,545 sqm.), two last generation autonomous miniature vehicles (one road, one aerial vehicle), and an autonomous driving demo case demonstrating the application.

Autonomous Driving (AD) represents a breakthrough technology with immense potential to make our transportation more intelligent, as many Autonomous Vehicles (AV) are already deployed into real products on public roads. The brain of an AV is the Autonomous Driving System (ADS), which relies on a combination of sensors and various machine learning algorithms to perceive the environment, make decisions, and navigate safely. However, real-world scenarios are complex and dynamic, with numerous factors that can affect the performance of ADS. Adver- sarial attacks, including training-time adversarial attacks (i.e., backdoor attacks) and testing-time adversarial attacks, have emerged as a destructive means of com- promising AV. Nonetheless, current research lacks a comprehensive evaluation of its effectiveness and robustness in physical environments. Furthermore, there is still a huge room for improvement in the current defense methods against physical adversarial attacks against ADS. However, due to the complexity of ADS, conducting a comprehensive evaluation poses significant challenges. To address this challenge, my thesis focuses on the systematic analysis of building secure ADS through evaluation in both simulator and real-world environments. This involves holistic testing and assessment of ADS, using realistic attacks, and actively discovering new security issues. To achieve this goal, my thesis first evaluates the performance of existing physical adversarial attacks in real-world settings, summarizing new insights and identifying new attack surfaces and methods. Specifically, we introduce BatAV, a comprehen- sive platform for benchmarking physical backdoor attacks to ADS perception. To uncover new vulnerabilities of ADS, we develop STFA, a dynamic physical world vulnerability testing platform for the ADS decision-making module. Based on this, we demonstrate that some existing adversarial attacks and vulnerabilities proposed by us could pose significant dangers to AVs. Subsequently, various studies have investigated adversarial attacks on the perception modules in ADS. Therefore, our second focus is on developing novel defense mechanisms against adversarial attacks on ADS. To have a deep understanding of physical adversarial attacks, we comprehensively evaluate 9 state-of-the-art methods in real-world scenarios. Based on that, we introduce two novel defense methods: (1) ADS-Lead, an effective collaborative anomaly detection method to safeguard ADS lane-following mechanisms. (2) VisionGuard, a unified defense framework capable of detecting and mitigating various physical adversarial attacks on ADS perception. Both defense measures utilize positioning and navigation sensors (e.g., GPS, IMU) to defend against attacks on visual sensors (e.g., cameras, LiDAR). In summary, this thesis is committed to evaluating existing adversarial attack methods, finding new vulnerabilities and attack surfaces, and designing novel ADS defense methods to build a secure ADS.

Autonomous driving is a challenging problem because the autonomous vehicle must understand complex and dynamic environment. This understanding consists of predicting future behavior of nearby vehicles and recognizing predefined rules. It is observed that not all rules have equivalent values, and the priority of the rules may change depending on the situation or the driver's driving style. In this work, we jointly reason both a future trajectories of vehicles and degree of satisfaction of each rule in the deep learning framework. Joint reasoning allows modeling interactions between vehicles, and leads to better prediction results. A rule is represented as a signal temporal logic (STL) formula, and a robustness slackness, a margin to the satisfaction of the rule, is predicted for the both autonomous and other vehicle, in addition to future trajectories. Learned robustness slackness decides which rule should be prioritized for the given situation for the autonomous vehicle, and filter out non-valid predicted trajectories for surrounding vehicles. The predicted information from the deep learning framework is used in model predictive control (MPC), which allows the autonomous vehicle navigate efficiently and safely. We test the feasibility of our approach in publicly available NGSIM datasets. Proposed method shows a driving style similar to the human one and considers the safety related to the rules through the future prediction of the surrounding vehicles.

Driverless Tractor into agriculture.The field of autonomous automation is of interest to researchers, and much has been accomplished in this area, of which this paper presents a detailed chronology. This paper can help one understand the trends in autonomous vehicle technology for the past, present, and future. We see a drastic change in autonomous vehicle technology since 1920s, when the first radio controlled vehicles were designed. In the subsequent decades, we see fairly autonomous electric cars powered by embedded circuits in the roads. By 1960s, autonomous cars having similar electronic guide systems came into picture. 1980s saw vision guided autonomous vehicles, which was a major milestone in technology and till date we use similar or modified forms of vision and radio guided technologies. Various semi-autonomous features introduced in modern cars such as lane keeping, automatic braking and adaptive cruise control are based on such systems. Extensive network guided systems in conjunction with vision guided features is the future of autonomous vehicles. It is predicted that most companies will launch fully autonomous vehicles by the advent of next decade. The future of autonomous vehicles is an ambitious era of safe and comfortable transportation. KEY WORDS: autonomous vehicles,adaptive cruise control,embedded circuits.

The development of intelligent driving systems has drawn significant attention to enhancing the safety of autonomous vehicles and their intended functionality. Despite this, current accelerated testing approaches remain inadequate in assessing system reliability, as they fail to simulate scenarios involving collisions between vehicles and pedestrians and identify unknown risks. To address these limitations, scenario-based testing methods have been proposed, which seek to identify critical scenarios with a high frequency of exposure to safety risks. A comprehensive review of these methods is thus of paramount significance. In this article, we provide a timely and systematic literature review of existing accelerated testing for autonomous vehicles. We propose a taxonomy of these methods, discuss each subfield, and highlight open problems and future directions. Our objective is to provide a clear and concise overview of the state of the art in this field and to offer insights into the effectiveness of scenario-based testing approaches. By doing so, we aim to facilitate the identification of critical scenarios and the assessment of risk exposure frequencies, which are essential for enhancing the safety and reliability of autonomous vehicles.

Intelligent Automation (IA) in automobiles combines robotic process automation and artificial intelligence, allowing digital transformation in autonomous vehicles. IA can completely replace humans with automation with better safety and intelligent movement of vehicles. This work surveys those recent methodologies and their comparative analysis, which use artificial intelligence, machine learning, and IoT in autonomous vehicles. With the shift from manual to automation, there is a need to understand risk mitigation technologies. Thus, this work surveys the safety standards and challenges associated with autonomous vehicles in context of object detection, cybersecurity, and V2X privacy. Additionally, the conceptual autonomous technology risks and benefits are listed to study the consideration of artificial intelligence as an essential factor in handling futuristic vehicles. Researchers and organizations are innovating efficient tools and frameworks for autonomous vehicles. In this survey, in-depth analysis of design techniques of intelligent tools and frameworks for AI and IoT-based autonomous vehicles was conducted. Furthermore, autonomous electric vehicle functionality is also covered with its applications. The real-life applications of autonomous truck, bus, car, shuttle, helicopter, rover, and underground vehicles in various countries and organizations are elaborated. Furthermore, the applications of autonomous vehicles in the supply chain management and manufacturing industry are included in this survey. The advancements in autonomous vehicles technology using machine learning, deep learning, reinforcement learning, statistical techniques, and IoT are presented with comparative analysis. The important future directions are offered in order to indicate areas of potential study that may be carried out in order to enhance autonomous cars in the future.

We tackle the problem of multi-label classification of fashion images, learning from noisy data with minimal human supervision. We present a new dataset of full body poses, each with a set of 66 binary labels corresponding to the information about the garments worn in the image obtained in an automatic manner. As the automatically-collected labels contain significant noise, we manually correct the labels for a small subset of the data, and use these correct labels for further training and evaluation. We build upon a recent approach that both cleans the noisy labels and learns to classify, and introduce simple changes that can significantly improve the performance.

In the last decades autonomous vehicles have been at the centre of the research in both the academic and the industrial fields, but not without difficulties. In particular, the problem of path planning and tracking at the limit of the handling capabilities of a vehicle poses many challenges from a control perspective, and it is yet to be understood whether the integration with stability controllers can improve the cornering performance of autonomous vehicles as much as it does for human drivers. This thesis aims to provide insights on these topics. The first part of the work is dedicated to the planning and tracking layers of an autonomous vehicle driving on racetracks. The analysis covers the offline optimisation of the trajectory and the description of a re-planning algorithm for the avoidance of obstacles. A comparison among several path tracking controllers is then provided, to understand whether the gain in performance obtained from advanced controllers justifies the design complexity. In the second part, the thesis highlights the benefits of yaw rate control on the behaviour of over-actuated vehicles. An algorithm for yaw rate control is introduced and implemented in a torque vectoring controller, and the proof of asymptotic stability of the system is provided. Several application examples are presented, with simulation and experimental results that demonstrate the potential and versatility of yaw rate control. Finally, the integration of torque vectoring and path tracking control in an autonomous racing vehicle is presented and assessed with a simulation study along obstacle avoidance tests. The results of the thesis show that: i) including road preview information in path tracking controllers improves the control action, resulting in better vehicle behaviour, and ii) torque vectoring control always improves the vehicle performance, and it also enhances the system robustness to variations in the tyre-road friction coefficient.

Ensuring safety and achieving human-level driving performance remain challenges for autonomous vehicles, especially in safety-critical situations. As a key component of artificial intelligence, reinforcement learning is promising and has shown great potential in many complex tasks; however, its lack of safety guarantees limits its real-world applicability. Hence, further advancing reinforcement learning, especially from the safety perspective, is of great importance for autonomous driving. As revealed by cognitive neuroscientists, the amygdala of the brain can elicit defensive responses against threats or hazards, which is crucial for survival in and adaptation to risky environments. Drawing inspiration from this scientific discovery, we present a fear-neuro-inspired reinforcement learning framework to realize safe autonomous driving through modeling the amygdala functionality. This new technique facilitates an agent to learn defensive behaviors and achieve safe decision making with fewer safety violations. Through experimental tests, we show that the proposed approach enables the autonomous driving agent to attain state-of-the-art performance compared to the baseline agents and perform comparably to 30 certified human drivers, across various safety-critical scenarios. The results demonstrate the feasibility and effectiveness of our framework while also shedding light on the crucial role of simulating the amygdala function in the application of reinforcement learning to safety-critical autonomous driving domains.

Blockchain is an underlying technology for securing many real-time applications and their data. The automobile is one such sector in which auto-manufacturers are looking forward to accepting the advantages of distributed ledger technology in autonomous vehicles or systems and improving their products, customer satisfaction, and other valuable experiences. This work aims to find the significance of blockchain technology in Autonomous Vehicles, including Autonomous Electric Vehicles (AEV), Autonomous Underwater Vehicles (AUV), Autonomous Guided Vehicles (AGV), Autonomous Aerial Vehicles (AAeV), and Autonomous Driving. In this work, a comparative analysis of blockchain-integrated autonomous vehicle systems is explored to identify the present scenario and futuristic challenges. In addition to blockchain technology, the uses and importance of sensors, architectures and infrastructure requirements, vehicle types, driving modes, vehicles target and tracking approaches, intelligent contracts, intelligent data handling, and industry-specific use cases are also explored. This study is based on the exploration of recent technologies and practices. As autonomous vehicles are expected to be the future of intelligent transportation, this paper surveys recent advances in autonomous vehicles and systems and how blockchain can help in improving user experiences and improving industry practices. Finally, limitations of work, future research directions, and challenges associated with different autonomous vehicles and systems are presented.

Unlike other autonomous working vehicles such as farm and construction machinery, autonomous cars have at least one occupant inside, even if driving operations are automated. Thus, the whole-body vibration problem must continue to be a research topic for autonomous vehicles. We believe that an autonomous vehicle can mitigate whole body vibration by utilizing autonomous functions such as lane change control. Therefore, the purpose of this paper is to verify the effectiveness of lane change control in reducing the whole-body vibration exposure of occupants in autonomous cars. We performed numerical simulations with experimental data obtained using the measurement techniques of ISO2631-1. Thus, we observed that if an autonomous car could skirt rough road surfaces using a lane change function, the degree of ride comfort would be enhanced during travel and the maximum acceptable exposure time as health effect evaluation would be lengthened.

Autonomous driving and its real-world implementation have been among the most actively studied topics in the past few years. In recent years, this growth has been accelerated by the development of advanced deep learning-based data processing technologies. Moreover, large automakers manufacture vehicles that can achieve partially or fully autonomous driving for driving on real roads. However, self-driving cars are limited to some areas with multi-lane roads, such as highways, and self-driving cars that drive in urban areas or residential complexes are still in the development stage. Among autonomous vehicles for various purposes, this paper focused on the development of autonomous vehicles for garbage collection in residential areas. Since we set the target environment of the vehicle as a residential complex, there is a difference from the target environment of a general autonomous vehicle. Therefore, in this paper, we defined ODD, including vehicle length, speed, and driving conditions for the development vehicle to drive in a residential area. In addition, to recognize the vehicle’s surroundings and respond to various situations, it is equipped with various sensors and additional devices that can notify the outside of the vehicle’s state or operate it in an emergency. In addition, an autonomous driving system capable of object recognition, lane recognition, route planning, vehicle manipulation, and abnormal situation detection was configured to suit the vehicle hardware and driving environment configured in this way. Finally, by performing autonomous driving in the actual experimental section with the developed vehicle, it was confirmed that the function of autonomous driving in the residential area works appropriately. Moreover, we confirmed that this vehicle would support garbage collection works through the experiment of work efficiency.