On-board Sensor Data Research Articles

Online High-Definition Map Construction for Autonomous Vehicles: A Comprehensive Survey

High-definition (HD) maps aim to provide detailed road information with centimeter-level accuracy, essential for enabling precise navigation and safe operation of autonomous vehicles (AVs). Traditional offline construction methods involve several complex steps, such as data collection, point cloud generation, and feature extraction, but these methods are resource-intensive and struggle to keep pace with the rapidly changing road environments. In contrast, online HD map construction leverages onboard sensor data to dynamically generate local HD maps, offering a bird’s-eye view (BEV) representation of the surrounding road environment. This approach has the potential to improve adaptability to spatial and temporal changes in road conditions while enhancing cost-efficiency by reducing the dependency on frequent map updates and expensive survey fleets. This survey provides a comprehensive analysis of online HD map construction, including the task background, high-level motivations, research methodology, key advancements, existing challenges, and future trends. We systematically review the latest advancements in three key sub-tasks: map segmentation, map element detection, and lane graph construction, aiming to bridge gaps in the current literature. We also discuss existing challenges and future trends, covering standardized map representation design, multitask learning, and multi-modality fusion, while offering suggestions for potential improvements.

Vehicle Sideslip Angle estimation under critical road conditions via nonlinear Kalman filter-based state-dependent Interacting Multiple Model approach

The knowledge of Vehicle Sideslip Angle (VSA) can play an essential role in active safety vehicle control systems. However, due to the high costs of sensing instruments, this information is difficult to be directly measured onboard of series production vehicles, restricting de facto its application in practice. It follows that there is a need for online VSA estimation methods only based on available measurements from low-cost sensors. From this perspective, this work proposes a strategy based on Interacting Multiple Model (IMM) filters, which does not require tyre–road friction coefficient knowledge. By integrating the available onboard sensor data, the IMM estimates relevant information in different driving conditions leveraging a 2-Degrees Of Freedom (DOF) single-track vehicle model embedding a Dugoff tyre representation. Two alternative IMM algorithms based on the Extended (EKF) and Unscented Kalman filter (UKF) are developed. Moreover, while usually the transition probabilities among models in classical IMMs are fixed and set on prior information and/or dedicated analysis, here these conservative hypotheses are relaxed introducing a state-dependent Markov transition matrix based on a novel model switching algorithm. The effectiveness of the new proposed methods is evaluated on extensive non-trivial simulation scenarios through a Monte Carlo analysis exploiting an accurate 15-DOF vehicle model via a purposely designed high-fidelity co-simulation platform embedding the dSPACE software Automotive Simulation Model (ASM). Results provide a meaningful comparative performance analysis between the IMMEKF and IMMUKF solutions, as well as with respect to traditional IMM based on constant probabilities transition matrix, blue in both the EKF and UKF configuration. Finally, the developed IMM-based estimation strategy is tested in two realistic driving scenarios to assess the VSA estimation accuracy in case of abrupt changes in road surface conditions.

Applications of using connected vehicle data for pavement quality analysis

Current quantitative methods to evaluate pavement conditions in the United States are most commonly focused on construction acceptance using the International Roughness Index (IRI). However, from an asset management perspective, qualitative visual inspection techniques are the most prevalent. Modern vehicles with factory-equipped sensors drive these roadways daily and can passively assess the condition of infrastructure at an accuracy level somewhere between qualitative assessment and rigorous construction acceptance techniques. This paper compares crowdsourced ride quality data with an industry standard inertial profiler on a 7-mile bi-directional construction zone. A linear correlation was performed on 14 miles of I-65 that resulted in an R2 of 0.7 and a p-value of <0.001, but with a modest fixed offset bias. The scalability of these techniques is illustrated with graphics characterizing IRI values obtained from 730,000 crowdsourced data segments over 5,800 miles of I-80 in April of 2022 and October 2022. This paper looks at the use of standard original equipment manufacturer (OEM) on-board sensor data from production vehicles to assess approximately 100 miles of roadway pavements before, during, and after construction. The completed construction projects observed IRI improvements of 10 in/mi to 100 in/mi. These results suggest that it is now possible to monitor pavement ride quality at a system level, even with a small proportion of connected vehicles (CV) providing roughness data.

UAV photogrammetry for red tide monitoring and geo-localization via onboard GNSS/IMU data

As one of the worst ecological disasters in the world’s oceans, red tide seriously threatens marine ecology. Red tide monitoring is essential for preserving the ocean ecosystem. However, traditional photogrammetric processing workflows, e.g. SfM (Structure from Motion) and MVS (Multi-view Stereo) based image orientation and dense matching, do not apply to offshore images due to the low texture of ocean color. The primary contribution of this study is a UAV (Unmanned Aerial Vehicle)-based photogrammetric solution for red tide monitoring and direct geo-localization. First, a direct geo-localization model has been created, which solely uses onboard sensor data for the 3D coordinate calculation of 2D image targets based on the collinear equation. Second, two UAVs are chosen as photogrammetric systems that can supply the necessary data for the direct geo-localization model, including high-resolution images, camera intrinsic and extrinsic parameters. Third, a human annotation technique has been utilized to identify red tide regions whose ground polygons can be further estimated using the direct geo-localization model, taking into account how difficult it is to detect red tides automatically. Finally, the accuracy of direct geo-localization and the viability of the proposed solution has been evaluated and verified using real UAV datasets. The experimental results show that the direct geo-localization precision is better than 20 m when only onboard sensor data is used. The suggested workflow is appropriate for monitoring red tides, which can provide critical information for managing marine ecosystems. The executable toolkit would be made publicly available at https://github.com/json87/PhotoDigitizer.

Comprehensive evaluation of machine learning models for predicting ship energy consumption based on onboard sensor data

Machine learning models for predicting ship energy consumption are built and their influencing factors are investigated. First, data collected from a real ship is preprocessed. Six machine learning methods are used to establish the prediction models of ship fuel consumption, and the performance of models is evaluated by Mean Absolute Error, Coefficient of Determination and training time. Then, by analysing the correlation and importance of the features, it's studied whether the model established complies with the laws of physics. Finally, the factors affecting the prediction performance of machine learning models are analysed. The results show that Random Forest and Extreme Gradient Boosting are the most suitable algorithms for ship fuel consumption prediction. Data preprocessing, data normalisation, training sample size, model type, ship operating conditions, as well as the thermotechnical parameters of main engine have impact on the prediction performance. In particular, when taking the thermotechnical parameters into consideration, R2 is increased by 0.32%, MAE is reduced by 5.0%.

A Novel Strong Tracking ETUKF for State Estimation of Preceding Vehicles Using V2V Communications

Real-time decision-making of the host vehicle relies on the motion information of the preceding vehicle (PV). Using some advanced sensors to estimate the PV state is a hot research topic. However, existing studies rarely use vehicle-to-vehicle (V2V) communication to estimate the PV state. Meanwhile, the influence of model parameter perturbation on estimation accuracy is not fully considered. In this article, a strong tracking event-triggered unscented Kalman filter (STETUKF) is designed to estimate the PV state utilizing V2V communication. First, we develop a nonlinear vehicle model. Then, the strong tracking filtering is combined with the ETUKF to establish STETUKF. Among them, the event-triggered mechanism determines if onboard sensor data from the PV will be delivered to the host vehicle. By appropriately adjusting the event-triggered threshold, an ideal compromise can be achieved between the transmission rate and estimation accuracy. Finally, experimental results demonstrate that STETUKF has better estimation performance than the traditional UKF. The proposed algorithm can save at least 64.93% of communication resources and it is robust to the model parameter perturbation.

Research Progress on Data-Driven Methods for Battery States Estimation of Electric Buses

Battery states are very important for the safe and reliable use of new energy vehicles. The estimation of power battery states has become a research hotspot in the development of electric buses and transportation safety management. This paper summarizes the basic workflow of battery states estimation tasks, compares, and analyzes the advantages and disadvantages of three types of data sources for battery states estimation, summarizes the characteristics and research progress of the three main models used for estimating power battery states such as machine learning models, deep learning models, and hybrid models, and prospects the development trend of estimation methods. It can be concluded that there are many data sources used for battery states estimation, and the onboard sensor data under natural driving conditions has the characteristics of objectivity and authenticity, making it the main data source for accurate power battery states estimation; Artificial neural network promotes the rapid development of deep learning methods, and deep learning models are increasingly applied in power battery states estimation, demonstrating advantages in accuracy and robustness; Hybrid models estimate the states of power batteries more accurately and reliably by comprehensively utilizing the characteristics of different types of models, which is an important development trend of battery states estimation methods. Higher accuracy, real-time performance, and robustness are the development goals of power battery states estimation methods.

Driver Identification Methods in Electric Vehicles, a Review

Driver identification is very important to realizing customized service for drivers and road traffic safety for electric vehicles and has become a research hotspot in the field of modern automobile development and intelligent transportation. This paper presents a comprehensive review of driver identification methods. The basic process of driver identification task is proposed as four steps, the advantages and disadvantages of different data sources for driver identification are analyzed, driver identification models are divided into three categories, and the characteristics and research progress of driver identification models are summarized, which can provide a reference for further research on driver identification. It is concluded that on-board sensor data in the natural driving state is objective and accurate and could be the main data source for driver identification. Emerging technologies such as big data, artificial intelligence, and the internet of things have contributed to building a deep learning hybrid model with high accuracy and robustness and representing an important gradual development trend of driver identification methods. Developing a driver identification method with high accuracy, real-time performance, and robustness is an important development goal in the future.

Seeing from space makes sense: Novel earth observation variables accurately map species distributions over Himalaya

Topical advances in earth observation have enabled spatially explicit mapping of species' fundamental niche limits that can be used for nature conservation and management applications. This study investigates the possibility of applying functional variables of ecosystem retrieved from Moderate Resolution Imaging Spectroradiometer (MODIS) onboard sensor data to map the species distribution of two alpine treeline species, namely Betula utilis D.Don and Rhododendron campanulatum D.Don over the Himalayan biodiversity hotspot. In this study, we have developed forty-nine Novel Earth Observation Variables (NEOVs) from MODIS products, an asset to the present investigation. To determine the effectiveness and ecological significance of NEOVs combinations, we built and compared four different models, namely, a bioclimatic model (BCM) with bioclimatic predictor variables, a phenology model (PhenoM) with earth observation derived phenological predictor variables, a biophysical model (BiophyM) with earth observation derived biophysical predictor variables, and a hybrid model (HM) with a combination of selected predictor variables from BCM, PhenoM, and BiophyM. All models utilized topographical variables by default. Models that include NEOVs were competitive for focal species, and models without NEOVs had considerably poor model performance and explanatory strength. To ascertain the accurate predictions, we assessed the congruence of predictions by pairwise comparisons of their performance. Among the three machine learning algorithms tested (artificial neural networks, generalised boosting model, and maximum entropy), maximum entropy produced the most promising predictions for BCM, PhenoM, BiophyM, and HM. Area under curve (AUC) and true skill statistic (TSS) scores for the BCM, PhenoM, BiophyM, and HM models derived from maximum entropy were AUC ≥0.9 and TSS ≥0.6 for the focal species. The overall investigation revealed the competency of NEOVs in the accurate prediction of species' fundamental niches, but conventional bioclimatic variables were unable to achieve such a level of precision. A principal component analysis of environmental spaces disclosed that niches of focal species substantially overlapped each other. We demonstrate that the use of satellite onboard sensors’ biotic and abiotic variables with species occurrence data can provide precision and resolution for species distribution mapping at a scale that is relevant ecologically and at the operational scale of most conservation and management actions.

An Auto-Encoder Based TinyML Approach for Real-Time Anomaly Detection

<div class="section abstract"><div class="htmlview paragraph">Condition monitoring plays a crucial role in the automotive space because they reduce the downtime and maintenance costs by preventing sudden catastrophic breakdown of vehicles. Condition monitoring solutions primarily monitor onboard sensor data to detect anomalies. However, with vehicles becoming more complex each day, the number of sensors to be monitored is also growing. This increases the data volume to be processed to make decisions. It is not feasible to send all the embedded sensor data captured to the cloud for processing. The network bandwidth, latency for transferring the data and finally the cost associated with data transfer are the factors which impede us from taking this approach. Much of the previous work done to address these issues has proposed Deep Learning driven onboard anomaly detection approaches. The deployment of these implementations requires power hungry and costly hardware with high computational resources. The recent advances in the field of Tiny Machine Learning have made it possible to design and deploy Deep Neural Networks on resource constrained low-cost embedded hardware. In this paper, we propose an Auto-Encoder based approach for anomaly detection in time-series vibration sensor data. The designed Auto-Encoder model has a 7.5 KB footprint and was finally deployed on a highly resource constrained ARM Cortex-M4 microcontroller with 256KB of SRAM and 1MB Flash. The model has been validated on the previous machine data. It has achieved an accuracy and precision close to 80%. Currently, by using post training quantization we are trading-off model accuracy for a reduction in model size. In future, we plan to use Quantization Aware Training which will help us in achieving even higher model accuracy.</div></div>

Deep Crash Detection From Vehicular Sensor Data With Multimodal Self-Supervision

The ability to detect vehicle accidents from on-board sensor data is of the utmost importance to provide prompt assistance to prevent injuries and fatalities. In this article, we present a novel deep learning method capable of analyzing time series recorded from Inertial Measurement Units (IMU) and GPS devices to recognize the presence of an accident along with its severity. We propose a neural architecture capable of exploiting the different sensor streams (i.e., acceleration, gyroscope, and GPS speed), a multimodal contrastive self-supervised training procedure, and an ad-hoc stack of data augmentation techniques, specifically designed to counteract the extreme class imbalance and to improve the generalization capabilities of the whole pipeline. The proposed method has been validated against several state-of-the-art methods on a large and highly imbalanced dataset, composed of more than 200 thousand time series collected from US vehicles, with different vehicle sizes and traveling on different types of road. Our method achieves an average-precision score (AP) of 0.9 in the detection of crashes and 0.76 in the detection of severe crashes, significantly outperforming all the other approaches, and has small footprint and latency, so that it can easily be deployed on embedded devices.

A Physics-Data Co-Operative Ship Dynamic Model for a Docking Operation

The development of onboard sensors is bringing us to the next level of ship digitalization. Its ultimate goal is to ensure safe & efficient marine operation by ship intelligence. In particular, during a docking operation, situation awareness based on precise motion prediction is of great importance. Knowledge-based ship models, developed based on the understanding of ship dynamics and simplifications, have played an important role in ship intelligence. However, they do not fully handle highly nonlinear and complex ship dynamics in the docking operation beyond our explicit understanding. On the contrary, data-driven models deal with such non-linearity and complexity in a non-parametric manner, however, the maritime industry does not regard them as reliable and applicable models due to the lack of interpretability and the physics foundation. To alleviate this dilemma, this study proposes a physics-data co-operative ship dynamic model for the docking operation; a knowledge-based ship model serves as a physics foundation with supportive data-driven models compensating a single-step-ahead velocity prediction error made by a physics foundation model. Neural networks trained with onboard sensor data are employed in supportive data-driven models. In a case study, we conducted full-scale docking operations of a 28.9m-length research vessel Gunnerus. The results show that mean prediction error in a position at 30s future is reduced by 34.6% compared to that made solely by a physics foundation model. The present approach will be the first step in the development of high-fidelity and cost-efficient ship dynamic models, thus contributing to ship autonomy in the future.

Attention Horizon as a Predictor for the Fuel Consumption Rate of Drivers.

Understanding the operation of complex assets such heavy-duty vehicles is essential for improving the efficiency, sustainability, and safety of future industry. Specifically, reducing energy consumption of transportation is crucially important for fleet operators, due to the impact it has on decreasing energy costs and lowering greenhouse gas emissions. Drivers have a high influence on fuel usage. However, reliably estimating driver performance is challenging. This is a key component of many eco-driving tools used to train drivers. Some key aspects of good, or efficient, drivers include being more aware of the surroundings, adapting to the road situations, and anticipating likely developments of the traffic conditions. With the development of IoT technologies and possibility of collecting high-precision and high-frequency data, even such vague concepts can be qualitatively measured, or at least approximated. In this paper, we demonstrate how the driver’s degree of attention to the road can be automatically extracted from onboard sensor data. More specifically, our main contribution is introduction of a new metric, called attention horizon (AH); it can, fully automatically and based on readily-available IoT data, capture, differentiate, and evaluate a driver’s behavior as the vehicle approaches a red traffic light. We suggest that our measure encapsulates complex concepts such as driver’s “awareness” and “carefulness” in itself. This metric is extracted from the pedal positions in a 150 m trajectory just before stopping. We demonstrate that this metric is correlated with normalized fuel consumption rate (FCR) in the long term, making it a suitable tool for ranking and evaluating drivers. For example, over weekly periods we found a negative median correlation between AH and FCR with the absolute value of ; while using monthly data, the value was .

Momentum-Based Extended Kalman Filter for Thrust Estimation on Flying Multibody Robots

Effective control design of flying vehicles requires a reliable estimation of the propellers’ thrust forces to secure a successful flight. Direct measurements of thrust forces, however, are seldom available in practice and on-line thrust estimation usually follows from the application of fusion algorithms that process on-board sensor data. This letter proposes a framework for the estimation of the thrust intensities on flying multibody systems that are not equipped with sensors for direct thrust measurement. The key ingredient of the proposed framework is the so-called centroidal momentum of a multibody system, which combined with the propeller model. It enables the design of Extended Kalman Filters (EKF) for on-line thrust estimation. The presented approach tackles the additional complexity in thrust estimation due to the possibly <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">large</i> number of degrees of freedom of the system and uncertainties in the propeller model. For instance, a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">covariance scheduling</i> approach based on the turbines RPM error is proposed to ensure a reliable estimation even in case of turbine failures. Simulations are presented to validate the proposed algorithm during robot flight. Moreover, an experimental setup is designed to evaluate the accuracy of the estimation algorithm using iRonCub, a jet-powered humanoid robot, while standing on the ground.

A Multiple-Output Hybrid Ship Trajectory Predictor With Consideration for Future Command Assumption

Onboard sensors contribute to data-driven understanding of complex and nonlinear ship dynamics in real time. By using sensors, precise ship trajectory prediction plays a key role in intelligent collision avoidance. A hybrid predictor makes prediction based on a mathematical model of which error is compensated by a black-box model. A Multiple-output Hybrid Predictor (MHP), which makes a long-horizon prediction at a time based on onboard sensor data, was developed in the previous study. However, it cannot handle a time series of future command assumption. This limitation hinders an MHP from being applied to the evaluation of future command assumption in the predictive decision making. A novel architecture of MHP presented in this study converts a long time series of future command assumption into a fixed-length model-based-predicted vessel state; then, it is included in inputs of a black-box error compensator. This idea is robust to multidimensionality of commands and long control horizon. Assuming a low-fidelity vessel model and a limited data are available, simulation experiments are conducted. The effect of environmental disturbances and maneuverings on the prediction performance is examined comprehensively for the first time. The present study successfully incorporates a long time series of future command assumption in an MHP. It reduces the mean error by 81.8% compared to a model-based predictor and by 45.6% compared to a data-driven predictor under Beaufort wind force scale 4 wave, wind, and ocean current. Present study expands the application of MHP to the predictive decision making of future autonomous ships.

Mining navigable water current information from ship-based big automated identification system data

High-resolution current information including direction and velocity in navigable water is essential for the safe navigation of ships. There is a lack of effective observation methods for current information in the navigable water environment. In this paper, the Current Information Mining Model (CIMM) is proposed to extract high-resolution current information in the navigable water environment based on on-board sensor data from the Automatic Identification System (AIS). In this model, a deep network based on the Gated Recurrent Unit (GRU) and bidirectional recurrent neural networks (BRNNs) is constructed to retrieval the spatial-temporal features of the vessel track, and an experience matrix (EM) is put forward to generate global attention. The results show that current direction prediction error of 83.5% of the data is less than 20°, current speed prediction error of 86.6% of the data is less than 0.3 m/s, the predicted data of the CIMM can reflect the real current information in navigable waterways. This study proposes a model for extracting current information in navigable waters, and explores a new way to obtain traffic environment information from sensor observation data carried by mobile vehicles.

Integrating Battery Aging in the Optimization for Bidirectional Charging of Electric Vehicles

Smart charging of Electric Vehicles (EVs) reduces operating cost, allows more sustainable battery usage, and promotes the rise of electric mobility. In addition, bidirectional charging and improved connectivity enable efficient power grid support. Today, however, uncoordinated charging, e.g., governed by users’ habits, is still the norm. Thus, the impact of upcoming smart charging applications is mostly unexplored. We aim to estimate the expenses inherent with smart charging, e.g., battery aging costs, and give suggestions for further research. Using typical onboard sensor data we concisely model and validate an EV battery. We then integrate the battery model into a realistic smart charging use case and compare it with measurements of real EV charging. The results show that i) the temperature dependence of battery aging calls for precise thermal models for charging power greater than 7 kW, ii) disregarding battery aging underestimates EVs’ operating cost by approx. 30%, and iii) the profitability of Vehicle-to-Grid (V2G) services based on bidirectional power flow, e.g., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">energy arbitrage</i> , depends on battery aging costs and the electricity price spread.

Rail-wheel contact forces and track irregularity estimation from on-board accelerometer data

Rolling stock approval protocols, such as UIC 518, demand monitoring of lateral and vertical forces at rail-wheel contact. Continuous monitoring through on-board instrumentation needs algorithms to convert the on-board sensor data into rail-wheel forces and track irregularities. In this work a Feed Forward Neural Network architecture, for fast and accurate estimation of rail-wheel forces and track irregularities from data obtained from a cluster of accelerometers mounted on the rolling stock, is presented. Simulation study is conducted for a passenger coach running over numerically generated set of one hundred rail tracks, each with a different irregularity. Power spectral density (PSD) function of ERRI B176 is used for creating rail irregularities. Multi-body rail coach model consists of wheel-sets, axle-box, bogie, bolster and car body. The input comprises acceleration, yaw and roll rate data from axle-box, bogie and car body, to produce contact forces and rail irregularities as outputs. Goodness of fit between actual and estimated values is illustrated through point-by-point graphs of actual and estimated track forces and irregularities. R-Squared () values, representing the fraction by which the variance of errors are less than the variance of actual values, are computed as indices of the accuracy of estimates.

A method to keep autonomous vehicles steadily drive based on lane detection

Existing studies on autonomous driving methods focus on the fusion of onboard sensor data. However, the driving behavior might be unsteady because of the uncertainties of environments. In this article, an expectation line is proposed to quantify the driving behavior motivated by the driving continuity of human drivers. Furthermore, the smooth driving could be achieved by predicting the future trajectory of the expectation line. First, a convolutional neural network-based method is applied to detect lanes in images sampled from driving video. Second, the expectation line is defined to model driving behavior of an autonomous vehicle. Finally, the long short-term memory-based method is applied to the expectation line so that the future trajectory of the vehicle could be predicted. By incorporating convolutional neural network- and long short-term memory-based methods, the autonomous vehicles could smoothly drive because of the prior information. The proposed method is evaluated using driving video data, and the experimental results demonstrate that the proposed method outperforms methods without trajectory predictions.

Vessel hydrodynamic model tuning by Discrete Bayesian updating using simulated onboard sensor data

Vessel and wave hydrodynamics are fundamental for vessel motion prediction. Improving hydrodynamic model accuracy without compromising computational efficiency has always been of high interest for safe and cost-effective marine operations. With continuous development of sensor technology and computational capacity, an improved digital twin concept for vessel motion prediction can be realized based on an onboard online adaptive hydrodynamic model. This article proposes and demonstrates a practical approach for tuning of important vessel hydrodynamic model parameters based on simulated onboard sensor data of vessel motion response. The algorithm relies fundamentally on spectral analysis, probabilistic modelling and the discrete Bayesian updating formula. All case studies show promising and reasonable tuning results. Sensitivities of the approach with respect to its key parameters were also studied. Sensor noise has been considered. The algorithm is found to be computationally efficient, robust and stable when tuning the values of hydrodynamic parameters and updating their uncertainties, within reasonable sensor noise levels.