Multi-sensor Data Integration Research Articles

Augmented reality navigation method based on image segmentation and sensor tracking registration technology

With the rapid development of modern science and technology, navigation technology provides great convenience for people's life, but the problem of inaccurate localization in complex environments has always been a challenge that navigation technology needs to be solved urgently. To address this challenge, this paper proposes an augmented reality navigation method that combines image segmentation and multi-sensor fusion tracking registration. The method optimizes the image processing process through the GA-OTSU-Canny algorithm and combines high-precision multi-sensor information in order to achieve accurate tracking of positioning and guidance in complex environments. Experimental results show that the GA-OTSU-Canny algorithm has a faster image edge segmentation rate, and the fastest start speed is only 1.8 s, and the fastest intersection selection time is 1.2 s. The navigation system combining the image segmentation and sensor tracking and registration techniques has a highly efficient performance in real-world navigation, and its building recognition rates are all above 99%. The augmented reality navigation system not only improves the navigation accuracy in high-rise and urban canyon environments, but also significantly outperforms traditional navigation solutions in terms of navigation startup time and target building recognition accuracy. In summary, this research not only provides a new framework for the theoretical integration of image processing and multi-sensor data, but also brings innovative technical solutions for the development and application of practical navigation systems.

High-resolution mapping of seasonal snow cover extent in the Pamir Hindu Kush using machine learning-based integration of multi-sensor data

High-resolution mapping of seasonal snow cover extent in the Pamir Hindu Kush using machine learning-based integration of multi-sensor data

Multisensor and Multiscale Data Integration Method of TLS and GPR for Three-Dimensional Detailed Virtual Reconstruction.

Integrated TLS and GPR data can provide multisensor and multiscale spatial data for the comprehensive identification and analysis of surficial and subsurface information, but a reliable systematic methodology associated with data integration of TLS and GPR is still scarce. The aim of this research is to develop a methodology for the data integration of TLS and GPR for detailed, three-dimensional (3D) virtual reconstruction. GPR data and high-precision geographical coordinates at the centimeter level were simultaneously gathered using the GPR system and the Global Navigation Satellite System (GNSS) signal receiver. A time synchronization algorithm was proposed to combine each trace of the GPR data with its position information. In view of the improved propagation model of electromagnetic waves, the GPR data were transformed into dense point clouds in the geodetic coordinate system. Finally, the TLS-based and GPR-derived point clouds were merged into a single point cloud dataset using coordinate transformation. In addition, TLS and GPR (250 MHz and 500 MHz antenna) surveys were conducted in the Litang fault to assess the feasibility and overall accuracy of the proposed methodology. The 3D realistic surface and subsurface geometry of the fault scarp were displayed using the integration data of TLS and GPR. A total of 40 common points between the TLS-based and GPR-derived point clouds were implemented to assess the data fusion accuracy. The difference values in the x and y directions were relatively stable within 2 cm, while the difference values in the z direction had an abrupt fluctuation and the maximum values could be up to 5 cm. The standard deviations (STD) of the common points between the TLS-based and GPR-derived point clouds were 0.9 cm, 0.8 cm, and 2.9 cm. Based on the difference values and the STD in the x, y, and z directions, the field experimental results demonstrate that the GPR-derived point clouds exhibit good consistency with the TLS-based point clouds. Furthermore, this study offers a good future prospect for the integration method of TLS and GPR for comprehensive interpretation and analysis of the surficial and subsurface information in many fields, such as archaeology, urban infrastructure detection, geological investigation, and other fields.

Three-Dimensional Mapping of Habitats Using Remote-Sensing Data and Machine-Learning Algorithms

Progress toward habitat protection goals can effectively be performed using satellite imagery and machine-learning (ML) models at various spatial and temporal scales. In this regard, habitat types and landscape structures can be discriminated against using remote-sensing (RS) datasets. However, most existing research in three-dimensional (3D) habitat mapping primarily relies on same/cross-sensor features like features derived from multibeam Light Detection And Ranging (LiDAR), hydrographic LiDAR, and aerial images, often overlooking the potential benefits of considering multi-sensor data integration. To address this gap, this study introduced a novel approach to creating 3D habitat maps by using high-resolution multispectral images and a LiDAR-derived Digital Surface Model (DSM) coupled with an object-based Random Forest (RF) algorithm. LiDAR-derived products were also used to improve the accuracy of the habitat classification, especially for the habitat classes with similar spectral characteristics but different heights. Two study areas in the United Kingdom (UK) were chosen to explore the accuracy of the developed models. The overall accuracies for the two mentioned study areas were high (91% and 82%), which is indicative of the high potential of the developed RS method for 3D habitat mapping. Overall, it was observed that a combination of high-resolution multispectral imagery and LiDAR data could help the separation of different habitat types and provide reliable 3D information.

Assessing the consistency of crop leaf area index derived from seasonal Sentinel-2 and Landsat 8 imagery over Manitoba, Canada

To take full advantage of cloud-free optical remote sensing data for crop leaf area index (LAI) retrieval throughout the growing season, integration of multi-sensor satellite data is increasingly resorted. However, the consistencies of LAI products derived from different satellites using different retrieval approaches should be assessed. This study aimed at understanding the spatiotemporal consistency of crop LAI derived from S2/MSI (Multi Spectral Instrument, Sentinel-2) and L8/OLI (Operational Land Imager, Landsat 8) data using two retrieving approaches, a field-data-driven (FDD) approach driven by field measured LAI and a hybrid approach based on simulations of the PROSAIL radiative transfer model. Results from the two approaches were assessed using field measured LAI for six types of crops in 2016 over Manitoba, one of the major agricultural provinces in the Canadian Prairies. LAI estimates from S2/MSI data obtained unbiased root-mean-square-error (ubRMSE) of 0.39 and 0.51 cm2 cm−2 for the FDD and hybrid approaches respectively. For the L8/OLI, ubRMSE of LAI estimates were 0.82 and 0.92 cm2cm−2 for the FDD and hybrid approaches, respectively. To evaluate the temporal consistency of crop LAI derived from the two satellites, daily crop LAI was reconstructed using the Parametric Double-Hyperbolic Tangent model (PDHT) with the aid of Bayesian statistical approach. Time series crop LAI of the hybrid approach had lower variability of estimated parameters in the PDHT and reconstructed daily LAI compared with that of the FDD approach. Based on the Spatial Efficiency Metric (SPAEF) that is a spatial performance metric for assessing spatial pattern similarity, the hybrid approach obtained relatively strong spatial consistency of LAI derived from the two satellites compared with the FDD approach. The study revealed that hybrid approach can achieve good spatiotemporal consistencies in seasonal LAI estimates from the two satellites combined when large numbers of field measurements are not available.

Multi-sensor based strategy learning with deep reinforcement learning for unmanned ground vehicle

As intelligent Unmanned Ground Vehicles (UGVs) find broader applications in areas such as transportation and logistics. The fusion of multiple sensors becomes crucial, since it not only amplifies UGV perception in dynamic scenarios but also underpins their autonomous decision-making capabilities. However, many existing methods only focus on single-sensor data, overlooking the multi-sensor data integration, thereby limiting UGV's scalability and adaptability. In this paper, we introduce the Multi-Sensor Collaborative Decision Network (MSCDN) for autonomous multi-sensor fusion policy learning designed specifically for UGVs. MSCDN is dedicated to integrate the data collected by multi-sensors in simulation environment and can be migrate to real environment. Firstly, a simulation environment mirroring real environment is created, using a framework that transfers UGV decision-making from simulated to real environment with deep reinforcement learning. Secondly, MSCDN uses a multi-sensor attention fusion network to adaptively integrate sensor data, refining UGV responses in dynamic settings. Thirdly, MSCDN's efficacy is tested on both simulated and real UGV lane-keeping tasks, showcasing its superior performance in comparative experiments. Compared to baseline methods, MSCDN reduces training steps and achieves a 35.71 % higher success rate and a 37.5 % quicker task completion time, underlining its proficient multi-sensor data fusion capability.

Improved deformation modelling of structures by least-squares variance component estimation based on multi-sensor data integration

In this contribution, to improve the deformation modelling based on data integration, the LS-VCE algorithm is proposed by obtaining a stochastic model of input multi-sensor data. So, one can achieve the accurate variance-covariance matrix of multi-sensor observations to participate in iterative least-squares. A practical application was made for the settlement observations from geotechnical settlement-meters and geodetic levelling (respectively known as internal and external sensors) to model the surface settlement variation of the Karkhe earth-dam. The determined variance component shows less contribution of the geotechnical settlements in the deformation modelling. An achievement of this paper is that the LS-VCE method improves the integration of the geotechnical with geodetic data by estimating an optimal stochastic model resulting in deformation model optimization. Validation results of estimated surface settlements on the check-points show an RMSE of about 3 cm and a relative-error of about 14%, which indicates the success of the modelling.

REMOTE SENSING AND CITY INFORMATION MODELING FOR REVEALING THE COMPLEXITY OF HISTORICAL CENTERS

Abstract. Historical centers represent the outcome of transformations and stratifications of the cities across the centuries. The knowledge of a historical urban environment requires an analytical methodology articulated on several interconnected levels of investigation to model a multi-layered complexity that encompasses the geometric and stylistic features of places (blocks irregularities, narrow streets, stratified buildings), the accessibility (pedestrial zone, no flyzone), the use of existing data (GIS, cartographies). Today the challenge for historical centers is dual: on the one side to make use of expeditious technologies to acquire data, on the other one to create 3D city models that allow to manage, visualize, enquire and use these data in a unique digital ecosystem. Our research deals with a multi-sensor data acquisition, evaluation and integration with the aim of creating informed and responsive 3D city models (CIM) that constitute a synthesis of the survey conducted and become the support for simulations in various contexts (seismic risk, hydraulic, energy performance).

Multi-scale, multi-sensor data integration for automated 3-D geological mapping

Enhanced digital outcrop models attributed with hyperspectral reflectance data, or hyperclouds, provide a flexible, three-dimensional medium for data-driven mapping of geological exposures, mine faces or cliffs. This approach facilitates the collection of spatially contiguous information on exposed mineralogy, and so helps to quantify mineralising processes, interpret 1-D drillhole data, and optimise mineral extraction. In this contribution we present an open-source python workflow, hylite, for creating hyperclouds by seamlessly fusing geometric information with data from a variety of hyperspectral imaging sensors and applying necessary atmospheric and illumination corrections. These rich datasets can be analysed using a variety of techniques, including minimum wavelength mapping and spectral indices, to accurately map geological objects from a distance. Reference spectra from spectral libraries, ground or laboratory measurements can also be included to derive supervised classifications using machine learning techniques. We demonstrate the potential of the hypercloud approach by integrating hyperspectral data from laboratory, tripod and unmanned aerial vehicle acquisitions to automatically map relevant lithologies and alterations associated with volcanic hosted massive sulphide (VHMS) mineralisation in the Corta Atalaya open-pit, Spain. These analyses allow quantitative and objective mineral mapping at the outcrop and open-pit scale, facilitating quantitative research and smart-mining approaches. Our results highlight the seamless sensor integration made possible with hylite and the power of data-driven mapping methods applied to hyperclouds. Significantly, we also show that random forests (RF) trained only on laboratory data from labelled hand-samples can be used to map appropriately corrected outcrop scale data.

Biomass retrieval based on genetic algorithm feature selection and support vector regression in Alpine grassland using ground-based hyperspectral and Sentinel-1 SAR data

ABSTRACT A general framework for the integration of multi-sensor data for dry and fresh biomass retrieval is proposed and tested in Alpine meadows and pastures. To this purpose, hyperspectral spectroradiometer (as simulation of hyperspectral imagery) and biomass samples were collected in field campaigns and Copernicus Sentinel-1 Interferometric Wide (IW) swath SAR backscattering coefficients were used. First, a genetic algorithm feature selection was performed on hyperspectral data, and afterwards the resulting most sensitive bands where combined with SAR data within a support vector regression (SVR) model. The most sensitive hyperspectral bands were mainly located in different regions of the SWIR range for both fresh and dry biomass, and in the red and near-infrared regions mainly for dry biomass, but with less influence for fresh biomass. The R 2 correlation values between the sampled and the estimated biomass range from 0.24 to 0.71. The relatively low performances are mainly related to the saturation effect in the optical bands, as well as to the paucity of points for high values of biomass. The methodology allows a better understanding of the interaction between grassland systems and the electromagnetic spectrum by offering a model with a reduced number of narrow bands in the context of a multi-sensor integration.

Human Falling Detection Algorithm Based on Multisensor Data Fusion with SVM

Falling is a common phenomenon in the life of the elderly, and it is also one of the 10 main causes of serious health injuries and death of the elderly. In order to prevent falling of the elderly, a real-time fall prediction system is installed on the wearable intelligent device, which can timely trigger the alarm and reduce the accidental injury caused by falls. At present, most algorithms based on single-sensor data cannot accurately describe the fall state, while the fall detection algorithm based on multisensor data integration can improve the sensitivity and specificity of prediction. In this study, we design a fall detection system based on multisensor data fusion and analyze the four stages of falls using the data of 100 volunteers simulating falls and daily activities. In this paper, data fusion method is used to extract three characteristic parameters representing human body acceleration and posture change, and the effectiveness of the multisensor data fusion algorithm is verified. The sensitivity is 96.67%, and the specificity is 97%. It is found that the recognition rate is the highest when the training set contains the largest number of samples in the training set. Therefore, after training the model based on a large amount of effective data, its recognition ability can be improved, and the prevention of fall possibility will gradually increase. In order to compare the applicability of random forest and support vector machine (SVM) in the development of wearable intelligent devices, two fall posture recognition models were established, respectively, and the training time and recognition time of the models are compared. The results show that SVM is more suitable for the development of wearable intelligent devices.

A Deep Learning Based Data Fusion Method for Degradation Modeling and Prognostics

Degradation modeling is a critical and challenging problem as it serves as the basis for system prognostics and evolution mechanism analysis. In practice, multiple sensors are used to monitor the status of a system. Thus, multisensor data fusion techniques have been proposed to capture comprehensive information for prognostic modeling and analysis, which aims at developing a composite health index (HI) through the fusion of multiple sensor signals. In the literature, most existing methods use a linear data-fusion model for integration of multisensor data to construct the HI, which is insufficient to model nonlinear relations between sensing signals and HI in a complicated system. This article proposes a novel data fusion method based on deep learning for HI construction for prognostic analysis. A pair of adversarial networks is proposed to enable the training procedure of neural networks. To guarantee the stability of the algorithm, we propose a root mean square propagation (i.e., RMSprop)-based sampling algorithm to estimate model parameters. A set of simulation studies and a case study on a set of degradation signals of aircraft engines are conducted. The results demonstrate that the proposed method has a significant improvement on remaining useful life prediction compared to existing data fusion methods.

Geospatial Technology Methods for Carbon Stock Assessment: A Comprehensive Review

Carbon stock estimation is becoming an important component for fighting against global warming. Assessment of biomass can give an idea of the amount of CO2 that can be removed from the atmosphere by forests and other plantations. Geo-Spatial technologies, including Remote Sensing (RS) and Geographic Information Systems (GIS), offer the mean to enable rapid assessment of terrestrial biomass over large areas in a timely and cost-effective manner, allowing for the estimation of above and below ground biomass. Hence, the deployment of an integrating RS-GIS approach for precision carbon management is of high significance. This article provides a review of various RS and GIS techniques used in forest aboveground biomass mapping and monitoring as well as highpoints the associated challenges and opportunities. The review concluded that the use of RS and GIS in large-scale forest aboveground biomass assessment provides a sound alternative when compared to the use of conventional approaches. It was noted that the freely available moderate resolution optical sensors could be used reliably for estimating forest carbon stock. Furthermore, the integration of multi-sensor data in a GIS environment increased the accuracy of the estimation results. This study helps contributing to the topic in a way that it illustrates the growing developments using geospatial technologies by identifying most sensitive RS variables to measurable biophysical parameters. Furthermore, it demonstrates the usefulness of geospatial technologies for estimating terrestrial carbon sequestered. Finally, gaps, limitations and the need for further studies are underlined.

Integration of Multi-Sensor Data to Estimate Plot-Level Stem Volume Using Machine Learning Algorithms–Case Study of Evergreen Conifer Planted Forests in Japan

The development of new methods for estimating precise forest structure parameters is essential for the quantitative evaluation of forest resources. Conventional use of satellite image data, increasing use of terrestrial laser scanning (TLS), and emerging trends in the use of unmanned aerial systems (UASs) highlight the importance of modern technologies in the realm of forest observation. Each technology has different advantages, and this work seeks to incorporate multiple satellite, TLS- and UAS-based remote sensing data sets to improve the ability to estimate forest structure parameters. In this paper, two regression analysis approaches are considered for the estimation: random forest regression (RFR) and support vector regression (SVR). To collect the dependent variable, in situ measurements of individual tree parameters (tree height and diameter at breast height (DBH)) were taken in a Japanese cypress forest using the nondestructive TLS method, which scans the forest to obtain dense and accurate point clouds under the tree canopy. Based on the TLS data, the stem volume was then computed and treated as ground truth information. Topographic and UAS information was then used to calculate various remotely sensed explanatory variables, such as canopy size, canopy cover, and tree height. Canopy cover and canopy shapes were computed via the orthoimages derived from the UAS and watershed segmentation method, respectively. Tree height was computed by combining the digital surface model (DSM) from the UAS and the digital terrain model (DTM) from the TLS data. Topographic variables were computed from the DTM. The backscattering intensity in the satellite imagery was obtained based on L-band (Advanced Land Observing Satellite-2 (ALOS-2) Phased Array type L-band Synthetic Aperture Radar-2 (PALSAR-2)) and C-band (Sentinel-1) synthetic aperture radar (SAR). All satellite (10–25 m resolution), TLS (3.4 mm resolution) and UAS (2.3–4.6 cm resolution) data were then combined, and RFR and SVR were trained; the resulting predictive powers were then compared. The RFR method yielded fitting R2 up to 0.665 and RMSE up to 66.87 m3/ha (rRMSE = 11.95%) depending on the input variables (best result with canopy height, canopy size, canopy cover, and Sentinel-1 data), and the SVR method showed fitting R2 up to 0.519 and RMSE up to 80.12 m3/ha (rRMSE = 12.67%). The RFR outperformed the SVR method, which could delineate the relationship between the variables for better model accuracy. This work has demonstrated that incorporating various remote sensing data to satellite data, especially adding finer resolution data, can provide good estimates of forest parameters at a plot level (10 by 10 m), potentially allowing advancements in precision forestry.

Integrating Stereo Images and Laser Altimeter Data of the ZY3-02 Satellite for Improved Earth Topographic Modeling

The positioning accuracy is critical for satellite-based topographic modeling in cases of exterior orientation parameters with high uncertainty and scarce ground control data. The integration of multi-sensor data can help to ensure precision topographical modeling in such situations. Presently, research on the combined processing of optical camera images and laser altimeter data has focused on planetary observations, especially on the Moon and Mars. This study presents an endeavor to establish a combined adjustment model with one constraint in image space for integration of ZY3-02 stereo images and laser altimeter data for improved Earth topographic modeling. The geometric models for stereo images and laser altimeter data were built first, and then, the laser ranging information was introduced to construct a combined adjustment model on the basis of the block adjustment model. One constraint that minimized the back-projection discrepancies in image space was incorporated into the combined adjustment. Datasets in several areas were collected as experimental data for the validation work. Experimental results demonstrated that the inconsistencies between stereo images and laser altimeter data for the ZY3-02 satellite can be reduced, and the elevation accuracy of stereo images can be significantly improved after applying the proposed combined adjustment. Experiments further proved that the improved height accuracy is insensitive to the number and relative position of laser altimeter points (LAPs) in stereo images. Moreover, additional plane control points (PCPs) were incorporated to achieve better planimetric accuracy. Experimental results in the Dengfeng area showed that the adjustment results derived by using LAPs and additional four PCPs were only slightly lower than those for the block adjustment with four ground control points (GCPs). Generally, the proposed approach can effectively improve the quality of Earth topographic model.

Multiple Target Tracking Based on Multiple Hypotheses Tracking and Modified Ensemble Kalman Filter in Multi-Sensor Fusion.

In multi-sensor fusion (MSF), the integration of multi-sensor observation data with different observation errors to achieve more accurate positioning of the target has always been a research focus. In this study, a modified ensemble Kalman filter (EnKF) is presented to substitute the traditional Kalman filter (KF) in the multiple hypotheses tracking (MHT) to deal with the high nonlinearity that always shows up in multiple target tracking (MTT) problems. In addition, the multi-source observation data fusion is also realized by using the modified EnKF, which enables the low-precision observation data to be corrected by high-precision observation data, and the accuracy of the corrected data can be calibrated by the statistical information provided by the EnKF. Numerical studies are given to demonstrate the effectiveness of our proposed method and the results show that the MHT-EnKF method can achieve remarkable enhancement in dealing with nonlinear movement variation and positioning accuracy for MTT problems in MSF scenario.

Multi-Sensor Data Integration Using Deep Learning for Characterization of Defects in Steel Elements.

Nowadays, there is a strong demand for inspection systems integrating both high sensitivity under various testing conditions and advanced processing allowing automatic identification of the examined object state and detection of threats. This paper presents the possibility of utilization of a magnetic multi-sensor matrix transducer for characterization of defected areas in steel elements and a deep learning based algorithm for integration of data and final identification of the object state. The transducer allows sensing of a magnetic vector in a single location in different directions. Thus, it enables detecting and characterizing any material changes that affect magnetic properties regardless of their orientation in reference to the scanning direction. To assess the general application capability of the system, steel elements with rectangular-shaped artificial defects were used. First, a database was constructed considering numerical and measurements results. A finite element method was used to run a simulation process and provide transducer signal patterns for different defect arrangements. Next, the algorithm integrating responses of the transducer collected in a single position was applied, and a convolutional neural network was used for implementation of the material state evaluation model. Then, validation of the obtained model was carried out. In this paper, the procedure for updating the evaluated local state, referring to the neighboring area results, is presented. Finally, the results and future perspective are discussed.

Gait analysis with multi-sensor data integration

Feasibility assessment was performed about a gait pattern analysis based on the temporal relationships in the multidimensional data obtained from inertial measurement units (IMUs). Temporal delays between time-series data of three 6-axes accelerometer and gyroscope sensors on the left and right forefoot and over the sacrum, under normal and fast 10 m walk by 35 elderly participants. Nine temporal delay parameters were estimated in terms of maximum and significant cross-correlation in the 2-second window between all possible combinations of the time-series data, including inter-sensor parameters, suggesting applicability of the method to the whole-body analysis. One parameter successfully predicted fall history. Other parameters successfully predicted levels of need for care (AUC=0.78, accuracy=82.4% for normal walk, AUC=0.66, accuracy=76.5% for fast walk). Future studies should address whether the analysis of the detailed temporal relationships enables more accurate estimation of gait kinematics as well as gait characterization directly.

Signal Averaging for Noise Reduction in Mobile Robot 3D Measurement System

Accurate information from sensors plays an important role in precision measurement. In such measurement small component of a noise can leads to a large variation in the output of the system. For the applications like multi-sensor data integration, sequences of observations from a number of different sensors are used to integrate into a single best estimate of the state of the environment. For integration of sensory information, sensor registration is considered as an important step. This research work is focused upon reduction of a noise from the repetitive motion of the pan tilt unit. The outcome of this paper is implemented on 3D feature extraction of the mobile robot’s environment using a laser range finder. In the proposed work averaging technique is implemented on the Pan Tilt Unit (PTU) of the system where laser range finder is fixed upon the PTU to extract the 3D feature of the target objects. Registration of the angular displacement of PTU and laser range finder is an essential element of the 3D feature extraction. An experimental result of sensor registration shows a substantial improvement in signal to noise (S/N) ratio and quality of 3D feature extraction is also found improved.

Vision-Based Human Tracking Control of a Wheeled Inverted Pendulum Robot.

In this paper, a vision-based adaptive control is designed for a wheeled inverted pendulum (WIP) robot to track a moving human target by integration of multisensor data. A new algorithm is employed in the system to combine an OptiTrack camera and a Kinect camera, such that more robust and efficient performance can be achieved for human target detection and tracking. Robust adaptive control has been developed for the WIP robot to maintain its balance on two wheels and to follow the human target using visual feedback. Leader-follower control, dynamic balance control and visual tracking are efficiently combined together to achieved desired tracking and balancing performance. Extensive experiment studies have been performed to test the effectiveness of the proposed control strategies.