Depth Fusion Research Articles

SDF-SLAM: A Deep Learning Based Highly Accurate SLAM Using Monocular Camera Aiming at Indoor Map Reconstruction With Semantic and Depth Fusion

Simultaneous localization and mapping (SLAM) is considered as a key technique in augmented reality (AR), robotics and unmanned driving. In the field of SLAM, solutions based on monocular sensors have gradually become important due to their ability to recognize more environmental information with simple structures and low costs. Feature-based ORB-SLAM is popular in many applications, but it has many limitations in complex indoor scenes. Firstly, camera pose estimation based on monocular images is greatly affected by the environment; secondly, monocular images lack scale information and cannot be used to obtain image depth information; thirdly, monocular based SLAM builds a fused map of feature points that lacks semantic information, which is incomprehensible for machine. To solve the aforementioned issues, this paper proposes an SDF-SLAM model based on deep learning, which can perform camera pose estimation in a wider indoor environment and can also perform depth estimation and semantic segmentation on monocular images to obtain an understandable three-dimensional map. SDF-SLAM is tested and verified using a CPU platform and two sets of indoor scenes. The results show that the average accuracy of the predicted point cloud coordinates reaches 90%, and the average accuracy of the semantic labels reaches 67%. Moreover, compared with the state-of-the-art SLAM frameworks, such as ORB-SLAM, LSD-SLAM, and CNN-SLAM, the absolute error of the camera trajectory on indoor data with more feature points is reduced from 0.436 m, 0.495 m, and 0.243 m to 0.037 m, respectively. On indoor data with fewer feature points, they decrease from 1.826 m, 1.206 m, and 0.264 m to 0.124 m, respectively.

Digital Empowerment: Path Choice of Physical Education Construction in Primary and Secondary Schools in China

Digital teaching is an important way to promote modern education technology, realize the digital technology and sports teaching depth fusion is an important boost to the development of school sports reform. This paper analyzes the current situation of digital physical education in primary and secondary schools in China under the background of big data by using the methods of literature and logical analysis, and puts forward the following three choices for the future construction path of digital physical education : first, to improve the quality assurance and popularization scope of school physical education electronic teaching materials ; second, improve the teaching staff, strengthen the digital teaching and training of physical education teachers ; third, improve teachers, strengthen the digital teaching and training of physical education teachers

Adaptive Illumination Based Depth Sensing Using Deep Superpixel and Soft Sampling Approximation

Dense depth map capture is challenging in existing active sparse illumination based depth acquisition techniques, such as LiDAR. Various techniques have been proposed to estimate a dense depth map based on fusion of the sparse depth map measurement with the RGB image. Recent advances in hardware enable adaptive depth measurements resulting in further improvement of the dense depth map estimation. In this paper, we study the topic of estimating dense depth from depth sampling. The adaptive sparse depth sampling network is jointly trained with a fusion network of an RGB image and sparse depth, to generate optimal adaptive sampling masks. Deep learning based superpixel sampling and soft sampling approximation are applied. We show that such adaptive sampling masks can generalize well to many RGB and sparse depth fusion algorithms under a variety of sampling rates (as low as 0.0625%). The proposed adaptive sampling method is fully differentiable and flexible to be trained end-to-end with upstream perception algorithms.

A Novel Lidar-Assisted Monocular Visual SLAM Framework for Mobile Robots in Outdoor Environments

In this paper, a novel 3D lidar-assisted monocular visual SLAM (LAMV-SLAM) framework is proposed for mobile robots in outdoor environments. LAMV-SLAM can run in real-time without a GPU and build a dense map with real scale. An online photometric calibration thread is integrated into LAMV-SLAM to eliminate the photometric disturbances in images. The tracking thread combines the lidar and vision data to estimate and refine the frame-to-frame transformation. In this thread, the depth fusion algorithm is proposed to provide accurate depth values for the extracted visual features by combining the lidar points, and a novel two-stage optimization method is proposed to utilize the fused lidar-vision data to estimate the camera transformation with the real scale. A parallel mapping thread generates new map points based on depth-filter and lidar-vision data fusion. A loop closing thread further reduces the accumulative errors of the system. To verify the accuracy and efficiency of the system, we evaluated the proposed pipeline on the KITTI odometry benchmark, and our LAMV-SLAM achieves a 0.81% of relative position drift while running at over 3X real-time speed. To verify the robustness of the system in challenging environments, experiments were carried out on the NCLT and NuScenes datasets. Moreover, real-world experiments were applied to our mobile robot platform to show the practicability and validity of the proposed approach.

Progress and Review of the Research on School Physical Education Management in China

Reviewing and summarizing the research of school physical education management can promote school physical education work. The purpose of this study was twofold: (1) sorting out the achievements and shortcomings of the current research of school physical education management, and (2) putting forward prospects for the future research of school physical education management. In this review study, we followed three steps. We used the following keywords: “school physical education management”, OR “physical education management” for the literature search via the libraries resources of a large university in Shanghai, China. A total of 396 relevant articles were retrieved and then saved to a data processing computer. Then we collated and classified all the articles and then analyzed the management mode, management system and management method at the macro level, as well as the management of physical education teaching in schools, the management of physical education teachers, and the management of field equipment at the micro level. (1) Achievements: The research in the field of school physical education management gradually realized that it was difficult to adapt the "experiential" management to the new requirements of school physical education development, and began to explore management concept renewal. The research on the management model of school sports began to draw lessons from the management of school sports in developed countries and put forward innovative new models. The research on the management system of school physical education puts forward the existing problems and inadequacies based on the current management system. (2) Shortcomings: the research field of school physical education management is relatively narrow, and the research on primary and secondary education themes is scarce; Lack of empirical research; The current situation of management and countermeasures were mainly studied using questionnaire survey, and some studies ignored the basic knowledge of sports management in countermeasures analysis. Future research should take into account era background and the perspective of teaching integration; specifically, to strengthen the empirical research on the management of school physical education, broaden the school sports management research field of vision, the sports management expertise and school sports management research depth fusion, make school sports research has a broader coverage, deeper theoretical depth.

Fusion of Depth and Thermal Imaging for People Detection

The methodology presented in this paper covers the topic of automatic detection of humans based on two types of images that do not rely on the visible light spectrum, namely on thermal and depth images. Various scenarios are considered with the use of deep neural networks being extensions of Faster R-CNN models. Apart from detecting people, independently, with the use of depth and thermal images, we proposed two data fusion methods. The first approach is the early fusion method with a 2-channel compound input. As it turned out, its performance surpassed that of all other methods tested. However, this approach requires that the model be trained on a dataset containing both types of spatially and temporally synchronized imaging sources. If such a training environment cannot be setup or if the specified dataset is not sufficiently large, we recommend the late fusion scenario, i.e. the other approach explored in this paper. Late fusion models can be trained with single-source data. We introduce the dual-NMS method for fusing the depth and thermal imaging approaches, as its results are better than those achieved by the common NMS.

Enhancing the Tracking of Seedling Growth Using RGB-Depth Fusion and Deep Learning.

The use of high-throughput phenotyping with imaging and machine learning to monitor seedling growth is a tough yet intriguing subject in plant research. This has been recently addressed with low-cost RGB imaging sensors and deep learning during day time. RGB-Depth imaging devices are also accessible at low-cost and this opens opportunities to extend the monitoring of seedling during days and nights. In this article, we investigate the added value to fuse RGB imaging with depth imaging for this task of seedling growth stage monitoring. We propose a deep learning architecture along with RGB-Depth fusion to categorize the three first stages of seedling growth. Results show an average performance improvement of correct recognition rate by comparison with the sole use of RGB images during the day. The best performances are obtained with the early fusion of RGB and Depth. Also, Depth is shown to enable the detection of growth stage in the absence of the light.

Beet seedling and weed recognition based on convolutional neural network and multi-modality images

Difficulties in the recognition of beet seedlings and weeds can arise from a complex background in the natural environment and a lack of light at night. In the current study, a novel depth fusion algorithm was proposed based on visible and near-infrared imagery. In particular, visible (RGB) and near-infrared images were superimposed at the pixel-level via a depth fusion algorithm and were subsequently fused into three-channel multi-modality images in order to characterize the edge details of beets and weeds. Moreover, an improved region-based fully convolutional network (R-FCN) model was applied in order to overcome the geometric modeling restriction of traditional convolutional kernels. More specifically, for the convolutional feature extraction layers, deformable convolution was adopted to replace the traditional convolutional kernel, allowing for the entire network to extract more precise features. In addition, online hard example mining was introduced to excavate the hard negative samples in the detection process for the retraining of misidentified samples. A total of four models were established via the aforementioned improved methods. Results demonstrate that the average precision of the improved optimal model for beets and weeds were 84.8% and 93.2%, respectively, while the mean average precision was improved to 89.0%. Compared with the classical R-FCN model, the performance of the optimal model was not only greatly improved, but the parameters were also not significantly expanded. Our study can provide a theoretical basis for the subsequent development of intelligent weed control robots under weak light conditions.

RGB-D scene analysis in the NICU

Continuity of care is achieved in the neonatal intensive care unit (NICU) through careful documentation of all events of clinical significance, including clinical interventions and routine care events (e.g., feeding, diaper change, weighing, etc.). As a step towards automating this documentation process, we propose a scene recognition algorithm that can automatically identify key features in a single image of the patient environment, paired with a rule-based sentence generator to caption the scene. Color and depth video were obtained from 29 newborn patients from the Children's Hospital of Eastern Ontario (CHEO) using an Intel RealSense SR300 RGB-D camera and manual bedside event annotation. Image processing techniques are implemented to classify two lighting conditions: brightness level and phototherapy. A deep neural network is developed for three image classification tasks: on-going intervention, bed occupancy, and patient coverage. Transfer learning is leveraged in the feature extraction layers, such that weights learned from a generic data-rich task are applied to the clinical domain where data collection is complex and costly. Different depth fusion techniques are implemented and compared among classification tasks, where the depth and color data are fused as an RGB-D image (image fusion) or separately at various layers in the network (network fusion). Promising results were obtained with >84% sensitivity and >73% F1 measure across all context variables despite the large class imbalance. RGBD-based models are shown to outperform RGB models on most tasks. In general, a 4-channel image fusion and network fusion at the 11th layer of the VGG-16 architecture were preferred. Ultimately, achieving complete scene understanding through multimodal computer vision could form the basis for a semi-automated charting system to assist clinical staff.

Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

Artificial intelligence including deep learning and 3D reconstruction methods is changing the daily life of people. Now, an unmanned aerial vehicle that can move freely in the air and avoid harsh ground conditions has been commonly adopted as a suitable tool for 3D reconstruction. The traditional 3D reconstruction mission based on drones usually consists of two steps: image collection and offline post-processing. But there are two problems: one is the uncertainty of whether all parts of the target object are covered, and another is the tedious post-processing time. Inspired by modern deep learning methods, we build a telexistence drone system with an onboard deep learning computation module and a wireless data transmission module that perform incremental real-time dense reconstruction of urban cities by itself. Two technical contributions are proposed to solve the preceding issues. First, based on the popular depth fusion surface reconstruction framework, we combine it with a visual-inertial odometry estimator that integrates the inertial measurement unit and allows for robust camera tracking as well as high-accuracy online 3D scan. Second, the capability of real-time 3D reconstruction enables a new rendering technique that can visualize the reconstructed geometry of the target as navigation guidance in the HMD. Therefore, it turns the traditional path-planning-based modeling process into an interactive one, leading to a higher level of scan completeness. The experiments in the simulation system and our real prototype demonstrate an improved quality of the 3D model using our artificial intelligence leveraged drone system.

High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision

Polarization imaging can retrieve inaccurate objects’ 3D shapes with fine textures, whereas coarse but accurate depths can be provided by binocular stereo vision. To take full advantage of these two complementary techniques, we investigate a novel 3D reconstruction method based on the fusion of polarization imaging and binocular stereo vision for high quality 3D reconstruction. We first generate the polarization surface by correcting the azimuth angle errors on the basis of registered binocular depth, to solve the azimuthal ambiguity in the polarization imaging. Then we propose a joint 3D reconstruction model for depth fusion, including a data fitting term and a robust low-rank matrix factorization constraint. The former is to transfer textures from the polarization surface to the fused depth by assuming their relationship linear, whereas the latter is to utilize the low-frequency part of binocular depth to improve the accuracy of the fused depth considering the influences of missing-entries and outliers. To solve the optimization problem in the proposed model, we adopt an efficient solution based on the alternating direction method of multipliers. Extensive experiments have been conducted to demonstrate the efficiency of the proposed method in comparison with state-of-the-art methods and to exhibit its wide application prospects in 3D reconstruction.

ROSEFusion

Online reconstruction based on RGB-D sequences has thus far been restrained to relatively slow camera motions (<1m/s). Under very fast camera motion (e.g., 3m/s), the reconstruction can easily crumble even for the state-of-the-art methods. Fast motion brings two challenges to depth fusion: 1) the high nonlinearity of camera pose optimization due to large inter-frame rotations and 2) the lack of reliably trackable features due to motion blur. We propose to tackle the difficulties of fast-motion camera tracking in the absence of inertial measurements using random optimization, in particular, the Particle Filter Optimization (PFO). To surmount the computation-intensive particle sampling and update in standard PFO, we propose to accelerate the randomized search via updating a particle swarm template (PST). PST is a set of particles pre-sampled uniformly within the unit sphere in the 6D space of camera pose. Through moving and rescaling the pre-sampled PST guided by swarm intelligence, our method is able to drive tens of thousands of particles to locate and cover a good local optimum extremely fast and robustly. The particles, representing candidate poses, are evaluated with a fitness function defined based on depth-model conformance. Therefore, our method, being depth-only and correspondence-free, mitigates the motion blur impediment as (ToF-based) depths are often resilient to motion blur. Thanks to the efficient template-based particle set evolution and the effective fitness function, our method attains good quality pose tracking under fast camera motion (up to 4m/s) in a realtime framerate without including loop closure or global pose optimization. Through extensive evaluations on public datasets of RGB-D sequences, especially on a newly proposed benchmark of fast camera motion, we demonstrate the significant advantage of our method over the state of the arts.

Obstacle detection based on depth fusion of lidar and radar in challenging conditions

PurposeTo the industrial application of intelligent and connected vehicles (ICVs), the robustness and accuracy of environmental perception are critical in challenging conditions. However, the accuracy of perception is closely related to the performance of sensors configured on the vehicle. To enhance sensors’ performance further to improve the accuracy of environmental perception, this paper aims to introduce an obstacle detection method based on the depth fusion of lidar and radar in challenging conditions, which could reduce the false rate resulting from sensors’ misdetection.Design/methodology/approachFirstly, a multi-layer self-calibration method is proposed based on the spatial and temporal relationships. Next, a depth fusion model is proposed to improve the performance of obstacle detection in challenging conditions. Finally, the study tests are carried out in challenging conditions, including straight unstructured road, unstructured road with rough surface and unstructured road with heavy dust or mist.FindingsThe experimental tests in challenging conditions demonstrate that the depth fusion model, comparing with the use of a single sensor, can filter out the false alarm of radar and point clouds of dust or mist received by lidar. So, the accuracy of objects detection is also improved under challenging conditions.Originality/valueA multi-layer self-calibration method is conducive to improve the accuracy of the calibration and reduce the workload of manual calibration. Next, a depth fusion model based on lidar and radar can effectively get high precision by way of filtering out the false alarm of radar and point clouds of dust or mist received by lidar, which could improve ICVs’ performance in challenging conditions.

Probabilistic Multi-View Fusion of Active Stereo Depth Maps for Robotic Bin-Picking

The reliable fusion of depth maps from multiple viewpoints has become an important problem in many 3D reconstruction pipelines. In this work, we investigate its impact on robotic bin-picking tasks such as 6D object pose estimation. The performance of object pose estimation relies heavily on the quality of depth data. However, due to the prevalence of shiny surfaces and cluttered scenes, industrial grade depth cameras often fail to sense depth or generate unreliable measurements from a single viewpoint. To this end, we propose a novel probabilistic framework for scene reconstruction in robotic bin-picking. Based on active stereo camera data, we first explicitly estimate the uncertainty of depth measurements for mitigating the adverse effects of both noise and outliers. The uncertainty estimates are then incorporated into a probabilistic model for incrementally updating the scene. To extensively evaluate the traditional fusion approach alongside our own approach, we will release a novel representative dataset with multiple views for each bin and curated parts. Over the entire dataset, we demonstrate that our framework outperforms a traditional fusion approach by a 12.8% reduction in reconstruction error, and 6.1% improvement in detection rate. The dataset will be available at https://www.trailab.utias.utoronto.ca/robi .

Snow Depth Fusion Based on Machine Learning Methods for the Northern Hemisphere

In this study, a machine learning algorithm was introduced to fuse gridded snow depth datasets. The input variables of the machine learning method included geolocation (latitude and longitude), topographic data (elevation), gridded snow depth datasets and in situ observations. A total of 29,565 in situ observations were used to train and optimize the machine learning algorithm. A total of five gridded snow depth datasets—Advanced Microwave Scanning Radiometer for the Earth Observing System (AMSR-E) snow depth, Global Snow Monitoring for Climate Research (GlobSnow) snow depth, Long time series of daily snow depth over the Northern Hemisphere (NHSD) snow depth, ERA-Interim snow depth and Modern-Era Retrospective Analysis for Research and Applications, version 2 (MERRA-2) snow depth—were used as input variables. The first three snow depth datasets are retrieved from passive microwave brightness temperature or assimilation with in situ observations, while the last two are snow depth datasets obtained from meteorological reanalysis data with a land surface model and data assimilation system. Then, three machine learning methods, i.e., Artificial Neural Networks (ANN), Support Vector Regression (SVR), and Random Forest Regression (RFR), were used to produce a fused snow depth dataset from 2002 to 2004. The RFR model performed best and was thus used to produce a new snow depth product from the fusion of the five snow depth datasets and auxiliary data over the Northern Hemisphere from 2002 to 2011. The fused snow-depth product was verified at five well-known snow observation sites. The R2 of Sodankylä, Old Aspen, and Reynolds Mountains East were 0.88, 0.69, and 0.63, respectively. At the Swamp Angel Study Plot and Weissfluhjoch observation sites, which have an average snow depth exceeding 200 cm, the fused snow depth did not perform well. The spatial patterns of the average snow depth were analyzed seasonally, and the average snow depths of autumn, winter, and spring were 5.7, 25.8, and 21.5 cm, respectively. In the future, random forest regression will be used to produce a long time series of a fused snow depth dataset over the Northern Hemisphere or other specific regions.

Satellite-Derived Aerosol Optical Depth Fusion Combining Active and Passive Remote Sensing Based on Bayesian Maximum Entropy

Satellite-derived aerosol optical depth (AOD) is an important parameter for studies related to atmospheric environment, climate change, and biogeochemical cycle. Unfortunately, the relatively high data missing ratio of satellite-derived AOD limits the atmosphere-related research and applications to a certain extent. Accordingly, numerous AOD fusion algorithms have been proposed in recent years. However, most of these algorithms focused on merging AOD products from multiple passive sensors, which cannot complementarily recover the AOD missing values due to cloud obscuration and the misidentification between optically thin cloud and aerosols. In order to address these issues, a spatiotemporal AOD fusion framework combining active and passive remote sensing based on Bayesian maximum entropy methodology (AP-BME) is developed to provide satellite-derived AOD data sets with high spatial coverage and good accuracy in large scale. The results demonstrate that AP-BME fusion significantly improves the spatial coverage of AOD, from an averaged spatial completeness of 27.9%–92.8% in the study areas, in which the spatial coverage improves from 91.1% to 92.8% when introducing Cloud–Aerosol Lidar with Orthogonal Polarization (CALIOP) AOD data sets into the fusion process. Meanwhile, the accuracy of recovered AOD nearly maintains that of the original satellite AOD products, based on evaluation against ground-based Aerosol Robotic Network (AERONET) AOD. Moreover, the efficacy of the active sensor in AOD fusion is discussed through overall accuracy comparison and two case analyses, which shows that the provision of key aerosol information by the active sensor on haze condition or under thin cloud is important for not only restoring the real haze situations but also avoiding AOD overestimation caused by cloud optical depth (COD) contamination in AOD fusion results.

Monocular Depth Estimation Based on Multi-Scale Depth Map Fusion

Monocular depth estimation is a basic task in machine vision. In recent years, the performance of monocular depth estimation has been greatly improved. However, most depth estimation networks are based on a very deep network to extract features that lead to a large amount of information lost. The loss of object information is particularly serious in the encoding and decoding process. This information loss leads to the estimated depth maps lacking object structure detail and have non-clear edges. Especially in a complex indoor environment, which is our research focus in this paper, the consequences of this loss of information are particularly serious. To solve this problem, we propose a Dense feature fusion network that uses a feature pyramid to aggregate various scale features. Furthermore, to improve the fusion effectiveness of decoded object contour information and depth information, we propose an adaptive depth fusion module, which allows the fusion network to fuse various scale depth maps adaptively to increase object information in the predicted depth map. Unlike other work predicting depth maps relying on U-NET architecture, our depth map predicted by fusing multi-scale depth maps. These depth maps have their own characteristics. By fusing them, we can estimate depth maps that not only include accurate depth information but also have rich object contour and structure detail. Experiments indicate that the proposed model can predict depth maps with more object information than other prework, and our model also shows competitive accuracy. Furthermore, compared with other contemporary techniques, our method gets state-of-the-art in edge accuracy on the NYU Depth V2 dataset.

GNSS-R-Based Snow Water Equivalent Estimation with Empirical Modeling and Enhanced SNR-Based Snow Depth Estimation

Snow depth and snow water equivalent (SWE) are two parameters for measuring snowfall. By exploiting the Global Navigation Satellite System reflectometry (GNSS-R) technique and thousands of existing GNSS Continuous Operating Reference Stations (CORS) deployed in the cryosphere, it is possible to improve the temporal and spatial resolutions of the SWE measurement. In this paper, a fusion model for combining multi-satellite SNR (Signal to Noise Ratio) snow depth estimations is proposed, which uses peak spectral powers associated with each of the snow depth estimations. To simplify the estimation of SWE, the complete snowfall period over a winter season is split into snow accumulation, transition, and melting period in accordance with the variation characteristics of snow depth and SWE. By extensively using in situ snow depth and SWE observations recorded by snow telemetry network (SNOTEL) and regression analysis, three empirical models are developed to describe the relationship between snow depth and SWE for the three periods, respectively. Based on the snow depth fusion model and the SWE empirical models, an SWE estimation algorithm is proposed. Three data sets recorded in different environments are used to test the proposed method. The results demonstrate that there exists good agreement between the in situ SWE measurements and the SWE estimates produced by the proposed method; the root-mean-square error of SWE estimations is smaller than 6 cm when the SWE is up to 80 cm.

Guided optimization framework for the fusion of time-of-flight with stereo depth

The fusion of depth acquired actively with the depth estimated passively proved its significance as an improvement strategy for gaining depth. This combination allows us to benefit from two sources of modalities such that they complement each other. To fuse two sensor data into a more accurate depth map, we must consider the limitations of active sensing such as low lateral resolution while combining it with a passive depth map. We present an approach for the fusion of active time-of-flight depth and passive stereo depth in an accurate way. We propose a multimodal sensor fusion strategy that is based on a weighted energy optimization problem. The weights are generated as a result of combining the edge information from a texture map and active and passive depth maps. The objective evaluation of our fusion algorithm shows an improved accuracy of the generated depth map in comparison with the depth map of every single modality and with the results of other fusion methods. Additionally, a visual comparison of our result shows a better recovery on the edges considering the wrong depth values estimated in passive stereo. Moreover, the left and right consistency check on the result illustrates the ability of our approach to consistently fuse sensors.

Occlusion gesture recognition based on improved SSD

SummaryGesture recognition has always been a research hotspot in the field of human‐computer interaction. Its purpose is to realize the natural interaction with the machine by recognizing the semantics expressed by gesture. In the process of gesture recognition, the occlusion of gesture is an inevitable problem. In the process of gesture recognition, some or even all of the gesture features will be lost due to the occlusion of the gesture, resulting in the wrong recognition or even unrecognizability of the gesture. Therefore, it is of great significance to study gesture recognition under occlusion. The single shot multibox detector (SSD) algorithm is analyzed, and the front‐end network is compared. Mobilenets is selected as the front‐end network, and the Mobilenets‐SSD network is improved. In tensorflow environment, based on the improved network model, the self‐occlusion gesture and object occluding gesture are trained in color map, depth map, and color and depth fusion respectively. The recognition models of self‐occlusion gestures and object‐occlusion gestures in color map, depth map, and color and depth fusion are obtained. And compare and analyze the learning rate, loss function, and average accuracy of various models obtained for occlusion gesture recognition.