Confidence Map Research Articles

Incorporating modelling uncertainty and prior knowledge into landslide susceptibility mapping using Bayesian neural networks

ABSTRACT In landslide susceptibility (LS) mapping, the reliance on deep learning (DL) models within the frequentist paradigm often leads to an oversight of modelling uncertainties and priors, undermining reliability. This study applied Bayesian neural network (BNN) using variational inference and Bayes backpropagation to LS mapping in Lin’an, China. We developed two BNN configurations: one employing a zero-mean Gaussian distribution as priors (BNN-normal) and another grounded in domain knowledge (BNN-knowledge), and benchmarked them against deep neural networks (DNNs). The study further investigates the models’ performance against random noise impacts on conditional factors and challenges posed by incomplete landslide inventories. Results indicated although BNNs were comparable to the DNNs in generalisation, they demonstrated superior calibration accuracy, resistance to noise and overfitting. Notably, BNN-knowledge effectively mitigated the adverse effects of incomplete data. A landslide confidence map was constructed by integrating the LS and epistemic entropy. Compared to LS maps without uncertainty analysis, the landslide densities in very high and very low LS area with low uncertainty increased by 8.34 and decreased by 0.1, respectively, providing more information for scientific and effective decision-making. Our results further confirmed that sparse vegetation, anthropogenic disturbances and economic forest expansion were main factors of high LS in the region.

Do as Sonographers Think: Contrast-Enhanced Ultrasound for Thyroid Nodules Diagnosis via Microvascular Infiltrative Awareness.

Dynamic contrast-enhanced ultrasound (CEUS) imaging can reflect the microvascular distribution and blood flow perfusion, thereby holding clinical significance in distinguishing between malignant and benign thyroid nodules. Notably, CEUS offers a meticulous visualization of the microvascular distribution surrounding the nodule, leading to an apparent increase in tumor size compared to gray-scale ultrasound (US). In the dual-image obtained, the lesion size enlarged from gray-scale US to CEUS, as the microvascular appeared to be continuously infiltrating the surrounding tissue. Although the infiltrative dilatation of microvasculature remains ambiguous, sonographers believe it may promote the diagnosis of thyroid nodules. We propose a deep learning model designed to emulate the diagnostic reasoning process employed by sonographers. This model integrates the observation of microvascular infiltration on dynamic CEUS, leveraging the additional insights provided by gray-scale US for enhanced diagnostic support. Specifically, temporal projection attention is implemented on time dimension of dynamic CEUS to represent the microvascular perfusion. Additionally, we employ a group of confidence maps with flexible Sigmoid Alpha Functions to aware and describe the infiltrative dilatation process. Moreover, a self-adaptive integration mechanism is introduced to dynamically integrate the assisted gray-scale US and the confidence maps of CEUS for individual patients, ensuring a trustworthy diagnosis of thyroid nodules. In this retrospective study, we collected a thyroid nodule dataset of 282 CEUS videos. The method achieves a superior diagnostic accuracy and sensitivity of 89.52% and 94.75%, respectively. These results suggest that imitating the diagnostic thinking of sonographers, encompassing dynamic microvascular perfusion and infiltrative expansion, proves beneficial for CEUS-based thyroid nodule diagnosis.

Quantifying uncertainty in landslide susceptibility mapping due to sampling randomness

The quality of landslide and non-landslide samples plays a crucial role in the landslide susceptibility maps (LSM) generated using machine learning algorithms. However, uncertainties arising from non-landslide sample collection can significantly compromise the reliability of these maps. Current methods, such as buffer-controlled sampling (BCS), often fail to address this issue adequately. This study aims to fill that gap by employing Monte Carlo simulations combined with BCS to quantify the uncertainties associated with non-landslide sampling and improve LSM accuracy. A novel framework is proposed, integrating landslide susceptibility confidence maps (LSCMs) to address the inherent uncertainty in BCS-based LSMs. The framework evaluates inconsistencies in LSMs, showing that maps generated by the same model differ in over 30% of the area due to variations in non-landslide sample selection. The proposed approach outperforms traditional methods by correctly classifying landslide-prone areas, particularly in low and very low susceptibility zones, while providing a more reliable quantification of uncertainty. These findings underscore the limitations of traditional LSM methods and demonstrate that LSCMs offer a more robust tool for landslide hazard assessment. The framework enhances the precision of susceptibility mapping and provides critical insights for better risk mitigation and disaster preparedness.

AI-smartphone markerless motion capturing of hip, knee, and ankle joint kinematics during countermovement jumps.

Recently, AI-driven skeleton reconstruction tools that use multistage computer vision pipelines were designed to estimate 3D kinematics from 2D video sequences. In the present study, we validated a novel markerless, smartphone video-based artificial intelligence (AI) motion capture system for hip, knee, and ankle angles during countermovement jumps (CMJs). Eleven participants performed six CMJs. We used 2D videos created by a smartphone (Apple iPhone X, 4K, 60 fps) to create 24 different keypoints, which together built a full skeleton including joints and their connections. Body parts and skeletal keypoints were localized by calculating confidence maps using a multilevel convolutional neural network that integrated both spatial and temporal features. We calculated hip, knee, and ankle angles in the sagittal plane and compared it with the angles measured by a VICON system. We calculated the correlation between both method's angular progressions, mean squared error (MSE), mean average error (MAE), and the maximum and minimum angular error and run statistical parametric mapping (SPM) analysis. Pearson correlation coefficients (r) for hip, knee, and ankle angular progressions in the sagittal plane during the entire movement were 0.96, 0.99, and 0.87, respectively. SPM group-analysis revealed some significant differences only for ankle angular progression. MSE was below 5.7°, MAE was below 4.5°, and error for maximum amplitudes was below 3.2°. The smartphone AI motion capture system with the trained multistage computer vision pipeline was able to detect, especially hip and knee angles in the sagittal plane during CMJs with high precision from a frontal view only.

Feature-Level Fusion Multi-Sensor Aggregation Temporal Network for Smartphone-Based Human Activity Recognition

Smartphone-based Human Activity Recognition (HAR) identifies human movements using inertial signals gathered from multiple smartphone sensors. Generally, these signals are stacked as one (data-level fusion) and fed into deep learning algorithms for feature extractions. This research studies feature-level fusion, individually processing inertial signals from each sensor, and proposes a lightweight deep temporal learning model, Feature-Level Fusion Multi-Sensor Aggregation Temporal Network (FLF-MSATN), that performs feature extraction on inertial signals from each sensor separately. The raw signals, segmented into equally sized time windows, are passed into individual Dilated-Pooled Convolutional Heads (DPC Heads) for temporal feature analysis. Each DPC Head has a spatiotemporal block containing dilated causal convolutions and average pooling, to extract underlying patterns. The DPC Heads’ outputs are concatenated and passed into a Global Average Pooling layer to generate a condensed confidence map before activity classification. FLF-MSATN is assessed using a subject-independent protocol on a publicly available HAR dataset, UCI HAR, and a self-collected HAR dataset, achieving 96.67% and 82.70% accuracies, respectively. A Data-Level Fusion MSATN is built to compare and verify the model performance attained by the proposed FLF-MSATN. The empirical results show that implementing FLF-MSATN enhances the accuracy by ~3.4% for UCI HAR and ~9.68% for self-collected datasets.

DVDS: A deep visual dynamic slam system

Simultaneous localization and mapping (SLAM) utilizing visual sensors represent an extensively investigated research area, holding significant potential for advancements in robotics and autonomous vehicular systems. Recently, dense SLAM systems underpinned by learning-based methodologies have showcased superior accuracy and robustness compared to conventional techniques. Nevertheless, contemporary learning-based SLAM systems exhibit notable discrepancies in pose estimation, particularly within dynamic environments. In addition, the constrained receptive field of convolutional features in these methods impedes their efficacy when confronted with homogeneous, texture-less images, rendering them vulnerable to noise perturbations. We develop a novel deep visual dynamic slam (DVDS) system that exploits solely static pixels within images to retrieve the camera poses. Specifically, we formulate a dynamic object exclusion mechanism that excises dynamic constituents within the scene before the optical flow computation, thus optimizing the precision of the estimation. In addition, we unveil an efficient dispersive transformer (DisFormer) that facilitates per-pixel features in assimilating long-range information from surrounding features, culminating in constructing more precise 4D correlation volumes. Building on the DisFormer, we suggest a Disformer-based gated recurrent unit (GRU) to generate a refined flow field coupled with a confidence map, which is subsequently employed by the dense bundle adjustment layer to iteratively rectify the residuals of inverse depths and associated camera poses. The global receptive field provided by the DisFormer promotes information integration from a wider contextual window, thus improving the robustness of our SLAM system. Comprehensive experiments underscore that our proposed DVDS system manifests superior efficacy compared with state-of-the-art works across both static and dynamic scenes.

Automated confidence estimation in deep learning auto-segmentation for brain organs at risk on MRI for radiotherapy.

We have built a novel AI-driven QA method called AutoConfidence (ACo), to estimate segmentation confidence on a per-voxel basis without gold standard segmentations, enabling robust, efficient review of automated segmentation (AS). We have demonstrated this method in brain OAR AS on MRI, using internal and external (third-party) AS models. Thirty-two retrospectives, MRI planned, glioma cases were randomly selected from a local clinical cohort for ACo training. A generator was trained adversarialy to produce internal autosegmentations (IAS) with a discriminator to estimate voxel-wise IAS uncertainty, given the input MRI. Confidence maps for each proposed segmentation were produced for operator use in AS editing and were compared with "difference to gold-standard" error maps. Nine cases were used for testing ACo performance on IAS and validation with two external deep learning segmentation model predictions [external model with low-quality AS (EM-LQ) and external model with high-quality AS (EM-HQ)]. Matthew's correlation coefficient (MCC), false-positive rate (FPR), false-negative rate (FNR), and visual assessment were used for evaluation. Edge removal and geometric distance corrections were applied to achieve more useful and clinically relevant confidence maps and performance metrics. ACo showed generally excellent performance on both internal and external segmentations, across all OARs (except lenses). MCC was higher on IAS and low-quality external segmentations (EM-LQ) than high-quality ones (EM-HQ). On IAS and EM-LQ, average MCC (excluding lenses) varied from 0.6 to 0.9, while average FPR and FNR were ≤0.13 and ≤0.21, respectively. For EM-HQ, average MCC varied from 0.4 to 0.8, while average FPR and FNR were ≤0.37 and ≤0.22, respectively. ACo was a reliable predictor of uncertainty and errors on AS generated both internally and externally, demonstrating its potential as an independent, reference-free QA tool, which could help operators deliver robust, efficient autosegmentation in the radiotherapy clinic.

Efficient occlusion avoidance based on active deep sensing for harvesting robots

With the increasing shortage of agricultural labor, the development of harvesting robots is becoming more and more urgent. Most of them require vision to locate the target, however, occlusion is common in agricultural environment, which restricts the accuracy of visual target recognition, and even leads to failure in serious cases. The active perception method is an effective means, but how to efficiently find the best observation position remains difficult to avoid the waste of time caused by repeated invalid motion. Targeting these problems, an active deep sensing method is proposed for harvesting clustered and single fruits. First, the region of interest of the target is extracted by a segmentation network, and then the occlusion status of it is obtained by image processing methods. Taking the current observation position as the starting point, the camera is moved within a matrix to form confidence and occlusion rate distribution maps. After establishing a series of occlusion rate and confidence matrix datasets, a designed deep network has been trained, which is used to predict the maximum confidence/minimum occlusion rate position after the current occlusion status is estimated. To verify the reliability of the method, laboratory and field experiments were carried out for apples and clustered tomatoes. After 1000 times of verification, results show that the successful pick/recognition rate is increased by 38.7 %, and the average successful recognition time is 5.2 s, which is 63.1 % and 46.4 % faster than that of a fixed movement method and a simple heuristic method.

Utilizing RT-DETR Model for Fruit Calorie Estimation from Digital Images

Estimating the calorie content of fruits is critical for weight management and maintaining overall health as well as aiding individuals in making informed dietary choices. Accurate knowledge of fruit calorie content assists in crafting personalized nutrition plans and preventing obesity and associated health issues. In this paper, we investigate the application of deep learning models for estimating the calorie content in fruits from digital images, aiming to provide a more efficient and accurate method for nutritional analysis. We create a dataset comprising images of various fruits and employ random data augmentation techniques during training to enhance model robustness. We utilize the RT-DETR model integrated into the ultralytics framework for implementation and conduct comparative experiments with YOLOv10 on the dataset. Our results show that the RT-DETR model achieved a precision rate of 99.01% and mAP50-95 of 94.45% in fruit detection from digital images, outperforming YOLOv10 in terms of F1- Confidence Curves, P-R curves, precision, and mAP. Conclusively, in this paper, we utilize a transformer architecture to detect fruits and estimate their calorie and nutritional content. The results of the experiments provide a technical reference for more accurately monitoring an individual’s dietary intake by estimating the calorie content of fruits.

Multi-Task Credible Pseudo-Label Learning for Semi-Supervised Crowd Counting.

As a widely used semi-supervised learning strategy, self-training generates pseudo-labels to alleviate the labor-intensive and time-consuming annotation problems in crowd counting while boosting the model performance with limited labeled data and massive unlabeled data. However, the noise in the pseudo-labels of the density maps greatly hinders the performance of semi-supervised crowd counting. Although auxiliary tasks, e.g., binary segmentation, are utilized to help improve the feature representation learning ability, they are isolated from the main task, i.e., density map regression and the multi-task relationships are totally ignored. To address the above issues, we develop a multi-task credible pseudo-label learning (MTCP) framework for crowd counting, consisting of three multi-task branches, i.e., density regression as the main task, and binary segmentation and confidence prediction as the auxiliary tasks. Multi-task learning is conducted on the labeled data by sharing the same feature extractor for all three tasks and taking multi-task relations into account. To reduce epistemic uncertainty, the labeled data are further expanded, by trimming the labeled data according to the predicted confidence map for low-confidence regions, which can be regarded as an effective data augmentation strategy. For unlabeled data, compared with the existing works that only use the pseudo-labels of binary segmentation, we generate credible pseudo-labels of density maps directly, which can reduce the noise in pseudo-labels and therefore decrease aleatoric uncertainty. Extensive comparisons on four crowd-counting datasets demonstrate the superiority of our proposed model over the competing methods. The code is available at: https://github.com/ljq2000/MTCP.

Identifying and fitting eclipse maps of exoplanets with cross-validation

ABSTRACT Eclipse mapping uses the shape of the eclipse of an exoplanet to measure its two-dimensional structure. Light curves are mostly composed of longitudinal information, with the latitudinal information only contained in the brief ingress and egress of the eclipse. This imbalance can lead to a spuriously confident map, where the longitudinal structure is constrained by out-of-eclipse data and the latitudinal structure is wrongly determined by the priors on the map. We present a new method to address this issue. The method tests for the presence of an eclipse mapping signal using k-fold cross-validation to compare the performance of a simple mapping model to the null hypothesis of a uniform disc. If a signal is found, the method fits a map with more degrees of freedom, optimizing its information content. The information content is varied by penalizing the model likelihood by a factor proportional to the spatial entropy of the map, optimized by cross-validation. We demonstrate this method for simulated data sets then apply it to three observational data sets. The method identifies an eclipse mapping signal for JWST MIRI/LRS observations of WASP-43b but does not identify a signal for JWST NIRISS/SOSS observations of WASP-18b or Spitzer Space Telescope observations of HD 189733b. It is possible to fit eclipse maps to these data sets, but we suggest that these maps are overfitting the eclipse shape. We fit a new map with more spatial freedom to the WASP-43b data set and show a flatter east–west structure than previously derived.

Interaction between clinicians and artificial intelligence to detect fetal atrioventricular septal defects on ultrasound: how can we optimize collaborative performance?

Artificial intelligence (AI) has shown promise in improving the performance of fetal ultrasound screening in detecting congenital heart disease (CHD). The effect of giving AI advice to human operators has not been studied in this context. Giving additional information about AI model workings, such as confidence scores for AI predictions, may be a way of further improving performance. Our aims were to investigate whether AI advice improved overall diagnostic accuracy (using a single CHD lesion as an exemplar), and to determine what, if any, additional information given to clinicians optimized the overall performance of the clinician-AI team. An AI model was trained to classify a single fetal CHD lesion (atrioventricular septal defect (AVSD)), using a retrospective cohort of 121 130 cardiac four-chamber images extracted from 173 ultrasound scan videos (98 with normal hearts, 75 with AVSD); a ResNet50 model architecture was used. Temperature scaling of model prediction probability was performed on a validation set, and gradient-weighted class activation maps (grad-CAMs) produced. Ten clinicians (two consultant fetal cardiologists, three trainees in pediatric cardiology and five fetal cardiac sonographers) were recruited from a center of fetal cardiology to participate. Each participant was shown 2000 fetal four-chamber images in a random order (1000 normal and 1000 AVSD). The dataset comprised 500 images, each shown in four conditions: (1) image alone without AI output; (2) image with binary AI classification; (3) image with AI model confidence; and (4) image with grad-CAM image overlays. The clinicians were asked to classify each image as normal or AVSD. A total of 20 000 image classifications were recorded from 10 clinicians. The AI model alone achieved an accuracy of 0.798 (95% CI, 0.760-0.832), a sensitivity of 0.868 (95% CI, 0.834-0.902) and a specificity of 0.728 (95% CI, 0.702-0.754), and the clinicians without AI achieved an accuracy of 0.844 (95% CI, 0.834-0.854), a sensitivity of 0.827 (95% CI, 0.795-0.858) and a specificity of 0.861 (95% CI, 0.828-0.895). Showing a binary (normal or AVSD) AI model output resulted in significant improvement in accuracy to 0.865 (P < 0.001). This effect was seen in both experienced and less-experienced participants. Giving incorrect AI advice resulted in a significant deterioration in overall accuracy, from 0.761 to 0.693 (P < 0.001), which was driven by an increase in both Type-I and Type-II errors by the clinicians. This effect was worsened by showing model confidence (accuracy, 0.649; P < 0.001) or grad-CAM (accuracy, 0.644; P < 0.001). AI has the potential to improve performance when used in collaboration with clinicians, even if the model performance does not reach expert level. Giving additional information about model workings such as model confidence and class activation map image overlays did not improve overall performance, and actually worsened performance for images for which the AI model was incorrect. © 2024 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Mapping interictal discharges using intracranial EEG-fMRI to predict postsurgical outcomes.

Various subjective and objective methods have been proposed to identify which interictal epileptiform discharge (IED)-related EEG-fMRI results are more likely to delineate seizure generating tissue in patients with drug-resistant focal epilepsy for the purposes of surgical planning. In this intracranial EEG-fMRI study, we evaluated the utility of these methods to localize clinically relevant regions pre-operatively and compared the extent of resection of these areas to post-operative outcome. Seventy patients admitted for intracranial video-EEG monitoring were recruited for a simultaneous intracranial EEG-fMRI study. For all analyses of blood oxygen level-dependent responses associated with IEDs, an experienced epileptologist identified the most Clinically Relevant brain activation cluster using available clinical information. The Maximum cluster (the cluster with the highest z-score) was also identified for all analyses and assigned to one of three confidence levels (low, medium, or high) based on the difference of the peak z-scores between the Maximum and Second Maximum cluster (the cluster with the second highest peak z-value). The distance was measured and compared between the peak voxel of the aforementioned clusters and the electrode contacts where the interictal discharge and seizure onset were recorded. In patients who subsequently underwent epilepsy surgery, the spatial concordance between the aforementioned clusters and the area of resection was determined and compared to post-operative outcome. We evaluated 106 different IEDs in 70 patients. Both subjective (identification of the Clinically Relevant cluster) and objective (Maximum cluster much more significant than the second maximum cluster) methods of culling non-localizing EEG-fMRI activation maps increased the spatial concordance between these clusters and the corresponding IED or seizure onset zone contacts. However, only the objective methods of identifying medium and high confidence maps resulted in a significant association between resection of the peak voxel of the Maximum cluster and post-operative outcome. Resection of this area was associated with good post-operative outcomes but was not sufficient for seizure freedom. On the other hand, we found that failure to resect the medium and high confidence Maximum clusters was associated with a poor post-surgical outcome (negative predictive value = 1.0, sensitivity = 1.0). Objective methods to identify higher confidence EEG-fMRI results are needed to localize areas necessary for good post-operative outcomes. However, resection of the peak voxel within higher confidence Maximum clusters is not sufficient for good outcomes. Conversely, failure to resect the peak voxel in these clusters is associated with a poor post-surgical outcome.

Where is my attention? An explainable AI exploration in water detection from SAR imagery

Attention mechanisms have found extensive application in Deep Neural Networks (DNNs), with numerous experiments over time showcasing their efficacy in improving the overall performance of DNNs. Considering the black-box nature of DNNs, it is unclear how attention mechanisms affect model decision-making processes. Therefore, it is of great significance to explore the internal relationship between the focused area of attention mechanisms and the prediction results. For the first time, an explainable framework for the attention mechanism of Synthetic Aperture Radar (SAR) image analytics is proposed, integrating DNN and eXplainable Artificial Intelligence (XAI), which realizes the effective explanation for attention mechanisms across different locations. The framework consists of three parts: the water extraction network which contains the attention mechanism at different locations; the proposed Attention- Gradient Class Activation Mapping (A-GCAM) method for attribution analysis of attention mechanisms; and we invent Category-specific channel Score of Confidence Mapping (CSCM) to perform geo-visualization for the output features of attention mechanisms. Experiments are conducted with the widely used Sentinel-1 system for water detection, in which three attention mechanisms with different characteristics are added to the encoder and decoder in DNNs. The results show that the framework can make the decision-making process of attention mechanisms transparent, thus improving their comprehensiveness and trustworthiness in various tasks and providing a reliable approach to selecting suitable attention mechanisms for the given task.

Data-limited and imbalanced bladder wall segmentation with confidence map-guided residual networks via transfer learning

Purpose: To develop an algorithm using a residual base network guided by the confidence map and transfer learning for limited dataset size and imbalanced bladder wall segmentation.Methods: The geometric transformation was made to the training data for data augmentation, and a pre-trained Resnet50 model on ImageNet was also adopted for transfer learning. Three loss functions were put into the pre-trained Resnet50 network, they are the cross-entropy loss function (CELF), the generalized Dice loss function (GDLF) and the Tversky loss function (TLF). Three models were obtained through training, and three corresponding confidence maps were output after entering a new image. By selecting the point with the maximum confidence values at the corresponding position, we merged the three images into one figure, performed threshold filtering to avoid external anomalies, and finally obtained the segmentation result.Results: The average Jaccard similarity coefficient of model training based on the CELF, GDLF and TLF is 0.9173, 0.8355, 0.8757, respectively, and the average Jaccard similarity coefficient of our algorithm can be achieved at 0.9282. In contrast, the classical 2D U-Net algorithm can only achieve 0.518. We also qualitatively give the reasons for the improvement of model performance.Conclusion: Our study demonstrates that a confidence map-assisted residual base network can accurately segment bladder walls on a limited-size data set. Compared with the segmentation results of each model alone, our method originally improves the accuracy of the segmentation results by combining confidence map guidance with threshold filtering.

Confidence maps for reliable estimation of proton density fat fraction and in the liver.

The objective was to develop a fully automated algorithm that generates confidence maps to identify regions valid for analysis of quantitative proton density fat fraction (PDFF) and maps of the liver, generated with chemical shift-encoded MRI (CSE-MRI). Confidence maps are urgently needed for automated quality assurance, particularly with the emergence of automated segmentation and analysis algorithms. Confidence maps for both PDFF and maps are generated based on goodness of fit, measured by normalized RMS error between measured complex signals and the CSE-MRI signal model. Based on Cramér-Rao lower bound and Monte-Carlo simulations, normalized RMS error threshold criteria were developed to identify unreliable regions in quantitative maps. Simulation, phantom, and in vivo clinical studies were included. To analyze the clinical data, a board-certified radiologist delineated regions of interest (ROIs) in each of the nine liver segments for PDFF and analysis in consecutive clinical CSE-MRI data sets. The percent area of ROIs in areas deemed unreliable by confidence maps was calculated to assess the impact of confidence maps on real-world clinical PDFF and measurements. Simulations and phantom studies demonstrated that the proposed algorithm successfully excluded regions with unreliable PDFF and measurements. ROI analysis by the radiologist revealed that 2.6% and 15% of the ROIs were placed in unreliable areas of PDFF and maps, as identified by confidence maps. A proposed confidence map algorithm that identifies reliable areas of PDFF and measurements from CSE-MRI acquisitions was successfully developed. It demonstrated technical and clinical feasibility.

Attentional pixel-wise deformation for pose-based human image generation

Human pose transfer aims to synthesize referred human images with target pose, bringing the substantial economic potential for E-commerce or virtual reality. In this paper, we propose a novel method, the Attentional Pixel-wise Deformation Network (APD-Net), for synthesizing human images with guided pose and referred images. Specifically, we leverage attention-based spatial transformation modules and affine transformation modules to generate accurate appearance and extract pixel-wise details in local regions to generate intermediate results. Additionally, we introduce a confidence map to refine spatial information during the final image synthesis. Domain alignment loss, cycle loss, perceptual and feature matching loss and contextual loss are applied to constrain the synthesized images while attention loss and fusion loss benefit warp images generation. We verify the efficacy of the model on the Market-1501 and DeepFashion datasets, using quantitative and qualitative measures. Our approach surpasses all previously published state-of-the-art results on most evaluation metrics, e.g., achieving 0.780 SSIM score, 9.55 Sliced Wasserstein Distance score, and a 0.963 Semantic Consistency score on DeepFashion and obtaining 0.303 SSIM score, 16.971 Sliced Wasserstein Distance score and 0.729 Semantic Consistency score on Market-1501 Code and pretrained models are available at: https://github.com/LiaoFJ/APD-Net/.

Depth Completion with Multiple Balanced Bases and Confidence for Dense Monocular SLAM.

Dense SLAM based on monocular cameras does indeed have immense application value in the field of AR/VR, especially when it is performed on a mobile device. In this paper, we propose a novel method that integrates a light-weight depth completion network into a sparse SLAM system using a multi-basis depth representation, so that dense mapping can be performed online even on a mobile phone. Specifically, we present a specifically optimized multi-basis depth completion network, called BBC-Net, tailored to the characteristics of traditional sparse SLAM systems. BBC-Net can predict multiple balanced bases and a confidence map from a monocular image with sparse points generated by off-the-shelf keypoint-based SLAM systems. The final depth is a linear combination of predicted depth bases that can be easily optimized by tuning the corresponding weights. To seamlessly incorporate the weights into traditional SLAM optimization and ensure efficiency and robustness, we design a set of depth weight factors, which makes our network a versatile plug-in module, facilitating easy integration into various existing sparse SLAM systems and significantly enhancing global depth consistency through bundle adjustment. To verify the portability of our method, we integrate BBC-Net into two representative SLAM systems. The experimental results on various datasets show that the proposed method achieves better performance in monocular dense mapping than the state-of-the-art methods. We provide an online demo running on a mobile phone, which verifies the efficiency and mapping quality of the proposed method in real-world scenarios.

A Deformation Field-Based Approach for Expression Processing in Head Motion Correction

It is a routine procedure to use Iterative Closest Point (ICP) algorithm to correct motion in medical imaging of the head. However, the basic assumption of ICP is the rigid body motion, and it is not strictly valid when there is non-rigid motion (facial expression). In this study, we propose a novel method to reduce the adverse effects of facial expressions on head motion estimation. First, we design a network named DFG-Net to generate a deformation field on the RGB domain and compute a confidence map. We then extend the original ICP by embedding the confidence map representing each point’s degree of non-rigid motion. Compared to other techniques, our proposed method eliminates the negative effects of facial expression while making the most of pose information. The experiments demonstrate that our approach reduces the position error by 19.23% and orientation error by 36.37% compared to the original ICP algorithm. They also show that our method is robust to variations of expressions.

Preliminary results of automatic cotton crops mapping using remote sensing data

The paper presents the results of application of the method of automatic generation of representative and unbiased set for in-season cotton crop mapping, based on crop simulation model, previously parameterized using ground truth and satellite data. The method provided confident mapping of cotton fields without using actual ground-truth information or apriori information about their in-season phenology. Overall mapping accuracy calculated using relevant ground truth data for cotton fields has reached 95.6 %. Consideration of time series of NDVI values as a model of phase characteristics allowed using relatively simple criteria to identify typical representatives of the selected crop on the basis of analysis of their seasonal phenology and made it possible to build a reference sample for modeling and further classification.