Image Feature Space Research Articles

HVTR++: Image and Pose Driven Human Avatars Using Hybrid Volumetric-Textural Rendering.

Recent neural rendering methods have made great progress in generating photorealistic human avatars. However, these methods are generally conditioned only on low-dimensional driving signals (e.g., body poses), which are insufficient to encode the complete appearance of a clothed human. Hence they fail to generate faithful details. To address this problem, we exploit driving view images (e.g., in telepresence systems) as additional inputs. We propose a novel neural rendering pipeline, Hybrid Volumetric-Textural Rendering (HVTR++), which synthesizes 3D human avatars from arbitrary driving poses and views while staying faithful to appearance details efficiently and at high quality. First, we learn to encode the driving signals of pose and view image on a dense UV manifold of the human body surface and extract UV-aligned features, preserving the structure of a skeleton-based parametric model. To handle complicated motions (e.g., self-occlusions), we then leverage the UV-aligned features to construct a 3D volumetric representation based on a dynamic neural radiance field. While this allows us to represent 3D geometry with changing topology, volumetric rendering is computationally heavy. Hence we employ only a rough volumetric representation using a pose- and image-conditioned downsampled neural radiance field (PID-NeRF), which we can render efficiently at low resolutions. In addition, we learn 2D textural features that are fused with rendered volumetric features in image space. The key advantage of our approach is that we can then convert the fused features into a high-resolution, high-quality avatar by a fast GAN-based textural renderer. We demonstrate that hybrid rendering enables HVTR++ to handle complicated motions, render high-quality avatars under user-controlled poses/shapes, and most importantly, be efficient at inference time. Our experimental results also demonstrate state-of-the-art quantitative results.

SGANnet: Super-resolution guided asymmetric stereo matching network

With asymmetric resolution stereo images as input, existing stereo matching algorithms significantly decline in prediction performance. To address this, we introduce SGANet (Super-resolution Guided Asymmetric Stereo Matching Network), a model that employs unsupervised training methods to overcome the difficulty of acquiring ground truth disparity. For the lower resolution side, this paper designs a stereo guided super-resolution module (SGSR), where the network generates a super-resolved image enriched with details guided by the higher resolution side. Additionally, for this module, we propose a feature consistency loss based on the image's feature space to measure the similarity between the real and super-resolved images. Experimental results on the autonomous driving dataset KITTI demonstrate the effectiveness of the SGSR module and the feature consistency loss in improving the disparity prediction performance of asymmetric resolution stereo images.

SGANnet: Super-resolution guided asymmetric stereo matching network

With asymmetric resolution stereo images as input, existing stereo matching algorithms significantly decline in prediction performance. To address this, we introduce SGANet (Super-resolution Guided Asymmetric Stereo Matching Network), a model that employs unsupervised training methods to overcome the difficulty of acquiring ground truth disparity. For the lower resolution side, this paper designs a stereo guided super-resolution module (SGSR), where the network generates a super-resolved image enriched with details guided by the higher resolution side. Additionally, for this module, we propose a feature consistency loss based on the image's feature space to measure the similarity between the real and super-resolved images. Experimental results on the autonomous driving dataset KITTI demonstrate the effectiveness of the SGSR module and the feature consistency loss in improving the disparity prediction performance of asymmetric resolution stereo images.

REN-GAN: Generative adversarial network-driven rebar clutter elimination network in GPR image for tunnel defect identification

In ground penetrating radar (GPR) images, the signals from reinforcing steel bars (rebars) can significantly obscure the signals of void defects beneath them, which hinders defect detection in tunnel linings. This study introduces a novel, end-to-end unsupervised deep learning-based network that is trained on unpaired GPR images. The primary objective is to suppress the rebar signal and enhance the accuracy of void defect identification in recorded GPR B-Scan images. The network incorporates an autoencoder generator with feature attention and feature mix-up modules, designed to recover textural information and generate improved images devoid of rebar interference. Crucially, this network leverages a strategy of comparing images in feature space, rather than pixel space. The features extracted by the autoencoder are constrained by cycle perceptual consistency loss, derived from a pretrained VGG19 network, to ensure fidelity and perceptual accuracy. Furthermore, this study introduces an enhanced Faster-RCNN network, specifically improved through a more robust feature extraction module, for detecting void defects beneath rebars. The effectiveness of the proposed method has been thoroughly validated using both synthetic and real-world data sets. The results indicate that our method not only surpasses other unsupervised techniques in performance but also effectively diminishes rebar signals and accurately reconstructs and identify void defect signals.

Vision-based adaptive LT sliding mode admittance control for collaborative robots with actuator saturation

Abstract In this paper, we propose a novel vision-based adaptive leakage-type (LT) sliding mode admittance control for actuator-constrained collaborative robots to realize the synchronous control of the precise path following and compliant interaction force. Firstly, we develop a vision-admittance-based model to couple the visual feedback and force sensing in the image feature space so that a reference image feature trajectory can be obtained concerning the contact force command and predefined trajectory. Secondly, considering the system uncertainty, external disturbance, and torque constraints of collaborative robots in reality, we propose an adaptive sliding mode controller in the image feature space to perform precise trajectory tracking. This controller employs a leakage-type (LT) adaptive control law to reduce the side effects of system uncertainties without knowing the upper bound of system uncertainties. Moreover, an auxiliary dynamic is considered in this controller to overcome the joint torque constraints. Finally, we prove the convergence of the tracking error with the Lyapunov stability analysis and operate various semi-physical simulations compared to the conventional adaptive sliding mode and parallel vision/force controller to demonstrate the efficacy of the proposed controller. The simulation results show that compared with the controller mentioned above, the path following accuracy and interaction force control precision of the proposed controller increased by 50% and achieved faster convergence.

Enhancing images for vision measurement in hazy tunnel construction

To address the negative impacts of image degradation on vision measurement in hazy tunnel construction, this paper presents a unified enhancement method of color and grayscale images in tunnel haze environments. Firstly, a tunnel haze environment light scattering model is introduced utilizing geometric optics principles and the conventional atmospheric scattering model. Secondly, a model for perceiving image haze is established within the image feature space to estimate ambient light and transmission. The efficacy of the method is assessed by constructing a tunnel haze simulation chamber and a platform for measurement of tunnel boring machine (TBM) cutter wear. Comparative experiments demonstrate the method reduces the error in 3D point cloud reconstruction by 77% and improves the measurement precision of TBM cutter wear by 62%. The results demonstrate the potential application of this method in automated tunnel construction. However, further research is needed to enhance its efficacy in dense haze images.

RL-SeqISP: Reinforcement Learning-Based Sequential Optimization for Image Signal Processing

Hardware image signal processing (ISP), aiming at converting RAW inputs to RGB images, consists of a series of processing blocks, each with multiple parameters. Traditionally, ISP parameters are manually tuned in isolation by imaging experts according to application-specific quality and performance metrics, which is time-consuming and biased towards human perception due to complex interaction with the output image. Since the relationship between any single parameter’s variation and the output performance metric is a complex, non-linear function, optimizing such a large number of ISP parameters is challenging. To address this challenge, we propose a novel Sequential ISP parameter optimization model, called the RL-SeqISP model, which utilizes deep reinforcement learning to jointly optimize all ISP parameters for a variety of imaging applications. Concretely, inspired by the sequential tuning process of human experts, the proposed model can progressively enhance image quality by seamlessly integrating information from both the image feature space and the parameter space. Furthermore, a dynamic parameter optimization module is introduced to avoid ISP parameters getting stuck into local optima, which is able to more effectively guarantee the optimal parameters resulting from the sequential learning strategy. These merits of the RL-SeqISP model as well as its high efficiency are substantiated by comprehensive experiments on a wide range of downstream tasks, including two visual analysis tasks (instance segmentation and object detection), and image quality assessment (IQA), as compared with representative methods both quantitatively and qualitatively. In particular, even using only 10% of the training data, our model outperforms other SOTA methods by an average of 7% mAP on two visual analysis tasks.

Study of the influence of synaptic plasticity on the formation of a feature space by a spiking neural network

The purpose of this study is to study the influence of synaptic plasticity on excitatory and inhibitory synapses on the formation of the feature space of the input image on the excitatory and inhibitory layers of neurons in a spiking neural network. Methods. To simulate the dynamics of the neuron, the computationally efficient model “Leaky integrate-and-fire” was used. The conductance-based synapse model was used as a synaptic contact model. Synaptic plasticity in excitatory and inhibitory synapses was modeled by the classical model of time dependent synaptic plasticity. A neural network composed of them generates a feature space, which is divided into classes by a machine learning algorithm. Results. A model of a spiking neural network was built with excitatory and inhibitory layers of neurons with adaptation of synaptic contacts due to synaptic plasticity. Various configurations of the model with synaptic plasticity were considered for the problem of forming the feature space of the input image on the excitatory and inhibitory layers of neurons, and their comparison was also carried out. Conclusion. It has been shown that synaptic plasticity in inhibitory synapses impairs the formation of an image feature space for a classification task. The model constraints are also obtained and the best model configuration is selected.

Alignment of multimodal rigid cardiac angiography images with an improved particle swarm algorithm

Abstract In this paper, we conduct a preliminary study on the current development status in medical image alignment and build up a basic framework for image alignment. The feature space, search space, similarity measure, and search strategy of cardiac angiography images are calculated and studied. The DGVF model is utilized to process the traditional snake model for optimization search and is combined with B-splines to construct the B-spline DGVF model. Optimize the traditional MsFCM algorithm by using the PSO algorithm and propose an MsFCM-PSO image segmentation method. It is applied together with the B-spline DGVF model to segment the vascular lumen in cardiac angiography ultrasound images. Finally, the model of this paper is analyzed in terms of segmentation performance, alignment stability, and evaluation of alignment results. The mean values of Dice, IoU, and HD of this paper’s MsFCM-PSO model in image segmentation of cardiac vessels are 94.27%, 92.60%, and 1.06, respectively (all optimal performances). In the ablation experiments, the MsFCMPSO model in this paper shows an increase of 6.02% and 5.47% in the mean values of Dice and IoU compared to the benchmark model. The stability calibration percentage of this paper’s MsFCM-PSO algorithm is 31.13% when the Gaussian factor is 0.5, which is significantly better than other algorithms. The algorithm in this paper is better than other methods in terms of alignment stability and alignment results.

Using Dimensionality Reduction to Visualize Phenotypic Changes in High-Throughput Microscopy.

High-throughput microscopy has enabled screening of cell phenotypes at unprecedented scale. Systematic identification of cell phenotype changes (such as cell morphology and protein localization changes) is a major analysis goal. Because cell phenotypes are high-dimensional, unbiased approaches to detect and visualize the changes in phenotypes are still needed. Here, we suggest that changes in cellular phenotype can be visualized in reduced dimensionality representations of the image feature space. We describe a freely available analysis pipeline to visualize changes in protein localization in feature spaces obtained from deep learning. As an example, we use the pipeline to identify changes in subcellular localization after the yeast GFP collection was treated with hydroxyurea.

Deep-Interior: A new pathway to interior tomographic image reconstruction via a weighted backprojection and deep learning.

In recent years, deep learning strategies have been combined with either the filtered backprojection or iterative methods or the direct projection-to-image by deep learning only to reconstruct images. Some of these methods can be applied to address the interior reconstruction problems for centered regions of interest (ROIs) with fixed sizes. Developing a method to enable interior tomography with arbitrarily located ROIs with nearly arbitrary ROI sizes inside a scanning field of view (FOV) remains an openquestion. To develop a new pathway to enable interior tomographic reconstruction for arbitrarily located ROIs with arbitrary sizes using a single trained deep neural network model. The method consists of two steps. First, an analytical weighted backprojection reconstruction algorithm was developed to perform domain transform from divergent fan-beam projection data to an intermediate image feature space, , for an arbitrary size ROI at an arbitrary location inside the FOV. Second, a supervised learning technique was developed to train a deep neural network architecture to perform deconvolution to obtain the true image from the new feature space . This two-step method is referred to as Deep-Interior for convenience. Both numerical simulations and experimental studies were performed to validate the proposed Deep-Interior method. The results showed that ROIs as small as a diameter of 5 cm could be accurately reconstructed (similarity index 0.985 ± 0.018 on internal testing data and 0.940 ± 0.025 on external testing data) at arbitrary locations within an imaging object covering a wide variety of anatomical structures of different body parts. Besides, ROIs of arbitrary size can be reconstructed by stitching small ROIs without additionaltraining. The developed Deep-Interior framework can enable interior tomographic reconstruction from divergent fan-beam projections for short-scan and super-short-scan acquisitions for small ROIs (with a diameter larger than 5 cm) at an arbitrary location inside the scanning FOV with high quantitative reconstruction accuracy.

Vision-admittance-based adaptive RBFNN control with a SMC robust compensator for collaborative parallel robots

This paper proposes a novel vision-admittance-based adaptive neural network control method for collaborative parallel robots tracking a predefined trajectory and force command. First of all, the vision-admittance model is creatively developed for coupling visual information and force-sensing information in the image feature space and is applied to generate the image feature reference trajectory online according to the desired trajectory and the command of the interaction force. Secondly, considering the system modelling uncertainties and external disturbances, an adaptive Radial basis functions neural network (RBFNN) controller is created to realize high-precision trajectory tracking. The adaptive tuning laws are designed based on the Lyapunov stability theorem so that the entire system’s stability and the convergence of the weight adaptation can be guaranteed. Moreover, a sliding mode control (SMC)-based robust compensator is added to the RBFNN controller as an auxiliary term to improve the robustness and stability of the control system. Finally, co-simulations between ADAMS/Simulink are performed on a collaborative Five-bar parallel robot to demonstrate the efficacy of the proposed controller.

Investigating comparisons on the coal and gangue in various scenarios using multidimensional image features

Online intelligent recognition of coal and gangue is an important aspect of coal mine intelligent development. Current research focuses more on improving recognition accuracy through optimizing image processing algorithms, but there is limited research on the characteristics of coal gangue materials themselves, particularly the comparative analysis of their appearance in images under different scenarios. Therefore, this paper proposes a multi-dimensional feature-based analysis method for coal-gangue images, which can be used to evaluate the characteristics and distinguishability of coal and gangue images. Firstly, the original feature space of coal and gangue images is constructed based on 55 global image feature operators. Then, the characteristics of coal and gangue images in both the original space and the two-dimensional space after dimensionality reduction are analyzed using the similarity between the left and right sides of the split violin plot and the distance of probability distributions. At the same time, different types of features are compared and analyzed in terms of their impact on classification performance when used as inputs with various machine learning classifiers. Furthermore, by leveraging the interpretability of tree models, the contribution of different feature types to coal gangue classification tasks is analyzed. The proposed method is applied to analyze a large number of raw coal images collected from three different conditions. The experimental results demonstrate that the distinguishability of coal gangue images varies under different scenarios, and the distance between the two distributions in the reduced space effectively measures this distinguishability. In coal gangue images, the contribution of color features and texture features to the classification task is related to the type of coal, with texture features playing a more significant role in classifying coking coal compared to thermal coal. However, overall, color features dominate the classification task. These findings are further validated in experiments using machine learning classifiers, which are crucial for designing more refined classification algorithms.

Interpretable thoracic pathologic prediction via learning group-disentangled representation.

Deep learning has brought a significant progress in medical image analysis. However, their lack of interpretability might bring high risk for wrong diagnosis with limited clinical knowledge embedding. In other words, we believe it's crucial for humans to interpret how deep learning work for medical analysis, thus appropriately adding knowledge constraints to correct the bias of wrong results. With such purpose, we propose Representation Group-Disentangling Network (RGD-Net) to explain the process of feature extraction and decision making inside deep learning framework, where we completely disentangle feature space of input X-ray images into independent feature groups, and each group would contribute to diagnose of a specific disease. Specifically, we first state problem definition for interpretable prediction with auto-encoder structure. Then, group-disentangled representations are extracted from input X-ray images with the proposed Group-Disentangle Module, which constructs semantic latent space by enforcing semantic consistency of attributes. Afterwards, adversarial constricts on mapping from features to diseases are proposed to prevent model collapse during training. Finally, a novel design of local tuning medical application is proposed based on RGB-Net, which is capable to aid clinicians for reasonable diagnosis. By conducting quantity of experiments on public datasets, RGD-Net have been superior to comparative studies by leveraging potential factors contributing to different diseases. We believe our work could bring interpretability in digging inherent patterns of deep learning on medical image analysis.

Visual-admittance-based model predictive control for nuclear collaborative robots

This paper presents a novel visual-admittance-based model predictive control scheme to cope with the problem of vision/force control and several constraints of a nuclear collaborative robotic visual servoing system. A visual-admittance model considering the desired image feature and force command in the image feature space is proposed. Moreover, a novel constraint scheme of the model predictive control (MPC) is proposed to cancel the overshoot in the interaction force control for most cases by taking the desired force command as the constraint of the proposed MPC. Via applying the robotic dynamic image-based visual servoing (IBVS) model, some other constraints, such as the actuator saturation, joint angle, and visual limits, can be satisfied simultaneously. The simulation results for the two-degrees-of-freedom (DOF) robot manipulator with an eye-to-hand camera are present to demonstrate the effectiveness of the proposed controller.

Semi-supervised aortic dissections segmentation: A time-dependent weighted feedback fusion framework.

The segmentation of true lumen (TL) and false lumen (FL) plays an important role in the diagnosis and treatment of aortic dissection (AD). Although the deep learning methods have achieved remarkable performance for this task, a large number of labeled data are required for training. In order to alleviate the burden of doctors' labeling, in this paper, a novel time-dependent weighted feedback fusion based semi-supervised aortic dissections segmentation framework is proposed by effectively leveraging the unlabeled data. A feedback network is additionally extended to encode the predicted output from the backbone network into high-level feature space, which is then fused with the original feature information of the image to fix previous potential mistakes, thereby segmentation accuracy is improved iteratively. To utilize both labeled data and unlabeled data, the fused feature space flows into the network again to generate the second feedback and make sure consistency with the previous one. The utilization of image feature space provides better robustness and accuracy for the proposed structure. Experiments show that our method outperforms five existing state-of-the-art semi-supervised segmentation methods on both a type-B AD dataset and a public dataset.

Compression-resistant backdoor attack against deep neural networks

In recent years, a number of backdoor attacks against deep neural networks (DNN) have been proposed. In this paper, we reveal that backdoor attacks are vulnerable to image compressions, as backdoor instances used to trigger backdoor attacks are usually compressed by image compression methods during data transmission. When backdoor instances are compressed, the feature of backdoor trigger will be destroyed, which could result in significant performance degradation for backdoor attacks. As a countermeasure, we propose the first compression-resistant backdoor attack method based on feature consistency training. Specifically, both backdoor images and their compressed versions are used for training, and the feature difference between backdoor images and their compressed versions are minimized through feature consistency training. As a result, the DNN treats the feature of compressed images as the feature of backdoor images in feature space. After training, the backdoor attack will be robust to image compressions. Furthermore, we consider three different image compressions (i.e., JPEG, JPEG2000, WEBP) during the feature consistency training, so that the backdoor attack can be robust to multiple image compression algorithms. Experimental results demonstrate that when the backdoor instances are compressed, the attack success rate of common backdoor attack is 6.63% (JPEG), 6.20% (JPEG2000) and 3.97% (WEBP) respectively, while the attack success rate of the proposed compression-resistant backdoor attack is 98.77% (JPEG), 97.69% (JPEG2000), and 98.93% (WEBP) respectively. The compression-resistant attack is robust under various parameters settings. In addition, extensive experiments have demonstrated that even if only one image compression method is used in the feature consistency training process, the proposed compression-resistant backdoor attack has the generalization ability to resist multiple unseen image compression methods.

TSCNet: Topological Structure Coupling Network for Change Detection of Heterogeneous Remote Sensing Images

With the development of deep learning, convolutional neural networks (CNNs) have been successfully applied in the field of change detection in heterogeneous remote sensing (RS) images and achieved remarkable results. However, most of the existing methods of heterogeneous RS image change detection only extract deep features to realize the whole image transformation and ignore the description of the topological structure composed of the image texture, edge, and direction information. The occurrence of change often means that the topological structure of the ground object has changed. As a result, these algorithms severely limit the performance of change detection. To solve these problems, this paper proposes a new topology-coupling-based heterogeneous RS image change detection network (TSCNet). TSCNet transforms the feature space of heterogeneous images using an encoder–decoder structure and introduces wavelet transform, channel, and spatial attention mechanisms. The wavelet transform can obtain the details of each direction of the image and effectively capture the image’s texture features. Unnecessary features are suppressed by allocating more weight to areas of interest via channels and spatial attention mechanisms. As a result of the organic combination of a wavelet, channel attention mechanism, and spatial attention mechanism, the network can focus on the texture information of interest while suppressing the difference of images from different domains. On this basis, a bitemporal heterogeneous RS image change detection method based on the TSCNet framework is proposed. The experimental results on three public heterogeneous RS image change detection datasets demonstrate that the proposed change detection framework achieves significant improvements over the state-of-the-art methods.

Total-body low-dose CT image denoising using a prior knowledge transfer technique with a contrastive regularization mechanism.

Reducing the radiation exposure experienced by patients in total-body computed tomography (CT) imaging has attracted extensive attention in the medical imaging community. A low radiation dose may result in increased noise and artifacts that greatly affect the subsequent clinical diagnosis. To obtain high-quality total-body low-dose CT (LDCT) images, previous deep learning-based research works developed various network architectures. However, most of these methods only employ normal-dose CT (NDCT) images as ground truths to guide the training process of the constructed denoising network. As a result of this simple restriction, the reconstructed images tend to lose favorable image details and easily generate oversmoothed textures. This study explores how to better utilize the information contained in the feature spaces of NDCT images to guide the LDCT image reconstruction process and achieve high-qualityresults. We propose a novel intratask knowledge transfer (KT) method that leverages the knowledge distilled from NDCT images as an auxiliary component of the LDCT image reconstruction process. Our proposed architecture is named the teacher-student consistency network (TSC-Net), which consists of teacher and student networks with identical architectures. By employing the designed KT loss, the student network is encouraged to emulate the teacher network in the representation space and gain robust prior content. In addition, to further exploit the information contained in CT scans, a contrastive regularization mechanism (CRM) built upon contrastive learning is introduced. The CRM aims to minimize and maximize the L2 distances from the predicted CT images to the NDCT samples and to the LDCT samples in the latent space, respectively. Moreover, based on attention and the deformable convolution approach, we design a dynamic enhancement module (DEM) to improve the network capability to transform input informationflows. By conducting ablation studies, we prove the effectiveness of the proposed KT loss, CRM, and DEM. Extensive experimental results demonstrate that the TSC-Net outperforms the state-of-the-art methods in both quantitative and qualitative evaluations. Additionally, the excellent results obtained for clinical readings also prove that our proposed method can reconstruct high-quality CT images for clinicalapplications. Based on the experimental results and clinical readings, the TSC-Net has better performance than other approaches. In our future work, we may explore the reconstruction of LDCT images by fusing the positron emission tomography (PET) and CT modalities to further improve the visual quality of the reconstructed CTimages.

A Multi-modal CBIR Framework with Image Segregation using Autoencoders and Deep Learning-based Pseudo-labeling

Various modulated techniques of Content-Based Image Retrieval (CBIR) using deep learning provide better search outputs even though they are computationally challenging. These methods can be enhanced further, if the search key can be tagged effectively and directed towards target images. In this paper, we have developed a new multilevel aggregation technique along with autoencoders, to be implemented on image features for precise feature selection and accurate search output. Locally significant datasets and generic ones are dealt separately for obtaining better hit rates along with the process of query expansion. The concept of pseudo-labelling by deep learning is introduced to classify images into positive and negative classes, where positive class consists of the images that are similar to the query image. The feature spaces of query images are compared with the images in the search pool based on their assigned weights. The target images are finally ranked and selected based on an adaptive threshold level. The methodology for this enhanced CBIR technique is tested on public datasets and its results are collated with recent approaches proposed in the literature. The proposed method has attained significant improvements in precision, recall and computation and thereby the use of images from connected devices has been proven to be significant. Given the importance of more accurate image retrieval in CBIR, this method can be experimented for other similar applications as well.