Unsupervised Learning Network Research Articles

A Developmental Approach for Training Deep Belief Networks

Deep belief networks (DBNs) are stochastic neural networks that can extract rich internal representations of the environment from the sensory data. DBNs had a catalytic effect in triggering the deep learning revolution, demonstrating for the very first time the feasibility of unsupervised learning in networks with many layers of hidden neurons. These hierarchical architectures incorporate plausible biological and cognitive properties, making them particularly appealing as computational models of human perception and cognition. However, learning in DBNs is usually carried out in a greedy, layer-wise fashion, which does not allow to simulate the holistic maturation of cortical circuits and prevents from modeling cognitive development. Here we present iDBN, an iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the model. We evaluate the proposed iterative algorithm on two different sets of visual stimuli, measuring the generative capabilities of the learned model and its potential to support supervised downstream tasks. We also track network development in terms of graph theoretical properties and investigate the potential extension of iDBN to continual learning scenarios. DBNs trained using our iterative approach achieve a final performance comparable to that of the greedy counterparts, at the same time allowing to accurately analyze the gradual development of internal representations in the deep network and the progressive improvement in task performance. Our work paves the way to the use of iDBN for modeling neurocognitive development.

Joint Deep Reinforcement Learning and Unsupervised Learning for Channel Selection and Power Control in D2D Networks.

Device-to-device (D2D) technology enables direct communication between devices, which can effectively solve the problem of insufficient spectrum resources in 5G communication technology. Since the channels are shared among multiple D2D user pairs, it may lead to serious interference between D2D user pairs. In order to reduce interference, effectively increase network capacity, and improve wireless spectrum utilization, this paper proposed a distributed resource allocation algorithm with the joint of a deep Q network (DQN) and an unsupervised learning network. Firstly, a DQN algorithm was constructed to solve the channel allocation in the dynamic and unknown environment in a distributed manner. Then, a deep power control neural network with the unsupervised learning strategy was constructed to output an optimized channel power control scheme to maximize the spectrum transmit sum-rate through the corresponding constraint processing. As opposed to traditional centralized approaches that require the collection of instantaneous global network information, the algorithm proposed in this paper used each transmitter as a learning agent to make channel selection and power control through a small amount of state information collected locally. The simulation results showed that the proposed algorithm was more effective in increasing the convergence speed and maximizing the transmit sum-rate than other traditional centralized and distributed algorithms.

A novel single robot image shadow detection method based on convolutional block attention module and unsupervised learning network.

Shadow detection plays a very important role in image processing. Although many algorithms have been proposed in different environments, it is still a challenging task to detect shadows in natural scenes. In this paper, we propose a convolutional block attention module (CBAM) and unsupervised domain adaptation adversarial learning network for single image shadow detection. The new method mainly contains three steps. Firstly, in order to reduce the data deviation between the domains, the hierarchical domain adaptation strategy is adopted to calibrate the feature distribution from low level to high level between the source domain and the target domain. Secondly, in order to enhance the soft shadow detection ability of the model, the boundary adversarial branch is proposed to obtain structured shadow boundary. Meanwhile, a CBAM is added in the model to reduce the correlation between different semantic information. Thirdly, the entropy adversarial branch is combined to further suppress the high uncertainty at the boundary of the prediction results, and it obtains the smooth and accurate shadow boundary. Finally, we conduct abundant experiments on public datasets, the RMSE has the lowest values with 9.6 and BER with 6.6 on ISTD dataset, the results show that the proposed shadow detection method has better edge structure compared with the existing deep learning detection methods.

Attraction and Repulsion: Unsupervised Domain Adaptive Graph Contrastive Learning Network

Graph convolutional networks (GCNs) are important techniques for analytics tasks related to graph data. To date, most GCNs are designed for a single graph domain. They are incapable of transferring knowledge from/to different domains (graphs), due to the limitation in graph representation learning and domain adaptation across graph domains. This paper proposes a novel Graph Contrastive Learning Network (GCLN) for unsupervised domain adaptive graph learning. The key innovation is to enforce attraction and repulsion forces within each single graph domain, and across two graph domains. Within each graph, an attraction force encourages local patch node features to be similar to global representation of the entire graph, whereas a repulsion force will repel node features so they can separate network from its permutations ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> domain-specific graph contrastive learning). Across two graph domains, an attraction force encourages node features from two domains to be largely consistent, whereas a repulsion force ensures features are discriminative to differentiate graph domains ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> cross-domain graph contrastive learning). The within- and cross-domain graph contrastive learning is carried out by optimizing an objective function, which combines source classifier and target classifier loss, domain-specific contrastive loss, and cross-domain contrastive loss. As a result, feature learning from graphs is facilitated using knowledge transferred between graphs. Experiments on real-world datasets demonstrate that GCLN outperforms state-of-the-art graph neural network algorithms.

Enhanced Local and Global Learning for Rotation-Invariant Point Cloud Representation

Various studies have been undertaken to learn point cloud representations that are both discriminative and robust. However, most of them suffer from rotation disturbance and insufficient labeled data. To solve the problem of rotation disturbance, we propose a novel rotation-invariant network called ELGANet that is equipped with the following two core modules: enhanced local representation learning module and global alignment module. The enhanced local representation learning module captures the geometric relationship among the neighbors defined in both 3-D Cartesian space and a latent space to exploit the local context and long-distance context. The global alignment module is devised to address the lack of global information and supplement the absolute locations of points by adaptively generating the rotation-invariant coordinates. For the issue of label dependence, we further propose an unsupervised learning network ELGANet-U that can still generate a discriminative and rotation-invariant representation without human supervision. Extensive experiments on both ModelNet and ScanObjectNN have demonstrated that our ELGANet is superior to other state-of-the-art methods on the premise of ensuring rotation invariance. Furthermore, the representation generated by our ELGANet-U also achieves a comparable performance to that of supervised learning.

Dual attention network for unsupervised medical image registration based on VoxelMorph

An accurate medical image registration is crucial in a variety of neuroscience and clinical studies. In this paper, we proposed a new unsupervised learning network, DAVoxelMorph to improve the accuracy of 3D deformable medical image registration. Based on the VoxelMorph model, our network presented two modifications, one is adding a dual attention architecture, specifically, we model semantic correlation on spatial and coordinate dimensions respectively, and the location attention module selectively aggregates the features of each location by weighting the features of all locations. The coordinate attention module further puts the location information into the channel attention. The other is introducing the bending penalty as regularization in the loss function to penalize the bending in the deformation field. Experimental results show that DAVoxelMorph achieved better registration performance including average Dice scores (0.714) and percentage of locations with non-positive Jacobian (0.345) compare with VoxelMorph (0.703, 0.355), CycleMorph (0.705, 0.133), ANTs SyN (0.707, 0.137) and NiftyReg (0.694, 0.549). Our model increases both model sensitivity and registration accuracy.

Identification of Abnormal Vibration Signal of Subway Track Bed Based on Ultra-Weak FBG Sensing Array Combined with Unsupervised Learning Network

The performance of the passing train and the structural state of the track bed are the concerns regarding the safe operation of subways. Monitoring the vibration response of the track bed structure and identifying abnormal signals within it will help address both of these concerns. Given that it is difficult to collect abnormal samples that are symmetric to those of the normal state of the structure in actual engineering, this paper proposes an unsupervised learning-based methodology for identifying the abnormal signals of the track beds detected by the ultra-weak fiber optic Bragg grating sensing array. For an actual subway tunnel monitoring system, an unsupervised learning network was trained by using a sufficient amount of vibration signals of the track bed collected when trains passed under normal conditions, which was used to quantify the deviations caused by anomalies. An experiment to validate the proposed procedures was designed and implemented according to the obtained normal and abnormal samples. The abnormal vibration samples of the track beds in the experiment came from two parts and were defined as three levels. One part of it stemmed from the vibration responses under the worn wheels of a train detected during system operation. The remaining abnormal samples were simulated by superimposing perturbations in the normal samples. The experimental results demonstrated that the established unsupervised learning network and the selected metric for quantifying error sequences can serve the threshold selection well based on the receiver operating characteristic curve. Moreover, the discussion results of the comparative tests also illustrated that the average results of accuracy and F1-score of the proposed network were at least 11% and 13% higher than those of the comparison networks, respectively.

UMA-Net: an unsupervised representation learning network for 3D point cloud classification.

The success of deep neural networks usually relies on massive amounts of manually labeled data, which is both expensive and difficult to obtain in many real-world datasets. In this paper, a novel unsupervised representation learning network, UMA-Net, is proposed for the downstream 3D object classification. First, the multi-scale shell-based encoder is proposed, which is able to extract the local features from different scales in a simple yet effective manner. Second, an improved angular loss is presented to get a good metric for measuring the similarity between local features and global representations. Subsequently, the self-reconstruction loss is introduced to ensure the global representations do not deviate from the input data. Additionally, the output point clouds are generated by the proposed cross-dim-based decoder. Finally, a linear classifier is trained using the global representations obtained from the pre-trained model. Furthermore, the performance of this model is evaluated on ModelNet40 and applied to the real-world 3D Terracotta Warriors fragments dataset. Experimental results demonstrate that our model achieves comparable performance and narrows the gap between unsupervised and supervised learning approaches in downstream object classification tasks. Moreover, it is the first attempt to apply the unsupervised representation learning for 3D Terracotta Warriors fragments. We hope this success can provide a new avenue for the virtual protection of cultural relics.

AutoAtlas: Neural Network for 3D Unsupervised Partitioning and Representation Learning.

We present a novel neural network architecture called AutoAtlas for fully unsupervised partitioning and representation learning of 3D brain Magnetic Resonance Imaging (MRI) volumes. AutoAtlas consists of two neural network components: one neural network to perform multi-label partitioning based on local texture in the volume, and a second neural network to compress the information contained within each partition. We train both of these components simultaneously by optimizing a loss function that is designed to promote accurate reconstruction of each partition, while encouraging spatially smooth and contiguous partitioning, and discouraging relatively small partitions. We show that the partitions adapt to the subject specific structural variations of brain tissue while consistently appearing at similar spatial locations across subjects. AutoAtlas also produces very low dimensional features that represent local texture of each partition. We demonstrate prediction of metadata associated with each subject using the derived feature representations and compare the results to prediction using features derived from FreeSurfer anatomical parcellation. Since our features are intrinsically linked to distinct partitions, we can then map values of interest, such as partition-specific feature importance scores onto the brain for visualization.

A novel method based on deep transfer unsupervised learning network for bearing fault diagnosis under variable working condition of unequal quantity

Recently, intelligent fault recognition means have been progressed rapidly and have attained marvelous achievement. Most of them have an assumption where those source and target domains have similar distributions. However, actual working conditions of bearing are variable, which makes the source domain and target domain data present large distribution discrepancy. In this paper, a new method named dilated convolution deep belief network-dynamic multilayer perceptron (DCDBN-DMLP) is proposed for bearing fault recognition under alterable running states. Firstly, dilated convolution deep belief network (DCDBN) is proposed to extract transferable characteristics from raw vibrational dataset of bearings under variable running conditions. The divergence between source domains and target domains is large because the dataset generally is gathered from multi-condition environments. Then, the multilayer domain adaptation and pseudo label technology are adopted to alleviate the cross and unequal quantity domains. Finally, dynamic multilayer perceptron (DMLP) is proposed to classify bearing faults, which is connected to DCDBN in a progressive manner. The performance of this proposed DCDBN-DMLP model is validated by three transfer tasks with bearing fault dataset. Experimental results present that the abundant transferable features can be learned by this proposed method, and its classification accuracies significantly outperform other methods by comparison.

Optimal text-to-image synthesis model for generating portrait images using generative adversarial network techniques

The advancements in artificial intelligence research, particularly in computer vision, have led to the development of previously unimaginable applications, such as generating new contents based on text description. In our work we focused on the text-to-image synthesis applications (TIS) field, to transform descriptive sentences into a real image. To tackle this issue, we use unsupervised deep learning networks that can generate high quality images from text descriptions, provided by eyewitnesses to assist law enforcement in their investigations, for the purpose of generating probable human faces. We analyzed a number of existing approaches and chose the best one. Deep fusion generative adversarial networks (DF-GAN) is the network that performs better than its peers, at multiple levels, like the generated image quality or the respect of the giving descriptive text. Our model is trained on the CelebA dataset and text descriptions (generated by our algorithm using existing attributes in the dataset). The obtained results from our implementation show that the learned generative model makes excellent quantitative and visual performances, the model is capable of generating realistic and diverse samples for human faces and create a complete portrait with respect of given text description.

Structure Guided Deep Neural Network for Unsupervised Active Learning.

Unsupervised active learning has become an active research topic in the machine learning and computer vision communities, whose goal is to choose a subset of representative samples to be labeled in an unsupervised setting. Most of existing approaches rely on shallow linear models by assuming that each sample can be well approximated by the span (i.e., the set of all linear combinations) of the selected samples, and then take these selected samples as the representative ones for manual labeling. However, the data do not necessarily conform to the linear models in many real-world scenarios, and how to model nonlinearity of data often becomes the key point of unsupervised active learning. Moreover, the existing works often aim to well reconstruct the whole dataset, while ignore the important cluster structure, especially for imbalanced data. In this paper, we present a novel deep unsupervised active learning framework. The proposed method can explicitly learn a nonlinear embedding to map each input into a latent space via a deep neural network, and introduce a selection block to select the representative samples in the learnt latent space through a self-supervised learning strategy. In the selection block, we aim to not only preserve the global structure of the data, but also capture the cluster structure of the data in order to well handle the data imbalance issue during sample selection. Meanwhile, we take advantage of the clustering result to provide self-supervised information to guide the above processes. Finally, we attempt to preserve the local structure of the data, such that the data embedding becomes more precise and the model performance can be further improved. Extensive experimental results on several publicly available datasets clearly demonstrate the effectiveness of our method, compared with the state-of-the-arts.

Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning.

Human action representation is derived from the description of human shape and motion. The traditional unsupervised 3-dimensional (3D) human action representation learning method uses a recurrent neural network (RNN)-based autoencoder to reconstruct the input pose sequence and then takes the midlevel feature of the autoencoder as representation. Although RNN can implicitly learn a certain amount of motion information, the extracted representation mainly describes the human shape and is insufficient to describe motion information. Therefore, we first present a handcrafted motion feature called pose flow to guide the reconstruction of the autoencoder, whose midlevel feature is expected to describe motion information. The performance is limited as we observe that actions can be distinctive in either motion direction or motion norm. For example, we can distinguish "sitting down" and "standing up" from motion direction yet distinguish "running" and "jogging" from motion norm. In these cases, it is difficult to learn distinctive features from pose flow where direction and norm are mixed. To this end, we present an explicit pose decoupled flow network (PDF-E) to learn from direction and norm in a multi-task learning framework, where 1 encoder is used to generate representation and 2 decoders are used to generating direction and norm, respectively. Further, we use reconstructing the input pose sequence as an additional constraint and present a generalized PDF network (PDF-G) to learn both motion and shape information, which achieves state-of-the-art performances on large-scale and challenging 3D action recognition datasets including the NTU RGB+D 60 dataset and NTU RGB+D 120 dataset.

MetaPan: Unsupervised Adaptation With Meta-Learning for Multispectral Pansharpening

Multispectral (MS) pansharpening aims to improve the spatial resolution of MS images (MSI) using the spatial details of panchromatic (PAN) images. Due to the gap of prior knowledge between the simulated data and real-world cases, unsupervised learning-based approaches have grown increasing interest. However, some key hyper-parameters, such as the initial weights of the networks, are set manually, which significantly impacts the fusion performance. To tackle this problem, we propose a novel unsupervised adaptation method with meta-learning for MS pansharpening (MetaPan), in which the meta-learning aims to automatically learn the initial parameters of a three-stream fusion network (TSFNet) for unsupervised adaptation learning (UAL). Specifically, the TSFNet consists of a PAN stream, an MS stream, and a fusion stream, where the fusion stream implicitly leverages domain-specific knowledge of input image pairs while the other two streams explicitly inject spatial details and spectral information into the fusion stream. The MetaPan consists of a pre-training stage, a meta-learning stage, and a UAL stage. At the pre-training stage, the TSFNet is trained with the supervision of simulated ground truth such that it is universal for all image pairs. Then, the process of meta-learning optimizes for an internal representation of network parameters that can adapt to a specific image pair with UAL through only a few steps. Finally, the learned internal representation is fine-tuned to a real-world image pair (a test image pair) with UAL. Experiments on two datasets show that our method performs better than state-of-the-art methods in both quantitative metrics and visual appearance.

A Class-Level Matching Unsupervised Transfer Learning Network for Rolling Bearing Fault Diagnosis Under Various Working Conditions

A Class-Level Matching Unsupervised Transfer Learning Network for Rolling Bearing Fault Diagnosis Under Various Working Conditions

Unsupervised Deep Learning for Single-Channel Earthquake Data Denoising and Its Applications in Event Detection and Fully Automatic Location

We propose to use unsupervised deep learning (DL) and attention networks to mute the unwanted components of the single-channel earthquake data. The proposed algorithm is an unsupervised technique that does not require any prior information about the input data, i.e., no need for the labeled data. The imaginary and real parts of the short-time frequency transform (STFT) are divided into several overlapped patches to be the input of the proposed DL network, while the output target is the absolute value of the STFT. The proposed DL network utilizes a customized loss function to reconstruct the signal mask, where the STFT components related to the seismic noise are muted. An adaptive thresholding technique is utilized to obtain the binary mask, which is multiplied by the real and imaginary parts of the input seismic data. The binary mask has zero values for the samples corresponding to the unwanted components and ones for the seismic signal components. Then, inverse STFT is used to reconstruct the denoised signal. The proposed algorithm is evaluated using samples from the STanford EArthquake Dataset (STEAD) and the results are compared to the benchmark denoising method, i.e., DeepDenoiser. As a result, the proposed algorithm shows a robust denoising performance and outperforms the DeepDenoiser method by 1.95 dB in terms of signal-to-noise ratio.

Unsupervised 3-D Random Noise Attenuation Using Deep Skip Autoencoder

Effective random noise attenuation is critical for subsequent processing of seismic data, such as velocity analysis, migration, and inversion. Thus, the removal of seismic random noise with an uncertainty level is meaningful. Attenuating 3-D random noise in a supervised way based on deep learning (DL) is challenging because clean labels are difficult to obtain. Therefore, it is necessary to develop an adaptive unsupervised-based method for random noise attenuation. In this article, we propose a deep-denoising unsupervised learning (DDUL) network to attenuate random noise in 2-D/3-D seismic data. A patching technique is used to split 2-D/3-D seismic data into several patches to be fed into the network, which helps to expand the number of samples for training. We use the fully symmetrical structure of the autoencoder to construct the network. In each corresponding encoder and decoder layer, skip connections are added to enhance the learning of seismic data features. We construct three blocks to extract waveform features in seismic data, i.e., encoder, decoder, and skip blocks. Among them, the skip is connected between the encoder and decoder blocks of each hidden layer. The use of multiple blocks not only improves the network&#x2019;s ability to extract seismic data features but also solves the problem of excessive training parameters caused by hidden layer stacking. Five 2-D/3-D synthetic and field seismic datasets are used to test the denoising performance of our proposed method. The denoising results demonstrate that our proposed method has good signal-preserving and noise attenuation capabilities in real-world applications.

Drug-likeness scoring based on unsupervised learning.

Drug-likeness prediction is important for the virtual screening of drug candidates. It is challenging because the drug-likeness is presumably associated with the whole set of necessary properties to pass through clinical trials, and thus no definite data for regression is available. Recently, binary classification models based on graph neural networks have been proposed but with strong dependency of their performances on the choice of the negative set for training. Here we propose a novel unsupervised learning model that requires only known drugs for training. We adopted a language model based on a recurrent neural network for unsupervised learning. It showed relatively consistent performance across different datasets, unlike such classification models. In addition, the unsupervised learning model provides drug-likeness scores that well separate distributions with increasing mean values in the order of datasets composed of molecules at a later step in a drug development process, whereas the classification model predicted a polarized distribution with two extreme values for all datasets presumably due to the overconfident prediction for unseen data. Thus, this new concept offers a pragmatic tool for drug-likeness scoring and further can be applied to other biochemical applications.

Blind Image Super Resolution Using Deep Unsupervised Learning

The goal of single image super resolution (SISR) is to recover a high-resolution (HR) image from a low-resolution (LR) image. Deep learning based methods have recently made a remarkable performance gain in terms of both the effectiveness and efficiency for SISR. Most existing methods have to be trained based on large-scale synthetic paired data in a fully supervised manner. With the available HR natural images, the corresponding LR images are usually synthesized with a simple fixed degradation operation, such as bicubic down-sampling. Then, the learned deep models with these training data usually face difficulty to be generalized to real scenarios with unknown and complicated degradation operations. This study exploits a novel blind image super-resolution framework using a deep unsupervised learning network. The proposed method can simultaneously predict the underlying HR image and its specific degradation operation from the observed LR image only without any prior knowledge. The experimental results on three benchmark datasets validate that our proposed method achieves a promising performance under the unknown degradation models.

CCT: Conditional Co-Training for Truly Unsupervised Remote Sensing Image Segmentation in Coastal Areas

As the fastest growing trend in big data analysis, deep learning technology has proven to be both an unprecedented breakthrough and a powerful tool in many fields, particularly for image segmentation tasks. Nevertheless, most achievements depend on high-quality pre-labeled training samples, which are labor-intensive and time-consuming. Furthermore, different from conventional natural images, coastal remote sensing ones generally carry far more complicated and considerable land cover information, making it difficult to produce pre-labeled references for supervised image segmentation. In our research, motivated by this observation, we take an in-depth investigation on the utilization of neural networks for unsupervised learning and propose a novel method, namely conditional co-training (CCT), specifically for truly unsupervised remote sensing image segmentation in coastal areas. In our idea, a multi-model framework consisting of two parallel data streams, which are superpixel-based over-segmentation and pixel-level semantic segmentation, is proposed to simultaneously perform the pixel-level classification. The former processes the input image into multiple over-segments, providing self-constrained guidance for model training. Meanwhile, with this guidance, the latter continuously processes the input image into multi-channel response maps until the model converges. Incentivized by multiple conditional constraints, our framework learns to extract high-level semantic knowledge and produce full-resolution segmentation maps without pre-labeled ground truths. Compared to the black-box solutions in conventional supervised learning manners, this method is of stronger explainability and transparency for its specific architecture and mechanism. The experimental results on two representative real-world coastal remote sensing datasets of image segmentation and the comparison with other state-of-the-art truly unsupervised methods validate the plausible performance and excellent efficiency of our proposed CCT.