Unsupervised Deep Learning Model Research Articles

Human-Level Comparable Control Volumes Mapping With an Unsupervised-Learning Model for CT-Guided Radiotherapy

To develop an unsupervised deep learning model with auto-mapped control volume (CV) from daily patient positioning CT (dCT) to planning computed tomography (pCT) for highly accurate patient positioning.An unsupervised learning framework is proposed to automatically generate the couch shifts (translations and rotations) for mapping CVs from dCT to pCT. Inputs to the network are the dCT, the pCT, and the CVs' locations within the pCT. The outputs are the transformational parameters of the dCT for head-and-neck cancer (HNC) patient positioning. We train the network to maximize image similarity between the CV in the pCT and dCT using normalized cross-correlation. We used a total of 158 HNC patients with 554 CT scans for network evaluation. Each patient underwent several CT scans at different time points. For the test cases, couch shifts are obtained by averaging translational and rotational parameters derived with different CVs. These means are then compared to ground-truth reference shifts obtained by the alignment of bony landmarks identified by an experienced radiation oncologist.Systematic/random positioning errors between the model prediction and the reference are smaller than 0.47/1.13 mm and 0.17/0.29° in translations and rotations, respectively. Pearson's correlation coefficient between model predictions and reference values exceeded 0.98. In comparison to standard registrations, the proposed method increased the proportion of cases registered within clinically accepted tolerance from 66.67% to 90.91%.A novel unsupervised learning technique was established to map CVs from pCT to dCT for HNC patient positioning. Our results show that fast and highly accurate HNC patient positioning is achievable by leveraging state-of-the-art deep learning strategies.

An Unsupervised Deep Feature Learning Model Based on Parallel Convolutional Autoencoder for Intelligent Fault Diagnosis of Main Reducer.

Traditional diagnostic framework consists of three parts: data acquisition, feature generation, and fault classification. However, manual feature extraction utilized signal processing technologies heavily depending on subjectivity and prior knowledge which affect the effectiveness and efficiency. To tackle these problems, an unsupervised deep feature learning model based on parallel convolutional autoencoder (PCAE) is proposed and applied in the stage of feature generation of diagnostic framework. Firstly, raw vibration signals are normalized and segmented into sample set by sliding window. Secondly, deep features are, respectively, extracted from reshaped form of raw sample set and spectrogram in time-frequency domain by two parallel unsupervised feature learning branches based on convolutional autoencoder (CAE). During the training process, dropout regularization and batch normalization are utilized to prevent over fitting. Finally, extracted representative features are feed into the classification model based on deep structure of neural network (DNN) with softmax. The effectiveness of the proposed approach is evaluated in fault diagnosis of automobile main reducer. The results produced in contrastive analysis demonstrate that the diagnostic framework based on parallel unsupervised feature learning and deep structure of classification can effectively enhance the robustness and enhance the identification accuracy of operation conditions by nearly 8%.

Image fusion algorithm based on unsupervised deep learning-optimized sparse representation

The image fusion method based on deep learning has problems such as the supervised learning of the model, the edge and noise of the fused image, and the setting of the image fusion weight map. To solve these problems, this paper proposes an end-to-end unsupervised deep learning model that performs one-to-many focus image fusion. It solves the training problem encountered by supervised deep learning models while avoiding the unreasonable image fusion weight maps. In addition, this paper proposes an optimized sparse representation method that divides an image into a target area and a background area. Then, it uses super complete dictionary learning to obtain a sparse representation of the image background area. This approach makes the proposed unsupervised deep learning image fusion method robust to noise. Finally, using this method to carry out image fusion experiments, the results show that the quality evaluation indicators of the fused image obtained by this method substantially outperform those of both mainstream machine learning and deep learning image fusion methods.

Automatic Sleep Staging Using CNN-HMM Based on Raw Single-Channel EEG

Although convolutional neural network (CNN) has become very popular, and has been applied to the sleep stage classification problem, almost all existing studies on sleep stage classification require a lot of labeled data. Obtaining labeled data is a subjective process and difficulty task. At the same time, due to different knowledge backgrounds, the sleep stage labels scored by different experts will be different. Therefore, a new unsupervised competition convolutional neural network (C-CNN) is proposed in this study. It consists of alternating layers containing a convolution operator, competitive operator, and pooling operator. The convolution operator is used to extract features from EEG signals. The competitive layer iteratively adjusts the weight vectors of the winning neurons according to the competition learning rules. By this way, the learned weight vectors can reflect the distribution of input samples. To evaluate the C-CNN model, two common datasets (UCD and Sleep-EDF) are used. The proposed model obtains a classification performance of 77.2% and 83.4% on UCD and Sleep-EDF datasets, respectively. The experimental results also show that our method outperforms the base models by 4.3% and 9.47%, respectively. This work provides avenues for further studies of unsupervised deep learning models.

A Tour of Unsupervised Deep Learning for Medical Image Analysis.

Interpretation of medical images for the diagnosis and treatment of complex diseases from high-dimensional and heterogeneous data remains a key challenge in transforming healthcare. In the last few years, both supervised and unsupervised deep learning achieved promising results in the area of medical image analysis. Several reviews on supervised deep learning are published, but hardly any rigorous review on unsupervised deep learning for medical image analysis is available. The objective of this review is to systematically present various unsupervised deep learning models, tools, and benchmark datasets applied to medical image analysis. Some of the discussed models are autoencoders and their variants, Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Deep Boltzmann Machine (DBM), and Generative Adversarial Network (GAN). Future research opportunities and challenges of unsupervised deep learning techniques for medical image analysis are also discussed. Currently, interpretation of medical images for diagnostic purposes is usually performed by human experts that may be replaced by computer-aided diagnosis due to advancement in machine learning techniques, including deep learning, and the availability of cheap computing infrastructure through cloud computing. Both supervised and unsupervised machine learning approaches are widely applied in medical image analysis, each of them having certain pros and cons. Since human supervisions are not always available or are inadequate or biased, therefore, unsupervised learning algorithms give a big hope with lots of advantages for biomedical image analysis.

Insurance fraud detection with unsupervised deep learning

AbstractThe objective of this paper is to propose a novel deep learning methodology to gain pragmatic insights into the behavior of an insured person using unsupervised variable importance. It lays the groundwork for understanding how insights can be gained into the fraudulent behavior of an insured person with minimum effort. Starting with a preliminary investigation of the limitations of the existing fraud detection models, we propose a new variable importance methodology incorporated with two prominent unsupervised deep learning models, namely, the autoencoder and the variational autoencoder. Each model's dynamics is discussed to inform the reader on how models can be adapted for fraud detection and how results can be perceived appropriately. Both qualitative and quantitative performance evaluations are conducted, although a greater emphasis is placed on qualitative evaluation. To broaden the scope of reference of fraud detection setting, various metrics are used in the qualitative evaluation.

Stacked Convolutional Sparse Auto-Encoders for Representation Learning

Deep learning seeks to achieve excellent performance for representation learning in image datasets. However, supervised deep learning models such as convolutional neural networks require a large number of labeled image data, which is intractable in applications, while unsupervised deep learning models like stacked denoising auto-encoder cannot employ label information. Meanwhile, the redundancy of image data incurs performance degradation on representation learning for aforementioned models. To address these problems, we propose a semi-supervised deep learning framework called stacked convolutional sparse auto-encoder, which can learn robust and sparse representations from image data with fewer labeled data records. More specifically, the framework is constructed by stacking layers. In each layer, higher layer feature representations are generated by features of lower layers in a convolutional way with kernels learned by a sparse auto-encoder. Meanwhile, to solve the data redundance problem, the algorithm of Reconstruction Independent Component Analysis is designed to train on patches for sphering the input data. The label information is encoded using a Softmax Regression model for semi-supervised learning. With this framework, higher level representations are learned by layers mapping from image data. It can boost the performance of the base subsequent classifiers such as support vector machines. Extensive experiments demonstrate the superior classification performance of our framework compared to several state-of-the-art representation learning methods.

Modeling Healthy Anatomy with Artificial Intelligence for Unsupervised Anomaly Detection in Brain MRI.

To develop an unsupervised deep learning model on MR images of normal brain anatomy to automatically detect deviations indicative of pathologic states on abnormal MR images. In this retrospective study, spatial autoencoders with skip-connections (which can learn to compress and reconstruct data) were leveraged to learn the normal variability of the brain from MR scans of healthy individuals. A total of 100 normal, in-house MR scans were used for training. Subsequently, as the model was unable to reconstruct anomalies well, this characteristic was exploited for detecting and delineating various diseases by computing the difference between the input data and their reconstruction. The unsupervised model was compared with a supervised U-Net- and threshold-based classifier trained on data from 50 patients with multiple sclerosis (in-house dataset) and 50 patients from The Cancer Imaging Archive. Both the unsupervised and supervised U-Net models were tested on five different datasets containing MR images of microangiopathy, glioblastoma, and multiple sclerosis. Precision-recall statistics and derivations thereof (mean area under the precision-recall curve, Dice score) were used to quantify lesion detection and segmentation performance. The unsupervised approach outperformed the naive thresholding approach in lesion detection (mean F1 scores ranging from 17% to 62% vs 6.4% to 15% across the five different datasets) and performed similarly to the supervised U-Net (20%-64%) across a variety of pathologic conditions. This outperformance was mostly driven by improved precision compared with the thresholding approach (mean precisions, 15%-59% vs 3.4%-10%). The model was also developed to create an anomaly heatmap display. The unsupervised deep learning model was able to automatically detect anomalies on brain MR images with high performance. Supplemental material is available for this article. Keywords: Brain/Brain Stem Computer Aided Diagnosis (CAD), Convolutional Neural Network (CNN), Experimental Investigations, Head/Neck, MR-Imaging, Quantification, Segmentation, Stacked Auto-Encoders, Technology Assessment, Tissue Characterization © RSNA, 2021.

Competition convolutional neural network for sleep stage classification

Although convolutional neural network (CNN) has become very popular, and has been applied to the sleep stage classification problem, almost all existing studies on sleep stage classification require a lot of labeled data. Obtaining labeled data is a subjective process and difficulty task. At the same time, due to different knowledge backgrounds, the sleep stage labels scored by different experts will be different. Therefore, a new unsupervised competition convolutional neural network (C-CNN) is proposed in this study. It consists of alternating layers containing a convolution operator, competitive operator, and pooling operator. The convolution operator is used to extract features from EEG signals. The competitive layer iteratively adjusts the weight vectors of the winning neurons according to the competition learning rules. By this way, the learned weight vectors can reflect the distribution of input samples. To evaluate the C-CNN model, two common datasets (UCD and Sleep-EDF) are used. The proposed model obtains a classification performance of 77.2% and 83.4% on UCD and Sleep-EDF datasets, respectively. The experimental results also show that our method outperforms the base models by 4.3% and 9.47%, respectively. This work provides avenues for further studies of unsupervised deep learning models.

Exploring Temporal Information for Dynamic Network Embedding

Representing nodes in a network as low-dimensional dense vectors can facilitate the analysis of complex networks, which is a challenging task and has attracted increasing attention. However, in the real world, networks are changing over time, such as cooperation in citation networks and communication in email networks. Most of the recent embedding methods only focus on static networks. Thus they ignore the critical temporal information, which serves as a supplement to structure information and has been proved to improve the quality of node embedding. In this work, we propose an unsupervised deep learning model called DTINE, which explores temporal information for further enhancing the robustness of node representations in dynamic networks. To preserve network topology, we pertinently design a temporal weight and sampling strategy to extract features from the neighborhoods. An attention mechanism will be applied on the recurrent neural network to measure the contributions of historical information and capture the evolution of the networks. Experimental results on four real-world networks demonstrate that the proposed method achieves better performance than state-of-the-art methods.

SESF-Fuse: an unsupervised deep model for multi-focus image fusion

Muti-focus image fusion is the extraction of focused regions from different images to create one all-in-focus fused image. The key point is that only objects within the depth-of-field have a sharp appearance in the photograph, while other objects are likely to be blurred. We propose an unsupervised deep learning model for multi-focus image fusion. We train an encoder–decoder network in an unsupervised manner to acquire deep features of input images. Then, we utilize spatial frequency, a gradient-based method to measure sharp variation from these deep features, to reflect activity levels. We apply some consistency verification methods to adjust the decision map and draw out the fused result. Our method analyzes sharp appearances in deep features instead of original images, which can be seen as another success story of unsupervised learning in image processing. Experimental results demonstrate that the proposed method achieves state-of-the-art fusion performance compared to 16 fusion methods in objective and subjective assessments, especially in gradient-based fusion metrics.

Unsupervised Denoising of Optical Coherence Tomography Images With Nonlocal-Generative Adversarial Network

Deep learning for image denoising has recently attracted considerable attentions due to its excellent performance. Since most of current deep learning-based denoising models require a large number of clean images for training, it is difficult to extend them to the denoising problems when the reference clean images are hard to acquire (e.g., optical coherence tomography (OCT) images). In this article, we propose a novel unsupervised deep learning model called as nonlocal-generative adversarial network (nonlocal-GAN) for OCT image denoising, where the deep model can be trained without reference clean images. Specifically, considering that the background areas of OCT images mainly contain pure real noise samples, we creatively train a discriminator to distinguish background real noise samples from the fake noise samples generated by the denoiser, that is the generator, and then the discriminator will guide the generator for denoising. To further enhance denoising performance, we introduce a nonlocal means layer into the generator of the nonlocal-GAN model. Furthermore, since nearby several OCT B-scans have strong correlations, we also propose a nonlocal-GAN-M model to utilize the high correlations within nearby B-scans. Extensive experimental results on clinical retinal OCT images demonstrate the effectiveness and efficiency of the proposed method.

Deep learning-based community detection in complex networks with network partitioning and reduction of trainable parameters

Community detection in complex networks is an important multidisciplinary research area and is considered crucial for understanding the structure of complex networks. Unsupervised deep learning models (e.g. stack autoencoders) have been successfully proposed for the problem of community detection, which can extract network features and use them in splitting the network into communities. Despite their effectiveness, these methods are not very efficient, especially when large networks are involved. Furthermore, existing models usually handle the network as a single object, which means that massive trainable parameters are required during training (which in normal complex networks often reach millions of parameters, not to mention large networks), thereby increasing the complexity of the model. To overcome these problems, this paper proposes a deep autoencoder method for network community detection based on three techniques: network-data partitioning, reduction and sharing of trainable parameters, which contribute significantly to improve the efficiency of the method. A new partitioning strategy is imposed on complex networks at different levels. The paper also proposes a parallel design for the proposed method. Furthermore, a new similarity constraint function is proposed to improve and preserve the performance of community detection task. We performed extensive experiments for different partitioning levels of a network-dataset to evaluate the method with CPU and GPU devices. The results showed that the proposed method significantly improved training speed and efficiency while maintaining performance. The results also showed that the efficiency of the method increases as we move to a deeper level of partitioning.

Deep Unsupervised Embedding for Remotely Sensed Images Based on Spatially Augmented Momentum Contrast

Convolutional neural networks (CNNs) have achieved great success when characterizing remote sensing (RS) images. However, the lack of sufficient annotated data (together with the high complexity of the RS image domain) often makes supervised and transfer learning schemes limited from an operational perspective. Despite the fact that unsupervised methods can potentially relieve these limitations, they are frequently unable to effectively exploit relevant prior knowledge about the RS domain, which may eventually constrain their final performance. In order to address these challenges, this article presents a new unsupervised deep metric learning model, called spatially augmented momentum contrast (SauMoCo), which has been specially designed to characterize unlabeled RS scenes. Based on the first law of geography, the proposed approach defines spatial augmentation criteria to uncover semantic relationships among land cover tiles. Then, a queue of deep embeddings is constructed to enhance the semantic variety of RS tiles within the considered contrastive learning process, where an auxiliary CNN model serves as an updating mechanism. Our experimental comparison, including different state-of-the-art techniques and benchmark RS image archives, reveals that the proposed approach obtains remarkable performance gains when characterizing unlabeled scenes since it is able to substantially enhance the discrimination ability among complex land cover categories. The source codes of this article will be made available to the RS community for reproducible research.

Deep Learning the Morphology of Dark Matter Substructure

Abstract Strong gravitational lensing is a promising probe of the substructure of dark matter halos. Deep-learning methods have the potential to accurately identify images containing substructure, and differentiate weakly interacting massive particle dark matter from other well motivated models, including vortex substructure of dark matter condensates and superfluids. This is crucial in future efforts to identify the true nature of dark matter. We implement, for the first time, a classification approach to identifying dark matter based on simulated strong lensing images with different substructure. Utilizing convolutional neural networks trained on sets of simulated images, we demonstrate the feasibility of deep neural networks to reliably distinguish among different types of dark matter substructure. With thousands of strong lensing images anticipated with the coming launch of Vera C. Rubin Observatory, we expect that supervised and unsupervised deep-learning models will play a crucial role in determining the nature of dark matter.

Stacked Fusion Supervised Auto-encoder with an Additional Classification Layer

Auto-encoders are unsupervised deep learning models, which try to learn hidden representations to reconstruct the inputs. While the learned representations are suitable for applications related to unsupervised reconstruction, they may not be optimal for classification. In this paper, we propose a supervised auto-encoder (SupAE) with an addition classification layer on the representation layer to jointly predict targets and reconstruct inputs, so it can learn discriminative features specifically for classification tasks. We stack several SupAE and apply a greedy layer-by-layer training approach to learn the stacked supervised auto-encoder (SSupAE). Then an adaptive weighted majority voting algorithm is proposed to fuse the prediction results of SupAE and the SSupAE, because each individual SupAE and the final SSupAE can both get the posterior probability information of samples belong to each class, we introduce Shannon entropy to measure the classification ability for different samples based on the posterior probability information, and assign high weight to sample with low entropy, thus more reasonable weights are assigned to different samples adaptively. Finally, we fuse the different results of classification layer with the proposed adaptive weighted majority voting algorithm to get the final recognition results. Experimental results on several classification datasets show that our model can learn discriminative features and improve the classification performance significantly.

A framework towards data analytics on host–pathogen protein–protein interactions

With the rapid development of high-throughput technologies, systems biology is now embracing a great opportunity made possible by the increased accumulation of data available online. Biological data analytics is considered as a critical means to contribute to a better understanding on such data through extraction of the latent features, relationships and the associated mechanisms. Therefore, it is important to evaluate how to involve data analytics from both computational and biological perspectives in practice. This paper has investigated interaction relationships in the proteomics area, which provide insights of the critical molecular processes within infection mechanisms. Specifically, we focused on host–pathogen protein–protein interactions, which represented the primary challenges associated with infectious diseases and drug design. Accordingly, a novel framework based on data analytics and machine learning techniques is detailed for analyzing these areas and we will describe the analytical results from host–pathogen protein–protein interactions (HP-PPI). Based on this framework, which serves as a pipeline solution for extracting and learning from the raw proteomics data, we have firstly evaluated several models from literature using different analytic technologies and performance measurements. An unsupervised deep learning model based on stacked denoising autoencoders, is subsequently proposed to capture higher level feature regarding the sequence information in the framework. The achieved performance indicates a superior capability of the unsupervised deep learning model in dealing with the host–pathogen protein interactions scenario among all of these models. The results will further help to enrich a theoretical and technical foundation for analyzing HP-PPI networks.

An Unsupervised Deep Learning Model for Early Network Traffic Anomaly Detection

Various attacks have emerged as the major threats to the success of a connected world like the Internet of Things (IoT), in which billions of devices interact with each other to facilitate human life. By exploiting the vulnerabilities of cheap and insecure devices such as IP cameras, an attacker can create hundreds of thousands of zombie devices and then launch massive volume attacks to take down any target. For example, in 2016, a record large-scale DDoS attack launched by millions of Mirai-injected IP cameras and smart printers blocked the accessibility of several high-profile websites. To date, the state-of-the-art defense systems against such attacks rely mostly on pre-defined features extracted from the entire flows or signatures. The feature definitions are manual, and it would be too late to block a malicious flow after extracting the flow features. In this work, we present an effective anomaly traffic detection mechanism, namely D-PACK, which consists of a Convolutional Neural Network (CNN) and an unsupervised deep learning model (e.g., Autoencoder) for auto-profiling the traffic patterns and filtering abnormal traffic. Notably, D-PACK inspects only the first few bytes of the first few packets in each flow for early detection. Our experimental results show that, by examining just the first two packets in each flow, D-PACK still performs with nearly 100% accuracy, while features an extremely low false-positive rate, e.g., 0.83%. The design can inspire the emerging efforts towards online anomaly detection systems that feature reducing the volume of processed packets and blocking malicious flows in time.

Diffeomorphic unsupervised deep learning model for mono- and multi-modality registration

Different from image segmentation, developing a deep learning network for image registration is less straightforward because training data cannot be prepared or supervised by humans unless they are trivial (e.g. pre-designed affine transforms). One approach for an unsupervised deep leaning model is to self-train the deformation fields by a network based on a loss function with an image similarity metric and a regularisation term, just with traditional variational methods. Such a function consists in a smoothing constraint on the derivatives and a constraint on the determinant of the transformation in order to obtain a spatially smooth and plausible solution. Although any variational model may be used to work with a deep learning algorithm, the challenge lies in achieving robustness. The proposed algorithm is first trained based on a new and robust variational model and tested on synthetic and real mono-modal images. The results show how it deals with large deformation registration problems and leads to a real time solution with no folding. It is then generalised to multi-modal images. Experiments and comparisons with learning and non-learning models demonstrate that this approach can deliver good performances and simultaneously generate an accurate diffeomorphic transformation.

Unsupervised deep-learning-powered anomaly detection for instrumented infrastructure

Deep learning methods have recently shown great success in numerous fields, including finance, healthcare, linguistics, robotics and even cybersports. Unsupervised learning methods identify the dominant patterns of variability that shape a data set. Such patterns may correspond to well-understood processes, previously unknown clusters or anomalies. This paper presents a case study where a state-of-the-art family of unsupervised deep learning models called variational autoencoder (VAE) is applied to data accrued from a network of fibre-optic sensors installed within a composite steel–concrete half-through railway bridge. The goals were (a) to characterise automatically the behaviour of the bridge based on sensor measurements and, (b) based on this characterisation, to determine when a train passes across a bridge. Based on the VAE model, an algorithm is presented to identify automatically the ‘train event’ points in an unsupervised setting. Two architectures for the VAE model are compared with commonly used baselines. The architecture tailored for modelling sequential data is shown to outperform other methods considered, on both seen and unseen data. No special hyperparameter optimisation is required. This study illustrates how state-of-the-art deep learning methods can be applied to a civil infrastructure engineering problem without directly modelling the physics of the objects or performing tedious hyperparameter optimisation.