Deep Autoencoder Network Research Articles

A novel deep quantile matrix completion model for top-N recommendation

Matrix completion models have been receiving keen attention due to their wide applications in science and engineering. However, the majority of these models assumes a symmetric noise distribution in their completion processes and uses conditional mean to characterize data distribution in a data set, the assumption of which incurs noticeable bias toward outliers. Recognizing the fact that noise distribution tends to be asymmetric in the real-world, this paper proposes a novel Deep Quantile Matrix Completion model, abbreviated as DQMC, which aims to accurately capture noise distribution in a data set by modeling conditional quantile of the data set instead of its conditional mean as traditionally handled by many state-of-the-art methods. Implemented via a deep computing paradigm, the newly proposed model maps a data set from its input space to the latent spaces through a two-branched deep autoencoder network. Such a mapping can effectively capture complex information latent in the data set. The proposed model is empowered by two key designed elements, including: (1) its two-branched deep autoencoder network that provides a flexible computing pathway to attain completion results with a high quality; (2) the introduction of a quantile loss function in combination with the proposed deep network, leading to a new unsupervised learning algorithm for tackling the matrix completion tasks with a superior capability. Comparative experimental results consistently demonstrate the superiority of the proposed DQMC model in conducting the top-N recommendation tasks involving both explicit and implicit rating data sets with respect to a series of state-of-the-art recommendation algorithms.

OutlierNets: Highly Compact Deep Autoencoder Network Architectures for On-Device Acoustic Anomaly Detection.

Human operators often diagnose industrial machinery via anomalous sounds. Given the new advances in the field of machine learning, automated acoustic anomaly detection can lead to reliable maintenance of machinery. However, deep learning-driven anomaly detection methods often require an extensive amount of computational resources prohibiting their deployment in factories. Here we explore a machine-driven design exploration strategy to create OutlierNets, a family of highly compact deep convolutional autoencoder network architectures featuring as few as 686 parameters, model sizes as small as 2.7 KB, and as low as 2.8 million FLOPs, with a detection accuracy matching or exceeding published architectures with as many as 4 million parameters. The architectures are deployed on an Intel Core i5 as well as a ARM Cortex A72 to assess performance on hardware that is likely to be used in industry. Experimental results on the model’s latency show that the OutlierNet architectures can achieve as much as 30× lower latency than published networks.

LRGAN: Visual anomaly detection using GAN with locality-preferred recoding

Deep neural networks, including deep auto-encoder (DAE) and generative adversarial networks (GAN), have been extensively applied for visual anomaly detection. These models generally assume that reconstruction errors should be lower for normal samples but higher for anomalies. However, it has been found that DAE based models can sometimes reconstruct anomalies very well and thus result in false alarms or misdetections. To address this problem, we propose a model using GAN with locality-preferred recoding, named LRGAN. LRGAN is inspired by the observation that both normal and abnormal samples are not completely scattered throughout the latent space but clustered separately at some local regions. Therefore, a locality-preferred recoding (LR) module is designed to compulsively represent the latent vectors of anomalies by normal ones. As a result, reconstructions of anomalies will approximate to normal samples and corresponding residuals can thus be enlarged. To partly avoid latent vectors of normal samples being recoded, we further present an improved model using GAN with an adaptive LR (ALR) module, named LRGAN+. ALR applies the clustering algorithm to generate a more compact codebook; more importantly, it helps LRGAN + automatically skip the LR module for possible normal samples with a threshold strategy. Our proposed method is evaluated on two public datasets (i.e., MNIST and CIFAR-10) and one real-world industrial dataset (i.e., Fasteners), considering both one-class and multi-class anomaly detection protocols. Experimental results demonstrate that LRGAN is comparable with state-of-the-art methods and LRGAN + outperforms these methods on all datasets.

Orthogonal Nonnegative Matrix Factorization using a novel deep Autoencoder Network

Orthogonal Nonnegative Matrix Factorization (ONMF) offers an important analytical vehicle for addressing many problems. Encouraged by record-breaking successes attained by neural computing models in solving an assortment of data analytics tasks, a rich collection of neural computing models has been proposed to perform ONMF with compelling performance. Such existing models can be broadly classified into the shallow-layered structure (SLS) based and deep-layered structure (DLS) based models. However, SLS models cannot capture complex relationships and hierarchical information latent in a matrix due to their simple network structures and DLS models rely on an iterative procedure to derive weights, leading to a less efficient solution process and cannot be reused to factorize new matrices. To overcome these shortcomings, this paper proposes a novel deep autoencoder network for ONMF, which is abbreviated as DAutoED-ONMF. Compared with SLS models, the newly proposed model is capable of generating solutions with good interpretability and solution uniqueness like original SLS models, yet the new model attains a superior learning capability thanks to its deep structure employed. In comparison with DLS models, the new model trains a reusable encoder network to directly factorize any given matrix with no need to repeatedly retrain the model for factorizing multiple matrices using a tailor-designed network training procedure. Proof of the procedure’s convergence is presented with an analysis of its computational complexity. The numerical experiments conducted on several publicly data sets convincingly demonstrate that the proposed DAutoED-ONMF model gains promising performance in terms of multiple metrics.

Recognition of multivariate geochemical anomalies associated with mineralization using an improved generative adversarial network

Deep learning (DL) algorithms have a strong ability to recognize high-level features in geochemical exploration data and have been widely employed for the recognition of multivariate geochemical anomalies linked to mineralization. In this study, the adversarially learned anomaly detection (ALAD) algorithm, an improved generative adversarial network (GAN), was employed to detect multivariate geochemical anomalies related to mineralization. Compared with other unsupervised deep learning algorithms, ALAD significantly improves anomaly detection performance by combining the advantages of deep variational autoencoder and generative adversarial network. Various experiments were performed to construct a well-designed network structure to process high-dimensional geochemical data to identify geochemical anomalies linked to tungsten (W) polymetallic mineralization in the south of Jiangxi Province of China. The extracted geochemical anomalies have a high spatial distribution correction with the locations of the discovered W polymetallic mineralization. Both the area under the receiver operating characteristic curve and the prediction-area plot indicate a good performance of the ALAD model. Furthermore, the extracted geochemical anomalies are spatially correlated with granites which control the spatial distribution of W polymetallic mineralization. These observations indicate that, as an unsupervised deep learning algorithm, ALAD is useful for detecting geochemical anomalies associated with mineralization.

Deep autoencoder with localized stochastic sensitivity for short-term load forecasting

This paper presents a short-term electric load forecasting model based on deep autoencoder with localized stochastic sensitivity (D-LiSSA). D-LiSSA can learn informative hidden representations from unseen samples by minimizing the perturbed error (including the training error and stochastic sensitivity) from historical load data. Specifically, this general deep autoencoder network as a deep learning model improves prediction accuracy and reliability. Moreover, a nonlinear fully connected feedforward neural network as a regression layer is applied to forecast the short-term load, with the generalization capability of the proposed model using hidden representations learned by D-LiSSA. The performance of D-LiSSA is evaluated using real-world public electric load markets of France (FR), Germany (GR), Romania (RO), and Spain (ES) from ENTSO-E. Extensive experimental results and comparisons with the classical and state-of-the-art models show that D-LiSSA yields accurate load forecasting results and achieves desired reliable capability. For instance, with the French case, D-LiSSA yields the lowest mean absolute error, mean absolute percentage error, root mean squared error; providing up to 61.89%, 63.20%, and 56.40% forecasting accuracy improvements as compared to the benchmark model for forecasting hourly horizon, respectively.

SAE based unified double JPEG compression detection system for Web image forensics

PurposeRecently, the double joint photographic experts group (JPEG) compression detection tasks have been paid much more attention in the field of Web image forensics. Although there are several useful methods proposed for double JPEG compression detection when the quantization matrices are different in the primary and secondary compression processes, it is still a difficult problem when the quantization matrices are the same. Moreover, those methods for the different or the same quantization matrices are implemented in independent ways. The paper aims to build a new unified framework for detecting the doubly JPEG compression.Design/methodology/approachFirst, the Y channel of JPEG images is cut into 8 × 8 nonoverlapping blocks, and two groups of features that characterize the artifacts caused by doubly JPEG compression with the same and the different quantization matrices are extracted on those blocks. Then, the Riemannian manifold learning is applied for dimensionality reduction while preserving the local intrinsic structure of the features. Finally, a deep stack autoencoder network with seven layers is designed to detect the doubly JPEG compression.FindingsExperimental results with different quality factors have shown that the proposed approach performs much better than the state-of-the-art approaches.Practical implicationsTo verify the integrity and authenticity of Web images, the research of double JPEG compression detection is increasingly paid more attentions.Originality/valueThis paper aims to propose a unified framework to detect the double JPEG compression in the scenario whether the quantization matrix is different or not, which means this approach can be applied in more practical Web forensics tasks.

Adaptive modem and interference suppression based on deep learning

AbstractWith the increasingly fierce competition of electromagnetic spectrum, developing intelligent communication systems that can reconfigure its waveform can effectively improve the communication system's ability to adapt to the complex electromagnetic environment. In this paper, an adaptive modem based on a deep autoencoder network (DAN) is designed, and the demodulation performance that is close to, consistent with, or better than traditional MPSK or QAM is achieved. This DAN can be trained by a unified loss function and a unified optimization algorithm to implement a variety of modems. For high‐order modems, the constellation diagrams generated by the DAN are radically different from the traditional modulation and very difficult to distinguish linearly, which is beneficial to improve the anti‐interception ability of the communication systems. Additionally, based on the trained DAN, a convolutional neural network (CNN) with a single convolutional layer is incorporated to suppress a variety of interferences. By using transfer learning, the new deep learning (DL) model with the CNN converges fast with a few epochs of training. Training and test results verify that the new DL model can improve the anti‐interference ability of the communication systems by learning and suppressing the interferences in both frequency and power domains.

A New Improved Algorithm for Aeromagnetic Compensation

Magnetic compensation is a necessary step in aeromagnetic data processing. The aeromagnetic compensation model is a linear regression model, but the model has multiple collinearity problems, which will reduce the performance of the compensation model. In view of this problem, we propose a deep autoencoder (DAE) aeromagnetic compensation algorithm. The DAE network extracts the features of the data by learning the compressed representation of the coefficient matrix, thereby weakening the correlation between the coefficient matrix variables. The feature obtained after dimension reduction is used for compensation calculation. The DAE algorithm is verified by Unmanned Aerial Vehicles, and the results show that the compensation quality of the DAE is better than the least squares algorithm.

Unsupervised Pixel-Wise Hyperspectral Anomaly Detection via Autoencoding Adversarial Networks

We propose a completely unsupervised pixel-wise anomaly detection method for hyperspectral images. The proposed method consists of three steps called data preparation, reconstruction, and detection. In the data preparation step, we apply a background purification to train the deep network in an unsupervised manner. In the reconstruction step, we propose to use three different deep autoencoding adversarial network (AEAN) models including 1D-AEAN, 2D-AEAN, and 3D-AEAN which are developed for working on spectral, spatial, and joint spectral-spatial domains, respectively. The goal of the AEAN models is to generate synthesized hyperspectral images (HSIs) which are close to real ones. A reconstruction error map (REM) is calculated between the original and the synthesized image pixels. In the detection step, we propose to use a WRX-based detector in which the pixel weights are obtained according to REM. We compare our proposed method with the classical RX, WRX, support vector data description-based (SVDD), collaborative representation-based detector (CRD), adaptive weight deep belief network (AW-DBN) detector and deep autoencoder anomaly detection (DAEAD) method on real hyperspectral datasets. The experimental results show that the proposed approach outperforms other detectors in the benchmark.

High imaging quality of Fourier single pixel imaging based on generative adversarial networks at low sampling rate

Single pixel imaging is an innovative imaging scheme using active light to obtain spatial information, which has attracted much attention in the computational imaging field. However, for single pixel imaging, it is a great challenge to find an efficient technique to obtain imaging results with high quality under low sampling conditions. In order to solve this problem, a Fourier single pixel imaging (FSPI) based on a generative adversarial network (GAN) is proposed in this paper. In the proposed GAN model, perceptual loss, pixel and frequency loss are incorporated into the total loss function to better preserve the details of the target. With the help of the GAN model, the FSPI can reconstruct results with high quality at low sampling rate conditions. The numerical simulation and experiment are implemented. Compared with conventional FSPI and FSPI based on a deep convolutional auto-encoder network, the proposed method has a better visual effect and image quality evaluation index. This approach is particularly important to high speed single pixel imaging applications due to its potential for reconstructing the high-quality target image with a low sampling rate.

A network security situation assessment method based on adversarial deep learning

Aiming at the poor performance and flexibility of traditional assessment methods of network security in dealing with a large number of network attack data, this paper proposes a network security situation assessment method based on adversarial deep learning. Firstly, establish the Deep Autoencoder-Deep Neural Network (AEDNN) model based on Deep Autoencoder (DAE) and Deep Neural Network (DNN). Conduct feature learning on the DAE network and apply the DNN network as a network attacks classifier. In the training process, for considering the results of feature learning in DAE, this paper builds an adversarial training process by changing the training weights. Besides, to increase the performance of the model on the minority network attacks, the Under–Over Sampling Weighted (UOSW) algorithm is designed; Finally, conduct model testing and calculate the network security situation value. The compared results of other models show the proposed model is more accurate for identifying network attacks and can evaluate the network situation more comprehensively and flexibly.

A characteristic standardization method for circuit input vectors based on Hash algorithm

The lengths of input vectors corresponding to different circuits are often quite different, which makes it difficult to standardize them. This paper proposes a characteristic standardization method for circuit input vectors based on hash algorithm. First, the collision rates of different hash algorithms are analyzed, and four representative hash algorithms are selected to process the input vectors. Then, based on the given dataset, their performance is analyzed from collision rate, stability and distribution characteristics, and the best performing RSHash algorithm is selected for further research. Next, we deal with the input vectors using RSHash mapping and partitioning strategy, respectively, and then apply processed input vectors to the deep autoencoder network to perform experiments. The experimental results show that the characteristic standardization method for circuit input vectors based on RSHash algorithm has better performance.

Autoencoder Constrained Clustering With Adaptive Neighbors.

The conventional subspace clustering method obtains explicit data representation that captures the global structure of data and clusters via the associated subspace. However, due to the limitation of intrinsic linearity and fixed structure, the advantages of prior structure are limited. To address this problem, in this brief, we embed the structured graph learning with adaptive neighbors into the deep autoencoder networks such that an adaptive deep clustering approach, namely, autoencoder constrained clustering with adaptive neighbors (ACC_AN), is developed. The proposed method not only can adaptively investigate the nonlinear structure of data via a parameter-free graph built upon deep features but also can iteratively strengthen the correlations among the deep representations in the learning process. In addition, the local structure of raw data is preserved by minimizing the reconstruction error. Compared to the state-of-the-art works, ACC_AN is the first deep clustering method embedded with the adaptive structured graph learning to update the latent representation of data and structured deep graph simultaneously.

Deep Learning Algorithms in EEG Signal Decoding Application: A Review

In recent years, deep learning algorithms have been developed rapidly, and they are becoming a powerful tool in biomedical engineering. Especially, there has been an increasing focus on the use of deep learning algorithms for decoding physiological or pathological status of the brain from electroencephalographic (EEG). This paper overviews current application of deep learning algorithms in various EEG decoding tasks, and introduces commonly used algorithms, typical application scenarios, important progresses and existing problems. Firstly, the basic principles of deep learning algorithms used in EEG decoding is briefly described, including convolutional neural network, deep belief network, auto-encoder and recurrent neural network. In this paper, existing applications of deep learning on EEG is discussed, including brain-computer interfaces, cognitive neuroscience and diagnosis of brain disorders. Finally, this paper outlines some key problems that will be addressed in future applications of deep learning for EEG decoding, such as parameter selection, computational complexity, and the capability of generalization.

Synthesis and Analysis-By-Synthesis of Modulated Diplophonic Glottal Area Waveforms

Diplophonia is a type of disordered voice in which two simultaneous pitches are perceived. Most commonly in diplophonic voices, the vocal folds are divided into two parts that vibrate at different frequencies. The glottal area is the projected area of the space between the vocal folds. The glottal area in time is referred to as the glottal area waveform (GAW). The GAW is modeled for diplophonic voice by superimposing two partial GAWs (pGAWs) that are trains of single-peak pulses with different pulse frequencies, i.e., fundamental frequencies ( $f_o$ s). In current kinematic models of diplophonic vocal fold vibration, the pGAWs are assumed to be quasiperiodic. This assumption is mitigated here by modulating pulse-to-pulse cycle length and amplitude. Both random and deterministic modulations are considered. Deterministic modulations depend on the difference of the pGAWs’ instantaneous phases. Model GAWs are fitted to input GAWs using an analysis-by-synthesis approach which we refer to as ‘modulated pulse trains decomposition’ (MPD). MPD is shown to be applicable to diplophonic as well as to nondiplophonic types of dysphonia, which include multi-pulse patterns, random timing behaviours, and chaos. It is mostly robust against modulations but degraded by large random modulations. MPD is compared to a deep autoencoder neural network, and the WaveGlow neural network. In terms of time-domain fitting errors, MPD outperforms the other two approaches unless random modulations are large. MPD outperforms the best of the other two approaches by up to approximately 5 dB. For large random modulations, the deep autoencoder network achieves the smallest fitting errors. In terms of magnitude spectrum fitting errors, WaveGlow is superior except for natural input GAWs containing only nondiplophonic types of dysphonia. Also pulse timing errors are shown to be advantageous for MPD.

Deep multi-kernel auto-encoder network for clustering brain functional connectivity data

In this study, we propose a deep-learning network model called the deep multi-kernel auto-encoder clustering network (DMACN) for clustering functional connectivity data for brain diseases. This model is an end-to-end clustering algorithm that can learn potentially advanced features and cluster disease categories. Unlike other auto-encoders, DMACN has an added self-expression layer and standard back-propagation is used to learn the features that are beneficial for clustering brain functional connectivity data. In the self-expression layer, the kernel matrix is constructed to extract effective features and a new loss function is proposed to constrain the clustering portion, which enables the training of a deep neural learning network that tends to cluster. To test the performance of the proposed algorithm, we applied the end-to-end deep unsupervised clustering algorithm to brain connectivity data. We then conducted experiments based on four public brain functional connectivity data sets and our own functional connectivity data set. The DMACN algorithm yielded good results in various evaluations compared with the existing clustering algorithm for brain functional connectivity data, the deep auto-encoder clustering algorithm, and several other relevant clustering algorithms. The deep-learning-based clustering algorithm has great potential for use in the unsupervised recognition of brain diseases.

Optimization of real-time wireless sensor based big data with deep autoencodernetwork: a tourism sector application with distributed computing

Internet usage has increased rapidly with the development of information communication technologies. The increase in internet usage led to the growth of data volumes on the internet and the emergence of the big data concept. Therefore, it has become even more important to analyze the data and make it meaningful. In this study, 690 million queries and approximately 5.9 quadrillion data collected daily from different servers were recorded on the Redis servers by using real-time big data analysis method and load balance structure for a company operating in the tourism sector. Here, wireless networks were used as a triggering factor to gather data from visitors of the hotels and the analysis was supportedwithanoptimizationapproachthroughthedeepautoencodernetwork. Accordingtothedatadensitygathered from the structure developed with distributed computing and the API software in C# language, server group numbers were increased to list the most affordable hotel in the desired times. Thanks to the developed architecture and software, response times of the servers were significantly reduced. In detail, it was seen that the HAProxy responded 11 times faster than NetScaler as the new architecture responded 1160 times faster than the old one. Also, the HashSet system in the newly developed architecture responded 18 times faster than the List system and as general, the new architecture was found to be 9 times faster than the old architecture.

Image Inpainting Based on Improved Deep Convolutional Auto‐encoder Network

This paper proposes an effective image inpainting method using an improved deep convolutional auto-encoder network. By analogy with exiting methods of image inpainting based on auto-decoders, inpainting methods using the deep convolutional auto-encoder networks are significantly more effective in capturing high-level features than classical methods based on exemplar. However, the inpainted regions would appear blurry and global inconsistency. To alleviate the fuzzy problem, we improved the network model by adding skip connections between mirrored layers in encoder and decoder stacks, so that the generative process of the inpainting area can directly use the low-level features information of the processing image. For making the inpainted result look both more plausible and consistent with its surrounding contexts, the model is trained with a combination of standard pixel-wise reconstruction loss and two adversarial losses which ensures pixel-accurate and local-global contents consistency. With extensive experimental on the ImageNet and Paris Streetview datasets, we demonstrate qualitatively and quantitatively that our approach performs better than state of the art.

A novel fusion approach of deep convolution neural network with auto-encoder and its application in planetary gearbox fault diagnosis

The fault features of gearbox are often influenced and interwoven with each other under the non-stationary condition. The traditional shallow intelligent diagnosis models are difficult to detect and identify gearbox faults with selected features according to prior knowledge. To solve this problem, a novel deep convolutional auto-encoding neural network is designed based on the fusion of the convolutional neural network with the automatic encoder in this research. The vibration signals of gearbox are transformed into Hilbert envelope spectrum by using Hilbert transform and Fourier transform, and the different characteristics of spectral spatial data are automatically learned by convolutional auto-encoding neural network with multiple convolution kernels. The parameters of the convolutional neural network are fine-tuned through a fully connected neural network with a small number of labeled samples. Through the analysis for gearbox fault experiments, the effectiveness and practicability of the proposed method in equipment fault diagnosis are verified. The deep convolutional neural network embedded in the auto-encoder has stronger learning ability, and the diagnosis performance is more stable and reliable in practical engineering application.