Unsupervised Learning Capability Research Articles

An enhanced structural developmental neural network with information saturation for continual unsupervised learning

In this paper, an enhanced structural developmental neural network with information saturation (ESDNNIS) is proposed for continual unsupervised learning. It improves SDNNIS in the following respects: (1) it makes a distinction between the developmental stages of neurons to better regulate the updating of neurons; (2) it adopts the forgetting mechanism to control the network size; (3) it changes the way coverage domain parameter is initialized and updated to divide clusters more accurately; and (4) it adds a parent–child relationship update mechanism to provide better robustness to sample input order. ESDNNIS inherits the unsupervised incremental learning and class segmentation capabilities of SDNNIS, and achieves better class incremental clustering for more complex datasets while using fewer neurons than SDNNIS. The experimental results show that the accuracy of clustering results of ESDNNIS is improved by 10.94% on average compared with SDNNIS, while the number of generated neurons is reduced by 71.93% on average.

Google Play App Forecast Utilizing Machine Learning

In today's dynamic environment, it is possible to accomplish various tasks through the use of machine learning methods. In this investigation, we provide an intricate explanation of the procedures and structures utilized in machine learning. According to future forecasts, it is possible that machine learning will generate the most fitting hypotheses to account for its observable phenomenon. As a result of the abundance of information available, it is not imperative to assign every single data point a specific name, thereby promoting the advancement of its unsupervised learning capabilities in the meantime. It is anticipated that the neural network arrangements will become increasingly unpredictable as they distribute semantic details into distinct categories. In addition, deep learning is set to become even more robust with better adaptation assistance, and utilizing these sites of interest could facilitate the completion of a greater number of tasks.

Linear leaky-integrate-and-fire neuron model based spiking neural networks and its mapping relationship to deep neural networks.

Spiking neural networks (SNNs) are brain-inspired machine learning algorithms with merits such as biological plausibility and unsupervised learning capability. Previous works have shown that converting Artificial Neural Networks (ANNs) into SNNs is a practical and efficient approach for implementing an SNN. However, the basic principle and theoretical groundwork are lacking for training a non-accuracy-loss SNN. This paper establishes a precise mathematical mapping between the biological parameters of the Linear Leaky-Integrate-and-Fire model (LIF)/SNNs and the parameters of ReLU-AN/Deep Neural Networks (DNNs). Such mapping relationship is analytically proven under certain conditions and demonstrated by simulation and real data experiments. It can serve as the theoretical basis for the potential combination of the respective merits of the two categories of neural networks.

Deep anomaly detection in horizontal axis wind turbines using Graph Convolutional Autoencoders for Multivariate Time series

Wind power is one of the fastest-growing renewable energy sectors instrumental in the ongoing decarbonization process. However, wind turbines are subjected to a wide range of dynamic loads which can cause more frequent failures and downtime periods, leading to ever-increasing attention to effective Condition Monitoring strategies. In this paper, we propose a novel unsupervised deep anomaly detection framework to detect anomalies in wind turbines based on SCADA data. We introduce a promising neural architecture, namely a Graph Convolutional Autoencoder for Multivariate Time series, to model the sensor network as a dynamical functional graph. This structure improves the unsupervised learning capabilities of Autoencoders by considering individual sensor measurements together with the nonlinear correlations existing among signals. On this basis, we developed a deep anomaly detection framework that was validated on 12 failure events occurred during 20 months of operation of four wind turbines. The results show that the proposed framework successfully detects anomalies and anticipates SCADA alarms by outperforming other two recent neural approaches.

A Cortical Learning Machine for Learning Real-Valued and Ranked Data

The cortical learning machine (CLM) introduced in [1-3] is a low-order computational model of the neocortex. It has the real-time, photogragraphic, unsupervised, and hierarchical learning capabilities, which existing learning machines such as the multilayer perceptron and convolutional neural network do not have. The CLM is a network of processing units (PUs) each comprising novel computational models of dendrites (for encoding), synapses (for storing code covariance matrices), spiking/nonspiking somas (for evaluating empirical probabilities and generating spikes), and unsupervised/supervised Hebbian learning schemes. In this paper, the masking matrix in the CLM in [1-3] is generalized to enable the CLM to learn ranked and real-valued data in the form of the binary numbers and unary (thermometer) codes. The general masking matrix assigns weights to the bits in the binary and unary code to reflect their relative significances. Numerical examples are provided to illustrate that a single PU with the general masking matrix is a pattern recognizer with an efficacy comparable to those of leading statistical and machine learning methods, showing the potential of CLMs with multiple PUs especially in consideration of the aforementioned capabilities of the CLM.

Texture-based latent space disentanglement for enhancement of a training dataset for ANN-based classification of fruit and vegetables

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables.

A GPU based virtual screening tool using SOM

This paper attempts to introduce the applicability of low cost graphics processing unit alternatives to a virtual screening technique using a novel self-organising map (SOM) based technique. This method combines the unsupervised learning capability of the SOM with a subsequent supervised labelling of the trained SOM neurons for building the prediction model. This novel iteration-based SOM technique can label molecule as undefined classes which can reduce the false positives in the screening. For running large datasets, the serial implementation of the proposed algorithm is very time-consuming and cannot be completed in a stipulated time frame. This has been overcome by exploiting the parallelism present in finding the winner neuron and neuron weight updating steps. A tool named SOMSCREEN is developed based on the proposed parallelised method to make the drug discovery process faster. It is observed that, the proposed method offers reduced false positive rate than the Random Forest based work.

Rolling Bearing Sub-Health Recognition via Extreme Learning Machine Based on Deep Belief Network Optimized by Improved Fireworks

Rolling bearings, as the main components of the large industrial rotating equipment, usually work under complex conditions and are prone to break down. It can provide a certain theoretical basis for identifying the sub-health state of the industrial equipment by the analysis from the incipient weak signals. Thus, a sub-health recognition offline algorithm based on Refined Composite Multiscale Dispersion Entropy (RCMDE) and Deep Belief Network-Extreme Learning Machine (DBN-ELM) optimized by Improved Firework Algorithm (IFWA) is proposed. First of all, in light of the drawbacks that it is easy to fall into local optima and cross the boundary for exploding fireworks in Firework Algorithm (FWA), Cauchy mutation and adaptive dynamic explosion radius factor coefficient is introduced into IFWA. Secondly, Maximum Correlation Kurtosis Deconvolution (MCKD) optimized by the improved parameters is used to process the incipient vibration signals with nonlinearity, nonstationary, and IFWA is used to adaptively adjust to the period T and the filter length L in MCKD(IFWA-MCKD). Then, each sequence of signals is further extracted the feature-RCMDE to rich sample diversity. Finally, combining the powerful unsupervised learning capability from DBN and the generalization capability from ELM, DBN-ELM can be established. What's more, in order to avoid the interference of human on the parameters, IFWA is used to optimize the number of hidden nodes in DBN-ELM, and the IFWA-DBN-ELM is established. It shows that the algorithm has the higher sub-health recognition accuracy, better robustness and generalization, which has a better industrial application prospect.

An acoustic-homologous transfer learning approach for acoustic emission–based rail condition evaluation

This article presents a novel transfer learning approach for evaluating structural conditions of rail in a progressive manner, by using acoustic emission monitoring data and knowledge transferred from an acoustic-related database. Specifically, the low-level layers of a model pre-trained on large audio data are leveraged in our model for feature extraction. Compared with conventional transfer learning approaches that transfer knowledge from models pre-trained on normal images, the proposed approach can handle acoustic emission spectrograms more naturally in terms of both data inner properties and the way of data intaking. The training and testing data used for rail condition evaluation contains two months of acoustic emission recordings, which were acquired from an in situ operating rail turnout with an integrated acoustic emission –based monitoring system. Results show that the evolving stages of rail from intact to critically cracked are successfully revealed using the proposed approach with excellent prediction accuracy and high computation efficiency. More importantly, the study quantitatively shows that audio source data are more relevant to the acoustic emission monitoring data than Image data, by introducing a maximum mean discrepancy metric, and the transfer learning model with smaller maximum mean discrepancy does lead to better performance. As a by-product of the study, it has also been found that the features extracted by the proposed transfer learning model (“bottleneck” features) already exhibit a clustering trend corresponding to the evolving stages of rail conditions even in the training process before any label is annotated, indicating the potential unsupervised learning capability of the proposed approach. Through the study, it is suggested that selecting source data more correspondingly and flexibly would maximize the benefit of transfer learning in structural health monitoring area when facing heterogenous monitoring data under varying practical scenarios.

FSpiNN: An Optimization Framework for Memory-Efficient and Energy-Efficient Spiking Neural Networks

Spiking Neural Networks (SNNs) are gaining interest due to their event-driven processing which potentially consumes low power/energy computations in hardware platforms, while offering unsupervised learning capability due to the spike-timing-dependent plasticity (STDP) rule. However, state-of-the-art SNNs require a large memory footprint to achieve high accuracy, thereby making them difficult to be deployed on embedded systems, for instance on battery-powered mobile devices and IoT Edge nodes. Towards this, we propose FSpiNN, an optimization framework for obtaining memory- and energy-efficient SNNs for training and inference processing, with unsupervised learning capability while maintaining accuracy. It is achieved by (1) reducing the computational requirements of neuronal and STDP operations, (2) improving the accuracy of STDP-based learning, (3) compressing the SNN through a fixed-point quantization, and (4) incorporating the memory and energy requirements in the optimization process. FSpiNN reduces the computational requirements by reducing the number of neuronal operations, the STDP-based synaptic weight updates, and the STDP complexity. To improve the accuracy of learning, FSpiNN employs timestep-based synaptic weight updates, and adaptively determines the STDP potentiation factor and the effective inhibition strength. The experimental results show that, as compared to the state-of-the-art work, FSpiNN achieves 7.5x memory saving, and improves the energy-efficiency by 3.5x on average for training and by 1.8x on average for inference, across MNIST and Fashion MNIST datasets, with no accuracy loss for a network with 4900 excitatory neurons, thereby enabling energy-efficient SNNs for edge devices/embedded systems.

A method to remove depositional background data based on the Modified Kernel Hebbian Algorithm

The seismic sedimentology is an emerging inter-discipline originating from the seismic stratigraphy and sequence stratigraphy. However, implementation of the seismic sedimentological research is found with high difficulties, due to influences imposed by structural and depositional background data (including strong reflections). In this paper, seismic records are regarded as a combination of the reflection from the depositional background and lithological data volumes, and moreover, the seismogram of the depositional background data is characterized by the low frequency and stable phase. Subsequently, the Kernel Hebbian Algorithm (KHA) has been modified to remove the influence of the depositional background data. The seismic trace data are used as the training set, and an innovative attempt has been made to incorporate the Ricker wavelet kernel function. Finally, a depositional background data volume extraction methodology with respect to input of higher-dimension seismic data has been developed, on the basis of the Modified KHA (MKHA), so as to obtain the lithological data volume. Utilizing the unsupervised online learning capabilities of the MKHA, iterative calculation of Kernel PCA can greatly reduce the computational complexity and can be adapted to big data problems. This paper introduces the Ricker wavelet kernel function to transform the original seismic data into the feature space through the inner-product operation, extract the non-linear features, and solve the problem that the seismic data of the original sample space is linearly inseparable. The seismic sedimentological analysis based on the lithological data volume that is able to reflect hidden sand bodies can achieve elaborate carving of the reservoir. The proposed method has been tested in Working Block A of the East-1 district in the Sulige gas field, the Ordos Basin, China. The case study demonstrates that the presented method is capable of efficiently removing the depositional background data, and making great contributions to improving accuracy of the seismic sedimentological analysis of the effective reservoir, with the help of higher-dimension seismic data.

Electroforming in Metal-Oxide Memristive Synapses.

Memristors have shown an extraordinary potential to emulate the plastic and dynamic electrical behaviors of biological synapses and have been already used to construct neuromorphic systems with in-memory computing and unsupervised learning capabilities; moreover, the small size and simple fabrication process of memristors make them ideal candidates for ultradense configurations. So far, the properties of memristive electronic synapses (i.e., potentiation/depression, relaxation, linearity) have been extensively analyzed by several groups. However, the dynamics of electroforming in memristive devices, which defines the position, size, shape, and chemical composition of the conductive nanofilaments across the device, has not been analyzed in depth. By applying ramped voltage stress (RVS), constant voltage stress (CVS), and pulsed voltage stress (PVS), we found that electroforming is highly affected by the biasing methods applied. We also found that the technique used to deposit the oxide, the chemical composition of the adjacent metal electrodes, and the polarity of the electrical stimuli applied have important effects on the dynamics of the electroforming process and in subsequent post-electroforming bipolar resistive switching. This work should be of interest to designers of memristive neuromorphic systems and could open the door for the implementation of new bioinspired functionalities into memristive neuromorphic systems.

Compact Graphene-Based Spiking Neural Network With Unsupervised Learning Capabilities

To fully unleash the potential of graphene-based devices for neuromorphic computing, we propose a graphene synapse and a graphene neuron that form together a basic Spiking Neural Network (SNN) unit, which can potentially be utilized to implement complex SNNs. Specifically, the proposed synapse enables two fundamental synaptic functionalities, i.e., Spike-Timing-Dependent Plasticity (STDP) and Long-Term Plasticity, and both Long-Term Potentiation (LTP) and Long-Term Depression (LTD) can be emulated with the same structure by properly adjusting its bias. The proposed neuron captures the essential Leaky Integrate and Fire spiking neuron behavior with post firing refractory interval. We demonstrate the proper operation of the graphene SNN unit by relying on a mixed simulation approach that embeds the high accuracy of atomistic level simulation of graphene structures conductance within the SPICE framework. Subsequently, we analyze the way graphene synaptic plasticity affects the behavior of a 2-layer SNN example consisting of 6 neurons and demonstrate that LTP significantly increases the number of firing events while LTD is diminishing them, as expected. To assess the plausibility of the graphene SNN reaction to input stimuli we simulate its behavior by means of both SPICE and NEST, a well established SNN simulation framework, and demonstrate that the obtained reactions, characterized in terms of total number of firing events and mean Inter-Spike Interval (ISI) length, are in close agreement, which clearly suggests that the proposed design exhibits a proper behavior. Further, we prove the unsupervised learning capabilities of the proposed design by considering a 2-layer SNN consisting of 30 neurons meant to recognize the characters “A,” “E,” “I,” “O,” and “U,” represented with a 5 by 5 black and white pixel matrix. The SPICE simulation results indicate that the graphene SNN is able to perform unsupervised character recognition associated learning and that its recognition ability is robust to input character variations. Finally, we note that our proposal results in a small real-estate footprint (max. 30 nm $^2$ are required by one graphene-based device) and operates at 200 mV supply voltage, which suggest its suitability for the design of large-scale energy-efficient computing systems.

Memristive crossbar arrays for brain-inspired computing.

With their working mechanisms based on ion migration, the switching dynamics and electrical behaviour of memristive devices resemble those of synapses and neurons, making these devices promising candidates for brain-inspired computing. Built into large-scale crossbar arrays to form neural networks, they perform efficient in-memory computing with massive parallelism by directly using physical laws. The dynamical interactions between artificial synapses and neurons equip the networks with both supervised and unsupervised learning capabilities. Moreover, their ability to interface with analogue signals from sensors without analogue/digital conversions reduces the processing time and energy overhead. Although numerous simulations have indicated the potential of these networks for brain-inspired computing, experimental implementation of large-scale memristive arrays is still in its infancy. This Review looks at the progress, challenges and possible solutions for efficient brain-inspired computation with memristive implementations, both as accelerators for deep learning and as building blocks for spiking neural networks.

Unsupervised Transfer Learning via Multi-Scale Convolutional Sparse Coding for Biomedical Applications.

The capabilities of (I) learning transferable knowledge across domains; and (II) fine-tuning the pre-learned base knowledge towards tasks with considerably smaller data scale are extremely important. Many of the existing transfer learning techniques are supervised approaches, among which deep learning has the demonstrated power of learning domain transferrable knowledge with large scale network trained on massive amounts of labeled data. However, in many biomedical tasks, both the data and the corresponding label can be very limited, where the unsupervised transfer learning capability is urgently needed. In this paper, we proposed a novel multi-scale convolutional sparse coding (MSCSC) method, that (I) automatically learns filter banks at different scales in a joint fashion with enforced scale-specificity of learned patterns; and (II) provides an unsupervised solution for learning transferable base knowledge and fine-tuning it towards target tasks. Extensive experimental evaluation of MSCSC demonstrates the effectiveness of the proposed MSCSC in both regular and transfer learning tasks in various biomedical domains.

A hybrid machine learning approach to network anomaly detection

Zero-day cyber attacks such as worms and spy-ware are becoming increasingly widespread and dangerous. The existing signature-based intrusion detection mechanisms are often not sufficient in detecting these types of attacks. As a result, anomaly intrusion detection methods have been developed to cope with such attacks. Among the variety of anomaly detection approaches, the Support Vector Machine (SVM) is known to be one of the best machine learning algorithms to classify abnormal behaviors. The soft-margin SVM is one of the well-known basic SVM methods using supervised learning. However, it is not appropriate to use the soft-margin SVM method for detecting novel attacks in Internet traffic since it requires pre-acquired learning information for supervised learning procedure. Such pre-acquired learning information is divided into normal and attack traffic with labels separately. Furthermore, we apply the one-class SVM approach using unsupervised learning for detecting anomalies. This means one-class SVM does not require the labeled information. However, there is downside to using one-class SVM: it is difficult to use the one-class SVM in the real world, due to its high false positive rate. In this paper, we propose a new SVM approach, named Enhanced SVM, which combines these two methods in order to provide unsupervised learning and low false alarm capability, similar to that of a supervised SVM approach. We use the following additional techniques to improve the performance of the proposed approach (referred to as Anomaly Detector using Enhanced SVM): First, we create a profile of normal packets using Self-Organized Feature Map (SOFM), for SVM learning without pre-existing knowledge. Second, we use a packet filtering scheme based on Passive TCP/IP Fingerprinting (PTF), in order to reject incomplete network traffic that either violates the TCP/IP standard or generation policy inside of well-known platforms. Third, a feature selection technique using a Genetic Algorithm (GA) is used for extracting optimized information from raw internet packets. Fourth, we use the flow of packets based on temporal relationships during data preprocessing, for considering the temporal relationships among the inputs used in SVM learning. Lastly, we demonstrate the effectiveness of the Enhanced SVM approach using the above-mentioned techniques, such as SOFM, PTF, and GA on MIT Lincoln Lab datasets, and a live dataset captured from a real network. The experimental results are verified by m-fold cross validation, and the proposed approach is compared with real world Network Intrusion Detection Systems (NIDS).

On-line monitoring of tool breakage in face milling using a self-organized neural network

This study introduces a new tool breakage monitoring methodology consisting of an unsupervised neural network combined with an adaptive time-series modeling algorithm. Cutting force signals are modeled by a discrete autoregressive model in which parameters are estimated recursively at each sampling instant using a parameter-adaptation algorithm based on a recursive least square. The experiment shows that monitoring the evolution of autoregressive parameters during milling is effective for detecting tool breakage. An adaptive resonance network based on Grossberg's adaptive resonance theory (ART 2) is employed for clustering tool states using model parameters, and this network has unsupervised learning capability. This system subsequently operates successfully with a fast monitoring time in a wide range of cutting conditions without a priori knowledge of the cutting process.

Towards an Intelligent Communication Network Routing System

Abstract Neural network information processing forms the basis of a novel Intelligent Communication Network Routing(ICNR) system designed to implement optimal routing in a data communication network. The main function of the ICNR system is similar to current optimization routing algorithms with two additional innovative features. First, the relevant communications network routing “state” information necessary for determining optimal routing paths, is acquired by using a neural network with unsupervised learning capability. This neural network operates on a history of incomplete state information by performing clustering of states to estimate the complete network state vector. Secondly, the routing decisions made are learned by a neural network with a supervised learning rule which adapts to acquire the mapping between the incomplete network state vector and the optimal routing assignment and generalizes the mapping to provide sub-optimal solutions for new cases. This paper demonstrates the feasibility of using a neural network with unsupervised learning capability to estimate the communication network traffic congestion conditions necessary to implement effective routing algorithms.