Self-organizing Incremental Neural Network Research Articles

SOINN Intrusion Detection Model Based on Three-Way Attribute Reduction

With a large number of intrusion detection datasets and high feature dimensionality, the emergent nature of new attack types makes it impossible to collect network traffic data all at once. The modified three-way attribute reduction method is combined with a Self-Organizing Incremental learning Neural Network (SOINN) algorithm to propose a self-organizing incremental neural network intrusion detection model based on three-way attribute reduction. Attribute importance is used to perform attribute reduction, and the data after attribute reduction are fed into a self-organized incremental learning neural network algorithm, which generalizes the topology of the original data through self-organized competitive learning. When the streaming data are transferred into the model, the inter-class insertion or node fusion operation is performed by comparing the inter-node distance and similarity threshold to achieve incremental learning of the model streaming data. The inter-node distance value is introduced into the weight update formulation to replace the traditional learning rate and to optimize the topological structure adjustment operation. The experimental results show that T-SOINN achieves high precision and recall when processing intrusion detection data.

Scalable cognitive developmental network: A strategy for integrating new perception online using relation evolution SOINN

Cognitive development can make robots more intelligent. However, current cognitive robots are still limited to learning fixed types of information. To this end, we propose a scalable cognitive developmental network (SCDN) based on self-organizing incremental neural network (SOINN), which allows new types of perception information to be added into the existing cognitive network online. SCDN has three hierarchical layers for developing concrete and abstract knowledge. Firstly, the sample layer learns the sample features of four modal information, including shape, color, name and taste. Secondly, the symbol layer generates respective symbol representations. Finally, the associative layer realizes the fusion of new perception information and native networks. Specifically, the Relation Evolution SOINN (RE-SOINN) is proposed in the associative layer to add new information into existing relationships, which contributes to the evolution of associative relationships. We evaluate the performance of SCDN on a fruits and vegetables dataset. Experimental results show that SCDN can effectively learn unknown sensory information online and form the correct link with the known information without learning from scratch.

A self-organizing incremental neural network for imbalance learning

A self-organizing incremental neural network for imbalance learning

SOINN-Based Abnormal Trajectory Detection for Efficient Video Condensation

With the evolution of video surveillance systems, the requirement of video storage grows rapidly; in addition, safe guards and forensic officers spend a great deal of time observing surveillance videos to find abnormal events. As most of the scene in the surveillance video are redundant and contains no information needs attention, we propose a video condensation method to summarize the abnormal events in the video by rearranging the moving trajectory and sort them by the degree of anomaly. Our goal is to improve the condensation rate to reduce more storage size, and increase the accuracy in abnormal detection. As the trajectory feature is the key to both goals, in this paper, a new method for feature extraction of moving object trajectory is proposed, and we use the SOINN (Self-Organizing Incremental Neural Network) method to accomplish a high accuracy abnormal detection. In the results, our method is able to shirk the video size to 10% storage size of the original video, and achieves 95% accuracy of abnormal event detection, which shows our method is useful and applicable to the surveillance industry.

A Continuous Learning Approach for Real-Time Network Intrusion Detection.

Network intrusion detection is becoming a challenging task with cyberattacks that are becoming more and more sophisticated. Failing the prevention or detection of such intrusions might have serious consequences. Machine learning approaches try to recognize network connection patterns to classify unseen and known intrusions but also require periodic re-training to keep the performances at a high level. In this paper, a novel continuous learning intrusion detection system, called Soft-Forgetting Self-Organizing Incremental Neural Network (SF-SOINN), is introduced. SF-SOINN, besides providing continuous learning capabilities, is able to perform fast classification, is robust to noise, and it obtains good performances with respect to the existing approaches. The main characteristic of SF-SOINN is the ability to remove nodes from the neural network based on their utility estimate. SF-SOINN has been validated on the well-known NSL-KDD and CIC-IDS-2017 intrusion detection datasets as well as on some artificial data to show the classification capability on more general tasks.

Automated Construction of Neural Network Potential Energy Surface: The Enhanced Self-Organizing Incremental Neural Network Deep Potential Method.

In recent years, the use of deep learning (neural network) potential energy surface (NNPES) in molecular dynamics simulation has experienced explosive growth as it can be as accurate as quantum chemistry methods while being as efficient as classical mechanic methods. However, the development of NNPES is highly nontrivial. In particular, it has been troubling to construct a dataset that is as small as possible yet can cover the target chemical space. In this work, an ESOINN-DP method is developed, which has the enhanced self-organizing incremental neural network (ESOINN) and a newly proposed error indicator at its core. With ESOINN-DP, one can construct the NNPES with little human intervention, and this method ensures that the constructed reference dataset covers the target chemical space with minimum redundancy. The performance of the ESOINN-DP method has been well validated by developing neural network potential energy surfaces for water clusters, tripeptides, and by de-redundancy of a sub-dataset of the ANI-1 database. We believe that the ESOINN-DP method provides a novel idea for the construction of NNPES and, especially, the reference datasets, and it can be used for molecular dynamics (MD) simulations of various gas-phase and condensed-phase chemical systems.

A self-organizing incremental neural network for continual supervised learning

Continual learning algorithms can adapt to changes of data distributions, new classes, and even completely new tasks without catastrophically forgetting previously acquired knowledge. Here, we present a novel self-organizing incremental neural network, GSOINN+, for continual supervised learning. GSOINN+ learns a topological mapping of the input data to an undirected network and uses a weighted nearest-neighbor rule with fractional distance for classification. GSOINN+ learns incrementally—new classification tasks do not need to be specified a priori, and no rehearsal of previously learned tasks with stored training sets is required. In a series of sequential learning experiments, we show that GSOINN+ can mitigate catastrophic forgetting, even when completely new tasks are to be learned.

Learning Then, Learning Now, and Every Second in Between: Lifelong Learning With a Simulated Humanoid Robot.

Long-term human-robot interaction requires the continuous acquisition of knowledge. This ability is referred to as lifelong learning (LL). LL is a long-standing challenge in machine learning due to catastrophic forgetting, which states that continuously learning from novel experiences leads to a decrease in the performance of previously acquired knowledge. Two recently published LL approaches are the Growing Dual-Memory (GDM) and the Self-organizing Incremental Neural Network+ (SOINN+). Both are growing neural networks that create new neurons in response to novel sensory experiences. The latter approach shows state-of-the-art clustering performance on sequentially available data with low memory requirements regarding the number of nodes. However, classification capabilities are not investigated. Two novel contributions are made in our research paper: (I) An extended SOINN+ approach, called associative SOINN+ (A-SOINN+), is proposed. It adopts two main properties of the GDM model to facilitate classification. (II) A new LL object recognition dataset (v-NICO-World-LL) is presented. It is recorded in a nearly photorealistic virtual environment, where a virtual humanoid robot manipulates 100 different objects belonging to 10 classes. Real-world and artificially created background images, grouped into four different complexity levels, are utilized. The A-SOINN+ reaches similar state-of-the-art classification accuracy results as the best GDM architecture of this work and consists of 30 to 350 times fewer neurons, evaluated on two LL object recognition datasets, the novel v-NICO-World-LL and the well-known CORe50. Furthermore, we observe an approximately 268 times lower training time. These reduced numbers result in lower memory and computational requirements, indicating higher suitability for autonomous social robots with low computational resources to facilitate a more efficient LL during long-term human-robot interactions.

An improved self-organizing incremental neural network model for short-term time-series load prediction

Load prediction is a crucial component for optimal building energy management. The challenge with buildings’ load prediction is the lack of historical data since not all sites have a large amount of collected data. One of the solutions is incremental learning that updates the trained model with the most recent data. It allows the model to be deployed as soon as possible and improving its prediction accuracy as time progress. This study studies the effect of incremental learning. This study also proposes a novel DB-SOINN-R model that is based on the enhanced self-organizing incremental neural network (ESOINN), which is one of the incremental learning models. The problems with ESOINN are the inappropriate node removal by the original denoising of ESOINN, the inappropriate Euclidean distance for training data with imbalanced dimensions that usually consist of less discrete timestamp data compared to time-series historical data, and the incapability to obtain unique predicted outputs. To tackle these problems, the DB-SOINN-R incorporates a new density-based denoising that replaces the original denoising, a new mean Euclidean distance as the distance metric to handle training data with imbalanced dimensions, and k-nearest-neighbor inverse distance weighting (kNN-IDW) regression to obtain unique predicted output for every different input. The proposed DB-SOINN-R is compared with five models: feedforward neural network, deep neural network with long-short-term memory, support vector regression, ESOINN, and kNN regression. They are tested on day-ahead and one-hour-ahead load predictions, using two different datasets. The proposed DB-SOINN-R has the highest prediction accuracy among all models with incremental learning in both datasets.

Autonomous cognition development with lifelong learning: A self-organizing and reflecting cognitive network

Lifelong learning is still a great challenge for cognitive robots since the continuous streaming data they encounter is usually enormous and no-stationary. Traditional cognitive methods suffer from large storage and computation consumption in this situation. Therefore, we propose a self-organizing and reflecting cognitive network (SORCN) to realize robotic lifelong cognitive development through incremental learning and regular reflecting. The network integrates a self-organizing incremental neural network (SOINN) with a modified CFS clustering algorithm. SOINN develops concise object concepts to alleviate storage consumption. Moreover, we modify SOINN by an efficient competitive method based on reflection results to reduce the learning computation. The modified CFS clustering algorithm is designed for reflecting knowledge learned by SOINN periodically. It improves the traditional CFS as a three-step clustering method including clustering, merging and splitting. Specifically, an autonomous center selection strategy is employed for CFS to cater to online learning. Moreover, a series of cluster merging and splitting strategies are proposed to enable CFS to cluster data incrementally and improve its clustering effect. Additionally, the reflection results are utilized to adjust the topological structure of SOINN and guide the future learning. Experimental results demonstrate that SORCN can achieve better learning effectiveness and efficiency over several state-of-art algorithms.

A Novel Algorithm for the Fault Diagnosis of a Redundant Inertial Measurement Unit

Although a number of fault diagnosis algorithms for inertial sensors have been proposed in previous decades, the performance of these algorithms needs to be improved with regard to small faults. In this paper, we introduce a data driven-based algorithm, namely, SaPD, for the anomaly detection and output reconstruction of a redundant inertial measurement unit (RIMU). SaPD implements the fault identification of an inertial apparatus by combining an artificial neural network with the Q contribution plots method in parity space. To improve the performance of the fault detection part, in particular for small faults, we introduce a novel hyperplane that measures the distances between inputs and the primary-neuron set obtained from a self-organizing incremental neural network (SOINN). We also employ the Q contribution plots of sensors in the fault isolation part by analyzing historical data with principal component analysis (PCA). We perform quantitative evaluations in a realistic simulation environment, which demonstrates that the proposed SaPD algorithm outperforms other related algorithms in terms of the fault identification accuracy of tiny faults with an acceptable computational complexity.

Online Topology Learning by a Gaussian Membership-Based Self-Organizing Incremental Neural Network.

In order to extract useful information from data streams, incremental learning has been introduced in more and more data mining algorithms. For instance, a self-organizing incremental neural network (SOINN) has been proposed to extract a topological structure that consists of one or more neural networks to closely reflect the data distribution of data streams. However, SOINN has the tradeoff between deleting previously learned nodes and inserting new nodes, i.e., the stability-plasticity dilemma. Therefore, it is not guaranteed that the topological structure obtained by the SOINN will closely represent data distribution. For solving the stability-plasticity dilemma, we propose a Gaussian membership-based SOINN (Gm-SOINN). Unlike other SOINN-based methods that allow only one node to be identified as a "winner" (the nearest node), the Gm-SOINN uses a Gaussian membership to indicate to which degree the node is a winner. Hence, the Gm-SOINN avoids the topological structure that cannot represent the data distribution because previously learned nodes overly deleted or noisy nodes inserted. In addition, an evolving Gaussian mixture model is integrated into the Gm-SOINN to estimate the density distribution of nodes, thereby avoiding the wrong connection between two nodes. Experiments involving both artificial and real-world data sets indicate that our proposed Gm-SOINN achieves better performance than other topology learning methods.

SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams

The goal of continuous learning is to acquire and fine-tune knowledge incrementally without erasing already existing knowledge. How to mitigate this erasure, known as catastrophic forgetting, is a grand challenge for machine learning, specifically when systems are trained on evolving data streams. Self-organizing incremental neural networks (SOINN) are a class of neural networks that are designed for continuous learning from non-stationary data. Here, we propose a novel method, SOINN+, which is fundamentally different from the previous generations of SOINN with respect to how “forgetting” is modeled. To achieve a more graceful forgetting, we developed three new concepts: idle time, trustworthiness, and unutility of a node. SOINN+ learns a topology-preserving mapping of the input data to a network structure, which reveals clusters of arbitrary shapes in streams of noisy data. Our experiments with synthetic and real-world data sets showed that SOINN+ can maintain discovered structures even under sudden and recurring concept drifts.

An interactive visual analytics approach for network anomaly detection through smart labeling

Network anomaly detection is an important means for safeguarding network security. On account of the difficulties encountered in traditional automatic detection methods such as lack of labeled data, expensive retraining costs for new data and non-explanation, we propose a novel smart labeling method, which combines active learning and visual interaction, to detect network anomalies through the iterative labeling process of the users. The algorithms and the visual interfaces are tightly integrated. The network behavior patterns are first learned by using the self-organizing incremental neural network. Then, the model uses a Fuzzy c-means-based algorithm to do classification on the basis of user feedback. After that, the visual interfaces are updated to present the improved results of the model, which can help users to choose meaningful candidates, judge anomalies and understand the model results. The experiments show that compared to labeling without our visualizations, our method can achieve a high accuracy rate of anomaly detection with fewer labeled samples.

Learning sampling distribution for motion planning with local reconstruction-based self-organizing incremental neural network

For sampling-based motion planners (e.g., PRM and RRT*), collision detection dominates the asymptotic running time and reduces the execution efficiency. The reason of this problem is that obtaining a high-dimensional implicit representation (i.e., configuration space distribution) of the state space is not easy, especially in the complicated environment with various obstacles. Though sampling-based planning algorithms and their variants perform well, most of these algorithms have strict restrictions and narrow applications. A possible ideal solution is to design a non-uniform sampling strategy to ensure the sampling process only occurs in collision-free region $$\chi _{free}$$ but not in collision region $$\chi _{col}$$ . Therefore, we propose a new methodology to learn the sampling distribution for non-uniform sampling. The sampling distribution is learned through a local reconstruction-based self-organizing incremental neural network and allows to generate samples from the learned latent distribution. Besides, our method can adapt well to environmental non-vigorous changes and adjust the learned distribution quickly. The method can effectively exploit the underlying structure of the planning problem and be spread for general use in combination with any sampling-based planning algorithms. Specifically, we use two typical planning problems to show that the proposed method can effectively learn and update the sampling distribution from the high-dimensional configuration space in the changed environment, resulting in a dominant performance in terms of the cost, running time and success rate.

Increasing the share of correct clustering of characteristic signal with random losses in self-organizing maps

Analysis of methods for optimizing algorithms of functioning of the Kohonen neural networks, self-organizing maps (SOM), in terms of training speed and percentage of correct clustering was made. Effective optimization of self-organizing maps was determined by the second criterion, the enhanced self-organizing incremental neural network (ESOINN). It was established that in the case of incomplete input signal, that is the signal with losses at unknown time points, the share of correct clustering is unacceptably low with any SOM algorithms, both basic and optimized. The incomplete signal was represented as the input vector of the neural network, the values of which are represented by a single array, that is without taking into consideration conformity of the moments of losses to the current values and without the possibility of determining these moments. A method for determining conformance of the incomplete input vector to the input layer of neurons to increase percentage of correct recognition was programmed and proposed. The method is based on finding the minimum distance between the current input vector and the vector of weights of each neuron. To reduce operating time of the algorithm, it was proposed to operate not with individual values of the input signal but their indivisible parts and the corresponding groups of input neurons. The proposed method was implemented for the SOM and ESOINN. To prove effectiveness of implementation of the basic algorithm of the SOM, its comparison with existing counterparts of other developers was made. A mathematical model was developed for formation of examples of complete signals of a training sample on the basis of reference curves of the second order and a training sample was generated. In accordance with the training sample, training of all neural networks implemented with and without the proposed method was made. A diagram of simulation of losses was developed and test samples were generated for computational experiments with incomplete signals. On the basis of experiments, efficiency of the proposed method for classification in terms of incomplete input signal on the basis of self-organizing maps was proved both for implementations of the basic algorithm of SOM and ESOINN

Semi-parametric training of autoencoders with Gaussian kernel smoothed topology learning neural networks

Autoencoders are essential for training multi-hidden layer neural networks. Parametric autoencoder trainings often require user selections of hidden neuron numbers and kernel types. In this paper, a semi-parametric autoencoder training method based on self-organized learning and incremental learning is proposed. The cost function is constructed incrementally by nonparametric learning, and the model parameter is trained by parametric learning. First, a topology learning neural network such as growing neural gas or self-organizing incremental neural network is trained to obtain a discrete representation of the training data. Second, the correlations between different dimensions are modeled as a joint distribution by the neural network representation and kernel smoothers. Finally, the loss function is defined to be the regression prediction errors with each dimension as a response variable in density regression. The parameter of kernels is selected by gradient descent which minimizes the reconstruction error on a data subset. The proposed architecture has the advantage of high training space efficiency because of incremental training, and the advantage of automated selection of hidden neuron numbers. Experiments are carried out on 4 UCI datasets and an image interpolation task. Results show that the proposed methods outperform the perceptron architecture autoencoders and the restricted Boltzmann machine in the task of nonlinear feature learning.

Simultaneous Forecasting of Meteorological Data Based on a Self-Organizing Incremental Neural Network

In this study, we propose a simultaneous forecasting model for meteorological time-series data based on a self-organizing incremental neural network (SOINN). Meteorological parameters (i.e., temperature, wet bulb temperature, humidity, wind speed, atmospheric pressure, and total solar radiation on a horizontal surface) are considered as input data for the prediction of meteorological time-series information. Based on a SOINN within normalized-refined-meteorological data, proposed model succeeded forecasting temperature, humidity, wind speed and atmospheric pressure simultaneously. In addition, proposed model does not take more than 2 s in training half-year period and 15 s in testing half-year period. This paper also elucidates the SOINN and the algorithm of the learning process. The effectiveness of our model is established by comparison of our results with experimental results and with results obtained by another model. Three advantages of our model are also described. The obtained information can be effective in applications based on neural networks, and the proposed model for handling meteorological phenomena may be helpful for other studies worldwide including energy management system.

An intelligent real-time power management system with active learning prediction engine for PV grid-tied systems

In this paper, an incremental unsupervised neural network algorithm namely memory self-organizing incremental neural network (M-SOINN) is proposed to first predict the output power and subsequently detect the occurrence of power fluctuation events in a photovoltaic microgrid system. The M-SOINN uses clustering technique to form the data map, identifies the most similar patterns to predict the photovoltaic output power and then detects power fluctuation events. A novel memory layer is incorporated to establish the time-series learning. By using real life environment data, the proposed M-SOINN based real-time prediction engine detects power fluctuation events with a high detection rate of 92.69% and it outperforms the conventional self-organizing map (SOM), k-nearest neighbour (KNN), focused time delay neural network (FTDNN), and nonlinear autoregressive with exogenous input (NARX) networks. The system is simulated in the PSCAD environment and later experimentally. The proposed M-SOINN is then integrated into a power management system to mitigate power fluctuation events of the photovoltaic grid-tied system. Results show that the proposed power management system reduces 79.62% of power fluctuation events with an energy loss of 2.16% and battery state-of-charge maintains within 30%–100%. The proposed system outperforms hourly rule-based controller and the ramp rate controller by 44.02% and 27.57%, respectively in terms of the mitigated power fluctuation events.

Time Series Prediction of Tropical Storm Trajectory Using Self-Organizing Incremental Neural Networks and Error Evaluation

This study proposes a route prediction method using a self-organizing incremental neural network. The route trajectory is predicted from two location parameters (the latitude and longitude of the middle of a tropical storm) and the meteorological information (the atmospheric pressure). The method accurately predicted the normalized atmospheric pressure data of East Asia in the topological space of latitude and longitude, with low calculation cost. This paper explains the algorithms for training the self-organizing incremental neural network, the procedure for refining the datasets and the method for predicting the storm trajectory. The effectiveness of the proposed method was confirmed in experiments. With the results of experiments, possibility of prediction model improvement is discussed. Additionally, this paper explains the limitations of proposed method and brief solution to resolve. Although the proposed method was applied only to typhoon phenomena in the present study, it is potentially applicable to a wide range of global problems.