Constructive Learning Algorithm Research Articles

IMPLICATIONS OF COMPUTE AND INTERPRETABILITY FOR THE DIFFERENT SDN SECURITY MODELS

The main focus of this project is to design and implement an IDS for SDNs that are currently causing great interest in the networking society. Making the control plane and the data plane modularity, SDNs provide a higher level of flexibility and programming capabilities to manage the network but concurrently budding novel security risks. Several machine learning algorithms are in use such as the Logistic Regression, Decision Tree, Random Forest, etc. , which are used to identify malice within the network. As for the dataset sdn_intrusion. csv, preprocessing steps included were cleaning where missing values were managed, categorical data was encoded as well as handling for outliers existed. This paper applied feature selection by analysing correlation between the input features and the target variable with the aim of enhancing the model accuracy. The models were assessed with cross validation and several metrics like accuracy, Precision, Recall, F1 score, AUC and the results are quantitatively and qualitatively presented in the form of confusion matrices, ROC and Precision Recall curves. As the findings indicate, Extended Trees (Extra Trees Classifier) and Histograms Gradient Boosting models are efficient in identifying intrusions within SDN architectures. In conclusion, the study highlights the necessity of the proper choice of the machine learning algorithm and preliminary data processing for the effective IDS construction in SDNs, which enhances the readiness level of the networks against cyber threats. KEYWORDS: Software Defined Networks, Intrusion Detection System, Machine Learning, Network Security, Logistic Regression, Extra Trees Classifier, Random Forest, LightGBM, Precision-Recall Curve, ROC Curve.

Conditional mean embedding and optimal feature selection via positive definite kernels

Motivated by applications, we consider new operator-theoretic approaches to conditional mean embedding (CME). Our present results combine a spectral analysis-based optimization scheme with the use of kernels, stochastic processes, and constructive learning algorithms. For initially given non-linear data, we consider optimization-based feature selections. This entails the use of convex sets of kernels in a construction o foptimal feature selection via regression algorithms from learning models. Thus, with initial inputs of training data (for a suitable learning algorithm), each choice of a kernel \(K\) in turn yields a variety of Hilbert spaces and realizations of features. A novel aspect of our work is the inclusion of a secondary optimization process over a specified convex set of positive definite kernels, resulting in the determination of "optimal" feature representations.

Hierarchical integrated machine learning model for predicting flight departure delays and duration in series

Flight delays may propagate through the entire aviation network and are becoming an important research topic. This paper proposes a novel hierarchical integrated machine learning model for predicting flight departure delays and duration in series rather than in parallel to avoid ambiguity in decision making. The paper analyses the proposed model using various machine learning algorithms in combination with different sampling techniques. The highly noisy, unbalanced, dispersed, and skewed historical high dimensional data provided by an international airline operating in Hong Kong was used to demonstrate the practical application of the model. The result shows that for a 4-h forecast horizon, a constructive neural network machine learning algorithm with the Synthetic Minority Over Sampling Technique-Tomek Links (SMOTETomek) sampling technique was able to achieve better average balanced recall accuracies of 65.5%, 61.5%, 59% for classifying delay status and predicting delay duration at thresholds of 60 min and 30 min, respectively. Similarly, for minority labels, the precision-recall and area under the curve showed that the proposed model achieved better results of 32.44% and 35.14% compared to the parallel model of 26.43% and 21.02% for thresholds of 60 min and 30 min, respectively. The effect of different sampling techniques, sampling approaches, and estimation mechanisms on prediction performance is also studied.

Efficient design of multicolumn RBF networks

To make radial basis function (RBF) networks efficient for large-scale learning tasks, the parallel technique provides a promising way for the construction of multicolumn RBF network (MCRN), where the task to be tackled is decomposed into several subtasks and RBF networks designed for these subtasks are integrated as a parallel structure to obtain the final model. This paper focuses on presenting an efficient design method for MCRN based on improved generalized hybrid constructive (GHC) learning algorithm such that desired performance and high efficiency are guaranteed. Specifically, a minimum difference scheme is firstly introduced to divide a large-scale dataset into several parts with similar distribution. For each subtask, a compact RBF network is designed by improved GHC learning algorithm. By fully considering the inherent properties of RBF networks, a K-nearest potential centers based criterion is established to calculate the output of testing sample. Experiments on some benchmark classification problems are conducted to verify the advantages of the proposed method. In terms of training and testing efficiencies and classification accuracy, it is shown that our model is superior over some existing ones.

A deep machine learning algorithm for construction of the Kolmogorov–Arnold representation

The Kolmogorov–Arnold representation is a proven adequate replacement of a continuous multivariate function by a hierarchical structure of multiple functions of one variable. The proven existence of such representation inspired many researchers to search for a practical way of its construction, since such model answers the needs of machine learning. This article shows that the Kolmogorov–Arnold representation is not only a composition of functions but also a particular case of a tree of the discrete Urysohn operators. The article introduces new, quick and computationally stable algorithm for constructing of such Urysohn trees. Besides continuous multivariate functions, the suggested algorithm covers the cases with quantised inputs and combination of quantised and continuous inputs. The article also contains multiple results of testing of the suggested algorithm on publicly available datasets, used also by other researchers for benchmarking.

Local Sigmoid Method: Non-Iterative Deterministic Learning Algorithm for Automatic Model Construction of Neural Network

A non-iterative learning algorithm for artificial neural networks is an alternative to optimize the neural network parameters with extremely fast convergence time. Extreme learning machine (ELM) is one of the fastest learning algorithms based on a non-iterative method for a single hidden layer feedforward neural network (SLFN) model. ELM uses a randomization technique that requires a large number of hidden nodes to achieve the high accuracy. This leads to a large and complex model, which is slow at the inference time. Previously, we reported analytical incremental learning (AIL) algorithm, which is a compact model and a non-iterative deterministic learning algorithm, to be used as an alternative. However, AIL cannot grow its set of hidden nodes, due to the node saturation problem. Here, we describe a local sigmoid method (LSM) that is also a sufficiently compact model and a non-iterative deterministic learning algorithm to overcome both the ELM randomization and AIL node saturation problems. The LSM algorithm is based on “divide and conquer” method that divides the dataset into several subsets which are easier to optimize separately. Each subset can be associated with a local segment represented as a hidden node that preserves local information of the subset. This technique helps us to understand the function of each hidden node of the network built. Moreover, we can use such a technique to explain the function of hidden nodes learned by backpropagation, the iterative algorithm. Based on our experimental results, LSM is more accurate than other non-iterative learning algorithms and one of the most compact models.

Integrating Classical Control into Reinforcement Learning Policy

Deep reinforcement learning has made impressive advances in sequential decision making problems recently. Constructive reinforcement learning (RL) algorithms have been proposed to focus on the policy optimization process, while further research on the effect of different policy network has not been fully explored. MLPs, LSTMs and linear layer are complementary in their controlling capabilities, as MLPs are appropriate for global control, LSTMs are able to exploit history information and linear layer is good at stabilizing system dynamics. In this paper, we propose a “Proportional-Integral” (PI) neural network architecture that could be easily combined with popular optimization algorithms. This PI-patterned policy network exploits the advantages of integral control and linear control that are widely applied in classic control systems, based on which an ensemble-learning-based model is trained to further improve the sample efficiency and training performance on most RL tasks. Experimental results on public RL simulation platforms demonstrate the proposed architecture could achieve better performance than generally used MLP and other existing applied models.

Brain-Inspired Constructive Learning Algorithms with Evolutionally Additive Nonlinear Neurons

In this article, inspired partially by the physiological evidence of brain’s growth and development, we developed a new type of constructive learning algorithm with evolutionally additive nonlinear neurons. The new algorithms have remarkable ability in effective regression and accurate classification. In particular, the algorithms are able to sustain a certain reduction of the loss function when the dynamics of the trained network are bogged down in the vicinity of the local minima. The algorithm augments the neural network by adding only a few connections as well as neurons whose activation functions are nonlinear, nonmonotonic, and self-adapted to the dynamics of the loss functions. Indeed, we analytically demonstrate the reduction dynamics of the algorithm for different problems, and further modify the algorithms so as to obtain an improved generalization capability for the augmented neural networks. Finally, through comparing with the classical algorithm and architecture for neural network construction, we show that our constructive learning algorithms as well as their modified versions have better performances, such as faster training speed and smaller network size, on several representative benchmark datasets including the MNIST dataset for handwriting digits.

Generalized Hybrid Constructive Learning Algorithm for Multioutput RBF Networks.

An efficient generalized hybrid constructive (GHC) learning algorithm for multioutput radial basis function (RBF) networks is proposed to obtain a compact network with good generalization capability. By this algorithm, one can train the adjustable parameters and determine the optimal network structure simultaneously. First, an initialization method based on the growing and pruning algorithm is utilized to select the important initial hidden neurons and candidate ones. Then, by introducing a generalized hidden matrix, a structured parameter optimization algorithm is presented to train multioutput RBF network with fixed size, which combines Levenberg-Marquardt (LM) algorithm with least-square method together. Beginning from an appropriate number of hidden neurons, new neurons chosen from the candidates are added one by one each time when the training entraps into local minima. By incorporating an improved incremental constructive scheme, the training is built on previous results after adding new neurons such that the GHC learning algorithm avoids a trial-and-error procedure. Furthermore, based on the improved computation for LM training, the memory limitation problem is solved. The computational complexity analysis and experimental results demonstrate that better performance is efficiently achieved by this algorithm.

SONFIS: Structure Identification and Modeling with a Self-Organizing Neuro-Fuzzy Inference System

AbstractThis paper presents a new adaptive learning algorithm to automatically design a neural fuzzy model. This constructive learning algorithm attempts to identify the structure of the model based on an architectural self-organization mechanism with a data-driven approach. The proposed training algorithm self-organizes the model with intuitive adding, merging and splitting operations. Sub-networks compete to learn specific training patterns and, to accomplish this task, the algorithm can either add new neurons, merge correlated ones or split existing ones with unsatisfactory performance. The proposed algorithm does not use a clustering method to partition the input-space like most of the state of the art algorithms. The proposed approach has been tested on well-known synthetic and real-world benchmark datasets. The experimental results show that our proposal is able to find the most suitable architecture with better results compared with those obtained with other methods from the literature.

Learning Assumptions for CompositionalVerification of Timed Systems

Compositional techniques such as assume-guarantee reasoning (AGR) can help to alleviate the state space explosion problem associated with model checking. However, compositional verification is difficult to be automated, especially for timed systems, because constructing appropriate assumptions for AGR usually requires human creativity and experience. To automate compositional verification of timed systems, we propose a compositional verification framework using a learning algorithm for automatic construction of timed assumptions for AGR. We prove the correctness and termination of the proposed learning-based framework, and experimental results show that our method performs significantly better than traditional monolithic timed model checking.

A convergent algorithm for a cascade network of multiplexed dual outputdiscrete perceptrons for linearly nonseparable classification

In this paper a new discrete perceptron model is introduced. The model forms a cascade structure and it is capable of realizing an arbitrary classification task designed by a constructive learning algorithm. The main idea is to copy a discrete perceptron neuron's output to have a complementary dual output for the neuron, and then to select, by using a multiplexer, the true output, which might be 0 or 1 depending on the given input. Hence, the problem of realization of the desired output is transformed into the realization of the selector signal of the multiplexer. In the next step, the selector signal is taken as the desired output signal for the remaining part of the network. The repeated applications of the procedure render the problem into a linearly separable one and eliminate the necessity of using the selector signal in the last step of the algorithm. The proposed modification to the discrete perceptron brings universality with the expense of getting just a slight modification in hardware implementation.

Border Pairs Method—constructive MLP learning classification algorithm

In this paper we present the Border Pairs Method, a constructive learning algorithm for multilayer perceptron (MLP). During learning with this method a near-minimal network architecture is found. MLP learning is conducted separately by individual layers and neurons. The algorithm is tested in computer simulation with simple learning patterns (XOR and triangles image), with traditional learning patterns (Iris and Pen-Based Recognition of Handwritten Digits) and with noisy learning patterns. During the learning process we observed the following behaviour of BPM: capability to focus on global minima, good generalisation, no problems in learning with noisy, multi-dimensional and numerous learning patterns. The Border Pairs Method also supports incremental and online learning. Both are realized with or without MLP reconstruction and with or without forgetting (unlearning). The learning results with the BPM method are comparable with results from other methods.

TOOL CONDITION MONITORING USING ACOUSTIC EMISSION, SURFACE ROUGHNESS AND GROWING CELL STRUCTURES NEURAL NETWORK

The monitoring of tool wear is a most difficult task in the case of various metal-cutting processes. Artificial Neural Networks (ANN) has been used to estimate or classify certain wear parameters, using continuous acquisition of signals from multi-sensor systems. Most of the research has been concentrated on the use of supervised neural network types like multi-layer perceptron (MLP), using back-propagation algorithm and Radial Basis Function (RBF) network. In this article, a new constructive learning algorithm proposed by Fritzke, namely Growing Cell Structures (GCS) has been used for tool wear estimation in face milling operations, thereby monitoring the condition of the tool. GCS generates compact network architecture in less training time and performs well on new untrained data. The performance of this network has been compared with that of another constructive learning algorithm-based neural network, namely the Resource Allocation Network (RAN). For the sake of establishing the effectiveness of GCS, results obtained have been compared with those obtained using Multi Layer Perceptron (MLP), which is a standard and widely used neural network.

Morphological perceptrons with competitive learning: Lattice-theoretical framework and constructive learning algorithm

A morphological neural network is generally defined as a type of artificial neural network that performs an elementary operation of mathematical morphology at every node, possibly followed by the application of an activation function. The underlying framework of mathematical morphology can be found in lattice theory. With the advent of granular computing, lattice-based neurocomputing models such as morphological neural networks and fuzzy lattice neurocomputing models are becoming increasingly important since many information granules such as fuzzy sets and their extensions, intervals, and rough sets are lattice ordered. In this paper, we present the lattice-theoretical background and the learning algorithms for morphological perceptrons with competitive learning which arise by incorporating a winner-take-all output layer into the original morphological perceptron model. Several well-known classification problems that are available on the internet are used to compare our new model with a range of classifiers such as conventional multi-layer perceptrons, fuzzy lattice neurocomputing models, k-nearest neighbors, and decision trees.

New Constructive Neural Network Architecture for Pattern Classification

Problem statement: Constructive neural network learning algorithms provide optimal ways to determine the architecture of a multi layer perceptron network along with learning algorithms for determining appropriate weights for pattern classification problems. These algorithms initially start with small network and dynamically allow the network to grow by adding and training neurons as needed until a satisfactory solution is found. The constructive neural network training is performed via feed forward paradigm under supervised training considerations. These supervised methods often make the network size grow exponentially, or, the network lacks generalization. To address these problems a new method for learning in constructive neural networks is necessary. Approach: To address these issues a new Multicategory Tiling architecture was chosen for its simple topology and an improved adaptive resonance theory unsupervised training algorithm was used with proper weight setting to train the constructive networks on binary sequence patterns. The results and performance of the new algorithm was compared with existing constructive neural network architectures and tabulated. Results: The new architecture with improved training algorithm offer faster convergence in learning, the nodes required for storage are less and the generalization of pattern classification was achieved in comparison with existing algorithms. Conclusion: Constructive neural networks could be trained using unsupervised algorithm to achieve better performance in comparison with existing supervised algorithms.

Gaussian kernel approximation algorithm for feedforward neural network design

A Gaussian kernel approximation algorithm for a feedforward neural network is presented. The approach used by the algorithm, which is based on a constructive learning algorithm, is to create the hidden units directly so that automatic design of the architecture of neural networks can be carried out. The algorithm is defined using the linear summation of input patterns and their randomized input weights. Hidden-layer nodes are defined so as to partition the input space into homogeneous regions, where each region contains patterns belonging to the same class. The largest region is used to define the center of the corresponding Gaussian hidden nodes. The algorithm is tested on three benchmark data sets of different dimensionality and sample sizes to compare the approach presented here with other algorithms. Real medical diagnoses and a biological classification of mushrooms are used to illustrate the performance of the algorithm. These results confirm the effectiveness of the proposed algorithm.

Long-term prediction of solar and geomagnetic activity daily time series using singular spectrum analysis and fuzzy descriptor models

Abstract Of the various conditions that affect space weather, Sun-driven phenomena are the most dominant. Cyclic solar activity has a significant effect on the Earth, its climate, satellites, and space missions. In recent years, space weather hazards have become a major area of investigation, especially due to the advent of satellite technology. As such, the design of reliable alerting and warning systems is of utmost importance, and international collaboration is needed to develop accurate short-term and long-term prediction methodologies. Several methods have been proposed and implemented for the prediction of solar and geomagnetic activity indices, but problems in predicting the exact time and magnitude of such catastrophic events still remain. There are, however, descriptor systems that describe a wider class of systems, including physical models and non-dynamic constraints. It is well known that the descriptor system is much tighter than the state-space expression for representing real independent parametric perturbations. In addition, the fuzzy descriptor models as a generalization of the locally linear neurofuzzy models are general forms that can be trained by constructive intuitive learning algorithms. Here, we propose a combined model based on fuzzy descriptor models and singular spectrum analysis (SSA) (FD/SSA) to forecast a number of geomagnetic activity indices in a manner that optimizes a fuzzy descriptor model for each of the principal components obtained from singular spectrum analysis and recombines the predicted values so as to transform the geomagnetic activity time series into natural chaotic phenomena. The method has been applied to predict two solar and geomagnetic activity indices: geomagnetic aa and solar wind speed (SWS) of the solar wind index. The results demonstrate the higher power of the proposed method—compared to other methods—for predicting solar activity.

GENERALIZED CELLULAR NEURAL NETWORKS (GCNNs) CONSTRUCTED USING PARTICLE SWARM OPTIMIZATION FOR SPATIO-TEMPORAL EVOLUTIONARY PATTERN IDENTIFICATION

Particle swarm optimization (PSO) is introduced to implement a new constructive learning algorithm for training generalized cellular neural networks (GCNNs) for the identification of spatio-temporal evolutionary (STE) systems. The basic idea of the new PSO-based learning algorithm is to successively approximate the desired signal by progressively pursuing relevant orthogonal projections. This new algorithm will thus be referred to as the orthogonal projection pursuit (OPP) algorithm, which is in mechanism similar to the conventional projection pursuit approach. A novel two-stage hybrid training scheme is proposed for constructing a parsimonious GCNN model. In the first stage, the orthogonal projection pursuit algorithm is applied to adaptively and successively augment the network, where adjustable parameters of the associated units are optimized using a particle swarm optimizer. The resultant network model produced at the first stage may be redundant. In the second stage, a forward orthogonal regression (FOR) algorithm, aided by mutual information estimation, is applied to refine and improve the initially trained network. The effectiveness and performance of the proposed method is validated by applying the new modeling framework to a spatio-temporal evolutionary system identification problem.

Constructive learning neural network applied to identification and control of a fuel-ethanol fermentation process

In the present work, a constructive learning algorithm was employed to design a near-optimal one-hidden layer neural network structure that best approximates the dynamic behavior of a bioprocess. The method determines not only a proper number of hidden neurons but also the particular shape of the activation function for each node. Here, the projection pursuit technique was applied in association with the optimization of the solvability condition, giving rise to a more efficient and accurate computational learning algorithm. As each activation function of a hidden neuron is defined according to the peculiarities of each approximation problem, better rates of convergence are achieved, guiding to parsimonious neural network architectures. The proposed constructive learning algorithm was successfully applied to identify a MIMO bioprocess, providing a multivariable model that was able to describe the complex process dynamics, even in long-range horizon predictions. The resulting identification model was considered as part of a model-based predictive control strategy, producing high-quality performance in closed-loop experiments.