Neural Networks For Classification Problems Research Articles

Parallel ensemble of a randomization-based online sequential neural network for classification problems using a frequency criterion

Randomization-based neural networks have gained wide acceptance in the scientific community owing to the simplicity of their algorithm and generalization capabilities. Random vector functional link (RVFL) networks and their variants are a class of randomization-based neural networks. RVFL networks have shown promising results in classification, regression, and clustering problems. For real-world applications, learning algorithms that can train with new samples over previous results are necessary because of to the constant generation of problems related to large-scale datasets. Various online sequential algorithms, commonly involving an initial learning phase followed by a sequential learning phase, have been proposed to address this issue. This paper presents a training algorithm based on multiple online sequential random vector functional link (OS-RVFL) networks for large-scale databases using a shared memory architecture. The training dataset is distributed among p OS-RVFL networks, which are trained in parallel using p threads. Subsequently, the test dataset samples are classified using each trained OS-RVFL network. Finally, a frequency criterion is applied to the results obtained from each OS-RVFL network to determine the final classification. Additionally, an equation was derived to reasonably predict the total training time of the proposed algorithm based on the learning time in the initial phase and the time scaling factor compared to the sequential learning phase. The results demonstrate a drastic reduction in training time because of data distribution and an improvement in accuracy because of the adoption of the frequency criterion.

Demonstrating a new evaluation method on ReLU based Neural Networks for classification problems

Deep neural networks, which have proven to be effective methods to solve complex problems, can even be applied in decision systems controlling critical processes. However, in such applications the outcomes of the neural network must be checked if it we have a clear understanding regarding the operation of network at given input intervals. The most straightforward, though often computationally expensive approach for this checking process evaluates the network at discrete input points and estimates the expected outputs at the given interval. The present research aims to develop a novel approach that can identify in the case of specific input intervals whether the operation process and the output of a neural network can be considered as known or unknown. During the presented case study, we investigated the ReLU (Rectified Linear Unit) and Sigmoid activation functions, using the double moon and the Banknote Authentication classification problems for demonstration.Our method can be applied to identify certain input intervals where the given neural network cannot support critical decisions. The evaluation is performed based on a nonlinear system of equations and inequalities built on arbitrary continuous activation functions. To define the critical intervals of the input variables (i.e. where the decision-making system should not be relied on), those input variable combinations are identified which result in a non-expected output value. This inverse logic is intended to identify intervals of the input variables where the response of the system and the correct decision are not identical. The presented demonstration examples supported our assumption that the number of the neurons and the dimension of the decision space have a significant impact on the complexity of the evaluation process.

Evaluation of Dataflow through layers of convolutional neural networks in classification problems

Evaluation of Dataflow through layers of convolutional neural networks in classification problems

Comparative characteristics of the ability of convolutional neural networks to the concept of transfer learning

The object of research is the ability to combine a previously trained model of a deep neural network of direct propagation with user data when used in problems of determining the class of one object in the image. That is, the processes of transfer learning in convolutional neural networks in classification problems are considered. The conducted researches are based on application of a method of comparison of theoretical and practical results received at training of convolutional neural networks. The main objective of this research is to conduct two different learning processes. Traditional training during which in each epoch of training there is an adjustment of values of all weights of each layer of a network. After that there is a process of training of a neural network on a sample of the data presented by images. The second process is learning using transfer learning methods, when initializing a pre-trained network, the weights of all its layers are «frozen» except for the last fully connected layer. This layer will be replaced by a new one with the number of outputs, which should be equal to the number of classes in the sample. After that, to initialize its parameters by the random values distributed according to the normal law. Then conduct training of such convolutional neural network on the set sample. When the training was conducted, the results were compared. In conclusion, learning from convolutional neural networks using transfer learning techniques can be applied to a variety of classification tasks, ranging from numbers to space objects (stars and quasars). The amount of computer resources spent on research is also quite important. Because not all model of a convolutional neural network can be fully taught without powerful computer systems and a large number of images in the training sample.

Experimental stability analysis of neural networks in classification problems with confidence sets for persistence diagrams

We investigate classification performance of neural networks (NNs) based on topological insight in an attempt to guarantee stability of their inference. NNs which can accurately classify a dataset map it into a hidden space while disentangling intertwined data. NNs sometimes acquire forcible mapping to disentangle the data, and this forcible mapping generates outliers. The mapping around the outliers is unstable because the outputs change drastically. Hence, we define stable NNs to mean that they do not generate outliers. To investigate the possibility of the existence of outliers, we use persistent homology and a method to estimate the confidence set for persistence diagrams. The combined use enables us to test whether the focused geometry is topologically simple, that is, no outliers. In this work, we use the MNIST and CIFAR-10 datasets and investigate the relationship between the classification performance and the topological characteristics with several NNs. Investigation results with the MNIST dataset show that the test accuracy of all the networks is superior, exceeding 98%, even though the transformed dataset is not topologically simple. Results with the CIFAR-10 dataset also show that the possibility of the existence of outliers is shown in the mapping by the accurate convolutional NNs. Therefore, we conclude that the presented investigation is necessary to guarantee that the NNs, in particular deep NNs, do not acquire unstable mapping for forcible classification.

WiFiNet: WiFi-based indoor localisation using CNNs

Different technologies have been proposed to provide indoor localisation: magnetic field, Bluetooth, WiFi, etc. Among them, WiFi is the one with the highest availability and highest accuracy. This fact allows for an ubiquitous accurate localisation available for almost any environment and any device. However, WiFi-based localisation is still an open problem.In this article, we propose a new WiFi-based indoor localisation system that takes advantage of the great ability of Convolutional Neural Networks in classification problems. Three different approaches were used to achieve this goal: a custom architecture called WiFiNet, designed and trained specifically to solve this problem, and the most popular pre-trained networks using both transfer learning and feature extraction.Results indicate that WiFiNet is as a great approach for indoor localisation in a medium-sized environment (30 positions and 113 access points) as it reduces the mean localisation error (33%) and the processing time when compared with state-of-the-art WiFi indoor localisation algorithms such as SVM.

Global convergence and geometric characterization of slow to fast weight evolution in neural network training for classifying linearly non-separable data

In this paper, we study the dynamics of gradient descent in learning neural networks for classification problems. Unlike in existing works, we consider the linearly non-separable case where the training data of different classes lie in orthogonal subspaces. We show that when the network has sufficient (but not exceedingly large) number of neurons, (1) the corresponding minimization problem has a desirable landscape where all critical points are global minima with perfect classification; (2) gradient descent is guaranteed to converge to the global minima. Moreover, we discovered a geometric condition on the network weights so that when it is satisfied, the weight evolution transitions from a slow phase of weight direction spreading to a fast phase of weight convergence. The geometric condition says that the convex hull of the weights projected on the unit sphere contains the origin.

Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness

The accuracy of deep learning, i.e., deep neural networks, can be characterized by dividing the total error into three main types: approximation error, optimization error, and generalization error. Whereas there are some satisfactory answers to the problems of approximation and optimization, much less is known about the theory of generalization. Most existing theoretical works for generalization fail to explain the performance of neural networks in practice. To derive a meaningful bound, we study the generalization error of neural networks for classification problems in terms of data distribution and neural network smoothness. We introduce the cover complexity (CC) to measure the difficulty of learning a data set and the inverse of the modulus of continuity to quantify neural network smoothness. A quantitative bound for expected accuracy/error is derived by considering both the CC and neural network smoothness. Although most of the analysis is general and not specific to neural networks, we validate our theoretical assumptions and results numerically for neural networks by several data sets of images. The numerical results confirm that the expected error of trained networks scaled with the square root of the number of classes has a linear relationship with respect to the CC. We also observe a clear consistency between test loss and neural network smoothness during the training process. In addition, we demonstrate empirically that the neural network smoothness decreases when the network size increases whereas the smoothness is insensitive to training dataset size.

$$\beta$$-Hill climbing algorithm with probabilistic neural network for classification problems

Classification is a crucial step in the data mining field. The probabilistic neural network (PNN) is an efficient method developed for classification problems. The success factor of using PNN for classification problems implies in finding the proper weight during classification process. The main goal of this paper is to improve the performance of PNN by finding the best weight for the PNN using the recent local search approach called $$\beta$$ -hill-climbing ( $$\beta$$ -HC) optimizer. This algorithm is an extension version of the traditional hill climbing algorithm in that it uses a stochastic operator to avoid local optima. The proposed approach is evaluated against 11 benchmark datasets ,and the experimental results showed that the proposed $$\beta$$ -HC with PNN approach performed better in terms of classification accuracy than the original PNN, HC-PNN and other six well-established approaches using the same experimented benchmarks.

Development of a Self-Regulating Evolving Spiking Neural Network for classification problem

This paper presents a new spiking neural network for pattern classification problems, referred to as the Self-Regulating Evolving Spiking Neural (SRESN) classifier, that regulates the learning process of the network. It uses a two layered spiking neural network and the input layer consists of receptive field neurons, which convert a real valued input to spikes using the population coding scheme without any delays. The output layer consists of leaky integrate-and-fire neurons. Since SRESN does not use any delays, the number of network parameters for SRESN is significantly lower than that used by other spiking neural networks, used in this study. During training, the learning algorithm for SRESN, automatically evolves neurons in the output layer based on the training data stream and the current knowledge stored in the network. Depending on the knowledge in the sample and the class specific knowledge stored in the network, it can choose to either add a neuron or update the network parameters or skip learning the sample resulting in self-regulation of the learning process. In case of neuron addition, the weights for the newly added neuron are initialized using a modified rank order scheme which facilitates SRESN for use in online/sequential as well as batch learning modes. The parameter update strategy in SRESN ensures that connections with non-zero postsynaptic potential at the time of the spike are alone updated which helps prevent over training. While evaluating the performance of SRESN, first a study is conducted to assess the impact of various parameters on its performance and establish guidelines to choose suitable values for these parameters. Next, the performance of SRESN, operating in batch mode, is compared with other spiking neural classifiers, including SpikeProp and MuSpiNN, for the UCI benchmark problems of Iris flower classification and Wisconsin breast cancer. Subsequently, the performance of SRESN in online and batch learning mode is compared with an evolving spiking neural classifier for five benchmark data sets from the UCI machine learning repository. Finally, SRESN is applied to solve the practical problem of Epilepsy detection. The performance comparison clearly indicates that SRESN provides a higher generalization accuracy using fewer network parameters.

Meta-cognitive Neural Network for classification problems in a sequential learning framework

In this paper, we propose a sequential learning algorithm for a neural network classifier based on human meta-cognitive learning principles. The network, referred to as Meta-cognitive Neural Network (McNN). McNN has two components, namely the cognitive component and the meta-cognitive component. A radial basis function network is the fundamental building block of the cognitive component. The meta-cognitive component controls the learning process in the cognitive component by deciding what-to-learn, when-to-learn and how-to-learn. When a sample is presented at the cognitive component of McNN, the meta-cognitive component chooses the best learning strategy for the sample using estimated class label, maximum hinge error, confidence of classifier and class-wise significance. Also sample overlapping conditions are considered in growth strategy for proper initialization of new hidden neurons. The performance of McNN classifier is evaluated using a set of benchmark classification problems from the UCI machine learning repository and two practical problems, viz., the acoustic emission for signal classification and a mammogram data set for cancer classification. The statistical comparison clearly indicates the superior performance of McNN over reported results in the literature.

Use of Linear Genetic Programming and Artificial Neural Network Methods to Solve Classification Task

This paper presents a comparative analysis of linear genetic programming and artificial neural network methods to solve classification tasks. Usually classification tasks have data sets containing a large number of attributes and records, and more than two classes that will be processed using, for example, created classification rules. As a result, by using classical method to classify a large number of records, a high classification error value will be obtained. The artificial neural networks are often used to solve classification task, mostly obtaining good results. The linear genetic programming is a new direction of evolution algorithms that is not widely researched and its application areas are not well defined. However, some advantages of linear genetic programming are based on genetic operators whose structure does not require complicated calculations. During this work approximately 400 experiments were conducted with linear genetic programming and artificial neural network methods, using various data sets with different quantity of records, attributes and classes. Based on the results received, conclusions on possibilities of using the methods of linear genetic programming and artificial neural networks in classification problems were drawn, and suggestions for improving their performance were proposed.

Learning Ensembles of Neural Networks by Means of a Bayesian Artificial Immune System

In this paper, we apply an immune-inspired approach to design ensembles of heterogeneous neural networks for classification problems. Our proposal, called Bayesian artificial immune system, is an estimation of distribution algorithm that replaces the traditional mutation and cloning operators with a probabilistic model, more specifically a Bayesian network, representing the joint distribution of promising solutions. Among the additional attributes provided by the Bayesian framework inserted into an immune-inspired search algorithm are the automatic control of the population size along the search and the inherent ability to promote and preserve diversity among the candidate solutions. Both are attributes generally absent from alternative estimation of distribution algorithms, and both were shown to be useful attributes when implementing the generation and selection of components of the ensemble, thus leading to high-performance classifiers. Several aspects of the design are illustrated in practical applications, including a comparative analysis with other attempts to synthesize ensembles.

Applying generalised feedforward neural networks to classifying industrial jobs in terms of risk of low back disorders

This paper describes a new approach for the development of artificial neural networks applied to classifying the risk of low back disorders (LBDs) presented by certain lifting jobs. The development process of neural networks for classification problems and the influence of network architecture on its prediction and generalisation capabilities are analysed. The phenomenon of overfitting and its relationship to the number of network connections, and the size of the training data set are discussed. The new approach uses complex architecture networks and early stopping of training procedures to avoid overfitting. It is thus compared with previous studies and its results and advantages over them are assessed. Such comparison shows that this approach allows for the development of neural networks for the classification of industrial jobs according to their risk of causing LBDs in a way faster than previous procedures. The neural network obtained is able to correctly classify 81.6% of highly repetitive lifting industrial jobs as posing low or high-risk for LBDs, from five independent variables representing biomechanical risk factors. The neural network obtained has been implemented in a software application which focuses on risk analysis and prevention of the injuries caused by tasks involving manual lifting in the industrial environment. Relevance to industry The approach described in this paper allows neural networks to be developed for the prediction of musculoskeletal disorders in a way that is faster and easier than with previous procedures. It can also be applied to risk prediction in other areas of ergonomics. The neural network obtained can be used by the ergonomist as a diagnostic system, enabling jobs to be classified into two categories (low-risk and high-risk) according to the associated likelihood of causing low back disorders. This system provides a higher proportion of correct classifications than other previous models.

A hybrid radial basis function and data envelopment analysis neural network for classification

We propose a hybrid radial basis function network-data envelopment analysis (RBFN-DEA) neural network for classification problems. The procedure uses the radial basis function to map low dimensional input data from input space ℜ to a high dimensional ℜ + feature space where DEA can be used to learn the classification function. Using simulated datasets for a non-linearly separable binary classification problem, we illustrate how the RBFN-DEA neural network can be used to solve it. We also show how asymmetric misclassification costs can be incorporated in the hybrid RBFN-DEA model. Our preliminary experiments comparing the RBFN-DEA with feed forward and probabilistic neural networks show that the RBFN-DEA fares very well.

An Empirical Comparison of Combinations of Evolutionary Algorithms and Neural Networks for Classification Problems

There are numerous combinations of neural networks (NNs) and evolutionary algorithms (EAs) used in classification problems. EAs have been used to train the networks, design their architecture, and select feature subsets. However, most of these combinations have been tested on only a few data sets and many comparisons are done inappropriately measuring the performance on training data or without using proper statistical tests to support the conclusions. This paper presents an empirical evaluation of eight combinations of EAs and NNs on 15 public-domain and artificial data sets. Our objective is to identify the methods that consistently produce accurate classifiers that generalize well. In most cases, the combinations of EAs and NNs perform equally well on the data sets we tried and were not more accurate than hand-designed neural networks trained with simple backpropagation.

The Effect of Misclassification Costs on Neural Network Classifiers

The potential of neural networks for classification problems has been established by numerous successful applications reported in the literature. One of the major assumptions used in almost all studies is the equal cost consequence of misclassification. With this assumption, minimizing the total number of misclassification errors is the sole objective in developing a neural network classifier. Often this is done simply to ease model development and the selection of classification decision points. However, it is not appropriate for many real situations such as quality assurance, direct marketing, bankruptcy prediction, and medical diagnosis where misclassification costs have unequal consequences for different categories. In this paper, we investigate the issue of unequal misclassification costs in neural network classifiers. Through an application in thyroid disease diagnosis, we find that different cost considerations have significant effects on the classification performance and that appropriate use of cost information can aid in optimal decision making. A cross‐validation technique is employed to alleviate the problem of bias in the training set and to examine the robustness of neural network classifiers with regard to sampling variations and cost differences.

Learning in the combinatorial neural model

The combinatorial neural model (CNM) is a type of fuzzy neural network for classification problems and, more generally, for the mapping between fuzzy multidimensional spaces. Learning in CNM is a complex task spanning the learning of input-neuron membership functions, of the network topology, and of connection weights. In this paper we are concerned with these various aspects of learning in CNM, most notably with the learning of connection weights, whose complexity comes from the existence of nondifferentiable, nonconvex error functions associated with the learning process. We introduce several algorithms for weight learning, most based on subgradient techniques borrowed from the field of nondifferentiable optimization. All algorithms are based on essentially "local" rules, and are therefore amenable to distributed/parallel implementations. Experimental results are provided on the large-scale problem of monitoring the deforestation of the Amazon region on satellite images. What these results indicate is that a hybrid CNM system outperforms previous results obtained with variations of error backpropagation techniques. In addition, this hybrid system has demonstrated robustness in the context under consideration, therefore constituting an attractive alternative.

GINN (Genetic Inside Neural Network): towards a non-parametric training

A genetic algorithm (steady state evolutionary algorithm without duplication) to train neural networks for classification problems is investigated. The algorithm is based on direct optimization of frequencies of both misclassifications and number of correct classifications with special attention to the parameters known as sensibility and specificity. This way, it is possible to find more information about the behaviour of artificial neural networks on classification.

A neural network architecture for classification of fuzzy inputs

A neural network for classification problems with fuzzy inputs is proposed. A fuzzy input is represented as an LR-type fuzzy set. A generalized pocket algorithm, called fuzzy pocket algorithm, that uses LR-type fuzzy sets operations and defuzzification method is proposed to train a linear threshold unit (LTU). This LTU node will classify as many fuzzy input instances as possible. Afterward, FV nodes that represent fuzzy vectors will then be generated and expanded, by proposed FVGE learning algorithm, to classify those fuzzy input instances that cannot be classified by the LTU node. The similarity degree between FV nodes and fuzzy inputs is measured by the fuzzy subsethood degree. The network structure is automatically generated. The number of hidden nodes generated depends on the overlapping degree of training instances. Besides, on-line learning is supplied, and parameters used are few and insensitive. The relationship between proposed model and hyperbox-based classifiers, e.g., Fuzzy ART series and Fuzzy Min-Max series, is also discussed. Two sample problems, heart disease and knowledge-based evaluator, are considered to illustrate the working of the proposed model. The experimental results are very encouraging.