Cascade-correlation Learning Research Articles

Improved Cascade Correlation Neural Network Model Based on Group Intelligence Optimization Algorithm

The Cascade Correlation learning algorithm is a special supervised learning algorithm for artificial neural network architecture. The optimization algorithm in the traditional neural network has the disadvantages of a single optimization goal, slow convergence speed, and can easily fall into local area, which cannot fully meet the key elements in the cascade correlation learning algorithm. In comparison, the group intelligence optimization algorithm can take into account these key elements in the optimization process at the same time, and obtain better optimization results. In this paper, we propose the single-objective optimization algorithm jDE-B and the multi-objective optimization algorithm MOEA-T, and improve the network expansion mode in the learning process of Cascade Correlation neural networks. We investigate the effect of applying the group intelligent optimization algorithm in the Cascade Correlation learning algorithm. Experimental results show that our improved algorithm is able to enhance the ability of the Cascade Correlation neural network to fit problems, reduce the number of hidden units and the depth of the network, and optimize the network structure.

Predicted Number of Pregnant Women in Aichi Prefecture, Japan: Estimation by Machine Learning Database Construction for Disaster Preparation.

Aichi prefecture, Japan is predicted to be hit by Mega-earthquake. Aichi Prefectural Association of Midwives has been making efforts to improve disaster preparedness for pregnant women. This project aims to acquire area data of pregnant women for simulated studies of rescue activities. Number of women in census survey areas in Nagoya City was acquired from nationwide data of pregnant women by machine learning (Cascade-Correlation Learning Architecture). Quite high correlation coefficients between actual data and estimation data were observed. Rescue simulations have been carried out based on the data acquired by this study.

A surrogate model based evolutionary game-theoretic approach for optimizing non-isothermal compression RTM processes

The Compression Resin Transfer Moulding (CRTM) process is a variant of the traditional RTM process and permits significantly faster fill times. However, the design parameters of CRTM processes must be carefully chosen in order to reduce cycle time, capital layout and running costs, while maximizing final part quality. These objectives are principally governed by the filling and curing phases which are strongly coupled in the case of non-isothermal processes. In this work the composites manufacturing cycle is modelled as a static Stackelberg game with two virtual decision makers (DMs) monitoring the filling and curing phases, respectively. The model is implemented through a Bilevel Multiobjective Genetic Algorithm (BMOGA), in conjunction with the Cascade-Correlation Learning Architecture Neural Network (CCA-NN) for function evaluations. The obtained results are efficient with respect to the objectives of both DMs and provide the manufacturer with a diverse set of solutions to choose from.

A Connectionist Study on the Interplay of Nouns and Pronouns in Personal Pronoun Acquisition

Cascade-correlation learning is used to model pronoun acquisition in children. The cascade-correlation algorithm is a feed-forward neural network that builds its own topology from input and output units. Personal pronoun acquisition is an interesting non-linear problem in psychology. A mother will refer to her son as you and herself as me, but the son must infer for himself that when he speaks to his mother, she becomes you and he becomes me. Learning the shifting reference of these pronouns is a difficult task that most children master. We show that learning of two different noun-and-pronoun addressee patterns is consistent with naturalistic studies. We observe a surprising factor in pronoun reversal: increasing the amount of exposure to noun patterns can decrease or eliminate reversal errors in children.

유전자알고리즘을 이용한 시그모이드 활성화 함수 파라미터의 최적화와 이중나선 문제의 입력공간 패턴인식 분석

본 논문에서는 유전자알고리즘을 이용한 시그모이드 활성화 함수 파라미터의 최적화와 이중나선기준문제(two spirals benchmark problem)의 입력공간 패턴인식 상태를 분석 한다. 실험을 위하여 캐스케이드 코릴레이션 학습 알고리즘(Cascade Correlation learning algorithm)을 이용한다. 첫 번째 실험에서는 기본적인 시그모이드 활성화 함수를 사용하여 이중나선 문제를 분석하고, 두 번째 실험에서는 시그모이드 활성화 함수(sigmoidal activation function)의 파라미터 값이 서로 다른 함수를 사용하여 8개의 풀을 구성한다. 세 번째 실험에서는 시그모이드 함수의 변위를 결정하는 세 개의 파라미터 값을 유전자 알고리즘을 이용하여 얻고 이 파라미터 값들이 적용된 시그모이드 함수들은 후보뉴런의 활성화를 위해서 사용된다. 이러한 알고리즘의 성능평가를 위하여 각 학습단계 마다 입력패턴공간에서 인식된 이중나선의 형태를 보여준다. This paper presents a optimization of sigmoid activation function parameter using genetic algorithms and pattern recognition analysis in input space of two spirals benchmark problem. To experiment, cascade correlation learning algorithm is used. In the first experiment, normal sigmoid activation function is used to analyze the pattern classification in input space of the two spirals problem. In the second experiment, sigmoid activation functions using different fixed values of the parameters are composed of 8 pools. In the third experiment, displacement of the sigmoid function to determine the value of the three parameters is obtained using genetic algorithms. The parameter values applied to the sigmoid activation functions for candidate neurons are used. To evaluate the performance of these algorithms, each step of the training input pattern classification shows the shape of the two spirals.

Radial Basis Function Cascade Correlation Networks

A cascade correlation learning architecture has been devised for the first time for radial basis function processing units. The proposed algorithm was evaluated with two synthetic data sets and two chemical data sets by comparison with six other standard classifiers. The ability to detect a novel class and an imbalanced class were demonstrated with synthetic data. In the chemical data sets, the growth regions of Italian olive oils were identified by their fatty acid profiles; mass spectra of polychlorobiphenyl compounds were classified by chlorine number. The prediction results by bootstrap Latin partition indicate that the proposed neural network is useful for pattern recognition.

일반화된 캐스케이드 코릴레이션 알고리즘과 일반화된 순환 캐스케이드 코릴레이션 알고리즘의 결합을 통한 학습 능력 향상

본 논문에서는 일반화된 캐스케이드 코릴레이션 학습 알고리즘과 일반화된 순환 캐스케이드 코릴레이션 학습 알고리즘의 결합을 통한 새로운 알고리즘을 소개한다. 이 새로운 알고리즘은 패턴분류문제(pattern classification problem)의 신속한 해결을 위하여 비순환 뉴런이 유리한지 순환 뉴런이 유리한지 또는 수직성장이 유리한지 수평성장이 유리한지 고민할 필요 없이 후보뉴런의 학습 중에 네트워크의 구성을 스스로 결정한다. 이 알고리즘의 성능평가를 위하여 학습 알고리즘에서 중요한 기준 문제(benchmark problem) 중의 하나인 콘택트렌즈 문제(Contact lens problem)와 밸런스 스케일 문제 (Balance scale problem)에 대하여 실험하였고 기존의 캐스케이드 코릴레이션 알고리즘 및 순환 캐스케이드 코릴레이션 알고리즘과 성능을 비교 하였다. 이 실험에서 활성화 함수는 일반적으로 많이 사용하는 시그모이드 함수(sigmoidal function) 와 하이퍼볼릭탄젠트 함수(hyperbolic tangent function)를 사용하였다. 이 새로운 알고리즘은 학습을 통하여 기존의 알고리즘보다 적은 수의 은닉뉴런을 생성하여 보다 빠른 학습 속도를 보여주었다. This paper presents a combination of the generalized Cascade Correlation and generalized Recurrent Cascade Correlation learning algorithms. The new network will be able to grow with vertical or horizontal direction and with recurrent or without recurrent units for the quick solution of the pattern classification problem. The proposed algorithm was tested learning capability with the sigmoidal activation function and hyperbolic tangent activation function on the contact lens and balance scale standard benchmark problems. And results are compared with those obtained with Cascade Correlation and Recurrent Cascade Correlation algorithms. By the learning the new network was composed with the minimal number of the created hidden units and shows quick learning speed. Consequently it will be able to improve a learning capability.

Improved ANN Based Tap-Changer Controller Using Modified Cascade-Correlation Algorithm

Artificial Neural Network (ANN) based tap changer control of closed primary bus and cross network connected parallel transformers has demonstrated potential use in power distribution system. In those research works the proposed ANN for application in this control were developed using various algorithms and concluded that a network trained by Bayesian Regularization (BR) backpropagation algorithm produced the best performance measured in terms of correct tap changing decisions. However, further improvement of ANN based transformer tap changer operation is always desirable. A general rule for obtaining good generalization is to use the smallest network that solves the problem. In this paper, we show that a small sized ANN is obtainable for further improvement of transformer tap changer operation by modifying the standard Cascade-Correlation algorithm. The modification incorporates weight smoothing of output layer weights in Cascade-Correlation learning using Bayesian frame work. Experimental results demonstrate that significant improvement in performance is achieved when an ANN is trained by modified Cascade-Correlation algorithm instead of standard Cascade-Correlation or Bayesian Regularization backpropagation algorithm. A comparison of performances of different algorithms in application to transformer tap changer operation is analyzed and the results are presented.

캐스케이드-상관 학습 알고리즘의 패밀리

Fahlman과 Lebiere의 캐스케이드-상관 (CC) 학습 알고리즘은 신경망의 구성 알고리즘에서 가장 널리 사용되는 것 중의 하나이며, 망에서 은닉 뉴런을 캐스케이드 형태로 취함으로서 매우 강력한 비선형을 표현할 수 있다. 비록 이 멱승이 유용할지 몰라도 대체로 문제를 푸는데는 강력한 비선형성이 요구되지 않으며 단점이 될 수도 있다. CC 알고리즘의 캐스케이드 구조 및 출력 뉴런의 가중치 훈련에 대한 변형된 형태인 3개 모델이 제안되고 경험적으로 비교되었다. 실험결과 다음과 같은 결론을 얻었다: (1) 패턴분류에 있어서, 새로 추가되는 은닉 뉴런과 출력층간 연결강도만 훈련시키는 모델이 가장 좋은 예측력을 나타내었다; (2) 함수근사 문제에 있어서는 입력-출력 연결강도를 제거하고 시그모이드-선형 작동함수를 사용하는 모델이 CasCor 알고리즘보다 좋은 결과를 나타내었다. The cascade-correlation (CC) learning algorithm of Fahlman and Lebiere is one of the most influential constructive algorithm in a neural network. Cascading the hidden neurons results in a network that can represent very strong nonlinearities. Although this power is in principle useful, it can be a disadvantage if such strong nonlinearity is not required to solve the problem. 3 models are presented and compared empirically. All of them are based on valiants of the cascade architecture and output neurons weights training of the CC algorithm. Empirical results indicate the followings： (1) In the pattern classification, the model that train only new hidden neuron to output layer connection weights shows the best predictive ability; (2) In the function approximation, the model that removed input-output connection and used sigmoid-linear activation function is better predictability than CasCor algorithm.

Convergence analysis of cascade error projection--an efficient learning algorithm for hardware implementation.

In this paper, we present a mathematical foundation, including a convergence analysis, for cascading architecture neural network. Our analysis also shows that the convergence of the cascade architecture neural network is assured because it satisfies Liapunov criteria, in an added hidden unit domain rather than in the time domain. From this analysis, a mathematical foundation for the cascade correlation learning algorithm can be found. Furthermore, it becomes apparent that the cascade correlation scheme is a special case from mathematical analysis in which an efficient hardware learning algorithm called Cascade Error Projection(CEP) is proposed. The CEP provides efficient learning in hardware and it is faster to train, because part of the weights are deterministically obtained, and the learning of the remaining weights from the inputs to the hidden unit is performed as a single-layer perceptron learning with previously determined weights kept frozen. In addition, one can start out with zero weight values (rather than random finite weight values) when the learning of each layer is commenced. Further, unlike cascade correlation algorithm (where a pool of candidate hidden units is added), only a single hidden unit is added at a time. Therefore, the simplicity in hardware implementation is also achieved. Finally, 5- to 8-bit parity and chaotic time series prediction problems are investigated; the simulation results demonstrate that 4-bit or more weight quantization is sufficient for learning neural network using CEP. In addition, it is demonstrated that this technique is able to compensate for less bit weight resolution by incorporating additional hidden units. However, generation result may suffer somewhat with lower bit weight quantization.

River flow prediction using artificial neural networks: generalisation beyond the calibration range

Artificial neural networks (ANNs) provide a quick and flexible means of creating models for river flow prediction, and have been shown to perform well in comparison with conventional methods. However, if the models are trained using a dataset that contains a limited range of values, they may perform poorly when encountering events containing previously unobserved values. This failure to generalise limits their use as a tool in applications where the data available for calibration is unlikely to cover all possible scenarios.This paper presents a method for improved generalisation during training by adding a guidance system to the cascade-correlation learning architecture. Two case studies from catchments in the UK are prepared so that the validation data contains values that are greater or less than any included in the calibration data. The ability of the developed algorithm to generalise on new data is compared with that of the standard error backpropagation algorithm. The ability of ANNs trained with different output activation functions to extrapolate beyond the calibration data is assessed.

Modified cascade-correlation learning for classification

The main advantages of cascade-correlation learning are the abilities to learn quickly and to determine the network size. However, recent studies have shown that in many problems the generalization performance of a cascade-correlation trained network may not be quite optimal. Moreover, to reach a certain performance level, a larger network may be required than with other training methods. Recent advances in statistical learning theory emphasize the importance of a learning method to be able to learn optimal hyperplanes. This has led to advanced learning methods, which have demonstrated substantial performance improvements. Based on these recent advances in statistical learning theory, we introduce modifications to the standard cascade-correlation learning that take into account the optimal hyperplane constraints. Experimental results demonstrate that with modified cascade correlation, considerable performance gains are obtained compared to the standard cascade-correlation learning. This includes better generalization, smaller network size, and faster learning.

Application of combined genetic algorithms with cascade correlation to diagnosis of delayed gastric emptying from electrogastrograms

The current standard method (radioscintigraphy) for the diagnosis of delayed gastric emptying (GE) of a solid meal involves radiation exposure and considerable expense. Based on combining genetic algorithms with the cascade correlation learning architecture, a neural network approach is proposed for the diagnosis of delayed GE from electrogastrograms (EGGs). EGGs were measured by placing surface electrodes on the abdominal skin over the stomach in 152 patients with suspected gastric motility disorders for 30 min in the fasting state and for 2 h after a standard test meal. The GE rate of the stomach was simultaneously monitored after the meal using radioscintigraphy. Five spectral parameters of EGG data in each patient were used as the inputs to a classifier. The classifier was designed by using genetic algorithms in conjunction with the cascade correlation learning architecture. The main advantage of this technique over the back-propagation (BP) for supervised learning is that it can automatically develop the architecture of neural networks to give a suitable network size for a specific problem. The resulted neural network with three hidden units exhibits 83% correct classification for the EGG data, and has comparable performance with the BP network. This study demonstrates the potential of the neural network approach based on combined genetic algorithms with cascade correlation for diagnosis of gastric emptying from the EGG.

Modelling with constructive backpropagation

Neural network methods have proven to be powerful tools in modelling of nonlinear processes. One crucial part of modelling is the training phase where the model parameters are adjusted so that the model performs the desired operation as well as possible. Besides parameter estimation, an important problem is to select a suitable model structure. With a bad structure we potentially run into problems like underfitting, overfitting or wasting computational resources. One approach for structure learning is to use constructive methods, where training begins with minimal structure, and then more parameters are added when needed according to some predefined rule. This kind of constructive solution has also become more attractive in neural networks literature where one of the most well known constructive techniques is cascade-correlation (CC) learning. Inspired by CC we propose and study a similar technique called constructive backpropagation (CBP). We show that CBP is computationally just as efficient as the CC algorithm even though we need to backpropagate the error through no more than one hidden layer. Further, CBP has the same constructive benefits as CC, but in addition CBP benefits from simpler implementation and the ability to utilize stochastic optimization routines. Moreover, we show how CBP can be extended to allow addition of multiple new units simultaneously and how it can be used to perform continuous automatic structure adaptation. This includes both addition and deletion of units. The performance of CBP learning is studied with time series modelling experiments which demonstrate that CBP can provide significantly better modelling capabilities compared to CC learning.

Fast initialization for cascade-correlation learning

Weight initialization in the cascade-correlation learning is considered. Most of the previous studies use the so called candidate training to deal with the initialization problem in the cascade-correlation learning. There several candidate hidden units are first trained, and then the one yielding the best value for the covariance criterion is installed to the network. In case there are many candidate units to be trained, the total computational cost of the training can become very large. Here we consider a new approach for weight initialization in the cascade-correlation learning. The proposed method is based on the concept of stepwise regression. Empirical simulations show that the new method can significantly speed-up the cascade-correlation learning compared to the case where the candidate training is used. Moreover, the overall performance remained similar or was even better than with the candidate training.

Analysis of Knowledge Representations in Cascade Correlation Networks

Feed-forward neural network models approximate nonlinear functions connecting inputs to outputs. The cascade correlation (CC) learning algorithm allows networks to grow dynamically starting from the simplest network topology to solve increasingly more difficult problems. It has been demonstrated that the CC network can solve a wide range of problems including those for which other kinds of networks (e.g., back-propagation networks) have been found to fail. In this paper we show the mechanism and characteristics of nonlinear function learning and representations in CC networks, their generalization capabilities, the effects of environmental bias, etc., using a variety of knowledge representation analysis tools.

Improvement of cascade correlation learning

Cascade Correlation (CC) as a constructive algorithm for artificial neural networks stands out for solving hard classification problems. In this note we propose an extension of this technique that uses genetic algorithm for finding a good initialization point and yields improved results. The performance of our method is demonstrated on benchmark problems.

Automated speech recognition using a plurality of different multilayer perception structures to model a plurality of distinct phoneme categories

For speech recognition systems a method for modeling context-dependent phonetic categories using artificial neural nets has been described. First, linguistically motivated context-clustering is employed to reduce the number of context-dependent categories. Second, phone-specific MLP structures are used where the number of outputs in each MLP is based on the number of left and right contexts occurring in a training database. The structure of each MLP can be automatically determined using the cascade-correlation learning algorithm.

Neural Network Studies. 3. Variable Selection in the Cascade-Correlation Learning Architecture

Pruning methods for feed-forward artificial neural networks trained by the cascade-correlation learning algorithm are proposed. The cascade-correlation algorithm starts with a small network and dynamically adds new nodes until the analyzed problem has been solved. This feature of the algorithm removes the requirement to predefine the architecture of the neural network prior to network training. The developed pruning methods are used to estimate the importance of large sets of initial variables for quantitative structure−activity relationship studies and simulated data sets. The calculated results are compared with the performance of fixed-size back-propagation neural networks and multiple regression analysis and are carefully validated using different training/test set protocols, such as leave-one-out and full cross-validation procedures. The results suggest that the pruning methods can be successfully used to optimize the set of variables for the cascade-correlation learning algorithm neural networks. The use of variables selected by the elaborated methods provides an improvement of neural network prediction ability compared to that calculated using the unpruned sets of variables.

The prediction of protein secondary structure with a Cascade Correlation Learning Architecture of neural networks

A Cascade Correlation Learning Architecture (CCLA) of neural networks is tested on the task of predicting the secondary structure of proteins. The results are compared with those obtained with Neural Networks (NN) trained with the back-propagation algorithm (BPNN) and generated with genetic algorithms. CCLA proceeds towards the global minimum of the error function more efficiently than BPNN. However, only a slight improvement in the average efficiency value is noticeable (61.82% as compared with 61.61% obtained with BPNN). The values of the three correlation coefficients for the discriminated secondary structures are also rather similar (Ct8,Cα,Cβ and Ccoil are 0.36, 0.29 and 0.36 with CCLA, and 0.36, 0.31 and 0.35 with BPNN). This indicates that the efficiency of the prediction does not depend upon the training algorithm, and confirms our previous observation that when single sequences are used as input code to the network system, different NN architectures can perform similarly.