Online Structure Learning Research Articles

Application of self-learning interval type-2 fuzzy neural network in PM2.5 concentration prediction

Abstract The change of PM2.5 concentration in air quality is nonlinear and difficult to predict. Therefore, a self-learning interval type-2 fuzzy neural network (SLIT2FNN) is proposed. SLIT2FNN has two parts: online structure learning and parameter learning. In structure learning, to improve the training accuracy and speed of the model, the Possibilistic Fuzzy C-Means (PFCM) algorithm is used to process the input data and obtain the number of initial rules before model training. The PFCM algorithm introduces the concept of possibility P to Fuzzy C-Means (FCM), allowing PFCM to overcome the shortcomings of FCM that cannot accurately cluster a large number of nonlinear problems. SLIT2FNN can establish an appropriate number of rules in the preparation stage, and then use the firing strength of the antecedents of the rules to judge whether to generate fuzzy rules for online self-learning, thereby optimizing its network structure. Then, the improved Levenberg–Marquardt (ILM) algorithm is used to modify the relevant parameters of SLIT2FNN. The ILM algorithm can address the challenge of numerous parameters in the Jacobian matrix and complex calculations and improve the calculation speed and adaptive ability of SLIT2FNN parameter learning. Finally, SLIT2FNN is applied to the prediction of air quality PM2.5 concentration, and the performance is compared with other models. The experiment proves that SLIT2FNN has a high prediction accuracy and fast convergence.

Online semi-supervised learning of composite event rules by combining structure and mass-based predicate similarity

Symbolic event recognition systems detect event occurrences using first-order logic rules. Although existing online structure learning approaches ease the discovery of such rules in noisy data streams, they assume the existence of fully labelled training data. Splice is a recent online graph-based approach that estimates the labels of unlabelled data and makes it possible to learn such rules from semi-supervised training sequences of logical interpretations. However, Splice labelling depends significantly on the metric used to compute the distances of unlabelled examples to their labelled counterparts. Moreover, there is no guarantee about the quality of the labelling found in the local graphs that are built as the data stream in. In this paper, we propose a new online learning method, which includes an enhanced hybrid measure that combines an optimised structural distance, and a data-driven one. The former is guided by feature selection targeted to kNN classification, while the latter is a mass-based dissimilarity. Additionally, the enhanced Splice method, improves the graph construction process, by storing a synopsis of the past, in order to achieve more informed labelling on the local graphs. We evaluate our approach by learning Event Calculus theories for the tasks of human activity recognition, maritime monitoring, and fleet management. The evaluation suggests that our approach outperforms its predecessor, in terms of inferring the missing labels and improving the predictive accuracy of the underlying structure learning system.

Recurrent general type-2 fuzzy neural networks for nonlinear dynamic systems identification

This paper introduces a recurrent general type-2 Takagi–Sugeno–Kang fuzzy neural network (RGT2-TSKFNN) for the identification of nonlinear systems. In the proposed structure, the general type-2 fuzzy set (GT2FS) and a recurrent fuzzy neural network (RFNN) are combined to obviate the data uncertainties. The fuzzy firing strengths in the developed structure are returned to the network input as internal variables. In the proposed structure, GT2FS is utilized to characterize the antecedent parts while the consequent parts are performed using TSK type. The issues of constructing a RGT2-TSKFNN involve type reduction, structure learning as well as parameter learning. An efficient strategy is developed by utilizing alpha-cuts to decompose a GT2FS into several interval type-2 fuzzy sets (IT2FSs). In order to solve the computation time of the type-reduction issue, a direct defuzzification method is used instead of iterative nature of Karnik–Mendel​ (KM) algorithm. Type-2 fuzzy clustering and Lyapunov criteria are utilized for online structure learning as well as the antecedent and consequent parameters, respectively for reducing the number of rules and guaranteeing the stability of the proposed RGT2-TSKFNN. The reported comparative analysis of the simulation results is utilized to estimate the performance of the proposed RGT2-TSKFNN with respect to other popular type-2 FNNs (T2FNNs) methodologies.

Discriminative structure learning of sum–product networks for data stream classification

Sum–product network (SPN) is a deep probabilistic representation that allows for exact and tractable inference. There has been a trend of online SPN structure learning from massive and continuous data streams. However, online structure learning of SPNs has been introduced only for the generative settings so far. In this paper, we present an online discriminative approach for SPNs for learning both the structure and parameters. The basic idea is to keep track of informative and representative examples to capture the trend of time-changing class distributions. Specifically, by estimating the goodness of model fitting of data points and dynamically maintaining a certain amount of informative examples over time, we generate new sub-SPNs in a recursive and top-down manner. Meanwhile, an outlier-robust margin-based log-likelihood loss is applied locally to each data point and the parameters of SPN are updated continuously using most probable explanation (MPE) inference. This leads to a fast yet powerful optimization procedure and improved discrimination capability between the genuine class and rival classes. Empirical results show that the proposed approach achieves better prediction performance than the state-of-the-art online structure learner for SPNs, while promising order-of-magnitude speedup. Comparison with state-of-the-art stream classifiers further proves the superiority of our approach.

A recurrent TSK interval type-2 fuzzy neural networks control with online structure and parameter learning for mobile robot trajectory tracking

This paper focuses on the design of a recurrent Takagi-Sugeno-Kang interval type-2 fuzzy neural network RTSKIT2FNN for mobile robot trajectory tracking problem. The RTSKIT2FNN is incorporating the recurrent frame of internal-feedback loops into interval type-2 fuzzy neural network which uses simple interval type-2 fuzzy sets in the antecedent part and the Takagi-Sugeno-Kang (TSK) type in the consequent part of the fuzzy rule. The antecedent part forms a local internal feedback loop by feeding the membership function of each node in the fuzzification layer to itself. Initially, the rule base in the RTSKIT2FNN is empty, after that, all rules are generated by online structure learning, and all the parameters of the RTSKIT2FNN are updated online using gradient descent algorithm with varied learning rates VLR. Through experimental results, we demonstrate the effectiveness of the proposed RTSKIT2FNN for mobile robot control.

Semi-supervised online structure learning for composite event recognition

Online structure learning approaches, such as those stemming from statistical relational learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel approach for completing the supervision of a semi-supervised structure learning task. We incorporate graph-cut minimisation, a technique that derives labels for unlabelled data, based on their distance to their labelled counterparts. In order to adapt graph-cut minimisation to first order logic, we employ a suitable structural distance for measuring the distance between sets of logical atoms. The labelling process is achieved online (single-pass) by means of a caching mechanism and the Hoeffding bound, a statistical tool to approximate globally-optimal decisions from locally-optimal ones. We evaluate our approach on the task of composite event recognition by using a benchmark dataset for human activity recognition, as well as a real dataset for maritime monitoring. The evaluation suggests that our approach can effectively complete the missing labels and eventually, improve the accuracy of the underlying structure learning system.

Semi-Supervised Online Structure Learning for Composite Event Recognition

Semi-Supervised Online Structure Learning for Composite Event Recognition

Self-evolving fuzzy model-based controller with online structure and parameter learning for hypersonic vehicle

This paper presents a self-evolving fuzzy model-based controller for a generic hypersonic vehicle (HV). The self-evolving fuzzy model can dynamically evolve its rule base through evaluating the influence of a rule and then is introduced to reconstruct the unknown dynamics so that the changing dynamics and unknown uncertainties of the HV are integrated. The control law is designed using the fuzzy identified model instead of the actual HV one. The stability analysis of the whole control system is presented from the Lyapunov function and shows that the tracking errors converge to zero. The bounds of the control inputs are considered in this study and ensured by means of the stable projection-type parameter adaptation laws. To reduce the computation complexity, only the parameters of a ‘winner’ rule closest to the current state are adjusted while those of other rules maintain unvaried. This differs from the existing studies in which the parameters of all rules need to be adjusted. The simulation results under the nominal conditions and parameter uncertainties demonstrate the superior performance of the proposed controller.

The application of an interactively recurrent self-evolving fuzzy CMAC classifier on face detection in color images

This study proposes an interactively recurrent self-evolving fuzzy cerebellar model articulation controller (IRSFCMAC) classifier to solve face detection problems. The learning methods of the proposed classifier are based on simultaneous structure and parameter learning. The structure learning is used to decide the proper input space partition, while the parameter learning is based on gradient descent method. The online structure learning does not need to set any initial structure in advance. In other words, the online structure learning algorithm enables the network along of the problem to efficiently identify the required network structure. The advantages of our proposed IRSFCMAC classifier include (1) using a non-constant differentiable Gaussian basis function to model the hypercube structure; (2) applying an interactively recurrent structure to serve as external loops and internal feedbacks by feeding the hypercube cell (rule) firing strength to itself and other hypercube cells (rules); and (3) requiring fewer computing memory. Finally, experimental results show that the proposed IRSFCMAC classifier is a more adaptive and effective face detection than the other classifiers.

On-line constructive fuzzy sliding-mode control for voice coil motors

In this paper, a voice coil motor (VCM) featuring fast dynamic performance and high position repeatability is developed. To achieve robust VCM control performance under different operating conditions, an on-line constructive fuzzy sliding-mode control (OCFSC) system, which comprises of a main controller and an exponential compensator, is proposed. In the main controller, a fuzzy observer is used to on-line approximate the unknown nonlinear term in the system dynamics with on-line structure learning and parameter learning using a gradient descent algorithm. According to the structure learning mechanism, the fuzzy observer can either increase or decrease the number of fuzzy rules based on tracking performance. The exponential compensator is applied to ensure the system stability with a nonlinear exponential reaching law. Thus, the chattering signal can be alleviated and the convergence of tracking error can be speed up. Finally, the experimental results show that not only the OCFSC system can achieve good position tracking accuracy but also the structure learning ability enables the fuzzy observer to evolve its structure on-line.

An Interval Type-2 Neural Fuzzy Classifier Learned Through Soft Margin Minimization and its Human Posture Classification Application

This paper proposes an interval type-2 neural fuzzy classifier learned through soft margin minimization (IT2NFC-SMM) and applies it to human body posture classification. The IT2NFC-SMM consists of interval type-2, zero-order Takagi–Sugeno (T–S) fuzzy rules established through online structure learning. The antecedent part of the IT2NFC-SMM uses interval type-2 fuzzy sets to decrease the number of rules and manage noisy data. For parameter learning, the consequent parameters are learned through a linear support vector machine (SVM) for soft margin minimization to improve the generalization ability. The proposed SVM-based learning addresses the problem that the orders of the fuzzy rules in computing the outputs of an interval type-2 fuzzy system depend on the consequent values that are unknown in advance. To address this problem, the IT2NFC-SMM uses weighted bound-set boundaries to simplify the type-reduction operation and a novel crisp-to-interval linear SVM learning algorithm. Based on the soft margin minimization, the antecedent parameters are tuned using the gradient descent algorithm. The IT2NFC-SMM is applied to a vision-based human body posture classification system. The system uses two cameras and novel classification features extracted from a silhouette of the human body to classify the four postures of standing, bending, sitting, and lying. The classification performance of the IT2NFC-SMM is verified through results in clean and noisy classification examples and through the posture classification problem, as well as through comparisons with various type-1 and type-2 fuzzy classifiers. The overall result shows that the IT2NFC-SMM achieves higher classification rates with a smaller or similar model size than the classifiers used for comparison, especially for noisy classification problems.

An interval type-2 neural fuzzy chip with on-chip incremental learning ability for time-varying data sequence prediction and system control.

This paper proposes a new circuit to implement a Mamdani-type interval type-2 neural fuzzy chip with on-chip incremental learning ability (IT2NFC-OL) for applications in changing environments. Traditional interval type-2 fuzzy systems use an iterative procedure to find the system outputs, which is computationally expensive, especially for hardware implementation. To address this problem, the IT2NFC-OL uses a simplified type reduction operation to reduce the hardware implementation cost without degrading the learning performance. The software-implemented IT2NFC-OL is characterized by online structure learning and parameter learning using a gradient descent algorithm. The learned fuzzy model is then implemented in a field-programmable gate array (FPGA) chip. The FPGA-implemented IT2NFC-OL performs not only fuzzy inference but also online consequent parameter learning for applications in changing environments. Novel circuits for the computation of system outputs and the update of interval consequent values are proposed. The learning performance of the software-implemented IT2NFC-OL and the on-chip learning ability are verified with applications to time-varying data sequence prediction and system control problems and by comparisons with different software-implemented type-1 and type-2 neural fuzzy systems and interval type-2 fuzzy chips.

Learning Structure Models with Context Information for Visual Tracking

Tracking objects that undergo abrupt appearance changes and heavy occlusions is a challenging problem which conventional tracking methods can barely handle. To address the problem, we propose an online structure learning algorithm that contains three layers: an object is represented by a mixture of online structure models (OSMs) which are learnt from block-based online random forest classifiers (BORFs). BORFs are able to handle occlusion problems since they model local appearances of the target. To further improve the tracking accuracy and reliability, the algorithm utilizes mixture relational models (MRMs) as multi-mode context information to integrate BORFs into OSMs. Furthermore, the mixture construction of OSMs can avoid over-fitting effectively and is more flexible to describe targets. Fusing BORFs with MRMs, OSMs capture the discriminative parts of the target, which guarantees the reliability and robustness of our tracker. In addition, OSMs incorporate with block occlusion reasoning to update our BORFs and MRMs, which can deal with appearance changes and drifting problems effectively. Experiments on challenging videos show that the proposed tracker performs better than several state-of-the-art algorithms.

Reduced Interval Type-2 Neural Fuzzy System Using Weighted Bound-Set Boundary Operation for Computation Speedup and Chip Implementation

This paper proposes a reduced interval type-2 neural fuzzy system using weighted bound-set boundaries (RIT2NFS-WB) for the simplification of type-reduction operations. The objective of this simplification is to reduce the system training time in software implementation and chip size in hardware implementation, especially when the number of rules is large. The antecedent part in the RIT2NFS-WB uses interval type-2 fuzzy sets (IT2FSs), and the consequent part can be of the Takagi-Sugeno-Kang (TSK) or Mamdani type. The RIT2NFS-WB is built through an online structure and parameter learning to improve model accuracy. In addition, the interpretability of the RIT2NFS-WB is improved by considering distributions of the IT2FSs in input variables. A distinguishability-oriented cost function is used in parameter learning to generate distinguishable IT2FSs and improve semantics-based interpretability. For highly overlapped IT2FSs, they are merged to reduce the number of IT2FSs and improve complexity-based interpretability. The software-implemented TSK-type RIT2NFS-WB is hardware-implemented on a field-programmable gate array chip. To accelerate the chip execution speed, the chip utilizes not only the parallel execution properties of fuzzy rules and bound-set boundaries but the pipeline technique as well. In particular, the flexibility of the chip is considered so that no redesign of the circuits is required when the RIT2NFS-WB is applied to different problems. The characteristics of the software- and hardware-implemented RIT2NFS-WB are verified through various examples and comparisons with various type-1 and interval type-2 fuzzy models.

A Mamdani-type Recurrent Interval Type-2 Fuzzy Neural Network for Identification of Dynamic Systems with Measurement Noise

AbstractRecurrent fuzzy neural networks (FNNs) have been widely applied to dynamic system processing problems. However, most recurrent FNNs focus on the use of type-1 fuzzy sets. This paper proposes a Mamdani-type recurrent interval type-2 FNN (M-RIT2FNN) that uses interval type-2 fuzzy sets in both rule antecedent and consequent parts. The reason for using interval type-2 fuzzy sets is to increase network noise resistance. The M-RIT2FNN uses self-feedback loops for memorizing past states and past control inputs of an identified plant. For identification problems, it is unnecessary to know the plant order or input delay number in advance when using the M-RIT2FNN. The M-RIT2FNN identifies a plant via online structure and parameter learning. Simulation results on noisy plant identification and comparisons with different recurrent FNNs verify the advantage of the M-RIT2FNN.

A Self-Evolving Interval Type-2 Fuzzy Neural Network With Online Structure and Parameter Learning

This paper proposes a self-evolving interval type-2 fuzzy neural network (SEIT2FNN) with online structure and parameter learning. The antecedent parts in each fuzzy rule of the SEIT2FNN are interval type-2 fuzzy sets and the fuzzy rules are of the Takagi-Sugeno-Kang (TSK) type. The initial rule base in the SEIT2FNN is empty, and the online clustering method is proposed to generate fuzzy rules that flexibly partition the input space. To avoid generating highly overlapping fuzzy sets in each input variable, an efficient fuzzy set reduction method is also proposed. This method independently determines whether a corresponding fuzzy set should be generated in each input variable when a new fuzzy rule is generated. For parameter learning, the consequent part parameters are tuned by the rule-ordered Kalman filter algorithm for high-accuracy learning performance. Detailed learning equations on applying the rule-ordered Kalman filter algorithm to the SEIT2FNN consequent part learning, with rules being generated online, are derived. The antecedent part parameters are learned by gradient descent algorithms. The SEIT2FNN is applied to simulations on nonlinear plant modeling, adaptive noise cancellation, and chaotic signal prediction. Comparisons with other type-1 and type-2 fuzzy systems in these examples verify the performance of the SEIT2FNN.

Nonlinear systems control using self-constructing wavelet networks

This paper describes a self-constructing wavelet network (SCWN) controller for nonlinear systems control. The proposed SCWN controller has a four-layer structure. We adopt the orthogonal wavelet functions as its node functions. An online learning algorithm, structure learning and parameter learning, allows the dynamic determining of the number of wavelet bases, and adjusting the shape of the wavelet bases and the connection weights. The SCWN controller is a highly autonomous system. Initially, there are no hidden nodes. They are created and begin to grow as learning proceeds. Computer simulations have been conducted to illustrate the performance and applicability of the proposed learning scheme.

A Self-Evolving Interval Type-2 Fuzzy Neural Network for Nonlinear Systems Identification

Abstract This paper proposes a Self-Evolving Interval Type-2 Fuzzy Neural Network (SEIT2FNN) for nonlinear systems identification. The SEIT2FNN has both on-line structure and parameter learning abilities. The antecedent parts in each fuzzy rule of the SEIT2FNN are interval type-2 fuzzy sets and the fuzzy rules are of the Takagi-Sugeno-Kang (TSK) type. An on-line clustering method is proposed to generate fuzzy rules which flexibly partition the input space. For parameter learning, the consequent part parameters are tuned by a rule-ordered Kalman filter algorithm for high accuracy learning performance. The antecedent part parameters are learned by gradient descent algorithms. Comparisons with identification using other type-1 and type-2 fuzzy systems verify the performance of the SEIT2FNN.

Mold temperature control of a rubber injection-molding machine by TSK-type recurrent neural fuzzy network

Practical mold temperature control of a rubber injection-molding machine is studied in this paper. The controller used is a recurrent fuzzy network called Takagi–Sugeno–Kang (TSK)-type recurrent fuzzy network (TRFN), which is characterized by its recurrent structure, on-line structure and parameter learning. Due to the powerful learning ability of TRFN, a simple controller design scheme using direct inverse configuration is proposed. With recurrent structure in TRFN, no a priori knowledge of the molding machine order is required. The designed TRFN controller performs well even if the sampling interval is different from the original one used for training. The design of TRFN consists of off-line and on-line training. For off-line learning, structure and parameter of TRFN are learned, and the consequent part parameters are tuned by Kalman filter algorithm. On-line learning is performed to fine tune the consequent parameters of TRFN and achieve a better control performance with the use a simple gradient descent algorithm. Practical experiments and comparisons with other types of controllers demonstrate the performance of the proposed TRFN controller.

Direction of arrival estimation based on phase differences using neural fuzzy network

A new high-resolution direction of arrival (DOA) estimation technique using a neural fuzzy network based on phase difference (PD) is proposed. The conventional DOA estimation method such as MUSIC and MLE, are computationally intensive and difficult to implement in real time. To attack these problems, neural networks have become popular for DOA estimation. However, the normal neural networks such as the multilayer perceptron (MLP) and radial basis function network (RBFN) usually produce the extra problems of low convergence speed and/or large network size (i.e., the number of network parameters is large). Also, the may to decide the network structure is heuristic. To overcome these defects and take use of neural learning ability, a powerful self-constructing neural fuzzy inference network (SONFIN) is used to develop a new DOA estimation algorithm. By feeding the PDs of the received radar-array signals, the trained SONFIN can give high-resolution DOA estimation. The proposed scheme is thus called PD-SONFIN. This new algorithm avoids the need of empirically determining the network size and parameters in normal neural networks due to the powerful on-line structure and parameter learning ability of SONFIN. The PD-SONFIN can always find itself an economical network size in the fast learning process. Our simulation results show that the performance of the new algorithm is superior to the RBFN in terms of convergence accuracy, estimation accuracy, sensitivity to noise, and network size.