Incremental Learning Algorithm Research Articles

Online Prediction Method of Transmission Line Icing Based on Robust Seasonal Decomposition of Time Series and Bilinear Temporal–Spectral Fusion and Improved Beluga Whale Optimization Algorithm–Least Squares Support Vector Regression

Due to the prevalent challenges of inadequate accuracy, unstandardized parameters, and suboptimal efficiency with regard to icing prediction, this study introduces an innovative online method for icing prediction based on Robust STL–BTSF and IBWO–LSSVR. Firstly, this study adopts the Robust Seasonal Decomposition of Time Series and Bilinear Temporal–Spectral Fusion (Robust STL–BTSF) approach, which is demonstrably effective for short-term and limited sample data preprocessing. Subsequently, injecting a multi-faceted enhancement approach to the Beluga Whale Optimization algorithm (BWO), which integrates a nonlinear balancing factor, a population optimization strategy, a whale fall mechanism, and an ascendant elite learning scheme. Then, using the Improved BWO (IBWO) above to optimize the key hyperparameters of Least Squares Support Vector Regression (LSSVR), a superior offline predictive part is constructed based on this approach. In addition, an Incremental Online Learning algorithm (IOL) is imported. Integrating the two parts, the advanced online icing prediction model for transmission lines is built. Finally, simulations based on actual icing data unequivocally demonstrate that the proposed method markedly enhances both the accuracy and speed of predictions, thereby presenting a sophisticated solution for the icing prediction on the transmission lines.

Probabilistic neural networks for incremental learning over time-varying streaming data with application to air pollution monitoring

This paper proposes a novel algorithm for incremental learning over streaming data in a non-stationary environment. The idea refers to the applicability of Probabilistic Neural Networks (PNNs), commonly used as a fast and robust method for solving classification problems in a stationary scenario. It is well known that PNNs have solid mathematical foundations but they fail in the non-stationary scenario and are not able to deal with concept drift. In this paper, the method based on the orthogonal series expansions of unknown probability densities is proposed. This approach significantly extends the applicability of the PNNs. The nonparametric procedures called Incremental Probabilistic Neural Networks (IPNNs) for tracking drifting probability densities or conditional class densities, in the case of classification, are presented. We prove their convergence as the stream data size is growing to infinity. More precisely, we show convergence in probability and with probability one as the sample size tends to infinity. Compared to the existing literature, almost exclusively based on various heuristics, the proposed method is mathematically justified and has solid theoretical foundations. Simulations performed on synthetic and air pollution data confirm the effectiveness of the algorithm.

An information entropy-based fuzzy stochastic configuration network for robust data modeling

In data-driven modeling, the generalization performance of a neural network based on the minimum mean square error criterion may be affected when the samples contain noise. To handle uncertain data modeling problems, this study contributes a novel robust data modeling method called information entropy-based fuzzy stochastic configuration networks (IEFSCNs), which incorporates the maximum correntropy criterion (MCC) to reflect the squared error of the training sample set and uses the quantized minimum error entropy (QMEE) criterion to mine the distribution structure of the modeling data and suppress outlier data. Fuzzy systems are designed to generate nodes in the fuzzy transformation layer. A nonlinear enhancement layer is established as a hidden layer to enhance the representational capacity of the target system. The fuzzy transformation and nonlinear enhancement layers are fully connected to determine the outputs of the IEFSCNs. In addition, an incremental parameter learning algorithm is developed based on a supervision mechanism for optimizing the cost function, and its convergence property is demonstrated. Owing to the superiorities of the cost function by synthesizing MCC and QMEE, IEFSCNs can improve the modeling accuracy and reduce noise suppression. The performances are evaluated using various regression and classification tasks. Compared with existing methods, including SCNs, RVFL, F-SCNs, RSC-KDE, RSC-MCC, and SM-RSC, the proposed method demonstrates an effective robust modeling performance by reducing network structure redundancy while enhancing its ability to suppress noisy data.

Development and Validation of Adaptable Skin Cancer Classification System Using Dynamically Expandable Representation.

Skin cancer is a prevalent type of malignancy, necessitating efficient diagnostic tools. This study aimed to develop an automated skin lesion classification model using the dynamically expandable representation (DER) incremental learning algorithm. This algorithm adapts to new data and expands its classification capabilities, with the goal of creating a scalable and efficient system for diagnosing skin cancer. The DER model with incremental learning was applied to the HAM10000 and ISIC 2019 datasets. Validation involved two steps: initially, training and evaluating the HAM10000 dataset against a fixed ResNet-50; subsequently, performing external validation of the trained model using the ISIC 2019 dataset. The model's performance was assessed using precision, recall, the F1-score, and area under the precision-recall curve. The developed skin lesion classification model demonstrated high accuracy and reliability across various types of skin lesions, achieving a weighted-average precision, recall, and F1-score of 0.918, 0.808, and 0.847, respectively. The model's discrimination performance was reflected in an average area under the curve (AUC) value of 0.943. Further external validation with the ISIC 2019 dataset confirmed the model's effectiveness, as shown by an AUC of 0.911. This study presents an optimized skin lesion classification model based on the DER algorithm, which shows high performance in disease classification with the potential to expand its classification range. The model demonstrated robust results in external validation, indicating its adaptability to new disease classes.

Incremental and sequence learning algorithms for weighted regularized extreme learning machines

Incremental and sequence learning algorithms for weighted regularized extreme learning machines

Cross-Class Feature Augmentation for Class Incremental Learning

We propose a novel class incremental learning approach, which incorporates a feature augmentation technique motivated by adversarial attacks. We employ a classifier learned in the past to complement training examples of previous tasks. The proposed approach has an unique perspective to utilize the previous knowledge in class incremental learning since it augments features of arbitrary target classes using examples in other classes via adversarial attacks on a previously learned classifier. By allowing the Cross-Class Feature Augmentations (CCFA), each class in the old tasks conveniently populates samples in the feature space, which alleviates the collapse of the decision boundaries caused by sample deficiency for the previous tasks, especially when the number of stored exemplars is small. This idea can be easily incorporated into existing class incremental learning algorithms without any architecture modification. Extensive experiments on the standard benchmarks show that our method consistently outperforms existing class incremental learning methods by significant margins in various scenarios, especially under an environment with an extremely limited memory budget.

Survey of continuous deep learning methods and techniques used for incremental learning

Neural networks and deep learning algorithms are designed to function similarly to biological synaptic structures. However, classical deep learning algorithms fail to fully capture the need for continuous learning; this has led to the advent of incremental learning. Incremental learning adds new challenges that are handled differently by modern state-of-the-art approaches. Some of these include: utilization of network memory as additional knowledge increases the size of the network, open-set recognition to be able to identify unrecognized information, and efficient knowledge distillation as most incremental learning algorithms are prone to catastrophic forgetting of previously learned knowledge. Recent advancements achieve incremental learning through a multitude of methods. Most methods are characterized by augmenting the normal algorithm of neural network training by both directly modifying the neural network structure and by adding additional learning steps. This paper analyzes and provides a comprehensive survey of existing methods and various techniques used for incremental learning. A novel categorization of the methods is also introduced based on recent trends of the state-of-the-art solutions. The study focuses on methods that provide incremental learning success as well as discusses emerging patterns in new research.

A Procedural Constructive Learning Mechanism with Deep Reinforcement Learning for Cognitive Agents

Recent advancements in AI and deep learning have created a growing demand for artificial agents capable of performing tasks within increasingly complex environments. To address the challenges associated with continuous learning constraints and knowledge capacity in this context, cognitive architectures inspired by human cognition have gained significance. This study contributes to existing research by introducing a cognitive-attentional system employing a constructive neural network-based learning approach for continuous acquisition of procedural knowledge. We replace an incremental tabular Reinforcement Learning algorithm with a constructive neural network deep reinforcement learning mechanism for continuous sensorimotor knowledge acquisition, thereby enhancing the overall learning capacity. The primary emphasis of this modification centers on optimizing memory utilization and reducing training time. Our study presents a learning strategy that amalgamates deep reinforcement learning with procedural learning, mirroring the incremental learning process observed in human sensorimotor development. This approach is embedded within the CONAIM cognitive-attentional architecture, leveraging the cognitive tools of CST. The proposed learning mechanism allows the model to dynamically create and modify elements in its procedural memory, facilitating the reuse of previously acquired functions and procedures. Additionally, it equips the model with the capability to combine learned elements to effectively adapt to complex scenarios. A constructive neural network was employed, initiating with an initial hidden layer comprising one neuron. However, it possesses the capacity to adapt its internal architecture in response to its performance in procedural and sensorimotor learning tasks, inserting new hidden layers or neurons. Experimentation conducted through simulations involving a humanoid robot demonstrates the successful resolution of tasks that were previously unsolved through incremental knowledge acquisition. Throughout the training phase, the constructive agent achieved a minimum of 40% greater rewards and executed 8% more actions when compared to other agents. In the subsequent testing phase, the constructive agent exhibited a 15% increase in the number of actions performed in contrast to its counterparts.

Graph-based few-shot incremental learning algorithm for unknown class detection

Few-shot learning, a promising technique for acquiring new concepts from limited data, assumes that testing samples belong to “unknown classes” and are regarded as new knowledge. However, real-world scenarios introduce uncertainty about the class membership of testing samples. To address this uncertainty, we propose a novel challenge of few-shot incremental unknown class detection, aligning more closely with practical situations. Open set recognition can classify known class samples and reject unknown class samples to mitigate the uncertainty, but it struggles to address the critical limitation of having few available samples. To tackle both uncertainty and limitation, we propose a graph-based few-shot incremental learning algorithm for unknown class detection, which includes four components. First, a feature extractor learns from the base dataset during training and is subsequently fixed for embedding node features from the novel dataset during inference. Then, embedded node features, along with their corresponding prototypes, contribute to graph generation and edge construction. Third, a mixed-rejection strategy is proposed to determine the class membership of testing samples. Finally, a novel class is treated as a new known class, engaging the embedded node features in graph update and edge reconstruction. Evaluation on benchmark datasets with varying structures, including USTC-TFC2016 and miniImageNet datasets, demonstrates that our proposed algorithm outperforms classical open set recognition algorithm in few-shot incremental learning for unknown class detection, which offers promising performance and potential for practical applications in real-world scenarios.

BASS: Broad Network Based on Localized Stochastic Sensitivity.

The training of the standard broad learning system (BLS) concerns the optimization of its output weights via the minimization of both training mean square error (MSE) and a penalty term. However, it degrades the generalization capability and robustness of BLS when facing complex and noisy environments, especially when small perturbations or noise appear in input data. Therefore, this work proposes a broad network based on localized stochastic sensitivity (BASS) algorithm to tackle the issue of noise or input perturbations from a local perturbation perspective. The localized stochastic sensitivity (LSS) prompts an increase in the network's noise robustness by considering unseen samples located within a Q -neighborhood of training samples, which enhances the generalization capability of BASS with respect to noisy and perturbed data. Then, three incremental learning algorithms are derived to update BASS quickly when new samples arrive or the network is deemed to be expanded, without retraining the entire model. Due to the inherent superiorities of the LSS, extensive experimental results on 13 benchmark datasets show that BASS yields better accuracies on various regression and classification problems. For instance, BASS uses fewer parameters (12.6 million) to yield 1% higher Top-1 accuracy in comparison to AlexNet (60 million) on the large-scale ImageNet (ILSVRC2012) dataset.

Kernel Path for Semisupervised Support Vector Machine.

Semisupervised support vector machine (S3VM) is a powerful semisupervised learning model that can use large amounts of unlabeled data to train high-quality classification models. The choice of kernel parameters in the kernel function determines the mapping between the input space and the feature space and is crucial to the performance of the S3VM. Kernel path algorithms have been widely recognized as one of the most efficient tools to trace the solutions with respect to a kernel parameter. However, existing kernel path algorithms are limited to convex problems, while S3VM is nonconvex problem. To address this challenging problem, in this article, we first propose a kernel path algorithm of S3VM (KPS3VM), which can track the solutions of the nonconvex S3VM with respect to a kernel parameter. Specifically, we estimate the position of the breakpoint by monitoring the change of the sample sets. In addition, we also use an incremental and decremental learning algorithm to deal with the Karush-Khun-Tucker violating samples in the process of tracking the solutions. More importantly, we prove the finite convergence of our KPS3VM algorithm. Experimental results on various benchmark datasets not only validate the effectiveness of our KPS3VM algorithm but also show the advantage of choosing the optimal kernel parameters.

OEBench: Investigating Open Environment Challenges in Real-World Relational Data Streams

How to get insights from relational data streams in a timely manner is a hot research topic. Data streams can present unique challenges, such as distribution drifts, outliers, emerging classes, and changing features, which have recently been described as open environment challenges for machine learning. While existing studies have been done on incremental learning for data streams, their evaluations are mostly conducted with synthetic datasets. Thus, a natural question is how those open environment challenges look like and how existing incremental learning algorithms perform on real-world relational data streams. To fill this gap, we develop an Open Environment Benchmark named OEBench to evaluate open environment challenges in real-world relational data streams. Specifically, we investigate 55 real-world relational data streams and establish that open environment scenarios are indeed widespread, which presents significant challenges for stream learning algorithms. Through benchmarks with existing incremental learning algorithms, we find that increased data quantity may not consistently enhance the model accuracy when applied in open environment scenarios, where machine learning models can be significantly compromised by missing values, distribution drifts, or anomalies in real-world data streams. The current techniques are insufficient in effectively mitigating these challenges brought by open environments. More researches are needed to address real-world open environment challenges. All datasets and code are open-sourced in https://github.com/Xtra-Computing/OEBench.

A data-knowledge-driven interval type-2 fuzzy neural network with interpretability and self-adaptive structure

Interval type-2 fuzzy neural networks (IT2FNNs) have gained sustainable attention and wide applications because of their power of adaptive fuzzy modeling. Although the existing models accurately simulate the relationships between inputs and outputs by various adaptive learning algorithms, the parameters and fuzzy rules characterizing the model behaviors no longer have explicit interpretations. To achieve good interpretability and generalization performance, this paper proposes a novel data-knowledge-driven IT2FNN (DKIT2FNN). Unlike accuracy-oriented model fitting, DKIT2FNN pioneers the use of famous association rule mining techniques to automatically explore the interpretable rule knowledge, which intuitively presents conditional relation patterns among features. Using the obtained knowledge, the DKIT2FNN model is accomplished in a single-hidden layer network framework, in which the antecedent structure of each fuzzy rule is determined by the relation patterns provided by rule knowledge. In addition, an adversarial incremental learning algorithm is proposed for the self-organizing modeling of DKIT2FNN. Besides automatically assembling a compact rule base, it integrates the label adversarial training to reinforce the model's capability against label noise. We validate the DKIT2FNN on classification benchmark datasets with label noise and compare it with state-of-the-art non-fuzzy and neuro-fuzzy methods. The results demonstrate that the proposed DKIT2FNN achieves robust classification performance and good interpretability.

Global Model Selection via Solution Paths for Robust Support Vector Machine.

Robust support vector machine (RSVM) using ramp loss provides a better generalization performance than traditional support vector machine (SVM) using hinge loss. However, the good performance of RSVM heavily depends on the proper values of regularization parameter and ramp parameter. Traditional model selection technique with gird search has extremely high computational cost especially for fine-grained search. To address this challenging problem, in this paper, we first propose solution paths of RSVM (SPRSVM) based on the concave-convex procedure (CCCP) which can track the solutions of the non-convex RSVM with respect to regularization parameter and ramp parameter respectively. Specifically, we use incremental and decremental learning algorithms to deal with the Karush-Khun-Tucker violating samples in the process of tracking the solutions. Based on the solution paths of RSVM and the piecewise linearity of model function, we can compute the error paths of RSVM and find the values of regularization parameter and ramp parameter, respectively, which corresponds to the minimum cross validation error. We prove the finite convergence of SPRSVM and analyze the computational complexity of SPRSVM. Experimental results on a variety of benchmark datasets not only verify that our SPRSVM can globally search the regularization and ramp parameters respectively, but also show a huge reduction of computational time compared with the grid search approach.

When Broad Learning System Meets Label Noise Learning: A Reweighting Learning Framework.

Broad learning system (BLS) is a novel neural network with efficient learning and expansion capacity, but it is sensitive to noise. Accordingly, the existing robust broad models try to suppress noise by assigning each sample an appropriate scalar weight to tune down the contribution of noisy samples in network training. However, they disregard the useful information of the noncorrupted elements hidden in the noisy samples, leading to unsatisfactory performance. To this end, a novel BLS with adaptive reweighting (BLS-AR) strategy is proposed in this article for the classification of data with label noise. Different from the previous works, the BLS-AR learns for each sample a weight vector rather than a scalar weight to indicate the noise degree of each element in the sample, which extends the reweighting strategy from sample level to element level. This enables the proposed network to precisely identify noisy elements and thus highlight the contribution of informative ones to train a more accurate representation model. Thanks to the separability of the model, the proposed network can be divided into several subnetworks, each of which can be trained efficiently. In addition, three corresponding incremental learning algorithms of the BLS-AR are developed for adding new samples or expanding the network. Substantial experiments are conducted to explicate the effectiveness and robustness of the proposed BLS-AR model.

A Memory-Efficient Federated Kernel Support Vector Machine for Edge Devices.

A federated learning (FL) scheme (denoted as Fed-KSVM) is designed to train kernel support vector machines (SVMs) over multiple edge devices with low memory consumption. To decompose the training process of kernel SVM, each edge device first constructs high-dimensional random feature vectors of its local data, and then trains a local SVM model over the random feature vectors. To reduce the memory consumption on each edge device, the optimization problem of the local model is divided into several subproblems. Each subproblem only optimizes a subset of the model parameters over a block of random feature vectors with a low dimension. To achieve the same optimal solution to the original optimization problem, an incremental learning algorithm called block boosting is designed to solve these subproblems sequentially. After training of the local models, the central server constructs a global SVM model by averaging the model parameters of these local models. Fed-KSVM only increases the iterations of training the local SVM models to save the memory consumption, while the communication rounds between the edge devices and the central server are not affected. Theoretical analysis shows that the kernel SVM model trained by Fed-KSVM converges to the optimal model with a linear convergence rate. Because of such a fast convergence rate, Fed-KSVM reduces the communication cost during training by up to 99% compared with the centralized training method. The experimental results also show that Fed-KSVM reduces the memory consumption on the edge devices by nearly 90% while achieving the highest test accuracy, compared with the state-of-the-art schemes.

Identification of optimal locations of adaptive traffic signal control using heuristic methods

Adaptive Traffic Signal Control (ATSC) adjusts signal timings to real-time traffic measurements, increasing operational efficiency within a network. However, ATSC is both expensive to install and operate making it infeasible to deploy at all signalized intersections within a network. This study presents a bi-level optimization framework that applies heuristic methods to identify a limited set of locations for ATSC deployment within an urban network. At the upper-level, the Population Based Incremental Learning (PBIL) algorithm is employed to generate, evaluate, learn, and update different ATSC configurations. The lower-level uses the delay-based Max-Pressure algorithm to simulate the ATSC configuration within a microsimulation platform. The study proposes improvements to the PBIL algorithm by considering constraints on the maximum number of intersections for ATSC deployment and incorporates prior information about the intersection performance (i.e., informed search). Simulation results on the traffic network of State College, PA reveal that the proposed PBIL algorithm consistently outperforms baseline methods that select locations only based on queue-lengths or delays in terms of reducing overall network travel times. The study also reveals that intersections experiencing the highest delays or longest queues are not always the best candidates for ATSC. Moreover, applying ATSC at all intersections does not always provide the best performance; in fact, ATSC applied to some locations could increase travel times by contributing additional congestion downstream. Additionally, the modified PBIL algorithm with the informed search strategy is more efficient at identifying promising solutions suggesting it can be readily applied to more generalized optimization problems.

Decentralized and collaborative machine learning framework for IoT

Decentralized machine learning has recently been proposed as a potential solution to the security issues of the canonical federated learning approach. In this paper, we propose a decentralized and collaborative machine learning framework specially oriented to resource-constrained devices, usual in IoT deployments. With this aim we propose the following construction blocks. First, an incremental learning algorithm based on prototypes that was specifically implemented to work in low-performance computing elements. Second, two random-based protocols to exchange the local models among the computing elements in the network. Finally, two algorithmics approaches for prediction and prototype creation. This proposal was compared to a typical centralized incremental learning approach in terms of accuracy, training time and robustness with very promising results.

Adaptive hierarchical positioning error compensation for long-term service of industrial robots based on incremental learning with fixed-length memory window and incremental model reconstruction

Industrial robots have been extensively used in industry, however, geometric errors mainly caused by connecting rod parameter error and non-geometric errors caused by deflection and friction, etc., limit its application in high-accuracy machining. Aiming at addressing these two types of errors, parametric methods for error compensation based on the kinematic model and non-parametric methods of directly establishing the mapping relationship between the actual and target poses of the robot end-effector are investigated and proposed. Currently both types of methods are mainly offline and will be no longer applicable when the pose of the end-effector in the workspace changes dramatically or the working performance of the robot degrades. Thus, to compensate the positioning error of an industrial robot during long-term operation, this research proposes an adaptive hierarchical compensation method based on fixed-length memory window incremental learning and incremental model reconstruction. Firstly, the correlation between positioning errors and robot poses is studied, a calibration sample library is created, and thus the actively evaluating mechanism of the pose mapping model is established to overcome the problem of the robot’ workspace having a differential distribution of error levels. Then, an incremental learning algorithm with fixed-length memory window and an incremental model reconstruction algorithm are designed to optimize the pose mapping model in terms of its parameters and architecture and overcome the problem that the performance degradation of the robot exacerbates the positioning error and affects the applicability of the pose mapping model, ensuring that the pose mapping model runs stably above the target accuracy level. Finally, the proposed method is applied to the long-term compensation case of a Stäubli industrial robot and a UR robot, and compared to state-of-art methods. Verification results show the proposed method reduces the position error of the Stäubli robot from 0.85mm to 0.13mm and orientation error from 0.68° to 0.07°, as well as reduces the position error of the UR robot from 2.11mm to 0.17mm, demonstrating that the proposed method works in real world scenarios and outperforms similar methods.

Mutual dimensionless improved bearing fault diagnosis based on Bp-increment broad learning system in computer vision

Efficient and accurate diagnosis of rotating machinery in the petrochemical industry is crucial for ensuring normal machinery operation. However, the nonlinear and non-stationary vibration signals generated by machinery in harsh environments pose significant challenges in distinguishing fault signals from normal ones. Although several fault diagnosis methods based on mutual dimensionless indicators (MDI) have been proposed, they often fail to achieve effective and accurate health monitoring. Hence, this paper proposes a BP-Incremental Broad Learning System (BP-INBLS) model based on the combined synergistic of two modules, to address the existing challenges. Firstly, a new mutual dimensionless indicator (VMDI) with high sensitivity and low overlap is refactored. Secondly, leveraging the advantages of incremental learning algorithms, a novel Broad Learning System (BLS) model for quickly identifying different fault types is constructed. Finally, the proposed method is validated using multiple datasets and verified through a comparative analysis with a published method based on dimensionless indicators (DI). The results demonstrate the effectiveness of the proposed method in fault diagnosis.