Incremental Learning Framework Research Articles

Self-organizing hierarchical incremental learning framework and universal approximation analysis based on stochastic configuration mechanism

Conventional machine learning algorithms face significant limitations when dealing with high-dimensional data. Besides, deep learning models often require substantial computational resources and have a high processing time despite their excellent performance. Hence, this paper proposes an expanded stochastic configuration network and a self-organizing hierarchical incremental learning (SHIL) framework to overcome these challenges. Specifically, this study introduces a novel supervised hierarchical clustering tree based on the minimum redundancy maximum correlation algorithm, which mines internal data structures to construct diverse hierarchies. Subsequently, by exploiting the parent-child node relationships in the tree structure, SHIL defines the maximum number of nodes as the switching condition between levels uses the supervisory mechanism as the parameter selection criterion, and adopts the tolerance error as the termination criterion for the training. Furthermore, the universal approximation property of the SHIL framework is provided. The proposed SHIL framework is validated on several benchmark datasets, image datasets, and industrial robot cases, with the corresponding experimental results demonstrating that SHIL significantly improves computational efficiency and ensures high accuracy.

Erratum to “Two-dimensional hybrid incremental learning (2DHIL) framework for semantic segmentation of skin tissues” [Image and Vision Computing. Vol148 (2024) 105098]

Erratum to “Two-dimensional hybrid incremental learning (2DHIL) framework for semantic segmentation of skin tissues” [Image and Vision Computing. Vol148 (2024) 105098]

CCSI: Continual Class-Specific Impression for data-free class incremental learning

In real-world clinical settings, traditional deep learning-based classification methods struggle with diagnosing newly introduced disease types because they require samples from all disease classes for offline training. Class incremental learning offers a promising solution by adapting a deep network trained on specific disease classes to handle new diseases. However, catastrophic forgetting occurs, decreasing the performance of earlier classes when adapting the model to new data. Prior proposed methodologies to overcome this require perpetual storage of previous samples, posing potential practical concerns regarding privacy and storage regulations in healthcare. To this end, we propose a novel data-free class incremental learning framework that utilizes data synthesis on learned classes instead of data storage from previous classes. Our key contributions include acquiring synthetic data known as Continual Class-Specific Impression (CCSI) for previously inaccessible trained classes and presenting a methodology to effectively utilize this data for updating networks when introducing new classes. We obtain CCSI by employing data inversion over gradients of the trained classification model on previous classes starting from the mean image of each class inspired by common landmarks shared among medical images and utilizing continual normalization layers statistics as a regularizer in this pixel-wise optimization process. Subsequently, we update the network by combining the synthesized data with new class data and incorporate several losses, including an intra-domain contrastive loss to generalize the deep network trained on the synthesized data to real data, a margin loss to increase separation among previous classes and new ones, and a cosine-normalized cross-entropy loss to alleviate the adverse effects of imbalanced distributions in training data. Extensive experiments show that the proposed framework achieves state-of-the-art performance on four of the public MedMNIST datasets and in-house echocardiography cine series, with an improvement in classification accuracy of up to 51% compared to baseline data-free methods. Our code is available at https://github.com/ubc-tea/Continual-Impression-CCSI.

RRA-FFSCIL: Inter-intra classes representation and relationship augmentation federated few-shot incremental learning

Federated learning (FL), as a distributed machine learning paradigm, enables on-device model training and inference without data updates or privacy breaches, promoting edge intelligence in Industrial Internet of Things (IIoTs) applications. However, FL applying in IIoTs faces some challenges in practice: (1) the catastrophic forgetting of old data resulting from a streaming reception for arriving data; (2) the forgetting exacerbated by the non-i.i.d. data distribution across devices; (3) the overfitting caused by insufficient sample size of new data (i.e., few samples). To address the above problems, we propose an inter-intra classes representation and relationship augmentation federated few-shot incremental learning framework called RRA-FFSCIL, effectively mitigating catastrophic forgetting from local and global perspectives. Specifically, forgetting from streaming data is alleviated by designed modules in the client side of RRA-FFSCIL: a random few-shot (FS) episodic adaptation strategy, a class-projection relation strengthening loss function, and a class-aware distribution balancing loss function. Meanwhile, the mitigation of global forgetting from non-i.i.d. of distributions across devices thanks to selecting the best old models on the server to assist local class relation enhancement. We conduct extensive experiments on three datasets (cifar100, tinyImageNet, miniImageNet) to validate the effectiveness and superiority of RRA-FFSCIL in addressing the forgetting problem associated with streaming data.

Two-dimensional hybrid incremental learning (2DHIL) framework for semantic segmentation of skin tissues

This study aims to enhance the robustness and generalization capability of a deep learning transformer model used for segmenting skin carcinomas and tissues through the introduction of incremental learning. Deep learning AI models demonstrate their claimed performance only for tasks and data types for which they are specifically trained. Their performance is severely challenged for the test cases which are not similar to training data thus questioning their robustness and ability to generalize. Moreover, these models require an enormous amount of annotated data for training to achieve desired performance. The availability of large annotated data, particularly for medical applications, is itself a challenge. Despite efforts to alleviate this limitation through techniques like data augmentation, transfer learning, and few-shot training, the challenge persists. To address this, we propose refining the models incrementally as new classes are discovered and more data becomes available, emulating the human learning process. However, deep learning models face the challenge of catastrophic forgetting during incremental training. Therefore, we introduce a two-dimensional hybrid incremental learning framework for segmenting non-melanoma skin cancers and tissues from histopathology images. Our approach involves progressively adding new classes and introducing data of varying specifications to introduce adaptability in the models. We also employ a combination of loss functions to facilitate new learning and mitigate catastrophic forgetting. Our extended experiments demonstrate significant improvements, with an F1 score reaching 91.78, mIoU of 93.00, and an average accuracy of 95%. These findings highlight the effectiveness of our incremental learning strategy in enhancing the robustness and generalization of deep learning segmentation models while mitigating catastrophic forgetting.

Towards retraining-free RNA modification prediction with incremental learning

RNA modifications are important for deciphering the function of cells and their regulatory mechanisms. In recent years, researchers have developed many deep learning methods to identify specific modifications. However, these methods require model retraining for each new RNA modification and cannot progressively identify the newly identified RNA modifications. To address this challenge, we propose an innovative incremental learning framework that incorporates multiple incremental learning methods. Our experimental results confirm the efficacy of incremental learning strategies in addressing the RNA modification challenge. By uniquely targeting 10 RNA modification types in a class incremental setting, our framework exhibits superior performance. Notably, it can be extended to new category methylation predictions without the need for retraining with previous data, improving computational efficiency. Through the accumulation of knowledge, the model is able to evolve and continuously learn the differences across methylation, mitigating the problem of catastrophic forgetting during deep learning model training. Overall, our framework provides various alternatives to enhance the prediction of novel RNA modifications and illuminates the potential of incremental learning in tacking numerous genome data.

An Incremental Knowledge Learning Framework for Continuous Defect Detection

An Incremental Knowledge Learning Framework for Continuous Defect Detection

An IoMT-Based Incremental Learning Framework With a Novel Feature Selection Algorithm for Intelligent Diagnosis in Smart Healthcare

An IoMT-Based Incremental Learning Framework With a Novel Feature Selection Algorithm for Intelligent Diagnosis in Smart Healthcare

Let model keep evolving: Incremental learning for encrypted traffic classification

Encrypted Traffic Classification (ETC) is valuable for many network management and security solutions as it provides insights into applications active on the network. However, the network environment constantly evolves, and new applications emerge in an endless stream daily, which gradually makes well-trained ETC models ineffective. The conventional approach to adapting new applications is to re-train the models on a re-formed dataset with both pre-existing and new application samples. The major limitation is that requiring redundant computing resources and sufficient storage spaces. In this work, we propose an Incremental Learning (IL) framework based on multi-view sequences fusion, MISS, to keep ETC models evolving with new applications. The key novelty of MISS is three-fold: extract cross-view information from multi-view sequences to capture sufficient knowledge; propose an exemplar selection algorithm from communication patterns to reduce redundant consumption; design a pair of branches from the learnability of parameters to mitigate accuracy loss during evolution. MISS outperforms the existing IL methods of ETC, and the state-of-the-art ETC models using the classic IL framework, on the real-world network traffic datasets, which achieves satisfactory improvements of 11.37%↑ and 1.58%↑. Furthermore, we comprehensively perform incremental experiments to evaluate the evolution ability of MISS, which is able to select representative exemplars of old applications, counteract the adverse effects of homogeneous applications, and keep evolving with unknown applications.

Open set maize seed variety classification using hyperspectral imaging coupled with a dual deep SVDD-based incremental learning framework

Open set maize seed variety classification using hyperspectral imaging coupled with a dual deep SVDD-based incremental learning framework

A crowdsourcing-based incremental learning framework for automated essays scoring

Automated Essay Scoring (AES) is a challenging topic in Natural Language Processing. Recently, deep learning models have achieved remarkable performance for the AES task. However, applying deep learning models to the AES system in practice is expensive when both data collection and model training are taken into consideration. This paper aims to tackle this problem by proposing the Crowdsourcing-based Automated Essay Scoring (CAES) framework. The proposed framework gradually collects data through crowdsourcing and incrementally trains the AES models. In particular, we propose the Incremental Learning with Dynamic Exemplar Herding (ILDEH) approach to simultaneously tackle catastrophic forgetting and concept drift. The proposed approach dynamically updates the exemplar set by the Dynamic Exemplar Herding algorithm to obtain the best approximation of the overall data distribution and selectively apply knowledge distillation on the model outputs by Linear Outlier Suppression loss to retain the learned knowledge. Moreover, we use a lightweight AES model for effective and efficient essay scoring. The experimental results show that our proposed ILDEH approach outperforms other strong baseline approaches for the AES task. Moreover, the CAES framework is able to steadily improve the AES performance in the crowdsourcing environment with only 10.6% training time of the conventional approach. Further analysis shows that one single CPU server can support daily updates of more than 300 AES models, which is sufficient for most practical AES systems.

Incremental Learning for Online Data Using QR Factorization on Convolutional Neural Networks.

Catastrophic forgetting, which means a rapid forgetting of learned representations while learning new data/samples, is one of the main problems of deep neural networks. In this paper, we propose a novel incremental learning framework that can address the forgetting problem by learning new incoming data in an online manner. We develop a new incremental learning framework that can learn extra data or new classes with less catastrophic forgetting. We adopt the hippocampal memory process to the deep neural networks by defining the effective maximum of neural activation and its boundary to represent a feature distribution. In addition, we incorporate incremental QR factorization into the deep neural networks to learn new data with both existing labels and new labels with less forgetting. The QR factorization can provide the accurate subspace prior, and incremental QR factorization can reasonably express the collaboration between new data with both existing classes and new class with less forgetting. In our framework, a set of appropriate features (i.e., nodes) provides improved representation for each class. We apply our method to the convolutional neural network (CNN) for learning Cifar-100 and Cifar-10 datasets. The experimental results show that the proposed method efficiently alleviates the stability and plasticity dilemma in the deep neural networks by providing the performance stability of a trained network while effectively learning unseen data and additional new classes.

Isolation and Impartial Aggregation: A Paradigm of Incremental Learning without Interference

This paper focuses on the prevalent stage interference and stage performance imbalance of incremental learning. To avoid obvious stage learning bottlenecks, we propose a new incremental learning framework, which leverages a series of stage-isolated classifiers to perform the learning task at each stage, without interference from others. To be concrete, to aggregate multiple stage classifiers as a uniform one impartially, we first introduce a temperature-controlled energy metric for indicating the confidence score levels of the stage classifiers. We then propose an anchor-based energy self-normalization strategy to ensure the stage classifiers work at the same energy level. Finally, we design a voting-based inference augmentation strategy for robust inference. The proposed method is rehearsal-free and can work for almost all incremental learning scenarios. We evaluate the proposed method on four large datasets. Extensive results demonstrate the superiority of the proposed method in setting up new state-of-the-art overall performance. Code is available at https://github.com/iamwangyabin/ESN.

Enhanced NILM load pattern extraction via variable-length motif discovery

Due to the diversity of appliances and users’ power consumption behaviors, it is challenging to accurately extract load signature samples for non-intrusive load monitoring (NILM) in unseen scenarios. To this end, this paper proposes an enhanced NILM load pattern extraction method via variable-length motif discovery. Firstly, the variable-length motif discovery method is used for the first time to discover similar subsequences from the aggregated power time series to extract power waveform signature (PWS) samples for appliances and obtain the power waveform motifs (PWMs). Furthermore, the appliance PWM sequence pattern mining method is proposed to detect complete power consumption patterns of appliance working cycles, and eventually construct the load signature library. Finally, the similarity between the original PWS samples and the templates in the signature library is measured to realize load identification. In practice, the proposed method is intended to integrate into the incremental learning framework, so that the unknown appliances can be identified gradually by continuous iterative learning. The comparison testing results on the public and private datasets demonstrate that the proposed method can effectively discover various PWMs of different appliances, and accurately model their power consumption patterns, thus exhibiting better performance on the unsupervised load identification compared with the existing methods, especially for the complex appliances.

Exemplar-free class incremental learning via discriminative and comparable parallel one-class classifiers

The exemplar-free class incremental learning (IL) requires classification models to learn new-class knowledge incrementally without retaining any old samples. Recently, the IL framework based on parallel one-class classifiers (POC) has demonstrated promising performance. It trains a one-class classifier (OCC) for each category and thus is immune to the catastrophic forgetting problem. However, the single-class training strategy may incur weak discriminability and low comparability between different classifiers in POC. To meet this challenge, we propose a new IL framework, referred to as Discriminative and Comparable Parallel One-class Classifiers (DCPOC). Instead of ordinary OCCs (e.g., deep SVDD) used in other POC methods, DCPOC adopts variational auto-encoders (VAE) as OCCs because VAEs can be used not only to identify classes for given samples but also to generate pseudo samples for the trained classes. With this advantage, DCPOC trains a new-class VAE in contrast with the old-class VAEs, which benefits the new-class VAE to reconstruct better for new-class samples but worse for old-class pseudo samples, thus enhancing the comparability. Furthermore, DCPOC introduces a hinge reconstruction loss to reinforce the discriminability. We evaluate our method on MNIST, CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet. The experimental results show that DCPOC achieves state-of-the-art performance on these datasets.11The source code is publicly available at https://github.com/SunWenJu123/DCPOC

Distilled Meta-learning for Multi-Class Incremental Learning

Meta-learning approaches have recently achieved promising performance in multi-class incremental learning. However, meta-learners still suffer from catastrophic forgetting, i.e., they tend to forget the learned knowledge from the old tasks when they focus on rapidly adapting to the new classes of the current task. To solve this problem, we propose a novel distilled meta-learning (DML) framework for multi-class incremental learning that integrates seamlessly meta-learning with knowledge distillation in each incremental stage. Specifically, during inner-loop training, knowledge distillation is incorporated into the DML to overcome catastrophic forgetting. During outer-loop training, a meta-update rule is designed for the meta-learner to learn across tasks and quickly adapt to new tasks. By virtue of the bilevel optimization, our model is encouraged to reach a balance between the retention of old knowledge and the learning of new knowledge. Experimental results on four benchmark datasets demonstrate the effectiveness of our proposal and show that our method significantly outperforms other state-of-the-art incremental learning methods.

Graph Deep Factors for Probabilistic Time-series Forecasting

Effective time-series forecasting methods are of significant importance to solve a broad spectrum of research problems. Deep probabilistic forecasting techniques have recently been proposed for modeling large collections of time-series. However, these techniques explicitly assume either complete independence (local model) or complete dependence (global model) between time-series in the collection. This corresponds to the two extreme cases where every time-series is disconnected from every other time-series in the collection or likewise, that every time-series is related to every other time-series resulting in a completely connected graph. In this work, we propose a deep hybrid probabilistic graph-based forecasting framework called Graph Deep Factors (GraphDF) that goes beyond these two extremes by allowing nodes and their time-series to be connected to others in an arbitrary fashion. GraphDF is a hybrid forecasting framework that consists of a relational global and relational local model. In particular, a relational global model learns complex non-linear time-series patterns globally using the structure of the graph to improve both forecasting accuracy and computational efficiency. Similarly, instead of modeling every time-series independently, a relational local model not only considers its individual time-series but also the time-series of nodes that are connected in the graph. The experiments demonstrate the effectiveness of the proposed deep hybrid graph-based forecasting model compared to the state-of-the-art methods in terms of its forecasting accuracy, runtime, and scalability. Our case study reveals that GraphDF can successfully generate cloud usage forecasts and opportunistically schedule workloads to increase cloud cluster utilization by 47.5% on average. Furthermore, we target addressing the common nature of many time-series forecasting applications where time-series are provided in a streaming version; however, most methods fail to leverage the newly incoming time-series values and result in worse performance over time. In this article, we propose an online incremental learning framework for probabilistic forecasting. The framework is theoretically proven to have lower time and space complexity. The framework can be universally applied to many other machine learning-based methods.

An incremental learning method with hybrid data over/down-sampling for sEMG-based gesture classification

Surface electromyography(sEMG)-based gesture classification methods have been widely developed in neural decoding. However, these decoding methods are usually constrained to a fixed set of gestures, which hinders flexibility in practical application. This paper contributes to an incremental learning framework to make classifiers learn different gesture sets (new tasks) gradually without catastrophic forgetting. First, this study analyzes the existing neural decoding methods with deep learning, and introduces an early and late fusion convolutional neural network (ELFCNN) structure based on frequency spectrum. Then, a sEMG-based gesture classification with incremental learning is demonstrated, which modifies the end to end learning method with hybrid data over/down-sampling (HDOD) method. By combining ELFCNN and the HDOD progress, the incremental learning method can make a comparable performance without data dimension reduction, and mitigate catastrophic forgetting while reducing data storage. Experimental results on both small and large size situations show consistent classification accuracy improvement from 0.47% to 0.71% compared with other popular incremental learning methods.

Forgetting to Remember: A Scalable Incremental Learning Framework for Cross-Task Blind Image Quality Assessment

Recent years have witnessed the great success of blind image quality assessment (BIQA) in various task-specific scenarios, which present invariable distortion types and evaluation criteria. However, due to the rigid structure and learning framework, they cannot apply to the cross-task BIQA scenario, where the distortion types and evaluation criteria keep changing in practical applications. This paper proposes a scalable incremental learning framework (SILF) that could sequentially conduct BIQA across multiple evaluation tasks with limited memory capacity. More specifically, we develop a dynamic parameter isolation strategy to sequentially update the task-specific parameter subsets, which are non-overlapped with each other. Each parameter subset is temporarily settled to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Remember</i> one evaluation preference toward its corresponding task, and the previously settled parameter subsets can be adaptively reused in the following BIQA to achieve better performance based on the task relevance. To suppress the unrestrained expansion of memory capacity in sequential tasks learning, we develop a scalable memory unit by gradually and selectively pruning unimportant neurons from previously settled parameter subsets, which enable us to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Forget</i> part of previous experiences and free the limited memory capacity for adapting to the emerging new tasks. Extensive experiments on eleven IQA datasets demonstrate that our proposed method significantly outperforms the other state-of-the-art methods in cross-task BIQA. The source code of the proposed method is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">github.com/maruiperfect/SILF.</uri>

Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks

Artificial neural networks are known to suffer from catastrophic forgetting: when learning multiple tasks sequentially, they perform well on the most recent task at the expense of previously learned tasks. In the brain, sleep is known to play an important role in incremental learning by replaying recent and old conflicting memory traces. Here we tested the hypothesis that implementing a sleep-like phase in artificial neural networks can protect old memories during new training and alleviate catastrophic forgetting. Sleep was implemented as off-line training with local unsupervised Hebbian plasticity rules and noisy input. In an incremental learning framework, sleep was able to recover old tasks that were otherwise forgotten. Previously learned memories were replayed spontaneously during sleep, forming unique representations for each class of inputs. Representational sparseness and neuronal activity corresponding to the old tasks increased while new task related activity decreased. The study suggests that spontaneous replay simulating sleep-like dynamics can alleviate catastrophic forgetting in artificial neural networks.