Incremental Object Research Articles

Towards Non Co-occurrence Incremental Object Detection with Unlabeled In-the-Wild Data

Towards Non Co-occurrence Incremental Object Detection with Unlabeled In-the-Wild Data

Non-exemplar Domain Incremental Object Detection via Learning Domain Bias

Domain incremental object detection (DIOD) aims to gradually learn a unified object detection model from a dataset stream composed of different domains, achieving good performance in all encountered domains. The most critical obstacle to this goal is the catastrophic forgetting problem, where the performance of the model improves rapidly in new domains but deteriorates sharply in old ones after a few sessions. To address this problem, we propose a non-exemplar DIOD method named learning domain bias (LDB), which learns domain bias independently at each new session, avoiding saving examples from old domains. Concretely, a base model is first obtained through training during session 1. Then, LDB freezes the weights of the base model and trains individual domain bias for each new incoming domain, adapting the base model to the distribution of new domains. At test time, since the domain ID is unknown, we propose a domain selector based on nearest mean classifier (NMC), which selects the most appropriate domain bias for a test image. Extensive experimental evaluations on two series of datasets demonstrate the effectiveness of the proposed LDB method in achieving high accuracy on new and old domain datasets. The code is available at https://github.com/SONGX1997/LDB.

TIB: Detecting Unknown Objects via Two-Stream Information Bottleneck.

Detecting diverse objects, including ones never-seen-before during training, is critical for the safe application of object detectors. To this end, a task of unsupervised out-of-distribution object detection (OOD-OD) is proposed to detect unknown objects without the reliance on an auxiliary dataset. For this task, it is important to reduce the impact of lacking unknown data for supervision and leverage in-distribution (ID) data to improve the model's discrimination. In this paper, we propose a method of Two-Stream Information Bottleneck (TIB), consisting of a standard IB and a dedicated Reverse Information Bottleneck (RIB). Specifically, after extracting the features of an ID image, we first define a standard IB network to disentangle instance representations that are beneficial for localizing and recognizing objects. Meanwhile, we present RIB to obtain simulative OOD features to alleviate the impact of lacking unknown data. Different from standard IB aiming to extract task-relevant compact representations, RIB is to obtain task-irrelevant representations by reversing the optimization objective of the standard IB. Next, to further enhance the discrimination, a mixture of information bottlenecks is designed to sufficiently capture object-related information. Experimental results on OOD-OD, open-vocabulary object detection, incremental object detection, and open-set object detection show the superiorities of our method.

Unsupervised Out-of-Distribution Object Detection via PCA-Driven Dynamic Prototype Enhancement.

To promote the application of object detectors in real scenes, out-of-distribution object detection (OOD-OD) is proposed to distinguish whether detected objects belong to the ones that are unseen during training or not. One of the key challenges is that detectors lack unknown data for supervision, and as a result, can produce overconfident detection results on OOD data. Thus, this task requires to synthesize OOD data for training, which achieves the goal of enhancing the ability of localizing and discriminating OOD objects. In this paper, we propose a novel method, i.e., PCA-Driven dynamic prototype enhancement, to explore exploiting Principal Component Analysis (PCA) to extract simulative OOD data for training and obtain dynamic prototypes that are related to the current input and are helpful for boosting the discrimination ability. Concretely, the last few principal components of the backbone features are utilized to calculate an OOD map that involves plentiful information that deviates from the correlation distribution of the input. The OOD map is further used to extract simulative OOD data for training, which alleviates the impact of lacking unknown data. Besides, for in-distribution (ID) data, the category-level semantic information of objects between the backbone features and the high-level features should be kept consistent. To this end, we utilize the residual principal components to extract dynamic prototypes that reflect the semantic information of the current backbone features. Next, we define a contrastive loss to leverage these prototypes to enlarge the semantic gap between the simulative OOD data and the features from the residual principal components, which improves the ability of discriminating OOD objects. In the experiments, we separately verify our method on OOD-OD and incremental object detection. The significant performance gains demonstrate the superiorities of our method.

Domain Incremental Object Detection Based on Feature Space Topology Preserving Strategy

Object detection with the capacity to incrementally adapt to new domains is a crucial yet relatively under-explored research topic. The <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">catastrophic forgetting</i> problem presents a significant challenge to achieve this goal, where the model’s performance improves quickly in new conditions but deteriorates sharply in old ones after several incremental learning sessions. Drawing on recent discoveries in visual memories of the human brain, we introduce the Topology-Preserving Domain Incremental Object Detection (TP-DIOD) approach, which aims to address the catastrophic forgetting problem by extracting the topological structure of the feature space learned by the Convolutional Neural Network (CNN) model and preserving this topology during the subsequent incremental learning sessions. Specifically, we model the feature space topology using the self-organizing map (SOM) and construct an anchor image set based on the centroid vectors of the SOM nodes to memorize the feature space topology. We then develop the anchor loss function to penalize the topological changes of the feature space during the subsequent incremental learning sessions. Experimental evaluations on two sets of datasets demonstrate the effectiveness of the proposed TP-DIOD method in mitigating the catastrophic forgetting problem and achieving high accuracy on both old and new domain datasets.

Simulation of SMA-based engineering applications considering large displacement and rotation, thermomechanical coupling and partial phase transformation

Shape memory alloys (SMAs) undergo large recoverable deformation due to their inherent diffusionless martensitic phase transformation. Two factors need to be taken into account to incisively emulate the behaviour of SMA based structures: (i) consideration of large displacement and rotation (LDR) effect, and (ii) capturing the consequence of transformation induced material level coupling. In this study, both these effects are apprehended by extending the infinitesimal strain-based constitutive model of SMA as proposed by Lagoudas et al. (2012), assuming small elastic strain but with finite inelastic strain. To preclude any spurious stress generation out of large rotation, the Jaumann stress rate is used, and incremental objectivity for finite rotation between two successive time instants is conserved by rotating the strain increment and spin tensor to the rotation-independent mid-point configuration following Hughes–Winget (Hughes and Winget, 1980) algorithm. In addition, the contribution of thermoelastic and latent heat evolved during transformation is taken into consideration in the thermal equilibrium equation. Moreover, the effect of partial phase transformation yielding minor hysteresis loop has also been captured. The equilibrium equation is expressed in the current configuration following the Updated Lagrangian formulation. The mechanical and thermal equilibrium equations are solved concurrently in block matrix form using the Newton–Raphson (NR) iterative technique, considering the coupling terms resulting from the phase transformation. The efficacy and robustness of the developed finite element model are corroborated through various practical applications of SMA-based members, e.g., SMA ring, SMA-actuated beam, morphing of corrugated airfoil, orthodontic palatal expander, compliant gripper, undergoing LDR while subjected to different thermomechanical loading conditions. The combined effects of LDR along with thermomechanical coupling yield a stiffening behaviour during loading and sluggish response at the time of thermal recovery.

A New Multinetwork Mean Distillation Loss Function for Open-World Domain Incremental Object Detection

The development of object detection networks has reached a high point, and there have been significant improvements in accuracy and detection speed. Object detection is widely used in intelligent robots, self-driving cars, and other edge-intelligent terminals. Unfortunately, when a detector is allowed to learn new objects in an unfamiliar environment, it can catastrophically forget the objects it has already learned. In particular, reliable and stable knowledge cannot be extracted from old models. Based on this, a new multinetwork mean distillation loss function for open-world domain incremental object detection is presented. To better extract reliable and stable knowledge from old models, we enhanced the distillation output of the detector with a ResNet50 backbone and an output RoI head. The distillation output of the intermediate RPN is softened by adaptive distillation. To obtain more stable results, the ResNet50 backbone and RPN on the channel are zero-averaged. Various incremental steps and stability experiments are performed on two benchmark datasets, PASCAL VOC and MS COCO. The experimental results show the excellent performance of our method in different experimental scenarios, and it is superior to the most advanced methods. For example, in the setting of the batch task, incremental object detection on the PASCAL VOC and MS COCO datasets is improved by 3.4% and 2.1%, respectively.

Class incremental learning of remote sensing images based on class similarity distillation.

When a well-trained model learns a new class, the data distribution differences between the new and old classes inevitably cause catastrophic forgetting in order to perform better in the new class. This behavior differs from human learning. In this article, we propose a class incremental object detection method for remote sensing images to address the problem of catastrophic forgetting caused by distribution differences among different classes. First, we introduce a class similarity distillation (CSD) loss based on the similarity between new and old class prototypes, ensuring the model's plasticity to learn new classes and stability to detect old classes. Second, to better extract class similarity features, we propose a global similarity distillation (GSD) loss that maximizes the mutual information between the new class feature and old class features. Additionally, we present a region proposal network (RPN)-based method that assigns positive and negative labels to prevent mislearning issues. Experiments demonstrate that our method is more accurate for class incremental learning on public DOTA and DIOR datasets and significantly improves training efficiency compared to state-of-the-art class incremental object detection methods.

Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning

Incremental few-shot object detection aims at detecting novel classes without forgetting knowledge of the base classes with only a few labeled training data from the novel classes. Most related prior works are on incremental object detection that rely on the availability of abundant training samples per novel class that substantially limits the scalability to real-world setting where novel data can be scarce. In this paper, we propose the Incremental-DETR that does incremental few-shot object detection via fine-tuning and self-supervised learning on the DETR object detector. To alleviate severe over-fitting with few novel class data, we first fine-tune the class-specific components of DETR with self-supervision from additional object proposals generated using Selective Search as pseudo labels. We further introduce an incremental few-shot fine-tuning strategy with knowledge distillation on the class-specific components of DETR to encourage the network in detecting novel classes without forgetting the base classes. Extensive experiments conducted on standard incremental object detection and incremental few-shot object detection settings show that our approach significantly outperforms state-of-the-art methods by a large margin. Our source code is available at https://github.com/dongnana777/Incremental-DETR.

One-Shot Replay: Boosting Incremental Object Detection via Retrospecting One Object

Modern object detectors are ill-equipped to incrementally learn new emerging object classes over time due to the well-known phenomenon of catastrophic forgetting. Due to data privacy or limited storage, few or no images of the old data can be stored for replay. In this paper, we design a novel One-Shot Replay (OSR) method for incremental object detection, which is an augmentation-based method. Rather than storing original images, only one object-level sample for each old class is stored to reduce memory usage significantly, and we find that copy-paste is a harmonious way to replay for incremental object detection. In the incremental learning procedure, diverse augmented samples with co-occurrence of old and new objects to existing training data are generated. To introduce more variants for objects of old classes, we propose two augmentation modules. The object augmentation module aims to enhance the ability of the detector to perceive potential unknown objects. The feature augmentation module explores the relations between old and new classes and augments the feature space via analogy. Extensive experimental results on VOC2007 and COCO demonstrate that OSR can outperform the state-of-the-art incremental object detection methods without using extra wild data.

Continual Object Detection: A review of definitions, strategies, and challenges

The field of Continual Learning investigates the ability to learn consecutive tasks without losing performance on those previously learned. The efforts of researchers have been mainly focused on incremental classification tasks. Yet, we believe that continual object detection deserves even more attention due to its vast range of applications in robotics and autonomous vehicles. This scenario is also more complex than conventional classification, given the occurrence of instances of classes that are unknown at the time but can appear in subsequent tasks as a new class to be learned, resulting in missing annotations and conflicts with the background label. In this review, we analyze the current strategies proposed to tackle the problem of class-incremental object detection. Our main contributions are: (1) a short and systematic review of the methods that propose solutions to traditional incremental object detection scenarios; (2) A comprehensive evaluation of the existing approaches using a new metric to quantify the stability and plasticity of each technique in a standard way; (3) an overview of the current trends within continual object detection and a discussion of possible future research directions.

DIODE: Dilatable Incremental Object Detection

To accommodate rapid changes in the real world, the cognition system of humans is capable of continually learning concepts. On the contrary, conventional deep learning models lack this capability of preserving previously learned knowledge. When a neural network is fine-tuned to learn new tasks, its performance on previously trained tasks will significantly deteriorate. Many recent works on incremental object detection tackle this problem by introducing advanced regularization. Although these methods have shown promising results, the benefits are often short-lived after the first incremental step. Under multi-step incremental learning, the trade-off between old knowledge preserving and new task learning becomes progressively more severe. Thus, the performance of regularization-based incremental object detectors gradually decays for subsequent learning steps. In this paper, we aim to alleviate this performance decay on multi-step incremental detection tasks by proposing a dilatable incremental object detector (DIODE). For the task-shared parameters, our method adaptively penalizes the changes of important weights for previous tasks. At the same time, the structure of the model is dilated or expanded by a limited number of task-specific parameters to promote new task learning. Extensive experiments on PASCAL VOC and COCO datasets demonstrate substantial improvements over the state-of-the-art methods. Notably, compared with the state-of-the-art methods, our method achieves up to 6.0% performance improvement by increasing the number of parameters by just 1.2% for each newly learned task.

Multi-View correlation distillation for incremental object detection

In real applications, new object classes often emerge after the detection model has been trained on a prepared dataset with fixed classes. Fine-tuning the old model with only new data will lead to a well-known phenomenon of catastrophic forgetting, which severely degrades the performance of modern object detectors. Due to the storage burden, data privacy and time consumption, sometimes it is impractical to train the model from scratch with all data of both old and new classes. In this paper, we propose a novel Multi-View Correlation Distillation (MVCD) based incremental object detection method, which explores the intra-feature correlations in the feature space of the object detector. To better transfer the knowledge learned from the old classes and maintain the ability to learn new classes, we select the sample-specific discriminative features from channel-wise, point-wise and instance-wise views. Meanwhile, the correlation distillation losses on the selective features are designed to regularize the learning of the incremental object detector. A new metric named Stability-Plasticity-mAP (SPmAP) is proposed to evaluate the incremental learning performance as a complementary metric to mAP, which integrates the metrics for the stability on old classes and the plasticity on new classes in incremental object detection. The extensive experiments conducted on VOC2007 and COCO demonstrate that MVCD achieves a better trade-off between stability and plasticity than state-of-the-art first-order distillation-based incremental object detection methods.

A dynamic approach for updating the optimal approximation of rough set

The approximate representation and calculation of target concept on approximate space are the foundation of rough set model study. For dynamic attribute reduction and related tasks, it is often necessary to update the approximation of the target concept of rough set theory, and most existing updating approaches are based on the lower and upper approximations of target concepts. This paper presents a dynamic approach for updating the optimal set when a single object is added into the decision table. Aiming at the dynamic change of condition class and decision class caused by the incremental object, the paper discusses the updating mechanisms of optimal approximation set in four different cases and then the dynamic updating algorithm of optimal approximation set is obtained. Finally, the experiments on different data sets from UCI show that the time complexity of optimal approximate updating of decision classes are different for four different cases and the incremental methods are feasible and efficient for decision systems of added one single object.

The Selectional Constraint of the Chinese Progressive Operator &lt;i&gt;zai&lt;/i&gt;

Previous studies on the Chinese progressive mainly focus on the distinctions between Chinese progressive <i>zai</i> and the continuous <i>-zhe</i>, but few has noticed that the progressive <i>zai</i> is sensitive to the plurality of the event encoded by the predicate in its scope. Inspired by the observations of the selectional constraints of the English progressive, this paper aims to find out the selectional constraints of Chinese progressive <i>zai</i>. It is argued that Chinese progressive <i>zai</i> requires its modified predicates to denote singular durative events as its English counterpart do. Given that no studies have illustrated the semantic criterion for identifying predicates denoting singular events in Mandarin, this study proposes that singular telic (incremental) predicates compatible with <i>zai</i> are those with singular atoms or singular groups as the incremental themes and singular atelic predicates co-occurring with <i>zai</i> are those being incremental homogeneous. An incremental accomplishment with a singular atom as the theme is expressed by the combination of an incremental verb and a quantized incremental object with the cardinal numeral quantifier <i>yi</i> ‘one’ as the measure phrase, while that with a singular group as the theme is expressed by the combination of an incremental verb and a definite object modified by the definite determiner like <i>na</i> ‘that/those’. In contrast, incremental homogeneous predicates are expressed by activities or plural achievements.

Incremental Detection of Remote Sensing Objects With Feature Pyramid and Knowledge Distillation

When a detection model that has been well-trained on a set of classes faces new classes, incremental learning is always necessary to adapt the model to detect the new classes. In most scenarios, it is required to preserve the learned knowledge of the old classes during incremental learning rather than reusing the training data from the old classes. Since the objects in remote sensing images often appear in various sizes, arbitrary directions, and dense distribution, it further makes incremental learning-based object detection more difficult. In this article, a new architecture for incremental object detection is proposed based on feature pyramid and knowledge distillation. Especially, by means of a feature pyramid network (FPN), the objects with various scales are detected in the different layers of the feature pyramid. Motivated by Learning without Forgetting (LwF), a new branch is expended in the last layer of FPN, and knowledge distillation is applied to the outputs of the old branch to maintain the old learning capability for the old classes. Multitask learning is adopted to jointly optimize the losses from two branches. Experiments on two widely used remote sensing data sets show our promising performance compared with state-of-the-art incremental object detection methods.

Faster ILOD: Incremental learning for object detectors based on faster RCNN

• RPN is relatively robust towards missing annotations on incremental detection. • Multi-network adaptive distillation is proposed for incremental detection. • Our framework is generic and can be applied to any object detectors with RPN. • We demonstrate the superior performance of our model on detection benchmark datasets. The human vision and perception system is inherently incremental where new knowledge is continually learned over time whilst existing knowledge is retained. On the other hand, deep learning networks are ill-equipped for incremental learning. When a well-trained network is adapted to new categories, its performance on the old categories will dramatically degrade. To address this problem, incremental learning methods have been explored which preserve the old knowledge of deep learning models . However, the state-of-the-art incremental object detector employs an external fixed region proposal method that increases overall computation time and reduces accuracy comparing to Region Proposal Network (RPN) based object detectors such as Faster RCNN. The purpose of this paper is to design an efficient end-to-end incremental object detector using knowledge distillation. We first evaluate and analyze the performance of the RPN-based detector with classic distillation on incremental detection tasks. Then, we introduce multi-network adaptive distillation that properly retains knowledge from the old categories when fine-tuning the model for new task. Experiments on the benchmark datasets, PASCAL VOC and COCO, demonstrate that the proposed incremental detector based on Faster RCNN is more accurate as well as being 13 times faster than the baseline detector.

IncDet: In Defense of Elastic Weight Consolidation for Incremental Object Detection.

Elastic weight consolidation (EWC) has been successfully applied for general incremental learning to overcome the catastrophic forgetting issue. It adaptively constrains each parameter of the new model not to deviate much from its counterpart in the old model during fine-tuning on new class data sets, according to its importance weight for old tasks. However, the previous study demonstrates that it still suffers from catastrophic forgetting when directly used in object detection. In this article, we show EWC is effective for incremental object detection if with critical adaptations. First, we conduct controlled experiments to identify two core issues why EWC fails if trivially applied to incremental detection: 1) the absence of old class annotations in new class images makes EWC misclassify objects of old classes in these images as background and 2) the quadratic regularization loss in EWC easily leads to gradient explosion when balancing old and new classes. Then, based on the abovementioned findings, we propose the corresponding solutions to tackle these issues: 1) utilize pseudobounding box annotations of old classes on new data sets to compensate for the absence of old class annotations and 2) adopt a novel Huber regularization instead of the original quadratic loss to prevent from unstable training. Finally, we propose a general EWC-based incremental object detection framework and implement it under both Fast R-CNN and Faster R-CNN, showing its flexibility and versatility. In terms of either the final performance or the performance drop with respect to the upper bound of joint training on all seen classes, evaluations on the PASCAL VOC and COCO data sets show that our method achieves a new state of the art.

An elasto-plastic self-consistent model for damaged polycrystalline materials: Theoretical formulation and numerical implementation

Elasto-plastic multiscale approaches are known to be suitable to model the mechanical behavior of metallic materials during forming processes. These approaches are classically adopted to explicitly link relevant microstructural effects to the macroscopic behavior. This paper presents a finite strain elasto-plastic self-consistent model for damaged polycrystalline aggregates and its implementation into ABAQUS/Standard finite element (FE) code. Material degradation is modeled by the introduction of a scalar damage variable at each crystallographic slip system for each individual grain. The single crystal plastic flow is described by both the classical and a regularized version of the Schmid criterion. To integrate the single crystal constitutive equations, two new numerical algorithms are developed (one for each plastic flow rule). Then, the proposed single crystal modeling is embedded into the self-consistent scheme to predict the mechanical behavior of elasto-plastic polycrystalline aggregates in the finite strain range. This strategy is implemented into ABAQUS/Standard FE code through a user-defined material (UMAT) subroutine. Special attention is paid to the satisfaction of the incremental objectivity and the efficiency of the convergence of the global resolution scheme, related to the computation of the consistent tangent modulus. The capability of the new constitutive modeling to capture the interaction between the damage evolution and the microstructural properties is highlighted through several simulations at both single crystal and polycrystalline scales. It appears from the numerical tests that the use of the classical Schmid criterion leads to a poor numerical convergence of the self-consistent scheme (due to the abrupt changes in the activity of the slip systems), which sometimes causes the computations to be prematurely stopped. By contrast, the use of the regularized version of the Schmid law allows a better convergence of the self-consistent approach, but induces an important increase in the computation time devoted to the integration of the single crystal constitutive equations (because of the high value of the power-law exponent used to regularize the Schmid yield function). To avoid these difficulties, a numerical strategy is built to combine the benefits of the two approaches: the classical Schmid criterion is used to integrate the single crystal constitutive equations, while its regularized version is used to compute the microscopic tangent modulus required for solving the self-consistent equations. The robustness and the accuracy of this novel numerical strategy are particularly analyzed through several numerical simulations (prediction of the mechanical behavior of polycrystalline aggregates and simulation of a circular cup-drawing forming process).

Incremental Learning of Object Models From Natural Human–Robot Interactions

In order to perform complex tasks in realistic human environments, robots need to be able to learn new concepts in the wild , incrementally, and through their interactions with humans. This article presents an end-to-end pipeline to learn object models incrementally during the human–robot interaction (HRI). The pipeline we propose consists of three parts: 1) recognizing the interaction type; 2) detecting the object that the interaction is targeting; and 3) learning incrementally the models from data recorded by the robot sensors. Our main contributions lie in the target object detection, guided by the recognized interaction, and in the incremental object learning. The novelty of our approach is the focus on natural, heterogeneous, and multimodal HRIs to incrementally learn new object models. Throughout the article, we highlight the main challenges associated with this problem, such as high degree of occlusion and clutter, domain change, low-resolution data, and interaction ambiguity. This article shows the benefits of using multiview approaches and combining visual and language features, and our experimental results outperform standard baselines. Note to Practitioners —This article was motivated by challenges in recognition tasks for dynamic and varying scenarios. Our approach learns to recognize new user interactions and objects. To do so, we use multimodal data from the user–robot interaction; visual data are used to learn the objects and speech is used to learn the label and help with the interaction-type recognition. We use state-of-the-art deep learning (DL) models to segment the user and the objects in the scene. Our algorithm for incremental learning is based on a classic incremental clustering approach. The pipeline we propose works with all sensors mounted on the robot, so it allows mobility on the system. This article uses the data recorded from a Baxter robot, which enables the use of the manipulation arms in future steps, but it would work with any robot that can have the same sensors mounted. The sensors used are two RGB-D cameras and a microphone. The pipeline currently has high computational requirements to run the two DL-based steps. We have tested it with a desktop computer, including a GTX 1060 and 32 GB of RAM.