Instance Level Research Articles

Multi-instance semantic similarity transferring for knowledge distillation

Knowledge distillation is a popular paradigm for learning portable neural networks by transferring the knowledge from a large model into a smaller one. Most existing approaches enhance the student model by utilizing the similarity information between the categories of instance level provided by the teacher model. However, these works ignore the similarity correlation between different instances that plays an important role in confidence prediction. To tackle this issue, we propose a novel method in this paper, called multi-instance semantic similarity transferring for knowledge distillation (STKD), which aims to fully utilize the similarities between categories of multiple samples. Furthermore, we propose to better capture the similarity correlation between different instances by the mixup technique, which creates virtual samples by a weighted linear interpolation. Note that, our distillation loss can fully utilize the incorrect classes similarities by the mixed labels. The proposed approach promotes the performance of student model as the virtual sample created by multiple images produces a similar probability distribution in the teacher and student networks. Experiments and ablation studies on several public classification datasets including CIFAR-10, CIFAR-100, CINIC-10 and Tiny-ImageNet verify that this light-weight method can effectively boost the performance of the compact student model. It shows that STKD has substantially outperformed the vanilla knowledge distillation and achieved superior accuracy over the state-of-the-art knowledge distillation methods.

Selective label enhancement for multi-label classification based on three-way decisions

Multi-label classification is a challenging issue in the data science community due to the ambiguity of label semantics. Existing studies mainly focus on improving label association with logical labels, but the performance suffers from the threshold setting. Although label distribution learning gains superior discrimination, the expenditure of collecting large-scale fine-grained numerical labels is intolerable. To address the uncertainty of logical label semantics, we propose a novel model called three-way decisions with label enhancement (3WDLE). For unseen instances, we implement a trisecting-acting-outcome framework. In the trisecting stage, an uncertainty measure called global uncertain-prone degree partitions these instances into uncertain and certain regions, where the trisecting procedure is completed from label level to instance level by leveraging the distributions of pseudo-label information. In the acting stage, instances recognized as certain regions directly take the results generated by label-specific learning, whereas the remaining are reclassified by conducting selective label enhancement. The enriched knowledge generated by the label enhancement module is learnt on trustworthy instances only. In the outcome stage, we adopt five evaluation metrics to evaluate the classification performance from the perspectives of both labels and instances. In this way, three-way decisions provide a systematic methodology to deal with uncertainty in multi-label classification, which combines logical label learning with numerical label learning into a unified framework to optimize the performance of the multi-label classification model. Extensive experiments demonstrate the superiority of 3WDLE over state-of-the-art multi-label classifications with logical labels only.

Understanding Synonymous Referring Expressions via Contrastive Features

Referring expression comprehension aims to localize objects identified by natural language descriptions. This is a challenging task as it requires understanding of both visual and language domains. One nature is that each object can be described by synonymous sentences with paraphrases, and such varieties in languages have critical impact on learning a comprehension model. While prior work usually treats each sentence and attends it to an object separately, we focus on learning a referring expression comprehension model that considers the property in synonymous sentences. To this end, we develop an end-to-end trainable framework to learn contrastive features on the image and object instance levels, where features extracted from synonymous sentences to describe the same object should be closer to each other after mapping to the visual domain. We conduct extensive experiments to evaluate the proposed algorithm on several benchmark datasets, and demonstrate that our method performs favorably against the state-of-the-art approaches. Furthermore, since the varieties in expressions become larger across datasets when they describe objects in different ways, we present the cross-dataset and transfer learning settings to validate the ability of our learned transferable features.

Hostility measure for multi-level study of data complexity

Complexity measures aim to characterize the underlying complexity of supervised data. These measures tackle factors hindering the performance of Machine Learning (ML) classifiers like overlap, density, linearity, etc. The state-of-the-art has mainly focused on the dataset perspective of complexity, i.e., offering an estimation of the complexity of the whole dataset. Recently, the instance perspective has also been addressed. In this paper, the hostility measure, a complexity measure offering a multi-level (instance, class, and dataset) perspective of data complexity is proposed. The proposal is built by estimating the novel notion of hostility: the difficulty of correctly classifying a point, a class, or a whole dataset given their corresponding neighborhoods. The proposed measure is estimated at the instance level by applying the k-means algorithm in a recursive and hierarchical way, which allows to analyze how points from different classes are naturally grouped together across partitions. The instance information is aggregated to provide complexity knowledge at the class and the dataset levels. The validity of the proposal is evaluated through a variety of experiments dealing with the three perspectives and the corresponding comparative with the state-of-the-art measures. Throughout the experiments, the hostility measure has shown promising results and to be competitive, stable, and robust.

Attention-based domain adaptation for single-stage detectors

While domain adaptation has been used to improve the performance of object detectors when the training and test data follow different distributions, previous work has mostly focused on two-stage detectors. This is because their use of region proposals makes it possible to perform local adaptation, which has been shown to significantly improve the adaptation effectiveness. Here, by contrast, we target single-stage architectures, which are better suited to resource-constrained detection than two-stage ones but do not provide region proposals. To nonetheless benefit from the strength of local adaptation, we introduce an attention mechanism that lets us identify the important regions on which adaptation should focus. Our method gradually adapts the features from global, image level to local, instance level. Our approach is generic and can be integrated into any Single-Shot Detector. We demonstrate this on standard benchmark datasets by applying it to both the single-shot detector (SSD) and a recent variant of the You Only Look Once detector (YOLOv5). Furthermore, for equivalent single-stage architectures, our method outperforms the state-of-the-art domain adaptation techniques even though they were designed for specific detectors.

Twin Contrastive Learning for Online Clustering

This paper proposes to perform online clustering by conducting twin contrastive learning (TCL) at the instance and cluster level. Specifically, we find that when the data is projected into a feature space with a dimensionality of the target cluster number, the rows and columns of its feature matrix correspond to the instance and cluster representation, respectively. Based on the observation, for a given dataset, the proposed TCL first constructs positive and negative pairs through data augmentations. Thereafter, in the row and column space of the feature matrix, instance- and cluster-level contrastive learning are respectively conducted by pulling together positive pairs while pushing apart the negatives. To alleviate the influence of intrinsic false-negative pairs and rectify cluster assignments, we adopt a confidence-based criterion to select pseudo-labels for boosting both the instance- and cluster-level contrastive learning. As a result, the clustering performance is further improved. Besides the elegant idea of twin contrastive learning, another advantage of TCL is that it could independently predict the cluster assignment for each instance, thus effortlessly fitting online scenarios. Extensive experiments on six widely-used image and text benchmarks demonstrate the effectiveness of TCL. The code will be released on GitHub.

AFP-Mask: Anchor-Free Polyp Instance Segmentation in Colonoscopy.

Colorectal cancer (CRC) is a common and lethal disease. Globally, CRC is the third most commonly diagnosed cancer in males and the second in females. The most effective way to prevent CRC is through using colonoscopy to identify and remove precancerous growths at an early stage. The detection and removal of colorectal polyps have been found to be associated with a reduction in mortality from colorectal cancer. However, the false negative rate of polyp detection during colonoscopy is often high even for experienced physicians. With recent advances in deep learning based object detection techniques, automated polyp detection shows great potential in helping physicians reduce false positive rate during colonoscopy. In this paper, we propose a novel anchor-free instance segmentation framework that can localize polyps and produce the corresponding instance level masks without using predefined anchor boxes. Our framework consists of two branches: (a) an object detection branch that performs classification and localization, (b) a mask generation branch that produces instance level masks. Instead of predicting a two-dimensional mask directly, we encode it into a compact representation vector, which allows us to incorporate instance segmentation with one-stage bounding-box detectors in a simple yet effective way. Moreover, our proposed encoding method can be trained jointly with object detector. Our experiment results show that our framework achieves a precision of 99.36% and a recall of 96.44% on public datasets, outperforming existing anchor-free instance segmentation methods by at least 2.8% in mIoU on our private dataset.

Learning Universal Adversarial Perturbation by Adversarial Example

Deep learning models have shown to be susceptible to universal adversarial perturbation (UAP), which has aroused wide concerns in the community. Compared with the conventional adversarial attacks that generate adversarial samples at the instance level, UAP can fool the target model for different instances with only a single perturbation, enabling us to evaluate the robustness of the model from a more effective and accurate perspective. The existing universal attack methods fail to exploit the differences and connections between the instance and universal levels to produce dominant perturbations. To address this challenge, we propose a new universal attack method that unifies instance-specific and universal attacks from a feature perspective to generate a more dominant UAP. Specifically, we reformulate the UAP generation task as a minimax optimization problem and then utilize the instance-specific attack method to solve the minimization problem thereby obtaining better training data for generating UAP. At the same time, we also introduce a consistency regularizer to explore the relationship between training data, thus further improving the dominance of the generated UAP. Furthermore, our method is generic with no additional assumptions about the training data and hence can be applied to both data-dependent (supervised) and data-independent (unsupervised) manners. Extensive experiments demonstrate that the proposed method improves the performance by a significant margin over the existing methods in both data-dependent and data-independent settings. Code is available at https://github.com/lisenxd/AT-UAP.

Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation

In object re-identification (ReID), the development of deep learning techniques often involves model updates and deployment. It is unbearable to re-embedding and re-index with the system suspended when deploying new models. Therefore, backward-compatible representation is proposed to enable ``new'' features to be compared with ``old'' features directly, which means that the database is active when there are both ``new'' and ``old'' features in it. Thus we can scroll-refresh the database or even do nothing on the database to update. The existing backward-compatible methods either require a strong overlap between old and new training data or simply conduct constraints at the instance level. Thus they are difficult in handling complicated cluster structures and are limited in eliminating the impact of outliers in old embeddings, resulting in a risk of damaging the discriminative capability of new features. In this work, we propose a Neighborhood Consensus Contrastive Learning (NCCL) method. With no assumptions about the new training data, we estimate the sub-cluster structures of old embeddings. A new embedding is constrained with multiple old embeddings in both embedding space and discrimination space at the sub-class level. The effect of outliers diminished, as the multiple samples serve as ``mean teachers''. Besides, we propose a scheme to filter the old embeddings with low credibility, further improving the compatibility robustness. Our method ensures the compatibility without impairing the accuracy of the new model. It can even improve the new model's accuracy in most scenarios.

Inter-cluster and intra-cluster joint optimization for unsupervised cross-domain person re-identification

The purpose of unsupervised cross-domain (UCD) person re-identification (re-ID) is to adapt the well pre-trained model on the labeled source domain to the unlabeled target domain, which tackles a more realistic problem. Most successful UCD methods are devoted to exploring the relationship at instance level in the target domain, but neglect the relationship at cluster level. However, person re-ID is the task of retrieving samples belonging to the same identity. Thus, exploring the cluster level relationship is critical to improve the performance of re-ID. In this paper, a novel inter-cluster and intra-cluster joint optimization (IcJO) model is proposed. Specifically, for the inter-cluster relationship, a novel centroid-based contrastive learning strategy is designed, making the sample close to its cluster centroid and far away from the heterogeneous centroids. In order to alleviate the influence of clustering noise, a cluster-based uncertainty estimation coefficient is introduced to guide the contrastive learning process. As for the intra-cluster relationship, a centroid-based maximum mean discrepancy (cMMD) loss is adopted to narrow the intra-cluster distribution. Moreover, in order to enhance the feature invariance of each identity, a centroid-based similarity loss is developed. Extensive experiments are conducted on three standard benchmarks. The results demonstrate that our IcJO model performs favorably against the state-of-the-art UCD person re-ID methods.

Intuitionistic Fuzzy-Based Three-Way Label Enhancement for Multi-Label Classification

Multi-label classification deals with the determination of instance-label associations for unseen instances. Although many margin-based approaches are delicately developed, the uncertainty classifications for those with smaller separation margins remain unsolved. The intuitionistic fuzzy set is an effective tool to characterize the concept of uncertainty, yet it has not been examined for multi-label cases. This paper proposed a novel model called intuitionistic fuzzy three-way label enhancement (IFTWLE) for multi-label classification. The IFTWLE combines label enhancement with an intuitionistic fuzzy set under the framework of three-way decisions. For unseen instances, we generated the pseudo-label for label uncertainty evaluation from a logical label-based model. An intuitionistic fuzzy set-based instance selection principle seamlessly bridges logical label learning and numerical label learning. The principle is hierarchically developed. At the label level, membership and non-membership functions are pair-wisely defined to measure the local uncertainty and generate candidate uncertain instances. After upgrading to the instance level, we select instances from the candidates for label enhancement, whereas they remained unchanged for the remaining. To the best of our knowledge, this is the first attempt to combine logical label learning with numerical label learning into a unified framework for minimizing classification uncertainty. Extensive experiments demonstrate that, with the selectively reconstructed label importance, IFTWLE achieves statistically superior over the state-of-the-art multi-label classification algorithms in terms of classification accuracy. The computational complexity of this algorithm is On2mk, where n, m, and k denote the unseen instances count, label count, and average label-specific feature size, respectively.

Road and Railway Smart Mobility: A High-Definition Ground Truth Hybrid Dataset.

A robust visual understanding of complex urban environments using passive optical sensors is an onerous and essential task for autonomous navigation. The problem is heavily characterized by the quality of the available dataset and the number of instances it includes. Regardless of the benchmark results of perception algorithms, a model would only be reliable and capable of enhanced decision making if the dataset covers the exact domain of the end-use case. For this purpose, in order to improve the level of instances in datasets used for the training and validation of Autonomous Vehicles (AV), Advanced Driver Assistance Systems (ADAS), and autonomous driving, and to reduce the void due to the no-existence of any datasets in the context of railway smart mobility, we introduce our multimodal hybrid dataset called ESRORAD. ESRORAD is comprised of 34 videos, 2.7 k virtual images, and 100 k real images for both road and railway scenes collected in two Normandy towns, Rouen and Le Havre. All the images are annotated with 3D bounding boxes showing at least three different classes of persons, cars, and bicycles. Crucially, our dataset is the first of its kind with uncompromised efforts on being the best in terms of large volume, abundance in annotation, and diversity in scenes. Our escorting study provides an in-depth analysis of the dataset’s characteristics as well as a performance evaluation with various state-of-the-art models trained under other popular datasets, namely, KITTI and NUScenes. Some examples of image annotations and the prediction results of our 3D object detection lightweight algorithms are available in ESRORAD dataset. Finally, the dataset is available online. This repository consists of 52 datasets with their respective annotations performed.

BUILDING SECTION INSTANCE SEGMENTATION WITH COMBINED CLASSICAL AND DEEP LEARNING METHODS

Abstract. In big cities, the complexity of urban infrastructure is very high. In city centers, one construction can consist of several building sections of different heights or roof geometries. Most of the existing approaches detect those buildings as a single construction in the form of binary building segmentation maps or as one instance of object-oriented segmentation. However, reconstructing complex buildings consisting of several parts requires a higher level of detail. In this work, we present a methodology for individual building section instance segmentation on satellite imagery. We show that fully convolutional networks (FCNs) can tackle the issue much better than the state-of-the-art Mask-RCNN. A ground truth raster image with pixel value 1 for building sections and 2 for their touching borders was generated to train models on predicting both classes as a semantic output. The semantic outputs were then post-processed with the help of morphology and watershed labeling to generate segmentation on the instance level. The combination of a deep learning-based approach and a classical image processing algorithm allowed us to fulfill the segmentation task on the instance level and reach high-quality results with an mAP of up to 42 %.

SeqSeg: A sequential method to achieve nasopharyngeal carcinoma segmentation free from background dominance.

Reliable nasopharyngeal carcinoma (NPC) segmentation plays an important role in radiotherapy planning. However, recent deep learning methods fail to achieve satisfactory NPC segmentation in magnetic resonance (MR) images, since NPC is infiltrative and typically has a small or even tiny volume with indistinguishable border, making it indiscernible from tightly connected surrounding tissues from immense and complex backgrounds. To address such background dominance problems, this paper proposes a sequential method (SeqSeg) to achieve accurate NPC segmentation. Specifically, the proposed SeqSeg is devoted to solving the problem at two scales: the instance level and feature level. At the instance level, SeqSeg is forced to focus attention on the tumor and its surrounding tissue through the deep Q-learning (DQL)-based NPC detection model by prelocating the tumor and reducing the scale of the segmentation background. Next, at the feature level, SeqSeg uses high-level semantic features in deeper layers to guide feature learning in shallower layers, thus directing the channel-wise and region-wise attention to mine tumor-related features to perform accurate segmentation. The performance of our proposed method is evaluated by extensive experiments on the large NPC dataset containing 1101 patients. The experimental results demonstrated that the proposed SeqSeg not only outperforms several state-of-the-art methods but also achieves better performance in multi-device and multi-center datasets.

Detection of plane in remote sensing images using super-resolution.

The object detection of remote sensing image often has low accuracy and high missed or false detection rate due to the large number of small objects, instance level noise and cloud occlusion. In this paper, a new object detection model based on SRGAN and YOLOV3 is proposed, which is called SR-YOLO. It solves the problems of SRGAN network sensitivity to hyper-parameters and modal collapse. Meanwhile, The FPN network in YOLOv3 is replaced by PANet, shortened the distance between the lowest and the highest layers, and the SR-YOLO model has strong robustness and high detection ability by using the enhanced path to enrich the characteristics of each layer. The experimental results on the UCAS-High Resolution Aerial Object Detection Dataset showed SR-YOLO has achieved excellent performance. Compared with YOLOv3, the average precision (AP) of SR-YOLO increased from 92.35% to 96.13%, the log-average miss rate (MR-2) decreased from 22% to 14%, and the Recall rate increased from 91.36% to 95.12%.

MISD-SLAM: Multimodal Semantic SLAM for Dynamic Environments

Simultaneous localization and mapping (SLAM) is one of the most essential technologies for mobile robots. Although great progress has been made in the field of SLAM in recent years, there are a number of challenges for SLAM in dynamic environments and high-level semantic scenes. In this paper, we propose a novel multimodal semantic SLAM system (MISD-SLAM), which removes the dynamic objects in the environments and reconstructs the static background with semantic information. MISD-SLAM builds three main processes: instance segmentation, dynamic pixels removal, and semantic 3D map construction. An instance segmentation network is used to provide semantic knowledge of surrounding environments in instance level. The ORB features located on the predefined dynamic objects are removed directly. In this way, MISD-SLAM effectively reduces the impact of dynamic objects to provide precise pose estimation. Then, combining multiview geometry constraint with K -means clustering algorithm, our system removes the undefined but moving pixels. Meanwhile, a 3D dense point cloud map with semantic information is reconstructed, which recovers the static background without the corruptions of dynamic objects. Finally, we evaluate MISD-SLAM by comparing to ORB-SLAM3 and the state-of-the-art dynamic SLAM systems in TUM RGB-D datasets and real-world dynamic indoor environments. The results indicate that our method significantly improves the localization accuracy and system robustness, especially in high-dynamic environments.

Improving unsupervised image clustering with spatial consistency

Unsupervised image clustering (UIC) is regularly employed to group images without manual annotation. One significant problem that occurs in the UIC context is that the visual-feature similarity across different semantic classes tends to introduce instance-dependent errors to clustering. The most successful recent approaches aimed at resolving this problem have focused on semisupervised reclustering, which utilizes reliable samples selected from the existing clusters. Despite this, virtually no previous work has considered the spatial consistency of the instance- and class-level representations which is crucial for error disambiguation. This makes it difficult to assess whether the selected reliable sample is reasonable. Accordingly, we propose a spatial consistency-based clustering (SCC) method to retain the alignment of representations learned from the instance and class levels in order to effectively select reliable samples from pre-existing clusters with errors thus leading to better clustering performance. More specifically, we first learn instance- and class-level representations by encouraging the semantic invariants of different instance augmentations and enforcing class alignment across semantically similar instances, respectively. We then assign instances with similar class-level representations to the same cluster to obtain the preliminary clusters. Subsequently we assess sample reliability by utilizing the spatial consistency constraint, which diffuses the class-level representations within the instance-level representation space. Finally, we employ semisupervised baselines combined with refinement techniques to perform reclustering based on the selected reliable samples. Extensive experimental results demonstrate that SCC can effectively obtain credible samples and outperform current SOTA clustering methods on the CIFAR-10 and CIFAR-100-20 benchmarks. The relevant code is available at https://github.com/RyanZhaoIc/SCC.git.

Close-set camera style distribution alignment for single camera person re-identification

The purpose of person re-identification (ReID) is to find the same person under different cameras and the basic difficulty lies in the need for large amounts of cross-camera pedestrian annotations. In reality, annotating cross-camera pedestrians is time-consuming especially in large-scale surveillance camera networks. This paper focuses on addressing the ReID problem under single-camera training (SCT) setting, where each person of the training set only appears in one camera. Due to the lack of cross-camera pedestrian annotations, it is difficult to effectively eliminate the camera style interference by narrowing the distance between the image features of the same person. To address this problem, we propose a close-set camera style distribution alignment (C2SDA) framework for SCT ReID. In order to reduce the camera-style interference from both instance- and distribution-levels simultaneously, we first design an instance-distribution camera style alignment module that directly aligns the feature distribution of input images under each camera and then trains the model at instance level with the aligned features. Secondly, we further design an augment close-set camera style distribution module that transforms the camera feature distribution alignment problem from open-set into close-set one, which preserving the discriminative ability of features during the alignment process. Experimental results verify that our framework can significantly improve the ReID performance under SCT setting and surpass the current SOTA methods. The source code is available athttps://github.com/HongweiZhang97/CCSDA.

Learning to optimise general TSP instances

The Travelling Salesman Problem is a classical combinatorial optimisation problem (COP). In recent years, learning to optimise approaches have shown success in solving TSP problems. However, they focus on one type of TSP instance, where the points are uniformly distributed in Euclidean spaces (easy instances). Such approaches cannot generalise to other embedding spaces that represent various levels of difficult instances, e.g., TSP instances where the points are distributed in a non-uniform manner and spherical spaces. Obtain optimal solutions for easy instances is achievable and can be used as training data to solve various TSP instances. However, acquire optimal solutions for complex TSP instances is difficult and time-consuming. Hence, this paper introduces a new learning-based approach based on a convolutional neural network combined with a Long Short-Term Memory, referred to as the non-Euclidean TSP network (NETSP), that utilises randomly generated instances (easy instances) to solve various common TSP instances (complex TSP instances). We have demonstrated its superiority over state-of-the-art methods for various TSP instances. We performed extensive experiments that indicate our approach generalises across many instances and scales to larger instances.

Identification and characterization of stress responsive homeodomain leucine zipper transcription factors in Medicago truncatula.

Homeodomain leucine zipper (HD-ZIP) transcription factors play roles in regulating plant development and responses to abiotic stresses; however, how HD-ZIP genes in Medicago truncatula are involved in abiotic stress response remains elusive. The HD-ZIP I genes in Medicago truncatula were identified and characterized, and their expression patterns in different tissues and under different abiotic stresses were analyzed. A total of 15 Medicago truncatula HD-ZIP I genes were identified and a phylogenetic analysis of HD-ZIP I proteins in Arabidopsis thaliana and Medicago truncatula was conducted. Fifteen HD-ZIP I genes showed diverse tissue preferences. Among them, expressions of MtHB22 and MtHB51 were specially detected in vegetative buds. In addition, they responded to various abiotic stresses, including salinity and osmotic stress and abscisic acid (ABA). For instance, MtHB7 and MtHB12 expression levels were found to be positively associated with salt, osmotic stress and ABA in both shoots and roots, while MtHB13 and MtHB23 were negatively associated with these stresses in Medicago truncatula. The HD-ZIP I genes in Medicago truncatula are evolutionarily conserved, but also exhibit gene duplication and gene loss events. Differential expression analysis of Medicago truncatula HD-ZIP I genes indicated their crucial roles in abiotic stress responses. Our genome-wide analysis of the HD-ZIP I transcription factor family in Medicago truncatula provided a valuable reference for further research.