Difficulty Of Instances Research Articles

Multi-instance imbalance semantic segmentation by instance-dependent attention and adaptive hard instance mining

Multi-instance imbalanced semantic segmentation (MISS) is a common issue where larger foreground instances can dominate smaller ones, yet still produce satisfactory Dice. Detecting small instances is crucial for many applications. For example, in the diagnosis of coronary artery disease, it is essential to locate small-scale lesions. To address the challenge of MISS, we propose MISS-Net (MISS-Network), which considers the distribution characteristics and segmentation difficulty of instances. On one hand, given the spatially sparse distribution characteristics of instances, we propose an instance-dependent SparseViT. SparseViT learns unstructured sparse attention to capture instance-related patterns by computing content-aware attention scores and selecting the top-K indices. On the other hand, we propose a hard instance-aware loss that identifies ‘hard-to-segment’ instances through a dynamic instance weighting strategy. Combined with a global segmentation loss, it produces a trade-off between the global-level and instance-level segmentation performance. We extensively evaluated MISS-Net on three challenging 3D semantic segmentation tasks: two coronary artery calcification (CAC) datasets and one liver tumor (LITS) dataset. These datasets contain a large number of instances with significant variability in instance volumes (e.g., in the LITS dataset, a patient has an average of 12.39 instances, with instance volumes ranging from 987.66cm3 to 0.04cm3). The results demonstrate that MISS-Net significantly improves both instance-level and global-level segmentation performance. For the CAC task, MISS-Net improves Dice by 3.6% and 6.4%, and instance F1 by 3.1% and 1.9%. For the LITS dataset, Dice and instance F1 are improved by 1.6% and 3.8%, respectively.

Data Complexity: A New Perspective for Analyzing the Difficulty of Defect Prediction Tasks

Defect prediction is crucial for software quality assurance and has been extensively researched over recent decades. However, prior studies rarely focus on data complexity in defect prediction tasks, and even less on understanding the difficulties of these tasks from the perspective of data complexity. In this article, we conduct an empirical study to estimate the hardness of over 33,000 instances, employing a set of measures to characterize the inherent difficulty of instances and the characteristics of defect datasets. Our findings indicate that: (1) instance hardness in both classes displays a right-skewed distribution, with the defective class exhibiting a more scattered distribution; (2) class overlap is the primary factor influencing instance hardness and can be characterized through feature, structural, and instance-level overlap; (3) no universal preprocessing technique is applicable to all datasets, and it may not consistently reduce data complexity, fortunately, dataset complexity measures can help identify suitable techniques for specific datasets; (4) integrating data complexity information into the learning process can enhance an algorithm’s learning capacity. In summary, this empirical study highlights the crucial role of data complexity in defect prediction tasks, and provides a novel perspective for advancing research in defect prediction techniques.

Long-Tailed Object Detection for Multimodal Remote Sensing Images

With the rapid development of remote sensing technology, the application of convolutional neural networks in remote sensing object detection has become very widespread, and some multimodal feature fusion networks have also been proposed in recent years. However, these methods generally do not consider the long-tailed problem that is widely present in remote sensing images, which limits the further improvement of model detection performance. To solve this problem, we propose a novel long-tailed object detection method for multimodal remote sensing images, which can effectively fuse the complementary information of visible light and infrared images and adapt to the imbalance between positive and negative samples of different categories. Firstly, the dynamic feature fusion module (DFF) based on image entropy can dynamically adjust the fusion coefficient according to the information content of different source images, retaining more key feature information for subsequent object detection. Secondly, the instance-balanced mosaic (IBM) data augmentation method balances instance sampling during data augmentation, providing more sample features for the model and alleviating the negative impact of data distribution imbalance. Finally, class-balanced BCE loss (CBB) can not only consider the learning difficulty of specific instances but also balances the learning difficulty between categories, thereby improving the model’s detection accuracy for tail instances. Experimental results on three public benchmark datasets show that our proposed method achieves state-of-the-art performance; in particular, the optimization of the long-tailed problem enables the model to meet various application scenarios of remote sensing image detection.

Understand Restart of SAT Solver Using Search Similarity Index (Student Abstract)

SAT solvers are widely used to solve many industrial problems because of their high performance, which is achieved by various heuristic methods. Understanding why these methods are effective is essential to improving them. One approach to this is analyzing them using qualitative measurements. In our previous study, we proposed search similarity index (SSI), a metric to quantify the similarity between searches. SSI significantly improved the performance of the parallel SAT solver. Here, we apply SSI to analyze the effect of restart, a key SAT solver technique. Experiments using SSI reveal the correlation between the difficulty of instances and the search change effect by restart, and the reason behind the effectiveness of the state-of-the-art restart method is also explained.

A regression analysis of the impact of routing and packing dependencies on the expected runtime

Problems with multiple interdependent components offer a better representation of the real-world situations where globally optimal solutions are preferred over optimal solutions for the individual components. One such model is the Travelling Thief Problem (TTP); while it may offer a better benchmarking alternative to the standard models, only one form of inter-component dependency is investigated. The goal of this paper is to study the impact of different models of dependency on the fitness landscape using performance prediction models (regression analysis). To conduct the analysis, we consider a generalised model of the TTP, where the dependencies between the two components of the problem are tunable through problem features. We use regression trees to predict the instance difficulty using an efficient memetic algorithm that is agnostic to the domain knowledge to avoid any bias. We report all the decision trees resulting from the regression model, which is the core in understanding the relationship between the dependencies (represented by the features) and problem difficulty (represented by the runtime). The regression model was able to predict the expected runtime of the algorithm based on the problem features. Furthermore, the results show that the contribution of the item value drop dependency is significantly higher than the velocity change dependency.

Crowdmeta: Crowdsourcing truth inference with meta-Knowledge transfer

Crowdsourcing provides a fast and low-cost solution to collect annotations for training data in computer vision. However, there are two challenges in crowdsourced image annotation: First, when crowdsourced workers perform annotation tasks in an unfamiliar domain, their accuracy will dramatically decline due to the lack of expertise; Second, the difficulties of tasks may be different due to the noises in images, which is only related to the features of images themselves and will affect the judgment of workers. It is well known that transferring knowledge from relevant domains can form a better representation for training samples, which benefits the estimation of workers’ expertise in truth inference models. However, the existing knowledge transfer processes for crowdsourcing require a considerable number of well-collected samples in source domains. Comprehensively considering the above issues, this paper proposes a novel probabilistic model for crowdsourcing truth inference, which fuses few-shot meta-learning and transfer learning. The proposed model transfers meta-knowledge from the source domain to form better high-level representations of the instances in the target domain. Simultaneously utilizing both high-level representations and instance features, the quality of workers and the difficulty of instances can be better modeled and inferred. Experimental results on a number of datasets show that the proposed model not only outperforms the state-of-the-art models but also significantly reduces the number of instances required in the source domain.

Multiple Instance Complementary Detection and Difficulty Evaluation for Weakly Supervised Object Detection in Remote Sensing Images

Weakly supervised object detection (WSOD) in remote sensing images (RSIs) has attracted lots of attention because it solely employs image-level labels to drive the model training. Most of the WSOD methods incline to mine salient object as positive instance, and the less salient objects are considered as negative instances, which will cause the problem of missing instances. In addition, the quantity of hard and easy instances is usually imbalanced, consequently, the cumulative loss of a large amount of easy instances dominates the training loss, which limits the upper bound of WSOD performance. To handle the first problem, a complementary detection network (CDN) is proposed, which consists of a complementary multiple instance detection network (CMIDN) and a complementary feature learning (CFL) module. The CDN can capture robust complementary information from two basic multiple instance detection networks (MIDNs) and mine more object instances. To handle the second problem, an instance difficulty evaluation metric named instance difficulty score (IDS) is proposed, which is employed as the weight of each instance in the training loss. Consequently, the hard instances will be assigned larger weights according to the IDS, which can improve the upper bound of WSOD performance. The ablation experiments demonstrate that our method significantly increases the baseline method by large margins, i.e. 23.6% (10.2%) mAP and 32.4% (13.1%) CorLoc gains on the NWPU VHR-10.v2 (DIOR) dataset. Our method obtains 58.1% (26.7%) mAP and 72.4% (47.9%) CorLoc on the NWPU VHR-10.v2 (DIOR) dataset, which achieves better performance compared with seven advanced WSOD methods.

Multiscale Image Splitting Based Feature Enhancement and Instance Difficulty Aware Training for Weakly Supervised Object Detection in Remote Sensing Images

Multiscale Image Splitting Based Feature Enhancement and Instance Difficulty Aware Training for Weakly Supervised Object Detection in Remote Sensing Images

Curriculum-Style Fine-Grained Adaption for Unsupervised Cross-Lingual Dependency Transfer

Unsupervised cross-lingual transfer has been shown great potentials for dependency parsing of the low-resource languages when there is no annotated treebank available. Recently, the self-training method has received increasing interests because of its state-of-the-art performance in this scenario. In this work, we advance the method further by coupling it with curriculum learning, which guides the self-training in an easy-to-hard manner. Concretely, we present a novel metric to measure the instance difficulty of a dependency parser which is trained mainly on a Treebank from a resource-rich source language. By using the metric, we divide a low-resource target language into several fine-grained sub-languages by their difficulties, and then apply iterative-self-training progressively on these sub-languages. To fully explore the auto-parsed training corpus from sub-languages, we exploit an improved parameter generation network to model the sub-languages for better representation learning. Experimental results show that our final curriculum-style self-training can outperform a range of strong baselines, leading to new state-of-the-art results on unsupervised cross-lingual dependency parsing. We also conduct detailed experimental analyses to examine the proposed approach in depth for comprehensive understandings.

Local Optima Network Analysis of Multi-Attribute Vehicle Routing Problems

Multi-Attribute Vehicle Routing Problems (MAVRP) are variants of Vehicle Routing Problems (VRP) in which, besides the original constraint on vehicle capacity present in Capacitated Vehicle Routing Problem (CVRP), other attributes that model diverse real-life system characteristics are present. Among the most common attributes studied in the literature are vehicle capacity and route duration constraints. The influence of these restrictions on the overall structure of the problem and the performance of local search algorithms used to solve it has yet to be well known. This paper aims to explain the impact of constraints present in different variants of VRP through the alterations of the structure of the underlying search space that they cause. We focus on Local Optima Network Analysis (LONA) for multiple Traveling Salesman Problem (m-TSP) and VRP with vehicle capacity (CVRP), route duration (DVRP), and both (DCVRP) constraints. We present results that indicate that measures obtained for a sample of local optima provide valuable information on the behavior of the landscape under modifications in the problem’s constraints. Additionally, we use the LONA measures to explain the difficulty of VRP instances for solving by local search algorithms.

Supervised Machine Learning for Understanding and Improving the Computational Performance of Chemical Production Scheduling MIP Models

We adopt a supervised learning approach to predict runtimes of batch production scheduling mixed-integer programming (MIP) models with the aim of understanding what instance features make a model computationally expensive. We introduce novel features to characterize instance difficulty according to problem type. The developed machine learning models trained on runtime data obtained from a wide variety of instances show good predictive performances. Then, we discuss informative features and their effects on computational performance. Finally, based on the derived insights, we propose solution methods for improving the computational performance of batch scheduling MIP models.

Instance difficulty-based noise correction for crowdsourcing

Crowdsourcing offers an efficient way to obtain a multiple noisy label set of each instance from different crowd workers and then label integration algorithms are used to infer its integrated label. In spite of the effectiveness of label integration algorithms, there always exists a certain degree of noise in the integrated labels, and thus noise correction algorithms have been proposed to reduce the effect of noise. However, existing noise correction algorithms seldom consider the effect of instance difficulty on noise correction. In this paper, we argue that the greater the difficulty of an instance, the fewer crowd workers can label it correctly, and the more likely the instance is a noise instance. Based on this premise, we propose a simple but very effective noise correction algorithm called instance difficulty-based noise correction (IDNC). In IDNC, we at first propose two methods to measure the difficulty of each instance. Then, we use the proposed two methods to filter the noise instances to obtain a clean set and a noise set. Finally, we build two different classifiers on the clean set to correct the noise instances in the noise set via the consensus voting. The extensive experiments on both simulated and real-world crowdsourced datasets validated the effectiveness and efficiency of our proposed IDNC.

Learning from biased crowdsourced labeling with deep clustering

With the rapid development of crowdsourcing learning, amount of labels can be obtained from crowd workers fast and cheaply. However, crowdsourcing learning also faces challenges due to the varied qualities of amateurish crowd workers. To improve the quality of crowd labels, many researchers focus on inferring the ground truth from noisy labels, and take different factors, e.g. the reliability of workers and the difficulty of instances, into consideration to infer the aggregated labels. Nevertheless, to the best of our knowledge, label aggregation for biased crowdsourced labeling scenarios has not been sufficiently studied. Actually, the phenomenon of biased labeling exists in many crowdsourcing annotation tasks and affects the performance of label aggregation. To this end, this paper proposes a novel framework termed Biased Crowdsourcing Learning with Deep Clustering (BCLDC), which involves label aggregation and prediction using deep clustering to improve the quality of aggregated labels and learned models in biased labeling scenarios. BCLDC utilizes a deep clustering method to detect the labeling bias and then eliminates the bias by adjusting the number of labels belonging to the minority class which has fewer labels. Finally, a classifier is trained simultaneously with the aggregated labels inferred by an EM algorithm. Experimental results on six real-world datasets and five synthetic datasets consistently show that the proposed BCLDC outperforms other state-of-the-art algorithms in terms of ground truth inference and prediction.

When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.

Variable Neighborhood Search for the Two-Echelon Electric Vehicle Routing Problem with Time Windows

Increasing environmental concerns and legal regulations have led to the development of sustainable technologies and systems in logistics, as in many fields. The adoption of multi-echelon distribution networks and the use of environmentally friendly vehicles in freight distribution have become major concepts for reducing the negative impact of urban transportation activities. In this line, the present paper proposes a two-echelon electric vehicle routing problem. In the first echelon of the distribution network, products are transported from central warehouses to satellites located in the surroundings of cities. This is achieved by means of large conventional trucks. Subsequently, relatively smaller-sized electric vehicles distribute these products from the satellites to demand points/customers located in the cities. The proposed problem also takes into account the limited driving range of electric vehicles that need to be recharged at charging stations when necessary. In addition, the proposed problem considers time window constraints for the delivery of products to customers. A mixed-integer linear programming formulation is developed and small-sized instances are solved using CPLEX. Furthermore, we propose a constructive heuristic based on a modified Clarke and Wright savings heuristic. The solutions of this heuristic serve as initial solutions for a variable neighborhood search metaheuristic. The numerical results show that the variable neighborhood search matches CPLEX in the context of small problems. Moreover, it consistently outperforms CPLEX with the growing size and difficulty of problem instances.

Instance Segmentation in Very High Resolution Remote Sensing Imagery Based on Hard-to-Segment Instance Learning and Boundary Shape Analysis

Although great success has been achieved in instance segmentation, accurate segmentation of instances remains difficult, especially at object edges. This problem is more prominent for instance segmentation in remote sensing imagery due to the diverse scales, variable illumination, smaller objects, and complex backgrounds. We find that most current instance segmentation networks do not consider the segmentation difficulty of different instances and different regions within the instance. In this paper, we study this problem and propose an ensemble method to segment instances from remote sensing images, considering the enhancement of hard-to-segment instances and instance edges. First, we apply a pixel-level Dice metric that reliably describes the segmentation quality of each instance to achieve online hard instance learning. Instances with low Dice values are studied with emphasis. Second, we generate a penalty map based on the analysis of boundary shapes to not only enhance the edges of objects but also discriminatively strengthen the edges of different shapes. That is, different areas of an object, such as internal areas, flat edges, and sharp edges, are distinguished and discriminatively weighed. Finally, the hard-to-segment instance learning and the shape-penalty map are integrated for precise instance segmentation. To evaluate the effectiveness and generalization ability of the proposed method, we train with the classic instance segmentation network Mask R-CNN and conduct experiments on two different types of remote sensing datasets: the iSAID-Reduce100 and the JKGW_WHU datasets, which have extremely different feature distributions and spatial resolutions. The comprehensive experimental results show that the proposed method improved the segmentation results by 2.78% and 1.77% in mask AP on the iSAID-Reduce100 and JKGW_WHU datasets, respectively. We also test other state-of-the-art (SOTA) methods that focus on inaccurate edges. Experiments demonstrate that our method outperforms these methods.

An instance-oriented performance measure for classification

Performance evaluation is significant in data classification. The existing evaluation methods ignore the characteristics (such as classification difficulty) of each instance. In practice, it is necessary to measure classification performance from the perspective of instances. In this paper, an instance-oriented classification performance metric is proposed based on the classification difficulty of each instance, named degree of credibility (Cr ). Cr conforms to the natural cognition that the lower the probability of misclassifying relatively easy instances, the more credible the classifier. It focuses on the credibility of each instance’s prediction, which opens up a new way for classifier evaluation. Moreover, several important properties of Cr are identified, laying solid theoretical foundation for classifier evaluation. Also, the concept of acceptable classifier is proposed to judge whether the trained model and its parameter set reach excellent ranks at the current technology level instead of relying entirely on human experience. The experimental results of twelve classifiers on twelve datasets indicate the physical significance and good statistical consistency and discriminatory ability of Cr, as well as the feasibility of acceptable classifiers for model selection and training. Furthermore, the proposal of approximate difficulty greatly improves the computation efficiency of instance difficulty.

Learning from Crowds by Modeling Common Confusions

Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low cost. However, the annotation quality of annotators varies considerably, which imposes new challenges in learning a high-quality model from the crowdsourced annotations. In this work, we provide a new perspective to decompose annotation noise into common noise and individual noise and differentiate the source of confusion based on instance difficulty and annotator expertise on a per-instance-annotator basis. We realize this new crowdsourcing model by an end-to-end learning solution with two types of noise adaptation layers: one is shared across annotators to capture their commonly shared confusions, and the other one is pertaining to each annotator to realize individual confusion. To recognize the source of noise in each annotation, we use an auxiliary network to choose from the two noise adaptation layers with respect to both instances and annotators. Extensive experiments on both synthesized and real-world benchmarks demonstrate the effectiveness of our proposed common noise adaptation solution.

Review and Arrange: Curriculum Learning for Natural Language Understanding

With the notable success of pretrained language models, the pretraining-fine-tuning paradigm has become a dominant solution for natural language understanding (NLU) tasks. Typically, the training instances of a target NLU task are introduced in a completely random order and treated equally at the fine-tuning stage. However, these instances can vary greatly in difficulty, and similar to human learning procedures, language models can benefit from an easy-to-difficult curriculum. Based on this concept, we propose a curriculum learning (CL) framework. Our framework consists of two stages, Review and Arrange, targeting the two main challenges in curriculum learning, i.e., how to define the difficulty of instances and how to arrange a curriculum based on the difficulty, respectively. In the first stage, we devise a cross-review (CR) method to train several teacher models first and then review the training set in a crossed manner to distinguish easy instances from difficult instances. In the second stage, two sampling algorithms, a coarse-grained arrangement (CGA) and a fine-grained arrangement (FGA), are proposed to arrange a curriculum for language models in which the learning materials start from the easiest instances, and more difficult instances are gradually added into the training procedure. Compared to previous heuristic CL methods, our framework can avoid the errors caused by a gap in difficulty between humans and machines and has strong generalization ability. We conduct comprehensive experiments, and the results show that our curriculum learning framework, without any manual model architecture design or use of external data, obtains significant and universal performance improvements on a wide range of NLU tasks in different languages.

BIDI: A classification algorithm with instance difficulty invariance

In artificial intelligence, an expert/intelligent systems can emulate the decision-making ability of human experts. A good classification algorithm can provide significant assistance to expert/intelligent systems in solving a variety of practical problems. In classification, the “hard” instances may be outliers or noisy instances that are difficult to learn, which may confuse the classifier and induce the overfitting problem in the case of placing much emphasis on them. In fact, the difficulty of instances is crucial for improving the generalization and credibility of classification. Unfortunately, nearly all the existing classifiers ignore this important information. In this paper, the classification difficulty of each instance is introduced from a statistical perspective, which is an inherent characteristic of the instance itself. Then, a new classification algorithm named “boosting with instance difficulty invariance (BIDI)” is proposed by incorporating the classification difficulty of instances. The BIDI conforms to the human cognition that easy instances are misclassified with a lower probability than difficult ones, and performs better with respect to generalization. The key insight of BIDI can provide relevant guidance for researchers to improve the generalization and credibility of classifiers in the expert systems of decision support systems. Experimental results demonstrate the effectiveness of BIDI in real-world data sets, indicating that it has great potential for solving many classification tasks of expert systems such as disease diagnosis and credit card fraud detection. Although the classification difficulty has strong statistical significance, its implementation remains computationally expensive. A fast method demonstrating rationality and feasibility is also proposed to approximate instances’ classification difficulty.