Set Of Training Instances Research Articles

Finding BERT Errors by Clustering Activation Vectors

The non-linear nature of deep neural networks makes it difficult to interpret the reason behind their output, thus reducing verifiability of the system where these models are applied. Understanding the patterns between activation vectors and predictions could give insight as to erroneous classifications and how to identify them. This paper explains a systematic approach to identifying the clusters with the most misclassifications or false label annotations. For this research, we extracted the activation vectors from a deep learning model, DNABERT, and visualized them using t-SNE to decode the reason behind the results that are produced. We applied K-means in a hierarchical fashion on the activation vectors for a set of training instances. We analyzed cluster mean activation vectors to find any patterns in the errors across K-means clusters. The cluster analysis revealed that the predictions were uniform, or nearly 100 percent the same, in clusters of similar activation vectors. It was found that two clusters containing most of their objects belonging to the same true class tend to be closer together than clusters of opposite classes. The means of objects of the same true label are closer if two clusters have the same predicted labels rather than opposite predicted labels, showing that the activation vectors reflect both predicted and true classes. We did a similar analysis for all 26 organisms in the dataset, showing the Euclidean distance can be used for identifying clusters with many errors. We propose a heuristic to find the clusters with a high number of misclassifications or incorrect label annotations using the vector analysis between clusters. This can aid in identifying misclassifications of DNA sequences or problems with sequence tagging.

Learn to optimize-a brief overview.

Most optimization problems of practical significance are typically solved by highly configurable parameterized algorithms. To achieve the best performance on a problem instance, a trial-and-error configuration process is required, which is very costly and even prohibitive for problems that are already computationally intensive, e.g.optimization problems associated with machine learning tasks. In the past decades, many studies have been conducted to accelerate the tedious configuration process by learning from a set of training instances. This article refers to these studies as learn to optimize and reviews the progress achieved.

Learning shared and non-redundant label-specific features for partial multi-label classification

Partial multi-label learning (PML) is designed to address the challenge of having both ground-truth labels and noisy labels in the label set of training instances. In real-world applications, there are often noisy features in addition to noisy labels, but existing PML methods fail to filter out these noisy features or account for feature correlations, label correlations, and feature-label correlations together in the feature learning process. These oversights lead to poor classification accuracy, as the comprehensive exploration of feature and label information in PML can contribute significantly to disambiguation. To address this issue, we propose a novel PML framework that disambiguates candidate labels and learns label-specific features while taking into account all three types of correlations between features and labels. First, we capture high-order label correlations by employing the low-rank representation to obtain a complementary label matrix induced from the partial label matrix, which accounts for the presence of noisy labels. Then, we learn a shared and non-redundant label-specific data representation by incorporating intrinsic feature correlations and the learned label correlations to mitigate the impact of noisy features. Finally, we build a classifier by mapping the label-specific feature representation to the complementary label matrix to model the feature-label correlations. Various experiments validate the superiority of our method.

Exemplar-model account of categorization and recognition when training instances never repeat.

In a novel version of the classic dot-pattern prototype-distortion paradigm of category learning, Homa et al. (2019) tested a condition in which individual training instances never repeated, and observed results that they claimed severely challenged exemplar models of classification and recognition. Among the results was a dissociation in which participants classified transfer items with high accuracy in the no-repeat condition, yet in old-new recognition tests showed no ability to discriminate between old and new items of the same level of distortion from the prototype. In addition, speed of classification learning was no faster in a condition in which a small set of training instances was repeated continuously compared with the no-repeat condition. Here we show through computer-simulation modeling that exemplar models naturally capture the classification-recognition dissociation in the no-repeat condition, as well as a wide variety of other qualitative effects reported by Homa et al. (2019). We also conduct new conceptual-replication experiments to investigate their reported null effect of repeated versus nonrepeated training instances on speed of classification learning. In contrast to Homa et al. (2019) we find that speed of learning is substantially faster in the repeat condition than in the no-repeat condition, precisely as exemplar models predict. The exemplar model also captures a wide variety of transfer effects observed following the completion of category learning, including the classification-recognition dissociation observed across the repeat and no-repeat conditions. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Framework of algorithm portfolios for strip packing problem

In this paper selection of fast algorithm portfolios for 2SP packing problem is considered. The 2SP problem consists in placing rectangles on a strip of the given width for minimum strip length. The 2SP packing has application in many industries, but suitability of the related algorithms is limited by their runtimes. While solving combinatorial optimization problems, longer runtimes increase chances of obtaining higher quality solutions. This means that runtime vs solution quality trade-off is important in solving problems such as strip packing. Given some limited runtime, a method is needed to provide the best solution possible. However, a single algorithm outperforming all other methods under all possible conditions usually does not exist. Therefore, algorithm portfolios can reliably provide high quality solutions in the limited runtime. We propose a method choosing algorithm portfolios on the basis of the algorithm performance on a set of training instances. A portfolio covers the instances with the best solutions which could be obtained in the given runtime, subject to the minimum computational cost of the selected algorithms. The portfolios are evaluated in extensive experiments carried out on designed and literature datasets. We demonstrate that our method is capable of carrying over solution quality from the training datasets to the testing datasets. In other words, our algorithm selection method can learn from the training instances. We also compare performance of our portfolio selection method with some other more straightforward approaches to the portfolio selection.

A Sampling-Based Stack Framework for Imbalanced Learning in Churn Prediction

Churn prediction is gaining popularity in the research community as a powerful paradigm that supports data-driven operational decisions. Datasets related to churn prediction are often skewed with imbalanced class distribution. Data-level solutions, like over-sampling and under-sampling, have been commonly used by researchers to address this problem. There are limited number of case studies that attempt to evolve these data-level solutions by integrating them with computationally advanced frameworks, like ensembles. Ensembles primarily employ algorithmic diversity using a fixed set of training instances to achieve superior performance. This study aims to introduce algorithmic diversity in ensembles by modifying the fixed set of training instances using diverse sampling strategies to increase predictive performance in imbalanced learning. Data is acquired from the world’s largest open hotel commerce platform company. A four-part series of experiments is conducted to analyze the effectiveness of sampling techniques and ensemble solutions on model performance. A new sampling-based stack framework called “Stacking of Samplers for Imbalanced Learning” is proposed. The framework combines the prediction capabilities of sampling solutions to stimulate the information gain of the meta features in ensemble. It is observed that the proposed framework leads to improvement in model performance with AUC of 86.4% and top-decile lift of 4.7 for customers of the hotel technology provider. Additionally, results show that the framework records a higher information gain for meta features used in a stack, compared to commonly used stack frameworks.

Enhancing performance of gene expression value prediction with cluster-based regression.

The inherent correlations among gene expressions have received attention. Recently, it was reported that a set of approximately 1000 landmark genes can be utilized for prediction of expression of other genes (target genes). The objective of this study is to predict expression values of target genes based on expression values of landmark genes. A cluster-based regression method is proposed. In the proposed method, clusters are obtained from a set of training instances of a gene and an estimator is obtained per cluster. A test instance is assigned to one of clusters then a regression model corresponding to the cluster predicts expression value. Performance of the proposed method is measured on the GEO (Gene Expression Omnibus) expression data and the GTEx (Genotype-Tissue Expression) expression data. In terms of mean absolute error averaged across target genes, the proposed method significantly outperforms previous approaches in the case of the GEO expression data. The experimental results report that the combination of clustering and regression can outperform the state-of-the art methods such as generative adversarial networks and a gradient boosting based method.

A Genetic Programming Approach for Evolving Variable Selectors in Constraint Programming

Operational researchers and decision modelers have aspired to optimization technologies with a self-adaptive mechanism to cope with new problem formulations. Self-adaptive mechanisms not only free users from low-level and complex development tasks to enhance optimization efficiency but also allow them to focus on addressing high-level real-world operational requirements. In recent years, there has been a growing interest in applying machine learning and artificial intelligence techniques to improve self-adaptive mechanisms. However, learning to optimize hard combinatorial optimization problems remains a challenging task. This article proposes a new genetic programming approach to evolve efficient variable selectors to enhance the search mechanism in constraint programming. Starting with a set of training instances for a specific combinatorial optimization problem, the proposed approach evaluates variable selectors and evolves them to be more efficient over a number of generations. The novelties of our proposed approach are threefold: 1) a new representation of variable selectors; 2) a new mechanism for fitness evaluations; and 3) a preselection technique. We examine performance of the proposed approach on different job-shop scheduling problems, and the results show that variable selectors can be evolved efficiently. In particular, there are substantial reductions in the computational effort required for the search component of the constraint solver as well as increased chances of finding the optimal solutions. Further analyses also confirm the efficacy of our approach in respect to scalability, generalization, and interpretability of the evolved variable selectors.

Local-Set Based-on Instance Selection Approach for Autonomous Object Modelling

With the increasing presence of robotic agents in our daily life, computationally efficient modelling of real-world objects by autonomous systems is of prime importance for enabling these artificial agents to automatically and effectively perform tasks such as visual object recognition. For this purpose, we introduce a novel, machine-learning approach for instance selection called Approach for Selection of Border Instances (ASBI). This method adopts the notion of local sets to select the most representative instances at the boundaries of the classes, in order to reduce the set of training instances and, consequently, to reduce the computational resources that are necessary to perform the learning process of real-world objects by the artificial agents. Our new algorithm was validated on 27 standard datasets and applied on 2 challenging object-modelling datasets to test the automated object recognition task. ASBI performances were compared to those of 6 state-of-art algorithms, considering three standard metrics, namely, accuracy, reduction, and effectiveness. All the obtained results show that the proposed method is promising for the autonomous recognition task, while presenting the best trade-off between the classification accuracy and the data size reduction.

Sketch recognition with few examples

Sketch recognition is the task of converting hand-drawn digital ink into symbolic computer representations. Since the early days of sketch recognition, the bulk of the work in the field focused on building accurate recognition algorithms for specific domains, and well defined data sets. Recognition methods explored so far have been developed and evaluated using standard machine learning pipelines and have consequently been built over many simplifying assumptions. For example, existing frameworks assume the presence of a fixed set of symbol classes, and the availability of plenty of annotated examples. However, in practice, these assumptions do not hold. In reality, the designer of a sketch recognition system starts with no labeled data at all, and faces the burden of data annotation. In this work, we propose to alleviate the burden of annotation by building systems that can learn from very few labeled examples, and large amounts of unlabeled data. Our systems perform self-learning by automatically extending a very small set of labeled examples with new examples extracted from unlabeled sketches. The end result is a sufficiently large set of labeled training data, which can subsequently be used to train classifiers. We present four self-learning methods with varying levels of implementation difficulty and runtime complexities. One of these methods leverages contextual co-occurrence patterns to build verifiably more diverse set of training instances. Rigorous experiments with large sets of data demonstrate that this novel approach based on exploiting contextual information leads to significant leaps in recognition performance. As a side contribution, we also demonstrate the utility of bagging for sketch recognition in imbalanced data sets with few positive examples and many outliers.

Severe-occluded 3D object identification via region-based descriptions

This paper describes a region-based strategy for part-based object identification with independence of the external factors that affect its captured image: light variations, capture point-of-view or occlusions. Starting from color images and depth estimations, i.e. not requiring 3-dimensional models, we focus on the identification of learned objects in severe-occlusion scenarios. To face this problem, we assume that objects have been preliminarily segregated from the scene. Strong changes of appearance—due to one or several of the aforementioned factors or to the object nature, e.g. deformable objects—substantially increase the problem complexity. The proposed algorithm operates by splitting segregated objects in successively coarser region-partitions, with each region representing a part of the object from which it was extracted. For the characterization of these parts, two region-driven descriptors are proposed: R-DAISY and R-SHOT. Their novelty relies on the use of a size-and-shape-variable description support which is automatically defined by the object part itself. Descriptions obtained in this way are self-organized in a single neural structure by an unsupervised learning process. Experimental results are promising in the identification of severe-occluded objects using a small set of training instances—1-to-8 short-varied views per object.

A discriminative model selection approach and its application to text classification

Classification is one of the fundamental problems in data mining, in which a classification algorithm attempts to construct a classifier from a given set of training instances with class labels. It is well known that some classification algorithms perform very well on some domains, and poorly on others. For example, NB performs well on some domains, and poorly on others that involve correlated features. C4.5, on the other hand, typically works better than NB on such domains. To integrate their advantages and avoid their disadvantages, many model hybrid approaches, such as model insertion and model combination, are proposed. In this paper, we focus on a novel view and propose a discriminative model selection approach, called discriminative model selection (DMS). DMS discriminatively chooses different single models for different test instances and retains the interpretability of single models. Empirical studies on a collection of 36 classification problems from the University of California at Irvine repository show that our discriminative model selection approach outperforms single models, model insertion approaches and model combination approaches. Besides, we apply the proposed discriminative model selection approach to some state-of-the-art naive Bayes text classifiers and also improve their performance.

An Optimization-Based Method for Feature Ranking in Nonlinear Regression Problems.

In this paper, we consider the feature ranking problem, where, given a set of training instances, the task is to associate a score with the features in order to assess their relevance. Feature ranking is a very important tool for decision support systems, and may be used as an auxiliary step of feature selection to reduce the high dimensionality of real-world data. We focus on regression problems by assuming that the process underlying the generated data can be approximated by a continuous function (for instance, a feedforward neural network). We formally state the notion of relevance of a feature by introducing a minimum zero-norm inversion problem of a neural network, which is a nonsmooth, constrained optimization problem. We employ a concave approximation of the zero-norm function, and we define a smooth, global optimization problem to be solved in order to assess the relevance of the features. We present the new feature ranking method based on the solution of instances of the global optimization problem depending on the available training data. Computational experiments on both artificial and real data sets are performed, and point out that the proposed feature ranking method is a valid alternative to existing methods in terms of effectiveness. The obtained results also show that the method is costly in terms of CPU time, and this may be a limitation in the solution of large-dimensional problems.

Dimensionality reduction by feature clustering for regression problems

One of the issues encountered in classification and regression is the processing inefficiency caused by a large number of input dimensions involved in the given training data set. Many dimensionality reduction approaches have been proposed to address this issue by reducing the number of input dimensions and maintaining the generalization capability of the original data set. However, less attention has been paid to regression than to classification. Besides, the computation with covariance matrices involved results in an inefficient reduction process in most existing methods. In this paper, we propose a machine learning based dimensionality reduction approach for regression problems. For a given set of training instances, a group of clusters are formed such that the instances included in the same cluster are similar to each other. Then one new feature is extracted from each cluster through a certain weighted combination of the training instances. Consequently, the dimensionality of the original data set is reduced. The clusters are created incrementally and automatically without the need of specifying the number of clusters in advance by the user. The characteristics of the original data set are substantially retained since all the original features are involved in the derivation of the extracted features. Also, the computation with covariance matrices is avoided, and thus efficiency is maintained. A number of experiments on real-world data sets are conducted to demonstrate the effectiveness of the proposed approach.

Planning through Automatic Portfolio Configuration: The PbP Approach

In the field of domain-independent planning, several powerful planners implementing different techniques have been developed. However, no one of these systems outperforms all others in every known benchmark domain. In this work, we propose a multi-planner approach that automatically configures a portfolio of planning techniques for each given domain. The configuration process for a given domain uses a set of training instances to: (i) compute and analyze some alternative sets of macro-actions for each planner in the portfolio identifying a (possibly empty) useful set, (ii) select a cluster of planners, each one with the identified useful set of macro-actions, that is expected to perform best, and (iii) derive some additional information for configuring the execution scheduling of the selected planners at planning time. The resulting planning system, called PbP (Portfolio- based Planner), has two variants focusing on speed and plan quality. Different versions of PbP entered and won the learning track of the sixth and seventh International Planning Competitions. In this paper, we experimentally analyze PbP considering planning speed and plan quality in depth. We provide a collection of results that help to understand PbP’s behavior, and demonstrate the effectiveness of our approach to configuring a portfolio of planners with macro-actions.

Heuristic Search When Time Matters

In many applications of shortest-path algorithms, it is impractical to find a provably optimal solution; one can only hope to achieve an appropriate balance between search time and solution cost that respects the user's preferences. Preferences come in many forms; we consider utility functions that linearly trade-off search time and solution cost. Many natural utility functions can be expressed in this form. For example, when solution cost represents the makespan of a plan, equally weighting search time and plan makespan minimizes the time from the arrival of a goal until it is achieved. Current state-of-the-art approaches to optimizing utility functions rely on anytime algorithms, and the use of extensive training data to compute a termination policy. We propose a more direct approach, called Bugsy, that incorporates the utility function directly into the search, obviating the need for a separate termination policy. We describe a new method based on off-line parameter tuning and a novel benchmark domain for planning under time pressure based on platform-style video games. We then present what we believe to be the first empirical study of applying anytime monitoring to heuristic search, and we compare it with our proposals. Our results suggest that the parameter tuning technique can give the best performance if a representative set of training instances is available. If not, then Bugsy is the algorithm of choice, as it performs well and does not require any off-line training. This work extends the tradition of research on metareasoning for search by illustrating the benefits of embedding lightweight reasoning about time into the search algorithm itself.

Heuristic search when time matters

In many applications of shortest-path algorithms, it is impractical to find a provably optimal solution; one can only hope to achieve an appropriate balance between search time and solution cost that respects the user's preferences. Preferences come in many forms; we consider utility functions that linearly trade-off search time and solution cost. Many natural utility functions can be expressed in this form. For example, when solution cost represents the makespan of a plan, equally weighting search time and plan makespan minimizes the time from the arrival of a goal until it is achieved. Current state-of-theart approaches to optimizing utility functions rely on anytime algorithms, and the use of extensive training data to compute a termination policy. We propose a more direct approach, called Bugsy, that incorporates the utility function directly into the search, obviating the need for a separate termination policy. We describe a new method based on off-line parameter tuning and a novel benchmark domain for planning under time pressure based on platform-style video games. We then present what we believe to be the first empirical study of applying anytime monitoring to heuristic search, and we compare it with our proposals. Our results suggest that the parameter tuning technique can give the best performance if a representative set of training instances is available. If not, then Bugsy is the algorithm of choice, as it performs well and does not require any off-line training. This work extends the tradition of research on metareasoning for search by illustrating the benefits of embedding lightweight reasoning about time into the search algorithm itself.

Anomalistic sequence detection

Given a set of training instances of sequences (time series), the problem of anomalistic sequence detection is to predict whether a newly observed time series novel or normal. Anomalistic sequence detection is very useful in many monitoring applications such as video surveillance and signal recognition. In this paper, we extend existing distance-based outlier detection algorithms to address the anomaly detection problem, and propose an instance-based anomaly detection algorithm. We study the effectiveness of these algorithms on some commonly used distance measures of time series. Experiments show that the instance-based algorithm under warping distances such as DTW and EDR achieves much better accuracy than the other combination. We observe that the actual distance calculation of warping distances contributes the main bulk of computational cost of anomaly detection. To improve the efficiency, we propose a local instance summarisation approach, called VarSpace, which reduces distance calculation by summarising similar training instances. Experiments show that the VarSpace approach can improve the efficiency of instance-based anomaly detection significantly.

An evolutionary approach for achieving scalability with general regression neural networks

In this paper, we present an approach to overcome the scalability issues associated with instance-based learners. Our system uses evolutionary computational techniques to determine the minimal set of training instances needed to achieve good classification accuracy with an instance-based learner. In this way, instance-based learners need not store all the training data available but instead store only those instances that are required for the desired accuracy. Additionally, we explore the utility of evolving the optimal feature set used by the learner for a given problem. In this way, we attempt to deal with the so-called "curse of dimensionality" associated with computational learning systems. To these ends, we introduce the Evolutionary General Regression Neural Network. This design uses an estimation of distribution algorithm to generate both the optimal training set as well as the optimal feature set for a general regression neural network. We compare its performance against a standard general regression neural network and an optimized support vector machine across four benchmark classification problems.

Learning Concepts by Arranging Appropriate Training Order

Machine learning has been proven useful for solving the bottlenecks in building expert systems. Noise in the training instances will, however, confuse a learning mechanism. Two main steps are adopted here to solve this problem. The first step is to appropriately arrange the training order of the instances. It is well known from Psychology that different orders of presentation of the same set of training instances to a human may cause different learning results. This idea is used here for machine learning and an order arrangement scheme is proposed. The second step is to modify a conventional noise-free learning algorithm, thus making it suitable for noisy environment. The generalized version space learning algorithm is then adopted to process the training instances for deriving good concepts. Finally, experiments on the Iris Flower problem show that the new scheme can produce a good training order, allowing the generalized version space algorithm to have a satisfactory learning result.