Instance Space Research Articles

Instance Space Analysis of Testing of Autonomous Vehicles in Critical Scenarios

Before being deployed on roads, Autonomous Vehicles (AVs) must undergo comprehensive testing. Safety-critical situations, however, are infrequent in usual driving conditions, so simulated scenarios are used to create them. A test scenario comprises static and dynamic features related to the AV and the test environment; the representation of these features is complex and makes testing a heavy process. A test scenario is effective if it identifies incorrect behaviors of the AV. In this article, we present a technique for identifying key features of test scenarios associated with their effectiveness using Instance Space Analysis (ISA). ISA generates a ( \(2D\) ) representation of test scenarios and their features. This visualization helps to identify combinations of features that make a test scenario effective. We present a graphical representation of each feature that helps identify how well each testing technique explores the search space. While identifying key features is a primary goal, this study specifically seeks to determine the critical features that differentiate the performance of algorithms. Finally, we present metrics to assess the robustness of testing algorithms and the scenarios generated. Collecting essential features in combination with their values associated with effectiveness can be used for selection and prioritization of effective test cases.

Characterising harmful data sources when constructing multi-fidelity surrogate models

Surrogate modelling techniques have seen growing attention in recent years when applied to both modelling and optimisation of industrial design problems. These techniques are highly relevant when assessing the performance of a particular design carries a high cost, as the overall cost can be mitigated via the construction of a model to be queried in lieu of the available high-cost source. The construction of these models can sometimes employ other sources of information which are both cheaper and less accurate. The existence of these sources however poses the question of which sources should be used when constructing a model. Recent studies have attempted to characterise harmful data sources to guide practitioners in choosing when to ignore a certain source. These studies have done so in a synthetic setting, characterising sources using a large amount of data that is not available in practice. Some of these studies have also been shown to potentially suffer from bias in the benchmarks used in the analysis. In this study, we approach the characterisation of harmful low-fidelity sources as an algorithm selection problem. We employ recently developed benchmark filtering techniques to conduct a bias-free assessment, providing objectively varied benchmark suites of different sizes for future research. Analysing one of these benchmark suites with the technique known as Instance Space Analysis, we provide an intuitive visualisation of when a low-fidelity source should be used. By performing this analysis using only the limited data available to train a surrogate model, we are able to provide guidelines that can be directly used in an applied industrial setting.

Feature extractor optimization for discriminative representations in Generalized Category Discovery

Generalized Category Discovery (GCD) task involves transferring knowledge from labeled known categories to recognize both known and novel categories within an unlabeled dataset. A significant challenge arises from the lack of prior information for novel categories. To address this, we develop a feature extractor that can learn discriminative features for both known and novel categories. Our approach leverages the observation that similar samples often belong to the same class. We construct a similarity matrix and employ similarity contrastive loss to increase the similarity between similar samples in the feature space. Additionally, we incorporate cluster labels to further refine the feature extractor, utilizing K-means clustering to assign these labels to unlabeled data, providing valuable supervision. Our feature extractor is optimized through the utilization of instance-level contrastive learning and class-level contrastive learning constraints. These constraints promote similarity maximization in both the instance space and the label space for instances sharing the same pseudo-labels. These three components complement each other, facilitating the learning of discriminative representations for both known and novel categories. Through comprehensive evaluations of generic image recognition datasets and challenging fine-grained datasets, we demonstrate that our proposed method achieves state-of-the-art performance.

Synthesising Diverse and Discriminatory Sets of Instances using Novelty Search in Combinatorial Domains.

Gathering sufficient instance data to either train algorithm-selection models or understand algorithm footprints within an instance space can be challenging. We propose an approach to generating synthetic instances that are tailored to perform well with respect to a target algorithm belonging to a predefined portfolio but are also diverse with respect to their features. Our approach uses a novelty search algorithm with a linearly weighted fitness function that balances novelty and performance to generate a large set of diverse and discriminatory instances in a single run of the algorithm. We consider two definitions of novelty: (1) with respect to discriminatory performance within a portfolio of solvers; (2) with respect to the features of the evolved instances. We evaluate the proposed method with respect to its ability to generate diverse and discriminatory instances in two domains (knapsack and bin-packing), comparing to another well-known quality diversity method, Multi-dimensional Archive of Phenotypic Elites (MAP-Elites) and an evolutionary algorithm that only evolves for discriminatory behaviour. The results demonstrate that the novelty search method outperforms its competitors in terms of coverage of the space and its ability to generate instances that are diverse regarding the relative size of the "performance gap" between the target solver and the remaining solvers in the portfolio. Moreover, for the Knapsack domain, we also show that we are able to generate novel instances in regions of an instance space not covered by existing benchmarks using a portfolio of state-of-the-art solvers. Finally, we demonstrate that the method is robust to different portfolios of solvers (stochastic approaches, deterministic heuristics and state-of-the-art methods), thereby providing further evidence of its generality.

Label distribution feature selection based on neighborhood rough set

SummaryIn label distribution learning (LDL), an instance is involved with many labels in different importance degrees, and the feature space of instances is accompanied with thousands of redundant and/or irrelevant features. Therefore, the main characteristic of feature selection in LDL is to evaluate the ability of each feature. Motivated by neighborhood rough set (NRS), which can be used to measure the dependency degree of feature via constructing neighborhood relations on feature space and label space, respectively, this article proposes a novel label distribution feature selection method. In this article, the neighborhood class of instance in label distribution space is defined, which is beneficial to recognize the logical class of target instance. Then, a new NRS model for LDL is proposed. Specially, the dependency degree of feature combining label weight is defined. Finally, a label distribution feature selection based on NRS is presented. Extensive experiments on 12 data sets show the effectiveness of the proposed algorithm.

Per-Instance Algorithm Configuration in Homogeneous Instance Spaces: A Use Case in Reconfigurable Assembly Systems

The physical capabilities of a reconfigurable assembly system (RAS) increase the agility and responsiveness of the system in highly volatile market conditions. However, achieving optimal RAS utilization entails solving complex optimization problems effectively and efficiently. These optimizations often define homogenous sets of problem instances. While algorithm configuration in such homogeneous contexts traditionally adopts a “one-size-fits-all” approach, recent studies have shown the potential of per-instance algorithm configuration (PIAC) methods in these settings. In this work, we evaluate and compare the performance of different PIAC methods in this context, namely Hydra—a state-of-the-art PIAC method—and a simpler case-based reasoning (CBR) approach. We evaluate the impact of the tuning time budget and/or the number of unique problem instances used for training on each of the method’s performance and robustness. Our experiments show that whilst Hydra fails to improve upon the default algorithm configuration, the CBR method can lead to 16% performance increase using as few as 100 training instances. Following these findings, we evaluate Hydra’s methodology when applied to homogenous instance spaces. This analysis shows the limitations of Hydra’s inference mechanisms in these settings and showcases the advantages of distance-based approaches used in CBR.

The maximum length car sequencing problem

This paper introduces the maximum length car sequencing problem to support the assembly operations of a multinational automotive company. We propose an integer linear programming (ILP) formulation to schedule the maximum number of cars without violating the so-called option constraints. In addition, we present valid combinatorial lower and upper bounds, which can be calculated in less than 0.01 s, as well as binary and iterative search algorithms to solve the problem when good primal bounds are not readily available. To quickly obtain high-quality solutions, we devise an effective iterated local search algorithm, and we use the heuristic solutions as warm start to further enhance the performance of the exact methods. Computational results demonstrate that relatively low gaps were achieved for benchmark instances within a time limit of ten minutes. We also conducted an instance space analysis to identify the features that make the problem more difficult to solve. Moreover, the instances reflecting the company’s needs could be solved to optimality in less than a second. Finally, simulations with real-world demands, divided into shifts, were conducted over a period of four months. In this case, we use the proposed ILP model in all shifts except the last one of each month, for which we employ an alternative ILP model to sequence the unscheduled cars, adjusting the pace of the assembly line in an optimal fashion. The results pointed out that the latter was necessary in only one of the months.

Reducing the feasible solution space of resource-constrained project instances

This paper present an instance transformation procedure to modify known instances of the resource-constrained project scheduling problem to make them easier to solve by heuristic and/or exact solution algorithms. The procedure makes use of a set of transformation rules that aim at reducing the feasible search space without excluding at least one possible optimal solution. The procedure will be applied to a set of 11,183 instances and it will be shown by a set of experiments that these transformations lead to 110 improved lower bounds, 16 new and better schedules (found by three meta-heuristic procedures and a set of branch-and-bound procedures) and even 64 new optimal solutions which were never not found before.

Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features

Ensuring the safety of autonomous vehicles (AVs) is of utmost importance, and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road’s structural features (lanes, turns, road speed, etc.). In this paper, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2 D . This visualisation helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritisation.

Instance space analysis for 2D bin packing mathematical models

In this paper, we apply Instance Space Analysis (ISA) to study the two-dimensional bin-packing problem. We consider classical and newly-generated instances to test the performance of four mixed-integer programming (MIP) models from the literature. This is the first time ISA is used to compare MIP models. We set as a performance metric the time taken by the black-box MIP solver CPLEX to obtain a proven optimal solution when running each model. Our results provide a new perspective on the different models’ performance according to each instance’s features.

On the impact of initialisation strategies on Maximum Flow algorithm performance

Due to its theoretical and practical importance in network theory, designing effective algorithms for the Maximum Flow Problem (MFP) remains a focus of research efforts. Although worst-case performance analysis is the main tool for examining performance, empirical analysis across a wide variety of benchmark cases can identify scenarios where practical performance may contradict theoretical worse-case. In our previous work, we used Instance Space Analysis (ISA) to identify the practical strengths and weaknesses of four state-of-the-art MFP algorithms, and identified that the arc/path finding strategies employed by the algorithms explain critical differences in the algorithms’ behaviours. In this paper, we leverage these insights to propose two new initialisation strategies, which are an essential part of the arc/path finding strategy. To employ these new strategies on our previously studied four algorithms, we propose modifications that result in 15 new algorithmic variants. Using a comprehensive experimental setup and ISA, we examine the impact of these proposed initialisation strategies on performance, and discuss the conditions under which each initialisation strategy is expected to improve performance. One of the novel initialisation strategies is shown to improve the performance of MFP algorithms in many instances, making it promising for even further improvements of the algorithms.

Verifying new instances of the multidemand multidimensional knapsack problem with instance space analysis

The optimization literature contains numerous studies defining a problem and showing a state-of-the-art implementation of a solution method. These studies may involve an empirical evaluation of the solution methods but assume the current collection of test instances is sufficient. This paper is concerned with instance generation methods for the multidemand multidimensional knapsack problem. We use instance space analysis to characterize the landscape of existing instances and validate the novelty of new instances. We discuss gaps present within the current instance space landscape and fill them with three new sets of instances. We find feasibility issues within instance generation methods and address them through a primal problem instance generator (PPIG). The instance generator is capable of producing feasible and diverse instances by directly controlling the problem features. PPIG contributes to the previous collections of instances and is validated through instance space analysis.

Multi-constructor CMSA for the maximum disjoint dominating sets problem

We propose the Multi-Constructor CMSA, a Construct, Merge, Solve and Adapt (CMSA) algorithm that employs multiple heuristic procedures, respectively solution constructors, for the Maximum Disjoint Dominating Sets Problem (MDDSP). At every iteration of the search procedure, the solution components built by the constructors are merged into a sub-instance, which is subsequently solved by an exact solver and then adapted to keep only beneficial solution components. In our CMSA the solution constructors are chosen at random according to their relative probabilities, which are adapted during the search, through a mechanism based on reinforcement learning. We test two variants of the new Multi-Constructor CMSA that employ, respectively, two and six solution constructors, on a new set of 3600 problem instances, encompassing random graphs, Watts–Strogatz networks and Barabási-Albert networks, generated through a Hammersley sampling procedure on the instance space. We compare our algorithm against six heuristics from the literature, as well as with the standard version of CMSA. Furthermore, we employ an integer linear programming (ILP) model that is able to achieve a good performance for small, sparse graphs. Overall, the experimental results show that all versions of CMSA outperform by a large margin the previous state of the art and that, among the variants of CMSA, the novel version that combines two constructors provides slightly better results than the other ones, more prominently on larger graphs.

An Instance Space Analysis of Constrained Multiobjective Optimization Problems

Constrained multi-objective optimization problems (CMOPs) are generally more challenging than unconstrained problems. This in part can be attributed to the infeasible region generated by the constraint functions, the interaction between constraints and objectives, or both. In this paper, we explore the relationship between the performance of constrained multi-objective evolutionary algorithms (CMOEAs) and the instance characteristics of CMOP using Instance Space Analysis (ISA). To do this, we extend recent work on Landscape Analysis features for characterising CMOPs. Specifically, we introduce new features to describe the multi-objective-violation landscape, formed by the interaction between constraint violation and multi-objective fitness. Detailed evaluation of the algorithm footprints, spanning eight CMOP benchmark suites and fifteen CMOEAs, demonstrates that ISA effectively captures the strength and weakness of the CMOEAs. We conclude that two characteristics, the isolation of non-dominate set and the correlation between constraints and objectives evolvability, have the greatest impact on algorithm performance. However, the current benchmarks problems lack of diversity to represent the real-world problems and to fully reveal the efficacy of CMOEAs evaluated.

New benchmark instances for the inventory routing problem

The existing sets of benchmark instances for the inventory routing problem (IRP) have been beneficial for investigating and illustrating the properties of the problem. However, they possess certain features and design choices that are not necessarily representative of all real-world IRPs. Therefore, we propose a new collection of 270 real-world-like instances ranging from 10 to 200 customers. These instances vary in terms of the number of vehicles and their capacity, the length of the planning horizon, the demand structure, and the geographical distribution of customers. Transportation and inventory holding costs resemble costs found in practice. We present an instance space analysis showing that the new instances nicely complement the original instances. To derive lower and upper bounds for each instance, we present computational experiments with two high-quality solution methods: a matheuristic and an exact branch-and-cut method. The results confirm that the new set of instances is hard to solve with the proposed methods, and they demonstrate the need for developing new solution methods.

Learning Instance-Level Label Correlation Distribution for Multilabel Classification With Fuzzy Rough Sets

In multi-label learning, research on label correlation provides an effective solution to compress the hypothesis space of classifiers. However, the current works focus on the label correlation adapted to overall data, while ignoring the locally targeted information presented by some instances. The lack exploration on the distribution of local label correlation in multi-label instance space undoubtedly limits the in-depth application of label correlation in multi-label learning. Based on fuzzy rough set theory, the instance-level label correlation distribution is first proposed in this paper and applied to design a novel multi-label learner. For each multi-label instance, the local importance of features to label is quantitatively analyzed, by considering the decisive influence of input information on decision-making. According to coincidence degree between local feature weight distribution for different labels, the instance-level label correlation is constructed. In order to reflect the internal relationship between label variables objectively, the instance-level label correlation distribution is integrated into the empirical label relevance. On the basis, the label relevance matrix is used to define the constraints of optimization function in a new form. The relative position of sub-separating hyperplanes in input space is quantitatively characterized to reduce the complexity of multi-label classifier and improve learning performance. The experiment results on eighteen multi-label datasets illustrate the effectiveness of our algorithm. The impact of core parameters on performance is also dissected.

On Error and Compression Rates for Prototype Rules

We study the close interplay between error and compression in the non-parametric multiclass classification setting in terms of prototype learning rules. We focus in particular on a recently proposed compression-based learning rule termed OptiNet. Beyond its computational merits, this rule has been recently shown to be universally consistent in any metric instance space that admits a universally consistent rule---the first learning algorithm known to enjoy this property. However, its error and compression rates have been left open. Here we derive such rates in the case where instances reside in Euclidean space under commonly posed smoothness and tail conditions on the data distribution. We first show that OptiNet achieves non-trivial compression rates while enjoying near minimax-optimal error rates. We then proceed to study a novel general compression scheme for further compressing prototype rules that locally adapts to the noise level without sacrificing accuracy. Applying it to OptiNet, we show that under a geometric margin condition further gain in the compression rate is achieved. Experimental results comparing the performance of the various methods are presented.

A lightweight semi-centralized strategy for the massive parallelization of branching algorithms

Several NP-hard problems are solved exactly using exponential-time branching strategies, whether it be branch-and-bound algorithms, or bounded search trees in fixed-parameter algorithms. The number of tractable instances that can be handled by sequential algorithms is usually small, whereas massive parallelization has been shown to significantly increase the space of instances that can be solved exactly. However, previous centralized approaches require too much communication to be efficient, whereas decentralized approaches are more efficient but have difficulty keeping track of the global state of the exploration.In this work, we propose to revisit the centralized paradigm while avoiding previous bottlenecks. In our strategy, the center has lightweight responsibilities, requires only a few bits for every communication, but is still able to keep track of the progress of every worker. In particular, the center never holds any task but is able to guarantee that a process with no work always receives the highest priority task globally.Our strategy was implemented in a generic C++ library called GemPBA, which allows a programmer to convert a sequential branching algorithm into a parallel version by changing only a few lines of code. An experimental case study on the vertex cover problem demonstrates that some of the toughest instances from the DIMACS challenge graphs that would take months to solve sequentially can be handled within two hours with our approach.

Features for the 0-1 knapsack problem based on inclusionwise maximal solutions

Decades of research on the 0-1 knapsack problem led to very efficient algorithms that are able to quickly solve large problem instances to optimality. This prompted researchers to also investigate the structure of problem instances that are hard for existing solvers. In the current paper we are interested in investigating which features make 0-1 knapsack problem instances hard to solve to optimality for the state-of-the-art 0-1 knapsack solver. We propose a set of 14 features based on previous work by the authors in which so-called inclusionwise maximal solutions (IMSs) play a central role. Calculating these features is computationally expensive and requires one to solve hard combinatorial problems. Based on new structural results about IMSs, we formulate polynomial and pseudopolynomial time algorithms for calculating these features. These algorithms were executed for two large datasets on a supercomputer in approximately 540 CPU-hours. We show that the proposed features contain important information related to the empirical hardness of a problem instance that was missing in earlier features from the literature by training machine learning models that can accurately predict the empirical hardness of a wide variety of 0-1 knapsack problem instances. Moreover, we show that these features can be cheaply approximated at the cost of less accurate hardness predictions. Using the instance space analysis methodology, we show that hard 0-1 knapsack problem instances are clustered together around a relatively dense region of the instance space and several features behave differently in the easy and hard parts of the instance space.

Instance Space Analysis of Search-Based Software Testing

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances - in the form of Instance Space Analysis (ISA) - enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.