Benchmark Collection Research Articles

A new validity index of feature subset for evaluating the dimensionality reduction algorithms

A critical aspect of dimensionality reduction is to assess the quality of selected (or produced) feature subsets properly. Feature subset assessment in machine learning refers to split a given feature subset into a training set, which is used to estimate the parameters of a classification model, and a test set used to estimate the predictive performance of the model. Then, averaging the results of multiple splitting (i.e., Cross-Validation, CV) is commonly used to decrease the variance of the estimator. But in practice, CV scheme is very computationally expensive. In this paper, we propose a new statistics index method called LW-index for evaluation of feature subset and dimensionality reduction algorithms in general. The proposed method is a type of “classical statistics” approach that uses the feature subset to compute an empirical estimate of the quality of feature subset. A large number of performance comparisons with the machine learning approach conducted on fourteen benchmark collections show that the proposed LW-index is highly correlated with the external indices (i.e., MacroF1, MicroF1) of SVM and Centroid-Based Classifier (CBC) trained by five-fold CV scheme. Furthermore, the experimental results indicate that LW-index has the same performance as the traditional CV scheme for evaluating the dimensionality reduction algorithms and it is more efficient than the traditional methodology. Therefore, one contribution of this paper is to present an alternative methodology, based on an internal index typically used in the unsupervised learning context, that is computationally cheaper than the traditional CV methodology. Another contribution is to propose a new internal index that behaves better than other similar indices widely used in clustering and shows high correlation with the results obtained by the traditional methodology.

Constrained Laplacian biogeography-based optimization algorithm

Biogeography-based optimization (BBO) is a relatively new nature inspired optimization technique proposed by Dan Simon for unconstrained optimization, which was later generalized and improved by Happing Ma and Dan Simon for constrained optimization, called blended biogeography-based optimization. In an earlier paper, the authors have proposed a Laplacian biogeography-based optimization algorithm (LX-BBO) for unconstrained optimization. The purpose of the present paper is to generalize the LX-BBO from the unconstrained case to the constrained case. This is done by using the Deb’s constrained handling method. In order to evaluate the performance of the proposed constrained LX-BBO for constrained optimization problems, five different constrained optimization problems and popular CEC 2006 benchmark collection is used. Based on the analysis of results it is shown that the proposed Constrained LX-BBO outperforms Blended BBO for constrained optimization.

Using Copac data to benchmark collections

Purpose – The Copac Collections Management (CCM) project has been underway since 2011 with the goal of exploring how Copac can make a real difference for collection managers. By making Copac data work harder and building prototype collection analysis features on its extensive database, the collections management tools can provide valuable information from the catalogues of the UK’s major research libraries. The paper aims to discuss these issues. Design/methodology/approach – Developed to support staff in collection management activities the CCM tools offer a web-based collection management support service. It is a “grass roots” project designed to meet the needs of library practitioners and its development has been a community-based collaboration with extensive design input, consultation, feedback and support from the wider library community. Findings – This paper presents a case study showing how the tools are now embedded into “business as usual” at the University of York library. The case study reflects on how York is using the tools to assist: identify collection strengths and contribute to the identification of “unique and distinctive collections”; identify collection gaps and produce lists of potential titles for purchase; inform stock editing policies; review and prioritize work on “hidden collections”. The paper outlines the benefits of using the tools and considers lessons learnt. Originality/value – The paper offers an overall view of the tools and the potential uses by the UK higher education community. The case study will be of interest to libraries in exploring new ways to approach collection management issues, with a view to making efficient use of available resources.

A collection of challenging motion segmentation benchmark datasets

An in-depth analysis of computer vision methodologies is greatly dependent on the benchmarks they are tested upon. Any dataset is as good as the diversity of the true nature of the problem enclosed in it. Motion segmentation is a preprocessing step in computer vision whose publicly available datasets have certain limitations. Some databases are not up-to-date with modern requirements of frame length and number of motions, and others do not have ample ground truth in them. In this paper, we present a collection of diverse multifaceted motion segmentation benchmarks containing trajectory- and region-based ground truth. These datasets enclose real-life long and short sequences, with increased number of motions and frames per sequence, and also real distortions with missing data. The ground truth is provided on all the frames of all the sequences. A comprehensive benchmark evaluation of the state-of-the-art motion segmentation algorithms is provided to establish the difficulty of the problem and to also contribute a starting point. All the resources of the datasets have been made publicly available at http://dixie.udg.edu/udgms/.

Hybrid Sampling-Based Clustering Ensemble With Global and Local Constitutions.

Among a number of ensemble learning techniques, boosting and bagging are the most popular sampling-based ensemble approaches for classification problems. Boosting is considered stronger than bagging on noise-free data set with complex class structures, whereas bagging is more robust than boosting in cases where noise data are present. In this paper, we extend both ensemble approaches to clustering tasks, and propose a novel hybrid sampling-based clustering ensemble by combining the strengths of boosting and bagging. In our approach, the input partitions are iteratively generated via a hybrid process inspired by both boosting and bagging. Then, a novel consensus function is proposed to encode the local and global cluster structure of input partitions into a single representation, and applies a single clustering algorithm to such representation to obtain the consolidated consensus partition. Our approach has been evaluated on 2-D-synthetic data, collection of benchmarks, and real-world facial recognition data sets, which show that the proposed technique outperforms the existing benchmarks for a variety of clustering tasks.

Overlaid Arrow Detection for Labeling Regions of Interest in Biomedical Images

A new template-free geometric signature-based technique detects arrow annotations on biomedical images. Arrow detection is a key first step to region-of-interest (ROI) labeling and image content analysis. Images are first binarized using a fuzzy binarization tool, and candidates are selected based on the connected component principle. For each candidate, the proposed method checks geometric properties of novel arrow signatures from key points associated with its boundary. These signatures are then compared with the theoretical (or idealized) arrow signatures, and a high similarity score indicates the presence of an arrow. The data was evaluated against the imageCLEFmed benchmark collection and achieved precision and recall of 93.14 percent and 86.12 percent, respectively, which outperforms previously reported arrow-detection methods.

A survey of direct methods for sparse linear systems

Wilkinson defined a sparse matrix as one with enough zeros that it pays to take advantage of them.1This informal yet practical definition captures the essence of the goal of direct methods for solving sparse matrix problems. They exploit the sparsity of a matrix to solve problems economically: much faster and using far less memory than if all the entries of a matrix were stored and took part in explicit computations. These methods form the backbone of a wide range of problems in computational science. A glimpse of the breadth of applications relying on sparse solvers can be seen in the origins of matrices in published matrix benchmark collections (Duff and Reid 1979a, Duff, Grimes and Lewis 1989a, Davis and Hu 2011). The goal of this survey article is to impart a working knowledge of the underlying theory and practice of sparse direct methods for solving linear systems and least-squares problems, and to provide an overview of the algorithms, data structures, and software available to solve these problems, so that the reader can both understand the methods and know how best to use them.

The first reactive synthesis competition (SYNTCOMP 2014)

We introduce the reactive synthesis competition (SYNTCOMP), a long-term effort intended to stimulate and guide advances in the design and application of synthesis procedures for reactive systems. The first iteration of SYNTCOMP is based on the controller synthesis problem for finite-state systems and safety specifications. We provide an overview of this problem and existing approaches to solve it, and report on the design and results of the first SYNTCOMP. This includes the definition of the benchmark format, the collection of benchmarks, the rules of the competition, and the five synthesis tools that participated. We present and analyze the results of the competition and draw conclusions on the state of the art. Finally, we give an outlook on future directions of SYNTCOMP.

Edge map analysis in chest X-rays for automatic pulmonary abnormality screening.

Our particular motivator is the need for screening HIV+ populations in resource-constrained regions for the evidence of tuberculosis, using posteroanterior chest radiographs (CXRs). The proposed method is motivated by the observation that abnormal CXRs tend to exhibit corrupted and/or deformed thoracic edge maps. We study histograms of thoracic edges for all possible orientations of gradients in the range [Formula: see text] at different numbers of bins and different pyramid levels, using five different regions-of-interest selection. We have used two CXR benchmark collections made available by the U.S. National Library of Medicine and have achieved a maximum abnormality detection accuracy (ACC) of 86.36% and area under the ROC curve (AUC) of 0.93 at 1 s per image, on average. We have presented an automatic method for screening pulmonary abnormalities using thoracic edge map in CXR images. The proposed method outperforms previously reported state-of-the-art results.

Parallelism analysis: Precise WCET values for complex multi-core systems

In the verification of safety-critical real-time systems, the problem of determining the worst-case execution time (WCET) of a task is of utmost importance. Safe formal methods have been established for solving the single-task, single-core WCET problem. The de-facto standard approach uses abstract interpretation to derive basic block execution times and a combinatorial path analysis which derives the longest path through the program. WCET analyses for multi-core computers have extended this methodology by assuming that shared resources are partitioned in either time or space and that therefore each core can still be analyzed separately. For real-world multi-cores this assumption is often not true, making the classic WCET analysis approach either inapplicable or very imprecise. To overcome this, we present a technique to explore the interleavings of a parallel task system as well as an exclusion criterion to prove that certain interleavings can never occur. We show how this technique can be integrated into existing WCET analysis approaches and finally show that the average WCET of a collection of real-time benchmarks could be reduced by a factor of up to 11.96 using this new analysis type. • First precise WCET computation for complex multi-core systems. • Proofs of correctness. • Extended evaluation including shared caches.

Performing Initiative Data Prefetching in Distributed File Systems for Cloud Computing

This paper presents an initiative data prefetching scheme on the storage servers in distributed file systems for cloud computing. In this prefetching technique, the client machines are not substantially involved in the process of data prefetching, but the storage servers can directly prefetch the data after analyzing the history of disk I/O access events, and then send the prefetched data to the relevant client machines proactively. To put this technique to work, the information about client nodes is piggybacked onto the real client I/O requests, and then forwarded to the relevant storage server. Next, two prediction algorithms have been proposed to forecast future block access operations for directing what data should be fetched on storage servers in advance. Finally, the prefetched data can be pushed to the relevant client machine from the storage server. Through a series of evaluation experiments with a collection of application benchmarks, we have demonstrated that our presented initiative prefetching technique can benefit distributed file systems for cloud environments to achieve better I/O performance. In particular, configuration-limited client machines in the cloud are not responsible for predicting I/O access operations, which can definitely contribute to preferable system performance on them.

Vectorization of apply to reduce interpretation overhead of R

R is a popular dynamic language designed for statistical computing. Despite R's huge user base, the inefficiency in R's language implementation becomes a major pain-point in everyday use as well as an obstacle to apply R to solve large scale analytics problems. The two most common approaches to improve the performance of dynamic languages are: implementing more efficient interpretation strategies and extending the interpreter with Just-In-Time (JIT) compiler. However, both approaches require significant changes to the interpreter, and complicate the adoption by development teams as a result. This paper presents a new approach to improve execution efficiency of R programs by vectorizing the widely used Apply class of operations. Apply accepts two parameters: a function and a collection of input data elements. The standard implementation of Apply iteratively invokes the input function with each element in the data collection. Our approach combines data transformation and function vectorization to convert the looping-over-data execution of the standard Apply into a single invocation of a vectorized function that contains a sequence of vector operations over the input data. This conversion can significantly speed-up the execution of Apply operations in R by reducing the number of interpretation steps. We implemented the vectorization transformation as an R package. To enable the optimization, all that is needed is to invoke the package, and the user can use a normal R interpreter without any changes. The evaluation shows that the proposed method delivers significant performance improvements for a collection of data analysis algorithm benchmarks. This is achieved without any native code generation and using only a single-thread of execution.

Performance of Laplacian Biogeography-Based Optimization Algorithm on CEC 2014 continuous optimization benchmarks and camera calibration problem

This paper provides three innovations. Firstly, a new Laplacian BBO is presented which introduces a Laplacian migration operator based on the Laplace Crossover of Real Coded Genetic Algorithms. Secondly, the performance of the Laplacian BBO and Blended BBO is exhibited on the latest benchmark collection of CEC 2014. (To the best of the knowledge of the authors, the complete CEC 2014 benchmarks have not been solved by Blended BBO). On the basis of the criteria laid down in CEC 2014 as well as popular evaluation criteria called Performance Index, It is shown that Laplacian BBO outperforms Blended BBO in terms of error value defined in CEC 2014 benchmark collection. T-Test has also been employed to strengthen the fact that Laplacian BBO performs better than Blended BBO. The third innovation of the paper is the use of the proposed Laplacian BBO and Blended BBO to solve a real life problem from the field of Computer Vision. It is concluded that proposed Laplacian BBO is an efficient and reliable algorithm for solving not only the continuous functions but also real life problems like camera calibration.

Chaim Potok: Confronting Modernity through the Lens of Tradition

Chaim Potok: Confronting Modernity through the Lens of Tradition

A robust semi-supervised learning approach via mixture of label information

Due to the fact that limited amounts of labeled data are normally available in real-world, semi-supervised learning has become a popular option, where we expect to use unlabeled data information to improve the learning performance. However, how to use such unlabeled information to make the predicted labels more reliable remains to be a key for any successful learning. In this paper, we propose a semi-supervised learning framework via combination of semi-supervised clustering and semi-supervised classification. In our approach, the predicted labels are selected by both the constrained k-means and safe semi-supervised SVM (S4VMs) to improve the reliability of the predicted labels. Extensive evaluations on collection of benchmarks and real-world action recognition datasets show that the proposed technique outperforms the others.

A cross-benchmark comparison of 87 learning to rank methods

Learning to rank is an increasingly important scientific field that comprises the use of machine learning for the ranking task. New learning to rank methods are generally evaluated on benchmark test collections. However, comparison of learning to rank methods based on evaluation results is hindered by the absence of a standard set of evaluation benchmark collections. In this paper we propose a way to compare learning to rank methods based on a sparse set of evaluation results on a set of benchmark datasets. Our comparison methodology consists of two components: (1) Normalized Winning Number, which gives insight in the ranking accuracy of the learning to rank method, and (2) Ideal Winning Number, which gives insight in the degree of certainty concerning its ranking accuracy. Evaluation results of 87 learning to rank methods on 20 well-known benchmark datasets are collected through a structured literature search. ListNet, SmoothRank, FenchelRank, FSMRank, LRUF and LARF are Pareto optimal learning to rank methods in the Normalized Winning Number and Ideal Winning Number dimensions, listed in increasing order of Normalized Winning Number and decreasing order of Ideal Winning Number.

Synthesis of circular compositional program proofs via abduction

This paper presents a new technique for synthesizing circular compositional proofs of program correctness. Our technique uses abductive inference to decompose the proof into small lemmas (i.e., compositionality) and proves that each lemma is not the first one to fail (i.e., circularity). Our approach represents lemmas as small program fragments annotated with pre and post-conditions and uses different tools to discharge each different lemma. This approach allows us to combine the strengths of different verifiers and addresses scalability concerns, as each lemma concerns the correctness of small syntactic fragments of the program. We have implemented our proposed technique for generating circular compositional proofs and used four different program analysis tools to discharge the proof subgoals. We evaluate our approach on a collection of synthetic and real-world benchmarks and show that our technique can successfully verify applications that cannot be verified by any individual technique.

An exact algorithm for sparse matrix bipartitioning

The sparse matrix partitioning problem arises when minimizing communication in parallel sparse matrix–vector multiplications. Since the problem is NP-hard, heuristics are usually employed to find solutions. Here, we present a purely combinatorial branch-and-bound method for computing optimal bipartitionings of sparse matrices, in the sense that they have the lowest communication volume out of all possible bipartitionings obeying a certain load balance constraint. The method is based on a way of partitioning similar to the recently proposed medium-grain heuristic, which reduces the number of solutions to be considered in the branch-and-bound method.We applied the proposed optimal bipartitioner to find the optimal communication volume of all matrices of the University of Florida sparse matrix collection with 1000 nonzeros or less. For 85% of the matrices, an optimal bipartitioning was found within a single day of computation and for 58% even within a second. We also present optimal results for selected larger matrices, up to 129,042 nonzeros. The optimal bipartitionings and corresponding communication volumes are made publicly available in a benchmark collection.

Quality improvement and patient safety organizations in anesthesiology.

The Anesthesiology Quality Institute contributes both to local quality improvement in the practice of anesthesiology through data collection and establishment of benchmarks and to patient safety in partnership with the Anesthesia Patient Safety Foundation.

Next step toward the automation of screening for cervical cancer.

The simulations have always been of a great importance as they substitute the real processes when those are too expensive to be performed or impossible to be observed. The latter case is typical for optical microscopy. Here, we observe fixed or living cells under assumption, that the optical system and the attached electronic acquisition device do not affect the quality of the original specimen too much. Even though we can measure the most of optical aberrations and estimate the dominant sources of noise that together cause the final observed image to be blurred and noisy, we are still not able to reveal the original unaffected image how it would appear without any damage. Although several deconvolution methods are capable of inverting this degradation process, they can improve the quality of the data only to some extent. As there exists no exact knowledge, how the microscopic specimens look like, it is very difficult to evaluate the quality of new emerging segmentation and tracking algorithms that are of a great importance in medicine and biology. The same issue arises when one wants to tune-up their parameters. In the past, the only available quality measurement of the algorithms was an expert’s knowledge. The expert either classified the results of selected algorithm or provided an annotation of some real image dataset that was further used for evaluation purposes. Both ways, however, suffer from two main issues. First, the expert’s evaluation is nondeterministic. Second, for higher dimensional data (sequences of 2D or 3D images) the handmade annotation is impractical or even impossible. For this reason, the synthetic data, naturally accompanied by their ground truth, have started to appear. In the very beginning, only the basic geometric shapes like spheres or disc without any texture representing the internal structure of the observed cells were used. Since the late 90s, computer generated images have started to be more complex as the computer capabilities rose and allowed for calculations that required higher performance and extensive memory and disk usage. Namely, in the last 10 years, several simulation frameworks able to gener- ate for example cells with detailed description of subcellular components, large cell populations, and time-lapse image sequences emerged.