Large-scale Performance Evaluation Research Articles

Large-Scale Performance Evaluation of Various Woven Silt Fence Installations under Nebraska Highway Conditions

Sediment barriers are used on construction sites to protect downstream waterbodies from the impacts of sediment-laden stormwater runoff. Although ubiquitous on construction sites, many sediment barrier practices lack performance-based testing to determine effectiveness and treatment mechanisms, with previous evaluations being limited to conditions local to the Southeastern U.S., with conditions in other regions remaining untested. Testing was conducted to determine the effectiveness of woven silt fence barriers and provide structural improvements to common installation methods. Testing was conducted using a large-scale sediment barrier testing apparatus at the Auburn University—Stormwater Research Facility. The results from testing indicate that Nebraska DOT standard silt fence installations can be improved to reduce the risk of structural failures such as undermining, complete installation failure, slow dewatering, and overtopping. To improve structural performance, four modifications (a 15.2 cm [6 in.] offset trench, wooden posts, a dewatering board with an overflow weir, and a dewatering board with an overflow weir with adjusted post spacing) were tested. On average, 83% of introduced sediment was retained behind the tested barriers. The water quality results across the testing of standard and modified installations indicated that stormwater treatment was due to sedimentation within the impoundment formed by silt fence installations and not filtration through geotextile fabric.

LSim: Fine-Grained Simulation Framework for Large-Scale Performance Evaluation

As large-scale workloads with massive parallelism emerge, the demand for large-scale systems such as datacenters and supercomputers is rising sharply. To accurately design a large-scale system, architects heavily rely on performance modeling at design phases. However, modeling a large-scale workload without a large-scale system is a challenging problem. This paper presents LSim, a framework for large-scale performance evaluation. Based on the captured behavior within small-scale workload traces, LSim extrapolates the behavior of the workload on a large-scale system. To do so, we propose two techniques: (1) representative trace model and (2) function latency model to synthesize a trace and to predict the latency of functions in the synthesized trace, respectively.

Detection of Manipulated Face Videos over Social Networks: A Large-Scale Study

The detection of manipulated videos represents a highly relevant problem in multimedia forensics, which has been widely investigated in the last years. However, a common trait of published studies is the fact that the forensic analysis is typically applied on data prior to their potential dissemination over the web. This work addresses the challenging scenario where manipulated videos are first shared through social media platforms and then are subject to the forensic analysis. In this context, a large scale performance evaluation has been carried out involving general purpose deep networks and state-of-the-art manipulated data, and studying different effects. Results confirm that a performance drop is observed in every case when unseen shared data are tested by networks trained on non-shared data; however, fine-tuning operations can mitigate this problem. Also, we show that the output of differently trained networks can carry useful forensic information for the identification of the specific technique used for visual manipulation, both for shared and non-shared data.

Local feature extraction based facial emotion recognition: a survey

Notwithstanding the recent technological advancement, the identification of facial and emotional expressions is still one of the greatest challenges scientists have ever faced. Generally, the human face is identified as a composition made up of textures arranged in micro-patterns. Currently, there has been a tremendous increase in the use of local binary pattern based texture algorithms which have invariably been identified to being essential in the completion of a variety of tasks and in the extraction of essential attributes from an image. Over the years, lots of LBP variants have been literally reviewed. However, what is left is a thorough and comprehensive analysis of their independent performance. This research work aims at filling this gap by performing a large-scale performance evaluation of 46 recent state-of-the-art LBP variants for facial expression recognition. Extensive experimental results on the well-known challenging and benchmark KDEF, JAFFE, CK and MUG databases taken under different facial expression conditions, indicate that a number of evaluated state-of-the-art LBP-like methods achieve promising results, which are better or competitive than several recent state-of-the-art facial recognition systems. Recognition rates of 100%, 98.57%, 95.92% and 100% have been reached for CK, JAFFE, KDEF and MUG databases, respectively.

Research funding: past performance is a stronger predictor of future scientific output than reviewer scores

• Basic research grants significantly increase scientific performance. • Grant review scores have a low correlation with subsequent publication output in the course of the grant time. • The past scientometric performance of the principal investigator is the best predictor of future performance. • International reviewers are significantly less efficient than national reviewers. Scientific grants are awarded almost exclusively on the basis of an independent peer review of a proposal submitted by the principal investigator (PI). The writing and reviewing of these applications consumes a significant amount of researchers’ time. Here, we perform a large-scale performance evaluation of review-based grant allocation via analysis of the grant proposals submitted to the Hungarian Scientific Research Fund. In total, 42,905 scored review reports prepared for 13,303 proposals submitted between 2006 and 2015 were analyzed. The publication and citation characteristics of the PIs were obtained from the Hungarian Scientific Work Archive ( www.mtmt.hu ). Each publication was assigned to its respective SCImago Journal Rank category, and only publications in the first quarter (Q1) were considered. Citation, H-index and publication data were derived for each analyzed year for each researcher. Of all proposals, 3455 were funded (26%). PIs with a funded proposal had significantly more Q1 articles and first/last authored Q1 articles (1.91 vs. 1.30, p<1e-16 and 0.82 vs 0.53, p<1e-16, respectively). Of the successful applications, those involving international collaborations and extended budget had higher publication output. Applicant age, grant duration, and submission year were not correlated with publication performance. Reviewer scores displayed a minor association (corr.coeff = 0.08-011) with the number of Q1 publications. International reviewers were significantly less efficient than national reviewers (p = 0.021). A strong correlation with output was observed for the scientometric characteristics of the applying PI at the time of submission, including H-index (corr.coeff = 0.45-0.54), independent citation (corr.coeff. = 0.46-0.62), and yearly average Q1 articles (corr.coeff = 0.63-0.79, p<1e-16). Similar correlations were observed for nonfunded applicants. We performed a comprehensive evaluation of review-based resource allocation efficiency in basic research funding. Evidence suggests that the past scientometric performance of the principal investigator is the best predictor of future output.

RGB-T object tracking: Benchmark and baseline

RGB-Thermal (RGB-T) object tracking receives more and more attention due to the strongly complementary benefits of thermal information to visible data. However, RGB-T research is limited by lacking a comprehensive evaluation platform. In this paper, we propose a large-scale video benchmark dataset for RGB-T tracking. It has three major advantages over existing ones: 1) Its size is sufficiently large for large-scale performance evaluation (total number of frames: 234K, maximum number of frames per sequence: 8K). 2) The alignment between RGB-T sequence pairs is highly accurate, which does not need pre- or post-processing. 3) The occlusion levels are annotated for occlusion-sensitive performance analysis of different tracking algorithms. Moreover, we propose a novel graph-based approach to learn a robust object representation for RGB-T tracking. In particular, the tracked object is represented with a graph with image patches as nodes. Given initial weights of nodes, this graph including graph structure, node weights and edge weights is dynamically learned in a unified optimization framework. Extensive experiments on the large-scale dataset are executed to demonstrate the effectiveness of the proposed tracker against other state-of-the-art tracking methods. We also provide new insights and potential research directions to the field of RGB-T object tracking.

A Comprehensive Performance Evaluation of Image Quality Assessment Algorithms

Image quality assessment (IQA) algorithms aim to predict perceived image quality by human observers. Over the last two decades, a large amount of work has been carried out in the field. New algorithms are being developed at a rapid rate in different areas of IQA, but are often tested and compared with limited existing models using out-of-date test data. There is a significant gap when it comes to large-scale performance evaluation studies that include a wide variety of test data and competing algorithms. In this work we aim to fill this gap by carrying out the largest performance evaluation study so far. We test the performance of 43 full-reference (FR), seven fused FR (22 versions), and 14 no-reference (NR) methods on nine subject-rated IQA datasets, of which five contain singly distorted images and four contain multiply distorted content. We use a variety of performance evaluation and statistical significance testing criteria. Our findings not only point to the top performing FR and NR IQA methods, but also highlight the performance gap between them. In addition, we have also conducted a comparative study on FR fusion methods, and an important discovery is that rank aggregation based FR fusion is able to outperform not only other FR fusion approaches but also the top performing FR methods. It may be used to annotate IQA datasets as a possible alternative to subjective ratings, especially in situations where it is not possible to obtain human opinions, such as in the case of large-scale datasets composed of thousands or even millions of images.

Ranking Saliency.

Most existing bottom-up algorithms measure the foreground saliency of a pixel or region based on its contrast within a local context or the entire image, whereas a few methods focus on segmenting out background regions and thereby salient objects. Instead of only considering the contrast between salient objects and their surrounding regions, we consider both foreground and background cues in this work. We rank the similarity of image elements with foreground or background cues via graph-based manifold ranking. The saliency of image elements is defined based on their relevances to the given seeds or queries. We represent an image as a multi-scale graph with fine superpixels and coarse regions as nodes. These nodes are ranked based on the similarity to background and foreground queries using affinity matrices. Saliency detection is carried out in a cascade scheme to extract background regions and foreground salient objects efficiently. Experimental results demonstrate the proposed method performs well against the state-of-the-art methods in terms of accuracy and speed. We also propose a new benchmark dataset containing 5,168 images for large-scale performance evaluation of saliency detection methods.

Large-scale performance evaluation of Accu-Chek inform II point-of-care glucose meters

Background: The aim of this study was to report the experience of large-scale performance evaluation of 238 Accu-Chek Inform II point-of-care (POC) glucose meters in a single medical setting.Methods: The repeatability of 238 POC devices, the within-site imprecision of 12 devices, and the linearity of 49 devices were evaluated using glucose control solutions. The glucose results of 24 POC devices and central laboratory were compared using patient samples.Results: Mean concentration of control solutions was 2.39 mmol/L for Level 1 and 16.52 mmol/L for Level 2. The pooled repeatability coefficient of variation (CV) of the 238 devices was 2.0% for Level 1 and 1.6% for Level 2. The pooled within-site imprecision CV and reproducibility CV of the 12 devices were 2.7% and 2.7% for Level 1, and 1.9%, and 1.9% for Level 2, respectively. The test results of all 49 devices were linear within analytical measurement range from 1.55–31.02 mmol/L. The correlation coefficient for individual POC devices ranged from 0.9967–0.9985. The total correlation coefficient for the 24 devices was 0.998.Conclusions: The Accu-Chek Inform II POC blood glucose meters performed well in terms of precision, linearity, and correlation evaluations. Consensus guidelines for the large-scale performance evaluations of POC devices are required.

Large-scale experimental evaluation of GPU strategies for evolutionary machine learning

Graphics Processing Units (GPUs) are effective tools for improving the efficiency of many computationally demanding algorithms. GPUs have been particularly effective at speeding up the evaluation stage of evolutionary machine learning systems. The speedups obtained in these tasks, depend on many factors: dataset characteristics, the parallel strategy of the GPU code and the fit of the GPU code within the rest of the learning system. A solid understanding of all these factors is required to choose and adjust the most suitable GPU strategy in different scenarios. In this paper we present a large-scale performance evaluation of two GPU strategies for speeding up the evaluation of evolutionary machine learning systems. We use highly-tuneable synthetic problems to exhaustively explore the space of problem characteristics and determine the type of problems where each strategy performs best. The lessons learnt from this extensive evaluation are further confirmed by running experiments on a broad range of real-world datasets. Through this thorough evaluation we obtain a solid understanding of the capabilities and limitations of the evaluated GPU strategies for boosting the efficiency of evolutionary machine learning systems.

Object Tracking Benchmark.

Object tracking has been one of the most important and active research areas in the field of computer vision. A large number of tracking algorithms have been proposed in recent years with demonstrated success. However, the set of sequences used for evaluation is often not sufficient or is sometimes biased for certain types of algorithms. Many datasets do not have common ground-truth object positions or extents, and this makes comparisons among the reported quantitative results difficult. In addition, the initial conditions or parameters of the evaluated tracking algorithms are not the same, and thus, the quantitative results reported in literature are incomparable or sometimes contradictory. To address these issues, we carry out an extensive evaluation of the state-of-the-art online object-tracking algorithms with various evaluation criteria to understand how these methods perform within the same framework. In this work, we first construct a large dataset with ground-truth object positions and extents for tracking and introduce the sequence attributes for the performance analysis. Second, we integrate most of the publicly available trackers into one code library with uniform input and output formats to facilitate large-scale performance evaluation. Third, we extensively evaluate the performance of 31 algorithms on 100 sequences with different initialization settings. By analyzing the quantitative results, we identify effective approaches for robust tracking and provide potential future research directions in this field.

A model-driven emulation approach to large-scale TCP performance evaluation

Small-scale experiments are insufficient for a comprehensive study of congestion control protocols. Similarly, results obtained from pure simulation platforms without exercising real protocols and applications lack realism. Motivated by these reasons, we developed scalable virtualised evaluation environment for TCP (SVEET), a TCP performance evaluation testbed where real implementations of TCP variants can be accurately evaluated under diverse network configurations and workloads from real applications in large-scale network settings. It is our purpose to provide the research community a standard environment through which quantitative assessment regarding TCP behaviour can be drawn from large-scale experiments. In order to accomplish our goal, we adopted a novel model-driven emulation approach combining real-time simulation, machine and time virtualisation techniques. We validate the testbed via extensive experiments to assess its potentials and limitations. Additionally, we performed case studies involving real web, streaming and peer-to-peer applications. Our results indicate that SVEET can accurately model the behaviour of the TCP variants and support large-scale network scenarios.