Evaluation Campaign Research Articles

A study about the future evaluation of Question-Answering systems

Evaluation campaigns of Question Answering (QA) systems have contributed to the development of such technologies. These campaigns have promoted some changes oriented to overcome results. However, at this period we see how systems have reached an upper bound, as well as systems are still far away from answering complex questions. In this paper, we overview the main QA evaluations over free text, paying special attention to the changes encouraged at such campaigns. We observe that systems still return a high proportion of incorrect answers and that the changes are almost not included in traditional approaches. Moreover, we analyze QA collections in order to obtain better insights about the main challenges for current QA systems. We detect that QA systems find very difficult to deal with different rewordings in questions and documents, as well as to infer information that is not explicitly mentioned in texts. Based on those observations, we recommend a set of directions for future evaluations, suggesting the application of textual inference and knowledge bases as a way for improving results.

Evaluating Performance of Containerized IoT Services for Clustered Devices at the Network Edge

The constant and fast increase in the number of heterogeneous Internet of Things (IoT) devices that populate everyday life environments brings new challenges to the full exploitation of the computation, memory, sensing, and actuation resources associated to them. In this context, device virtualization solutions and platforms may definitely play a key role in enabling the desired tradeoff between flexibility and performance. This paper focuses on lightweight virtualization technologies for IoT devices, suitably thought to effectively deploy new integrated applications and to create a novel distributed and virtualized ecosystem. Two different frameworks for container-based IoT service provisioning are compared, the one based on a direct interaction between two cooperating devices and the other based on the presence of a manager supervising the operations between cooperating devices forming a cluster. In the latter case, accounting for the growing impetus to move intelligence toward the edge of the network, management features are implemented at the network access point to provide short latency responses. We also introduce the outcomes of a thorough performance evaluation campaign conducted via a real IoT testbed. The measurements, performed by accounting for the constraints of typical IoT nodes, shed light on the actual feasibility of container-based IoT frameworks.

A wikipedia-based semantic relatedness framework for effective dimensions classification in online reputation management

Social media repositories serve as a significant source of evidence when extracting information related to the reputation of a particular entity (e.g., a particular politician, singer or company). Reputation management experts manually mine the social media repositories (in particular Twitter) for monitoring the reputation of a particular entity. Recently, the online reputation management evaluation campaign known as RepLab at CLEF has turned attention to devising computational methods for facilitating reputation management experts. A quite significant research challenge related to the above issue is to classify the reputation dimension of tweets with respect to entity names. More specifically, finding various aspects of a brand’s reputation is an important task which can help companies in monitoring areas of their strengths and weaknesses in an effective manner. To address this issue in this paper we use dominant Wikipedia categories related to a reputation dimension; the dominant Wikipedia categories are then utilised within a semantic relatedness scoring framework to generate “associativities” with respect to the various reputation dimensions, and another version of “associativity” normalized by the “content entropy” of Wikipedia categories. The Wikipedia categories obtained through our applied methods are finally used in a random forest classifier for the task of reputation dimensions classification. The experimental evaluations show a significant improvement over the baseline accuracy.

Rehabilitation-Oriented Serious Game Development and Evaluation Guidelines for Musculoskeletal Disorders.

BackgroundThe progress in information and communication technology (ICT) led to the development of a new rehabilitation technique called “serious game for functional rehabilitation.” Previous works have shown that serious games can be used for general health and specific disease management. However, there is still lack of consensus on development and evaluation guidelines. It is important to note that the game performance depends on the designed scenario.ObjectiveThe objective of this work was to develop specific game scenarios and evaluate them with a panel of musculoskeletal patients to propose game development and evaluation guidelines.MethodsA two-stage workflow was proposed using determinant framework. The development guideline includes the selection of three-dimensional (3D) computer graphics technologies and tools, the modeling of physical aspects, the design of rehabilitation scenarios, and the implementation of the proposed scenarios. The evaluation guideline consists of the definition of evaluation metrics, the execution of the evaluation campaign, the analysis of user results and feedbacks, and the improvement of the designed game.ResultsThe case study for musculoskeletal disorders on the healthy control and patient groups showed the usefulness of these guidelines and associated games. All participants enjoyed the 2 developed games (football and object manipulation), and found them challenging and amusing. In particular, some healthy subjects increased their score when enhancing the level of difficulty. Furthermore, there were no risks and accidents associated with the execution of these games.ConclusionsIt is expected that with the proven effectiveness of the proposed guidelines and associated games, this new rehabilitation game may be translated into clinical routine practice for the benefit of patients with musculoskeletal disorders.

EVALITA Goes Social: Tasks, Data, and Community at the 2016 Edition

EVALITA, the evaluation campaign of Natural Language Processing and Speech Tools for the Italian language, was organised for the fifth time in 2016. Six tasks, covering both re-reruns as well as completely new tasks, and an IBM-sponsored challenge, attracted a total of 34 submissions. An innovative aspect at this edition was the focus on social media data, especially Twitter, and the use of shared data across tasks, yielding a test set with layers of annotation concerning PoS tags, sentiment information, named entities and linking, and factuality information. Differently from the previous edition(s), many systems relied on a neural architecture, and achieved best results when used. From the experience and success of this edition, also in terms of dissemination of information and data, and in terms of collaboration between organisers of different tasks, we collected some reflections and suggestions that prospective EVALITA chairs might be willing to take into account for future editions.

Distance measures in author profiling

Determining some demographics about the author of a document (e.g., gender, age) has attracted many studies during the last decade. To solve this author profiling task, various classification models have been proposed based on stylistic features (e.g., function word frequencies, n-gram of letters or words, POS distributions), as well as various vocabulary richness or overall stylistic measures. To determine the targeted category, different distance measures have been suggested without one approach clearly dominating all others. In this paper, 24 distance measures are studied, extracted from five general families of functions. Moreover, six theoretical properties are presented and we show that the Tanimoto or Matusita distance measures respect all proposed properties. To complement this analysis, 13 test collections extracted from the last CLEF evaluation campaigns are employed to evaluate empirically the effectiveness of these distance measures. This test set covers four languages (English, Spanish, Dutch, and Italian), four text genres (blogs, tweets, reviews, and social media) with respect to two genders and between four to five age groups. The empirical evaluations indicate that the Canberra or Clark distance measures tend to produce better effectiveness than the rest, at least in the context of an author profiling task. Moreover, our experiments indicate that having a training set closely related to the test set (e.g., the same collection) has a clear impact on the overall performance. The gender accuracy rate is decreased by 7% (19% for the age) when using the same text genre during the training compared to using the same collection (leaving-one-out methodology). Employing a different text genre in the training and in the test phases tends to hurt the overall performance, showing a decrease of the final accuracy rate of around 11% for the gender classification to 26% for the age.

Blind Speech Separation and Enhancement With GCC-NMF

We present a blind source separation algorithm named GCC-NMF that combines unsupervised dictionary learning via non-negative matrix factorization NMF with spatial localization via the generalized cross correlation GCC method. Dictionary learning is performed on the mixture signal, with separation subsequently achieved by grouping dictionary atoms, at each point in time, according to their spatial origins. The resulting source separation algorithm is simple yet flexible, requiring no prior knowledge or information. Separation quality is evaluated for three tasks using stereo recordings from the publicly available SiSEC signal separation evaluation campaign: 3 and 4 concurrent speakers in reverberant environments, speech mixed with real-world background noise, and noisy recordings of a moving speaker. Performance is quantified using perceptually motivated and SNR-based measures with the PEASS and BSS Eval toolkits, respectively. We evaluate the effects of model parameters on separation quality, and compare our approach with other unsupervised and semi-supervised speech separation and enhancement approaches. We show that GCC-NMF is a flexible source separation algorithm, outperforming task-specific approaches in each of the three settings, including both blind as well as several informed approaches that require prior knowledge or information.

An automated method for the evaluation of the pointing accuracy of Sun-tracking devices

Abstract. The accuracy of solar radiation measurements, for direct (DIR) and diffuse (DIF) radiation, depends significantly on the precision of the operational Sun-tracking device. Thus, rigid targets for instrument performance and operation have been specified for international monitoring networks, e.g., the Baseline Surface Radiation Network (BSRN) operating under the auspices of the World Climate Research Program (WCRP). Sun-tracking devices that fulfill these accuracy requirements are available from various instrument manufacturers; however, none of the commercially available systems comprise an automatic accuracy control system allowing platform operators to independently validate the pointing accuracy of Sun-tracking sensors during operation. Here we present KSO-STREAMS (KSO-SunTRackEr Accuracy Monitoring System), a fully automated, system-independent, and cost-effective system for evaluating the pointing accuracy of Sun-tracking devices. We detail the monitoring system setup, its design and specifications, and the results from its application to the Sun-tracking system operated at the Kanzelhöhe Observatory (KSO) Austrian radiation monitoring network (ARAD) site. The results from an evaluation campaign from March to June 2015 show that the tracking accuracy of the device operated at KSO lies within BSRN specifications (i.e., 0.1° tracking accuracy) for the vast majority of observations (99.8 %). The evaluation of manufacturer-specified active-tracking accuracies (0.02°), during periods with direct solar radiation exceeding 300 W m−2, shows that these are satisfied in 72.9 % of observations. Tracking accuracies are highest during clear-sky conditions and on days where prevailing clear-sky conditions are interrupted by frontal movement; in these cases, we obtain the complete fulfillment of BSRN requirements and 76.4 % of observations within manufacturer-specified active-tracking accuracies. Limitations to tracking surveillance arise during overcast conditions and periods of partial solar-limb coverage by clouds. On days with variable cloud cover, 78.1 % (99.9 %) of observations meet active-tracking (BSRN) accuracy requirements while for days with prevailing overcast conditions these numbers reduce to 64.3 % (99.5 %).

Developing a benchmark for emotional analysis of music.

Music emotion recognition (MER) field rapidly expanded in the last decade. Many new methods and new audio features are developed to improve the performance of MER algorithms. However, it is very difficult to compare the performance of the new methods because of the data representation diversity and scarcity of publicly available data. In this paper, we address these problems by creating a data set and a benchmark for MER. The data set that we release, a MediaEval Database for Emotional Analysis in Music (DEAM), is the largest available data set of dynamic annotations (valence and arousal annotations for 1,802 songs and song excerpts licensed under Creative Commons with 2Hz time resolution). Using DEAM, we organized the ‘Emotion in Music’ task at MediaEval Multimedia Evaluation Campaign from 2013 to 2015. The benchmark attracted, in total, 21 active teams to participate in the challenge. We analyze the results of the benchmark: the winning algorithms and feature-sets. We also describe the design of the benchmark, the evaluation procedures and the data cleaning and transformations that we suggest. The results from the benchmark suggest that the recurrent neural network based approaches combined with large feature-sets work best for dynamic MER.

The European Lead Factory: A Blueprint for Public-Private Partnerships in Early Drug Discovery.

The European Lead Factory (ELF) is a public–private partnership (PPP) that provides researchers in Europe with a unique platform for translation of innovative biology and chemistry into high-quality starting points for drug discovery. It combines an exceptional collection of small molecules, high-throughput screening (HTS) infrastructure, and hit follow-up capabilities to advance research projects from both private companies and publicly funded researchers. By active interactions with the wider European life science community, ELF connects and unites bright ideas, talent, and experience from several disciplines. As a result, ELF is a unique, collaborative lead generation engine that has so far resulted in >4,500 hit compounds with a defined biological activity from 83 successfully completed HTS and hit evaluation campaigns. The PPP has also produced more than 120,000 novel innovative library compounds that complement the 327,000 compounds contributed by the participating pharmaceutical companies. Intrinsic to its setup, ELF enables breakthroughs in areas with unmet medical and societal needs, where no individual entity would be able to create a comparable impact in such a short time.

Predicting the Best System Parameter Configuration: the (Per Parameter Learning) PPL method

Search engines aim at delivering the most relevant information whatever the query is. To proceed, search engines employ various modules (indexing, matching, ranking), each of these modules having different variants (e.g. different stemmers, different retrieval models or weighting functions). The international evaluation campaigns in information retrieval such as TREC revealed system variability which makes it impossible to find a single system that would be the best for any of the queries.While some approaches aim at optimizing the system parameters to improve the system effectiveness in average over a set of queries, in this paper we consider a different approach that aims at optimizing the system configuration on a per-query basis. Our method learns the configuration models in a training phase and then explores the system feature space and decides what should be the system configuration for any new query.The experimental results draw significant conclusions: (i) Predicting the best value for each system feature separately is more effective than predicting the best predefined system configuration; (ii) the method predicts successfully the optimal or most optimal system configurations for unseen queries; (iii) the mean average precision (MAP) of the system configurations predicted by our approach is much higher than the MAP of the best unique system.

Estimating the Structural Segmentation of Popular Music Pieces Under Regularity Constraints

Music structure estimation has recently emerged as a central topic within the field of music information retrieval. Indeed, as music is a highly structured information stream, knowledge of how a music piece is organized represents a key challenge to enhance the management and exploitation of large music collections. This paper focuses on the benefits that can be expected from a regularity constraint on the structural segmentation of popular music pieces. Specifically, here, we study how a constraint that favors structural segments of comparable size provides a better conditioning of the boundary estimation process. First, we propose a formulation of the structural segmentation task as an optimization process, which separates the contribution from the audio features and the one from the constraint. We illustrate how the corresponding cost function can be minimized using a Viterbi algorithm. We present briefly its implementation and results in three systems designed for and submitted to the MIREX 2010, 2011, and 2012 evaluation campaigns. Then, we explore the benefits of the regularity constraint as an efficient mean for combining the outputs of a selection of systems presented at MIREX between 2010 and 2015, yielding a level of performance competitive to that of the state-of-the-art on the “MIREX10” dataset (100 J-Pop songs from the RWC database).

Enhancing instance search with weak geometric correlation consistency

Finding object instances from within in large image collections is a challenging problem with many practical applications. Recent methods inspired by text retrieval have achieved good results, however a re-ranking stage based on spatial verification may still be required to boost performance. To improve the effectiveness of such instance retrieval systems while avoiding the computational complexity of a re-ranking stage, we explore the geometric correlations among local features, and we incorporate these correlations with each individual match to form a transformation consistency in rotation and scale space.This weak geometric correlation consistency can be used to effectively eliminate inconsistent feature matches in instance retrieval and can be applied to all candidate images at a low computational cost. Experimental results on three standard evaluation benchmarks show that the proposed approach results in a substantial performance improvement when compared with other state-of-the-art methods. In addition, the evaluation results from participating in the Instance Search Task in the TRECVid evaluation campaign also suggest that our proposed approach enhances retrieval performance for large scale video collections.

Feasibility study of a serious game based on Kinect system for functional rehabilitation of the lower limbs

Summary Introduction Conventional functional rehabilitation costs time, money and effort for the patients and for the medical staff. Serious games have been used as a new approach to improve the performance as well as to possibly reduce medical cost in the future for cognitive rehabilitation and body balance control. The objective of this present work was to perform a feasibility study on the use of a new real-time serious game system for improving the musculoskeletal rehabilitation of the lower limbs. Materials and methods A basic functional rehabilitation exercise database was established with different levels of difficulties. A 3D virtual avatar was created and scaled to represent each subject-specific body. A portable and affordable Kinect sensor was used to capture real-time kinematics during each exercise. A specific data coupling process was developed. An evaluation campaign was established to assess the developed system. Results The squats exercise was the hardest challenge. Moreover, the performance of each functional rehabilitation exercise depended on the physiological profile of each participant. Our game system was clear and attractive for all functional rehabilitation exercises. All testing subjects felt motivated and secure when playing the rehabilitation game. Discussion The comparison with other systems showed that our system was the first one focusing on the functional rehabilitation exercises of the lower limbs. Conclusions Our system showed useful functionalities for a large range of applications (rehabilitation at home, sports training). Looking forward, new in-situation exercises will be investigated for specific musculoskeletal disorders.

GridBox pilot project

GridBox is an open platform for monitoring and active control of distribution grids. It is based on an innovative concept that comprehensively addresses the challenges DSOs will be exposed to in the context of increasing amount of decentralized and often fluctuating generation as well as the electrification of the heat and transportation sector. In this paper, we outline the principles of the GridBox concept, we describe its key elements in terms of hardware and software and we specify functionalities and applications. The practical implementation of the concept is illustrated by presenting an overview of the results from field tests in two different regions in Switzerland one in an urban grid area in the city of Zurich and one in a rural grid area in the canton of Bern. Results from the evaluation campaigns for state estimation and optimization algorithms are presented.

Combination of multiple acoustic models with unsupervised adaptation for lecture speech transcription

Automatic speech recognition systems (ASR) have achieved considerable progress in real applications because of skilled design of the architecture with advanced techniques and algorithms. However, how to design a system efficiently integrating these various techniques to obtain advanced performance is still a challenging task. In this paper, we introduced an ensemble model combination and adaptation based ASR system with two characteristics: (1) large-scale combination of multiple ASR systems based on a Recognizer Output Voting Error Reduction (ROVER) system, and (2) multi-pass unsupervised speaker adaptation for deep neural network acoustic models and topic adaptation on language model. The multiple acoustic models were trained with different acoustic features and model architectures which helped to provide complementary and discriminative information in the ROVER process. With these multiple acoustic models, a better estimation of word confidence could be obtained from ROVER process which helped in selecting data for unsupervised adaptation on the previously trained acoustic models. The final recognition result was obtained using multi-pass decoding, ROVER, and adaptation processes. We tested the system on lecture speeches with topics related to Technology, Entertainment and Design (TED) that were used in the international workshop on spoken language translation (IWSLT) evaluation campaign, and obtained 6.5%, 7.0%, 10.6%, and 8.4% word error rates for test sets in 2011, 2012, 2013, and 2014, which to our knowledge are the best results for these evaluation sets.

A Large-Scale Open-Source Acoustic Simulator for Speaker Recognition

The state-of-the-art speaker-recognition systems suffer from significant performance loss on degraded speech conditions and acoustic mismatch between enrolment and test phases. Past international evaluation campaigns, such as the NIST speaker recognition evaluation (SRE), have partly addressed these challenges in some evaluation conditions. This work aims at further assessing and compensating for the effect of a wide variety of speech-degradation processes on speaker-recognition performance. We present an open-source simulator generating degraded telephone, VoIP, and interview-speech recordings using a comprehensive list of narrow-band, wide-band, and audio codecs, together with a database of over 60 h of environmental noise recordings and over 100 impulse responses collected from publicly available data. We provide speaker-verification results obtained with an $i$ -vector-based system using either a clean or degraded PLDA back-end on a NIST SRE subset of data corrupted by the proposed simulator. While error rates increase considerably under degraded speech conditions, large relative equal error rate (EER) reductions were observed when using a PLDA model trained with a large number of degraded sessions per speaker.

A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research

In recent years, substantial progress has been made in the field of reverberant speech signal processing, including both single- and multichannel dereverberation techniques and automatic speech recognition (ASR) techniques that are robust to reverberation. In this paper, we describe the REVERB challenge, which is an evaluation campaign that was designed to evaluate such speech enhancement (SE) and ASR techniques to reveal the state-of-the-art techniques and obtain new insights regarding potential future research directions. Even though most existing benchmark tasks and challenges for distant speech processing focus on the noise robustness issue and sometimes only on a single-channel scenario, a particular novelty of the REVERB challenge is that it is carefully designed to test robustness against reverberation, based on both real, single-channel, and multichannel recordings. This challenge attracted 27 papers, which represent 25 systems specifically designed for SE purposes and 49 systems specifically designed for ASR purposes. This paper describes the problems dealt within the challenge, provides an overview of the submitted systems, and scrutinizes them to clarify what current processing strategies appear effective in reverberant speech processing.

Comparison of ALBAYZIN query-by-example spoken term detection 2012 and 2014 evaluations

Query-by-example spoken term detection (QbE STD) aims at retrieving data from a speech repository given an acoustic query containing the term of interest as input. Nowadays, it is receiving much interest due to the large volume of multimedia information. This paper presents the systems submitted to the ALBAYZIN QbE STD 2014 evaluation held as a part of the ALBAYZIN 2014 Evaluation campaign within the context of the IberSPEECH 2014 conference. This is the second QbE STD evaluation in Spanish, which allows us to evaluate the progress in this technology for this language. The evaluation consists in retrieving the speech files that contain the input queries, indicating the start and end times where the input queries were found, along with a score value that reflects the confidence given to the detection of the query. Evaluation is conducted on a Spanish spontaneous speech database containing a set of talks from workshops, which amount to about 7 h of speech. We present the database, the evaluation metric, the systems submitted to the evaluation, the results, and compare this second evaluation with the first ALBAYZIN QbE STD evaluation held in 2012. Four different research groups took part in the evaluations held in 2012 and 2014. In 2014, new multi-word and foreign queries were added to the single-word and in-language queries used in 2012. Systems submitted to the second evaluation are hybrid systems which integrate letter transcription- and template matching-based systems. Despite the significant improvement obtained by the systems submitted to this second evaluation compared to those of the first evaluation, results still show the difficulty of this task and indicate that there is still room for improvement.

A simple and efficient algorithm for authorship verification

This paper describes and evaluates an unsupervised and effective authorship verification model called Spatium‐L1. As features, we suggest using the 200 most frequent terms of the disputed text (isolated words and punctuation symbols). Applying a simple distance measure and a set of impostors, we can determine whether or not the disputed text was written by the proposed author. Moreover, based on a simple rule we can define when there is enough evidence to propose an answer or when the attribution scheme is unable to make a decision with a high degree of certainty. Evaluations based on 6 test collections (PAN CLEF 2014 evaluation campaign) indicate that Spatium‐L1 usually appears in the top 3 best verification systems, and on an aggregate measure, presents the best performance. The suggested strategy can be adapted without any problem to different Indo‐European languages (such as English, Dutch, Spanish, and Greek) or genres (essay, novel, review, and newspaper article).