Robust Performance Estimation Research Articles

A protocol for trustworthy EEG decoding with neural networks

Deep learning solutions have rapidly emerged for EEG decoding, achieving state-of-the-art performance on a variety of decoding tasks. Despite their high performance, existing solutions do not fully address the challenge posed by the introduction of many hyperparameters, defining data pre-processing, network architecture, network training, and data augmentation. Automatic hyperparameter search is rarely performed and limited to network-related hyperparameters. Moreover, pipelines are highly sensitive to performance fluctuations due to random initialization, hindering their reliability. Here, we design a comprehensive protocol for EEG decoding that explores the hyperparameters characterizing the entire pipeline and that includes multi-seed initialization for providing robust performance estimates. Our protocol is validated on 9 datasets about motor imagery, P300, SSVEP, including 204 participants and 26 recording sessions, and on different deep learning models. We accompany our protocol with extensive experiments on the main aspects influencing it, such as the number of participants used for hyperparameter search, the split into sequential simpler searches (multi-step search), the use of informed vs. non-informed search algorithms, and the number of random seeds for obtaining stable performance. The best protocol included 2-step hyperparameter search via an informed search algorithm, with the final training and evaluation performed using 10 random initializations. The optimal trade-off between performance and computational time was achieved by using a subset of 3-5 participants for hyperparameter search. Our protocol consistently outperformed baseline state-of-the-art pipelines, widely across datasets and models, and could represent a standard approach for neuroscientists for decoding EEG in a trustworthy and reliable way.

Dataset Artefacts in Anti-Spoofing Systems: A Case Study on the ASVspoof 2017 Benchmark

The Automatic Speaker Verification Spoofing and Countermeasures Challenges motivate research in protecting speech biometric systems against a variety of different access attacks. The 2017 edition focused on replay spoofing attacks, and involved participants building and training systems on a provided dataset (ASVspoof 2017). More than 60 research papers have so far been published with this dataset, but none have sought to answer why countermeasures appear successful in detecting spoofing attacks. This article shows how artefacts inherent to the dataset may be contributing to the apparent success of published systems. We first inspect the ASVspoof 2017 dataset and summarize various artefacts present in the dataset. Second, we demonstrate how countermeasure models can exploit these artefacts to appear successful in this dataset. Third, for reliable and robust performance estimates on this dataset we propose discarding nonspeech segments and silence before and after the speech utterance during training and inference. We create speech start and endpoint annotations in the dataset and demonstrate how using them helps countermeasure models become less vulnerable from being manipulated using artefacts found in the dataset. Finally, we provide several new benchmark results for both frame-level and utterance-level models that can serve as new baselines on this dataset.

Kinematic features of a simple and short movement task to predict autism diagnosis.

Autism is a developmental condition primarily identified by social and communication deficits. However, over 70% of autistic individuals also show motor function deficits, which are evident even when simple stereotyped movements are performed. In this study, we have asked 24 autistic and 22 non-autistic adults to perform pointing movements between two markers 30 cm apart as quickly and as accurately as they can for 10 seconds. Motion tracking was employed to collect data and calculate kinematic features of the movement and aiming accuracy. At the group level, the results showed that autistic individuals performed pointing movements slower but more accurately compared to non-autistic individuals. At the individual level, we have used Machine Learning methods to predict autism diagnosis. Nested result Cross-Validation was used, which in contrast to commonly used K-fold Cross-Validation avoids pooling training and testing data and provides robust performance estimates. Our developed models achieved a statistically significant classification accuracy of 71% and showed that even a simple and short motor task enables discrimination between autistic and non-autistic individuals.

Diagnostic performance of quantitative flow ratio in prospectively enrolled patients: An individual patient-data meta-analysis.

We aimed to provide robust performance estimates for quantitative flow ratio (QFR) in assessment of intermediary coronary lesions. Angiography-based functional lesion assessment by QFR may appear as a cost saving and safe approach to expand the use of physiology-guided percutaneous coronary interventions. QFR was proven feasible and showed good diagnostic performance in mid-sized off-line and on-line studies with fractional flow reserve (FFR) as reference standard. We performed a collaborative individual patient-data meta-analysis of all available prospective studies with paired assessment of QFR and FFR using the CE-marked QFR application. The main outcome was agreement of QFR and FFR using a two-step analysis strategy with a multilevel mixed model accounting for study and center level variation. Of 16 studies identified, four studies had prospective enrollment and provided patient level data reaching a total of 819 patients and 969 vessels with paired FFR and QFR: FAVOR Pilot (n = 73); WIFI II (n = 170); FAVOR II China (n = 304) and FAVOR II Europe-Japan (n = 272). We found an overall agreement (mean difference 0.009 ± 0.068, I2 = 39.6) of QFR with FFR. The diagnostic performance was sensitivity 84% (95%CI: 77-90, I2 = 70.1), specificity 88% (95%CI: 84-91, I2 = 60.1); positive predictive value 80% (95%CI: 76-85, I2 = 33.4), and negative predictive value 95% (95%CI: 93-96, I2 = 75.9). Diagnostic performance of QFR was good with FFR as reference in this meta-analysis of high quality studies. QFR could provide an easy, safe, and cost-effective solution for functional evaluation of coronary artery stenosis.

Local stability analysis and H∞ performance for Lipschitz digital filters with saturation nonlinearity and external interferences

Abstract This paper proposes a novel method to analyze the local stability of Lipschitz nonlinear digital filtering schemes under saturation overflow nonlinearity. Conditions for the stability analysis and robust performance estimation are provided in the form of matrix inequalities by utilizing Lyapunov theory, local saturation overflow arithmetic, and Lipschitz condition. The proposed criterion ascertains (local) asymptotic stability in the absence of perturbations. Under the effects of external interferences, a condition for the local stability, ensuring the H∞ performance objective, is developed. The proposed approach offers a less conservative and more accurate estimate of H∞ performance index than the global method by utilizing a bound on the interferences energy. Moreover, the proposed criterion, in contrast to the existing global methods, can be employed to choose an adequate word length of a digital hardware for the specified values of tolerable perturbations energy, H∞ performance index, and fixed-point resolution. It is worth mentioning that analysis approaches have not been completely reported in the literature, in which local stability criteria for nonlinear discrete-time filtering prototypes under both overflow and disturbances have been developed. A detailed stability analysis for a nonlinear recurrent neural network is performed for demonstrating the effectiveness of the proposed scheme.

An efficient method for uncertainty propagation in robust software performance estimation

Software engineers often have to estimate the performance of a software system before having full knowledge of the system parameters, such as workload and operational profile. These uncertain parameters inevitably affect the accuracy of quality evaluations, and the ability to judge if the system can continue to fulfil performance requirements if parameter results are different from expected. Previous work has addressed this problem by modelling the potential values of uncertain parameters as probability distribution functions, and estimating the robustness of the system using Monte Carlo-based methods. These approaches require a large number of samples, which results in high computational cost and long waiting times.To address the computational inefficiency of existing approaches, we employ Polynomial Chaos Expansion (PCE) as a rigorous method for uncertainty propagation and further extend its use to robust performance estimation. The aim is to assess if the software system is robust, i.e., it can withstand possible changes in parameter values, and continue to meet performance requirements. PCE is a very efficient technique, and requires significantly less computations to accurately estimate the distribution of performance indices. Through three very different case studies from different phases of software development and heterogeneous application domains, we show that PCE can accurately ( > 97%) estimate the robustness of various performance indices, and saves up to 225 h of performance evaluation time when compared to Monte Carlo Simulation.

How Costly Is Forced Gender-Balancing of Corporate Boards?

In 2005, Norway became the first country to mandate gender-balanced corporate boards. We hypothesize that a gender quota reduces director CEO experience and increases board independence. Contrary to prior research, our robust performance estimates fail to reject an overall value-neutral effect of the quota, even for firms with all-male boards. We also show that, while boards lost some CEO experience, firms did not increase board size (to retain key male directors) or change legal form (to avoid the quota), and managed to maintain board network power. We conclude that investors and firms alike viewed the quota as a relatively low-cost constraint.

Heterogeneous Feature Models and Feature Selection Applied to Bearing Fault Diagnosis

Distinct feature extraction methods are simultaneously used to describe bearing faults. This approach produces a large number of heterogeneous features that augment discriminative information but, at the same time, create irrelevant and redundant information. A subsequent feature selection phase filters out the most discriminative features. The feature models are based on the complex envelope spectrum, statistical time- and frequency-domain parameters, and wavelet packet analysis. Feature selection is achieved by conventional search of the feature space by greedy methods. For the final fault diagnosis, the k-nearest neighbor classifier, feedforward net, and support vector machine are used. Performance criteria are the estimated error rate and the area under the receiver operating characteristic curve (AUC-ROC). Experimental results are shown for the Case Western Reserve University Bearing Data. The main contribution of this paper is the strategy to use several different feature models in a single pool, together with feature selection to optimize the fault diagnosis system. Moreover, robust performance estimation techniques usually not encountered in the context of engineering are employed.

Robustness of performance and stability for multistep and updated multistep MPC schemes

We consider a model predictive control approach to approximate the solution of infinite horizon optimal control problems for perturbed nonlinear discrete time systems. By reducing the number of re-optimizations, the computational load can be lowered considerably at the expense of reduced robustness of the closed-loop solution against perturbations. In this paper, we propose and analyze an update strategy based on re-optimizations on shrinking horizons which is computationally less expensive than that based on full horizon re-optimization, and at the same time allowing for rigorously quantifiable robust performance estimates.

Robust clinical outcome prediction based on Bayesian analysis of transcriptional profiles and prior causal networks.

Motivation: Understanding and predicting an individual’s response in a clinical trial is the key to better treatments and cost-effective medicine. Over the coming years, more and more large-scale omics datasets will become available to characterize patients with complex and heterogeneous diseases at a molecular level. Unfortunately, genetic, phenotypical and environmental variation is much higher in a human trial population than currently modeled or measured in most animal studies. In our experience, this high variability can lead to failure of trained predictors in independent studies and undermines the credibility and utility of promising high-dimensional datasets.Methods: We propose a method that utilizes patient-level genome-wide expression data in conjunction with causal networks based on prior knowledge. Our approach determines a differential expression profile for each patient and uses a Bayesian approach to infer corresponding upstream regulators. These regulators and their corresponding posterior probabilities of activity are used in a regularized regression framework to predict response.Results: We validated our approach using two clinically relevant phenotypes, namely acute rejection in kidney transplantation and response to Infliximab in ulcerative colitis. To demonstrate pitfalls in translating trained predictors across independent trials, we analyze performance characteristics of our approach as well as alternative feature sets in the regression on two independent datasets for each phenotype. We show that the proposed approach is able to successfully incorporate causal prior knowledge to give robust performance estimates.Contact: daniel.ziemek@pfizer.comSupplementary information: Supplementary data are available at Bioinformatics online.