Detection Error Tradeoff Curves Research Articles

Plasma microRNA-15a/16-1-based machine learning for early detection of hepatitis B virus-related hepatocellular carcinoma

Background and aimsHepatocellular carcinoma (HCC), which is prevalent worldwide and has a high mortality rate, needs to be effectively diagnosed. We aimed to evaluate the significance of plasma microRNA-15a/16-1 (miR-15a/16) as a biomarker of hepatitis B virus-related HCC (HBV-HCC) using the machine learning model. This study was the first large-scale investigation of these two miRNAs in HCC plasma samples. MethodsUsing quantitative polymerase chain reaction, we measured the plasma miR-15a/16 levels in a total of 766 participants, including 74 healthy controls, 335 with chronic hepatitis B (CHB), 47 with compensated liver cirrhosis, and 310 with HBV-HCC. The diagnostic performance of miR-15a/16 was examined using a machine learning model and compared with that of alpha-fetoprotein (AFP). Lastly, to validate the diagnostic efficiency of miR-15a/16, we performed pseudotemporal sorting of the samples to simulate progression from CHB to HCC. ResultsPlasma miR-15a/16 was significantly decreased in HCC than in all control groups (P < 0.05 for all). In the training cohort, the area under the receiver operating characteristic curve (AUC), sensitivity, and average precision (AP) for the detection of HCC were higher for miR-15a (AUC = 0.80, 67.3%, AP = 0.80) and miR-16 (AUC = 0.83, 79.0%, AP = 0.83) than for AFP (AUC = 0.74, 61.7%, AP = 0.72). Combining miR-15a/16 with AFP increased the AUC to 0.86 (sensitivity 85.9%) and the AP to 0.85 and was significantly superior to the other markers in this study (P < 0.05 for all), as further demonstrated by the detection error tradeoff curves. Moreover, miR-15a/16 impressively showed potent diagnostic power in early-stage, small-tumor, and AFP-negative HCC. A validation cohort confirmed these results. Lastly, the simulated follow-up of patients further validated the diagnostic efficiency of miR-15a/16. ConclusionsWe developed and validated a plasma miR-15a/16-based machine learning model, which exhibited better diagnostic performance for the early diagnosis of HCC compared to that of AFP.

A Hybrid Bald Eagle-Crow Search Algorithm for Gaussian mixture model optimisation in the speaker verification framework

A mutualism of nature-inspired algorithms, the Bald Eagle Search Algorithm (BESA) with the Crow Search Algorithm (CSA), is proposed for Gaussian Mixture Model optimisation in the speaker verification framework. BESA simulates bald eagles’ hunting behaviour, whereas CSA simulates crows’ food-hiding behaviour. BESA excels in exploitation but has limitations in exploration, whereas CSA showcases strong exploration capabilities but faces challenges in effective exploitation. BESA’s hunting characteristic combines crows’ memory update to generate a Hybrid Bald Eagle-Crow Search Algorithm (HBE-CSA). The ability of the proposed HBE-CSA is tested on clean speech utterances from the Librispeech dataset and eight real-life noisy conditions from the AURORA dataset at four different Signal-to-Noise Ratios (SNRs). The comparative analysis shows that the proposed HBE-CSA converges with higher log-likelihood than individual search algorithms and the state-of-the-art Expectation Maximisation algorithm. Detection error trade-off curves validate the results of convergence behaviour, indicating that best speaker models are built using the proposed HBE-CSA. The noise analysis shows the smallest equal error rate for the proposed HBE-CSA at different SNRs than competing algorithms. The proposed HBE-CSA holds promise for optimising models not only in speaker verification but also across other areas of speech processing.

Multilingual Customized Keyword Spotting Using Similar-Pair Contrastive Learning

The goal of keyword spotting (KWS) is to find words and voice commands. KWS research has received a lot of attention recently. However, the majority of them emphasize predefined KWS. More often than not, customers wish to choose individualized unique keywords. In this work, we present a brand-new Similar-Pair Contrastive Learning (SPCL) training method called MCKWS-SPCL to address Multilingual Customized Keyword Spotting. The three key processes in our MCKWS-SPCL are acoustic feature extraction, embedding model, and similarity computation. By pushing similar-pair samples closer together in projection space, the SPCL training approach is specifically created to obtain similar features with noise existing. Additionally, multiple embedding neural network structures are carefully constructed to examine the trade-off between a limited number of parameters and excellent performance in order to reduce power consumption. For the first time, we complete an all-around multi-dimensional comparison to demonstrate the superior performance of our SPCL over conventional Contrastive Learning in KWS. Results from all trials, including metrics for average accuracy(96.84%), the detection error tradeoff curve, the false reject rate under 5% false alarm rate per hour(1.901%), and visualization using t-Stochastic Neighbor Embedding(same class is closer than CL), conclusively demonstrate our framework's effectiveness and superiority to Contrastive Learning-based approaches. Additionally, the analysis of four distinct language datasets and implementation on the NVIDIA JETSON TX2 hardware platform demonstrates the versatility, affordability, and hardware-friendliness of our new approach.

Development of Convolutional Neural Network for Classification of Amplitude-Frequency Characteristics of Audio Signals

This paper studies the application of deep convolutional neural networks for the processing of audio files, particularly for classifying amplitude-frequency characteristics of audio signals. The mapping of audio fragments to each other is reduced to verifying objects by their representation. A large representative sample of audio signals was collected and supplemented with a satisfying Free Music Archive dataset to produce a dataset for training a deep convolutional neural network. The CQT-Net architecture is taken as a predictive model with cosine similarity being used to compare feature vectors. Four types of augmentation, including Gaussian noise, reverberation, change in pitch frequency, and change in tempo of the audio signal, are used to prevent retraining of the predictive model. The verification quality of the predictive model is tested on two separate datasets consisting of 1500 audio recordings excluded from the training dataset. Detection error tradeoff curves are plotted for all datasets, including testing ones with a changed pace and with a changed "pitch." Equal Error Rate is used as a model quality metric. The probability of identification of commonly used distortions of audio signals in the amplitude-frequency domain is evaluated to be higher than 92%. It signifies the reliability of the developed model.

Performance metrics for marine mammal signal detection and classification.

Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm.

Performance enhancement of the machine‐fault diagnosis system using feature mapping, normalisation and decision fusion

A unified fault modelling approach for machines with varying operating speeds is of interest in automating production facilities. Building a unified fault model is challenging due to the presence of speed-specific attributes in the features derived. In this work, it is shown how feature selection, mapping, normalisation and fusion of heterogeneous systems can help enhance the performance of speed-independent (SI) machine-fault diagnosis systems. Statistical features, obtained after applying feature selection, are used as input to a support vector machine (SVM) back-end classifier as the baseline system. Entropy-based feature selection algorithm is proposed to improve the performance of the fault diagnosis system. Furthermore, to make the fault diagnosis system independent of speed, locality constrained linear coding (LLC), Fisher vector encoding (FVE) and mean and variance normalisation (MVN) are used. The LLC-MVN system and FVE-MVN system map the input features in terms of SI basis vectors to make the features robust to speed-specific variations. Finally, the decision scores of the time-domain LLC-MVN-SVM, frequency-domain LLC-MVN-SVM systems and variational mode decomposition-based FVE-MVN-SVM system were fused with appropriate weighting factors. The detection error trade-off curve is also used as a performance measure for intelligent fault diagnosis systems.

A Speaker Verification Method Based on TDNN–LSTMP

In speaker recognition, a robust recognition method is essential. This paper proposes a speaker verification method that is based on the time-delay neural network (TDNN) and long short-term memory with recurrent project layer (LSTMP) model for the speaker modeling problem in speaker verification. In this work, we present the application of the fusion of TDNN and LSTMP to the i-vector speaker recognition system that is based on the Gaussian mixture model-universal background model. By using a model that can establish long-term dependencies to create a universal background model that contains a larger amount of speaker information, it is possible to extract more feature parameters, which are speaker dependent, from the speech signal. We conducted experiments with this method on four corpora: two in Chinese and two in English. The equal error rate, minimum detection cost function and detection error tradeoff curve are used as criteria for system performance evaluation. The experimental results show that the TDNN–LSTMP/i-vector speaker recognition method outperforms the baseline system on both Chinese and English corpora and has better robustness.

Classification assessment methods

Classification techniques have been applied to many applications in various fields of sciences. There are several ways of evaluating classification algorithms. The analysis of such metrics and its significance must be interpreted correctly for evaluating different learning algorithms. Most of these measures are scalar metrics and some of them are graphical methods. This paper introduces a detailed overview of the classification assessment measures with the aim of providing the basics of these measures and to show how it works to serve as a comprehensive source for researchers who are interested in this field. This overview starts by highlighting the definition of the confusion matrix in binary and multi-class classification problems. Many classification measures are also explained in details, and the influence of balanced and imbalanced data on each metric is presented. An illustrative example is introduced to show (1) how to calculate these measures in binary and multi-class classification problems, and (2) the robustness of some measures against balanced and imbalanced data. Moreover, some graphical measures such as Receiver operating characteristics (ROC), Precision-Recall, and Detection error trade-off (DET) curves are presented with details. Additionally, in a step-by-step approach, different numerical examples are demonstrated to explain the preprocessing steps of plotting ROC, PR, and DET curves.

Making Likelihood Ratios Digestible for Cross-Application Performance Assessment

Performance estimation is crucial to the assessment of novel algorithms and systems. In detection error tradeoff (DET) diagrams, discrimination performance is solely assessed targeting one application, where cross-application performance considers risks resulting from decisions, depending on application constraints. For the purpose of interchangeability of research results across different application constraints, we propose to augment DET curves by depicting systems regarding their support of security and convenience levels. Therefore, application policies are aggregated into levels based on verbal likelihood ratio scales, providing an easy to use concept for business-to-business communication to denote operative thresholds. We supply a reference implementation in Python, an exemplary performance assessment on synthetic score distributions, and a fine-tuning scheme for Bayes decision thresholds, when decision policies are bounded rather than fix.

Practical security and privacy attacks against biometric hashing using sparse recovery

Biometric hashing is a cancelable biometric verification method that has received research interest recently. This method can be considered as a two-factor authentication method which combines a personal password (or secret key) with a biometric to obtain a secure binary template which is used for authentication. We present novel practical security and privacy attacks against biometric hashing when the attacker is assumed to know the user’s password in order to quantify the additional protection due to biometrics when the password is compromised. We present four methods that can reconstruct a biometric feature and/or the image from a hash and one method which can find the closest biometric data (i.e., face image) from a database. Two of the reconstruction methods are based on 1-bit compressed sensing signal reconstruction for which the data acquisition scenario is very similar to biometric hashing. Previous literature introduced simple attack methods, but we show that we can achieve higher level of security threats using compressed sensing recovery techniques. In addition, we present privacy attacks which reconstruct a biometric image which resembles the original image. We quantify the performance of the attacks using detection error tradeoff curves and equal error rates under advanced attack scenarios. We show that conventional biometric hashing methods suffer from high security and privacy leaks under practical attacks, and we believe more advanced hash generation methods are necessary to avoid these attacks.

Elastic strips normalisation model for higher iris recognition performance

Iris recognition is among the best biometric systems. Characterised by the iris's uniqueness, universality, distinctiveness, permanence and collectability, the iris recognition system achieves high performance and real time response. In this study, the authors propose an improved iris normalisation model applied after a precise iris segmentation process. The normalisation model defines a new reference space for iris features. It normalises the iris using radial strips with a shape that changes between the pupil's boundary and the circular approximation of the iris's outer boundary. Moreover, the effect of the centres of the normalisation strips is evaluated by assessing the recognition performance when comparing three different centres configurations. The approach is tested on 2491 images from the CASIA V3 database. The system's performance is measured at the matching stage. Higher decidability and recognition accuracy at equal error rate is obtained. Detection error tradeoff curves are estimated by using the proposed model and compared with Daugman's reference system for assessing performance improvement.

Robust heart sound detection in respiratory sound using LRT with maximum a posteriori based online parameter adaptation

This paper investigates the utility of a likelihood ratio test (LRT) combined with an efficient adaptation procedure for the purpose of detecting the heart sound (HS) with lung sound and the lung sound only (non-HS) segments in a respiratory signal. The proposed detection method has four main stages: feature extraction, training of the models, detection, and adaptation of the model parameter. In the first stage, the logarithmic energy features are extracted for each frame of respiratory sound. In the second stage, the probabilistic models for HS and non-HS segments are constructed by training Gaussian mixture models (GMMs) with an expectation maximization algorithm in a subject-independent manner, and then the HS and non-HS segments are detected by the results of the LRT based on the GMMs. In the adaptation stage, the subject-independent trained model parameter is modified online using the observed test data to fit the model parameter of the target subject. Experiments were performed on the database from 24 healthy subjects. The experimental results indicate that the proposed heart sound detection algorithm outperforms two well-known heart sound detection methods in terms of the values of the normalized area under the detection error trade-off curve (NAUC), the false negative rate (FNR), and the false positive rate (FPR).

Universal background modeling for acoustic surveillance of urban traffic

Traffic congestion in modern cities is an increasing problem having significant consequences in our daily lives. This work proposes a non-intrusive, passive monitoring framework based on the acoustic modality which can be used either autonomously or as a part of a multimodal system and provide valuable information to an intelligent transportation system. We consider a large number of audio classes which are typically encountered in urban areas. We introduce a combination of a powerful audio representation mechanism based on time, frequency and wavelet domain features with universal background modeling which leads to higher recognition accuracies and detection rates (in terms of false alarm and miss probability rates) with respect to commonly employed methodologies. The basic advantage of a class-specific model derived using the universal background modeling logic is its tolerance to data which belong to other sound classes. Another important feature of the proposed system is its ability to detect crash incidents, which apart from their catastrophic impact on human life and property, have negative consequences on the traffic flow. Our experiments are based on the concurrent usage of professional sound effect collections which include audio recordings of high quality. We thoroughly examine the performance of the proposed system on isolated sound events as well as continuous audio streams using confusion matrices and detection error trade-off curves.

MODELLING THE ADAPTATION OF BUSINESS CONTINUITY PLANNING BY BUSINESSES USING NEURAL NETWORKS

SUMMARY Business continuity planning is an important element of business continuity management and is regarded as a fundamental step towards reducing the negative impacts of business disruptions caused by internal and external hazardous events. Many businesses are not prepared for such events, and very few studies have tried to examine and model the factors that contribute to business continuity management planning by various companies. In this paper we propose and develop a feed-forward neural network for modelling businesses continuity planning by businesses based on a dataset of 283 businesses operating in the Greater Toronto Area in Ontario, Canada. The fully connected neural network applied was trained on 65 % of the dataset records using different subsets of input variables. In order to preserve the generalization ability of the trained network, 15 % of the dataset records were used as a validation set for early stopping during the network's training process. Prediction capability of the trained networks was evaluated on 20 % and never-seen records of the dataset. The classification ability of the networks was then analysed using receiver operating characteristic and detection error trade-off curves, where the results obtained were promising. The equal error rate for the best models was 12 %, which reflects a very good accuracy of these models in predicting the existence of business continuity planning for a generic company. Copyright © 2011 John Wiley & Sons, Ltd.

Feature Selection Based on Minimizing the Area Under the Detection Error Tradeoff Curve

Feature selection is crucial to select an “optimized” subset of features from the original feature set based on a certain objective function. In general, feature selection removes redundant or irrelevant data while retaining classification accuracy. This paper proposes a feature selection algorithm that aims to minimize the area under the curve of detection error trade-off (DET) curve. Particle swarm optimization (PSO) is employed to search for the optimal feature subset. The proposed method is implemented in face recognition and iris recognition systems. The result shows that the proposed method is able to find an optimal subset of features that sufficiently describes iris and face images by removing unwanted and redundant features and at the same time improving the classification accuracy in terms of total error rate (TER).

Eyelid and eyelash detection method in the normalized iris image using the parabolic Hough model and Otsu’s thresholding method

Eyelids and eyelashes occluding the iris region are noise factors that degrade the performance of iris recognition. If they are incorrectly classified as an iris region, the false iris region information decreases the recognition rate. Thus, reliable detection of eyelids and eyelashes is required to improve the performance of iris recognition. In this paper, we propose an automatic eyelid and eyelash detection method based on the parabolic Hough model and Otsu’s thresholding method. By applying the parabolic Hough transform to the normalized iris image, rather than to the original image, we reduce the dimension of the parameter space and limit the parameter search range, decreasing the computational load. In addition, for automatically separating the eyelash region we apply Otsu’s method to the proposed feature that is obtained by combining the intensity and local standard deviation values. The proposed method is applied to the CASIA version 3 database and the performance of the proposed and six existing methods is assessed in terms of the decidability, equal error rate, and detection error trade-off curve. In terms of these performance measures, the proposed method shows the better performance than conventional methods.

Performance Generalization in Biometric Authentication Using Joint User-Specific and Sample Bootstraps

Biometric authentication performance is often depicted by a detection error trade-off (DET) curve. We show that this curve is dependent on the choice of samples available, the demographic composition and the number of users specific to a database. We propose a two-step bootstrap procedure to take into account the three mentioned sources of variability. This is an extension to the Bolle et al.'s bootstrap subset technique. Preliminary experiments on the NIST2005 and XM2VTS benchmark databases are encouraging, e.g., the average result across all 24 systems evaluated on NIST2005 indicates that one can predict, with more than 75 percent of DET coverage, an unseen DET curve with eight times more users. Furthermore, our finding suggests that with more data available, the confidence intervals become smaller and, hence, more useful.

Improved prediction of bacterial transcription start sites

Identifying bacterial promoters is an important step towards understanding gene regulation. In this paper, we address the problem of predicting the location of promoters and their transcription start sites (TSSs) in Escherichia coli. The accepted method for this problem is to use position weight matrices (PWMs), which define conserved motifs at the sigma-factor binding site. However this method is known to result in large numbers of false positive predictions. Our approaches to TSS prediction are based upon an ensemble of support vector machines (SVMs) employing a variant of the mismatch string kernel. This classifier is subsequently combined with a PWM and a model based on distribution of distances from TSS to gene start. We investigate the effect of different scoring techniques and quantify performance using area under a detection-error tradeoff curve. When tested on a biologically realistic task, our method provides performance comparable with or superior to the best reported for this task. False positives are significantly reduced, an improvement of great significance to biologists. The trained ensemble-SVM model with instructions on usage can be downloaded from http://eresearch.fit.qut.edu.au/downloads

The NIST speaker recognition evaluation – Overview, methodology, systems, results, perspective

This paper, based on three presentations made in 1998 at the RLA2C Workshop in Avignon, discusses the evaluation of speaker recognition systems from several perspectives. A general discussion of the speaker recognition task and the challenges and issues involved in its evaluation is offered. The NIST evaluations in this area and specifically the 1998 evaluation, its objectives, protocols and test data, are described. The algorithms used by the systems that were developed for this evaluation are summarized, compared and contrasted. Overall performance results of this evaluation are presented by means of detection error trade-off (DET) curves. These show the performance trade-off of missed detections and false alarms for each system and the effects on performance of training condition, test segment duration, the speakers' sex and the match or mismatch of training and test handsets. Several factors that were found to have an impact on performance, including pitch frequency, handset type and noise, are discussed and DET curves showing their effects are presented. The paper concludes with some perspective on the history of this technology and where it may be going.