Two-class Classification Task Research Articles

An EEG channel selection method for motor imagery based on Fisher score and local optimization

Objective. Multi-channel electroencephalogram (EEG) technology in brain–computer interface (BCI) research offers the advantage of enhanced spatial resolution and system performance. However, this also implies that more time is needed in the data processing stage, which is not conducive to the rapid response of BCI. Hence, it is a necessary and challenging task to reduce the number of EEG channels while maintaining decoding effectiveness. Approach. In this paper, we propose a local optimization method based on the Fisher score for within-subject EEG channel selection. Initially, we extract the common spatial pattern characteristics of EEG signals in different bands, calculate Fisher scores for each channel based on these characteristics, and rank them accordingly. Subsequently, we employ a local optimization method to finalize the channel selection. Main results. On the BCI Competition IV Dataset IIa, our method selects an average of 11 channels across four bands, achieving an average accuracy of 79.37%. This represents a 6.52% improvement compared to using the full set of 22 channels. On our self-collected dataset, our method similarly achieves a significant improvement of 24.20% with less than half of the channels, resulting in an average accuracy of 76.95%. Significance. This research explores the importance of channel combinations in channel selection tasks and reveals that appropriately combining channels can further enhance the quality of channel selection. The results indicate that the model selected a small number of channels with higher accuracy in two-class motor imagery EEG classification tasks. Additionally, it improves the portability of BCI systems through channel selection and combinations, offering the potential for the development of portable BCI systems.

Aflatoxins detection in almonds via fluorescence imaging and deep neural network approach

The detection of aflatoxin B in raw food materials represents a topic of great interest worldwide because of the huge health and economic impact of aflatoxin contamination. In this paper, we present an original approach to aflatoxin detection, using a portable instrument to acquire fluorescence images, among other spectral responses. The acquired images are processed by combining a color space conversion from the RGB scale to Y′CbCr, and a neural network approach to encode a vector of features. After a feature reduction using a Receiving Operating Curve method, two-class and three-class classification tasks of contaminated vs non-contaminated samples are accomplished. This procedure has been applied to artificially contaminated grained almond samples in the range of 0–320.2 ng/g, achieving an overall classification accuracy between 84.7% and 93.0%, depending on the number of classes. Thus, in this setting, we show that good classification performance can be achieved using only an image acquisition and analysis approach. The proposed procedure can represent a cheap, rapid, non-destructive yet sensitive method for the assessment of aflatoxin B contamination in food matrices, and its monitoring and tracing throughout the food chain.

GeNetOntology: identifying affected gene ontology terms via grouping, scoring, and modeling of gene expression data utilizing biological knowledge-based machine learning.

Introduction: Identifying significant sets of genes that are up/downregulated under specific conditions is vital to understand disease development mechanisms at the molecular level. Along this line, in order to analyze transcriptomic data, several computational feature selection (i.e., gene selection) methods have been proposed. On the other hand, uncovering the core functions of the selected genes provides a deep understanding of diseases. In order to address this problem, biological domain knowledge-based feature selection methods have been proposed. Unlike computational gene selection approaches, these domain knowledge-based methods take the underlying biology into account and integrate knowledge from external biological resources. Gene Ontology (GO) is one such biological resource that provides ontology terms for defining the molecular function, cellular component, and biological process of the gene product. Methods: In this study, we developed a tool named GeNetOntology which performs GO-based feature selection for gene expression data analysis. In the proposed approach, the process of Grouping, Scoring, and Modeling (G-S-M) is used to identify significant GO terms. GO information has been used as the grouping information, which has been embedded into a machine learning (ML) algorithm to select informative ontology terms. The genes annotated with the selected ontology terms have been used in the training part to carry out the classification task of the ML model. The output is an important set of ontologies for the two-class classification task applied to gene expression data for a given phenotype. Results: Our approach has been tested on 11 different gene expression datasets, and the results showed that GeNetOntology successfully identified important disease-related ontology terms to be used in the classification model. Discussion: GeNetOntology will assist geneticists and scientists to identify a range of disease-related genes and ontologies in transcriptomic data analysis, and it will also help doctors design diagnosis platforms and improve patient treatment plans.

Statistically significant features improve binary and multiple Motor Imagery task predictions from EEGs.

In recent studies, in the field of Brain-Computer Interface (BCI), researchers have focused on Motor Imagery tasks. Motor Imagery-based electroencephalogram (EEG) signals provide the interaction and communication between the paralyzed patients and the outside world for moving and controlling external devices such as wheelchair and moving cursors. However, current approaches in the Motor Imagery-BCI system design require effective feature extraction methods and classification algorithms to acquire discriminative features from EEG signals due to the non-linear and non-stationary structure of EEG signals. This study investigates the effect of statistical significance-based feature selection on binary and multi-class Motor Imagery EEG signal classifications. In the feature extraction process performed 24 different time-domain features, 15 different frequency-domain features which are energy, variance, and entropy of Fourier transform within five EEG frequency subbands, 15 different time-frequency domain features which are energy, variance, and entropy of Wavelet transform based on five EEG frequency subbands, and 4 different Poincare plot-based non-linear parameters are extracted from each EEG channel. A total of 1,364 Motor Imagery EEG features are supplied from 22 channel EEG signals for each input EEG data. In the statistical significance-based feature selection process, the best one among all possible combinations of these features is tried to be determined using the independent t-test and one-way analysis of variance (ANOVA) test on binary and multi-class Motor Imagery EEG signal classifications, respectively. The whole extracted feature set and the feature set that contain statistically significant features only are classified in this study. We implemented 6 and 7 different classifiers in multi-class and binary (two-class) classification tasks, respectively. The classification process is evaluated using the five-fold cross-validation method, and each classification algorithm is tested 10 times. These repeated tests provide to check the repeatability of the results. The maximum of 61.86 and 47.36% for the two-class and four-class scenarios, respectively, are obtained with Ensemble Subspace Discriminant among all these classifiers using selected features including only statistically significant features. The results reveal that the introduced statistical significance-based feature selection approach improves the classifier performances by achieving higher classifier performances with fewer relevant components in Motor Imagery task classification. In conclusion, the main contribution of the presented study is two-fold evaluation of non-linear parameters as an alternative to the commonly used features and the prediction of multiple Motor Imagery tasks using statistically significant features.

A deep domain adaptation framework with correlation alignment for EEG-based motor imagery classification

It is impractical to collect sufficient and well-labeled EEG data in Brain-computer interface because of the time-consuming data acquisition and costly annotation. Conventional classification methods reusing EEG data from different subjects and time periods (across domains) significantly decrease the classification accuracy of motor imagery. In this paper, we propose a deep domain adaptation framework with correlation alignment (DDAF-CORAL) to solve the problem of distribution divergence for motor imagery classification across domains. Specifically, a two-stage framework is adopted to extract deep features for raw EEG data. The distribution divergence caused by subjected-related and time-related variations is further minimized by aligning the covariance of the source and target EEG feature distributions. Finally, the classification loss and adaptation loss are optimized simultaneously to achieve sufficient discriminative classification performance and low feature distribution divergence. Extensive experiments on three EEG datasets demonstrate that our proposed method can effectively reduce the distribution divergence between the source and target EEG data. The results show that our proposed method delivers outperformance (an average classification accuracy of 92.9% for within-session, an average kappa value of 0.761 for cross-session, and an average classification accuracy of 83.3% for cross-subject) in two-class classification tasks compared to other state-of-the-art methods.

D2AFNet: A dual-domain attention cascade network for accurate and interpretable atrial fibrillation detection

Atrial fibrillation is one of the common and potentially dangerous persistent cardiac arrhythmias that are generally associated with the risk of stroke and heart failure. Manual electrocardiography diagnosis is the gold standard for the clinical detection of atrial fibrillation, but it has some drawbacks, such as being time-consuming and prone to misclassification due to inter-patient variability. Due to the powerful ability of deep learning to learn and extract rich features from huge datasets, end-to-end deep learning models are generally designed to detect abnormal atrial fibrillation signals automatically. However, these approaches usually ignore the key factors that feature maps from different channels and sequences may contribute differently to atrial fibrillation detection, making it challenging to implement accurate and interpretable models with better generalization performance. To tackle this challenge, we develop a dual-domain attention cascade D2AFNet for accurate and interpretable atrial fibrillation detection by cascading attention-based bidirectional gated recurrent units and densely connected networks embedded with channel-spatial information fusion modules. The D2AFNet can take full advantage of channel-spatial features to enhance the feature representation in the spatial domain, and then combine with the time series features in the temporal domain to form spatial–temporal fusion attention mechanisms to mine discriminative atrial fibrillation patterns. Besides, the D2AFNet can profoundly explore the different contributions of different spatial and temporal segments of feature maps for excellent interpretation. The proposed D2AFNet method is performed ten-fold cross-validation on the publicly available CPSC 2018 dataset, and achieves the accuracies of 99.49% and 99.28% in the two-class and three-class classification tasks, outperforming cutting-edge atrial fibrillation detection methods. In addition, the powerful generalization performance and inference efficiency of the D2AFNet method are also proved on another publicly available MIT-BIH dataset. The advantages of high performance and interpretability indicate that the D2AFNet method has huge potential in the computer-aided diagnosis of atrial fibrillation.

High-Precision Real-Time Forest Fire Video Detection Using One-Class Model

Machine learning-base fire detection methods play a vital role in the current forest fire monitoring systems. In the literature, fire detection is usually viewed as a two-class (or multi-class) classification task. However, this violates the most foundational hypothesis in machine learning, e.g., independent and identical distribution (i.i.d.), especially for the non-fire samples drawn from a complex forest background. Moreover, for omni-directional video -monitoring, the background is also always changing, which leads this violation to a worse situation. In this work, by relaxing the i.i.d. of non-fire samples, we aim to learn a one-class model that just relies on the fire samples. Considering the requirements of high-precision and real-time detection, training samples are directly constructed on the fire pixels without a complex feature transformation. Additionally, we also provide a batch decision-making strategy to speed up fire detection. This work also includes an extensive experimental comparison on the public forest fire videos, obtained by ground- or unmanned aerial vehicle (UAV)-monitoring cameras. Compared with the state-of-the-art methods, the results show the superiority of our proposal in terms of a high-fire detection rate, low-error warning rate, accurate fire location positioning, and real-time detection.

Probabilistic Pocket Druggability Prediction via One-Class Learning.

The choice of target pocket is a key step in a drug discovery campaign. This step can be supported by in silico druggability prediction. In the literature, druggability prediction is often approached as a two-class classification task that distinguishes between druggable and non-druggable (or less druggable) pockets (or voxels). Apart from obvious cases, however, the non-druggable class is conceptually ambiguous. This is because any pocket (or target) is only non-druggable until a drug is found for it. It is therefore more appropriate to adopt a one-class approach, which uses only unambiguous information, namely, druggable pockets. Here, we propose using the import vector domain description (IVDD) algorithm to support this task. IVDD is a one-class probabilistic kernel machine that we previously introduced. To feed the algorithm, we use customized DrugPred descriptors computed via NanoShaper. Our results demonstrate the feasibility and effectiveness of the approach. In particular, we can remove or mitigate biases chiefly due to the labeling.

Development of variational quantum deep neural networks for image recognition

Parametrized quantum circuits are widely used for supervised learning tasks such as image classification in the noisy intermediate scale quantum era. However, normally, it can only handle low-dimensional data. This study presented a variational quantum deep neural network (VQDNN) model for various scale image recognition tasks. Three classifiers were designed to verify the classification performance of the proposed VQDNN model. In the first classifier, to accommodate the limitations of qubits in both simulation hardware and real quantum hardware – we adopted hybrid principal component analysis – VQDNN architecture. Moreover, the amplitude encoding scheme and the rotation angle coding scheme were employed in the subsequent two classifiers to handle large-size images. Finally, we used the classical neural network and VQDNN model to conduct a comparative experiment of the ten-label classification learning task on the same dataset. The quantum numerical experiment was implemented on two benchmark datasets: the MNIST and UCI databases of handwritten digits. The simulation results showed that the proposed VQDNN classified the two datasets with an accuracy of 100% for the two-class classification task, while the UCI dataset has an accuracy of 90.87% for the ten-label classification task. The proposed VQDNN achieved better classification accuracy than the original classical neural network even under a limited number of qubits available in current hardware, indicating the promising application potential of VQDNN in image recognition.

Predicting Brain Amyloid-β PET Grades with Graph Convolutional Networks Based on Functional MRI and Multi-Level Functional Connectivity.

The detection of amyloid-β (Aβ) deposition in the brain provides crucial evidence in the clinical diagnosis of Alzheimer's disease (AD). However, the current positron emission tomography (PET)-based brain Aβ examination suffers from the problems of coarse visual inspection (in many cases, with 2-class stratification) and high scanning cost. 1) To characterize the non-binary Aβ deposition levels in the AD continuum based on clustering of PET data, and 2) to explore the feasibility of predicting individual Aβ deposition grades with non-invasive functional magnetic resonance imaging (fMRI). 1) Individual whole-brain Aβ-PET images from the OASIS-3 dataset (N = 258) were grouped into three clusters (grades) with t-SNE and k-means. The demographical data as well as global and regional standard uptake value ratios (SUVRs) were compared among the three clusters with Chi-square tests or ANOVA tests. 2) From resting-state fMRI, both conventional functional connectivity (FC) and high-order FC networks were constructed and the topological architectures of the two networks were jointly learned with graph convolutional networks (GCNs) to predict the Aβ-PET grades for each individual. We found three clearly separated clusters, indicating three Aβ-PET grades. There were significant differences in gender, age, cognitive ability, APOE type, as well as global and regional SUVRs among the three grades we found. The prediction of Aβ-PET grades with GCNs on FC for the 258 samples in the AD continuum reached a satisfactory averaged accuracy (78.8%) in the two-class classification tasks. The results demonstrated the feasibility of using deep learning on a non-invasive brain functional imaging technique to approximate PET-based Aβ deposition grading.

Multi-class WHMBoost: An ensemble algorithm for multi-class imbalanced data

The imbalanced data problem is widespread in the real world. In the process of training machine learning models, ignoring imbalanced data problems will cause the performance of the model to deteriorate. At present, researchers have proposed many methods to deal with the imbalanced data problems, but these methods mainly focus on the imbalanced data problems in two-class classification tasks. Learning from multi-class imbalanced data sets is still an open problem. In this paper, an ensemble method for classifying multi-class imbalanced data sets is put forward, called multi-class WHMBoost. It is an extension of WHMBoost that we proposed earlier. We do not use the algorithm used in WHMBoost to process the data, but use random balance based on average size so as to balance the data distribution. The weak classifiers we use in the boosting algorithm are support vector machine and decision tree classifier. In the process of training the model, they participate in training with given weights in order to complement each other’s advantages. On 18 multi-class imbalanced data sets, we compared the performance of multi-class WHMBoost with state of the art ensemble algorithms using MAUC, MG-mean and MMCC as evaluation criteria. The results demonstrate that it has obvious advantages compared with state of the art ensemble algorithms and can effectively deal with multi-class imbalanced data sets.

A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery Based Brain-Computer Interface

The motor imagery (MI) based brain-computer interfaces (BCIs) have been proposed as a potential physical rehabilitation technology. However, the low classification accuracy achievable with MI tasks is still a challenge when building effective BCI systems. We propose a novel MI classification model based on measurement of functional connectivity between brain regions and graph theory. Specifically, motifs describing local network structures in the brain are extracted from functional connectivity graphs. A graph embedding model called Ego-CNNs is then used to build a classifier, which can convert the graph from a structural representation to a fixed-dimensional vector for detecting critical structure in the graph. We validate our proposed method on four datasets, and the results show that our proposed method produces high classification accuracies in two-class classification tasks (92.8% for dataset 1, 93.4% for dataset 2, 96.5% for dataset 3, and 80.2% for dataset 4) and multiclass classification tasks (90.33% for dataset 1). Our proposed method achieves a mean Kappa value of 0.88 across nine participants, which is superior to other methods we compared it to. These results indicate that there is a local structural difference in functional connectivity graphs extracted under different motor imagery tasks. Our proposed method has great potential for motor imagery classification in future studies.

Self-supervised Learning for Reading Activity Classification

Reading analysis can relay information about user's confidence and habits and can be used to construct useful feedback. A lack of labeled data inhibits the effective application of fully-supervised Deep Learning (DL) for automatic reading analysis. We propose a Self-supervised Learning (SSL) method for reading analysis. Previously, SSL has been effective in physical human activity recognition (HAR) tasks, but it has not been applied to cognitive HAR tasks like reading. We first evaluate the proposed method on a four-class classification task on reading detection using electrooculography datasets, followed by an evaluation of a two-class classification task of confidence estimation on multiple-choice questions using eye-tracking datasets. Fully-supervised DL and support vector machines (SVMs) are used as comparisons for the proposed SSL method. The results show that the proposed SSL method is superior to the fully-supervised DL and SVM for both tasks, especially when training data is scarce. This result indicates the proposed method is the superior choice for reading analysis tasks. These results are important for informing the design of automatic reading analysis platforms.

SeBioGraph: Semi-supervised Deep Learning for the Graph via Sustainable Knowledge Transfer.

Semi-supervised deep learning for the biomedical graph and advanced manufacturing graph is rapidly becoming an important topic in both academia and industry. Many existing types of research focus on semi-supervised link prediction and node classification, as well as the application of these methods in sustainable development and advanced manufacturing. To date, most manufacturing graph neural networks are mainly evaluated on social and information networks, which improve the quality of network representation y integrating neighbor node descriptions. However, previous methods have not yet been comprehensively studied on biomedical networks. Traditional techniques fail to achieve satisfying results, especially when labeled nodes are deficient in number. In this paper, a new semi-supervised deep learning method for the biomedical graph via sustainable knowledge transfer called SeBioGraph is proposed. In SeBioGraph, both node embedding and graph-specific prototype embedding are utilized as transferable metric space characterized. By incorporating prior knowledge learned from auxiliary graphs, SeBioGraph further promotes the performance of the target graph. Experimental results on the two-class node classification tasks and three-class link prediction tasks demonstrate that the SeBioGraph realizes state-of-the-art results. Finally, the method is thoroughly evaluated.

Acrophobia Quantified by EEG Based on CNN Incorporating Granger Causality.

The aim of this study is to quantify acrophobia and provide safety advices for high-altitude workers. Considering that acrophobia is a fuzzy quantity that cannot be accurately evaluated by conventional detection methods, we propose a comprehensive solution to quantify acrophobia. Specifically, this study simulates a virtual reality environment called High-altitude Plank Walking Challenge, which provides a safe and controlled experimental environment for subjects. Besides, a method named Granger Causality Convolutional Neural Network (GCCNN) combining convolutional neural network and Granger causality functional brain network is proposed to analyze the subjects' noninvasive scalp EEG signals. Here, the GCCNN method is used to distinguish the subjects with severe acrophobia, moderate acrophobia, and no acrophobia in a three-class classification task or no acrophobia and acrophobia in a two-class classification task. Compared with the mainstream methods, the GCCNN method achieves better classification performance, with an accuracy of 98.74% for the two-class classification task (no acrophobia versus acrophobia) and of 98.47% for the three-class classification task (no acrophobia versus moderate acrophobia versus severe acrophobia). Consequently, our proposed GCCNN method can provide more accurate quantitative results than the comparative methods, making it to be more competitive in further practical applications.

A novel multi-focus image fusion method for improving imaging systems by using cascade-forest model

Image fusion technology combines information from different source images of the same target and performs extremely effective information complementation, which is widely used for the transportation field, medicine field, and surveillance field. Specifically, due to the limitation of depth of field in imaging device, images cannot focus on all objects and miss partial details. To deal with this problem, an effective multi-focus image fusion method is proposed in this paper. We interpret the production of the focus map as a two-class classification task and solve this problem by using a method based on the cascade-forest model. Firstly, we extract the specific features from overlapping patches to represent the clarity level of source images. To obtain the focus map, feature vectors are fed into the pre-trained cascade-forest model. Then, we utilize consistency check to acquire the initial decision map. Afterward, guided image filtering is used for edge-reservation to refine the decision map. Finally, the result is obtained through pixel-wise weighted average strategy. Extensive experiments demonstrate that the proposed method achieves outstanding visual performance and excellent objective indicators.

Identification of tissue-specific tumor biomarker using different optimization algorithms.

Identification of differentially expressed genes, i.e., genes whose transcript abundance level differs across different biological or physiological conditions, was indeed a challenging task. However, the inception of transcriptome sequencing (RNA-seq) technology revolutionized the simultaneous measurement of the transcript abundance levels for thousands of genes. In this paper, such next-generation sequencing (NGS) data is used to identify biomarker signatures for several of the most common cancer types (bladder, colon, kidney, brain, liver, lung, prostate, skin, and thyroid) METHODS: Here, the problem is mapped into the comparison of optimization algorithms for selecting a set of genes that lead to the highest classification accuracy of a two-class classification task between healthy and tumor samples. As the optimization algorithms Artificial Bee Colony (ABC), Ant Colony Optimization, Differential Evolution, and Particle Swarm Optimization are chosen for this experiment. A standard statistical method called DESeq2 is used to select differentially expressed genes before being feed to the optimization algorithms. Classification of healthy and tumor samples is done by support vector machine RESULTS: Cancer-specific validation yields remarkably good results in terms of accuracy. Highest classification accuracy is achieved by the ABC algorithm for Brain lower grade glioma data is 99.10%. This validation is well supported by a statistical test, gene ontology enrichment analysis, and KEGG pathway enrichment analysis for each cancer biomarker signature CONCLUSION: The current study identified robust genes as biomarker signatures and these identified biomarkers might be helpful to accurately identify tumors of unknown origin.

Genome-wide analysis of NGS data to compile cancer-specific panels of miRNA biomarkers.

MicroRNAs are small non-coding RNAs that influence gene expression by binding to the 3’ UTR of target mRNAs in order to repress protein synthesis. Soon after discovery, microRNA dysregulation has been associated to several pathologies. In particular, they have often been reported as differentially expressed in healthy and tumor samples. This fact suggested that microRNAs are likely to be good candidate biomarkers for cancer diagnosis and personalized medicine. With the advent of Next-Generation Sequencing (NGS), measuring the expression level of the whole miRNAome at once is now routine. Yet, the collaborative effort of sharing data opens to the possibility of population analyses. This context motivated us to perform an in-silico study to distill cancer-specific panels of microRNAs that can serve as biomarkers. We observed that the problem of finding biomarkers can be modeled as a two-class classification task where, given the miRNAomes of a population of healthy and cancerous samples, we want to find the subset of microRNAs that leads to the highest classification accuracy. We fulfill this task leveraging on a sensible combination of data mining tools. In particular, we used: differential evolution for candidate selection, component analysis to preserve the relationships among miRNAs, and SVM for sample classification. We identified 10 cancer-specific panels whose classification accuracy is always higher than 92%. These panels have a very little overlap suggesting that miRNAs are not only predictive of the onset of cancer, but can be used for classification purposes as well. We experimentally validated the contribution of each of the employed tools to the selection of discriminating miRNAs. Moreover, we tested the significance of each panel for the corresponding cancer type. In particular, enrichment analysis showed that the selected miRNAs are involved in oncogenesis pathways, while survival analysis proved that miRNAs can be used to evaluate cancer severity. Summarizing: results demonstrated that our method is able to produce cancer-specific panels that are promising candidates for a subsequent in vitro validation.

Two-class 3D-CNN classifiers combination for video copy detection

3D-CNN is the latest CNN model used for video classification. However, the required amount of computation and training data for training 3D-CNN, especially for complex classification tasks with large video data, hinders the wide application of 3D-CNN. In this paper, inspired by the exclusion method in human’s judgment, a parallel 3D-CNN architecture is proposed to decompose the multi-class classification task using one 3D-CNN into the combination of multiple two-class classification tasks. 3D-CNN is used as a two-class classifier for each of the two-class classification tasks, and the difficulty and the data requirement on training such a 3D-CNN is reduced greatly comparing with the 3D-CNN for multi-class classification. In addition, the combination of two-class classifiers provides the ability of recognizing unknown class to the proposed 3D-CNN model. The feasibility of this proposed 3D-CNN model is verified via its application on video copy detection on the CC_WEB_VIDEO dataset. The experimental results show the potentiality of the proposed parallel two-class 3D-CNN model in video classification.

The correctness problem: evaluating the ordering of binary features in rankings

In machine learning, feature ranking (FR) algorithms are used to rank features by relevance to the class variable. FR algorithms are mostly investigated for the feature selection problem and less studied for the problem of ranking. This paper focuses on the latter. A question asked about the problem of ranking given in the terminology of FR is: as different FR criteria estimate the relationship between a feature and the class variable differently on a given data, can we determine which criterion better captures the "true" feature-to-class relationship and thus generates the most "correct" order of individual features? This is termed as the "correctness" problem. It requires a reference ordering against which the ranks assigned to features by a FR algorithm are directly compared. The reference ranking is generally unknown for real-life data. In this paper, we show through theoretical and empirical analysis that for two-class classification tasks represented with binary data, the ordering of binary features based on their individual predictive powers can be used as a benchmark. Thus, allowing us to test how correct is the ordering of a FR algorithm. Based on these ideas, an evaluation method termed as FR evaluation strategy (FRES) is proposed. Rankings of three different FR criteria (relief, mutual information, and the diff-criterion) are investigated on five artificially generated and four real-life binary data sets. The results indicate that FRES works equally good for synthetic and real-life data and the diff-criterion generates the most correct orderings for binary data.