Feature selection refers to the problem of finding the optimal subset of features by removing irrelevant and redundant features to improve classification accuracy. The determination of the most effective distance measures to evaluate the relevance and redundancy of features has not been investigated precisely to date. Moreover, the relation between relevancy and redundancy is still uncertain. This paper presents a novel relevancy-redundancy measurement based on distance applying the idea of the mRMR criteria to an unsupervised method. In addition, a supervised method is proposed, in which the features are ranked in terms of the distance between each pair of samples in different classes of the feature vector. Then an ensemble of the proposed supervised and unsupervised methods is applied to choose the most relevant features subset. This study investigates and compares the effects of 24 distance measures selected from five major families of distance functions on the performance of the proposed feature selection methods. The highest-ranked features are selected using an empirically achieved threshold. To evaluate the selected features, three classifiers, i.e., Decision Tree, Support Vector Machine and Naive Bayes were applied to biomedical datasets representing binary problems from the UCI data repository. The experimental results demonstrate the superiority of the proposed methods over the state-of-the-art and also classical feature selection ones in terms of improving stability, classification accuracy, Recall (Sensitivity), Precision, F-measure, and Specificity.