Random Subset Of Features Research Articles

Short-Term Load Forecasting Based on Optimized Random Forest and Optimal Feature Selection

Short-term load forecasting (STLF) plays a vital role in ensuring the safe, efficient, and economical operation of power systems. Accurate load forecasting provides numerous benefits for power suppliers, such as cost reduction, increased reliability, and informed decision-making. However, STLF is a complex task due to various factors, including non-linear trends, multiple seasonality, variable variance, and significant random interruptions in electricity demand time series. To address these challenges, advanced techniques and models are required. This study focuses on the development of an efficient short-term power load forecasting model using the random forest (RF) algorithm. RF combines regression trees through bagging and random subspace techniques to improve prediction accuracy and reduce model variability. The algorithm constructs a forest of trees using bootstrap samples and selects random feature subsets at each node to enhance diversity. Hyperparameters such as the number of trees, minimum sample leaf size, and maximum features for each split are tuned to optimize forecasting results. The proposed model was tested using historical hourly load data from four transformer substations supplying different campus areas of the University of Beira Interior, Portugal. The training data were from January 2018 to December 2021, while the data from 2022 were used for testing. The results demonstrate the effectiveness of the RF model in forecasting short-term hourly and one day ahead load and its potential to enhance decision-making processes in smart grid operations.

Utilization of Stockwell Transform and Random Forest Algorithm for Efficient Detection and Classification of Power Quality Disturbances

Power quality disturbances (PQDs) can lead to significant operational and financial losses in power systems. Accurate detection and classification of PQDs are essential for maintaining power quality and preventing power system failures. This research article introduces an innovative approach for the precise detection and classification of single- and multiple-state power quality disturbances (PQDs) using the Stockwell transform (ST) and a random forest classifier. To create realistic PQD signals, seventeen distinct classes are generated in accordance with IEEE Standard 1159, employing mathematical equations implemented in MATLAB software. The ST is employed to extract relevant features from the PQD signals, which are subsequently utilized as input for the random forest classifier. The classifier employs bootstrapping sampling to generate multiple training sets from the original dataset. Each training set is used to construct a decision tree by recursively partitioning the data based on significant features. To mitigate overfitting and enhance robustness, a random subset of features is selected at each node of the decision tree, thereby reducing tree correlation. The performance of the random forest classifier is compared with other widely utilized machine learning classifiers. The results exhibit the efficacy of the proposed approach in accurately detecting and classifying PQ events, highlighting its superiority over alternative methods.

A [formula omitted] nearest neighbour ensemble via extended neighbourhood rule and feature subsets

kNN based ensemble methods minimise the effect of outliers by identifying a set of data points in the given feature space that are nearest to an unseen observation in order to predict its response by using majority voting. The ordinary ensembles based on kNN find out the k nearest observations in a region (bounded by a sphere) based on a predefined value of k. This scenario, however, might not work in situations where the test observation follows the pattern of the closest data points with the same class that lie on a certain path not contained in the given sphere. This paper proposes a k nearest neighbour ensemble where the neighbours are determined in k steps. Starting from the first nearest observation of the test point, the algorithm identifies a single observation that is closest to the observation at the previous step. At each base learner in the ensemble, this search is extended to k steps on a random bootstrap sample with a random subset of features selected from the feature space. The final predicted class of the test point is determined by using a majority vote in the predicted classes given by all base models. This new ensemble method is applied on 20 benchmark datasets and compared with other classical methods, including kNN based models, in terms of classification accuracy, kappa and Brier score as performance metrics. Boxplots are also utilised to illustrate the difference in the results given by the proposed and other state-of-the-art methods. The proposed method outperformed the considered classical methods in the majority of cases. The proposed method is further assessed through a detailed simulation study.

Optimal model selection for k-nearest neighbours ensemble via sub-bagging and sub-sampling with feature weighting

This paper proposes two novel approaches based on feature weighting and model selection for building more accurate kNN ensembles. The first approach identifies the nearest observations using a feature weighting scheme concerning the response variable via support vectors. A randomly selected subset of features is used for the feature weighting and model construction. After building a sufficiently large number of base models on bootstrap samples, a subset of the models is selected based on out-of-bag prediction error for the final ensemble. The second approach builds base learners build on random subsamples instead of bootstrap samples with a random subset of features. The method uses feature weighting while building the models. The remaining observations from each sample are used to assess the corresponding base learner and select a subset of the models for the final ensemble. The suggested ensemble methods are assessed on 12 benchmark datasets against other classical methods, including kNN-based models. The analyses reveal that the proposed methods are often better than the others.

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Integrated data-driven modeling and experimental optimization of granular hydrogel matrices

Supervised models to predict the Stunting in East Aceh

Nowadays, Undernutrition is the main cause of child death in developing countries. There are many people and organizations try to mitigate or minimize case of child death. Thus, this paper aimed to has excellent method to handle undernutrition case by exploring the efficacy of machine learning (ML) approaches to predict Stunting in East Aceh administrative zones of Indonesia and to identify the most important predictors. The study employed ML techniques using retrospective cross-sectional survey data from East Aceh, a national-representative data is collected from government by using 2019 about stunting data. We explored Random forest commonly used ML algorithms. Random Forest (RF) as an extension of bagging that in addition for taking random sample of data and also uses random subset of features which mitigates over fitting. Our results showed that the considered machine learning classification algorithms by random forest can effectively predict the stunting status in East Aceh administrative zones. Persistent stunting status was found in the east part of Aceh. The identification of high-risk zones can provide more useful information and data to decision-makers for trying to reduce child undernutrition.

LGBM-CBFS: A Heuristic Feature Sampling Method Based on Tree Ensembles

Gradient boosting decision tree (GBDT) is widely used because of its state-of-art performance in academia, industry, and data science competitions. The efficiency of the model is limited by the overwhelming training cost with the surge of data. A common solution is data reduction by sampling on training data. Current popular implementations of GBDT such as XGBoost and LightGBM both supports cut the search space by using only a random subset of features without any prior knowledge, which is ineffective and may lead the model fail to converge when sampling on a high-dimensional feature space with a small sampling rate assigned. To mitigate this problem, we proposed a heuristic sampling algorithm LGBM-CBFS, which samples features based on an available prior knowledge named “importance scores” to improve the performance and the effectiveness of GBDT. Experimental results indicate that LGBM-CBFS obtains a higher level of model accuracy than uniform sampling without introducing unacceptable time cost in the sparse high-dimensional scenarios.

Multi‐view rank‐based random forest: A new algorithm for prediction in eSports

AbstractThe main problem associated with the random forest (RF) algorithm is its application of random feature subset selection technique over a single vector. In this technique, the irrelevant or redundant features in a single‐view data have equal chances with the important features to be selected for training a classifier, leading to misclassification. To overcome this problem, this article proposes a novel algorithm, called multi‐view rank‐based random forest (MVRRF). The proposed algorithm builds a set of decision trees from multi‐view data by using a rank‐based feature selection strategy. The main advantages of our method are that (i) it extends the RF algorithm for multi‐view learning, and (ii) it reduces the chances of irrelevant and redundant feature selection, and thus it usually improves the accuracy, generalizability and robustness of the classification models. The aim of our study is to predict the match result of the game League of Legends in electronic sports (eSports). Thus, the eSports teams can define trustworthy strategies through important features. The proposed method can be successfully used in other fields as well as eSports. The experiments that were conducted on a publicly available eSports dataset show that the proposed MVRRF algorithm (93.32%) outperforms the standard RF algorithm (86.38%) on multi‐view data in terms of accuracy. Furthermore, the experimental results also show that our method achieved higher performance than the methods tested in the state of the art studies on the same dataset.

A k-Nearest Neighbours Based Ensemble via Optimal Model Selection for Regression

Ensemble methods based on k-NN models minimise the effect of outliers in a training dataset by searching groups of the k closest data points to estimate the response of an unseen observation. However, traditional k-NN based ensemble methods use the arithmetic mean of the training points' responses for estimation which has several weaknesses. Traditional k-NN based models are also adversely affected by the presence of non-informative features in the data. This paper suggests a novel ensemble procedure consisting of a class of base k-NN models each constructed on a bootstrap sample drawn from the training dataset with a random subset of features. In the k nearest neighbours determined by each k-NN model, stepwise regression is fitted to predict the test point. The final estimate of the target observation is then obtained by averaging the estimates from all the models in the ensemble. The proposed method is compared with some other state-of-the-art procedures on 16 benchmark datasets in terms of coefficient of determination (R 2 ), Pearson's product-moment correlation coefficient (r), mean square predicted error (MSPE), root mean squared error (RMSE) and mean absolute error (MAE) as performance metrics. Furthermore, boxplots of the results are also constructed. The suggested ensemble procedure has outperformed the other procedures on almost all the datasets. The efficacy of the method has also been verified by assessing the proposed method in comparison with the other methods by adding non-informative features to the datasets considered. The results reveal that the proposed method is more robust to the issue of non-informative features in the data as compared to the rest of the methods.

Multiclass classification of motor imagery EEG signals using ensemble classifiers & cross-correlation

To communicate without any muscle movement and purely based on brain signal has been the goal of Brain computer interfacing (BCI). Recent BCI based studies reported more and more accurate detection of brain states. This paper proposes a study that detects EEG signal belonging todifferent imaginary motor activities (Right leg, right hand, left leg and left hand). The Electroencephalogram (EEG) signal has been conditioned by band pass filter (BPF) to improve signal to noise ratio (SNR). The proposed method is based on similarity between signals to extract features. For measuring the similarity between signals, Cross correlation (CC) is used. An ensemble set of five classifiers (Linear Discriminant Analysis (LDA), K-Nearest Neighbors (KNN), Support Vector Machine (SVM), Naïve Bayes (NB) and Binary Decision Tree) was used collectively. As the similarity measurement was binary in nature, one versus rest (OVR) approach was used for multi class classification. Random subset of features was used to train the ensemble of classifiers. The classification label was obtained by using majority voting. An average accuracy of 89.57% was observed among all 10 subjects.

Spatial rich model steganalysis feature normalization on random feature-subsets

Spatial rich model (SRM) steganalysis feature is formed by high-order statistics collected from image noise residuals. These statistics are simply rescaled before machine learning. It is noted that SRM features of different cover images are very different. In this paper, we propose a feature normalization method based on random feature-subsets (NRS) for SRM. We randomly draw feature-subsets from SRM feature. Then these feature-subsets are normalized by using per-sample rescaling method to make the feature-subsets of different images have the same 1-norm (the sum of all elements). The proposed NRS method can adjust the feature distribution and increase the feature diversity. The normalized feature-subsets can achieve better detection performance and also can be used as a useful complement for the existing steganalysis features. Experimental results show that: (1) a small amount of normalized feature-subset supplement can obviously improve the detection performance of SRM feature; (2) under the same dimensionality, the proposed NRS version feature can achieve a better detection accuracy than that of original feature; (3) the proposed NRS method is applicable to projections of spatial rich model feature; (4) compared with steganalysis feature extraction, the computational time of NRS can be negligible.

Ensemble of One-Class Classifiers for Personal Risk Detection Based on Wearable Sensor Data.

This study introduces the One-Class K-means with Randomly-projected features Algorithm (OCKRA). OCKRA is an ensemble of one-class classifiers built over multiple projections of a dataset according to random feature subsets. Algorithms found in the literature spread over a wide range of applications where ensembles of one-class classifiers have been satisfactorily applied; however, none is oriented to the area under our study: personal risk detection. OCKRA has been designed with the aim of improving the detection performance in the problem posed by the Personal RIsk DEtection(PRIDE) dataset. PRIDE was built based on 23 test subjects, where the data for each user were captured using a set of sensors embedded in a wearable band. The performance of OCKRA was compared against support vector machine and three versions of the Parzen window classifier. On average, experimental results show that OCKRA outperformed the other classifiers for at least 0.53% of the area under the curve (AUC). In addition, OCKRA achieved an AUC above 90% for more than 57% of the users.

부도예측을 위한 KNN 앙상블 모형의 동시 최적화

Bankruptcy involves considerable costs, so it can have significant effects on a country`s economy. Thus, bankruptcy prediction is an important issue. Over the past several decades, many researchers have addressed topics associated with bankruptcy prediction. Early research on bankruptcy prediction employed conventional statistical methods such as univariate analysis, discriminant analysis, multiple regression, and logistic regression. Later on, many studies began utilizing artificial intelligence techniques such as inductive learning, neural networks, and case-based reasoning. Currently, ensemble models are being utilized to enhance the accuracy of bankruptcy prediction. Ensemble classification involves combining multiple classifiers to obtain more accurate predictions than those obtained using individual models. Ensemble learning techniques are known to be very useful for improving the generalization ability of the classifier. Base classifiers in the ensemble must be as accurate and diverse as possible in order to enhance the generalization ability of an ensemble model. Commonly used methods for constructing ensemble classifiers include bagging, boosting, and random subspace. The random subspace method selects a random feature subset for each classifier from the original feature space to diversify the base classifiers of an ensemble. Each ensemble member is trained by a randomly chosen feature subspace from the original feature set, and predictions from each ensemble member are combined by an aggregation method. The k-nearest neighbors (KNN) classifier is robust with respect to variations in the dataset but is very sensitive to changes in the feature space. For this reason, KNN is a good classifier for the random subspace method. The KNN random subspace ensemble model has been shown to be very effective for improving an individual KNN model. The k parameter of KNN base classifiers and selected feature subsets for base classifiers play an important role in determining the performance of the KNN ensemble model. However, few studies have focused on optimizing the k parameter and feature subsets of base classifiers in the ensemble. This study proposed a new ensemble method that improves upon the performance KNN ensemble model by optimizing both k parameters and feature subsets of base classifiers. A genetic algorithm was used to optimize the KNN ensemble model and improve the prediction accuracy of the ensemble model. The proposed model was applied to a bankruptcy prediction problem by using a real dataset from Korean companies. The research data included 1800 externally non-audited firms that filed for bankruptcy (900 cases) or non-bankruptcy (900 cases). Initially, the dataset consisted of 134 financial ratios. Prior to the experiments, 75 financial ratios were selected based on an independent sample t-test of each financial ratio as an input variable and bankruptcy or non-bankruptcy as an output variable. Of these, 24 financial ratios were selected by using a logistic regression backward feature selection method. The complete dataset was separated into two parts: training and validation. The training dataset was further divided into two portions: one for the training model and the other to avoid overfitting. The prediction accuracy against this dataset was used to determine the fitness value in order to avoid overfitting. The validation dataset was used to evaluate the effectiveness of the final model. A 10-fold cross-validation was implemented to compare the performances of the proposed model and other models. To evaluate the effectiveness of the proposed model, the classification accuracy of the proposed model was compared with that of other models. The Q-statistic values and average classification accuracies of base classifiers were investigated. The experimental results showed that the proposed model outperformed other models, such as the single model and random subspace ensemble model.

Resampling-based similarity measures for high-dimensional data.

An important issue in classification is the assessment of sample similarity. This is nontrivial in high-dimensional or megavariate datasets--datasets that are comprised of simultaneous measurements on thousands of features, many of which carry little or no information regarding consistent sample differences. Conventional similarity measures do not work particularly well for such data. As an alternative, we propose a distance measure that is based on a refiltering process: at each step of the process a random subset of features is selected and a cluster analysis is performed using only this subset; the relative frequency with which a pair of samples clusters together across several such random subsets forms the similarity measure. The features chosen at any step may be completely random or enriched by awarding the more informative features a higher chance of selection; this enrichment turns out to be particularly effective. We use actual datasets from the burgeoning genomics literature to demonstrate the superior performance of this similarity measure, especially the enriched form of the similarity measure, compared to more conventional measures such as Euclidean distance or correlation, or, if the data are categorical, Hamming distance.

Simultaneous feature and parameter selection using multiobjective optimization: application to named entity recognition

In this paper, we propose an efficient algorithm based on the concept of multiobjective optimization (MOO) for performing feature selection and parameter optimization of any machine learning technique. Feature and parameter combinations have significant effect to the accuracy of the classifier. We perform feature selection and parameter optimization for four different classifiers, namely conditional random field, support vector machine, memory based learner and maximum entropy. The proposed algorithms are evaluated for solving the problems of named entity recognition, an important component in many text processing applications. Currently we experiment with four different languages, namely Bengali, Hindi, Telugu and English. At first the proposed MOO based technique is used to determine the appropriate features and parameters. For each of the classifiers, the algorithm produces a set of solutions on the final Pareto optimal front. Each solution represents a classifier with a particular feature and parameter combination. All these solutions are thereafter combined using a MOO based classifier ensemble technique. Evaluation results show that the proposed approach attains the F-measure (harmonic mean of recall and precision) values of 90.48, 90.44, 78.71 and 88.68 % for Bengali, Hindi, Telugu and English, respectively. We also show that for all the experimental settings the proposed feature and parameter optimization technique performs reasonably better than the baseline systems, developed with random feature subsets. Comparisons with the existing works also show the efficacy of our proposed algorithm.

Exploring the diversity in cluster ensemble generation: Random sampling and random projection

Cluster ensemble first generates a large library of different clustering solutions and then combines them into a more accurate consensus clustering. It is commonly accepted that for cluster ensemble to work well the member partitions should be different from each other, and meanwhile the quality of each partition should remain at an acceptable level. Many different strategies have been used to generate different base partitions for cluster ensemble. Similar to ensemble classification, many studies have been focusing on generating different partitions of the original dataset, i.e., clustering on different subsets (e.g., obtained using random sampling) or clustering in different feature spaces (e.g., obtained using random projection). However, little attention has been paid to the diversity and quality of the partitions generated using these two approaches. In this paper, we propose a novel cluster generation method based on random sampling, which uses the nearest neighbor method to fill the category information of the missing samples (abbreviated as RS-NN). We evaluate its performance in comparison with k-means ensemble, a typical random projection method (Random Feature Subset, abbreviated as FS), and another random sampling method (Random Sampling based on Nearest Centroid, abbreviated as RS-NC). Experimental results indicate that the FS method always generates more diverse partitions while RS-NC method generates high-quality partitions. Our proposed method, RS-NN, generates base partitions with a good balance between the quality and the diversity and achieves significant improvement over alternative methods. Furthermore, to introduce more diversity, we propose a dual random sampling method which combines RS-NN and FS methods. The proposed method can achieve higher diversity with good quality on most datasets.

Ensemble of Nested Dichotomies 기법을 이용한 스마트폰 가속도 센서 데이터 기반의 동작 인지

Ensemble of Nested Dichotomies 기법을 이용한 스마트폰 가속도 센서 데이터 기반의 동작 인지

Investigating a novel GA-based feature selection method using improved KNN classifiers

This paper presents a novel scheme to select a subset of features from a dataset. We apply genetic algorithm (GA) with a random small subset of features. The GA explores stochastically a better subset of features using various combinations of lengths and features over a number of generations. The classification accuracy due to different classifiers in presence of these subsets of features is taken as the performance criteria (objective function) of GA. The proposed scheme is tested on a few UCI datasets. The performances of the KNN, informative KNN (local LI-KNN and global GI-KNN), and LI-KNN with boosting in presence of all features and those in presence of only selected subset of features are compared with reported results. With extensive simulation study, it is observed that the proposed scheme produces a reasonably good accuracy with a reduced subset of features in these datasets.

Learn ++.MF: A random subspace approach for the missing feature problem

We introduce Learn ++.MF, an ensemble-of-classifiers based algorithm that employs random subspace selection to address the missing feature problem in supervised classification. Unlike most established approaches, Learn ++.MF does not replace missing values with estimated ones, and hence does not need specific assumptions on the underlying data distribution. Instead, it trains an ensemble of classifiers, each on a random subset of the available features. Instances with missing values are classified by the majority voting of those classifiers whose training data did not include the missing features. We show that Learn ++.MF can accommodate substantial amount of missing data, and with only gradual decline in performance as the amount of missing data increases. We also analyze the effect of the cardinality of the random feature subsets, and the ensemble size on algorithm performance. Finally, we discuss the conditions under which the proposed approach is most effective.

Ensemble of Linear Models for Predicting Drug Properties

We propose a new classification method for the prediction of drug properties, called random feature subset boosting for linear discriminant analysis (LDA). The main novelty of this method is the ability to overcome the problems with constructing ensembles of linear discriminant models based on generalized eigenvectors of covariance matrices. Such linear models are popular in building classification-based structure-activity relationships. The introduction of ensembles of LDA models allows for an analysis of more complex problems than by using single LDA, for example, those involving multiple mechanisms of action. Using four data sets, we show experimentally that the method is competitive with other recently studied chemoinformatic methods, including support vector machines and models based on decision trees. We present an easy scheme for interpreting the model despite its apparent sophistication. We also outline theoretical evidence as to why, contrary to the conventional AdaBoost ensemble algorithm, this method is able to increase the accuracy of LDA models.