Feature Engineering Techniques Research Articles

Feature engineering and long short-term memory for energy use of appliances prediction

Electric energy consumption in a residential household is one of the key factors that affect the overall national electricity demand. Household appliances are one of the most electricity consumers in a residential household. Therefore, it is crucial to make a proper prediction for the electricity consumption of these appliances. This research implemented feature engineering technique and long short-term memory (LSTM) as a model predictor. Principal component analysis (PCA) was implemented as a feature extractor by reducing the final 62 features to 25 principal components for the LSTM inputs. Based on the experiments, the two-layered LSTM model (composed by 25 and 20 neurons for the first and second later respectively) with lookback number of 3 found to give the best performance with the error rates of 62.013 and 26.982 for RMSE and MAE, respectively.

Elderly Fall Detection With Vital Signs Monitoring Using CW Doppler Radar

Falling is the main cause of disability and fatality of elderly. In this work we used a 24 GHz continuous-wave Doppler radar to develop a low price and a highly accurate fall detection system aiming at observing indoor human activities and detecting fall accidents, thus limiting the consequences of undiscovered falls. A radar sensor was selected due to its capability of tracking human motions, passing through obstacles, and not being affected by light conditions. The sampled signals from the sensor were subjected to different feature engineering and machine learning techniques in order to determine the most characteristic features. Consequently, we were able to extract nontraditional radar features. Moreover, to improve the systems responsiveness and to lessen the hardware constraints, we added an additional pre-processing step to detect major physical activities with O(N) complexity. As part of fall detection, we also implemented vital signs monitoring to reduce false positives and to alert concerned authorities if necessary. Experimental results show that our proposed system has 90% recall rate for fall detection and 97.7% and 95.3% accuracy rates for respiration and heartbeat detection respectively.

Automated detection of acute respiratory distress syndrome from chest X-Rays using Directionality Measure and deep learning features

Acute respiratory distress syndrome (ARDS) is a life-threatening lung injury with global prevalence and high mortality. Chest x-rays (CXR) are critical in the early diagnosis and treatment of ARDS. However, imaging findings may not result in proper identification of ARDS due to a number of reasons, including nonspecific appearance of radiological features, ambiguity in a patient's case due to the pathological stage of the disease, and poor inter-rater reliability from interpretations of CXRs by multiple clinical experts. This study demonstrates the potential capability of methodologies in artificial intelligence, machine learning, and image processing to overcome these challenges and quantitatively assess CXRs for presence of ARDS. We propose and describe Directionality Measure, a novel feature engineering technique used to capture the “cloud-like” appearance of diffuse alveolar damage as a mathematical concept. This study also examines the effectiveness of using an off-the-shelf, pretrained deep learning model as a feature extractor in addition to standard features extracted from the histogram and gray-level co-occurrence matrix (GLCM). Data was collected from hospitalized patients at Michigan Medicine's intensive care unit and the cohort's inclusion criteria was specifically designed to be representative of patients at risk of developing ARDS. Multiple machine learning models were used to evaluate these features with 5-fold cross-validation and the final performance was reported on a hold-out, temporally distinct test set. With AdaBoost, Directionality Measure achieved an accuracy of 78% and AUC of 74% – outperforming classification results using features from the histogram (75% accuracy and 73% AUC), GLCM (76% accuracy and 73% AUC), and ResNet-50 (77% accuracy and 73% AUC). Further experimental results demonstrated that using all feature sets in combination achieved the best overall performance, yielding an accuracy of 83% and AUC of 79% with AdaBoost. These results demonstrate the potential capability of using the proposed methodologies to complement current clinical analysis for detection of ARDS from CXRs.

Machine Learning for Detecting Data Exfiltration

Context : Research at the intersection of cybersecurity, Machine Learning (ML), and Software Engineering (SE) has recently taken significant steps in proposing countermeasures for detecting sophisticated data exfiltration attacks. It is important to systematically review and synthesize the ML-based data exfiltration countermeasures for building a body of knowledge on this important topic. Objective : This article aims at systematically reviewing ML-based data exfiltration countermeasures to identify and classify ML approaches, feature engineering techniques, evaluation datasets, and performance metrics used for these countermeasures. This review also aims at identifying gaps in research on ML-based data exfiltration countermeasures. Method : We used Systematic Literature Review (SLR) method to select and review 92 papers. Results : The review has enabled us to: (a) classify the ML approaches used in the countermeasures into data-driven, and behavior-driven approaches; (b) categorize features into six types: behavioral, content-based, statistical, syntactical, spatial, and temporal; (c) classify the evaluation datasets into simulated, synthesized, and real datasets; and (d) identify 11 performance measures used by these studies. Conclusion : We conclude that: (i) The integration of data-driven and behavior-driven approaches should be explored; (ii) There is a need of developing high quality and large size evaluation datasets; (iii) Incremental ML model training should be incorporated in countermeasures; (iv) Resilience to adversarial learning should be considered and explored during the development of countermeasures to avoid poisoning attacks; and (v) The use of automated feature engineering should be encouraged for efficiently detecting data exfiltration attacks.

Categorizing Natural Language-Based Customer Satisfaction: An Implementation Method Using Support Vector Machine and Long Short-Term Memory Neural Network

Analyzing natural language-based Customer Satisfaction (CS) is a tedious process. This issue is practically true if one is to manually categorize large datasets. Fortunately, the advent of supervised machine learning techniques has paved the way toward the design of efficient categorization systems used for CS. This paper presents the feasibility of designing a text categorization model using two popular and robust algorithms – the Support Vector Machine (SVM) and Long Short-Term Memory (LSTM) Neural Network, in order to automatically categorize complaints, suggestions, feedbacks, and commendations. The study found that, in terms of training accuracy, SVM has best rating of 98.63% while LSTM has best rating of 99.32%. Such results mean that both SVM and LSTM algorithms are at par with each other in terms of training accuracy, but SVM is significantly faster than LSTM by approximately 35.47s. The training performance results of both algorithms are attributed on the limitations of the dataset size, high-dimensionality of both English and Tagalog languages, and applicability of the feature engineering techniques used. Interestingly, based on the results of actual implementation, both algorithms are found to be 100% effective in accurately predicting the correct CS categories. Hence, the extent of preference between the two algorithms boils down on the available dataset and the skill in optimizing these algorithms through feature engineering techniques and in implementing them toward actual text categorization applications.

Machine Learning and feature engineering-based study into sarcasm and irony classification with application to cyberbullying detection

Irony and sarcasm detection is considered a complex task in Natural Language Processing. This paper set out to explore the sarcasm and irony on Twitter, using Machine Learning and Feature Engineering techniques. First we review and clarify the definition of irony and sarcasm by discussing various studies focusing on the terms. Next the first experiment is conducted comparing between various types of classification methods including some popular classifiers for text classification task. For the second experiment, different types of data preprocessing methods were compared and analyzed. Finally, the relationship between irony, sarcasm, and cyberbullying are discussed. The results are interesting as we observed high similarity between them.

Cardiovascular risk assessment using data mining inferencing and feature engineering techniques

With the frequent decline in people’s health due to the hectic lifestyle, increased levels of workload and intake of fast food, there has been an unfortunate growth in the number of patients suffering from cardiovascular diseases each year. Around the world, millions of people die each year due to cardiovascular diseases. While the statistics are eye-opening, with the vast amount of data about heart patients in our hands, we can save millions by detecting the risk at an early stage. With the recent advances in soft computing and fuzzy logic, various algorithmic approaches are employed to tackle the issue of cardiovascular risk assessment through machine learning. Using some of the algorithms of machine learning like Logistic Regression (LR), Naive Bayes (NB), Support vector machine (SVM), and Decision tree (DT), Random Forest (RF) and K-Nearest Neighbours (KNN) classifiers, a model can be built to predict the risk accurately. In this paper, we have analysed each of the above methods normally and through feature engineering techniques like transformation through Principal Component Axes and considering different train-test folds to find the best performing model, which was found to be KNN in terms of all metrics and Logistic Regression in terms of accuracy.

A novel custom ensemble learning model for an improved reservoir permeability and water saturation prediction

With the advances of technology, many new well logs have been acquired over the past decade that carries vital information about the reservoir and subsurface layers. Thus, identifying the most relevant data that can improve the determination and prediction of petrophysical parameters has become very challenging. There has been an increase in the application of machine learning models that can accurately determine the petrophysical parameters of reservoirs, but further studies are still in demand. In this study, enhanced data analytics were used together with the visualisation techniques to pre-process the wireline logs acquired from the Volve field in the North Sea.Descriptive statistical methods were used to understand the relationship between the variables (input and output parameters), followed by applying the Extreme Gradient Boosting (XGBoost) regression model to predict the reservoir permeability and water saturation. A new ensemble model of Random Forest and Lasso Regularisation with an enhanced feature engineering technique was then proposed to improve the accuracy of the results. It appeared that the proposed ensemble model has a better performance than the traditional XGBoost and the hybrid PCA-XGBoost models in terms of precision, consistency and accuracy. The immense potential of ensemble modelling to enhance reservoir characterisation has been demonstrated by the success of this research.

Predicting stock returns of Tehran exchange using LSTM neural network and feature engineering technique

Prediction is defined as the expression of events that will occur in the future, before they occur, based on scientific and logical principles and rules. Due to the importance of financial markets for economic activists, prediction in this field has received much attention from scholars. Prediction of the stock market, as one of the largest financial markets can be very profitable to the predictors. The dynamic and complexity of the market has added to its appeal to researchers. To date, many researchers have reported good returns for prediction in this market using neural network methods. In this paper, we attempted to obtain better results on Tehran Stock Exchange by using their findings and by applying the Long Short-Term Memory (LSTM) deep neural network. In the area of feature engineering, we have tried to reduce the number of features using AutoEncoder-based feature selection to improve stock returns and reduce prediction error. To evaluate the proposed method, a return measure that is closer to the real world of stock trading was used. Experimental results showed that using the proposed method yielded a better output with a lower error mean.

Passenger data analysis of Titanic using machine learning approach in the context of chances of surviving the disaster

Titanic disaster occurred about 100 years back but still it attracts the researchers to understand and study that how some passengers survived and others perished. In this work, the characteristics of the passengers will be identified and the relationship of survival chance from the disaster is found. Feature engineering techniques will be performed, where the alphabetic values will be changed to numeric values, the family size will be calculated. Also, we will extract the title from the name, and deck label from ticket number. Classification is done using Decision tree machine learning classification algorithm using two classes which are survived and not survived. R programming has been used for its implementation. Clustering is performed using KMeans machine learning algorithm. Its implementation has been done using Python programming.

Diagnosis of COVID-19 using artificial intelligence based model

Scientists have shown that early diagnosis of COVID-19 and self-isolation can help curb the spread of the virus. Hence, there is a need to develop methods for early and fast diagnosis of the disease. This research focuses on the development and use of an AI based model that would help medical professionals in easy and fast detection of COVID-19 that can be found in X-ray images, CT scan images, and patient symptoms. The model would then be deployed to the web for easy accessibility. To increase the confident level of decision made by the model, different data augmentation technique was deployed to create variance to the dataset and thereby increasing the accuracy and validation of the model. This project proposes the use of Convolutional Neural Network for classification purpose of both the X-ray image of the Lung of a positive and negative pneumonia patient. Further processes were done with CT scan images of both positive and negative COVID-19 patients. This eliminated biasness of the model. Lastly, symptoms of the disease were added to the model as conditional statement convolutional neural network model. The model was able to reach an overall accuracy of 95% for Pneumonia X-ray image and 89.65% for CT scan image of unseen data (test/evaluation). Artificial intelligence based model is therefore, encouraged for better, easier and more accurate diagnosis of COVID-19. Also, the accuracy of the model can further be improved by certain feature engineering technique and adding more complex deep neural network technique known as ensembles.

Predicting the Risk of Alcohol Use Disorder Using Machine Learning: A Systematic Literature Review

The number of deaths caused by alcohol-related diseases may be reduced by predicting alcohol use disorder (AUD). Many researchers have worked on AUD prediction using machine learning (ML) techniques. However, to the best of our knowledge, there is a lack of a comprehensive systematic literature review (SLR) that summarizes the existing studies on AUD prediction using ML in the last ten years. To address this knowledge gap, this article provides an SLR of academic articles on AUD prediction using ML techniques dated from January 2010 to July 2021. This SLR highlights technical decision analysis related to five aspects: data collection site, characteristics, and type of dataset; data sampling and data pre-processing techniques; feature types and feature engineering techniques; and characteristics of ML techniques and evaluation metrics. Six bibliographic databases were searched, and the identified studies were rigorously reviewed based on the above five aspects. In the selected studies, public datasets were not used very often for AUD prediction. Given that, the current paper identified two different types of data collection sites for review. Imbalanced class distribution in datasets was the primary focus of the pre-processing and sampling steps. Various features, including demographics, family history, drinking behaviour, and electronic health records, were introduced as the more widely used AUD prediction features. The filter, wrapper, and embedded methods were identified as the primary feature selection methods. Support vector machine was the most widely employed algorithm for predicting AUD; however, the lack of deep neural network techniques is evident in this field. Moreover, considering gender disparities, early detection of AUD, and identifying trajectories towards AUD are suggested for future work. For the purpose of evaluating the performance of the prediction approaches, most studies considered the overall accuracy and the area under the receiver operating characteristic curve. Nevertheless, external validation was not performed in any of the selected studies. This paper also discusses challenges and open issues of AUD prediction for future research. This SLR represents a valuable resource for scholars investigating the prediction of AUD.

JONNEE: Joint Network Nodes and Edges Embedding

Recently, graph embedding models significantly improved the quality of graph machine learning tasks, such as node classification and link prediction. In this work, we propose a model called JONNEE (JOint Network Nodes and Edges Embedding), which learns node and edge embeddings under self-supervision via joint constraints in a given graph and its edge-to-vertex dual representation as a Line graph. The model uses two graph autoencoders with additional structural feature engineering and several regularization techniques to train for an adjacency matrix reconstruction task in an unsupervised setting. Experimental results show that our model performs on par with state-of-the-art undirected attribute graph embedding models and requires less number of epochs to achieve the same quality due to Line graph self-supervision under a unified embedding framework.

Feature Engineering Techniques to improve Identification Accuracy for Offline Signature Case-Bases

Handwritten signatures have been widely acclaimed for personal identification viability in educated human society. But, the astronomical growth of population in recent years warrant developing mechanized systems to remove the tedium and bias associated with manual checking. Here the proposed system, performing identification with Nearest Neighbor matching between offline signature images collected temporally. The raw images and their extracted features are preserved using Case Based Reasoning and Feature Engineering principles. Image patterns are captured through standard global and local features, along with some profitable indigenously developed features. Outlier feature values, on detection, are automatically replaced by their nearest statistically determined limit values. Search space reduction possibilities within the case base are probed on a few selected key features, applying Hierarchical clustering and Dendogram representation. Signature identification accuracy is found promising when compared with other machine learning techniques and a few existing well known approaches.

Using Wearables and Machine Learning to Enable Personalized Lifestyle Recommendations to Improve Blood Pressure.

Background: Blood pressure (BP) is an essential indicator for human health and is known to be greatly influenced by lifestyle factors, like activity and sleep factors. However, the degree of impact of each lifestyle factor on BP is unknown and may vary between individuals. Our goal is to investigate the relationships between BP and lifestyle factors and provide personalized and precise recommendations to improve BP, as opposed to the current practice of general lifestyle recommendations. Method: Our proposed system consists of automated data collection using home BP monitors and wearable activity trackers and feature engineering techniques to address time-series data and enhance interpretability. We propose Random Forest with Shapley-Value-based Feature Selection to offer personalized BP modeling and top lifestyle factor identification, and subsequent generation of precise recommendations based on the top factors. Result: In collaboration with UC San Diego Health and Altman Clinical and Translational Research Institute, we performed a clinical study, applying our system to 25 patients with elevated BP or stage I hypertension for three consecutive months. Our study results validate our system’s ability to provide accurate personalized BP models and identify the top features which can vary greatly between individuals. We also validate the effectiveness of personalized recommendations in a randomized controlled experiment. After receiving recommendations, the subjects in the experimental group decreased their BPs by 3.8 and 2.3 for systolic and diastolic BP, compared to the decrease of 0.3 and 0.9 for the subjects without recommendations. Conclusion: The study demonstrates the potential of using wearables and machine learning to develop personalized models and precise lifestyle recommendations to improve BP.

Forecasting the 2020 US Elections with Decision Desk HQ: Methodology for Modern American Electoral Dynamics

Optimus has constructed models to predict the outcomes of the 2020 presidential and congressional general elections in collaboration with Decision Desk HQ. The model is an iteration from its 2018 U.S. Congressional model designed to predict the outcome of the election as if it were held today. The congressional model predicts the probability of a Republican (GOP) victory in individual House and Senate elections, as well as the number of aggregate seats expected to be won by each party (to predict partisan control of each chamber). The presidential model uses a similar framework to estimate vote shares and probabilities of victory for each major party candidate in each of the states. These estimates are then used to proxy electoral college predictions that determine who is elected as the next President of the United States. We provide a survey of features, feature engineering techniques, models, and ensembling techniques. We also provide some empirical results.

Deep Recurrent Neural Networks for Automatic Detection of Sleep Apnea from Single Channel Respiration Signals.

Sleep apnea is a common sleep disorder that causes repeated breathing interruption during sleep. The performance of automated apnea detection methods based on respiratory signals depend on the signals considered and feature extraction methods. Moreover, feature engineering techniques are highly dependent on the experts’ experience and their prior knowledge about different physiological signals and conditions of the subjects. To overcome these problems, a novel deep recurrent neural network (RNN) framework is developed for automated feature extraction and detection of apnea events from single respiratory channel inputs. Long short-term memory (LSTM) and bidirectional long short-term memory (BiLSTM) are investigated to develop the proposed deep RNN model. The proposed framework is evaluated over three respiration signals: Oronasal thermal airflow (FlowTh), nasal pressure (NPRE), and abdominal respiratory inductance plethysmography (ABD). To demonstrate our results, we use polysomnography (PSG) data of 17 patients with obstructive, central, and mixed apnea events. Our results indicate the effectiveness of the proposed framework in automatic extraction for temporal features and automated detection of apneic events over the different respiratory signals considered in this study. Using a deep BiLSTM-based detection model, the NPRE signal achieved the highest overall detection results with true positive rate (sensitivity) = 90.3%, true negative rate (specificity) = 83.7%, and area under receiver operator characteristic curve = 92.4%. The present results contribute a new deep learning approach for automated detection of sleep apnea events from single channel respiration signals that can potentially serve as a helpful and alternative tool for the traditional PSG method.

Enriching feature engineering for short text samples by language time series analysis

In this case study, we are extending feature engineering approaches for short text samples by integrating techniques which have been introduced in the context of time series classification and signal processing. The general idea of the presented feature engineering approach is to tokenize the text samples under consideration and map each token to a number, which measures a specific property of the token. Consequently, each text sample becomes a language time series, which is generated from consecutively emitted tokens, and time is represented by the position of the respective token within the text sample. The resulting language time series can be characterised by collections of established time series feature extraction algorithms from time series analysis and signal processing. This approach maps each text sample (irrespective of its original length) to 3970 stylometric features, which can be analysed with standard statistical learning methodologies. The proposed feature engineering technique for short text data is applied to two different corpora: the Federalist Papers data set and the Spooky Books data set. We demonstrate that the extracted language time series features can be successfully combined with standard machine learning approaches for natural language processing and have the potential to improve the classification performance. Furthermore, the suggested feature engineering approach can be used for visualizing differences and commonalities of stylometric features. The presented framework models the systematic feature engineering based on approaches from time series classification and develops a statistical testing methodology for multi-classification problems.

ECG-Derived Heart Rate Variability Interpolation and 1-D Convolutional Neural Networks for Detecting Sleep Apnea.

Feature extraction from ECG-derived heart rate variability signal has shown to be useful in classifying sleep apnea. In earlier works, time-domain features, frequency-domain features, and a combination of the two have been used with classifiers such as logistic regression and support vector machines. However, more recently, deep learning techniques have outperformed these conventional feature engineering and classification techniques in various applications. This work explores the use of convolutional neural networks (CNN) for detecting sleep apnea segments. CNN is an image classification technique that has shown robust performance in various signal classification applications. In this work, we use it to classify one-dimensional heart rate variability signal, thereby utilizing a one-dimensional CNN (1-D CNN). The proposed technique resizes the raw heart rate variability data to a common dimension using cubic interpolation and uses it as a direct input to the 1-D CNN, without the need for feature extraction and selection. The performance of the method is evaluated on a dataset of 70 overnight ECG recordings, with 35 recordings used for training the model and 35 for testing. The proposed method achieves an accuracy of 88.23% (AUC=0.9453) in detecting sleep apnea epochs, outperforming several baseline techniques.

Refractive index prediction models for polymers using machine learning

The refractive index (RI) is an important material property and is necessary for making informed materials selection decisions when optical properties are important. Acquiring accurate empirical measurements of RI is time consuming, and while semi-empirical and computational determination of RI is generally faster than empirical determination, predictions are less accurate. In this work, we utilized experimentally measured RI data of polymers to build a machine learning model capable of making accurate near-instantaneous predictions of RI. The Gaussian process regression model is trained using data of 527 unique polymers. Feature engineering techniques were also used to optimize model performance. This new model is one of the most chemically diverse and accurate RI prediction models to date and improves upon our previous work. We also concluded that the model is capable of providing insights about structure–property relationships important for estimating the RI when designing new polymer backbones.