Non-linear Machine Learning Algorithms Research Articles

Exploration of the Hidden Influential Factors on Crime Activities: A Big Data Approach

Crime activities have long been a great concern of all the countries. Analysis of crime data has been a key part yet a considerable challenge for discovering crime patterns and reducing crimes. In recent year, along with the development of data collection and data mining techniques, lots of big data-related studies have been conducted to analyze the crime data. Studying the numerical influential factors is one important yet challenging problem, especially for those indirect features. Though a number of studies have been conducted to analyze the influential factors of crime activities, most of them have some limitations in the era of “big data”. Some adopted the linear statistical methods, of which the basic assumption is opposite to the non-linear real world. Some limited their studied factors within one or two aspects. Some overlooked the importance of ranking the influence of factors. To fill these research gaps, this paper proposes a big data approach to analyze the influential factors on the crime activities, and experimented it on New York City. More than 1515 different factors ranging from demographic, housing, education, economy, social, and city planning were considered and analyzed. The proposed framework combines non-linear machine learning algorithms and geographical information system (GIS) to study the spatial determinants of crimes. Recursive feature elimination (RFE) is used to select the optimum feature set. Performance of gradient boost decision tree (GBDT), logistic regression (LR), support vector machine (SVM), artificial neural network (ANN) and random forest (RF) are compared to generate the optimum model. Important impact factors were then investigated using GBDT and GIS. The experimental results demonstrate that the combined GBDT and GIS model can find out the most important factors of crime rate with high efficiency and accuracy.

A Nonlinear Tensor-Based Machine Learning Algorithm for Image Classification

A Nonlinear Tensor-Based Machine Learning Algorithm for Image Classification

Ranking of environmental heat stressors for dairy cows using machine learning algorithms

Heat stress is harmful to the health and productivity of dairy cows. While several models have been developed to assess heat stress conditions for dairy cows, many of these models assume a particular relationship, chosen by the researcher, between environmental conditions and their physiological responses to heat stress. These assumptions may not accurately represent the underlying effects. This study uses machine learning algorithms to evaluate how environmental heat stressors (air temperature, AT; relative humidity, RH; solar radiation, SR; and wind speed, U) influence physiological responses (respiration rate, RR; skin temperature, ST; and vaginal temperature, VT) of dairy cows. The advantage of this approach is that many machine learning algorithms automatically consider nonlinearity in data, which removes subjectivity from researchers choosing the relationship between the predictor and response variables. Four algorithms were considered in this study: penalized linear regression, random forests, gradient boosted machines, and neural networks. Nonlinear machine learning algorithms (random forests and neural networks) were consistently the most accurate in predicting the three physiological responses. The root mean squared error, RMSE, for RR was 9.695 respirations per minute, which was obtained with a random forest model; RMSE for ST was 0.334 °C, which was obtained with a random forest model; and RMSE for VT was 0.434 °C, which was obtained with a neural network model. Air temperature had the highest ranking for effect on RR, ST, and VT. Wind speed and its interaction terms displayed much lower effects as environmental heat stressors. Ranking of environmental heat stressors could help farmers make evidence-based interventions before anticipated stressful environmental conditions occur. Early intervention could improve animal health, which would increase production and reduce the costs associated with treating heat stressed livestock.

Direct prediction of petrophysical and petroelastic reservoir properties from seismic and well-log data using nonlinear machine learning algorithms

An analytical comparison of seismic inversion with several multivariate predictive techniques is made. Statistical data reduction techniques are examined that incorporate various machine learning algorithms, such as linear regression, alternating conditional expectation regression, random forest, and neural network. Seismic and well-log data are combined to estimate petrophysical or petroelastic properties, like bulk density. Currently, spatial distribution and estimation of reservoir properties is leveraged by inverting 3D seismic data calibrated to elastic properties (VP, VS, and bulk density) obtained from well-log data. Most commercial seismic inversions are based on linear convolution, i.e., one-dimensional models that involve a simplified plane-parallel medium. However, in cases that are geophysically more complex, such as fractured and/or fluid-rich layers, the conventional straightforward prediction relationship breaks down. This is because linear convolution operators no longer adequately describe seismic wavefield propagation due to nonlinear energy absorption. Such nonlinearity is also suggested by the seismic nonstationarity phenomenon, expressed by vertical and horizontal changes in the shape of the seismic wavelet (amplitude and frequency variations). The nonlinear predictive operator, extracted by machine learning algorithms, makes it possible in certain cases to estimate petrophysical reservoir properties more accurately and with less influence of interpretational bias.

Identification of high impact factors of air quality on a national scale using big data and machine learning techniques

Abstract To effectively control and prevent air pollution, it is necessary to study the influential factors of air quality. A number of previous studies have explored the relationships between air pollution and related factors. However, the methods currently used either cannot well address the multicollinearity problem or fail to explain the importance of the influential factors. Moreover, most of the existing literature limited their studied area in a city or a small region and studied factors in one aspect. There is a lack of studies that analyze the influential factors from the perspective of a country or take into consideration multiple variables. To fill the research gap, this paper proposes a multivariate analysis in the national scale to investigate the most important factors of air quality. In order to study as much influential factors as possible, 171 features ranging from environmental, demographical, economic, meteorological, and energy, were collected and analyzed. To tackle such a “big data” problem, a non-linear machine learning algorithm namely Extreme Gradient Boosting (XGBoost) is utilized to model the relationship and measure the variable importance. Geographical Information System (GIS) is employed to preprocess the diversified variables and visualize the results. Performance of XGBoost is compared with other models and its parameters are tuned using Bayesian Optimization. Experimental results of a case study in the U.S. show that our methodology framework can effectively uncover the important factors of air quality. Six kinds of factors are found to have the largest impact on air quality. Practical suggestions are also proposed from the six aspects to control and prevent air pollution.

Multiple Machine Learning Classifiers for Student’s Admission to University Prediction

Data is the most important asset for any organization which is further processed to produce useful information. Machine Learning and Big Data techniques are widely used for industrial sectors to generate useful patterns helpful for earning more profits and expand businesses. From the past few years, a lot of research works have been done by using Big Data techniques on educational data for improvement in Education System. Machine Learning and Big Data can be useful for predicting the students’ admission, performance of teaching, performance of a student, identifying the group of students of similar behavior. However, the manual process of record checking is time consuming, tedious, and error prone; due to the inherent volume and complexity of data. In this study, the combination of linear and non-linear machine learning algorithms; Logistic Regression, Decision Tree, k-NN, and Naïve Bayes have been chosen to perform prediction of the target class for an unseen observation by polling. As the models built in this work are predicting the likelihood of a student taking up the admission into any university based on the student data collected by any marketing agency, the combined models are collectively called as the Admission Predictor. The administrative officials of any academic institution can use this kind of an application to explore and analyze the patterns that are affecting the student admission and come up with enhanced strategies to improve admission. Such an application not only plays a vital role in administration, but also help the management in reformulating the marketing criteria for overall development of academic institution.

A Comparative Analysis of Nonlinear Machine Learning Algorithms for Breast Cancer Detection

A Comparative Analysis of Nonlinear Machine Learning Algorithms for Breast Cancer Detection

Comparison between linear and nonlinear machine-learning algorithms for the classification of thyroid nodules

BackgroundA key challenge in thyroid carcinoma is preoperatively diagnosing malignant thyroid nodules. The purpose of this study was to compare the classification performance of linear and nonlinear machine-learning algorithms for the evaluation of thyroid nodules using pathological reports as reference standard. MethodsEthical approval was obtained for this retrospective analysis, and the informed consent requirement was waived. A total of 1179 thyroid nodules (training cohort, n = 700; validation cohort, n = 479) were confirmed by pathological reports or fine-needle aspiration (FNA) biopsy. The following ultrasonography (US) featu res were measured for each nodule: size (maximum diameter), margins, shape, aspect ratio, capsule, hypoechoic halo, composition, echogenicity, calcification pattern, vascularity, and cervical lymph node status. We analyzed five nonlinear and three linear machine-learning algorithms. The diagnostic performance of each algorithm was compared by using the area under the curve (AUC) of the receiver operating characteristic curve. We repeated this process 1000 times to obtain the mean AUC and 95% confidence interval (CI). ResultsOverall, nonlinear machine-learning algorithms demonstrated similar AUCs compared with linear algorithms. The Random Forest and Kernel Support Vector Machines algorithms achieved slightly greater AUCs in the validation cohort (0.954, 95% CI: 0.939–0.969; 0.954 95%CI: 0.939–0.969, respectively) than other algorithms. ConclusionsOverall, nonlinear machine-learning algorithms share similar performance compared with linear algorithms for the evaluation the malignancy risk of thyroid nodules.

Subject-specific and group-based running pattern classification using a single wearable sensor

The objective of this study was to determine whether subject-specific or group-based models provided better classification accuracy to identify changes in biomechanical running gait patterns across different inclination conditions. The classification process was based on measurements from a single wearable sensor using a total of 41,780 strides from eleven recreational runners while running in real-world and uncontrolled environment. Biomechanical variables included pelvic drop, ground contact time, braking, vertical oscillation of pelvis, pelvic rotation, and cadence were recorded during running on three inclination grades: downhill, −2° to −7°; level, −0.2° to +0.2°; and uphill, +2° to +7°. An ensemble and non-linear machine learning algorithm, random forest (RF), was used to classify inclination condition and determine the importance of each of the biomechanical variables. Classification accuracy was determined for subject-specific and group-based RF models. The mean classification accuracy of all subject-specific RF models was 86.29%, while group-based classification accuracy was 76.17%. Braking was identified as the most important variable for all the runners using the group-based model and for most of the runners based on a subject-specific models. In addition, individual runners used different strategies across different inclination conditions and the ranked order of variable importance was unique for each runner. These results demonstrate that subject-specific models can better characterize changes in gait biomechanical patterns compared to a more traditional group-based approach.

Environmental factors influencing snowfall and snowfall prediction in the Tianshan Mountains, Northwest China

Snowfall is one of the dominant water resources in the mountainous regions and is closely related to the development of the local ecosystem and economy. Snowfall predication plays a critical role in understanding hydrological processes and forecasting natural disasters in the Tianshan Mountains, where meteorological stations are limited. Based on climatic, geographical and topographic variables at 27 meteorological stations during the cold season (October to April) from 1980 to 2015 in the Tianshan Mountains located in Xinjiang of Northwest China, we explored the potential influence of these variables on snowfall and predicted snowfall using two methods: multiple linear regression (MLR) model (a conventional measuring method) and random forest (RF) model (a non-parametric and non-linear machine learning algorithm). We identified the primary influencing factors of snowfall by ranking the importance of eight selected predictor variables based on the relative contribution of each variable in the two models. Model simulations were compared using different performance indices and the results showed that the RF model performed better than the MLR model, with a much higher R2 value (R2=0.74; R2, coefficient of determination) and a lower bias error (RSR=0.51; RSR, the ratio of root mean square error to standard deviation of observed dataset). This indicates that the non-linear trend is more applicable for explaining the relationship between the selected predictor variables and snowfall. Relative humidity, temperature and longitude were identified as three of the most important variables influencing snowfall and snowfall prediction in both models, while elevation, aspect and latitude were of secondary importance, followed by slope and wind speed. These results will be beneficial to understand hydrological modeling and improve management and prediction of water resources in the Tianshan Mountains.

Predicting the vessel lumen area tree-ring parameter of <i>Quercus robur</i> with linear and nonlinear machine learning algorithms

Climate-growth relationships in <i>Quercus robur</i> chronologies for vessel lumen area (VLA) from two oak stands (QURO-1 and QURO-2) showed a consistent temperature signal: VLA is highly correlated with mean April temperature and the temperature at the end of the previous growing season. QURO-1 showed significant negative correlations with winter sums of precipitation. Selected climate variables were used as predictors of VLA in a comparison of various linear and nonlinear machine learning methods: Artificial Neural Networks (ANN), Multiple Linear Regression (MLR), Model Trees (MT), Bagging of Model Trees (BMT) and Random Forests of Regression Trees (RF). ANN outperformed all the other regression algorithms at both sites. Good performance also characterised RF and BMT, while MLR, and especially MT, displayed weaker performance. Based on our results, advanced machine learning algorithms should be seriously considered in future climate reconstructions.

Using wearable sensors to classify subject-specific running biomechanical gait patterns based on changes in environmental weather conditions.

Running-related overuse injuries can result from a combination of various intrinsic (e.g., gait biomechanics) and extrinsic (e.g., running surface) risk factors. However, it is unknown how changes in environmental weather conditions affect running gait biomechanical patterns since these data cannot be collected in a laboratory setting. Therefore, the purpose of this study was to develop a classification model based on subject-specific changes in biomechanical running patterns across two different environmental weather conditions using data obtained from wearable sensors in real-world environments. Running gait data were recorded during winter and spring sessions, with recorded average air temperatures of -10° C and +6° C, respectively. Classification was performed based on measurements of pelvic drop, ground contact time, braking, vertical oscillation of pelvis, pelvic rotation, and cadence obtained from 66,370 strides (~11,000/runner) from a group of recreational runners. A non-linear and ensemble machine learning algorithm, random forest (RF), was used to classify and compute a heuristic for determining the importance of each variable in the prediction model. To validate the developed subject-specific model, two cross-validation methods (one-against-another and partitioning datasets) were used to obtain experimental mean classification accuracies of 87.18% and 95.42%, respectively, indicating an excellent discriminatory ability of the RF-based model. Additionally, the ranked order of variable importance differed across the individual runners. The results from the RF-based machine-learning algorithm demonstrates that processing gait biomechanical signals from a single wearable sensor can successfully detect changes to an individual’s running patterns based on data obtained in real-world environments.

Optimal Deep Learning LSTM Model for Electric Load Forecasting using Feature Selection and Genetic Algorithm: Comparison with Machine Learning Approaches †

Background: With the development of smart grids, accurate electric load forecasting has become increasingly important as it can help power companies in better load scheduling and reduce excessive electricity production. However, developing and selecting accurate time series models is a challenging task as this requires training several different models for selecting the best amongst them along with substantial feature engineering to derive informative features and finding optimal time lags, a commonly used input features for time series models. Methods: Our approach uses machine learning and a long short-term memory (LSTM)-based neural network with various configurations to construct forecasting models for short to medium term aggregate load forecasting. The research solves above mentioned problems by training several linear and non-linear machine learning algorithms and picking the best as baseline, choosing best features using wrapper and embedded feature selection methods and finally using genetic algorithm (GA) to find optimal time lags and number of layers for LSTM model predictive performance optimization. Results: Using France metropolitan’s electricity consumption data as a case study, obtained results show that LSTM based model has shown high accuracy then machine learning model that is optimized with hyperparameter tuning. Using the best features, optimal lags, layers and training various LSTM configurations further improved forecasting accuracy. Conclusions: A LSTM model using only optimally selected time lagged features captured all the characteristics of complex time series and showed decreased Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) for medium to long range forecasting for a wider metropolitan area.

Large-Scale Dynamic Predictive Regressions

We propose a novel approach for forecasting the equity premium within a data-rich environment based on ensembling small-scale linear models. The economic nature of the predictors is exploited to efficiently retain all of the information available without assuming a priori that some predictor might be irrelevant or easily reducible to a latent factor. Empirically, our results lend strong support for transparent linear predictive models and the use of accounting-based information when forecasting both industry and aggregate stock market excess returns: positive statistical and economic out-of-sample performance compared to sparse predictive regressions, forecast combination strategies and complex non-linear machine learning algorithms.

Identification of the numerical patterns behind the leading counties in the U.S. local green building markets using data mining

The U.S. is reported to have one of the most developed green building markets. The country wide success must start from the success of different local markets. So how many important local green building markets are there in the U.S. and why those areas became important? The answers can provide useful implications for developing countries like China to better promote their green building markets. To explore the question in a numerical way, this study therefore collected the data of 17,636 green building projects in the U.S., clustered them into 39 important regions, and analyzed the numerical patterns behind based on 82 different features covering demography, economy, education, climate and policy. Non-linear machine learning algorithms help find that economic factors and educational factors are one of the most influential features. Discoveries were also implemented in China to suggest 20 areas to be the focus of developing Chinese green building market.

Investigation of optical neuro-monitoring technique for detection of maintenance and emergence states during general anesthesia.

The American Society of Anesthesiologist recommends peripheral physiological monitoring during general anesthesia, which offers no information regarding the effects of anesthetics on the brain. Since no "gold standard" method exists for this evaluation, such a technique is needed to ensure patient comfort, procedure quality and safety. In this study we investigated functional near infrared spectroscopy (fNIRS) as possible monitor of anesthetic effects on the prefrontal cortex. Anesthetic drugs, such as sevoflurane, suppress the cerebral metabolism and alter the cerebral blood flow. We hypothesize that fNIRS derived features carry information on the effects of anesthetics on the prefrontal cortex (PFC) that can be used for the classification of the anesthetized state. In this study, patients were continuously monitored using fNIRS, BIS and standard monitoring during surgical procedures under sevoflurane general anesthesia. Maintenance and emergence states were identified and fNIRS features were identified and compared between states. Linear and non-linear machine learning algorithms were investigated as methods for the classification of maintenance/emergence. The results show that changes in oxygenated (HbO2) and deoxygenated hemoglobin (HHb) concentration and blood volume measured by fNIRS were associated with the transition between maintenance and emergence that occurs as a result of sevoflurane washout. We observed that during maintenance the signal is relatively more stable than during emergence. Maintenance and emergence states were classified with 94.7% accuracy with a non-linear model using the locally derived mean total hemoglobin, standard deviation of HbO2, minimum and range of HbO2 and HHb as features. These features were found to be correlated with the effects of sevoflurane and to carry information that allows real time and automatic classification of the anesthetized state with high accuracy.

Box-office forecasting based on sentiments of movie reviews and Independent subspace method

Box-office forecasting is a challenging but important task for movie distributors in their decision making process. Many previous studies have tried to determine a way to accurately predict the box-office, but the results reported have not been satisfactory for two main reasons: (1) lack of variable diversity and (2) simplicity of forecasting algorithms. Although the importance of word-of-mouth (WOM) has consistently emphasized in past studies, only summarized information, such as volume or valence of user ratings is commonly used. In forecasting algorithms, multiple linear regression is the most popular algorithm because it generates not only predicted values but also variable significances. In this study, new box-office forecasting models are presented to enhance the forecasting accuracy by utilizing review sentiments and employing non-linear machine learning algorithms. Viewer sentiments from review texts are used as input variables in addition to conventional predictors, whereas three machine learning-based algorithms, i.e., classification and regression tree (CART), artificial neural network (ANN), and support vector regression (SVR), are employed to capture non-linear relationship between the box-office and its predictors. In order to provide variable importance for machine learning-based forecasting algorithms, an independent subspace method (ISM) is applied. Forecasting results from six different forecasting periods show that the presented methods can make accurate and robust forecasts.

Localized Heuristic Inverse Quantitative Structure Activity Relationship with Bulk Descriptors Using Numerical Gradients

State-of-the-art quantitative structure-activity relationship (QSAR) models are often based on nonlinear machine learning algorithms, which are difficult to interpret. From a pharmaceutical perspective, QSARs are used to enhance the chemical design process. Ultimately, they should not only provide a prediction but also contribute to a mechanistic understanding and guide modifications to the chemical structure, promoting compounds with desirable biological activity profiles. Global ranking of descriptor importance and inverse QSAR have been used for these purposes. This paper introduces localized heuristic inverse QSAR, which provides an assessment of the relative ability of the descriptors to influence the biological response in an area localized around the predicted compound. The method is based on numerical gradients with parameters optimized using data sets sampled from analytical functions. The heuristic character of the method reduces the computational requirements and makes it applicable not only to fragment based methods but also to QSARs based on bulk descriptors. The application of the method is illustrated on congeneric QSAR data sets, and it is shown that the predicted influential descriptors can be used to guide structural modifications that affect the biological response in the desired direction. The method is implemented into the AZOrange Open Source QSAR package. The current implementation of localized heuristic inverse QSAR is a step toward a generally applicable method for elucidating the structure activity relationship specifically for a congeneric region of chemical space when using QSARs based on bulk properties. Consequently, this method could contribute to accelerating the chemical design process in pharmaceutical projects, as well as provide information that could enhance the mechanistic understanding for individual scaffolds.

Prediction of plant promoters based on hexamers and random triplet pair analysis

BackgroundWith an increasing number of plant genome sequences, it has become important to develop a robust computational method for detecting plant promoters. Although a wide variety of programs are currently available, prediction accuracy of these still requires further improvement. The limitations of these methods can be addressed by selecting appropriate features for distinguishing promoters and non-promoters.MethodsIn this study, we proposed two feature selection approaches based on hexamer sequences: the Frequency Distribution Analyzed Feature Selection Algorithm (FDAFSA) and the Random Triplet Pair Feature Selecting Genetic Algorithm (RTPFSGA). In FDAFSA, adjacent triplet-pairs (hexamer sequences) were selected based on the difference in the frequency of hexamers between promoters and non-promoters. In RTPFSGA, random triplet-pairs (RTPs) were selected by exploiting a genetic algorithm that distinguishes frequencies of non-adjacent triplet pairs between promoters and non-promoters. Then, a support vector machine (SVM), a nonlinear machine-learning algorithm, was used to classify promoters and non-promoters by combining these two feature selection approaches. We referred to this novel algorithm as PromoBot.ResultsPromoter sequences were collected from the PlantProm database. Non-promoter sequences were collected from plant mRNA, rRNA, and tRNA of PlantGDB and plant miRNA of miRBase. Then, in order to validate the proposed algorithm, we applied a 5-fold cross validation test. Training data sets were used to select features based on FDAFSA and RTPFSGA, and these features were used to train the SVM. We achieved 89% sensitivity and 86% specificity.ConclusionsWe compared our PromoBot algorithm to five other algorithms. It was found that the sensitivity and specificity of PromoBot performed well (or even better) with the algorithms tested. These results show that the two proposed feature selection methods based on hexamer frequencies and random triplet-pair could be successfully incorporated into a supervised machine learning method in promoter classification problem. As such, we expect that PromoBot can be used to help identify new plant promoters. Source codes and analysis results of this work could be provided upon request.

Pol II promoter prediction using characteristic 4-mer motifs: a machine learning approach

BackgroundEukaryotic promoter prediction using computational analysis techniques is one of the most difficult jobs in computational genomics that is essential for constructing and understanding genetic regulatory networks. The increased availability of sequence data for various eukaryotic organisms in recent years has necessitated for better tools and techniques for the prediction and analysis of promoters in eukaryotic sequences. Many promoter prediction methods and tools have been developed to date but they have yet to provide acceptable predictive performance. One obvious criteria to improve on current methods is to devise a better system for selecting appropriate features of promoters that distinguish them from non-promoters. Secondly improved performance can be achieved by enhancing the predictive ability of the machine learning algorithms used.ResultsIn this paper, a novel approach is presented in which 128 4-mer motifs in conjunction with a non-linear machine-learning algorithm utilising a Support Vector Machine (SVM) are used to distinguish between promoter and non-promoter DNA sequences. By applying this approach to plant, Drosophila, human, mouse and rat sequences, the classification model has showed 7-fold cross-validation percentage accuracies of 83.81%, 94.82%, 91.25%, 90.77% and 82.35% respectively. The high sensitivity and specificity value of 0.86 and 0.90 for plant; 0.96 and 0.92 for Drosophila; 0.88 and 0.92 for human; 0.78 and 0.84 for mouse and 0.82 and 0.80 for rat demonstrate that this technique is less prone to false positive results and exhibits better performance than many other tools. Moreover, this model successfully identifies location of promoter using TATA weight matrix.ConclusionThe high sensitivity and specificity indicate that 4-mer frequencies in conjunction with supervised machine-learning methods can be beneficial in the identification of RNA pol II promoters comparative to other methods. This approach can be extended to identify promoters in sequences for other eukaryotic genomes.