Continuous Target Variables Research Articles

Concrete Breaking Strength Prediction Using Machine Learning

Concrete Breaking Strength Prediction Using Machine Learning

Breast Cancer Disease Prediction Using Random Forest Regression and Gradient Boosting Regression

The current research endeavors to evaluate the efficacy of regression-based machine learning algorithms through an assessment of their performance using diverse metrics. The focus of our study involves the implementation of the breast cancer Wisconsin (Diagnostic) dataset, employing both the random forest and gradient-boosting regression algorithms. In our comprehensive performance analysis, we utilized key metrics such as Mean Squared Error (MSE), R-squared, Mean Absolute Error (MAE), and Coefficient of Determination (COD), supplemented by additional metrics. The evaluation aimed to gauge the algorithms' accuracy and predictive capabilities. Notably, for continuous target variables, the gradient-boosting regression model emerged as particularly noteworthy in terms of performance when compared to other models. The gradient-boosting regression model exhibited remarkable results, highlighting its superiority in handling the breast cancer dataset. The model achieved an impressively low MSE value of 0.05, indicating minimal prediction errors. Furthermore, the R-squared value of 0.89 highlighted the model's ability to explain the variance in the data, affirming its robust predictive power. The Mean Absolute Error (MAE) of 0.14 reinforced the model's accuracy in predicting continuous outcomes. Beyond these core metrics, the study incorporated additional measures to provide a comprehensive understanding of the algorithms' performance. The findings underscore the potential of gradient-boosting regression in enhancing predictive accuracy for datasets with continuous target variables, particularly evident in the context of breast cancer diagnosis. This research contributes valuable insights to the ongoing exploration of machine learning algorithms, providing a basis for informed decision-making in medical and predictive analytics domains.

Utilizing Machine Learning to Forecast Smoking Behavior

Abstract: The examination of smoking habits, extensively studied over time, has posed challenges in accurately predicting and thoroughly analyzing its determinants. Previous research efforts struggled to precisely predict smoking behavior due to the presence of continuous target variables, which hindered the application of crucial feature selection techniques like mutual information. This study aims to tackle these hurdles through an innovative approach that integrates data preprocessing, feature engineering, and advanced machine learning methods. To address the issue of continuous target variables, our methodology involves categorizing smoking behavior into discrete groups, enabling the utilization of feature selection techniques such as mutual information scores. Logistic regression, Gaussian Naive Bayes, and Random Forest Classifier models are utilized to achieve highly accurate predictions of smoking behavior. The Select KBest method is employed to evaluate the importance of features based on mutual information scores. The investigation delves into various health markers, including BMI, haemoglobin levels, and cholesterol, offering comprehensive insights into their influence on smoking habits. Furthermore, Principal Component Analysis (PCA) is implemented to reduce dimensionality while preserving essential information from the dataset. Through this novel approach and a steadfast commitment to ethical data collection practices, our objective is to advance the comprehension of smoking behavior, surmounting past challenges, and providing valuable insights for public health initiatives and smoking cessation endeavors. The paper assesses outcomes using specified algorithms and parameters, presenting a comparative analysis to enhance the clarity and reliability of our findings.

Predicting Smoking Status of a Person using Machine Learning

Abstract: The study of smoking behavior, a topic extensively researched over time, has presented difficulties in accurately predicting and thoroughly analyzing its determinants. Previous studies struggled to predict smoking behavior accurately, mainly due to the presence of continuous target variables, hindering the application of vital feature selection methods like mutual information. This research aims to address these challenges through an innovative approach that incorporates data preprocessing, feature engineering, and advanced machine learning techniques. To overcome the issue of continuous target variables, our methodology involves categorizing smoking behavior into discrete groups, allowing the use of feature selection methods such as mutual information scores. Logistic regression, Gaussian Naive Bayes, and Random Forest Classifier models are employed in this study to achieve highly accurate predictions of smoking behavior. The Select KBest method is utilized to assess the significance of features based on mutual information scores. The research explores various health indicators, including BMI, haemoglobin levels, and cholesterol, providing comprehensive insights into their impact on smoking behavior. The principal component analysis, or PCA, is another technique used to lower multiplicity. while retaining essential information from the dataset. Through this innovative approach and a rigorous commitment to ethical data collection practices, our goal is to advance the understanding of smoking behavior, overcoming previous challenges, and offering valuable insights for public health initiatives and smoking cessation efforts. The study evaluates results using specified algorithms and parameters, presenting a comparative analysis to enhance the clarity and robustness of our findings.

Affine combination‐based over‐sampling for imbalanced regression

AbstractImbalanced domain prediction analysis is currently one of the hot research topics. Many real‐world data mining analyses involve using imbalanced data to obtain predictive models. In the context of imbalance, research on classification problems has been extensive, but research on regression problems is negligible. Rare values rarely occur in imbalanced regression problems, but the focus is on accurately predicting the continuous target variables of rare instances. One of the challenges in imbalanced regression is finding a suitable strategy to rebalance the original dataset in order to improve the predictive performance of the model in rare instances. In this study, two algorithms are proposed: sigma nearest over‐sampling based on convex combination for regression (SNOCCR) and affine combination‐based over‐sampling (ACOS). ACOS rebalances the original dataset by generating new instances through the affine combinations of the original examples. The region where the new instances are generated can be adjusted based on the distribution of the data, ensuring that the generated cases better mimic the distribution of the original examples. The comparison among ACOS, SNOCCR, and other preprocessing methods was conducted on 15 datasets to validate the predictive performance of models trained on rebalanced datasets for rare instances. The experimental results indicate that ACOS outperforms other existing methods.

Information Consistency-Based Measures for Spatial Stratified Heterogeneity

As a typical form of spatial heterogeneity, spatial stratified heterogeneity is widely observed in geographical phenomena. Although the q statistic provides a measure of spatial stratified heterogeneity using variance differences, it is not suitable for nominal target variables and neglects information differences between strata at higher order moments. Based on the mutual information and relative entropy between variables, two spatial stratified heterogeneity measures are proposed for nominal and continuous target variables, respectively. Permutation tests are then used to determine their statistical significance. The proposed measures are suitable for either nominal or continuous target variables. They make no assumptions regarding the distribution of target variables, and return a value of zero only when the distribution of the target variable is independent of the explanatory variable. Experiments on five illustrative data sets and three publicly accessible data sets show that the proposed measures are consistent with the q statistic and can detect the existence of spatial stratified heterogeneity when the q statistic fails, so long as there are significant differences between the distributions in different strata.

Prediction of concrete compressive strength employing machine learning techniques

Machine learning (ML) methods have proven to be reliable techniques in estimating, classifying, and predicting material strength based on varying material properties. It is observed that the nature of the problem and the data at hand governs the selection of an appropriate ML method. Therefore, to determine the accuracy of ML models to predict concrete compressive strength, fifteen different ML algorithms were applied to a given concrete compressive strength dataset. Among all the ML algorithms used, the prediction accuracy of the Support Vector Regressor (SVR) was the highest (88.18%) due to its effective performance when dealing with continuous target variables and nonlinear relationships among the features and the target.

A novel adaptive network intrusion detection system for internet of things.

Cyber-attack is one of the most challenging aspects of information technology. After the emergence of the Internet of Things, which is a vast network of sensors, technology started moving towards the Internet of Things (IoT), many IoT based devices interplay in most of the application wings like defence, healthcare, home automation etc., As the technology escalates, it gives an open platform for raiders to hack the network devices. Even though many traditional methods and Machine Learning algorithms are designed hot, still it "Have a Screw Loose" in detecting the cyber-attacks. To "Pull the Plug on" an effective "Intrusion Detection System (IDS)" is designed with "Deep Learning" technique. This research work elucidates the importance in detecting the cyber-attacks as "Anomaly" and "Normal". Fast Region-Based Convolution Neural Network (Fast R-CNN), a deep convolution network is implemented to develop an efficient and adaptable IDS. After hunting many research papers and articles, "Gradient Boosting" is found to be a powerful optimizer algorithm that gives us a best results when compared to other existing methods. This algorithm uses "Regression" tactics, a statistical technique to predict the continuous target variable that correlates between the variables. To create a structured valid dataset, a stacked model is made by implementing the two most popular dimensionality reduction techniques Principal Component Analysis (PCA) and Singular Value Decomposition (SVD) algorithms. The brainwaves made us to hybridize Fast R-CNN and Gradient Boost Regression (GBR) which reduces the loss function, processing time and boosts the model's performance. All the above said methods are trained and tested with NIDS dataset V.10 2017. Finally, the "Decision Making" model decides the best result by giving an alert message. Our proposed model attains a high accuracy of 99.5% in detecting the "Cyber Attacks". The experiment results revealed that the effectiveness of our proposed model surpasses other deep neural network and machine learning techniques which have less accuracy.

Dimensionality reduction strategy for Multi-Target Regression paradigm

Multi-Target Regression (MTR) is used to study the relationship between the same set of input variables and multiple continuous target variables simultaneously. A dataset with many input and output variables is the prime issue to address in the MTR, which is computationally complex to build a prediction model. Also, dimensionality reduction from multiple target variables is a challenging and essential task that aims to reduce the size of the dataset to optimize the time complexity of analysis and remove the redundant and irrelevant variables. This paper proposes an efficient feature selection strategy, Multi-Target Feature Subset Selection (MTFSS), for MTR that constructs a unique subset of features by considering multiple targets. On the other hand, two feature evaluators, correlation and ReliefF, support the MTR dataset without discretization. Furthermore, two new score functions, weighted mean aggregation strategy and threshold function, are introduced to identify the significant features. To evaluate the effectiveness of the proposed MTFSS, experiments were carried out on a benchmark dataset. The experimental results demonstrate that the proposed MTFSS can select fewer features and perform better than the original dataset results. Also, the correlation-based feature evaluator performs better than ReliefF with better performance.

Classification random forest with exact conditioning for spatial prediction of categorical variables

Machine learning methods are increasingly used for spatially predicting a categorical target variable when spatially exhaustive predictor variables are available within the study region. Even though these methods exhibit competitive spatial prediction performance, they do not exactly honor the categorical target variable's observed values at sampling locations by construction. On the other side, competitor geostatistical methods perfectly match the categorical target variable's observed values at sampling locations by essence. In many geoscience applications, it is often desirable to perfectly match the observed values of the categorical target variable at sampling locations, especially when the categorical target variable's measurements can be reasonably considered error-free. This paper addresses the problem of exact conditioning of machine learning methods for the spatial prediction of categorical variables. It introduces a classification random forest-based approach in which the categorical target variable is exactly conditioned to the data, thus having the exact conditioning property like competitor geostatistical methods. The proposed method extends a previous work dedicated to continuous target variables by using an implicit representation of the categorical target variable. The basic idea consists of transforming the ensemble of classification tree predictors' (categorical) resulting from the traditional classification random forest into an ensemble of signed distances (continuous) associated with each category of the categorical target variable. Then, an orthogonal representation of the ensemble of signed distances is created through the principal component analysis, thus allowing to reformulate the exact conditioning problem as a system of linear inequalities on principal component scores. Then, the sampling of new principal component scores ensuring the data's exact conditioning is performed via randomized quadratic programming. The resulting conditional signed distances are turned out into an ensemble of categorical outputs, which perfectly honor the categorical target variable's observed values at sampling locations. Then, the majority vote is used to aggregate the ensemble of categorical outputs. The effectiveness of the proposed method is illustrated on a simulated dataset for which ground-truth is available and showcased on a real-world dataset, including geochemical data. A comparison with geostatistical and traditional machine learning methods show that the proposed technique can perfectly match the categorical target variable's observed values at sampling locations while maintaining competitive out-of-sample predictive performance.

Conservative Confidence Interval Prediction in Fused Deposition Modeling Process With Linear Optimization Approach

Abstract Regression models are widely used as data-driven methods for predicting a continuous target variable. From a set of input variables, regression models predict a deterministic point value for the target variable. But the deterministic point value prediction is not always sufficient because a target variable value often varies due to diverse sources of uncertainty. For instance, in the fused deposition modeling process, the inconsistent results of replications are associated with natural randomness, environmental condition, and noisy process parameters. The point value estimation is not sufficient to represent the variability in this kind of scenario. Instead of point estimation, the interval prediction of a target variable is more useful in this application. In this paper, linear optimization-based techniques are proposed to predict conservative confidence intervals for linear and polynomial regression models. Two linear optimization models, one for ordinary least squares (OLS) regression and the other for weighted least squares (WLS) regression, are proposed. The proposed methods are implemented on several datasets, including an experimental fused deposition modeling dataset to demonstrate the effectiveness of the proposed methods. The results show that the proposed method is useful for the fused deposition modeling process where the level of uncertainty or the lack of knowledge of uncertainty sources is high. The proposed method can also be leveraged to the Bayesian neural network (BNN), where the optimization techniques for interval prediction will be nonlinear optimization instead of linear optimization.

Multi-target regression via self-parameterized Lasso and refactored target space

Multi-target regression (MTR) aims at simultaneously predicting multiple continuous target variables based on the same set of input variables. It has been used to solve some challenging problems. A self-parameterized Lasso for MTR is proposed in this paper, which is applied to refactored target space via linear combinations of existing targets. Our approach can simultaneously model intrinsic inter-target correlations and input-target correlations, which makes full use of the information contained in the data. Meanwhile, this information helps automatically generate the parameters needed in the model. Compared with the common method, which requires a manual setting of parameters and is expensive to optimize these parameters, our method can save a lot of time while no reduction in performance. Besides, our method can be used not only for MTR tasks but also for multi-classification tasks. The experimental results show that our method performs well in different tasks and has a wide range of applications.

Personal characteristics affecting the biological age of the individual

The aim of the research is to study the influence of personality traits and the life path on the rate of biological aging of adults, depending on their gender and age. Methods: diagnostic (questioning the life path, self-esteem of personality, determination of biological age) and statistical (automatic neural networks). Subjects: 987 people (including 575 women) aged 35-70 years from various regions of the Russian Federation. Continuous target variable: relative biological aging of the organism (Biological Age – Proper Biological Age). Continuous input variables: indicators of life path and personality self-esteem. Results: the general factors influencing the rate of biological aging were identified: negative factors that enhance aging - addictions and overwork; and positive factors that slow down aging are having children and professionalization. Personal indicators that changed their sign after a person retired were also found. Women changed their sign from negative to positive for indicators “work at the present time” and “study at the present time”, for men – “a positive attitude to pension reform” and professional activities of the types “Human-Technology”, “Human-Sign”. The negative strategies of organizing the life path leading to the acceleration of biological aging are highlighted: “forming the addictions and striving for excesses” and “multiplication of the load”. Positive strategies are also highlighted: “positive activity strategy” (for women and men of retirement age) and “family strategy” (for men of pre-retirement age).

Performing Multi-Target Regression via a Parameter Sharing-Based Deep Network.

Multi-target regression (MTR) comprises the prediction of multiple continuous target variables from a common set of input variables. There are two major challenges when addressing the MTR problem: the exploration of the inter-target dependencies and the modeling of complex input-output relationships. This paper proposes a neural network model that is able to simultaneously address these two challenges in a flexible way. A deep architecture well suited for learning multiple continuous outputs is designed, providing some flexibility to model the inter-target relationships by sharing network parameters as well as the possibility to exploit target-specific patterns by learning a set of nonshared parameters for each target. The effectiveness of the proposal is analyzed through an extensive experimental study on 18 datasets, demonstrating the benefits of using a shared representation that exploits the commonalities between target variables. According to the experimental results, the proposed model is competitive with respect to the state-of-the-art in MTR.

Resampling strategies for regression

AbstractSeveral real world prediction problems involve forecasting rare values of a target variable. When this variable is nominal, we have a problem of class imbalance that was thoroughly studied within machine learning. For regression tasks, where the target variable is continuous, few works exist addressing this type of problem. Still, important applications involve forecasting rare extreme values of a continuous target variable. This paper describes a contribution to this type of tasks. Namely, we propose to address such tasks by resampling approaches that change the distribution of the given data set to decrease the problem of imbalance between the rare target cases and the most frequent ones. We present two modifications of well‐known resampling strategies for classification tasks: the under‐sampling and the synthetic minority over‐sampling technique (SMOTE) methods. These modifications allow the use of these strategies on regression tasks where the goal is to forecast rare extreme values of the target variable. In an extensive set of experiments, we provide empirical evidence for the superiority of our proposals for these particular regression tasks. The proposed resampling methods can be used with any existing regression algorithm, which means that they are general tools for addressing problems of forecasting rare extreme values of a continuous target variable.

Efficient algorithms for finding optimal binary features in numeric and nominal labeled data

An important subproblem in supervised tasks such as decision tree induction and subgroup discovery is finding an interesting binary feature (such as a node split or a subgroup refinement) based on a numeric or nominal attribute, with respect to some discrete or continuous target variable. Often one is faced with a trade-off between the expressiveness of such features on the one hand and the ability to efficiently traverse the feature search space on the other hand. In this article, we present efficient algorithms to mine binary features that optimize a given convex quality measure. For numeric attributes, we propose an algorithm that finds an optimal interval, whereas for nominal attributes, we give an algorithm that finds an optimal value set. By restricting the search to features that lie on a convex hull in a coverage space, we can significantly reduce computation time. We present some general theoretical results on the cardinality of convex hulls in coverage spaces of arbitrary dimensions and perform a complexity analysis of our algorithms. In the important case of a binary target, we show that these algorithms have linear runtime in the number of examples. We further provide algorithms for additive quality measures, which have linear runtime regardless of the target type. Additive measures are particularly relevant to feature discovery in subgroup discovery. Our algorithms are shown to perform well through experimentation and furthermore provide additional expressive power leading to higher-quality results.

Uncertainty causes rounding: an experimental study

Rounding is a common phenomenon when subjects provide an answer to an open-ended question, both in experimental tasks and in survey responses. From a statistical perspective, rounding implies that the measured variable is a coarsened version of the underlying continuous target variable. Since the coarsening process is non-random, inference from rounded data is generally biased. Despite the potentially severe consequences of rounding, little is known about its causes. In this paper, we focus on subjects' uncertainty about the target variable as one potential cause for rounding behavior. We present a novel experimental method that induces uncertainty in a controlled way, thus providing causal evidence for the effect of subjects' uncertainty on the extent of rounding. Then, we specify and estimate a mixture model that relates uncertainty and rounding. The results suggest that an increase in the exogenous level of uncertainty translates into higher variance of the subjects' beliefs, which in turn results in more rounding.

Ordinal and nominal classification of wind speed from synoptic pressurepatterns

Wind speed reconstruction is a challenging problem in areas (mainly wind farms) where there are not direct wind measures available. Different approaches have been applied to this reconstruction, such as measure-correlate-predict algorithms, approaches based on physical models such as reanalysis methods, or more recently, indirect measures such as pressure, and its relation to wind speed. This paper adopts the latter method, and deals with wind speed estimation in wind farms from pressure measures, but including different novelties in the problem treatment. Existing synoptic pressure-based indirect approaches for wind speed estimation are based on considering the wind speed as a continuous target variable, estimating then the corresponding wind series of continuous values. However, the exact wind speed is not always needed by wind farm managers, and a general idea of the level of speed is, in the majority of cases, enough to set functional operations for the farm (such as wind turbines stop, for example). Moreover, the accuracy of the models obtained is usually improved for the classification task, given that the problem is simplified. Thus, this paper tackles the problem of wind speed prediction from synoptic pressure patterns by considering wind speed as a discrete variable and, consequently, wind speed prediction as a classification problem, with four wind level categories: low, moderate, high or very high. Moreover, taking into account that these four different classes are associated to four values in an ordinal scale, the problem can be considered as an ordinal regression problem. The performance of several ordinal and nominal classifiers and the improvement achieved by considering the ordering information are evaluated. The results obtained in this paper present the support vector machine as the best tested classifier for this task. In addition, the use of the intrinsic ordering information of the problem is shown to significantly improve ranks with respect to nominal classification, although differences in accuracy aresmall.

Predicting Waterflood Responses With Decision Trees

Abstract Adding new wells and new production in existing fields under Enhanced Oil Recovery (EOR) is particularly important in mature fields that are characterized by a long history of field activity. Different drilling programs, a variety of field treatments, well conversions and new injectors add many layers of complexity and uncertainty to the existing effects of geological, completion and production factors. Surveillance and prediction of responses caused by injected fluids in fields with dozens of patterns and hundreds of wells calls for computer-based systems that estimate responses based on numerical and statistical solutions. This is especially important when geological understanding is very weak (i.e. no core, no log data). This paper shows how results from EOR surveillance programs can be integrated with geological data. Furthermore, this paper shows how to build predictive models for production estimates based on injection responses and geology. These models support a two to three times more accurate selection of wells with high oil production during EOR than historically implemented selections. Included in the paper are practical tips on how to select the best model and derive solutions with decision trees that are equivalent to sets of English-based rules. Solutions from decision trees are compared with solutions from logistic regression and neural networks. This comparison deals with the statistical accuracy of model predictions, interpretation ability and assisting in applying these models to support field decisions. Introduction The main goal of the study was to develop and test a model that would predict production performance during waterfloods. In predictive modelling, regression is traditionally applied to calculate continuous target variables(1). Models that predict binary response variables use logistic regression(2). A binary target variable is characterized by two events. They can be of a numerical nature (0 and 1) where zero represents a non-event and one represents an event. Alternatively, a character string with two outcomes (e.g. No and Yes) is often applied. In the case of a continuous target variable, we may predict the fluid rates of oil, water, gas or the total fluids. The binary 0 or 1 target variable can represent a low or high production output, respectively. A production cutoff can be based on economical or engineering criteria applied to the actual rates or volumes in a specific time period. This paper presents the principles of a numerical model development for predicting well performance during waterflood in the Pekisko B Field. The production performance predictions were based on geological and injection response parameters. The injection response parameter definition and non-numerical integration with geological data was presented in an earlier paper(3). In this paper, the performance predictions were done for a binary target variable that identified wells with good performance. The performance was based on the actual normalized volume of production. A well's normalized production was defined as a ratio of the well production in a specific time period, to the total field production in the same time period. A binary target variable (indicator) of two levels (0 and 1) was assigned based on the normal-ized production.

Decision tree approaches for zero-inflated count data

There have been many methodologies developed about zero-inflated data in the field of statistics. However, there is little literature in the data mining fields, even though zero-inflated data could be easily found in real application fields. In fact, there is no decision tree method that is suitable for zero-inflated responses. To analyze continuous target variable with decision trees as one of data mining techniques, we use F-statistics (CHAID) or variance reduction (CART) criteria to find the best split. But these methods are only appropriate to a continuous target variable. If the target variable is rare events or zero-inflated count data, the above criteria could not give a good result because of its attributes. In this paper, we will propose a decision tree for zero-inflated count data, using a maximum of zero-inflated Poisson likelihood as the split criterion. In addition, using well-known data sets we will compare the performance of the split criteria. In the case when the analyst is interested in lower value groups (e.g. no defect areas, customers who do not claim), the suggested ZIP tree would be more efficient.