Classical Regression Techniques Research Articles

Forecasting Wind–Photovoltaic Energy Production and Income with Traditional and ML Techniques

Hybrid production plants harness diverse climatic sources for electricity generation, playing a crucial role in the transition to renewable energies. This study aims to forecast the profitability of a combined wind–photovoltaic energy system. Here, we develop a model that integrates predicted spot prices and electricity output forecasts, incorporating relevant climatic variables to enhance accuracy. The jointly modeled climatic variables and the spot price constitute one of the innovative aspects of this work. Regarding practical application, we considered a hypothetical wind–photovoltaic plant located in Italy and used the relevant climate series to determine the quantity of energy produced. We forecast the quantity of energy as well as income through machine learning techniques and more traditional statistical and econometric models. We evaluate the results by splitting the dataset into estimation windows and test windows, and using a backtesting technique. In particular, we found evidence that ML regression techniques outperform results obtained with traditional econometric models. Regarding the models used to achieve this goal, the objective is not to propose original models but to verify the effectiveness of the most recent machine learning models for this important application, and to compare them with more classic linear regression techniques.

Benchmarking AutoML solutions for concrete strength prediction: Reliability, uncertainty, and dilemma

Building precise machine learning and deep learning models has traditionally required a combination of mathematical skills and hands-on experience to meticulously adjust hyperparameters that significantly impact the learning process. As datasets continue to expand across various engineering domains, researchers increasingly turn to machine learning methods to uncover hidden insights that may elude classic regression techniques. This surge in adoption raises concerns about the adequacy of resultant meta-models and the interpretation of findings. In response to these challenges, automated machine learning (AutoML) emerges as a promising solution, aiming to construct machine learning models with minimal intervention or guidance from human experts. This paper benchmarks AutoML solutions by providing an overview of their principles and applying them to predict the most important mechanical properties of different concrete datasets, i.e., compressive strength. Nine datasets from various concrete types, sample sizes, and features are utilized, with a detailed discussion on the benchmark dataset from high-performance concrete, applying best practices to the other eight datasets. For each case, the importance of hyperparameter tuning is discussed, alongside the ensemble and stacking models. Tree-based models are employed for each dataset to develop SHAP plots, interpret results, and understand the contribution of each component in the mix design to the overall strength of the concrete. This paper further explores three unique aspects of benchmarking AutoML in material science: (1) “reliability” by contrasting the benchmark dataset’s error metric with literature collected over the past 20 years, (2) “uncertainty” by quantifying the variability in the mean and standard deviation of the error metric from different datasets and its correlation with the sample-to-feature ratio, and (3) “dilemma” by discussing the shortcomings of AutoML in specific concrete datasets.

Monitoring Disease Severity of Mild Cognitive Impairment from Single-Channel EEG Data Using Regression Analysis.

A deviation in the soundness of cognitive health is known as mild cognitive impairment (MCI), and it is important to monitor it early to prevent complicated diseases such as dementia, Alzheimer's disease (AD), and Parkinson's disease (PD). Traditionally, MCI severity is monitored with manual scoring using the Montreal Cognitive Assessment (MoCA). In this study, we propose a new MCI severity monitoring algorithm with regression analysis of extracted features of single-channel electro-encephalography (EEG) data by automatically generating severity scores equivalent to MoCA scores. We evaluated both multi-trial and single-trail analysis for the algorithm development. For multi-trial analysis, 590 features were extracted from the prominent event-related potential (ERP) points and corresponding time domain characteristics, and we utilized the lasso regression technique to select the best feature set. The 13 best features were used in the classical regression techniques: multivariate regression (MR), ensemble regression (ER), support vector regression (SVR), and ridge regression (RR). The best results were observed for ER with an RMSE of 1.6 and residual analysis. In single-trial analysis, we extracted a time-frequency plot image from each trial and fed it as an input to the constructed convolutional deep neural network (CNN). This deep CNN model resulted an RMSE of 2.76. To our knowledge, this is the first attempt to generate automated scores for MCI severity equivalent to MoCA from single-channel EEG data with multi-trial and single data.

Parametric Metamodeling Based on Optimal Transport Applied to Uncertainty Evaluation

When training a parametric surrogate to represent a real-world complex system in real time, there is a common assumption that the values of the parameters defining the system are known with absolute confidence. Consequently, during the training process, our focus is directed exclusively towards optimizing the accuracy of the surrogate’s output. However, real physics is characterized by increased complexity and unpredictability. Notably, a certain degree of uncertainty may exist in determining the system’s parameters. Therefore, in this paper, we account for the propagation of these uncertainties through the surrogate using a standard Monte Carlo methodology. Subsequently, we propose a novel regression technique based on optimal transport to infer the impact of the uncertainty of the surrogate’s input on its output precision in real time. The OT-based regression allows for the inference of fields emulating physical reality more accurately than classical regression techniques, including advanced ones.

Performance and healthcare analysis in elite sports teams using artificial intelligence: a scoping review.

In competitive sports, teams are increasingly relying on advanced systems for improved performance and results. This study reviews the literature on the role of artificial intelligence (AI) in managing these complexities and encouraging a system thinking shift. It found various AI applications, including performance enhancement, healthcare, technical and tactical support, talent identification, game prediction, business growth, and AI testing innovations. The main goal of the study was to assess research supporting performance and healthcare. Systematic searches were conducted on databases such as Pubmed, Web of Sciences, and Scopus to find articles using AI to understand or improve sports team performance. Thirty-two studies were selected for review. The analysis shows that, of the thirty-two articles reviewed, fifteen focused on performance and seventeen on healthcare. Football (Soccer) was the most researched sport, making up 67% of studies. The revised studies comprised 2,823 professional athletes, with a gender split of 65.36% male and 34.64% female. Identified AI and non-AI methods mainly included Tree-based techniques (36%), Ada/XGBoost (19%), Neural Networks (9%), K-Nearest Neighbours (9%), Classical Regression Techniques (9%), and Support Vector Machines (6%). This study highlights the increasing use of AI in managing sports-related healthcare and performance complexities. These findings aim to assist researchers, practitioners, and policymakers in developing practical applications and exploring future complex systems dynamics.

A multi-factor quantitative model based on weighted SIR-SAVE and model averaging

In financial quantitative research, there are many kinds of predictors that are closely related to stock prices, so how to design effective machine learning algorithms for such high-dimensional prediction problems is a hot issue. Based on this, this paper proposes a stock factor quantization and prediction model based on the idea of sufficient dimensionality reduction and model averaging. The proposed method is based on the principle of conditional independence, which can greatly retain the validity of the predictors on the stock price while solving the dimensionality catastrophe problem. In addition, the method adaptively learns the link function between the quantized factors and stock prices by using the model averaging method to effectively weigh the variance and bias of the prediction model. In the actual data analysis, this paper chooses the weighted SIR-SAVE as the adequate dimensionality reduction method, the mutual information criterion as the assignment rule of model averaging, and some classical high-dimensional regression techniques as the comparison method, and the empirical results show that the proposed method has a higher accuracy under the evaluation indexes of mean squared error and absolute error, and possesses a certain degree of robustness. Finally, since the sub-models in the model averaging are free, the proposed method is also applicable to the problem of classifying the prediction of stock ups and downs.

Prediction of tapered steel plate girders shear strength using multigene genetic programming

Tapered steel plate girders have been widely used in the construction of bridges and heavy industry structures. However, predicting their structural behavior, particularly in shear, is challenging due to their non-prismatic sections and ill-understood influencing factors. Therefore, simplified analytical and classic regression techniques may not be able to capture the underlying nonlinear relationships controlling the shear behavior of tapered plate girders. In this study, multigene genetic programming (MGGP) was utilized to explore such complex relationships, and subsequently develop robust predictive expressions for the tapered steel plate girders shear strength. Attributed to the lack of a large experimental dataset, a nonlinear finite element model (FEM) was first developed and validated against available experimental results in literature. The FEM was subsequently employed to generate a matrix of 211 numerical results to augment 200 more FEM results compiled from previous studies, to cover a wider range of design parameters. The entire dataset was then used in the training and testing of the MGGP predictive expressions. The prediction accuracy of the developed expressions was evaluated against that of other existing expressions. The results showed that the adopted MGGP approach, guided be mechanics understanding, produced elegant predictive expressions with high level of accuracy and generalizability compared to other existing ones examined herein. As such, the developed expressions present an efficient prediction tool that can be adopted by design standards for estimating the ultimate shear strength of tapered steel girders. Finally, reliability analysis is performed on the developed expressions to introduce strength reduction factors in order to satisfy target design conservatism.

Forecasting solar energy production in Spain: A comparison of univariate and multivariate models at the national level

Renewable energies, such as solar power, offer a clean and cost-effective energy source. However, their integration into national electricity grids poses challenges due to their dependence on climate and geography. While numerous studies have focused on solar energy time series, few have specifically addressed the critical task of forecasting solar energy production at the national level. Accurate national-level forecasting is crucial for optimizing energy management, informing policy development, and promoting environmental sustainability. This study aims to address the challenges associated with the significant variability in renewable energy production and its impact on grid stability by improving the accuracy of existing forecasting approaches. To achieve this goal, we evaluate the effectiveness of univariate and multivariate approaches for time series forecasting of national solar energy production data from ESIOS (the Spanish System Operator). Our primary focus is on leveraging external solar variables, such as solar irradiance data. To this end, we propose a methodology to integrate solar irradiance forecasts with historical data from solar power plants in Spain to improve the performance of multivariate models. Subsequently, we compare the performance of classical regression techniques and state-of-the-art deep learning algorithms, presenting univariate and multivariate models for three forecast horizons (1 h, 24 h, and 48 h). Finally, we assess the performance of our best univariate and multivariate models by comparing them with the official forecast of ESIOS. Our findings indicate that the best-performing models are deep-learning multivariate approaches, which benefit from incorporating solar irradiance forecasts, particularly for longer forecast horizons (24 h and 48 h), and avoid the detrimental effects of the Hughes Phenomenon, which seems to hamper non-deep-learning forecasters. The top-performing models, based on Convolutional Networks and Convolutional + Recurrent Neural Networks, outperform ESIOS by reducing mean absolute error by 41% and 47.58%, respectively.

Applicability of several machine learning models in estimation of vortex tube trapping efficiency

Abstract A vortex tube ejector comprises a tube with a slitted crown that lies flush across the entire width of a channel bed surface. The bed and suspended loads are ejected with minimal flushing water through the slit with the same efficacy as any other alternative extractor. The whirling flow phenomena through the vortex duct are very complex, so ordinary classical models have results contrary to required design guidelines. So, the machine learning (ML) models of the artificial neural network (ANN), deep neural network (DNN), gradient boosting machine (GBM), stacked ensemble (SE), and adaptive neuro-fuzzy inference system (ANFIS) are used to predict vortex tube trapping efficiency (VTE). The input dataset takes the size of the sediment (Sz), concentration (I) of the sediment, the ratio of slit thickness to diameter of the tube (th/dia), and extraction ratio (Extro) while trapping efficiency (TE) is taken as output. Based on statistical assessments, GBM appears to be better than all proposed models. However, other proposed ML models give comparable performance. The classical models, multivariate linear, and nonlinear regression techniques also provide comparatively good results. According to sensitivity analyses, the extraction ratio is the most relevant parameter in evaluating the VTE.

Gene Expression Programming (GEP) Modelling of Sustainable Building Materials including Mineral Admixtures for Novel Solutions

In this study, the employment of the gene expression programming (GEP) technique in forecasting models on sustainable construction materials including mineral admixtures and civil engineering quantities (e.g., compressive strength), was investigated. Compared to the artificial neural networks (ANN) based formulations, which are often too complicated to be used, GEP-based derived models provide estimation equations that are reasonably simple and may be used for practical design purposes and even for hand calculations. Many popular models, such as best-fitted curves based on regression analyses, multi-linear regression (MLR), multinomial logistic regression (MNLR), and multinomial variate regression (MNVR), can also be used for construction materials properties modeling. However, due to the nonlinearity and complexity of the target properties, the models established using linear regression analyses may not reveal the precise behavior. Additionally, regression models lack generality, and this comes from the fact that some functions are defined for regression in classical regression techniques; while in the GEP approach, there is no predefined function to be considered, and it reproduces or omits various combinations of parameters to provide the formulation that fits the experimental outcomes. If the input parameters can be evaluated through simple laboratory or rapid measurements, and also a comprehensive experimental database is made available, the models can be constructed with optimal flexibility. Flexibility in choosing the complexity and fitness functions, such as RMSE, MAE, and MSE, might lead to better performance of the approach and well-capturing the governing pattern behind the material’s characteristics. There may be minor inaccuracies with this technique; however, the explicit mathematical expressions, which can be easily implemented in the design and analysis process, may cover the minor inaccuracies compared to ANN, support vector machine (SVM), and other intelligent approaches. Based on the presented study, sometimes it would be better to provide more than one GEP model and consider different combinations of input contributing variables to afford the possible initial feed for a more settled and comprehensive model. Mostly, GEP’s strengths as a superior machine learning technique in modeling the behavior of construction materials including mineral admixtures, leading to innovative solutions in civil engineering, have been presented.

CoCoA: conditional correlation models with association size.

Many scientific questions can be formulated as hypotheses about conditional correlations. For instance, in tests of cognitive and physical performance, the trade-off between speed and accuracy motivates study of the two variables together. A natural question is whether speed-accuracy coupling depends on other variables, such as sustained attention. Classical regression techniques, which posit models in terms of covariates and outcomes, are insufficient to investigate the effect of a third variable on the symmetric relationship between speed and accuracy. In response, we propose a conditional correlation model with association size, a likelihood-based statistical framework to estimate the conditional correlation between speed and accuracy as a function of additional variables. We propose novel measures of the association size, which are analogous to effect sizes on the correlation scale while adjusting for confound variables. In simulation studies, we compare likelihood-based estimators of conditional correlation to semiparametric estimators adapted from genomic studies and find that the former achieves lower bias and variance under both ideal settings and model assumption misspecification. Using neurocognitive data from the Philadelphia Neurodevelopmental Cohort, we demonstrate that greater sustained attention is associated with stronger speed-accuracy coupling in a complex reasoning task while controlling for age. By highlighting conditional correlations as the outcome of interest, our model provides complementary insights to traditional regression modeling and partitioned correlation analyses.

Tax proceeds and fiscal advancement: Pragmatic configuration in Cross River State, Nigeria

This study examined tax proceeds and fiscal advancement: pragmatic configuration in Cross River State, Nigeria. The specific objectives of the study were to determine the relationship between personal proceeds tax and fiscal advancement of Cross River State and the extent to which tourism development tax affects the fiscal advancement of Cross River State. The study adopted the ex-post facto research design and scope comprised of the entire data related to personal proceeds tax and tourism development tax from the Cross River State Internal Revenue Service from 1990-2020. This study adopted the classical ordinary least squares regression techniques as the technique of estimation. Findings from the study revealed that, tax proceeds exerted a negative and insignificant effect on fiscal advancement in Cross River State. The study therefore recommended that, regarding road tax, there should be institutional reforms in order to plug leakages and enhance collection mechanism used by tax officials so as to free the collection of these taxes from corrupt tax payers and minimized embezzlement by insincere tax collectors. Government should make it impossible for companies to evade taxes by penalizing any registered company that evades tax.

More Accurate Predictions of Daily Soil Temperature

More Accurate Predictions of Daily Soil Temperature

Machine Learning Approaches for Hospital Acquired Pressure Injuries: A Retrospective Study of Electronic Medical Records.

BackgroundMany machine learning heuristics integrate well with Electronic Medical Record (EMR) systems yet often fail to surpass traditional statistical models for biomedical applications.ObjectiveWe sought to compare predictive performances of 12 machine learning and traditional statistical techniques to predict the occurrence of Hospital Acquired Pressure Injuries (HAPI).MethodsEMR information was collected from 57,227 hospitalizations acquired from Dartmouth Hitchcock Medical Center (April 2011 to December 2016). Twelve classification algorithms, chosen based upon classic regression and recent machine learning techniques, were trained to predict HAPI incidence and performance was assessed using the Area Under the Receiver Operating Characteristic Curve (AUC).ResultsLogistic regression achieved a performance (AUC = 0.91 ± 0.034) comparable to the other machine learning approaches. We report discordance between machine learning derived predictors compared to the traditional statistical model. We visually assessed important patient-specific factors through Shapley Additive Explanations.ConclusionsMachine learning models will continue to inform clinical decision-making processes but should be compared to traditional modeling approaches to ensure proper utilization. Disagreements between important predictors found by traditional and machine learning modeling approaches can potentially confuse clinicians and need to be reconciled. These developments represent important steps forward in developing real-time predictive models that can be integrated into EMR systems to reduce unnecessary harm.

Infrared spectroscopy of quasi-ideal binary liquid mixtures: The challenges of conventional chemometric regression

We have recorded ATR-IR spectra of binary mixtures in the (quasi-)ideal systems Benzene-Toluene, Benzene-Carbon tetrachloride and Benzene-Cyclohexane and performed classical least squares, inverse least squares and principal component regression on the resulting spectra. In contrast to the general expectation, the spectra of ideal mixtures follow only roughly Beer’s approximation, in particular stronger bands show shifts and increased intensities for intermediary compositions since the polarization of matter by light cannot be neglected. As a consequence, these conventional regression techniques lead to principle and unavoidable errors, even though some of the classical regression techniques are assumed to be able to cope with nonlinearities. In particular in the system Benzene-Carbon tetrachloride large errors result and the relative average error of the volume fraction determination is about 10 % for all three methods. Especially remarkable is that the multivariate regression methods do not perform better than the classical univariate calibration if for the latter a peak due to an oscillator with comparably low oscillator strength is selected, since for such bands polarization effects are weak and Beer’s approximation holds comparably well.

Intelligent cost-effective winter road maintenance by predicting road surface temperature using machine learning techniques

Since Winter Road Maintenance (WRM) is an important activity in Nordic countries, accurate intelligent cost-effective WRM can create precise advance plans for developing decision support systems to improve traffic safety on the roads, while reducing cost and negative environmental impacts. Lack of comprehensive knowledge and inaccurate WRM information would lead to a certain loss of WRM budget, safety reduction, and irreparable environmental damage. This study proposes an intelligent methodology that uses data envelopment analysis and machine learning techniques. In the proposed methodology, WRM efficiency is calculated by data envelopment analysis for different decision-making units (roads), and inefficient units need to be considered for further assessments. Therefore, road surface temperature is predicted by means of machine learning methods, in order to achieve efficient and effective WRM on the roads during winter in cold regions. In total, four different methods have been used to predict road surface temperature on an inefficient road. One of these is linear regression, which is a classical statistical regression technique (ordinary least square regression); the other three methods are machine-learning techniques, including support vector regression, multilayer perceptron artificial neural network, and random forest regression. Graphical and numerical results indicate that support vector regression is the most accurate method.

A Map-Based Recommendation System and House Price Prediction Model for Real Estate

In 2015, global real estate was worth $217 trillion, which is approximately 2.7 times the global GDP; it also accounts for roughly 60% of all conventional global resources, making it one of the key factors behind any country’s economic growth and stability. The accessibility of spatial big data will help real estate investors make better judgement calls and earn additional profit. Since location is deemed necessary for real estate and consequent decision-making, digital maps have become a prime resource for real estate purchases, planning and development. Personalisation can assist in making judgments by identifying user desires and inclinations, which can then be recorded or captured as a user performs some interactions with a digital map. A personalised real estate portal can use this information to suggest properties, assist homeowners and provide valuable real estate analytics. This article presents a novel framework for recommending real estate to users. By monitoring user interactions through an online real estate portal, the framework can make personalised recommendations of real estate based on content, collaboration and location. The effectiveness of the recommendations was tested by the user feedback mechanism through a method of mean absolute precision, and the results show that 79% precise suggestions were generated, i.e., out of 5 recommendations produced, users were interested in at least 3. Along with that, a separate house price prediction model based on neural networks and classical regression techniques was also implemented to assist users in making an informed decision regarding prospects of real estate purchase.

Artificial neural network application in predicting probabilistic seismic demands of bridge components

AbstractProbabilistic seismic demand models (PSDMs) of bridge components such as column and abutment are commonly developed through classical linear regression techniques in which a univariate model format is predefined in the logarithmically transformed space. The more advanced machine learning (ML)‐based PSDMs incorporate various sources of uncertainties, which eventually leads to a more credible prediction of the seismic demands of bridge components and enhances the vulnerability assessment of the overall bridge systems. Despite the emerging advancements in ML approaches, many of them have not yet been introduced to estimate bridge seismic responses. To this end, the present study seeks to develop predictive PSDMs using a reputable ML approach, the artificial neural network (ANN). Relative to the classical univariate PSDMs, the ANN‐based PSDMs improve the median estimation of demands, particularly over the large and small range of ground motion intensities and reduce the total prediction variability. Moreover, the proposed ANN‐based approach provides a generalizable model with an unbiased prediction of the seismic demands. The ANN‐based PSDMs can be further used in estimating the probability of structural damage in the fragility and risk assessment process.

Surrogate parametric metamodel based on Optimal Transport

The description of a physical problem through a model necessarily involves the introduction of parameters. Hence, one wishes to have a solution of the problem that is a function of all these parameters: a parametric solution. However, the construction of such parametric solutions exhibiting localization in space is only ensured by costly and time-consuming tests, which can be both numerical or experimental. Numerical methodologies used classically imply enormous computational efforts for exploring the design space. Therefore, parametric solutions obtained using advanced nonlinear regressions are an essential tool to address this challenge. However, classical regression techniques, even the most advanced ones, can lead to non physical interpolation in some fields such as fluid dynamics, where the solution localizes in different regions depending on the problem parameters choice. In this context, Optimal Transport (OT) offers a mathematical approach to measure distances and interpolate between general objects in a, sometimes, more physical way than the classical interpolation approach. Thus, OT has become fundamental in some fields such as statistics or computer vision, and it is being increasingly used in fields such as computational mechanics. However, the OT problem is usually computationally costly to solve and not adapted to be accessed in an online manner. Therefore, the aim of this paper is combining advanced nonlinear regressions with Optimal Transport in order to implement a parametric real-time model based on OT. To this purpose, a parametric model is built offline relying on Model Order Reduction and OT, leading to a real-time interpolation tool following Optimal Transport theory. Such a tool is of major interest in design processes, but also within the digital twin rationale.

Detection of Influential Observations in Spatial Regression Model Based on Outliers and Bad Leverage Classification

Influential observations (IOs), which are outliers in the x direction, y direction or both, remain a problem in the classical regression model fitting. Spatial regression models have a peculiar kind of outliers because they are local in nature. Spatial regression models are also not free from the effect of influential observations. Researchers have adapted some classical regression techniques to spatial models and obtained satisfactory results. However, masking or/and swamping remains a stumbling block for such methods. In this article, we obtain a measure of spatial Studentized prediction residuals that incorporate spatial information on the dependent variable and the residuals. We propose a robust spatial diagnostic plot to classify observations into regular observations, vertical outliers, good and bad leverage points using a classification based on spatial Studentized prediction residuals and spatial diagnostic potentials, which we refer to as ISRs−Posi and ESRs−Posi. Observations that fall into the vertical outliers and bad leverage points categories are referred to as IOs. Representations of some classical regression measures of diagnostic in general spatial models are presented. The commonly used diagnostic measure in spatial diagnostics, the Cook’s distance, is compared to some robust methods, Hi2 (using robust and non-robust measures), and our proposed ISRs−Posi and ESRs−Posi plots. Results of our simulation study and applications to real data showed that the Cook’s distance, non-robust Hsi12 and robust Hsi22 were not very successful in detecting IOs. The Hsi12 suffered from the masking effect, and the robust Hsi22 suffered from swamping in general spatial models. Interestingly, the results showed that the proposed ESRs−Posi plot, followed by the ISRs−Posi plot, was very successful in classifying observations into the correct groups, hence correctly detecting the real IOs.