Nonlinear Machine Research Articles

Performance analysis of linear and non-linear machine learning models for forecasting compressive strength of concrete

Performance analysis of linear and non-linear machine learning models for forecasting compressive strength of concrete

Forecasting realized volatility: Does anything beat linear models?

We evaluate the performance of several linear and nonlinear machine learning (ML) models in forecasting the realized volatility (RV) of ten global stock market indices in the period from January 2000 to December 2021. We train models using a dataset that includes past values of the RV and additional predictors, including lagged returns, implied volatility, macroeconomic and sentiment variables. We compare these models to widely used heterogeneous autoregressive (HAR) models. Our main conclusions are that (i) the additional predictors improve the out-of-sample forecasts at the daily and weekly forecast horizons; (ii) we find no evidence that nonlinear ML models can statistically outperform linear models in general; and (iii) in terms of the economic value that an investor would derive from monthly RV forecasts to build volatility-timing portfolios, simpler models without additional predictors work better.

Integrating temporal decomposition and data-driven approaches for predicting coastal harmful algal blooms

Frequent coastal harmful algal blooms (HABs) threaten the ecological environment and human health. Biscayne Bay in southeastern Florida also faces algal bloom issues; however, the mechanisms driving these blooms are not fully understood, emphasizing the importance of HAB prediction for effective environmental management. The overarching goal of this study is to offer a robust HAB predictive framework and try to enhance the understanding of HAB dynamics. This study established three scenarios to predict chlorophyll-a concentrations, a recognized representative of HABs: Scenario 1 (S1) using single nonlinear machine learning (ML) algorithms, hybrid Scenario 2 (S2) combining linear models and nonlinear ML algorithms, and hybrid Scenario 3 (S3) combining temporal decomposition and ML (TD-ML) algorithms. The novel-developed S3 TD-ML hybrid models demonstrated superior predictive capabilities, achieving all R2 values above 0.9 and MAPE under 30% in tests, significantly outperforming the S1 with an average R2 of 0.16 and the S2 with an R2 of −0.06. S3 models effectively captured the algal dynamics, successfully predicting complex time series with extremes and noise. In addition, we unveiled the relationship between environmental variables and chlorophyll-a through correlation analysis and found that climate change might intensify the HABs in Biscayne Bay. This research developed a precise predictive framework for early warning and proactive management of HABs, offering potential global applicability and improved prediction accuracy to address HAB challenges.

Epistatic Features and Machine Learning Improve Alzheimer's Disease Risk Prediction Over Polygenic Risk Scores.

Polygenic risk scores (PRS) are linear combinations of genetic markers weighted by effect size that are commonly used to predict disease risk. For complex heritable diseases such as late-onset Alzheimer's disease (LOAD), PRS models fail to capture much of the heritability. Additionally, PRS models are highly dependent on the population structure of the data on which effect sizes are assessed and have poor generalizability to new data. The goal of this study is to construct a paragenic risk score that, in addition to single genetic marker data used in PRS, incorporates epistatic interaction features and machine learning methods to predict risk for LOAD. We construct a new state-of-the-art genetic model for risk of Alzheimer's disease. Our approach innovates over PRS models in two ways: First, by directly incorporating epistatic interactions between SNP loci using an evolutionary algorithm guided by shared pathway information; and second, by estimating risk via an ensemble of non-linear machine learning models rather than a single linear model. We compare the paragenic model to several PRS models from the literature trained on the same dataset. The paragenic model is significantly more accurate than the PRS models under 10-fold cross-validation, obtaining an AUC of 83% and near-clinically significant matched sensitivity/specificity of 75%. It remains significantly more accurate when evaluated on an independent holdout dataset and maintains accuracy within APOE genotype strata. Paragenic models show potential for improving disease risk prediction for complex heritable diseases such as LOAD over PRS models.

Machine learning models for predicting blood pressure phenotypes by combining multiple polygenic risk scores

We construct non-linear machine learning (ML) prediction models for systolic and diastolic blood pressure (SBP, DBP) using demographic and clinical variables and polygenic risk scores (PRSs). We developed a two-model ensemble, consisting of a baseline model, where prediction is based on demographic and clinical variables only, and a genetic model, where we also include PRSs. We evaluate the use of a linear versus a non-linear model at both the baseline and the genetic model levels and assess the improvement in performance when incorporating multiple PRSs. We report the ensemble model’s performance as percentage variance explained (PVE) on a held-out test dataset. A non-linear baseline model improved the PVEs from 28.1 to 30.1% (SBP) and 14.3% to 17.4% (DBP) compared with a linear baseline model. Including seven PRSs in the genetic model computed based on the largest available GWAS of SBP/DBP improved the genetic model PVE from 4.8 to 5.1% (SBP) and 4.7 to 5% (DBP) compared to using a single PRS. Adding additional 14 PRSs computed based on two independent GWASs further increased the genetic model PVE to 6.3% (SBP) and 5.7% (DBP). PVE differed across self-reported race/ethnicity groups, with primarily all non-White groups benefitting from the inclusion of additional PRSs. In summary, non-linear ML models improves BP prediction in models incorporating diverse populations.

Multi-Endpoint Acute Toxicity Assessment of Organic Compounds Using Large-Scale Machine Learning Modeling.

In recent years, alternative animal testing methods such as computational and machine learning approaches have become increasingly crucial for toxicity testing. However, the complexity and scarcity of available biomedical data challenge the development of predictive models. Combining nonlinear machine learning together with multicondition descriptors offers a solution for using data from various assays to create a robust model. This work applies multicondition descriptors (MCDs) to develop a QSTR (Quantitative Structure-Toxicity Relationship) model based on a large toxicity data set comprising more than 80,000 compounds and 59 different end points (122,572 data points). The prediction capabilities of developed single-task multi-end point machine learning models as well as a novel data analysis approach with the use of Convolutional Neural Networks (CNN) are discussed. The results show that using MCDs significantly improves the model and using them with CNN-1D yields the best result (R2train = 0.93, R2ext = 0.70). Several structural features showed a high level of contribution to the toxicity, including van der Waals surface area (VSA), number of nitrogen-containing fragments (nN+), presence of S-P fragments, ionization potential, and presence of C-N fragments. The developed models can be very useful tools to predict the toxicity of various compounds under different conditions, enabling quick toxicity assessment of new compounds.

A neural network framework for portfolio optimization under second-order stochastic dominance

Traditional portfolio optimization strategies often rely on statistical methods and linear programming tools to achieve a balance between return and risk. Despite their usefulness for portfolio optimization, they cannot efficiently capture the specific differences and complexity of real-world financial markets. Several modern approaches attempt to overcome these limitations by using nonlinear models, machine learning, and advanced risk measures. In this study, we propose a novel strategy for optimizing portfolios that incorporates second-order stochastic dominance constraints and solves them with neural networks. we show that portfolios subject to second-order stochastic dominance constraints outperform their traditional counterparts, especially in tail-risk situations.

Performance of double-arch tunnels under internal BLEVE

Double-arch tunnels, as one of the popular forms of tunnels, might be exposed to boiling liquid expanding vapour explosions (BLEVEs) associated with transported liquified petroleum gas (LPG), which could cause damage to the tunnel and even catastrophic collapse of the tunnel in extreme cases. However, very limited study has investigated the performance of double-arch tunnels when exposed to internal BLEVEs and in most analyses of tunnel responses to accidental explosions. The TNT-equivalence method was used to approximate the explosion load, which may lead to inaccurate tunnel response predictions. This study numerically investigates the response of typical double-arch tunnels to an internal BLEVE resulting from the instantaneous rupture of a 20 m3 LPG tank. Effects of various factors, including in-situ stresses, BLEVE locations, and lining configurations on tunnel responses are examined. The results show that the double-arch tunnels at their early-operation ages are more vulnerable to severe damage when exposed to the BLEVE due to the low action of in-situ stress of rock mass on the response of early-age tunnels. It is also found that directing the LPG tank to different driving lanes inside tunnels can affect the BLEVE-induced tunnel response more significantly than varying the configurations of tunnel lining. Moreover, installing section-steel arches in the mid-wall can effectively improve the blast resistance of the double-arch tunnels against the internal BLEVE. In addition, the prediction models based on multi-variate nonlinear regressions and machine learning methods are developed to predict the BLEVE-induced damage levels of the double-arch tunnels without and with section-steel arches.

Bipartite networks represent causality better than simple networks: evidence, algorithms, and applications.

A network, whose nodes are genes and whose directed edges represent positive or negative influences of a regulatory gene and its targets, is often used as a representation of causality. To infer a network, researchers often develop a machine learning model and then evaluate the model based on its match with experimentally verified "gold standard" edges. The desired result of such a model is a network that may extend the gold standard edges. Since networks are a form of visual representation, one can compare their utility with architectural or machine blueprints. Blueprints are clearly useful because they provide precise guidance to builders in construction. If the primary role of gene regulatory networks is to characterize causality, then such networks should be good tools of prediction because prediction is the actionable benefit of knowing causality. But are they? In this paper, we compare prediction quality based on "gold standard" regulatory edges from previous experimental work with non-linear models inferred from time series data across four different species. We show that the same non-linear machine learning models have better predictive performance, with improvements from 5.3% to 25.3% in terms of the reduction in the root mean square error (RMSE) compared with the same models based on the gold standard edges. Having established that networks fail to characterize causality properly, we suggest that causality research should focus on four goals: (i) predictive accuracy; (ii) a parsimonious enumeration of predictive regulatory genes for each target gene g; (iii) the identification of disjoint sets of predictive regulatory genes for each target g of roughly equal accuracy; and (iv) the construction of a bipartite network (whose node types are genes and models) representation of causality. We provide algorithms for all goals.

The impact of environmental variables on surface Conductance: Advancing simulation with a nonlinear Machine learning model

Surface conductance (Gs) is a key factor in the Penman-Monteith (PM) equation; the interaction between environmental variables such as CO2 concentration, air temperature (TA), vapor pressure deficit (VPD), soil water content (SWC), and net radiation (R) affects Gs, evapotranspiration and thus impacts the hydrological cycle. These interactions are highly nonlinear and vary among different vegetation types. However, conventional Gs simulation models use fixed interactions between environmental variables in their equations for all vegetation types. Moreover, the characterisation and parameterisation of conventional Gs models is highly uncertain due to the high spatiotemporal variability in key environmental variables and plant parameters, which inhibits their generalisation. This study investigates whether Gs could be estimated more accurately by nonlinear statistical techniques that capture the multiple interactions between the environmental variables that affect Gs for each vegetation type. We compare mixed generalized additive model (MGAM) for Gs simulation with semi-empirical and empirical models at 20 eddy covariance flux tower sites with four different vegetation types at daily and monthly timescales. The results show that the Nash-Sutcliffe Efficiency (NSE) in Gs simulation increased by up to 50% in MGAM model in comparison to the semi-empirical and empirical models. The MGAM model highlighted the interactive effects of CO2, VPD, and SWC for crops and grasses. The interactive effects of CO2, VPD, and TA were important for trees and grasses. The results from this study expand our understanding of the ability of Gs simulation models to identify and include the interactive effects of crucial environmental variables on plant transpiration and hydrological processes.

Predicting 2-year neurodevelopmental outcomes in preterm infants using multimodal structural brain magnetic resonance imaging with local connectivity

The neurodevelopmental outcomes of preterm infants can be stratified based on the level of prematurity. We explored brain structural networks in extremely preterm (EP; < 28 weeks of gestation) and very-to-late (V-LP; ≥ 28 and < 37 weeks of gestation) preterm infants at term-equivalent age to predict 2-year neurodevelopmental outcomes. Using MRI and diffusion MRI on 62 EP and 131 V-LP infants, we built a multimodal feature set for volumetric and structural network analysis. We employed linear and nonlinear machine learning models to predict the Bayley Scales of Infant and Toddler Development, Third Edition (BSID-III) scores, assessing predictive accuracy and feature importance. Our findings revealed that models incorporating local connectivity features demonstrated high predictive performance for BSID-III subsets in preterm infants. Specifically, for cognitive scores in preterm (variance explained, 17%) and V-LP infants (variance explained, 17%), and for motor scores in EP infants (variance explained, 15%), models with local connectivity features outperformed others. Additionally, a model using only local connectivity features effectively predicted language scores in preterm infants (variance explained, 15%). This study underscores the value of multimodal feature sets, particularly local connectivity, in predicting neurodevelopmental outcomes, highlighting the utility of machine learning in understanding microstructural changes and their implications for early intervention.

A hybrid approach integrating genetic algorithm and machine learning to solve the order picking batch assignment problem considering learning and fatigue of pickers

Modeling human behaviors has become increasingly relevant to improving the performance of manual order-picking systems. However, although a vast corpus of literature has recently started to consider the human factors in these systems, several gaps remain uncovered. Specifically, mental and physical human factors, like learning and fatigue, and quantitative and spatial features of picking orders have never been considered jointly to estimate the time a human order picker requires to execute a specific picking mission. Furthermore, little attention has been given to assigning and sequencing orders to pickers to minimize the picking time acting on their individual learning and fatigue characteristics. This study thus proposes a novel approach integrating machine learning and genetic algorithms to solve the problem. A non-linear machine learning-based predictive model has been adopted to predict the picking time of batches of orders based on quantitative and spatial features of batches and learning and fatigue indicators of pickers. These predictions have thus been adopted to guide a genetic algorithm to find the best assignment of future planned batches of orders to pickers. One year of picking data collected from the warehouse of a grocery retailer has been adopted to investigate the potential of the proposed approach. Furthermore, multiple comparisons have been performed. First, the advantages of predicting the batch-picking time with the proposed non-linear model have been compared with predictions executed based on linear models. In addition, an ablation analysis has been performed to investigate the advantages of predicting the batch picking time while simultaneously considering the quantitative and spatial features of batches and the learning and fatigue indicators of pickers. Moreover, the advantages of the proposed batch assignment strategy, which considers learning and fatigue indicators, have been compared with an assignment strategy that does not optimize these elements. Lastly, an explainability analysis of the predictive model has been performed to understand how and how much quantitative and spatial features of batches and learning and fatigue indicators of pickers affect the batch picking time.

The Importance of Reaction Energy in Predicting Chemical Reaction Barriers with Machine Learning Models.

Improving our fundamental understanding of complex heterocatalytic processes increasingly relies on electronic structure simulations and microkinetic models based on calculated energy differences. In particular, calculation of activation barriers, usually achieved through compute-intensive saddle point search routines, remains a serious bottleneck in understanding trends in catalytic activity for highly branched reaction networks. Although the well-known Brønsted-Evans-Polyani (BEP) scaling - a one-feature linear regression model - has been widely applied in such microkinetic models, they still rely on calculated reaction energies and may not generalize beyond a single facet on a single class of materials, e. g., a terrace sites on transition metals. For highly branched and energetically shallow reaction networks, such as electrochemical CO2 reduction or wastewater remediation, calculating even reaction energies on many surfaces can become computationally intractable due to the combinatorial explosion of states that must be considered. Here, we investigate the feasibility of activation barrier prediction without knowledge of the reaction energy using linear and nonlinear machine learning (ML) models trained on a new database of over 500 dehydrogenation activation barriers. We also find that inclusion of the reaction energy significantly improves both classes of ML models, but complex nonlinear models can achieve performance similar to the simplest BEP scaling when predicting activation barriers on new systems. Additionally, inclusion of the reaction energy significantly improves generalizability to new systems beyond the training set. Our results suggest that the reaction energy is a critical feature to consider when building models to predict activation barriers, indicating that efforts to reliably predict reaction energies through, e. g., the Open Catalyst Project and others, will be an important route to effective model development for more complex systems.

Estimation of instantaneous peak flows in Canadian rivers: an evaluation of conventional, nonlinear regression, and machine learning methods.

Instantaneous peak flows (IPFs) are often required to derive design values for sizing various hydraulic structures, such as culverts, bridges, and small dams/levees, in addition to informing several water resources management-related activities. Compared to mean daily flows (MDFs), which represent averaged flows over a period of 24 h, information on IPFs is often missing or unavailable in instrumental records. In this study, conventional methods for estimating IPFs from MDFs are evaluated and new methods based on the nonlinear regression framework and machine learning architectures are proposed and evaluated using streamflow records from all Canadian hydrometric stations with natural and regulated flow regimes. Based on a robust model selection criterion, it was found that multiple methods are suitable for estimating IPFs from MDFs, which precludes the idea of a single universal method. The performance of machine learning-based methods was also found reasonable compared to conventional and regression-based methods. To build on the strengths of individual methods, the fusion modeling concept from the machine learning area was invoked to synthesize outputs of multiple methods. The study findings are expected to be useful to the climate change adaptation community, which currently heavily relies on MDFs simulated by hydrologic models.

Contribution to advancing aquifer geometric mapping using machine learning and deep learning techniques: a case study of the AL Haouz-Mejjate aquifer, Marrakech, Morocco

Groundwater resources in Morocco often face sustainability challenges due to increased exploitation and climate change. Specifically, the Al-Haouz-Mejjate groundwater in the Marrakesh region is faced with overexploitation and insufficient recharge. However, the complex subsurface geometries hamper hydrogeological modeling, characterization, and effective management. Reliably estimating aquifer substrate topography is critical for groundwater models but is challenged by limited direct measurements. This study develops nonlinear machine learning models to infer substrate depths by fusing sparse borehole logs with regional geospatial data. A Gaussian process regression approach provided robust holistic mapping, leveraging flexibility, and uncertainty quantification. Supplementary neural network architectures focus on isolating specific variable relationships, like surface elevation–substrate. Model accuracy exceeded 0.8 R-squared against validation boreholes. Spatial visualizations confirmed consistency across landscape transects. Elevation and piezometric data proved most predictive, though multivariate inputs were required for the lowest errors. The results highlight the power of statistical learning to extract meaningful patterns from disparate hydrological data. However, model opacity and the need for broader training datasets remain barriers. Overall, the work demonstrates advanced machine learning as a promising avenue for illuminating complex aquifer geometries essential for sustainability. Hybrid approaches that use both data-driven and physics-based methods can help solve long-standing problems with hydrogeological characterization.

Tensor Product Alternatives for Nonlinear Field-Oriented Control of Induction Machines

The paper presents a nonlinear field-oriented control technique based on the tensor product representation of the nonlinear induction machine model and the solvability of linear matrix inequalities. The nonlinear model has 32 quasi linear parameter-varying equivalent variants, and it is shown that only half of the models result in feasible controller. Two control goals are realized: torque control and speed control. The controller is a nonlinear state feedback controller completed by integral action. A new block diagram is investigated for speed control. The controller gains are designed by the solution of linear matrix inequalities to solve the Lyapunov inequality to obtain a stable and fast response and constraints on the control signal. The presented methods are verified and compared by simulations.

Seismic Discrimination Between Nuclear Explosions and Natural Earthquakes using Multi-Machine Learning Techniques

AbstractIn the field of seismic signal analysis, it is of utmost importance to accurately differentiate between earthquakes and underground nuclear explosions. As a contribution for the verification regime of the Comprehensive Nuclear Test Ban Treaty (CTBT), Various methods have been employed for this purpose, including Complexity, Spectral ratio, mb—Ms (body wave and surface wave magnitudes), and corner frequency of P and S waves. These discrimination techniques have been examined to manually identify natural seismic events from nuclear explosions across different regions worldwide, such as China, India, Pakistan, North Korea, and the United States. To gather the necessary data, a comprehensive dataset comprising nuclear explosions and earthquakes of the same magnitude range (4 ≤ mb ≤ 6.5) of 35 seismic events from 1945 to 2017 has been compiled from the International Research Institute for Seismology (IRIS) using broadband and long period seismic stations. The objective of this study is to employ a range of linear and nonlinear Machine Learning (ML) models with the aim of automatically distinguishing between underground nuclear explosions and large earthquakes to enhance the accuracy of manual feature extraction. For this purpose, time domain waveforms and different classifier techniques focused on feature extraction have been used. The ML models employed include logistic regression, K-nearest neighbours classifier, decision tree classifier, random forest classifier, voting classifier, and Naive Bayes. The outcomes of the ROC and AUC analyses were employed to validate the validity of our proposed discrimination algorithm. The results show that the Random Forest Classifier is the most effective model, obtaining 100% accuracy in the case of feature extraction, while the best model for the time domain waveform classifier that achieved 75.5% accuracy is the voting classifier.

Sex determination based on craniometric parameters: a comparative approach between linear and non-linear machine learning algorithms

Introduction: Determining sex based on cranial characteristics is of great relevance in forensic anthropology. Most studies have employed linear methods (such as logistic regression) for this estimation with accuracies around 70%, rarely exceeding 90% accuracy. Several authors have tested non-linear models such as neural networks, support vector machines, and decision trees with good results, surpassing linear models. Objective: To compare linear models (logistic regression, linear regression, and linear discriminant analysis) with non-linear models (neural networks, extreme gradient boosting, support vector machine, naive Bayes, random forest, decision tree, k-nearest neighbors, and adaptive multivariate spline regression). Materials and Methods: 241 skulls used in this study were obtained from the collection of Center for Study and Research in Anatomy and Forensic Anthropology at Tiradentes University, Farolândia campus in Aracaju, Sergipe. Each skull in the collection has secure detailed records. Eighty-nine skulls with signs of craniotomy (n=58) or damage (n=30) and one unidentified were excluded. The 152 eligible skulls underwent cranial measurements. Using the Anaconda platform and Jupyter editor, the data were divided into a training group (80% of the sample) and then were tested (20% of the sample). Eleven machine learning algorithms, including both linear and non-linear models, were applied. Results: The best machine learning algorithm was a neural network with average accuracy of 93%, after 50 runs. The difference to logistic regression, which had an accuracy of 68%, was significantly (p-value of 0.01016). Conclusion: This study demonstrated the potential of the neural network for solving the sex classification problem. The study has a limitation in that neural networks perform better with a large volume of data, and this study used data from a single center. Nevertheless, in the future, more studies should be conducted testing neural networks with larger samples and skulls from other continents.

Predictive modeling of skin permeability for molecules: Investigating FDA-approved drug permeability with various AI algorithms.

The transdermal route of drug administration has gained popularity for its convenience and bypassing the first-pass metabolism. Accurate skin permeability prediction is crucial for successful transdermal drug delivery (TDD). In this study, we address this critical need to enhance TDD. A dataset comprising 441 records for 140 molecules with diverse LogKp values was characterized. The descriptor calculation yielded 145 relevant descriptors. Machine learning models, including MLR, RF, XGBoost, CatBoost, LGBM, and ANN, were employed for regression analysis. Notably, LGBM, XGBoost, and gradient boosting models outperformed others, demonstrating superior predictive accuracy. Key descriptors influencing skin permeability, such as hydrophobicity, hydrogen bond donors, hydrogen bond acceptors, and topological polar surface area, were identified and visualized. Cluster analysis applied to the FDA-approved drug dataset (2326 compounds) revealed four distinct clusters with significant differences in molecular characteristics. Predicted LogKp values for these clusters offered insights into the permeability variations among FDA-approved drugs. Furthermore, an investigation into skin permeability patterns across 83 classes of FDA-approved drugs based on the ATC code showcased significant differences, providing valuable information for drug development strategies. The study underscores the importance of accurate skin permeability prediction for TDD, emphasizing the superior performance of nonlinear machine learning models. The identified key descriptors and clusters contribute to a nuanced understanding of permeability characteristics among FDA-approved drugs. These findings offer actionable insights for drug design, formulation, and prioritization of molecules with optimum properties, potentially reducing reliance on costly experimental testing. Future research directions include offering promising applications in pharmaceutical research and formulation within the burgeoning field of computer-aided drug design.

Data-driven nonlinear turbulent flow scaling with Buckingham Pi variables

Nonlinear machine learning for turbulent flows can exhibit robust performance even outside the range of training data. This is achieved when machine-learning models can accommodate scale-invariant characteristics of turbulent flow structures. This study presents a data-driven approach to reveal scale-invariant vortical structures across Reynolds numbers that provide insights for supporting nonlinear machine-learning-based studies of turbulent flows. To uncover conditions for which nonlinear models are likely to perform well, we use a Buckingham-Pi-based sparse nonlinear scaling to find the influence of the Pi groups on the turbulent flow data. We consider nonlinear scalings of the invariants of the velocity gradient tensor for an example of three-dimensional decaying isotropic turbulence. The present scaling not only enables the identification of vortical structures that are interpolatory and extrapolatory for the given flow field data but also captures non-equilibrium effects of the energy cascade. As a demonstration, the present findings are applied to machine-learning-based super-resolution analysis of three-dimensional isotropic turbulence. We show that machine-learning models reconstruct vortical structures well in the interpolatory space with reduced performance in the extrapolatory space revealed by the nonlinearly scaled invariants. The present approach enables us to depart from labelling turbulent flow data with a single parameter of Reynolds number and comprehensively examine the flow field to support training and testing of nonlinear machine-learning techniques.