Low-data Limit Research Articles

Leveraging large language models for predictive chemistry

Machine learning has transformed many fields and has recently found applications in chemistry and materials science. The small datasets commonly found in chemistry sparked the development of sophisticated machine learning approaches that incorporate chemical knowledge for each application and, therefore, require specialized expertise to develop. Here we show that GPT-3, a large language model trained on vast amounts of text extracted from the Internet, can easily be adapted to solve various tasks in chemistry and materials science by fine-tuning it to answer chemical questions in natural language with the correct answer. We compared this approach with dedicated machine learning models for many applications spanning the properties of molecules and materials to the yield of chemical reactions. Surprisingly, our fine-tuned version of GPT-3 can perform comparably to or even outperform conventional machine learning techniques, in particular in the low-data limit. In addition, we can perform inverse design by simply inverting the questions. The ease of use and high performance, especially for small datasets, can impact the fundamental approach to using machine learning in the chemical and material sciences. In addition to a literature search, querying a pre-trained large language model might become a routine way to bootstrap a project by leveraging the collective knowledge encoded in these foundation models, or to provide a baseline for predictive tasks.

Regularized regressions for parametric models based on separated representations

Regressions created from experimental or simulated data enable the construction of metamodels, widely used in a variety of engineering applications. Many engineering problems involve multi-parametric physics whose corresponding multi-parametric solutions can be viewed as a sort of computational vademecum that, once computed offline, can be then used in a variety of real-time engineering applications including optimization, inverse analysis, uncertainty propagation or simulation based control. Sometimes, these multi-parametric problems can be solved by using advanced model order reduction—MOR-techniques. However, solving these multi-parametric problems can be very costly. In that case, one possibility consists in solving the problem for a sample of the parametric values and creating a regression from all the computed solutions. The solution for any choice of the parameters is then inferred from the prediction of the regression model. However, addressing high-dimensionality at the low data limit, ensuring accuracy and avoiding overfitting constitutes a difficult challenge. The present paper aims at proposing and discussing different advanced regressions based on the proper generalized decomposition (PGD) enabling the just referred features. In particular, new PGD strategies are developed adding different regularizations to the s-PGD method. In addition, the ANOVA-based PGD is proposed to ally them.

Material Forming Digital Twins: The Alliance between Physics-Based and Data-Driven Models

This paper aims at introducing the main building blocks of a digital twin, embracing physics-based and data-driven functionalities, both enriching mutually. Both should proceed in almost real-time, and the last being able to proceed in the scarce data limit. When applied to materials and processes, model order reduction technologies enable the construction of the so-called “computational vademecum”, whereas data-driven modelling, based in advanced regressions, must be informed by the physics to encompass rapidity and accuracy, in the low data limit. Despite of the recent advances, a lot of functionalities are needed and are under progress, some of them representing real scientific challenges. A number of them, the ones that we estimate being the most crucial, will be discussed in the present work.

Real-time prediction by data-driven models applied to induction heating process

Data-driven modeling approach constitutes an appealing alternative to the finite element method for optimizing complex multiphysics parametrized problems. In this context, this paper aims at proposing a parametric solution for the temperature-time evolution during the multiphysics induction heating process by using a data-driven non-intrusive modeling approach. To achieve this goal, firstly, a set of synthetic solutions was collected, at some sparse sensors in the space domain and for properly selected process parameters, by solving the full-order finite element models using FORGE® software. Then, the gappy proper orthogonal decomposition method was used to complete the missing data. Next, the proper orthogonal decomposition method coupled with the nonlinear sparse proper generalized decomposition regression method was applied to find a low-dimensional space onto which the original solutions were projected and a model for the low-dimensional representations was, therefore, created. Hence, a real-time prediction of the temperature-time evolution and for any new process parameters could be efficiently computed at the predefined positions (sensors) in the space domain. Finally, spatial interpolation was carried out to extend the solutions everywhere in the spatial domain by applying a strategy based on the nonlinear dimensionality reduction by locally linear embedding method and the proper orthogonal decomposition method with radial basis functions interpolation. It was shown that the results are promising and the applied approaches provide good approximations in the low-data limit case.

Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control.

Sparse model identification enables the discovery of nonlinear dynamical systems purely from data; however, this approach is sensitive to noise, especially in the low-data limit. In this work, we leverage the statistical approach of bootstrap aggregating (bagging) to robustify the sparse identification of the nonlinear dynamics (SINDy) algorithm. First, an ensemble of SINDy models is identified from subsets of limited and noisy data. The aggregate model statistics are then used to produce inclusion probabilities of the candidate functions, which enables uncertainty quantification and probabilistic forecasts. We apply this ensemble-SINDy (E-SINDy) algorithm to several synthetic and real-world datasets and demonstrate substantial improvements to the accuracy and robustness of model discovery from extremely noisy and limited data. For example, E-SINDy uncovers partial differential equations models from data with more than twice as much measurement noise as has been previously reported. Similarly, E-SINDy learns the Lotka Volterra dynamics from remarkably limited data of yearly lynx and hare pelts collected from 1900 to 1920. E-SINDy is computationally efficient, with similar scaling as standard SINDy. Finally, we show that ensemble statistics from E-SINDy can be exploited for active learning and improved model predictive control.

Data-Driven Modelling of Polyethylene Recycling under High-Temperature Extrusion.

Two main problems are studied in this article. The first one is the use of the extrusion process for controlled thermo-mechanical degradation of polyethylene for recycling applications. The second is the data-based modelling of such reactive extrusion processes. Polyethylenes (high density polyethylene (HDPE) and ultra-high molecular weight polyethylene (UHMWPE)) were extruded in a corotating twin-screw extruder under high temperatures (350 °C < T < 420 °C) for various process conditions (flow rate and screw rotation speed). These process conditions involved a decrease in the molecular weight due to degradation reactions. A numerical method based on the Carreau-Yasuda model was developed to predict the rheological behaviour (variation of the viscosity versus shear rate) from the in-line measurement of the die pressure. The results were successfully compared to the viscosity measured from offline measurement assuming the Cox-Merz law. Weight average molecular weights were estimated from the resulting zero-shear rate viscosity. Furthermore, the linear viscoelastic behaviours (Frequency dependence of the complex shear modulus) were also used to predict the molecular weight distributions of final products by an inverse rheological method. Size exclusion chromatography (SEC) was performed on five samples, and the resulting molecular weight distributions were compared to the values obtained with the two aforementioned techniques. The values of weight average molecular weights were similar for the three techniques. The complete molecular weight distributions obtained by inverse rheology were similar to the SEC ones for extruded HDPE samples, but some inaccuracies were observed for extruded UHMWPE samples. The Ludovic® (SC-Consultants, Saint-Etienne, France) corotating twin-screw extrusion simulation software was used as a classical process simulation. However, as the rheo-kinetic laws of this process were unknown, the software could not predict all the flow characteristics successfully. Finally, machine learning techniques, able to operate in the low-data limit, were tested to build predicting models of the process outputs and material characteristics. Support Vector Machine Regression (SVR) and sparsed Proper Generalized Decomposition (sPGD) techniques were chosen to predict the process outputs successfully. These methods were also applied to material characteristics data, and both were found to be effective in predicting molecular weights. More precisely, the sPGD gave better results than the SVR for the zero-shear viscosity prediction. Stochastic methods were also tested on some of the data and showed promising results.

Structured time-delay models for dynamical systems with connections to Frenet-Serret frame.

Time-delay embedding and dimensionality reduction are powerful techniques for discovering effective coordinate systems to represent the dynamics of physical systems. Recently, it has been shown that models identified by dynamic mode decomposition on time-delay coordinates provide linear representations of strongly nonlinear systems, in the so-called Hankel alternative view of Koopman (HAVOK) approach. Curiously, the resulting linear model has a matrix representation that is approximately antisymmetric and tridiagonal; for chaotic systems, there is an additional forcing term in the last component. In this paper, we establish a new theoretical connection between HAVOK and the Frenet–Serret frame from differential geometry, and also develop an improved algorithm to identify more stable and accurate models from less data. In particular, we show that the sub- and super-diagonal entries of the linear model correspond to the intrinsic curvatures in the Frenet–Serret frame. Based on this connection, we modify the algorithm to promote this antisymmetric structure, even in the noisy, low-data limit. We demonstrate this improved modelling procedure on data from several nonlinear synthetic and real-world examples.

Data-Driven Modeling for Multiphysics Parametrized Problems-Application to Induction Hardening Process

Data-driven modeling provides an efficient approach to compute approximate solutions for complex multiphysics parametrized problems such as induction hardening (IH) process. Basically, some physical quantities of interest (QoI) related to the IH process will be evaluated under real-time constraint, without any explicit knowledge of the physical behavior of the system. Hence, computationally expensive finite element models will be replaced by a parametric solution, called metamodel. Two data-driven models for temporal evolution of temperature and austenite phase transformation, during induction heating, were first developed by using the proper orthogonal decomposition based reduced-order model followed by a nonlinear regression method for temperature field and a classification combined with regression for austenite evolution. Then, data-driven and hybrid models were created to predict hardness, after quenching. It is shown that the results of artificial intelligence models are promising and provide good approximations in the low-data limit case.

A non-intrusive model order reduction approach for multi-physics parametrized problems - Application to induction heating process

Finite element modeling (FEM) has recently become the most attractive computational tool to predict and optimize many industrial problems. However, the FEM becomes ineffective as far as complex multi-physics parameterized problems, such as induction heating process, are concerned because of high computational cost. This work aims at studying the possibility of applying a new approach based on the reduced order modeling (ROM) to obtain approximate solutions of a parametric problem. Basically, the effect of induction heating process parameters on some physical quantities of interest (QoI) will be analyzed under the real-time constraint. To achieve this dimensionality reduction, a set of precomputed solutions is first collected, at some sparse points in the space domain and for a properly selected process parameters, by solving the full-order models implemented in the commercial finite element software FORGE®. A Proper Orthogonal Decomposition (POD) based reducedorder model is then applied to the collected data to find a low dimensional space onto which the solution manifold could be projected and an approximated solution for new process parameters could be efficiently computed in real time. Besides, the POD is applied to build a reduced basis and to compute their corresponding modal coefficients. It is then followed by artificial intelligence techniques for regression purpose, such as sparse Proper Generalized Decomposition, to fit the low dimensional POD modal coefficients. Hence, the problem can be solved with a much lower dimension compared to the initial one. It was shown that a good approximation of the QoI was provided, in low-data limit, using a single POD modal coefficient as a response for the regression methods. However, the obtained approximation accuracy needs to be enhanced.

Application of Machine Learning Tools for the Improvement of Reactive Extrusion Simulation

AbstractThe purpose of this paper is to combine a classical 1D twin‐screw extrusion model with machine learning techniques to obtain accurate predictions of a complex system despite few data. Systems involving reactive polyethylene oligomer dispersed in situ in a polypropylene matrix by reactive twin‐screw extrusion are studied for this purpose. The twin‐screw extrusion simulation software LUDOVIC is used and machine learning techniques dealing with low data limit are used as a correction of the simulation.

On the Data-Driven Modeling of Reactive Extrusion

This paper analyzes the ability of different machine learning techniques, able to operate in the low-data limit, for constructing the model linking material and process parameters with the properties and performances of parts obtained by reactive polymer extrusion. The use of data-driven approaches is justified by the absence of reliable modeling and simulation approaches able to predict induced properties in those complex processes. The experimental part of this work is based on the in situ synthesis of a thermoset (TS) phase during the mixing step with a thermoplastic polypropylene (PP) phase in a twin-screw extruder. Three reactive epoxy/amine systems have been considered and anhydride maleic grafted polypropylene (PP-g-MA) has been used as compatibilizer. The final objective is to define the appropriate processing conditions in terms of improving the mechanical properties of these new PP materials by reactive extrusion.

Incremental dynamic mode decomposition: A reduced-model learner operating at the low-data limit

The present work aims at proposing a new methodology for learning reduced models from a small amount of data. It is based on the fact that discrete models, or their transfer function counterparts, have a low rank and then they can be expressed very efficiently using few terms of a tensor decomposition. An efficient procedure is proposed as well as a way for extending it to nonlinear settings while keeping limited the impact of data noise. The proposed methodology is then validated by considering a nonlinear elastic problem and constructing the model relating tractions and displacements at the observation points.

The cooperative free volume rate model for segmental dynamics: Application to glass-forming liquids and connections with the density scaling approach⋆.

In this paper, we apply the cooperative free volume (CFV) rate model for pressure-dependent dynamics of glass-forming liquids and polymer melts. We analyze segmental relaxation times, [Formula: see text] , as a function of temperature (T and free volume ( [Formula: see text] , and make substantive comparisons with the density scaling approach. [Formula: see text] , the difference between the total volume (V and the volume at close-packing, is predicted independently of the dynamics for any temperature and pressure using the locally correlated lattice (LCL) equation-of-state (EOS) analysis of characteristic thermodynamic data. We discuss the underlying physical motivation in the CFV and density scaling models, and show that their key, respective, material parameters are connected, where the CFV b parameter and the density scaling [Formula: see text] parameter each characterize the relative sensitivity of dynamics to changes in T , vs. changes in V . We find [Formula: see text] , where [Formula: see text] is the value predicted by the LCL EOS at the ambient [Formula: see text] . In comparing the predictive power of the two models we highlight the CFV advantage in yielding a universal linear collapse of relaxation data using a minimal set of parameters, compared to the same parameter space yielding a changing functional form in the density scaling approach. Further, we demonstrate that in the low data limit, where there is not enough data to characterize the density scaling model, the CFV model may still be successfully applied, and we even use it to predict the correct [Formula: see text] parameter.

Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning.

Predicting bioactivity and physical properties of small molecules is a central challenge in drug discovery. Deep learning is becoming the method of choice but studies to date focus on mean accuracy as the main metric. However, to replace costly and mission-critical experiments by models, a high mean accuracy is not enough: outliers can derail a discovery campaign, thus models need to reliably predict when it will fail, even when the training data is biased; experiments are expensive, thus models need to be data-efficient and suggest informative training sets using active learning. We show that uncertainty quantification and active learning can be achieved by Bayesian semi-supervised graph convolutional neural networks. The Bayesian approach estimates uncertainty in a statistically principled way through sampling from the posterior distribution. Semi-supervised learning disentangles representation learning and regression, keeping uncertainty estimates accurate in the low data limit and allowing the model to start active learning from a small initial pool of training data. Our study highlights the promise of Bayesian deep learning for chemistry.

Advanced model order reduction and artificial intelligence techniques empowering advanced structural mechanics simulations: application to crash test analyses

This paper proposes a general framework for expressing parametrically quantities of interest related to the solution of complex structural mechanics models, in particular the ones involved in crash analyses where strongly coupled nonlinear and dynamic behaviors coexist with space-time localized mechanisms. Advanced nonlinear regressions able to proceed in the low-data limit, enabling to accommodate heterogeneous parameters, will be proposed and their performances evaluated in the case of crash simulations. As soon as these parametric expressions will be determined, they can be used for generating large amounts of realizations of the quantity of interest for different choices of the parameters, for supporting data-analytics. On the other hand, such parametric representations allow the use advanced optimization techniques, evaluate sensitivities and propagate uncertainty all them under the stringent real-time constraint.

Analytical and Critical Review from Cloud to Fog to Internet of Things (IoT)

Internet of Things (IoT) has become one of the crucial parts of our everyday life. So there must be rapid advancement in this field with the goal of better Quality of Service for IoT. But, with the advancement of Internet of Things (IoT) applications, the cloud computing model face few troubles, for instance low data limit and network issues. To solve these challenges, fog computing brings the cloud closer to Internet of Things (IoT). This paper presents the architecture of Internet of Things (IoT), a comparative analysis of fog and cloud, advantages of fog over cloud. It also presents the challenges and uses of fog computing w.r.t. Internet of Things.

Sparse identification of nonlinear dynamics for model predictive control in the low-data limit.

Data-driven discovery of dynamics via machine learning is pushing the frontiers of modelling and control efforts, providing a tremendous opportunity to extend the reach of model predictive control (MPC). However, many leading methods in machine learning, such as neural networks (NN), require large volumes of training data, may not be interpretable, do not easily include known constraints and symmetries, and may not generalize beyond the attractor where models are trained. These factors limit their use for the online identification of a model in the low-data limit, for example following an abrupt change to the system dynamics. In this work, we extend the recent sparse identification of nonlinear dynamics (SINDY) modelling procedure to include the effects of actuation and demonstrate the ability of these models to enhance the performance of MPC, based on limited, noisy data. SINDY models are parsimonious, identifying the fewest terms in the model needed to explain the data, making them interpretable and generalizable. We show that the resulting SINDY-MPC framework has higher performance, requires significantly less data, and is more computationally efficient and robust to noise than NN models, making it viable for online training and execution in response to rapid system changes. SINDY-MPC also shows improved performance over linear data-driven models, although linear models may provide a stopgap until enough data is available for SINDY. SINDY-MPC is demonstrated on a variety of dynamical systems with different challenges, including the chaotic Lorenz system, a simple model for flight control of an F8 aircraft, and an HIV model incorporating drug treatment.

Sparse identification of nonlinear dynamics for rapid model recovery.

Big data have become a critically enabling component of emerging mathematical methods aimed at the automated discovery of dynamical systems, where first principles modeling may be intractable. However, in many engineering systems, abrupt changes must be rapidly characterized based on limited, incomplete, and noisy data. Many leading automated learning techniques rely on unrealistically large data sets, and it is unclear how to leverage prior knowledge effectively to re-identify a model after an abrupt change. In this work, we propose a conceptual framework to recover parsimonious models of a system in response to abrupt changes in the low-data limit. First, the abrupt change is detected by comparing the estimated Lyapunov time of the data with the model prediction. Next, we apply the sparse identification of nonlinear dynamics (SINDy) regression to update a previously identified model with the fewest changes, either by addition, deletion, or modification of existing model terms. We demonstrate this sparse model recovery on several examples for abrupt system change detection in periodic and chaotic dynamical systems. Our examples show that sparse updates to a previously identified model perform better with less data, have lower runtime complexity, and are less sensitive to noise than identifying an entirely new model. The proposed abrupt-SINDy architecture provides a new paradigm for the rapid and efficient recovery of a system model after abrupt changes.