Mori-Zwanzig Formalism Research Articles

Data-driven model reduction, Wiener projections, and the Koopman-Mori-Zwanzig formalism

Model reduction methods aim to describe complex dynamic phenomena using only relevant dynamical variables, decreasing computational cost, and potentially highlighting key dynamical mechanisms. In the absence of special dynamical features such as scale separation or symmetries, the time evolution of these variables typically exhibits memory effects. Recent work has found a variety of data-driven model reduction methods to be effective for representing such non-Markovian dynamics, but their scope and dynamical underpinning remain incompletely understood. Here, we study data-driven model reduction from a dynamical systems perspective. For both chaotic and randomly-forced systems, we show the problem can be naturally formulated within the framework of Koopman operators and the Mori-Zwanzig projection operator formalism. We give a heuristic derivation of a NARMAX (Nonlinear Auto-Regressive Moving Average with eXogenous input) model from an underlying dynamical model. The derivation is based on a simple construction we call Wiener projection, which links Mori-Zwanzig theory to both NARMAX and to classical Wiener filtering. We apply these ideas to the Kuramoto-Sivashinsky model of spatiotemporal chaos and a viscous Burgers equation with stochastic forcing.

Reconstructing complex system dynamics from time series: a method comparison

Modeling complex systems with large numbers of degrees of freedom has become a grand challenge over the past decades. In many situations, only a few variables are actually observed in terms of measured time series, while the majority of variables—which potentially interact with the observed ones—remain hidden. A typical approach is then to focus on the comparably few observed, macroscopic variables, assuming that they determine the key dynamics of the system, while the remaining ones are represented by noise. This naturally leads to an approximate, inverse modeling of such systems in terms of stochastic differential equations (SDEs), with great potential for applications from biology to finance and Earth system dynamics. A well-known approach to retrieve such SDEs from small sets of observed time series is to reconstruct the drift and diffusion terms of a Langevin equation from the data-derived Kramers–Moyal (KM) coefficients. For systems where interactions between the observed and the unobserved variables are crucial, the Mori–Zwanzig formalism (MZ) allows to derive generalized Langevin equations that contain non-Markovian terms representing these interactions. In a similar spirit, the empirical model reduction (EMR) approach has more recently been introduced. In this work we attempt to reconstruct the dynamical equations of motion of both synthetical and real-world processes, by comparing these three approaches in terms of their capability to reconstruct the dynamics and statistics of the underlying systems. Through rigorous investigation of several synthetical and real-world systems, we confirm that the performance of the three methods strongly depends on the intrinsic dynamics of the system at hand. For instance, statistical properties of systems exhibiting weak history-dependence but strong state-dependence of the noise forcing, can be approximated better by the KM method than by the MZ and EMR approaches. In such situations, the KM method is of a considerable advantage since it can directly approximate the state-dependent noise. However, limitations of the KM approximation arise in cases where non-Markovian effects are crucial in the dynamics of the system. In these situations, our numerical results indicate that methods that take into account interactions between observed and unobserved variables in terms of non-Markovian closure terms (i.e., the MZ and EMR approaches), perform comparatively better.

Projection operators in statistical mechanics: a pedagogical approach

The Mori–Zwanzig projection operator formalism is one of the central tools of nonequilibrium statistical mechanics, allowing one to derive macroscopic equations of motion from the microscopic dynamics through a systematic coarse-graining procedure. It is important as a method in the physical sciences and gives many insights into the general structure of nonequilibrium transport equations and the general procedure of microscopic derivations. Therefore, it is a valuable ingredient of basic and advanced courses in statistical mechanics. However, accessible introductions to this method—in particular in its more advanced forms—are extremely rare. In this article, we give a simple and systematic introduction to the Mori–Zwanzig formalism, which allows students to understand the methodology in the form it is used in current research. This includes both basic and modern versions of the theory. Moreover, we relate the formalism to more general aspects of statistical mechanics and quantum mechanics. Thereby, we explain how this method can be incorporated into a lecture course on statistical mechanics as a way to give a general introduction to the study of nonequilibrium systems. Applications, in particular to spin relaxation and dynamical density functional theory, are also discussed.

The Adjoint Petrov–Galerkin method for non-linear model reduction

We formulate a new projection-based reduced-order modeling technique for non-linear dynamical systems. The proposed technique, which we refer to as the Adjoint Petrov–Galerkin (APG) method, is derived by decomposing the generalized coordinates of a dynamical system into a resolved coarse-scale set and an unresolved fine-scale set. A Markovian finite memory assumption within the Mori–Zwanzig formalism is then used to develop a reduced-order representation of the coarse scales. This procedure leads to a closed reduced-order model that displays commonalities with the adjoint stabilization method used in finite elements. The formulation is shown to be equivalent to a Petrov–Galerkin method with a non-linear, time-varying test basis, thus sharing some similarities with the Least-Squares Petrov–Galerkin method. Theoretical analysis examining a priori error bounds and computational cost is presented. Numerical experiments on the compressible Navier–Stokes equations demonstrate that the proposed method can lead to improvements in numerical accuracy, robustness, and computational efficiency over the Galerkin method on problems of practical interest. Improvements in numerical accuracy and computational efficiency over the Least-Squares Petrov–Galerkin method are observed in most cases.

Recurrent neural network closure of parametric POD-Galerkin reduced-order models based on the Mori-Zwanzig formalism

Closure modeling based on the Mori-Zwanzig formalism has proven effective to improve the stability and accuracy of projection-based model order reduction. However, closure models are often expensive and infeasible for complex nonlinear systems. Towards efficient model reduction of general problems, this paper presents a recurrent neural network (RNN) closure of parametric POD-Galerkin reduced-order model. Based on the short time history of the reduced-order solutions, the RNN predicts the memory integral which represents the impact of the unresolved scales on the resolved scales. A conditioned long short term memory (LSTM) network is utilized as the regression model of the memory integral, in which the POD coefficients at a number of time steps are fed into the LSTM units, and the physical/geometrical parameters are fed into the initial hidden state of the LSTM. The reduced-order model is integrated in time using an implicit-explicit (IMEX) Runge-Kutta scheme, in which the memory term is integrated explicitly and the remaining right-hand-side term is integrated implicitly to improve the computational efficiency. Numerical results demonstrate that the RNN closure can significantly improve the accuracy and efficiency of the POD-Galerkin reduced-order model of nonlinear problems. The POD-Galerkin reduced-order model with the RNN closure is also shown to be capable of making accurate predictions, well beyond the time interval of the training data.

Time-series learning of latent-space dynamics for reduced-order model closure

We study the performance of long short-term memory networks (LSTMs) and neural ordinary differential equations (NODEs) in learning latent-space representations of dynamical equations for an advection-dominated problem given by the viscous Burgers equation. Our formulation is devised in a nonintrusive manner with an equation-free evolution of dynamics in a reduced space with the latter being obtained through a proper orthogonal decomposition. In addition, we leverage the sequential nature of learning for both LSTMs and NODEs to demonstrate their capability for closure in systems that are not completely resolved in the reduced space. We assess our hypothesis for two advection-dominated problems given by the viscous Burgers equation. We observe that both LSTMs and NODEs are able to reproduce the effects of the absent scales for our test cases more effectively than does intrusive dynamics evolution through a Galerkin projection. This result empirically suggests that time-series learning techniques implicitly leverage a memory kernel for coarse-grained system closure as is suggested through the Mori–Zwanzig formalism.

Generalized Langevin Equation: An Introductory Review for Biophysicists

An introductory, pedagogical review of the generalized Langevin equation (GLE) within the classical regime is presented. It is intended to be accessible to biophysicists with an interest in molecular dynamics (MD). Section 1 presents why the equation may be of interest within biophysical modeling. A detailed elementary first principles derivation of the (multidimensional) Kac–Zwanzig model is presented. The literature is reviewed with a focus on biophysical applications and representation by Markovian stochastic differential equations. The relationship with the Mori–Zwanzig formalism is discussed. The framework of model reduction and approximation is emphasized. Some open problems are identified.

Derivation of delay equation climate models using the Mori-Zwanzig formalism.

Models incorporating delay have been frequently used to understand climate variability phenomena, but often the delay is introduced through an ad hoc physical reasoning, such as the propagation time of waves. In this paper, the Mori-Zwanzig formalism is introduced as a way to systematically derive delay models from systems of partial differential equations and hence provides a better justification for using these delay-type models. The Mori-Zwanzig technique gives a formal rewriting of the system using a projection onto a set of resolved variables, where the rewritten system contains a memory term. The computation of this memory term requires solving the orthogonal dynamics equation, which represents the unresolved dynamics. For nonlinear systems, it is often not possible to obtain an analytical solution to the orthogonal dynamics and an approximate solution needs to be found. Here, we demonstrate the Mori-Zwanzig technique for a two-strip model of the El Niño Southern Oscillation (ENSO) and explore methods to solve the orthogonal dynamics. The resulting nonlinear delay model contains an additional term compared to previously proposed ad hoc conceptual models. This new term leads to a larger ENSO period, which is closer to that seen in observations.

Mori-Zwanzig projection operator formalism for far-from-equilibrium systems with time-dependent Hamiltonians.

The Mori-Zwanzig projection operator formalism is a powerful method for the derivation of mesoscopic and macroscopic theories based on known microscopic equations of motion. It has applications in a large number of areas including fluid mechanics, solid-state theory, spin relaxation theory, and particle physics. In its present form, however, the formalism cannot be directly applied to systems with time-dependent Hamiltonians. Such systems are relevant in many scenarios such as driven soft matter or nuclear magnetic resonance. In this article we derive a generalization of the present Mori-Zwanzig formalism that is able to treat also time-dependent Hamiltonians. The extended formalism can be applied to classical and quantum systems, close to and far from thermodynamic equilibrium, and even in the case of explicitly-time-dependent observables. Moreover, we develop a variety of approximation techniques that enhance the practical applicability of our formalism. Generalizations and approximations are developed for both equations of motion and correlation functions. Our formalism is demonstrated for the important case of spin relaxation in a time-dependent external magnetic field. The Bloch equations are derived together with microscopic expressions for the relaxation times.

The multi-dimensional generalized Langevin equation for conformational motion of proteins.

Using the generalized Langevin equation (GLE) is a promising approach to build coarse-grained (CG) models of molecular systems since the GLE model often leads to more accurate thermodynamic and kinetic predictions than Brownian dynamics or Langevin models by including a more sophisticated friction with memory. The GLE approach has been used for CG coordinates such as the center of mass of a group of atoms with pairwise decomposition and for a single CG coordinate. We present a GLE approach when CG coordinates are multiple generalized coordinates, defined, in general, as nonlinear functions of microscopic atomic coordinates. The CG model for multiple generalized coordinates is described by the multidimensional GLE from the Mori-Zwanzig formalism, which includes an exact memory matrix. We first present a method to compute the memory matrix in a multidimensional GLE using trajectories of a full system. Then, in order to reduce the computational cost of computing the multidimensional friction with memory, we introduce a method that maps the GLE to an extended Markovian system. In addition, we study the effect of using a nonconstant mass matrix in the CG model. In particular, we include mass-dependent terms in the mean force. We used the proposed CG model to describe the conformational motion of a solvated alanine dipeptide system, with two dihedral angles as the CG coordinates. We showed that the CG model can accurately reproduce two important kinetic quantities: the velocity autocorrelation function and the distribution of first passage times.

A Model Framework for Stochastic Representation of Uncertainties Associated with Physical Processes in NOAA’s Next Generation Global Prediction System (NGGPS)

Abstract In this study, we propose a physical-process-based stochastic parameterization scheme using cellular automata for NOAA’s Next Generation Global Prediction System. The cellular automata, used to simulate stochastic processes such as the production and destruction of subgrid convective elements, are conditioned on unresolved vertical motion that follows a prescribed stochastically generated skewed distribution (SGS). The SGS is described by a stochastic differential equation and linked to observations by taking into account the first four moments from an observed dataset. In the proposed parameterization framework, we emphasize the need for a dynamical memory term to be included in physical-process-based stochastic parameterizations, and we illustrate the requirement for the dynamical memory using the Mori–Zwanzig formalism. Although this paper focuses on the methodology, early results indicate that if we apply our stochastic framework to deep cumulus convection, it is found that the frequency distribution of precipitation is improved in a single-member stochastic forecast, and some improved spread–skill relationship in ensemble runs can be found in state variables in the tropics, as well as in the subtropics.

Mori-Zwanzig reduced models for uncertainty quantification

In many time-dependent problems of practical interest the parameters and/or initial conditions entering the equations describing the evolution of the various quantities exhibit uncertainty. One way to address the problem of how this uncertainty impacts the solution is to expand the solution using polynomial chaos expansions and obtain a system of differential equations for the evolution of the expansion coefficients. We present an application of the Mori-Zwanzig (MZ) formalism to the problem of constructing reduced models of such systems of differential equations. In particular, we construct reduced models for a subset of the polynomial chaos expansion coefficients that are needed for a full description of the uncertainty caused by uncertain parameters or initial conditions. Even though the MZ formalism is exact, its straightforward application to the problem of constructing reduced models for estimating uncertainty involves the computation of memory terms whose cost can become prohibitively expensive. For those cases, we present a Markovian reformulation of the MZ formalism which can lead to approximations that can alleviate some of the computational expense while retaining an accuracy advantage over reduced models that discard the memory altogether. Our results support the conclusion that successful reduced models need to include memory effects.

Correction: Implicit-solvent coarse-grained modeling for polymer solutions via Mori–Zwanzig formalism

Correction for 'Implicit-solvent coarse-grained modeling for polymer solutions via Mori-Zwanzig formalism' by Shu Wang et al., Soft Matter, 2019, DOI: .

Implicit-solvent coarse-grained modeling for polymer solutionsviaMori-Zwanzig formalism

We present a bottom-up coarse-graining (CG) method to establish implicit-solvent CG modeling for polymers in solution, which conserves the dynamic properties of the reference microscopic system. In particular, tens to hundreds of bonded polymer atoms (or Lennard-Jones beads) are coarse-grained as one CG particle, and the solvent degrees of freedom are eliminated. The dynamics of the CG system is governed by the generalized Langevin equation (GLE) derived via the Mori-Zwanzig formalism, by which the CG variables can be directly and rigorously linked to the microscopic dynamics generated by molecular dynamics (MD) simulations. The solvent-mediated dynamics of polymers is modeled by the non-Markovian stochastic dynamics in GLE, where the memory kernel can be computed from the MD trajectories. To circumvent the difficulty in direct evaluation of the memory term and generation of colored noise, we exploit the equivalence between the non-Markovian dynamics and Markovian dynamics in an extended space. To this end, the CG system is supplemented with auxiliary variables that are coupled linearly to the momentum and among themselves, subject to uncorrelated Gaussian white noise. A high-order time-integration scheme is used to solve the extended dynamics to further accelerate the CG simulations. To assess, validate, and demonstrate the established implicit-solvent CG modeling, we have applied it to study four different types of polymers in solution. The dynamic properties of polymers characterized by the velocity autocorrelation function, diffusion coefficient, and mean square displacement as functions of time are evaluated in both CG and MD simulations. Results show that the extended dynamics with auxiliary variables can construct arbitrarily high-order CG models to reproduce dynamic properties of the reference microscopic system and to characterize long-time dynamics of polymers in solution.

Model Reduction with Memory and the Machine Learning of Dynamical Systems

The well-known Mori-Zwanzig theory tells us that model reduction leads to memory effect. For a long time, modeling the memory effect accurately and efficiently has been an important but nearly impossible task in developing a good reduced model. In this work, we explore a natural analogy between recurrent neural networks and the Mori-Zwanzig formalism to establish a systematic approach for developing reduced models with memory. Two training models-a direct training model and a dynamically coupled training model-are proposed and compared. We apply these methods to the Kuramoto-Sivashinsky equation and the Navier-Stokes equation. Numerical experiments show that the proposed method can produce reduced model with good performance on both short-term prediction and long-term statistical properties.

Thermal and thermoelectric transport coefficients for a two-dimensional SDW metal with weak disorder: A memory matrix calculation

We calculate the thermal and thermoelectric transport coefficients for a two-dimensional spin-density-wave metal in the presence of weak disorder and under an external magnetic field using the Mori-Zwanzig memory matrix formalism. As a consequence, we obtain that although the Seebeck coefficient S displays a relatively conventional linear dependence on the temperature of the system, the Nernst coefficient ν exhibits a clear signature of bad-metallicity within the so-called strange metal regime described by the model. This result agrees qualitatively with experimental transport data obtained for many cuprate superconductors around optimal doping.

Mesoscopic coarse-grained representations of fluids rigorously derived from atomistic models.

Mesoscopic models are widely used to study complex organization and transport phenomena in chemical and biological systems. Defining a rigorous procedure by which a mesoscopic coarse-grained (CG) representation for a fluid can be constructed from an atomistic fine-grained (FG) model is a long-standing question in the field. The connection of these CG models with the FG level of description, which might be built by CG mappings from the FG model, is often unclear. The present paper introduces a new CG mapping scheme that uses dynamically self-consistent smooth centroidal Voronoi tessellation to address this challenging problem. The new mapping scheme is applied to the coarse-graining of supercritical Lennard-Jones fluid systems at different CG resolutions under both quiescent conditions and non-equilibrium shear flow. The method generates continuous, stable, and ergodic CG trajectories and quantitatively captures the slow collective motions of the underlying FG fluids. A parameterization of the CG models from the mapped CG trajectory is then developed based on the Mori-Zwanzig formalism. The Generalized Langevin Equation describes the dynamics of CG variables, and the parameterized result is shown to reproduce the structural and dynamical correlations of the CG system. The new dynamical mapping scheme and the parameterization protocol open up an avenue for direct bottom-up construction of mesoscopic models of fluids in a Lagrangian description.

A note on hydrodynamics from dissipative particle dynamics

We calculate current correlation functions (CCFs) of dissipative particle dynamics (DPD) and compare them with results of molecular dynamics (MD) and solutions of linearized hydrodynamic equations. In particular, we consider three versions of DPD, the empirical/classical DPD, coarse-grained (CG) DPD with radial-direction interactions only and full (radial, transversal, and rotational) interactions between particles. To facilitate quantitative discussions, we consider specifically a star-polymer melt system at a moderate density. For bonded molecules, it is straightforward to define the CG variables and to further derive CG force fields for DPD within the framework of the Mori-Zwanzig formalism. For both transversal and longitudinal current correlation functions (TCCFs and LCCFs), we observe that results of MD, DPD, and hydrodynamic solutions agree with each other at the continuum limit. Below the continuum limit to certain length scales, results of MD deviate significantly from hydrodynamic solutions, whereas results of both empirical and CG DPD resemble those of MD. This indicates that the DPD method with Markovian force laws possibly has a larger applicability than the continuum description of a Newtonian fluid. This is worth being explored further to represent generalized hydrodynamics.

Construction of non-Markovian coarse-grained models employing the Mori-Zwanzig formalism and iterative Boltzmann inversion.

We propose a new coarse-grained (CG) molecular simulation technique based on the Mori-Zwanzig (MZ) formalism along with the iterative Boltzmann inversion (IBI). Non-Markovian dissipative particle dynamics (NMDPD) taking into account memory effects is derived in a pairwise interaction form from the MZ-guided generalized Langevin equation. It is based on the introduction of auxiliary variables that allow for the replacement of a non-Markovian equation with a Markovian one in a higher dimensional space. We demonstrate that the NMDPD model exploiting MZ-guided memory kernels can successfully reproduce the dynamic properties such as the mean square displacement and velocity autocorrelation function of a Lennard-Jones system, as long as the memory kernels are appropriately evaluated based on the Volterra integral equation using the force-velocity and velocity-velocity correlations. Furthermore, we find that the IBI correction of a pair CG potential significantly improves the representation of static properties characterized by a radial distribution function and pressure, while it has little influence on the dynamic processes. Our findings suggest that combining the advantages of both the MZ formalism and IBI leads to an accurate representation of both the static and dynamic properties of microscopic systems that exhibit non-Markovian behavior.

On the Asymptotic Behavior of the Kernel Function in the Generalized Langevin Equation: A One-Dimensional Lattice Model

We present some estimates for the memory kernel function in the generalized Langevin equation, derived using the Mori-Zwanzig formalism from a one-dimensional lattice model, in which the particles interactions are through nearest and second nearest neighbors. The kernel function can be explicitly expressed in a matrix form. The analysis focuses on the decay properties, both spatially and temporally, revealing a power-law behavior in both cases. The dependence on the level of coarse-graining is also studied.