Irreversible Statistical Mechanics Research Articles

The Mori–Zwanzig formulation of deep learning

We develop a new formulation of deep learning based on the Mori–Zwanzig (MZ) formalism of irreversible statistical mechanics. The new formulation is built upon the well-known duality between deep neural networks and discrete dynamical systems, and it allows us to directly propagate quantities of interest (conditional expectations and probability density functions) forward and backward through the network by means of exact linear operator equations. Such new equations can be used as a starting point to develop new effective parameterizations of deep neural networks and provide a new framework to study deep learning via operator-theoretic methods. The proposed MZ formulation of deep learning naturally introduces a new concept, i.e., the memory of the neural network, which plays a fundamental role in low-dimensional modeling and parameterization. By using the theory of contraction mappings, we develop sufficient conditions for the memory of the neural network to decay with the number of layers. This allows us to rigorously transform deep networks into shallow ones, e.g., by reducing the number of neurons per layer (using projection operators), or by reducing the total number of layers (using the decay property of the memory operator).

Statistical analysis and simulation of random shocks in stochastic Burgers equation

We study the statistical properties of random shock waves in stochastic Burgers equation subject to random space–time perturbations and random initial conditions. By using the response–excitation probability density function (PDF) method and the Mori–Zwanzig (MZ) formulation of irreversible statistical mechanics, we derive exact reduced-order equations for the one-point and two-point PDFs of the solution field. In particular, we compute the statistical properties of random shock waves in the inviscid limit by using an adaptive (shock-capturing) discontinuous Galerkin method in both physical and probability spaces. We consider stochastic flows generated by high-dimensional random initial conditions and random additive forcing terms, yielding multiple interacting shock waves collapsing into clusters and settling down to a similarity state. We also address the question of how random shock waves in space and time manifest themselves in probability space. The proposed new mathematical framework can be applied to different conservation laws, potentially leading to new insights into high-dimensional stochastic dynamical systems and more efficient computational algorithms.

Convolutionless Nakajima-Zwanzig equations for stochastic analysis in nonlinear dynamical systems.

Determining the statistical properties of stochastic nonlinear systems is of major interest across many disciplines. Currently, there are no general efficient methods to deal with this challenging problem that involves high dimensionality, low regularity and random frequencies. We propose a framework for stochastic analysis in nonlinear dynamical systems based on goal-oriented probability density function (PDF) methods. The key idea stems from techniques of irreversible statistical mechanics, and it relies on deriving evolution equations for the PDF of quantities of interest, e.g. functionals of the solution to systems of stochastic ordinary and partial differential equations. Such quantities could be low-dimensional objects in infinite dimensional phase spaces. We develop the goal-oriented PDF method in the context of the time-convolutionless Nakajima-Zwanzig-Mori formalism. We address the question of approximation of reduced-order density equations by multi-level coarse graining, perturbation series and operator cumulant resummation. Numerical examples are presented for stochastic resonance and stochastic advection-reaction problems.

Renormalized reduced models for singular PDEs

We present a novel way of constructing reduced models for systems of ordinary differential equations. The reduced models we construct depend on coefficients which measure the importance of the different terms appearing in the model and need to be estimated. The proposed approach allows the estimation of these coefficients on the fly by enforcing the equality of integral quantities of the solution as computed from the original system and the reduced model. In particular, the approach combines the concepts of renormalization and effective field theory developed in the context of high energy physics and the Mori-Zwanzig formalism of irreversible statistical mechanics. It allows to construct stable reduced models of higher order than was previously possible. The method is applied to the problem of computing reduced models for ordinary differential equation systems resulting from Fourier expansions of singular (or near-singular) time-dependent partial differential equations. Results for the 1D Burgers and the 3D incompressible Euler equations are used to illustrate the construction. We also present, for the 1D Burgers and the 3D Euler equations, a simple and efficient recursive algorithm for calculating the higher order terms.

Problem reduction, renormalization, and memory

We present methods for the reduction of the complexity of computational problems, both time-dependent and stationary, together with connections to renormalization, scaling, and irreversible statistical mechanics. Most of the methods have been presented before; what is new here is the common framework which relates the several constructions to each other and to methods of theoretical physics, as well as the analysis of the approximate reductions for time-dependent problems. The key conclusions are: (i) in time dependent problems, it is not in general legitimate to average equations without taking into account memory effects and noise; (ii) mathematical tools developed in physics for carrying out renormalization group transformations yield effective block Monte Carlo methods; (iii) the Mori–Zwanzig formalism, which in principle yields exact reduction methods but is often hard to use, can be tamed by approximation; and (iv) more generally, problem reduction is a search for hidden similarities.

A comparative study of two stochastic mode reduction methods

We present a comparative study of two methods for the reduction of the dimensionality of a system of ordinary differential equations that exhibits time-scale separation. Both methods lead to a reduced system of stochastic differential equations. The novel feature of these methods is that they allow the use, in the reduced system, of higher-order terms in the resolved variables. The first method, proposed by Majda, Timofeyev and Vanden-Eijnden, is based on an asymptotic strategy developed by Kurtz. The second method is a short-memory approximation of the Mori–Zwanzig projection formalism of irreversible statistical mechanics, as proposed by Chorin, Hald and Kupferman. We present conditions under which the reduced models arising from the two methods should have similar predictive ability. We apply the two methods to test cases that satisfy these conditions. The form of the reduced models and the numerical simulations show that the two methods have similar predictive ability as expected.

Existence proof for orthogonal dynamics and the Mori-Zwanzig formalism

We study the existence of solutions to the orthogonal dynamics equation, which arises in the Mori-Zwanzig formalism in irreversible statistical mechanics. This equation generates the random noise associated with a reduction in the number of variables. IfL is the Liouvillian, or Lie derivative associated with a Hamiltonian system, andP an orthogonal projection onto a closed subspace ofL 2, then the orthogonal dynamics is generated by the operator (I −P)L. We prove the existence of classical solutions for the case whereP has finite-dimensional range. In the general case, we prove the existence of weak solutions.

Optimal prediction with memory

Optimal prediction methods estimate the solution of nonlinear time-dependent problems when that solution is too complex to be fully resolved or when data are missing. The initial conditions for the unresolved components of the solution are drawn from a probability distribution, and their effect on a small set of variables that are actually computed is evaluated via statistical projection. The formalism resembles the projection methods of irreversible statistical mechanics, supplemented by the systematic use of conditional expectations and new methods of solution for an auxiliary equation, the orthogonal dynamics equation, needed to evaluate a non-Markovian memory term. The result of the computations is close to the best possible estimate that can be obtained given the partial data. We present the constructions in detail together with several useful variants, provide simple examples, and point out the relation to the fluctuation–dissipation formulas of statistical physics.

On the Statistical Foundations of Irreversible Thermodynamics

We briefly, and partially, consider aspects of the present status of phenomenological irreversible thermodynamics and nonequilibrium statistical mechanics. After short comments on Classical Irreversible Thermodynamics, its conceptual and practical shortcomings are pointed out, as well as the efforts undertaken to go beyond its limits, consisting of particular approaches to a more general theory of Irreversible Thermodynamics. In particular, a search for statistical-mechanical foundations of Irreversible Thermodynamics, namely, the construction of a statistical thermodynamics, are based on the Non-equilibrium Statistical Operator Method. This important theory for the treatment of phenomena at the macroscopic level, is based on a microscopic molecular description in the context of a nonequilibrium ensemble formalism. We draw attention to the fact that this method may be considered to be emcompassed within Jaynes' Predictive Statistical Mechanics and based on the principle of maximization of informational entropy. Finally, we describe how, in fact, the statistical method provides foundations to phenomenological irreversible thermodynamics, thus giving rise to what can be referred to as Informational Statistical Thermodynamics.

Microscopic Kinetic Mechanism in Current-Induced Conversion on Si(001) Vicinal Surface

The microscopic mechanism of current-induced domain conversion phenomena on the Si(001) vicinal surface during annealing is studied using the kinetic equation derived by the path probability method (PPM) in irreversible statistical mechanics along with the Monte Carlo simulation. In addition to evaporation, our model takes account of the three effects related to migration of surface atoms: anisotropic migration on the Si(001) 2×1 reconstructed surface, the electromigration effect and asymmetry in step kinetics (Schwoebel effect) which takes the difference in the kinetics between two types of steps into account. The numerical calculation of the kinetic equation reproduces the domain conversion when the Schwoebel effect exists. The differences in the movements of two types of steps and the spreading velocities of major domains observed during domain conversion are also shown. The results suggest that the combination of the three migration effects causes the difference in the kinetics of atoms between two types of steps, which leads to the domain conversion. The results of the Monte Carlo simulation are in good agreement with those of the PPM.

Hyperbolic type transport equations

In recent years hyperbolic type transport equations have acquired a great deal of importance in problems ranging from theoretical physics to biology. In spite of their greater mathematical difficulty as compared with their parabolic type analogs arising from the framework of Linear Irreversible Thermodynamics, they have, in many ways, superseded the latter ones. Although the use of this type of equations is well known since the last century through the telegraphist equation of electromagnetic theory, their use in studying several problems in transport theory is hardly fifty years old. In fact the first appearance of a hyperbolic type transport equation for the problem of heat conduction dates back to Cattaneos' work in 1948. Three years later, in 1951 S. Goldstein showed how in the theory of stochastic processes this type of an equation is obtained in the continuous limit of a one-dimensional persistent random walk problem. After that, other phenomenological derivations have been offered for such equations. The main purpose of this paper is to critically discuss a derivation of a hyperbolic type Fokker-Planck equation recently presented using the same ideas as M.S. Green did in 1952 to provide the stochastic foundations of irreversible statistical mechanics. Arguments are given to show that such an equation as well as transport equations derived from it by taking appropriate averages are at most approximate and that a much more detailed analysis is required before asserting their validity.

Frequency dependence of ionic conductivity in lattice gas models

The frequency dependence of the ionic conductivity has been treated based on the relaxation process of ionic current using the pair approximation of the path probability method (PPM) of irreversible statistical mechanics. In order to deal with the time development of the time correlation of the motion of particles, the formalism of the PPM which deals with the flow based on the averaged distribution of particles has been changed to follow the fluctuation of the system with time. This allows us to evaluate the development of the time correlation of the motion of particles in the pair approximation. The Kohlrausch-Williams-Watts type relaxation behavior of ionic current when a constant field is applied at time t=0 can be attributed to the change of the time correlation with time. The Jonscher power-law exponent p is thus derived for several important examples in the pair approximation. In the case of the motion of a tagged particle in the assembly of particles of the same species in a lattice gas model, the results show an excellent agreement with those obtained by the Monte Carlo simulation method.

A critical investigation of the generalized Langevin equation

It is shown that the formulation of the generalized Langevin equation leads to a non-decaying correlation function 〈 x, R( t)〉 = 〈 x, R〉 where R( t) is the random acceleration at time t, x is the position of a Brownian particle at t=0 and R R(0). This is a serious difficulty in view of the fundamental requirement form irreversible statistical mechanics that lim t→∞ 〈 x, R( t)〉=〈 x〉〈 R( t)〉=0.

Non Debye Relaxation Process of Hopping Ionic Conduction in Lattice Gas Models

Statistical treatments of the relaxation process of hopping ionic conduction in lattice gas models generally lead to a linear relaxation process. This situation was investigated critically by means of the pair approximation of the Path Probability method of irreversible statistical mechanics for the purpose of understanding the strong non Debye relaxation process in the ionic conduction in disordered solid electrolytes known as «the universal dynamic response.» It is found that the formalism of these treatments is to follow quantities in the ensemble average and that the usage of such treatments for flow which requires the time correlation of the motion of particles is responsible for the above results independent of the degree of approximation

Atomic Relaxation of a Junction Profile

Nonlinear relaxation of a sharp concentration profile typical in layered semiconductor junctions is investigated using the Path Probability Method (PPM) of irreversible statistical mechanics. We employ the vacancy mechanism for atomic migration and the pair approximation for the statistical treatment. The PPM is a microscopic method, from which we can derive macroscopic parameters. Our results show that at the initial stage of the relaxation of a sharp concentration profile, atoms near the junction may diffuse up against the concentration gradient. More surprisingly, our numerical examples convincingly demonstrate that the atom flux goes up against the local chemical potential gradient near a sharp profile, indicating that in such highly nonlinear regime the usual linear diffusion theory in which the atom flux is linearly proportional to the chemical potential gradient breaks down. It is shown that the cause of the uphill diffusion is the repulsion among different species, which is also the physical origin of the square gradient term.

The path probability method: An atomistic technique of diffusion

The path probability method (PPM) of irreversible statistical mechanics has been successfully applied to various diffusion problems. The major advantage of this atomistic treatment over the phenomenological approach of irreversible thermodynamics is that all Onsager matrix coefficients can be derived analytically so that relations among measurable quantities can be clearly understood in terms of microscopic parameters. This review article attempts to present the PPM in the simplest possible form. The importance of the PPM as an atomistic technique is illustrated using a simple example. The applicability and limitations of the technique are also emphasized.

Path Probability Method of Irreversible Statistical Mechanics

Formalism of the Path Probability method (PPM) of irreversible statistical mechanics has been examined in order to explore the applicability of the method to a variety of kinetic problems. It is found that the PPM works very well when the time dependence of macrovariables in homogeneous systems such as the order-disorder transformations in alloys is studied. If the method is applied to problems in which deviations from the above conditions are involved, however, the reliability of the method decreases. In this category, applications to transport problems are discussed exclusively. Here, the degree of approximation is found to be not as good as that expected from the method taken. The examination of the cause leads to a conversion method which gives results with the same degree of approximation as that in the homogeneous case.

Theory of self diffusion in electrolytes

Starting from the concept of marked ions, the problem of self diffusion in electrolyte solutions is discussed from the point of view of irreversible thermodynamics and statistical mechanics. On the basis of the diffusion approach to the theory of transport processes in electrolytes, we derive the statistical theory of self diffusion and an expression for the self diffusion coefficient in terms of the radial distribution function. Results for the concentration dependence of the self-diffusion coefficient are presented.

Demixing: A source of material deterioration

Demixing is a significant cause of deterioration of materials used in high temperature technologies. It is important to devise ways to relate such deterioration behavior to microscopic properties of materials so that precise predictions can be made. An atomistic rather than a customary phenomenological treatment would be required for this purpose. An analytical treatment based on the Path Probability Method (PPM) of irreversible statistical mechanics has been proved useful towards this end. This article reviews the topic of demixing emphasizing particularly the importance of such an atomistic treatment.

Time Evolution of Demixing in Oxides under an Oxygen Potential Gradient

Time evolution of concentration profiles leading to steady‐state demixing in oxides under oxygen partial pressure gradient is treated based on a microscopic model by applying the path probability method of irreversible statistical mechanics. The redistribution of vacancies and cations toward the steady‐state distribution has essentially been reduced to the diffusion of vacancies and the interdiffusion of cations along the specimen. Atomistic features of the problem, such as a vast difference in the relaxation times for the redistribution of vacancies and cations, are emphasized. Time required for the system to reach the steady state is then estimated in terms of fundamental material properties.