Thermodynamics Of Computation Research Articles

The Differential Entropy of Mixtures: New Bounds and Applications

Mixture distributions are extensively used as a modeling tool in diverse areas from machine learning to communications engineering to physics, and obtaining bounds on the entropy of mixture distributions is of fundamental importance in many of these applications. This article provides sharp bounds on the entropy concavity deficit, which is the difference between the differential entropy of the mixture and the weighted sum of differential entropies of constituent components. Toward establishing lower and upper bounds on the concavity deficit, results that are of importance in their own right are obtained. In order to obtain nontrivial upper bounds, properties of the skew-divergence are developed and notions of “skew” <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$f$ </tex-math></inline-formula> -divergences are introduced; a reverse Pinsker inequality and a bound on Jensen-Shannon divergence are obtained along the way. Complementary lower bounds are derived with special attention paid to the case that corresponds to independent summation of a continuous and a discrete random variable. Several applications of the bounds are delineated, including to mutual information of additive noise channels, thermodynamics of computation, and functional inequalities.

Shakin' All Over: Proving Landauer's principle without neglect of fluctuations

Landauer's principle is, roughly, the principle that logically irreversible operations cannot be performed without dissipation of energy, with a specified lower bound on that dissipation. Though widely accepted in the literature on the thermodynamics of computation, it has been the subject of considerable dispute in the philosophical literature. Proofs of the principle have been questioned on the grounds of insufficient generality and on the grounds of the assumption, used in the proofs, of the availability of reversible processes at the microscale. The relevance of the principle, should it be true, has also been questioned, as it has been argued that microscale fluctuations entail dissipation that always greatly exceeds the Landauer bound. In this article Landauer's principle is treated within statistical mechanics, and a proof of the principle is given that neither relies on neglect of fluctuations nor assumes the availability of thermodynamically reversible processes. In addition, it is argued that microscale fluctuations are no obstacle to approximating thermodynamic reversibility, in the appropriate sense, as closely as one would like.

Energy transfer and thermodynamics of quantum gravity computation

In a quantum gravity environment, the processes and events are causally non-separable because the term of time and the time-steps have no interpretable meaning in a non-fixed causality structure. Here, we study the energy transfer and thermodynamics of quantum gravity computations. We show that a non-fixed causality stimulates entropy transfer between the quantum gravity environment and the independent local systems of the quantum gravity space. We prove that the entropy transfer reduces the entropies of the contributing local systems and increases the entropy of the quantum gravity environment. We reveal on a smooth Cauchy slice that the space-time geometry of the quantum gravity environment dynamically adapts to the vanishing causality. We define the corresponding Hamiltonians and the causal development of the quantum gravity environment.

On Norton’s No-Go Result

Physicists in particular should pay close attention: John Norton’s no-go result refers to a process that Rolf Landauer took for granted in his formulation of the thermodynamics of computation.

Thermodynamics of computation and linear stability limits of superfluid refrigeration of a model computing array

We analyze the stability of the temperature profile of an array of computing nanodevices refrigerated by flowing superfluid helium, under variations in temperature, computing rate, and barycentric velocity of helium. It turns out that if the variation in dissipated energy per bit with respect to temperature variations is higher than some critical values, proportional to the effective thermal conductivity of the array, then the steady-state temperature profiles become unstable and refrigeration efficiency is lost. Furthermore, a restriction on the maximum rate of variation in the local computation rate is found.

Number of hidden states needed to physically implement a given conditional distribution

We consider the problem of how to construct a physical process over a finite state space X that applies some desired conditional distribution P to initial states to produce final states. This problem arises often in the thermodynamics of computation and nonequilibrium statistical physics more generally (e.g. when designing processes to implement some desired computation, feedback controller, or Maxwell demon). It was previously known that some conditional distributions cannot be implemented using any master equation that involves just the states in X. However, here we show that any conditional distribution P can in fact be implemented—if additional ‘hidden’ states not in X are available. Moreover, we show that it is always possible to implement P in a thermodynamically reversible manner. We then investigate a novel cost of the physical resources needed to implement a given distribution P: the minimal number of hidden states needed to do so. We calculate this cost exactly for the special case where P represents a single-valued function, and provide an upper bound for the general case, in terms of the non-negative rank of P. These results show that having access to one extra binary degree of freedom, thus doubling the total number of states, is sufficient to implement any P with a master equation in a thermodynamically reversible way, if there are no constraints on the allowed form of the master equation. (Such constraints can greatly increase the minimal needed number of hidden states.) Our results also imply that for certain P that can be implemented without hidden states, having hidden states permits an implementation that generates less heat.

The Physics of Forgetting: Thermodynamics of Information at IBM 1959–1982

The origin and history of Landauer’s principle is traced through the development of the thermodynamics of computation at IBM from 1959 to 1982. This development was characterized by multiple conceptual shifts: memory came to be seen not as information storage, but as delayed information transmission; information itself was seen not as a disembodied logical entity, but as participating in the physical world; and logical irreversibility was connected with physical, thermodynamic, irreversibility. These conceptual shifts were characterized by an ambivalence opposing strong metaphysical claims to practical considerations. Three sorts of practical considerations are discussed. First, these conceptual shifts engaged materials central to IBM’s business practice. Second, arguments for metaphysical certainties were made with reference to the practical functioning of typical computers. Third, arguments for metaphysical certainties were made in the context of establishing the thermodynamics of information as a sub-discipline of physics.

DEMONIC programming: a computational language for single-particle equilibrium thermodynamics, and its formal semantics.

Maxwell's Demon, 'a being whose faculties are so sharpened that he can follow every molecule in its course', has been the centre of much debate about its abilities to violate the second law of thermodynamics. Landauer's hypothesis, that the Demon must erase its memory and incur a thermodynamic cost, has become the standard response to Maxwell's dilemma, and its implications for the thermodynamics of computation reach into many areas of quantum and classical computing. It remains, however, still a hypothesis. Debate has often centred around simple toy models of a single particle in a box. Despite their simplicity, the ability of these systems to accurately represent thermodynamics (specifically to satisfy the second law) and whether or not they display Landauer Erasure, has been a matter of ongoing argument. The recent Norton-Ladyman controversy is one such example. In this paper we introduce a programming language to describe these simple thermodynamic processes, and give a formal operational semantics and program logic as a basis for formal reasoning about thermodynamic systems. We formalise the basic single-particle operations as statements in the language, and then show that the second law must be satisfied by any composition of these basic operations. This is done by finding a computational invariant of the system. We show, furthermore, that this invariant requires an erasure cost to exist within the system, equal to kTln2 for a bit of information: Landauer Erasure becomes a theorem of the formal system. The Norton-Ladyman controversy can therefore be resolved in a rigorous fashion, and moreover the formalism we introduce gives a set of reasoning tools for further analysis of Landauer erasure, which are provably consistent with the second law of thermodynamics.

Thermodynamics of stochastic Turing machines.

In analogy to Brownian computers we explicitly show how to construct stochastic models which mimic the behavior of a general-purpose computer (a Turing machine). Our models are discrete state systems obeying a Markovian master equation, which are logically reversible and have a well-defined and consistent thermodynamic interpretation. The resulting master equation, which describes a simple one-step process on an enormously large state space, allows us to thoroughly investigate the thermodynamics of computation for this situation. Especially in the stationary regime we can well approximate the master equation by a simple Fokker-Planck equation in one dimension. We then show that the entropy production rate at steady state can be made arbitrarily small, but the total (integrated) entropy production is finite and grows logarithmically with the number of computational steps.

Going Round in Circles: Landauer vs. Norton on the Thermodynamics of Computation

There seems to be a consensus among physicists that there is a connection between information processing and thermodynamics. In particular, Landauer’s Principle (LP) is widely assumed as part of the foundation of information theoretic/computational reasoning in diverse areas of physics including cosmology. It is also often appealed to in discussions about Maxwell’s demon and the status of the Second Law of Thermodynamics. However, LP has been challenged. In his 2005, Norton argued that LP has not been proved. LPSG offered a new proof of LP. Norton argued that the LPSG proof is unsound and Ladyman and Robertson defended it. However, Norton’s latest work also generalizes his critique to argue for a no go result that he purports to be the end of the thermodynamics of computation. Here we review the dialectic as it currently stands and consider Norton’s no go result.

Thermodynamic and logical reversibilities revisited

We review and investigate the general theory of the thermodynamics of computation, and derive the fundamental inequalities that set the lower bounds of the work requirement and the heat emission during a computation. These inequalities constitute the generalized Landauer principle, where the information contents are involved in the second law of thermodynamics. We discuss in detail the relationship between the thermodynamic and logical reversibilities; the former is related to the entropy production in the total system including a heat bath, while the latter is related to the entropy change only in the logical states of the memory. In particular, we clarify that any logically irreversible computation can be performed in a thermodynamically reversible manner in the quasi-static limit, which does not contradict the conventional Landauer principle. Our arguments would serve as the theoretical foundation of the thermodynamics of computation in terms of modern statistical physics.

The End of the Thermodynamics of Computation: A No-Go Result

The thermodynamics of computation assumes that computational processes at the molecular level can be brought arbitrarily close to thermodynamic reversibility and that thermodynamic entropy creation is unavoidable only in data erasure or the merging of computational paths, in accord with Landauer’s principle. The no-go result shows that fluctuations preclude completion of thermodynamically reversible processes. Completion can be achieved only by irreversible processes that create thermodynamic entropy in excess of the Landauer limit.

All Shook Up: Fluctuations, Maxwell’s Demon and the Thermodynamics of Computation

The most successful exorcism of Maxwell’s demon is Smoluchowski’s 1912 observation that thermal fluctuations would likely disrupt the operation of any molecular-scale demonic machine. A later tradition sought to exorcise Maxwell’s demon by assessing the entropic cost of the demon’s processing of information. This later tradition fails since these same thermal fluctuations invalidate the molecular-scale manipulations upon which the thermodynamics of computation is based. A new argument concerning conservation of phase space volume shows that all Maxwell’s demons must fail.

Ubiquity symposium: Evolutionary computation and the processes of life

While evolution has inspired algorithmic methods of heuristic optimization, little has been done in the way of using concepts of computation to advance our understanding of salient aspects of biological phenomena. The authors argue under reasonable assumptions, interesting conclusions can be drawn that are of relevance to behavioral evolution. The authors will focus on two important features of life---robustness and fitness---which, they will argue, are related to algorithmic probability and to the thermodynamics of computation, disciplines that may be capable of modeling key features of living organisms, and which can be used in formulating new algorithms of evolutionary computation.

Information Theory and Computational Thermodynamics: Lessons for Biology from Physics

We survey a few aspects of the thermodynamics of computation, connecting information, thermodynamics, computability and physics. We suggest some lines of research into how information theory and computational thermodynamics can help us arrive at a better understanding of biological processes. We argue that while a similar connection between information theory and evolutionary biology seems to be growing stronger and stronger, biologists tend to use information simply as a metaphor. While biologists have for the most part been influenced and inspired by information theory as developed by Claude Shannon, we think the introduction of algorithmic complexity into biology will turn out to be a much deeper and more fruitful cross-pollination.

Life as Thermodynamic Evidence of Algorithmic Structure in Natural Environments

In evolutionary biology, attention to the relationship between stochastic organisms and their stochastic environments has leaned towards the adaptability and learning capabilities of the organisms rather than toward the properties of the environment. This article is devoted to the algorithmic aspects of the environment and its interaction with living organisms. We ask whether one may use the fact of the existence of life to establish how far nature is removed from algorithmic randomness. The paper uses a novel approach to behavioral evolutionary questions, using tools drawn from information theory, algorithmic complexity and the thermodynamics of computation to support an intuitive assumption about the near optimal structure of a physical environment that would prove conducive to the evolution and survival of organisms, and sketches the potential of these tools, at present alien to biology, that could be used in the future to address different and deeper questions. We contribute to the discussion of the algorithmic structure of natural environments and provide statistical and computational arguments for the intuitive claim that living systems would not be able to survive in completely unpredictable environments, even if adaptable and equipped with storage and learning capabilities by natural selection (brain memory or DNA).

Complexity at large

Complexity at large

Some Computational Aspects of Essential Properties of Evolution and Life

AbstractWhile evolution has inspired algorithmic methods of heuristic optimisation, little has been done in the way of using concepts of computation to advance our understanding of salient aspects of biological evolution. We argue that under reasonable assumptions, interesting conclusions can be drawn that are of relevance to behavioural evolution. We will focus on two important features of life--robustness and fitness optimisation--which, we will argue, are related to algorithmic probability and to the thermodynamics of computation, subjects that may be capable of explaining and modelling key features of living organisms, and which can be used in understanding and formulating algorithms of evolutionary computation.

Waiting for Landauer

Landauer's Principle asserts that there is an unavoidable cost in thermodynamic entropy creation when data is erased. It is usually derived from incorrect assumptions, most notably, that erasure must compress the phase space of a memory device or that thermodynamic entropy arises from the probabilistic uncertainty of random data. Recent work seeks to prove Landauer's Principle without using these assumptions. I show that the processes assumed in the proof, and in the thermodynamics of computation more generally, can be combined to produce devices that both violate the second law and erase data without entropy cost, indicating an inconsistency in the theoretical system. Worse, the standard repertoire of processes selectively neglects thermal fluctuations. Concrete proposals for how we might measure dissipationlessly and expand single molecule gases reversibly are shown to be fatally disrupted by fluctuations. Reversible, isothermal processes on molecular scales are shown to be disrupted by fluctuations that can only be overcome by introducing entropy creating, dissipative processes.

When–and how–can a cellular automaton be rewritten as a lattice gas?

Both cellular automata (CA) and lattice-gas automata (LG) provide finite algorithmic presentations for certain classes of infinite dynamical systems studied by symbolic dynamics; it is customary to use the terms ‘cellular automaton’ and ‘lattice gas’ for a dynamic system itself as well as for its presentation. The two kinds of presentation share many traits but also display profound differences on issues ranging from decidability to modeling convenience and physical implementability. Following a conjecture by Toffoli and Margolus, it had been proved by Kari that any invertible CA, at least up to two dimensions, can be rewritten as an isomorphic LG. But until now it was not known whether this is possible in general for noninvertible CA—which comprise “almost all” CA and represent the bulk of examples in theory and applications. Even circumstantial evidence–whether in favor or against–was lacking. Here, for noninvertible CA, (a) we prove that an LG presentation is out of the question for the vanishingly small class of surjective ones. We then turn our attention to all the rest–noninvertible and nonsurjective–which comprise all the typical ones, including Conway’s ‘Game of Life’. For these (b) we prove by explicit construction that all the one-dimensional ones are representable as LG, and (c) we present and motivate the conjecture that this result extends to any number of dimensions. The tradeoff between dissipation rate and structural complexity implied by the above results have compelling implications for the thermodynamics of computation at a microscopic scale.