Flat Minima Research Articles

Flat minima generalize for low-rank matrix recovery

Abstract Empirical evidence suggests that for a variety of overparameterized nonlinear models, most notably in neural network training, the growth of the loss around a minimizer strongly impacts its performance. Flat minima—those around which the loss grows slowly—appear to generalize well. This work takes a step towards understanding this phenomenon by focusing on the simplest class of overparameterized nonlinear models: those arising in low-rank matrix recovery. We analyse overparameterized matrix and bilinear sensing, robust principal component analysis, covariance matrix estimation and single hidden layer neural networks with quadratic activation functions. In all cases, we show that flat minima, measured by the trace of the Hessian, exactly recover the ground truth under standard statistical assumptions. For matrix completion, we establish weak recovery, although empirical evidence suggests exact recovery holds here as well. We complete the paper with synthetic experiments that illustrate our findings.

Improving Differentiable Architecture Search via self-distillation

Differentiable Architecture Search (DARTS) is a simple yet efficient Neural Architecture Search (NAS) method. During the search stage, DARTS trains a supernet by jointly optimizing architecture parameters and network parameters. During the evaluation stage, DARTS discretizes the supernet to derive the optimal architecture based on architecture parameters. However, recent research has shown that during the training process, the supernet tends to converge towards sharp minima rather than flat minima. This is evidenced by the higher sharpness of the loss landscape of the supernet, which ultimately leads to a performance gap between the supernet and the optimal architecture. In this paper, we propose Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap. We utilize self-distillation to distill knowledge from previous steps of the supernet to guide its training in the current step, effectively reducing the sharpness of the supernet’s loss and bridging the performance gap between the supernet and the optimal architecture. Furthermore, we introduce the concept of voting teachers, where multiple previous supernets are selected as teachers, and their output probabilities are aggregated through voting to obtain the final teacher prediction. Experimental results on real datasets demonstrate the advantages of our novel self-distillation-based NAS method compared to state-of-the-art alternatives.

The inverse variance–flatness relation in stochastic gradient descent is critical for finding flat minima

Despite tremendous success of the stochastic gradient descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions at flat minima of the loss function in high-dimensional weight space. Here, we investigate the connection between SGD learning dynamics and the loss function landscape. A principal component analysis (PCA) shows that SGD dynamics follow a low-dimensional drift-diffusion motion in the weight space. Around a solution found by SGD, the loss function landscape can be characterized by its flatness in each PCA direction. Remarkably, our study reveals a robust inverse relation between the weight variance and the landscape flatness in all PCA directions, which is the opposite to the fluctuation-response relation (aka Einstein relation) in equilibrium statistical physics. To understand the inverse variance-flatness relation, we develop a phenomenological theory of SGD based on statistical properties of the ensemble of minibatch loss functions. We find that both the anisotropic SGD noise strength (temperature) and its correlation time depend inversely on the landscape flatness in each PCA direction. Our results suggest that SGD serves as a landscape-dependent annealing algorithm. The effective temperature decreases with the landscape flatness so the system seeks out (prefers) flat minima over sharp ones. Based on these insights, an algorithm with landscape-dependent constraints is developed to mitigate catastrophic forgetting efficiently when learning multiple tasks sequentially. In general, our work provides a theoretical framework to understand learning dynamics, which may eventually lead to better algorithms for different learning tasks.

Interactions of selected organic molecules with a blue phosphorene monolayer: self-assembly, solvent effect, enhanced binding and fixation through coadsorbed gold clusters.

In this paper we investigate the interaction between a pristine blue phosphorene monolayer and selected organic molecules like amino acids and nucleic acid bases. These molecules are bound to the substrate by a weak van der Waals interaction leading to their physisorption. When isolated, they tend to orient themselves parallel to the surface and are located in flat minima with very low libration frequencies; thus the electronic structures of the substrate and physisorbed molecules are not affected except for relative shifts. Even though the regular self-assembly of these molecules on the pristine blue phosphorene cannot be realized under this weak interaction, only their irregular coating of the substrate can occur due to increased intermolecular coupling. In a solvent like water, the weak binding energy is further decreased. Gold adatoms and gold clusters can form strong chemical bonds with pristine blue phosphorene and modify its electronic and magnetic state depending on the coverage. While full coverage of a blue phosphorene monolayer by gold adatoms leads to instabilities followed by clustering, relatively lower coverage can attribute very interesting magnetic and electronic states, like a spin gapless semiconductor. When bound to the gold clusters already adsorbed on the blue phosphorene monolayer, amino acid and nucleic acid base molecules form relatively strong chemical bonds and hence can be fixed to the surface; they are reoriented to gain self-assembly character and the whole system acquires new functionalities.

Quantum Monte Carlo and genetic algorithm study of the potential energy surface of the H molecule

AbstractIn this article we present a characterization of the vibrational spectrum of the H molecule using the correlation function quantum Monte Carlo (CFQMC) method and a genetic algorithm study of the topology of the potential energy surface (PES) used. The vibrational modes associated with the H‐H2 torsion and stretching possess very flat minima. As a consequence the fundamental frequencies corresponding to these modes are poorly described in the harmonic approximation. In our genetic algorithm study of the PES using Cartesian coordinates we have found some unexpected minima. A careful analysis shows that, in the curvilinear coordinates in which the potential is described, some of these minima have identical coordinates. Nevertheless, they represent nonequivalent molecular geometries. The vibrational frequencies obtained in this work are not in good agreement with available experimental data as well as other computational methods. It shows that our method of detailed analysis of PESs could reveal shortcomings introduced by the use of curvilinear coordinates. © 2008 Wiley Periodicals, Inc. Int J Quantum Chem, 2008

Strong Jahn–Teller effect in the excited state: Anomalous temperature dependence of the zero-phonon line

We are studying the optical spectra of impurity centers in the case when the optical transition takes place between nondegenerate and two-fold degenerate electronic E-states characterized by a strong vibronic coupling. In the latter state the adiabatic surface has a shape of a deformed Mexican hat with three flat minima on the trough of the “hat.” We have found that in this case the usual T7-dependence of the homogeneous width of the zero-phonon line at low temperatures is replaced by a dependence close to the T3-one. Besides, the usual T4-dependent redshift of the line is replaced by the T2-dependent shift. Depending on the flatness of the adiabatic surface, the latter shift may be to the red or to the blue. The theory is applied to the zero-phonon line 637 nm of the nitrogen-containing N-V center in a diamond. In this system the temperature dependence of the ZPL results from the common action of the Jahn–Teller effect and the pseudo-Jahn–Teller effect in the excited E-state: the pseudo-Jahn–Teller effect “helps” to get the system close to the dynamical instability due to the enlargement of the Jahn–Teller distortion.

Compaction in a mantle with a very small melt concentration: Implications for the generation of carbonatitic and carbonate-bearing high alkaline mafic melt impregnations

Xenoliths entrained in alkaline basalts and kimberlites give strong evidence that mantle carbonatitic and carbonated high alkaline mafic silicate melts, which are initially produced at very low degrees of partial melting (≪1%), percolate and accumulate to form impregnations with a melt concentration of up to 10%. At present no compaction model has explained such huge local amplification of melt concentration. Recently, Bercovici et al. [1] have shown that the commonly used equations of compaction are not sufficiently general to describe all melt percolation processes in the mantle. In particular, they show that, when the melt concentration in the mantle is very low, the pressure jump ΔP between the solid and liquid fractions of the mantle mush is very important and plays a driving role during compaction. 1-D compaction waves generated with two different systems of equations are computed. Three types of wave-trains are observed, i.e. (1) sinusoidal waves; (2) periodic waves with flat minima and very acute maxima (‘witch hat waves’); (3) periodic solitary waves with flat maxima and extremely narrow minima (‘bowler hat waves’). When the initial melt distribution in the mantle is quite homogeneous, the compaction waves have sinusoidal shapes and can locally amplify the melt concentration by a factor less than two. When there is a drastic obstruction at the top of the wetted domain, the pressure jump ΔP between solid and liquid controls the shape of the waves. If the computation assumes the equality of pressure between the two phases (ΔP=0), the compaction wave has a ‘bowler hat shape’, and locally amplifies the melt concentration by a factor less than 5. Alternatively, simulations taking into account the pressure jump between phases ΔP predict compaction waves with ‘witch hat shape’. These waves collect a large quantity of melt promoting the development of magmons with local melt concentration exceeding 100× the background melt concentration. It is inferred that in a mantle with very low concentrations of carbonatitic or high alkaline mafic silicate melt the magmons are about 1 km thick and reach, in less than 1 Ma, a melt concentration of about 10%. The magmons are likely generated below the lithosphere at some distance away from the center of hot spots. This can explain the development of mantle carbonatitic eruptions in the African rift and the carbonatite and high alkaline mafic silicate volcanic activity in oceanic islands.

An MNDO approach to the symmetry of bromine bridging in substituted bromonium ions

Calculated energy profiles for the opening of six methyl bromonium ions show single, shallow or very flat minima corresponding to symmetrical or highly asymmetrical bridged structures depending on the symmetry of the substitution but not on the number of methyl groups.

Topological, stereochemical and NMR line assignments in alcohols and ketones by normalized NMR shifts induced by ytterbium

Yb(fod) 3-Induced 13C- and 1H-NMR shifts exhibit regular differences if normalized by setting the shift at the nucleus closest to the binding site to RS = 100%. 13C-NMR data from 70 alcohols show RS = 47± 4%, for Cβ (43±3% for primary alcohols); the normalized RS at the other carbon atoms depend on the anti (a)/gauche (g) arrangements at the intervening bonds as follows: Cy a 23%, g 33%; Cδ aa 11%, ag 16%, gg 25%; Ce aaa 5%, aga 9%, gga 14%, ggg 22%, etc. (±2-4%). Similar observations are made with 20 ketones. The RS values agree with relative geometry factors RG predicted by calculations of the pseudo contact shifts for all significant conformations. Very flat minima are usually observed in these analyses; the impact of this and other ambiguities for accurate geometry investigations with the LIS method is discussed.

Excitation of large amplitude electron plasma waves by laser

A hydrodynamic description is used to derive a nonlinear wave equation in the geometry of the capacitor model. For a streaming cold plasma the equation can be solved analytically. For the warm plasma case numerical results of high amplitude waves and wave breaking are presented. Flat minima and peaked maxima of the electron density and the oscillatory velocity are the most characteristic features of highly nonlinear electron waves. Their Bohm–Gross dispersion relation is modified by self-interaction of the fundamental mode.

PdPtAu alloys for low-temperature precision resistors

Palladium-rich PdPt and PdAu alloys exhibit a resistance-temperature behavior characterized by flat minima even after eliminating the Kondo effect by oxidation of the impurities. The opposite signs of their thermoelectric potentials against a superconductor permit the preparation of ternary PdPtAu alloys free of a thermoelectric power. The measurements resulted in a nomogram suitable for determining the composition of the alloy necessary for a resistivity minimum at a given temperature between 1.5 and 4.2 K. Alloys manufactured in this way have a temperature coefficient of resistivity of less than 10(-6)/K within a temperature interval of 0.5 K and a thermoelectric power against a superconductor of less than 1 nV/K.

The thermal expansion of alkali borate glasses and the boric oxide anomaly

A brief review is presented of the major theories for the so-called “boric-oxide anomaly” in alkali borate glasses, and a comparison is effected with recent published information on the structure of B 2O 3 and alkali borate glasses. These indicate serious shortcomings in current theories. An experimental study is reported of the mean linear thermal expansion coefficients between 25°C and −196°C of a series of glasses, prepared at composition intervals of 2 mole% alkali oxide, in all five alkali borate systems. The thermal expansion versus composition relations are characterized by broad, flat minima, with no single composition being uniquely identifiable with the minimum. It is concluded that the compositional variation of the thermal expansion coefficient of alkali borate glasses is not truly anomalous, but rather seems to vary with structural variations in a generally predictable manner. It is suggested that the term “anomaly” be reserved for the more serious problem of explaining the change in boron coordination with composition.