Abstract

A tool for benchmarking one of the algorithms most commonly used in machine-learning provides insight into its performance and could lead to a better theoretical understanding of how similar algorithms work.

Highlights

  • Motivated by the general aim to shed light on the behavior and performance of noisy-gradient-descent algorithms that are widely used in machine learning, we analytically investigate the performance of the Langevin algorithm in the noisy high-dimensional limit of a spiked matrix-tensor model

  • While both of these algorithms are designed with the aim to sample the posterior measure of the model, we show that the Langevin algorithm fails to find correlation with the signal in a considerable part of the approximate message passing (AMP) easy region

  • Our analysis is based on the Langevin state evolution equations, a generalization of the dynamical theory for mean field spin glasses, that describe the evolution of the algorithm in the large size limit

Read more

Summary

MOTIVATION

Algorithms based on noisy variants of gradients descent [1,2] stand at the root of many modern applications of data science and are being used in a wide range of highdimensional nonconvex optimization problems. The analysis of this setting has been carried out rigorously in a wide range of teacher-student models for high-dimensional inference and learning tasks as diverse as a planted clique [7], generalized linear models such as compressed sensing or phase retrieval [8], factorization of matrices and tensors [9,10], or simple models of neural networks [11] In these works, the information-theoretic optimal performances—the ones obtained by an ideal Bayes-optimal estimator, not limited in time and memory—have been computed. We bring these powerful methods and ideas into the realm of statistical learning

SPIKED MATRIX-TENSOR MODEL
BAYES-OPTIMAL ESTIMATION AND MESSAGE-PASSING ALGORITHM
LANGEVIN ALGORITHM AND ITS ANALYSIS
BEHAVIOR OF THE LANGEVIN ALGORITHM
GLASSY NATURE OF THE LANGEVIN HARD PHASE
F ðqth
DISCUSSION AND PERSPECTIVES
Approximate message passing and Bethe free entropy
Ykixtk k
X Ni log Zi þ
Averaged free entropy and its proof
Ai i supAi: i
Δ2ðN þ xÃ0xÃi xi þ ðp ΔpðN
State evolution of AMP and its analysis
Phase diagrams of spiked matrix-tensor model
Dynamical mean-field equations
Tpðt0Þ i1 ðt0Þx0i2
Integrodifferential equations
T 2 ðtÞΔ2
T 2 ðt00 Þ t00Þ: ðC17Þ ðC18Þ
Numerical checks on the dynamical algorithm
Extrapolation procedure
Initial conditions
Annealing protocol
Computation of the complexity through the replica method
Δ2 : ðE9Þ
Breakdown of the fluctuation-dissipation theorem in the Langevin hard phase
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.