Implicit Deep Learning

Laurent El Ghaoui,Armin Askari,Fangda Gu,Bertrand Travacca,Alicia Tsai

doi:10.1137/20m1358517

Abstract

Related DatabasesWeb of Science You must be logged in with an active subscription to view this.Article DataHistorySubmitted: 7 August 2020Accepted: 18 June 2021Published online: 16 September 2021Keywordsdeep learning, deep equilibrium models, Perron--Frobenius theory, fixed-point equations, robustness, adversarial attacksAMS Subject Headings690C26, 49M99, 65K10, 62M45, 26B10Publication DataISSN (online): 2577-0187Publisher: Society for Industrial and Applied MathematicsCODEN: sjmdaq

Highlights

We consider a new class of deep learning models that are based on implicit prediction rules
Such rules are not obtained via a recursive procedure through several layers, as in current neural networks. They are based on solving a fixed-point equation in some single “state” vector x ∈ Rn
Robustness analysis: We describe how to analyze the robustness properties of a given implicit model, deriving bounds on the state under input perturbations, and generating adversarial attacks

Summary

Introduction

We consider a new class of deep learning models that are based on implicit prediction rules. Such rules are not obtained via a recursive procedure through several layers, as in current neural networks. Implicit rules open up the possibility of using novel architectures and prediction rules for deep learning, which are not based on any notion of “network” or “layers,” as is classically understood They allow one to consider rigorous approaches to challenging problems in deep learning, including robustness analysis, sparsity, and interpretability, and feature selection. Well-posedness and composition (section 2): In contrast with standard deep networks, implicit models may not be well-posed, in the sense that the equilibrium equation may have no or multiple solutions.

Lipschitz

B1C2 B1

Findings

A11 A12 B1