Reducing the Memory Requirement in Reverse Mode Automatic Differentiation by Solving TBR Flow Equations

Abstract

The fast computation of gradients in reverse mode Automatic Differentiation (AD) requires the generation of adjoint versions of every statement in the original code. Due to the resulting reversal of the control flow certain intermediate values have to be made available in reverse order to compute the local partial derivatives. This can be achieved by storing these values or by recomputing them when they become required. In any case one is interested in minimizing the size of this set. Following an extensive introduction of the “To-Be-Recorded” (TBR) problem we will present flow equations for propagating the TBR status of variables in the context of reverse mode AD of structured programs.KeywordsMemory RequirementIntermediate VariableLocal Partial DerivativeAdjoint ModelArray IndexThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.