Using Directed Edge Separators to Increase Efficiency in the Determination of Jacobian Matrices via Automatic Differentiation

Abstract

Every numerical function evaluation can be represented as a directed acyclic graph (DAG), beginning at the initial input variable settings, and terminating at the output or corresponding function value(s). The “reverse mode” of automatic differentiation (AD) generates a “tape” which is a representation of this underlying DAG. In this work we illustrate that a directed edge separator in this underlying DAG can yield space and time efficiency gains in the application of AD. Use of directed edge separators to increase AD efficiency in different ways than proposed here has been suggested by other authors (Bischof and Haghighat, Hierarchical approaches to automatic differentiation. In: Berz M, Bischof C, Corliss G, Griewank A (eds) Computational differentiation: techniques, applications, and tools, SIAM, Philadelphia, PA, pp 83–94, 1996; Bucker and Rasch, ACM Trans Math Softw 29(4):440–457, 2003). In contrast to these previous works, our focus here is primarily on space. Furthermore, we explore two simple algorithms to find good directed edge separators, and show how these ideas can be applied recursively to great advantage. Initial numerical experiments are presented.