Mini-batch optimization enables training of ODE models on large-scale datasets

Paul Stapor,Leonard Schmiester,Christoph Wierling,Bodo M H Lange,Daniel Weindl,Jan Hasenauer,Simon Merkt,Dilan Pathirana

doi:10.1038/s41467-021-27374-6

Abstract

Quantitative dynamic models are widely used to study cellular signal processing. A critical step in modelling is the estimation of unknown model parameters from experimental data. As model sizes and datasets are steadily growing, established parameter optimization approaches for mechanistic models become computationally extremely challenging. Mini-batch optimization methods, as employed in deep learning, have better scaling properties. In this work, we adapt, apply, and benchmark mini-batch optimization for ordinary differential equation (ODE) models, thereby establishing a direct link between dynamic modelling and machine learning. On our main application example, a large-scale model of cancer signaling, we benchmark mini-batch optimization against established methods, achieving better optimization results and reducing computation by more than an order of magnitude. We expect that our work will serve as a first step towards mini-batch optimization tailored to ODE models and enable modelling of even larger and more complex systems than what is currently possible.

Highlights

Quantitative dynamic models are widely used to study cellular signal processing
Full-batch optimizers use line-search or trust-region approaches[50], which can deal with these non-evaluable points by adapting the step-size (Fig. 3a). We found these problems present in our benchmark examples (Fig. 3b, left), leading to failure of local optimization processes as available mini-batch optimization methods cannot handle failures of the objective function evaluation
We presented a framework for using mini-batch optimization in combination with advanced methods from dynamic modelling for the parameter estimation of ordinary differential equation (ODE) models in systems biology

Summary

Introduction

Quantitative dynamic models are widely used to study cellular signal processing. A critical step in modelling is the estimation of unknown model parameters from experimental data. We adapt, apply, and benchmark mini-batch optimization for ordinary differential equation (ODE) models, thereby establishing a direct link between dynamic modelling and machine learning. On our main application example, a large-scale model of cancer signaling, we benchmark mini-batch optimization against established methods, achieving better optimization results and reducing computation by more than an order of magnitude. We expect that our work will serve as a first step towards mini-batch optimization tailored to ODE models and enable modelling of even larger and more complex systems than what is currently possible. For large-scale ODE models with several hundred chemical species and thousands of experimental conditions, this can take tens of thousands of computing hours, even with state-of-the-art methods, such as adjoint sensitivity analysis and hierarchical optimization[10,14]. In multi-start local optimization, local optimization initializes at many random starting points in order to globally explore the parameter space

Methods

Results

Conclusion