C++ Code Generation for Fast Inference of Deep Learning Models in ROOT/TMVA

Abstract

We report the latest development in ROOT/TMVA, a new tool that takes trained ONNX deep learning models and emits C++ code that can be easily included and invoked for fast inference of the model, with minimal dependency. An introduction to SOFIE (System for Optimized Fast Inference code Emit) is presented, with examples of interface and generated code. We discuss the latest expanded support of a variety of neural network operators, including convolutional and recurrent layers, as well as the integration with RDataFrame. We demonstrate the latest performance of this framework with a set of benchmarks.