Mathematical formula detection in document images: A new dataset and a new approach

Abstract

Most of existing mathematical formula detectors focus on detecting formula entities through object detection or instance segmentation techniques. However, these methods often fail to convey complete messages due to the absence of the contextual and layout information of mathematical formulas. For a more comprehensive understanding of mathematical formulas in document images, it is preferable to detect logical formula blocks that include one or multiple formula entities arranged in their natural reading order. These logical formula blocks enable the transmission of complete contextual messages of mathematical formulas and aid in the reconstruction of layout information of the document images, resulting in a more accurate mathematical formula detection. In this paper, we present a novel perspective on mathematical formula detection by framing it as a joint task of formula entity detection and formula relation extraction for identifying logical formula blocks. To this end, we introduce a new, large-scale dataset, called ArxivFormula, that includes well-annotated formula entity bounding boxes and formula relationships. We also propose a new approach, called FormulaDet, to address these two sub-tasks simultaneously. FormulaDet first employs a dynamic convolution-based formula entity detector, named DynFormula, to detect formula entities. It then uses a multi-modal transformer-based relation extraction method, named RelFormer, to group these detected formula entities into logical formula blocks. Extensive experiments on standard benchmarks in this field and the proposed dataset demonstrate that our FormulaDet can achieve significantly improved performance on formula entity detection and formula relation extraction compared to previous state-of-the-art methods. The joint detection and relation extraction approach provides a more thorough understanding of mathematical formulas in document images and effectively supports downstream tasks such as document layout analysis and scientific document digitization. The ArxivFormula dataset is publicly available at https://github.com/microsoft/ArxivFormula.