Mathematical expression recognition using a new deep neural model

Abstract

In this paper, we propose a novel deep neural model for Mathematical Expression Recognition (MER). The proposed model uses encoder–decoder transformer architecture that is supported by additional pre/post-processing modules, to recognize the image of mathematical formula and convert it to a well-formed language. A novel pre-processing module based on domain prior knowledge is proposed to generate random pads around the formula’s image to create more efficient feature maps and keeps all the encoder neurons active during the training process. Also, a new post-processing module is developed which uses a sliding window to extract additional position-based information from the feature map, that is proved to be useful in the recognition process. The recurrent decoder module uses the combination of feature maps and the additional position-based information, which takes advantage of a soft attention mechanism, to extract the formula context into the LaTeX well-formed language. Finally, a novel Reinforcement Learning (RL) module processes the decoder output and tunes its results by sending proper feedbacks to the previous steps. The experimental results on im2latex-100k benchmark dataset indicate that each devised pre/post-processing as well as the RL refinement module has a positive effect on the performance of the proposed model. The results also demonstrate the higher accuracy of the proposed model compared to the state-of-the-art methods.