Mathematical Expression Recognition Research Articles

Contrastive representation enhancement and learning for handwritten mathematical expression recognition

Handwritten mathematical expression recognition (HMER) is an appealing task due to its wide applications and research challenges. Previous deep learning-based methods used string decoder to emphasize on expression symbol awareness and achieved considerable recognition performance. However, these methods still meet an obstacle in recognizing handwritten symbols with varying appearance, in which huge appearance variations significantly lead to the ambiguity of symbol representation. To this end, our intuition is to employ printed expressions with unified appearance to serve as the template of handwritten expressions, alleviating the effects brought by varying symbol appearance. In this paper, we propose a contrastive learning method, where handwritten symbols with identical semantic are clustered together through the guidance of printed symbols, leading model to enhance the robustness of symbol semantic representations. Specifically, we propose an anchor generation scheme to obtain printed expression images corresponding with handwritten expressions. We propose a contrastive learning objective, termed Semantic-NCE Loss, to pull together printed and handwritten symbols with identical semantic. Moreover, we employ a string decoder to parse the calibrated semantic representations, outputting satisfactory expression symbols. The experiment results on benchmark datasets CROHME 14/16/19 demonstrate that our method noticeably improves recognition accuracy of handwritten expressions and outperforms the standard string decoder methods.

SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

Recognition of Handwritten Mathematical Expressions Using Systems of Convolutional Neural Networks

Accurate recognition of handwritten mathematical expressions has proven difficult due to their two-dimensional structure. Various machine-learning techniques have previously been employed to transcribe handwritten math, including approaches based on convolutional neural networks (CNNs) and larger encoder/decoder-based models. In this work, we explore a CNN-based method for transcribing handwritten math expressions into the typesetting language known as LaTeX. This approach utilizes machine learning not only for classifying individual characters but also for extracting individual characters from handwritten inputs and determining what forms of two-dimensionality exist within the expression. This approach achieves significant reliability when recognizing common mathematical expressions.

Investigating Models for the Transcription of Mathematical Formulas in Images

The automated transcription of mathematical formulas represents a complex challenge that is of great importance for digital processing and comprehensibility of mathematical content. Consequently, our goal was to analyze state-of-the-art approaches for the transcription of printed mathematical formulas on images into spoken English text. We focused on two approaches: (1) The combination of mathematical expression recognition (MER) models and natural language processing (NLP) models to convert formula images first into LaTeX code and then into text, and (2) the direct conversion of formula images into text using vision-language (VL) models. Since no dataset with printed mathematical formulas and corresponding English transcriptions existed, we created a new dataset, Formula2Text, for fine-tuning and evaluating our systems. Our best system for (1) combines the MER model LaTeX-OCR and the NLP model BART-Base, achieving a translation error rate of 36.14% compared with our reference transcriptions. In the task of converting LaTeX code to text, BART-Base, T5-Base, and FLAN-T5-Base even outperformed ChatGPT, GPT-3.5 Turbo, and GPT-4. For (2), the best VL model, TrOCR, achieves a translation error rate of 42.09%. This demonstrates that VL models, predominantly employed for classical image captioning tasks, possess significant potential for the transcription of mathematical formulas in images.

Improving Handwritten Mathematical Expression Recognition via Similar Symbol Distinguishing

Handwritten mathematical expression recognition (HMER) is an essential task in the OCR community, which consists of two sub-tasks, i.e., symbol recognition and structure parsing. Modern literature treats HMER as a LaTeX sequence predicting problem that simultaneously recognizes symbols and parses the structures of MEs. Although deep learning-based HMER methods have been achieving promising results on public benchmarks, it is admitted that the misclassification error between visually similar symbols still prevents these approaches from more generalized scenes. In this paper, we try to solve this issue from three aspects. 1) We enhanced the feature extraction progress by introducing path signature features, which incorporates local writing details and global spatial information. 2) We developed a language model that uses contextual information to correct the symbols misclassified by vision-only-based recognition models. 3) We solved the misalignment problem in existing ensemble method by designing a dynamic time warping (DTW) based algorithm. By combining the above improvements, our method achieved state-of-the-art results on three CROHME benchmarks, outperforming previous methods by a large margin.

A tree-based model with branch parallel decoding for handwritten mathematical expression recognition

Handwritten mathematical expression recognition (HMER) is a challenging task in the field of computer vision due to the complex two-dimensional spatial structure and diverse handwriting styles of mathematical expressions (MEs). Recent mainstream approach treats MEs as objects with tree structures, modeled by sequence decoders or tree decoders. These decoders recognize the symbols and relationships between symbols in MEs in depth-first order, resulting in long decoding steps that can harm their performance, particularly for MEs with complex structures. In this paper, we propose a novel tree-based model with branch parallel decoding for HMER, which parses the structures of ME trees by explicitly predicting the relationships between symbols. In addition, a query constructing module is proposed to assist the decoder in decoding the branches of ME trees in parallel, thus reducing the number of decoding time steps and alleviating the problem of long sequence attention decoding. As a result, our model outperforms existing models on three widely-used benchmarks and demonstrates significant improvements in HMER performance.

Offline handwritten mathematical expression recognition with graph encoder and transformer decoder

Handwritten mathematical expression recognition (HMER) has attracted extensive attention. Despite the significant progress achieved in recent years attributed to the development of deep learning approaches, HMER remains a challenge due to the complex spatial structure and variable writing styles. Encoder–decoder models with attention mechanism, which treats HMER as an image-to-sequence (i.e. LaTeX) generation task, have boosted the accuracy, but suffer from low interpretability in that the symbols are not segmented explicitly. Symbol segmentation is desired for facilitating post-processing and human interaction in real applications. In this paper, we formulate the mathematical expression as a graph and propose a Graph-Encoder-Transformer-Decoder (GETD) approach for HMER. For constructing the graph from input image, candidate symbols are first detected using an object detector and represented as the nodes of a graph, called symbol graph, and the edges of the graph encodes the between-symbol relationship. The spatial information is aggregated in a graph neural network (GNN), and a Transformer-based decoder is used to identify the symbol classes and structure from the graph. Experiments on public datasets demonstrate that our GETD model achieves competitive expression recognition performance while offering good interpretability compared with previous methods.

Mathematical expression recognition using a new deep neural model

In this paper, we propose a novel deep neural model for Mathematical Expression Recognition (MER). The proposed model uses encoder–decoder transformer architecture that is supported by additional pre/post-processing modules, to recognize the image of mathematical formula and convert it to a well-formed language. A novel pre-processing module based on domain prior knowledge is proposed to generate random pads around the formula’s image to create more efficient feature maps and keeps all the encoder neurons active during the training process. Also, a new post-processing module is developed which uses a sliding window to extract additional position-based information from the feature map, that is proved to be useful in the recognition process. The recurrent decoder module uses the combination of feature maps and the additional position-based information, which takes advantage of a soft attention mechanism, to extract the formula context into the LaTeX well-formed language. Finally, a novel Reinforcement Learning (RL) module processes the decoder output and tunes its results by sending proper feedbacks to the previous steps. The experimental results on im2latex-100k benchmark dataset indicate that each devised pre/post-processing as well as the RL refinement module has a positive effect on the performance of the proposed model. The results also demonstrate the higher accuracy of the proposed model compared to the state-of-the-art methods.

The IBEM dataset: A large printed scientific image dataset for indexing and searching mathematical expressions

Searching for information in printed scientific documents is a challenging problem that has recently received special attention from the Pattern Recognition research community. Mathematical expressions are complex elements that appear in scientific documents, and developing techniques for locating and recognizing them requires the preparation of datasets that can be used as benchmarks. Most current techniques for dealing with mathematical expressions are based on Machine Learning techniques which require a large amount of annotated data. These datasets must be prepared with ground-truth information for automatic training and testing. However, preparing large datasets with ground-truth is a very expensive and time-consuming task. This paper introduces the IBEM dataset, consisting of scientific documents that have been prepared for mathematical expression recognition and searching. This dataset consists of 600 documents, more than 8200 page images with more than 160000 mathematical expressions. It has been automatically generated from the ▪ version of the documents and can be enlarged easily. The ground-truth includes the position at the page level and the ▪ transcript for mathematical expressions both embedded in the text and displayed. This paper also reports a baseline classification experiment with mathematical symbols and a baseline experiment of Mathematical Expression Recognition performed on the IBEM dataset. These experiments aim to provide some benchmarks for comparison purposes so that future users of the IBEM dataset can have a baseline framework.

AdamR-GRUs: Adaptive momentum-based Regularized GRU for HMER problems

Handwritten Mathematical Expression Recognition (HMER) is essential to online education and scientific research. However, discerning the length and characters of handwritten mathematical expressions remains a formidable challenge. In this study, we propose an encoder–decoder architecture incorporating a novel Adaptive momentum-based Regularized Gated Recurrent Unit (AdamR-GRU) to solve this problem. It incorporates a novel gate within the GRU framework to maintain input histories, enabling extended memory retention and faster convergence. To assess the performance of AdamR-GRU, we conducted a series of comprehensive experiments on the CROHME 2014, 2016, and 2019 datasets. The results indicate that the AdamR-GRU model exhibits competitive performance with existing state-of-the-art methods regarding Expression Recognition Rate (exprate). Moreover, our model achieves state-of-the-art accuracy concerning 1, 2, and 3 symbol-level errors (top 1, 2, 3 error accuracy) on the CROHME 2014 and 2016 datasets. For the CROHME 2019 dataset, the model attains state-of-the-art performance in top 2 and 3 error accuracy. AdamR-GRU demonstrates reduced computational complexity and memory usage compared to alternative contemporary models.

Offline handwritten mathematical expression recognition based on YOLOv5s

Offline handwritten mathematical expression recognition based on YOLOv5s

Discriminative estimation of probabilistic context-free grammars for mathematical expression recognition and retrieval

We present a discriminative learning algorithm for the probabilistic estimation of two-dimensional probabilistic context-free grammars (2D-PCFG) for mathematical expressions recognition and retrieval. This algorithm is based on a generalization of the H-criterion as the objective function and the growth transformations as the optimization method. For the development of the discriminative estimation algorithm, the N-best interpretations provided by the 2D-PCFG have been considered. Experimental results are reported on two available datasets: Im2Latex and IBEM. The first experiment compares the proposed discriminative estimation method with the classic Viterbi-based estimation method. The second one studies the performance of the estimated models depending on the length of the mathematical expressions and the number of admissible errors in the metric used.

A Step Toward an Automatic Handwritten Homework Grading System for Mathematics

An automatic system that helps teachers and students verify the correctness of handwritten derivation in mathematics homework is proposed. The system acquires input image containing handwritten mathematical derivation. In our preliminary study, the system that comprises only mathematical expression recognition (MER) and computer algebra system (CAS) did not perform well due to high misrecognition rate. Therefore, our study focuses on fixing the misrecognized symbols by using symbols replacement and the surrounding information. If all the original mathematical expressions (MEs) in the derivation sequence are already equivalent, the derivation is marked as “correct”. Otherwise, the symbols with low recognition confidence will be replaced by other possible candidates to maximize the number of equivalent MEs in that derivation. If there is none of symbols replacement that makes every line equivalent, the derivation is marked as “incorrect”. The recursive expression tree comparison was applied to report the types of mistake for those problems marked as incorrect. Finally, the performance of the system was evaluated by the digitally generated dataset of 6,000 handwritten mathematical derivations. The results showed that the symbols replacement improve the F1-score of derivation step marking from 69.41 to 95.95 % for the addition/ subtraction dataset and from 61.45 to 89.95 % for the multiplication dataset when compared to the case of using raw recognized string without symbols replacement.

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

Synthetic Data Generation and Shuffled Multi-Round Training Based Offline Handwritten Mathematical Expression Recognition

Synthetic Data Generation and Shuffled Multi-Round Training Based Offline Handwritten Mathematical Expression Recognition

Real Time Air-Written Mathematical Expression Recognition for Children’s Enhanced Learning

Real Time Air-Written Mathematical Expression Recognition for Children’s Enhanced Learning

Tree-based data augmentation and mutual learning for offline handwritten mathematical expression recognition

Recently, thanks to the successful application of the attention-based encoder-decoder framework, handwritten mathematical expression recognition (HMER) has achieved significant improvement. However, HMER is still a challenging task in the handwriting recognition area, which suffers from the ambiguity of handwritten symbols, the two-dimensional structure of mathematical expressions, and the lack of labeled data. In this paper, we attempt to improve the recognition performance and generalization ability of the existing state-of-the-art method from two perspectives: data augmentation and model design. We first propose a tree-based multi-level (including symbol level, sub-expression level, and image level) data augmentation strategy, which can generate many synthetic images. Then, we present a novel encoder-decoder hybrid model via tree-based mutual learning to fully utilize the complementarity between tree decoder and string decoder. Benefitting from our data augmentation strategy, we achieve 58.47%/57.82%/62.67% and 74.45% expression recognition accuracy respectively on the CROHME14/16/19 competition datasets and the OffRaSHME20 competition dataset. Moreover, tree-based data augmentation is a key technology to our champion system for the OffRaSHME20 competition. Our tree-based mutual learning method further improves the recognition accuracy to 61.63%/59.81%/64.38% and 75.68% on these datasets. Further quantitative and qualitative analyses also demonstrate the effectiveness and robustness of our proposed methods.

Automated Math Symbol Classification Using SVM

Handwritten character/symbol recognition is an important area of research in the present digital world. The solving of problems such as recognizing handwritten characters/symbols written in different styles can make the human job easier. Mathematical expression recognition using machines has become a subject of serious research. The main motivation for this work is both recognizing of the handwritten mathematical symbol, digits and characters which will be used for mathematical expression recognition. The system first identifies the contour in handwritten document segmentation and features extracted are given into SVM classifier for classification. GLCM and Zernike Moments are the two different feature extraction techniques used in this work. SVM with RBF kernel is used for classification. Zernike Moment features overperforms than GLCM. Zernike Moment achieves 97.89% accuracy and GLCM achieves 87.61% accuracy.

Deep Learning based Handwritten Polynomial Equation Solver

Abstract: Polynomials are algebraic expressions involving a sum of powers in one or more variables multiplied by its coefficients. If x is a variable, a0x n +a1x n-1 +a2x n-2 +....+an then it's a (n) powers polynomial. Human is capable to solve this type of mathematical problems. In this work, we propose a system in which machines can achieve the cognitional skills that can understand the problem by visual context. By taking an input image of Handwritten polynomial equations and simplifies the problem by generating the answer as an output. Here machine can able to solve quadratic, cubic, quartic, quantic, sextic as well as (n) powers polynomials. This proposed work can be workable in an embedded system as well as a mobile application. In this scope for recognition purposes, we use a CNN model. Robust handwritten character recognition is a tricky job in the area of image processing. Among all the problem handwritten mathematical expression recognition is one of the complicated issue in the area of computer vision research. Segmentation and classification of specific character makes the task more difficult. In this paper a group of handwritten quadratic equation as well as a single quadratic equation are considered to recognize and make a solution for those equations. Horizontal compact projection analysis and combined connected component analysis methods are used for segmentation. For classification of specific character we apply Convolutional Neural Network. Each of the correct detection, character string operation is used for the solution of the equation. The proposed workflow system automatically simplifies the Handwritten polynomial equation and has been done a really good performance. Developing an automatic equation recognizer and solver has been a desire of the researchers who worked in the field of NLP for many years.

TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition

In recent years, tree decoders become more popular than LaTeX string decoders in the field of handwritten mathematical expression recognition (HMER) as they can capture the hierarchical tree structure of mathematical expressions. However previous tree decoders converted the tree structure labels into a fixed and ordered sequence, which could not make full use of the diversified expression of tree labels. In this study, we propose a novel tree decoder (TDv2) to fully utilize the tree structure labels. Compared with previous tree decoders, this new model does not require a fixed priority for different branches of a node during training and inference, which can effectively improve the model generalization capability. The input and output of the model make full use of the tree structure label, so that there is no need to find the parent node in the decoding process, which simplifies the decoding process and adds a prior information to help predict the node. We verified the effectiveness of each part of the model through comprehensive ablation experiments and attention visualization analysis. On the authoritative CROHME 14/16/19 datasets, our method achieves the state-of-the-art results.