Handwritten Mathematical Expression Recognition Research Articles

Contrastive representation enhancement and learning for handwritten mathematical expression recognition

Handwritten mathematical expression recognition (HMER) is an appealing task due to its wide applications and research challenges. Previous deep learning-based methods used string decoder to emphasize on expression symbol awareness and achieved considerable recognition performance. However, these methods still meet an obstacle in recognizing handwritten symbols with varying appearance, in which huge appearance variations significantly lead to the ambiguity of symbol representation. To this end, our intuition is to employ printed expressions with unified appearance to serve as the template of handwritten expressions, alleviating the effects brought by varying symbol appearance. In this paper, we propose a contrastive learning method, where handwritten symbols with identical semantic are clustered together through the guidance of printed symbols, leading model to enhance the robustness of symbol semantic representations. Specifically, we propose an anchor generation scheme to obtain printed expression images corresponding with handwritten expressions. We propose a contrastive learning objective, termed Semantic-NCE Loss, to pull together printed and handwritten symbols with identical semantic. Moreover, we employ a string decoder to parse the calibrated semantic representations, outputting satisfactory expression symbols. The experiment results on benchmark datasets CROHME 14/16/19 demonstrate that our method noticeably improves recognition accuracy of handwritten expressions and outperforms the standard string decoder methods.

SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

SATD: syntax-aware handwritten mathematical expression recognition based on tree-structured transformer decoder

Recognition of Handwritten Mathematical Expressions Using Systems of Convolutional Neural Networks

Accurate recognition of handwritten mathematical expressions has proven difficult due to their two-dimensional structure. Various machine-learning techniques have previously been employed to transcribe handwritten math, including approaches based on convolutional neural networks (CNNs) and larger encoder/decoder-based models. In this work, we explore a CNN-based method for transcribing handwritten math expressions into the typesetting language known as LaTeX. This approach utilizes machine learning not only for classifying individual characters but also for extracting individual characters from handwritten inputs and determining what forms of two-dimensionality exist within the expression. This approach achieves significant reliability when recognizing common mathematical expressions.

Improving Handwritten Mathematical Expression Recognition via Similar Symbol Distinguishing

Handwritten mathematical expression recognition (HMER) is an essential task in the OCR community, which consists of two sub-tasks, i.e., symbol recognition and structure parsing. Modern literature treats HMER as a LaTeX sequence predicting problem that simultaneously recognizes symbols and parses the structures of MEs. Although deep learning-based HMER methods have been achieving promising results on public benchmarks, it is admitted that the misclassification error between visually similar symbols still prevents these approaches from more generalized scenes. In this paper, we try to solve this issue from three aspects. 1) We enhanced the feature extraction progress by introducing path signature features, which incorporates local writing details and global spatial information. 2) We developed a language model that uses contextual information to correct the symbols misclassified by vision-only-based recognition models. 3) We solved the misalignment problem in existing ensemble method by designing a dynamic time warping (DTW) based algorithm. By combining the above improvements, our method achieved state-of-the-art results on three CROHME benchmarks, outperforming previous methods by a large margin.

A tree-based model with branch parallel decoding for handwritten mathematical expression recognition

Handwritten mathematical expression recognition (HMER) is a challenging task in the field of computer vision due to the complex two-dimensional spatial structure and diverse handwriting styles of mathematical expressions (MEs). Recent mainstream approach treats MEs as objects with tree structures, modeled by sequence decoders or tree decoders. These decoders recognize the symbols and relationships between symbols in MEs in depth-first order, resulting in long decoding steps that can harm their performance, particularly for MEs with complex structures. In this paper, we propose a novel tree-based model with branch parallel decoding for HMER, which parses the structures of ME trees by explicitly predicting the relationships between symbols. In addition, a query constructing module is proposed to assist the decoder in decoding the branches of ME trees in parallel, thus reducing the number of decoding time steps and alleviating the problem of long sequence attention decoding. As a result, our model outperforms existing models on three widely-used benchmarks and demonstrates significant improvements in HMER performance.

Offline handwritten mathematical expression recognition with graph encoder and transformer decoder

Handwritten mathematical expression recognition (HMER) has attracted extensive attention. Despite the significant progress achieved in recent years attributed to the development of deep learning approaches, HMER remains a challenge due to the complex spatial structure and variable writing styles. Encoder–decoder models with attention mechanism, which treats HMER as an image-to-sequence (i.e. LaTeX) generation task, have boosted the accuracy, but suffer from low interpretability in that the symbols are not segmented explicitly. Symbol segmentation is desired for facilitating post-processing and human interaction in real applications. In this paper, we formulate the mathematical expression as a graph and propose a Graph-Encoder-Transformer-Decoder (GETD) approach for HMER. For constructing the graph from input image, candidate symbols are first detected using an object detector and represented as the nodes of a graph, called symbol graph, and the edges of the graph encodes the between-symbol relationship. The spatial information is aggregated in a graph neural network (GNN), and a Transformer-based decoder is used to identify the symbol classes and structure from the graph. Experiments on public datasets demonstrate that our GETD model achieves competitive expression recognition performance while offering good interpretability compared with previous methods.

AdamR-GRUs: Adaptive momentum-based Regularized GRU for HMER problems

Handwritten Mathematical Expression Recognition (HMER) is essential to online education and scientific research. However, discerning the length and characters of handwritten mathematical expressions remains a formidable challenge. In this study, we propose an encoder–decoder architecture incorporating a novel Adaptive momentum-based Regularized Gated Recurrent Unit (AdamR-GRU) to solve this problem. It incorporates a novel gate within the GRU framework to maintain input histories, enabling extended memory retention and faster convergence. To assess the performance of AdamR-GRU, we conducted a series of comprehensive experiments on the CROHME 2014, 2016, and 2019 datasets. The results indicate that the AdamR-GRU model exhibits competitive performance with existing state-of-the-art methods regarding Expression Recognition Rate (exprate). Moreover, our model achieves state-of-the-art accuracy concerning 1, 2, and 3 symbol-level errors (top 1, 2, 3 error accuracy) on the CROHME 2014 and 2016 datasets. For the CROHME 2019 dataset, the model attains state-of-the-art performance in top 2 and 3 error accuracy. AdamR-GRU demonstrates reduced computational complexity and memory usage compared to alternative contemporary models.

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

Offline Handwritten Mathematical Expression Recognition based on CLIP enhancement

Synthetic Data Generation and Shuffled Multi-Round Training Based Offline Handwritten Mathematical Expression Recognition

Synthetic Data Generation and Shuffled Multi-Round Training Based Offline Handwritten Mathematical Expression Recognition

Tree-based data augmentation and mutual learning for offline handwritten mathematical expression recognition

Recently, thanks to the successful application of the attention-based encoder-decoder framework, handwritten mathematical expression recognition (HMER) has achieved significant improvement. However, HMER is still a challenging task in the handwriting recognition area, which suffers from the ambiguity of handwritten symbols, the two-dimensional structure of mathematical expressions, and the lack of labeled data. In this paper, we attempt to improve the recognition performance and generalization ability of the existing state-of-the-art method from two perspectives: data augmentation and model design. We first propose a tree-based multi-level (including symbol level, sub-expression level, and image level) data augmentation strategy, which can generate many synthetic images. Then, we present a novel encoder-decoder hybrid model via tree-based mutual learning to fully utilize the complementarity between tree decoder and string decoder. Benefitting from our data augmentation strategy, we achieve 58.47%/57.82%/62.67% and 74.45% expression recognition accuracy respectively on the CROHME14/16/19 competition datasets and the OffRaSHME20 competition dataset. Moreover, tree-based data augmentation is a key technology to our champion system for the OffRaSHME20 competition. Our tree-based mutual learning method further improves the recognition accuracy to 61.63%/59.81%/64.38% and 75.68% on these datasets. Further quantitative and qualitative analyses also demonstrate the effectiveness and robustness of our proposed methods.

Deep Learning based Handwritten Polynomial Equation Solver

Abstract: Polynomials are algebraic expressions involving a sum of powers in one or more variables multiplied by its coefficients. If x is a variable, a0x n +a1x n-1 +a2x n-2 +....+an then it's a (n) powers polynomial. Human is capable to solve this type of mathematical problems. In this work, we propose a system in which machines can achieve the cognitional skills that can understand the problem by visual context. By taking an input image of Handwritten polynomial equations and simplifies the problem by generating the answer as an output. Here machine can able to solve quadratic, cubic, quartic, quantic, sextic as well as (n) powers polynomials. This proposed work can be workable in an embedded system as well as a mobile application. In this scope for recognition purposes, we use a CNN model. Robust handwritten character recognition is a tricky job in the area of image processing. Among all the problem handwritten mathematical expression recognition is one of the complicated issue in the area of computer vision research. Segmentation and classification of specific character makes the task more difficult. In this paper a group of handwritten quadratic equation as well as a single quadratic equation are considered to recognize and make a solution for those equations. Horizontal compact projection analysis and combined connected component analysis methods are used for segmentation. For classification of specific character we apply Convolutional Neural Network. Each of the correct detection, character string operation is used for the solution of the equation. The proposed workflow system automatically simplifies the Handwritten polynomial equation and has been done a really good performance. Developing an automatic equation recognizer and solver has been a desire of the researchers who worked in the field of NLP for many years.

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

Handwritten mathematical expression recognition aims to automatically generate LaTeX sequences from given images. Currently, attention-based encoder-decoder models are widely used in this task. They typically generate target sequences in a left-to-right (L2R) manner, leaving the right-to-left (R2L) contexts unexploited. In this paper, we propose an Attention aggregation based Bi-directional Mutual learning Network (ABM) which consists of one shared encoder and two parallel inverse decoders (L2R and R2L). The two decoders are enhanced via mutual distillation, which involves one-to-one knowledge transfer at each training step, making full use of the complementary information from two inverse directions. Moreover, in order to deal with mathematical symbols in diverse scales, an Attention Aggregation Module (AAM) is proposed to effectively integrate multi-scale coverage attentions. Notably, in the inference phase, given that the model already learns knowledge from two inverse directions, we only use the L2R branch for inference, keeping the original parameter size and inference speed. Extensive experiments demonstrate that our proposed approach achieves the recognition accuracy of 56.85 % on CROHME 2014, 52.92 % on CROHME 2016, and 53.96 % on CROHME 2019 without data augmentation and model ensembling, substantially outperforming the state-of-the-art methods. The source code is available in https://github.com/XH-B/ABM.

TDv2: A Novel Tree-Structured Decoder for Offline Mathematical Expression Recognition

In recent years, tree decoders become more popular than LaTeX string decoders in the field of handwritten mathematical expression recognition (HMER) as they can capture the hierarchical tree structure of mathematical expressions. However previous tree decoders converted the tree structure labels into a fixed and ordered sequence, which could not make full use of the diversified expression of tree labels. In this study, we propose a novel tree decoder (TDv2) to fully utilize the tree structure labels. Compared with previous tree decoders, this new model does not require a fixed priority for different branches of a node during training and inference, which can effectively improve the model generalization capability. The input and output of the model make full use of the tree structure label, so that there is no need to find the parent node in the decoding process, which simplifies the decoding process and adds a prior information to help predict the node. We verified the effectiveness of each part of the model through comprehensive ablation experiments and attention visualization analysis. On the authoritative CROHME 14/16/19 datasets, our method achieves the state-of-the-art results.

CAC: A Learning Context Recognition Model Based on AI for Handwritten Mathematical Symbols in e-Learning Systems

The e-learning environment should support the handwriting of mathematical expressions and accurately recognize inputted handwritten mathematical expressions. To this end, expression-related information should be fully utilized in e-learning environments. However, pre-existing handwritten mathematical expression recognition models mainly utilize the shape of handwritten mathematical symbols, thus limiting the models from improving the recognition accuracy of a vaguely represented symbol. Therefore, in this paper, a context-aided correction (CAC) model is proposed that adjusts an output of handwritten mathematical symbol (HMS) recognition by additionally utilizing information related to the HMS in an e-learning system. The CAC model collects learning contextual data associated with the HMS and converts them into learning contextual information. Next, contextual information is recognized through artificial intelligence to adjust the recognition output of the HMS. Finally, the CAC model is trained and tested using a dataset similar to that of a real learning situation. The experiment results show that the recognition accuracy of handwritten mathematical symbols is improved when using the CAC model.

R-GRU: Regularized gated recurrent unit for handwritten mathematical expression recognition

R-GRU: Regularized gated recurrent unit for handwritten mathematical expression recognition

Syntactic data generation for handwritten mathematical expression recognition

This paper proposes tree-based decomposition and sub-expression interchange for generating new syntactically valid handwritten mathematical expressions (HMEs) from a given set of HMEs to train an HME recognition model and a mathematical language model (LM). The recognition model is dual trained using weakly supervised learning and encoder-decoder attention loss on the generated samples. Recognition experiments indicate that the proposed data generation method is superior to other such methods for offline HMEs. The HME recognition model increases the expression recognition rates by 1.47, 2.88, and 2.67 points on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014, 2016, and 2019 testing sets, respectively. The LM also increases them by 8.92, 6.88, and 2.59 points on the testing sets. Further adding extra LaTeX sequences is cost effective in strengthening the LM for the expression recognition rates being 2.54, 2.8, and 1.25 points higher than without them on the CROHME testing sets, respectively. Among academic systems, the trained HME recognition system achieves the best performance with 64.60% and 66.08% expression recognition rates on the CROHME 2014 and 2016 testing sets and a comparable expression recognition rate of 58.72% on the CROHME 2019 testing set, respectively. Comparison with top systems from companies on CROHME 2019 suggests that more real and/or generated HME patterns will improve the performance of HME recognition models as well as mathematical language models.

Offline Handwritten Mathematical Expression Recognition Based on Square Loss Sequence-to-Sequence Neural Network

Offline Handwritten Mathematical Expression Recognition Based on Square Loss Sequence-to-Sequence Neural Network

Human-in-the-Loop Based Online Handwriting Mathematical Expressions Recognition

<p indent=0mm>Online handwritten mathematical expression recognition method faces some challenges, such as the uncertainty of writing characters, the complexity of mathematical expression structure and the various written styles of expression. Existing no-human-involved recognition algorithms have low recognition accuracy, especially when accidental errors and complex structures occur in mathematical expression writing. A human-in- the-loop based online handwritten mathematical expression recognition method is proposed, which mainly introduces human participation in the structural analysis stage to define and perfect the structural strokes or strokes within the mathematical structure, by means of human modification for the ambiguous strokes in the structure and operation of structural stroke makeup. To evaluate the effectiveness of the proposed method, the method is compared with a no-human-involved recognition algorithm in rate of structural analysis and expression recognition rate. The results show that proposed method can effectively promote users to participate in the process of recognition, and that incorporating user in the recognition processes can improve the accuracy of recognition of structure (9.26%) and expression (13.99%) in handwritten mathematical expression.

HANDWRITTEN MATHEMATICAL EXPRESSION RECOGNITION AND GRADING SYSTEM

Handwritten Mathematical Expression recognition and grading system is a challenging task in the field of pattern recognition. A lot of researchers have already worked on Handwritten Mathematical Expression recognition and have used various classifiers. In past, Convolutional Neural Network, also called CNN, has been highly used for recognizing patterns. In this paper, We propose an idea to recognize HME and evaluate offline using CNN for classification. The steps included are, first the worksheet is scanned and is sent to the work-spaces detection module where it will return all the rectangular work-spaces from the given worksheet, then the workspaces are sent to the line extraction module to extract all the lines. The extracted lines are then passed to the character segmentation module where it will segment the character and then characters will be classified using deep learning model DCCNN. Finally, the evaluation module will assess the line and draw a green/red bounding box depending on whether the line is correct or not.

Stroke constrained attention network for online handwritten mathematical expression recognition

In this paper, we propose a novel stroke constrained attention network (SCAN) which treats stroke as the basic unit for encoder-decoder based online handwritten mathematical expression recognition (HMER). Unlike previous methods which use trace points or image pixels as basic units, SCAN makes full use of stroke-level information for better alignment and representation. The proposed SCAN can be adopted in both single-modal (online or offline) and multi-modal HMER. For single-modal HMER, SCAN first employs a CNN-GRU encoder to extract point-level features from input traces in online mode and employs a CNN encoder to extract pixel-level features from input images in offline mode, then use stroke constrained information to convert them into online and offline stroke-level features. Using stroke-level features can explicitly group points or pixels belonging to the same stroke, therefore reduces the difficulty of symbol segmentation and recognition via the decoder with attention mechanism. For multi-modal HMER, other than fusing multi-modal information in decoder, SCAN can also fuse multi-modal information in encoder by utilizing the stroke based alignments between online and offline modalities. The encoder fusion is a better way for combining multi-modal information as it implements the information interaction one step before the decoder fusion so that the advantages of multiple modalities can be exploited earlier and more adequately. Besides, we propose an approach combining the encoder fusion and decoder fusion, namely encoder-decoder fusion, which can further improve the performance. Evaluated on a benchmark published by CROHME competition, the proposed SCAN achieves the state-of-the-art performance. Furthermore, by conducting experiments on an additional task: online handwritten Chinese character recognition (HCCR), we demonstrate the generality of our proposed method.