Handwritten Formulas Research Articles

Multimodal Dependence Attention and Large-Scale Data Based Offline Handwritten Formula Recognition

Multimodal Dependence Attention and Large-Scale Data Based Offline Handwritten Formula Recognition

Weakly Supervised Neural Symbolic Learning for Cognitive Tasks

Despite the recent success of end-to-end deep neural networks, there are growing concerns about their lack of logical reasoning abilities, especially on cognitive tasks with perception and reasoning processes. A solution is the neural symbolic learning (NeSyL) method that can effectively utilize pre-defined logic rules to constrain the neural architecture making it perform better on cognitive tasks. However, it is challenging to apply NeSyL to these cognitive tasks because of the lack of supervision, the non-differentiable manner of the symbolic system, and the difficulty to probabilistically constrain the neural network. In this paper, we propose WS-NeSyL, a weakly supervised neural symbolic learning model for cognitive tasks with logical reasoning. First, WS-NeSyL employs a novel back search algorithm to sample the possible reasoning process through logic rules. This sampled process can supervise the neural network as the pseudo label. Based on this algorithm, we can backpropagate gradients to the neural network of WS-NeSyL in a weakly supervised manner. Second, we introduce a probabilistic logic regularization into WS-NeSyL to help the neural network learn probabilistic logic. To evaluate WS-NeSyL, we have conducted experiments on three cognitive datasets, including temporal reasoning, handwritten formula recognition, and relational reasoning datasets. Experimental results show that WS-NeSyL not only outperforms the end-to-end neural model but also beats the state-of-the-art neural symbolic learning models.

Optical Recognition of Handwritten Logic Formulas Using Neural Networks

In this paper, we present a handwritten character recognition (HCR) system that aims to recognize first-order logic handwritten formulas and create editable text files of the recognized formulas. Dense feedforward neural networks (NNs) are utilized, and their performance is examined under various training conditions and methods. More specifically, after three training algorithms (backpropagation, resilient propagation and stochastic gradient descent) had been tested, we created and trained an NN with the stochastic gradient descent algorithm, optimized by the Adam update rule, which was proved to be the best, using a trainset of 16,750 handwritten image samples of 28 × 28 each and a testset of 7947 samples. The final accuracy achieved is 90.13%. The general methodology followed consists of two stages: the image processing and the NN design and training. Finally, an application has been created that implements the methodology and automatically recognizes handwritten logic formulas. An interesting feature of the application is that it allows for creating new, user-oriented training sets and parameter settings, and thus new NN models.

Graph-to-Graph: Towards Accurate and Interpretable Online Handwritten Mathematical Expression Recognition

Recent handwritten mathematical expression recognition (HMER) approaches treat the problem as an image-to-markup generation task where the handwritten formula is translated into a sequence (e.g. LaTeX). The encoder-decoder framework is widely used to solve this image-to-sequence problem. However, (i) for structured mathematical formula, the hierarchical structure neither in the formula nor in the markup has been explored adequately. In addition, (ii) existing image-to-markup methods could not explicitly segment mathematical symbols in the formula corresponding to each target markup token. In this paper, we address the above issues by formulating the HMER as a graph-to-graph (G2G) learning problem. Graph is more flexible and general for structure representation and learning compared with image or sequence. At the core of our method lies the embedding of input formula and output markup into graphs on primitives, with Graph Neural Networks (GNN) to explore the structural information, and a novel sub-graph attention mechanism to match primitives in the input and output graphs. We conduct extensive experiments on CROHME datasets to demonstrate the benefits of the proposed G2G model. Our method yields significant improvements over previous SOTA image-to-markup systems. Moreover, it explicitly resolves the symbol segmentation problem while still being trained end-to-end, making the whole system much more accurate and interpretable.

Multi-Feature Learning by Joint Training for Handwritten Formula Symbol Recognition

Given the similarity of handwritten formula symbols and various handwriting styles, this paper proposes a squeeze-extracted multi-feature convolution neural network (SE-MCNN) to improve the recognition rate of handwritten formula symbols. The system proposed in this paper integrates the eight-directional feature of the original sequence in the convolutional layer, which significantly compensates for the lost dynamic trajectory information in the handwritten formula symbol. Meanwhile, the joint loss is constructed to improve the discriminability of features in the way of supervised learning, which enlarges the inter-class difference and decreases inner-class similarity. The standard mathematical formula symbol library provided by the Competition Organization on Recognition of Online Handwritten Mathematical Expression (CROHME) is used to verify the effectiveness of the proposed algorithm. Experiments show that the proposed SE-MCNN approach outperforms the state-of-the-art methods even at the condition of without using the data augmentation.

基于多特征卷积神经网络的手写公式符号识别

基于多特征卷积神经网络的手写公式符号识别

A New Algorithm of Online Handwritten Mathematical Formula Structure Recognition

As a natural, fast mathematical formula input, online handwritten mathematical formula input have great prospects. The paper described the algorithm of recognizing the common online handwritten mathematical formula structure. Firstly, the forms were defined, the convention of formulas were expanded. A method of calculating weight for the minimum spanning tree(MST) was proposed according to the characteristics of hand-written formula. On the basis of them, MST algorithm was applied to formula structural analysis. Then two new algorithms were proposed to recognize the mix form and table structure respectively. Compared with other classical algorithms, the algorithm proposed in this paper can recognized the expansive structure, while recognition rate was improved.

Recognition algorithm of online handwritten mathematical formula structure

As a natural, fast mathematical formula input method, the online handwritten mathematical formula input method has many application prospects. The author described the algorithm of recognizing the common online handwritten mathematical formula structure, defined the fractal, the dominant relationships of the mathematical formulas structure and expanded the hard rules. Then a new method of calculating weight for the Minimum Spanning Tree ( MST) was proposed according to the characteristics of handwritten formulas. On the basis of them, MST algorithm was applied to formula structure analysis. Compared with other classical algorithms, the proposed algorithm can recognize more structures and improve the recognition rate.

What is the purpose or goal of a presidential address?

What is the purpose or goal of a presidential address?