Text Recognizer Research Articles

Air-Writing Recognition Enabled by a Flexible Dual-Network Hydrogel-Based Sensor and Machine Learning.

Accurate air-writing recognition is pivotal for advancing state-of-the-art text recognizers, encryption tools, and biometric technologies. However, most existing air-writing recognition systems rely on image-based sensors to track hand and finger motion trajectories. Additionally, users' writing is often guided by delimiters and imaginary axes which restrict natural writing movements. Consequently, recognition accuracy falls short of optimal levels, hindering performance and usability for practical applications. Herein, we have developed an approach utilizing a one-dimensional convolutional neural network (1D-CNN) algorithm coupled with an ionic conductive flexible strain sensor based on a sodium chloride/sodium alginate/polyacrylamide (NaCl/SA/PAM) dual-network hydrogel for intelligent and accurate air-writing recognition. Taking advantage of the excellent characteristics of the hydrogel sensor, such as high stretchability, good tensile strength, high conductivity, strong adhesion, and high strain sensitivity, alongside the enhanced analytical ability of the 1D-CNN machine learning (ML) algorithm, we achieved a recognition accuracy of ∼96.3% for in-air handwritten characters of the English alphabets. Furthermore, comparative analysis against state-of-the-art methods, such as the widely used residual neural network (ResNet) algorithm, demonstrates the competitive performance of our integrated air-writing recognition system. The developed air-writing recognition system shows significant potential in advancing innovative systems for air-writing recognition and paving the way for exciting developments in human-machine interface (HMI) applications.

SterAF: A Scene Text Recognizer with Appearance-Flow Rectification

As the demand for recognizing irregular text in natural scenes increases, people are increasingly realizing the value of such applications, such as license plate recognition systems, image search, handwriting recognition, and autonomous driving, which are profoundly changing our lives in the field of text recognition. Recent studies have shown that the recognition of curved text and perspective text has become an important challenge in the field of text recognition, and the correction of curved text is a key step to achieve accurate recognition. However, current methods use strained text image correction methods, resulting in poor recognition accuracy when recognizing curved text. Therefore, we propose an end-to-end framework called Scene Text Recognizer with Appearance-Flow rectification (SterAF), which includes a correction network and a recognition network. Specifically, the framework’s steps are as follows: first, the input text image is deformed through an appearance flow-based correction network to adaptively warp the text image, to prevent irregular and unnatural deformations of the text image. Second, a sequence-to-sequence recognition network predicts the sequence of characters in the corrected text image to accurately recognize the text in the image. Through subjective and objective experiments, our SterAF model has shown excellent performance in both qualitative and quantitative experiments.

An overview of statistical and neural-based line segmentation methods for offline handwriting recognition task

The object of the research is the line segmentation task. To recognize the handwritten text from the documents in image format offline handwriting recognition technology is used. The text recognizer module accepts input as separate lines, so one of the important preprocessing steps is the detection and splitting of all handwritten text into distinct lines. In this paper, the handwritten text line segmentation task, its requirements, problems, and challenges are examined. Two main approaches for this task that are used in modern recognition systems are reviewed. These approaches are statistical projection-based methods and neural-based methods. Multiple works and research papers for each type of approach are reviewed analyzing their strengths and weaknesses considering the described tasks, constraints, and input data peculiarities. Overall acquired results are formed in a single table for comparison. Based on the latest works that utilize deep neural networks the new possibilities of using these methods in recognition systems are described that were unavailable with traditional statistical segmentation approaches. The constructive conclusions are made based on the review, describing the main pros and cons of these two approaches for the line segmentation task. These results can be further used for the correct selection of suitable methods in handwriting recognition systems to improve their performance and quality, and for further research in this area.

A textual question answering and handwritten answer evaluation system for hindi language

Textual Question Answering targets answering questions defined in natural language. Question Answering Systems offer an automated approach to procuring answers to queries expressed in natural language. The need for Multilingual Question Answering without performing machine translation is ever existing. Besides that, automating tasks with the help of technology to assist humans, has been the main aim of research in recent years. This paper presents an automated answer evaluation system for reading comprehension-based questions in the Hindi language without requiring translation in any other language. The system accepts text, question, and handwritten answer of a student in the form of an image for answer evaluation. This is accomplished by developing a textual question-answering system for reading comprehension. It is an extractive approach that utilizes RoBERTa transformer model and fine-tunes it for Hindi question-answering. The answer to the question is extracted as a span from the provided text. Further, a handwritten text recognizer model is developed employing a Convolutional Recurrent Neural Network with Connectionist Temporal Classification module along with two layers of Bidirectional LSTM. Experimentation is performed using existing as well as self-created datasets to show the effectiveness of the proposed approach. An accuracy of 98.69% is obtained on the self-created Hindi-QA dataset and the proposed system outperformed the other existing methods. The paper also discusses potential research directions in the field.

SLOGAN: Handwriting Style Synthesis for Arbitrary-Length and Out-of-Vocabulary Text.

Large amounts of labeled data are urgently required for the training of robust text recognizers. However, collecting handwriting data of diverse styles, along with an immense lexicon, is considerably expensive. Although data synthesis is a promising way to relieve data hunger, two key issues of handwriting synthesis, namely, style representation and content embedding, remain unsolved. To this end, we propose a novel method that can synthesize parameterized and controllable handwriting S tyles for arbitrary-Length and O ut-of-vocabulary text based on a G enerative A dversarial N etwork (GAN), termed SLOGAN. Specifically, we propose a style bank to parameterize specific handwriting styles as latent vectors, which are input to a generator as style priors to achieve the corresponding handwritten styles. The training of the style bank requires only writer identification of the source images, rather than attribute annotations. Moreover, we embed the text content by providing an easily obtainable printed style image, so that the diversity of the content can be flexibly achieved by changing the input printed image. Finally, the generator is guided by dual discriminators to handle both the handwriting characteristics that appear as separated characters and in a series of cursive joins. Our method can synthesize words that are not included in the training vocabulary and with various new styles. Extensive experiments have shown that high-quality text images with great style diversity and rich vocabulary can be synthesized using our method, thereby enhancing the robustness of the recognizer.

Practical single-line diagram recognition based on digital image processing and deep vision models

The Single-Line Diagram (SLD) is a standardized graphical representation of power systems. Power engineering software typically requires users to model power systems by drawing SLDs. Although existing computer vision-based SLD recognition approaches facilitate system modeling to some extent, their performance is usually limited by the low resolution of images, the imbalance in size and number of electrical symbols, and the interference between adjacent graphical elements on practical SLDs. This paper proposes a novel SLD recognition framework to address these challenges. First, a text and symbol separation module based on digital image processing is developed to enable independent analysis of text labels and electrical symbols to avoid mutual interference. To accurately recognize electrical symbols of different sizes, the proposed framework integrates the Swin Transformer into the Faster Region-based Convolutional Neural Network (Faster R-CNN), which ultimately achieves a minimum precision of 98.9% and a minimum recall of 98.12%. In addition, the framework modifies the Text Super-Resolution Network (TSRN) to improve text recognition performance on low-resolution images. Combined with the Attentional Scene Text Recognizer with Flexible Rectification (ASTER) model, the proposed text recognition model results in an average accuracy increase of 1% on the TextZoom dataset and 97.1% accuracy in recognizing practical SLD text labels. A synthetic SLD generation method is also proposed to augment the actual SLD datasets to overcome the class imbalance of electrical symbols. The proposed framework significantly improves the performance of existing engineering drawing recognition frameworks. Relevant source code is available online at https://github.com/Lattle-y/work.git.

TextStyleBrush: Transfer of Text Aesthetics From a Single Example.

We present a novel approach for disentangling the content of a text image from all aspects of its appearance. The appearance representation we derive can then be applied to new content, for one-shot transfer of the source style to new content. We learn this disentanglement in a self-supervised manner. Our method processes entire word boxes, without requiring segmentation of text from background, per-character processing, or making assumptions on string lengths. We show results in different text domains which were previously handled by specialized methods, e.g., scene text, handwritten text. To these ends, we make a number of technical contributions: (1) We disentangle the style and content of a textual image into a non-parametric, fixed-dimensional vector. (2) We propose a novel approach inspired by StyleGAN but conditioned over the example style at different resolution and content. (3) We present novel self-supervised training criteria which preserve both source style and target content using a pre-trained font classifier and text recognizer. Finally, (4) we also introduce Imgur5K, a new challenging dataset for handwritten word images. We offer numerous qualitative photo-realistic results of our method. We further show that our method surpasses previous work in quantitative tests on scene text and handwriting datasets, as well as in a user study.

Improving Scene Text Image Super-resolution via Dual Prior Modulation Network

Scene text image super-resolution (STISR) aims to simultaneously increase the resolution and legibility of the text images, and the resulting images will significantly affect the performance of downstream tasks. Although numerous progress has been made, existing approaches raise two crucial issues: (1) They neglect the global structure of the text, which bounds the semantic determinism of the scene text. (2) The priors, e.g., text prior or stroke prior, employed in existing works, are extracted from pre-trained text recognizers. That said, such priors suffer from the domain gap including low resolution and blurriness caused by poor imaging conditions, leading to incorrect guidance. Our work addresses these gaps and proposes a plug-and-play module dubbed Dual Prior Modulation Network (DPMN), which leverages dual image-level priors to bring performance gain over existing approaches. Specifically, two types of prior-guided refinement modules, each using the text mask or graphic recognition result of the low-quality SR image from the preceding layer, are designed to improve the structural clarity and semantic accuracy of the text, respectively. The following attention mechanism hence modulates two quality-enhanced images to attain a superior SR result. Extensive experiments validate that our method improves the image quality and boosts the performance of downstream tasks over five typical approaches on the benchmark. Substantial visualizations and ablation studies demonstrate the advantages of the proposed DPMN. Code is available at: https://github.com/jdfxzzy/DPMN.

Perceiving Multiple Representations for scene text image super-resolution guided by text recognizer

Single image super-resolution (SISR) aims to recover clear high-resolution images from low-resolution images, which has made great progress with the development of deep learning these years. Scene text image super-resolution (STISR) is a subfield of SISR with the goal of increasing the resolution of a low-resolution text image and enhancing the readability of characters in the image. Despite significant improvements in recent approaches, STISR remains a challenging task due to the diversity of background, text appearances and layouts, etc. This paper presents a Perceiving Multiple Representations (PerMR) method for better super-resolution performances in scene text images. PerMR is a unified network that combines super-resolution with text recognition and exploits the recognizer’s feedback to facilitate super-resolution. Specifically, contextual information from the text decoder is extracted to provide sequence-specific guidance and enable the super-resolution model to pay more attention to the text region. Meanwhile, low-level and high-level visual features from the vision backbone of the recognition network are integrated to further improve visual quality. Additionally, we incorporate a frequency branch into the vanilla convolution unit, which efficiently enhances global and local feature representations. Experiments on the STISR benchmark dataset TextZoom validate that PerMR can not only generate more distinguishable images, but also outperforms the current state-of-the-art methods. PerMR boosts the average recognition accuracy by 5.9% using ASTER, 5.8% using MORAN and 10.6% using CRNN compared to the baseline model TSRN. PerMR outperforms the advanced method TPGSR-3 by 1.4% on ASTER, 0.1% on MORAN, 0.2% on CRNN and boosts TATT by 0.6% on ASTER and 1.1% on MORAN respectively. Furthermore, PerMR demonstrates good robustness and generalization when tackling low-quality text images in multiple scene text recognition datasets. The experiment results verify the capabilities of PerMR to boost text recognition performance.

A robust and efficient algorithm for Chinese historical document analysis and recognition.

This paper presents a novel and efficient algorithm for Chinese historical document understanding, incorporating three key components: a multi-oriented text detector, a dual-path learning-based text recognizer, and a heuristic-based reading order predictor.

Image-to-Character-to-Word Transformers for Accurate Scene Text Recognition.

Leveraging the advances of natural language processing, most recent scene text recognizers adopt an encoder-decoder architecture where text images are first converted to representative features and then a sequence of characters via 'sequential decoding'. However, scene text images suffer from rich noises of different sources such as complex background and geometric distortions which often confuse the decoder and lead to incorrect alignment of visual features at noisy decoding time steps. This paper presents I2C2W, a novel scene text recognition technique that is tolerant to geometric and photometric degradation by decomposing scene text recognition into two inter-connected tasks. The first task focuses on image-to-character (I2C) mapping which detects a set of character candidates from images based on different alignments of visual features in an non-sequential way. The second task tackles character-to-word (C2W) mapping which recognizes scene text by decoding words from the detected character candidates. The direct learning from character semantics (instead of noisy image features) corrects falsely detected character candidates effectively which improves the final text recognition accuracy greatly. Extensive experiments over nine public datasets show that the proposed I2C2W outperforms the state-of-the-art by large margins for challenging scene text datasets with various curvature and perspective distortions. It also achieves very competitive recognition performance over multiple normal scene text datasets.

Arbitrary Oriented Scene Text Recognizer (AOTR)

Arbitrary Oriented Scene Text Recognizer (AOTR)

Lexicon and attention based handwritten text recognition system

The handwritten text recognition problem is widely studied by the researchers of computer vision community due to its scope of improvement and applicability to daily lives. It is a sub-domain of pattern recognition. Due to advancement of computational power of computers since last few decades neural networks based systems heavily contributed towards providing the state-of-the-art handwritten text recognizers. In the same direction, we have taken two state-of-the art neural networks systems and merged the attention mechanism with it. The attention technique has been widely used in the domain of neural machine translations and automatic speech recognition and now is being implemented in text recognition domain. In this study, we are able to achieve 4.15% character error rate and 9.72% word error rate on IAM dataset, 7.07% character error rate and 16.14% word error rate on GW dataset after merging the attention and word beam search decoder with existing Flor et al. architecture. To analyse further, we have also used system similar to Shi et al. neural network system with greedy decoder and observed 23.27% improvement in character error rate from the base model.

SLPR: A Deep Learning Based Chinese Ship License Plate Recognition Framework

Automatic ship license plate recognition (SLPR) for ship identification is of great significance to waterway shipping management. But few attention has been paid to SLPR in the past. In this paper, a novel cascaded Chinese SLPR framework consisting of the quadrangle-based ship license plate detection (QSLPD) algorithm and the rectification-based text recognition network (RTRNet) is developed. Concretely, in QSLPD algorithm, detection is performed based on the pyramid feature fusion architecture ameliorated by the proposed variable receptive field feature enhancement strategy and three task-specific output heads. In addition, a new loss function combining the dice coefficient and cross entropy is explored in the proposed SLPR which can generate significant improvement over the baseline. In RTRNet, regions of interest (RoIs) extraction and irregular text line rectification based on the vertices information predicted by QSLPD are performed before text recognition. Data augmentation are also applied to cope with the problem of limited text recognizer training data and the extremely imbalance distribution of corpus. Extensive experiments are carried out to demonstrate the reliability of the proposed cascaded SLPR framework, that can achieve the highest F-measure of 87.78% and 76.59% with IoU and TIoU metric on the collected dataset, surpasses many existing advanced methods.

Multi-Task Learning for Scene Text Image Super-Resolution with Multiple Transformers

Scene text image super-resolution aims to improve readability by recovering text shapes from low-resolution degraded text images. Although recent developments in deep learning have greatly improved super-resolution (SR) techniques, recovering text images with irregular shapes, heavy noise, and blurriness is still challenging. This is because networks with Convolutional Neural Network (CNN)-based backbones cannot sufficiently capture the global long-range correlations of text images or detailed sequential information about the text structure. In order to address this issue, this paper proposes a Multi-task learning-based Text Super-resolution (MTSR) Network to approach this problem. MTSR is a multi-task architecture for image reconstruction and SR. It uses transformer-based modules to transfer complementary features of the reconstruction model, such as noise removal capability and text structure information, to the SR model. In addition, another transformer-based module using 2D positional encoding is used to handle irregular deformations of the text. The feature maps generated from these two transformer-based modules are fused to attempt improvement of the visual quality of images with heavy noise, blurriness, and irregular deformations. Experimental results on the TextZoom dataset and several scene text recognition benchmarks show that our MTSR significantly improves the accuracy of existing text recognizers.

Text Gestalt: Stroke-Aware Scene Text Image Super-resolution

In the last decade, the blossom of deep learning has witnessed the rapid development of scene text recognition. However, the recognition of low-resolution scene text images remains a challenge. Even though some super-resolution methods have been proposed to tackle this problem, they usually treat text images as general images while ignoring the fact that the visual quality of strokes (the atomic unit of text) plays an essential role for text recognition. According to Gestalt Psychology, humans are capable of composing parts of details into the most similar objects guided by prior knowledge. Likewise, when humans observe a low-resolution text image, they will inherently use partial stroke-level details to recover the appearance of holistic characters. Inspired by Gestalt Psychology, we put forward a Stroke-Aware Scene Text Image Super-Resolution method containing a Stroke-Focused Module (SFM) to concentrate on stroke-level internal structures of characters in text images. Specifically, we attempt to design rules for decomposing English characters and digits at stroke-level, then pre-train a text recognizer to provide stroke-level attention maps as positional clues with the purpose of controlling the consistency between the generated super-resolution image and high-resolution ground truth. The extensive experimental results validate that the proposed method can indeed generate more distinguishable images on TextZoom and manually constructed Chinese character dataset Degraded-IC13. Furthermore, since the proposed SFM is only used to provide stroke-level guidance when training, it will not bring any time overhead during the test phase. Code is available at https://github.com/FudanVI/FudanOCR/tree/main/text-gestalt.

Character-Level Street View Text Spotting Based on Deep Multisegmentation Network for Smarter Autonomous Driving

Urban scenes are full of street entities with sign boards. Therefore, in autonomous driving, street view text spotting techniques will play a significant role in the precise understanding of surrounding scenes during driving, because texts contained in the images usually provide important clues for accurate image understanding, while it is often ambiguous for existing computer vision algorithms to understand scene images without texts. In this work, we propose a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">M</b> ulti- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">S</b> egmentation network for character-level scene <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b> ext <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> etection (MSTD). The MSTD introduces a densely connected atrous spatial pyramid pooling module to enlarge the receptive field of the feature extraction layer, so as to localize long as well as large-sized text instances. Moreover, it devises a double segmentation subnetwork to utilize two independent but inherently complementary losses to co-optimize the network and increase the reliability of the confidence scores in predicting the text/nontext areas. With the character instances detected by the MSTD, one can easily perform scene text spotting with classic object recognition networks such as ResNet and DenseNet. We carried out extensive experiments on nine scene text datasets to demonstrate the outstanding performance of the MSTD on character-level and line-level text instance localization and scene text recognition, where the MSTD significantly outperforms the state-of-the-art scene text detection methods and the sequence-to-sequence-learning-based scene text recognizers.

ResNet CNN with LSTM Based Tamil Text Detection from Video Frames

Text content in videos includes applications such as library video retrievals, live-streaming advertisements, opinion mining, and video synthesis. The key components of such systems include video text detection and acknowledgments. This paper provides a framework to detect and accept text video frames, aiming specifically at the cursive script of Tamil text. The model consists of a text detector, script identifier, and text recognizer. The identification in video frames of textual regions is performed using deep neural networks as object detectors. Textual script content is associated with convolutional neural networks (CNNs) and recognized by combining ResNet CNNs with long short-term memory (LSTM) networks. A residual mapping underlies the network. In ResNet, skipping condenses the network into few layers to help it learn more quickly. Bidirectional LSTM enables networks to always have information about the sequence, backward as well as forward. For fast pattern learning processes, this combination uses the residual learning process. In comparison to the existing approaches, the proposed approach’s results show improved accuracy, precision and F-measures. The combination of ResNet CNNs and bidirectional LSTMs has high recognition rates for detecting video texts in Tamil cursive script.

PETR: Rethinking the Capability of Transformer-Based Language Model in Scene Text Recognition.

The exploration of linguistic information promotes the development of scene text recognition task. Benefiting from the significance in parallel reasoning and global relationship capture, transformer-based language model (TLM) has achieved dominant performance recently. As a decoupled structure from the recognition process, we argue that TLM's capability is limited by the input low-quality visual prediction. To be specific: 1) The visual prediction with low character-wise accuracy increases the correction burden of TLM. 2) The inconsistent word length between visual prediction and original image provides a wrong language modeling guidance in TLM. In this paper, we propose a Progressive scEne Text Recognizer (PETR) to improve the capability of transformer-based language model by handling above two problems. Firstly, a Destruction Learning Module (DLM) is proposed to consider the linguistic information in the visual context. DLM introduces the recognition of destructed images with disordered patches in the training stage. Through guiding the vision model to restore patch orders and make word-level prediction on the destructed images, visual prediction with high character-wise accuracy is obtained by exploring inner relationship between the local visual patches. Secondly, a new Language Rectification Module (LRM) is proposed to optimize the word length for language guidance rectification. Through progressively implementing LRM in different language modeling steps, a novel progressive rectification network is constructed to handle some extremely challenging cases (e.g. distortion, occlusion, etc.). By utilizing DLM and LRM, PETR enhances the capability of transformer-based language model from a more general aspect, that is, focusing on the reduction of correction burden and rectification of language modeling guidance. Compared with parallel transformer-based methods, PETR obtains 1.0% and 0.8% improvement on regular and irregular datasets respectively while introducing only 1.7M additional parameters. The extensive experiments on both English and Chinese benchmarks demonstrate that PETR achieves the state-of-the-art results.

TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance

Scene text recognition (STR) is an important bridge between images and text, attracting abundant research attention. While convolutional neural networks (CNNS) have achieved remarkable progress in this task, most of the existing works need an extra module (context modeling module) to help CNN to capture global dependencies to solve the inductive bias and strengthen the relationship between text features. Recently, the transformer has been proposed as a promising network for global context modeling by self-attention mechanism, but one of the main short-comings, when applied to recognition, is the efficiency. We propose a 1-D split to address the challenges of complexity and replace the CNN with the transformer encoder to reduce the need for a context modeling module. Furthermore, recent methods use a frozen initial embedding to guide the decoder to decode the features to text, leading to a loss of accuracy. We propose to use a learnable initial embedding learned from the transformer encoder to make it adaptive to different input images. Above all, we introduce a novel architecture for text recognition, named TRansformer-based text recognizer with Initial embedding Guidance (TRIG), composed of three stages (transformation, feature extraction, and prediction). Extensive experiments show that our approach can achieve state-of-the-art on text recognition benchmarks.