Scene Text Recognition Method Research Articles

Scene Chinese Recognition with Local and Global Attention

Recently, scene Chinese recognition has attracted increasing attention. While mainstream scene text recognition methods exhibit outstanding performance in English recognition, they are considerably limited in Chinese recognition, due to inter-class similarity, intra-class variability, and complex combination of components in scene Chinese text. In this paper, we design Adaptive Position Encoding(APE) to enhance the model’s ability to perceive spatial information. Based on APE, we have innovatively designed Local Attention Module (LAM) and Global Attention Module (GAM). Specifically, LAM captures local features to identify common characteristics among characters of the same category, addressing the issue of intra-class variability. Meanwhile, LAM captures global features to identify the subordination relationships of Chinese character components. By integrating LAM and GAM, combining both local and global features, it is possible to find differences in the details among features that are fundamentally similar, thus solving the problem of inter-class similarity. Further, we contrive the transformer encoder–decoder structure to identify the vast variety of Chinese characters. Based on the Local/Global Attention Module and transformer encoder–decoder framework, we devise the novel sequence-to-sequence Local and Global Attention Network(LGANet), where both the backbone and the encoder/decoder are composed of attention mechanisms. Subsequent experiments on the Chinese scene dataset show that the recognition accuracy of our proposed LGANet is 77.3% and the normalized editing distance is 88.6%, both of which achieve the SOTA results in Fig. 1.

Scene text recognition with context-aware autonomous bidirectional iterative models

This research focuses on Scene Text Recognition (STR), a crucial component in various applications of artificial intelligence such as image retrieval, office automation, and intelligent traffic systems. Recent studies have shown that semantic-aware approaches significantly improve the performance of STR tasks, with context-aware STR methods becoming mainstream. Among these, the fusion of visual and language models has shown remarkable effectiveness. We propose a novel method (PABINet) that incorporates three key components: a Visual-Language Decoder, a Language Model, and a Fusion Model. First, during training, the Visual-Language Decoder masks the original labels in the Transformer decoder using permutation masks, with each mask being unique. This enhances word memorization and learning through contextual semantic information, resulting in robust semantic knowledge. During the inference stage, the Visual-Language Decoder employs autonomous Autoregressive model (AR) inference to generate results. Subsequently, the Language Model scrutinizes and corrects the output of the Visual-Language Encoder using a cloze mask approach, achieving context-aware, autonomous, bidirectional inference. Finally, the Fusion Model concatenates and refines the outputs of both models through iterative layers.Experimental results demonstrate that our PABINet performs exceptionally well when handling various quality images. When trained with synthetic data, PABINet achieves a new STR benchmark (average accuracy of 92.41%), and when trained with real data, it establishes new state-of-the-art results (average accuracy of 96.28%).

NDOrder: Exploring a novel decoding order for scene text recognition

Text recognition in scene images is still considered as a challenging task for the computer vision and pattern recognition community. For text images affected by multiple adverse factors, such as occlusion (due to obstacles) and poor quality (due to blur and low resolution), the performance of the state-of-the-art scene text recognition methods degrades. The key reason is that the existing encoder–decoder framework follows fixed left-to-right decoding order, which lacks sufficient contextual information. In this paper, we present a novel decoding order where good-quality characters can first be decoded followed by low-quality characters, which preserves the contextual information irrespective of the aforementioned difficult scenarios. Our method, named NDOrder, extracts visual features with a ViT encoder and then decodes with the Random Order Generation module (ROG) for learning to decode with random decoding orders and the Vision-Content-Position module (VCP) for exploiting the connections among visual information, content and position. In addition, a new dataset named OLQT (Occluded and Low-Quality Text) is created by manually collecting text images that suffer from occlusion or low-quality from several standard text recognition datasets. The dataset is now available at https://github.com/djzhong1/OLQT. Experiments on OLQT and public scene text recognition benchmarks show that the proposed method achieves state-of-the-art performance.

Sequential visual and semantic consistency for semi-supervised text recognition

Scene text recognition (STR) is a challenging task that requires large-scale annotated data for training. However, collecting and labeling real text images is expensive and time-consuming, which limits the availability of real data. Therefore, most existing STR methods resort to synthetic data, which may introduce domain discrepancy and degrade the performance of STR models. To alleviate this problem, recent semi-supervised STR methods exploit unlabeled real data by enforcing character-level consistency regularization between weakly and strongly augmented views of the same image. However, these methods neglect word-level consistency, which is crucial for sequence recognition tasks. This paper proposes a novel semi-supervised learning method for STR that incorporates word-level consistency regularization from both visual and semantic aspects. Specifically, we devise a shortest path alignment module to align the sequential visual features of different views and minimize their distance. Moreover, we adopt a reinforcement learning framework to optimize the semantic similarity of the predicted strings in the embedding space. We conduct extensive experiments on several standard and challenging STR benchmarks and demonstrate the superiority of our proposed method over existing semi-supervised STR methods.

SAME-net： scene text recognition method based on soft attention mask embedding

SAME-net： scene text recognition method based on soft attention mask embedding

Dynamic receptive field adaptation for scene text recognition

Scene text recognition methods of the Encoder–Decoder framework, generally assume that the proportion of characters in the same text instance are basically the same. However, this assumption does not always hold in the context of complex scene images. For adaptively revising the receptive field according to the different font in the scene text image, we propose a Dynamic Receptive Field Adaption Framework which consists of Memory Attention (MA) module and Dynamic Feature Adaptive (DFA) module. MA percepts the historical location information to adapt to the change of character position in the decoder. DFA selects the most distinguishing features from feature maps of different levels dynamically. Additionally, MA and DFA can be easily extended to the existing attention-based and transformer-based text recognition methods to improve their performance. With extension experiments on public benchmark datasets, including IIIT-5K, SVT, SVTP, CUTE80, RECTS, LSVT, and RCTW, our method has shown effectiveness and robustness.

HAFE: A Hierarchical Awareness and Feature Enhancement Network for Scene Text Recognition

Scene text recognition (STR) is to obtain text information from natural text images. Compared to the optical character recognition, STR has attracted much attentions due to the variability of scene images. However, most STR methods tend to loss some spatial information due to only simply adding Resnet-45 to the front end of the encoder. To solve this problem, we propose a Hierarchical Awareness and Feature Enhancement (HAFE) network. Our proposed Hierarchical Awareness (HA) module can effectively extract the depth information of scene text images from multiple dimensions and then improve the anti-interference ability of the network. Besides, the proposed Feature Enhancement (FE) module can enhance the stability of the model through strengthening the global features, which is helpful to improve the accuracy of text recognition. On the whole, our HAFE network tends to strengthen the information in encoder, better learn the feature maps from different levels, and then improve the recognition accuracy of non-horizontal texts. Extensive comparative experiments on seven benchmark datasets show that the proposed method has the most advanced performance and outperforms most state-of-the-art ones. Our code will be released at website: https://github.com/HAFE in the future.

Reading Scene Text with Aggregated Temporal Convolutional Encoder

Reading scene text in the natural image is of fundamental importance in many real-world problems. Text recognition has a profound effect on information processing by enabling automated extraction and interpretation. Recent scene text recognition methods employ the encoder-decoder framework, which constructs the encoder by obtaining the visual representations based on the last layer of the backbone network and then feeding them into a sequence model. In this article, we propose a novel encoder structure that performs the feature extractor and the sequence modeling within a unified framework. The introduced Aggregated Temporal Convolutional Encoder (ATCE) first incorporates the temporal convolutional layers to consider the long-term temporal relationship in the encoder stage. The aggregation of these temporal convolution modules is designed to utilize visual features from different levels, by augmenting the standard architecture with deeper aggregation to better fuse information across modules. We also study the impact of different attention modules in convolutional blocks for learning accurate text representations. We conduct comparisons on several scene text recognition benchmarks for both Chinese and English; the experiments demonstrate the complementary ability with different decoder variants and the effectiveness of our proposed approach.

Scene Uyghur Recognition Based on Visual Prediction Enhancement.

Aiming at the problems of Uyghur oblique deformation, character adhesion and character similarity in scene images, this paper proposes a scene Uyghur recognition model with enhanced visual prediction. First, the content-aware correction network TPS++ is used to perform feature-level correction for skewed text. Then, ABINet is used as the basic recognition network, and the U-Net structure in the vision model is improved to aggregate horizontal features, suppress multiple activation phenomena, better describe the spatial characteristics of character positions, and alleviate the problem of character adhesion. Finally, a visual masking semantic awareness (VMSA) module is added to guide the vision model to consider the language information in the visual space by masking the corresponding visual features on the attention map to obtain more accurate visual prediction. This module can not only alleviate the correction load of the language model, but also distinguish similar characters using the language information. The effectiveness of the improved method is verified by ablation experiments, and the model is compared with common scene text recognition methods and scene Uyghur recognition methods on the self-built scene Uyghur dataset.

Rethinking text rectification for scene text recognition

Existing scene text recognition methods have incorporated text rectification to lessen text irregularity in images for accurate recognition. Previous text rectification methods aim to convert an irregular text image into a regular form, making it easier to be recognized. In this study, we explore text rectification for text recognition and discover the issues: performance degradation of the recognition network and the unreliable situation of text rectification, which are ignored by all previous works. Therefore, we rethink what is causing two issues, and propose a rectification-based text recognition network to mitigate the above issues. The proposed network consists of text rectification and text recognition, and designs a multi-level feature aggregation module to enhance feature learning for character representation. Concretely, we devise a mixed batch training strategy to address the performance degradation of the recognition network, and design a confidence decoding scheme to avoid the unreliable situation of text rectification. Extensive ablation studies verified the positive role of the feature aggregation module in feature learning and the effectiveness of the proposed training strategy and decoding scheme in addressing the issues. Experimental results outperform the state-of-the-art results on public benchmarks.

Dual Relation Network for Scene Text Recognition

Local visual and long-range contextual features yield two complementary cues for human reading text in natural scene. Existing scene text recognition methods mainly extract local features in low level features and then model long-range dependencies in high level, this sequential pipeline may be sub-optimal to construct complete and effective representation. Except for high level features, long-range contextual relation is of importance in low level features as well since it can help separate different characters based on the intervals between characters and thus enhance the features in character area. To address this issue, we develop a dual relation module to extract complementary features in a parallel manner for scene text recognition. Dual relation module consists of a local visual branch and a long-range contextual branch from low level to high level. The local visual branch employs a topological-aware operation to model intra-character characteristic and extract discriminative features of different characters. Meanwhile, the long-range contextual branch utilizes a simple but effective strategy to incorporate inter-character relations into feature maps. Our dual relation module is a plug-and-play block which can be easily incorporated into modern deep architectures. Experimental results demonstrate that our methods achieved top performance on several standard benchmarks. Code and models will become publicly available in the future.

RMFPN: End-to-End Scene Text Recognition Using Multi-Feature Pyramid Network

Scene text recognition (STR) plays an important role in various computer vision activities. STR has been a desirable research topic in the computer community, and deep learning-based STR methods have gained tremendous outcomes over the past few years. Earlier state-of-the-art scene text recognition approaches even deliver a notable quantity of inaccurate yields when applied to images caught in real-world environments. Because these images lose precise text content information, previous methods generate less robust features and semantic information about text content. To address this issue, we propose a new approach called Residual Multi-Feature Pyramid Network(RMFPN), which integrates ResNet and Multi-Feature Pyramid Networks to grab multi-level relations, enrich the functionality, and generalization of the feature extractor. We build RMFPN with two convolutional pyramids as a feature extractor, which improves the robustness of features and semantic information to endure scene text recognition of various scales. Comprehensive experiments on diverse datasets demonstrate that our proposed method can acquire significant performance accuracy. The proposed RMFPN acquires a 0.61%, 1.2%, 1%, and 0.2% improvement on SVT, IC15, SVTP, and CUTE datasets.

A Deep Learning-Based Text Detection and Recognition Approach for Natural Scenes

In this paper, we design a natural scene text detection and recognition model based on deep learning by model construction and in-depth study of wild scene text detection and recognition. This paper proposes a scene text recognition method based on connection time classification and attention mechanism for the situation where natural scene text is challenging to recognize due to the high complexity of text and background. The method converts the text recognition problem in natural scenes into a sequence recognition problem, avoiding the drawback of overall recognition performance degradation due to the difficulty of character segmentation. At the same time, the attention mechanism introduced can reduce the network complexity and improve the recognition accuracy. The performance of the improved PSE-based text detection algorithm in this paper is tested on the curved text datasets SCUT-ctw1500 and ICDAR2017 in natural scenes for comparison. The results show that the proposed algorithm achieves 88.5%, 77%, and 81.3% in the three indexes of accuracy, recall, and F1 value, respectively, without adding the pre-training module. The algorithm can detect text in any direction well without adding the pre-training module; the improved text recognition algorithm based on CRNN in this paper is tested on the natural scene dataset ICDAR2017, and the results show that the accuracy rate reaches 94.5% under the condition of no constraint, which is a good performance.

Spatial attention contrastive network for scene text recognition

At present, most scene text recognition methods achieve good performance by training models on many synthetic data. However, many data lead to huge storage space and large amount of calculation. And there is a gap between synthetic and real data. To solve these problems, we use a few real data to train a novel proposed model named spatial attention contrastive network (SAC-Net). The SAC-Net consists of a background suppression network (BSNet), a feature encoder, an attention decoder (ADEer), and a feature contrastive network (FCNet). The BSNet based on U-Net is used to reduce the interference of background. For relatively low prediction accuracy brought by connectionist temporal classification, we design an ADEer to improve performance by using convolutional attention mechanism. Based on data augmentation, we design a FCNet which belongs to contrastive learning. Finally, our SAC-Net is almost equivalent to the state-of-the-art model trained on a few real data for word accuracy on six benchmark test datasets.

Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of the 1D character sequence predicted by a visual recognition (VR) model, which ignore the 2D spatial context of visual semantics within and between character instances, making them not generalize well to arbitrary shape scene text. To address this issue, we make the first attempt to perform textual reasoning based on visual semantics in this paper. Technically, given the character segmentation maps predicted by a VR model, we construct a subgraph for each instance, where nodes represent the pixels in it and edges are added between nodes based on their spatial similarity. Then, these subgraphs are sequentially connected by their root nodes and merged into a complete graph. Based on this graph, we devise a graph convolutional network for textual reasoning (GTR) by supervising it with a cross-entropy loss. GTR can be easily plugged in representative STR models to improve their performance owing to better textual reasoning. Specifically, we construct our model, namely S-GTR, by paralleling GTR to the language model in a segmentation-based STR baseline, which can effectively exploit the visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art on six challenging STR benchmarks and generalizes well to multi-linguistic datasets. Code is available at https://github.com/adeline-cs/GTR.

Deep-learning generation of POI data with scene images

Point of interest (POI) is essential to urban scene understanding and location-based services. However, most of the POI data sets are collected manually on the spot, which is time-consuming and laborious. In this study, we propose a deep learning-based three-stage framework to automatically generate POI data sets from scene images by integrating instance segmentation, scene text recognition (STR), and multimodal technology. Firstly, we utilize an instance segmentation model to extract the region of interest (ROI) that contains POI text information from the scene images. Secondly, a STR method is used to locate and identify the text lines from the ROI. Thirdly, we develop a novel visual-linguistic multi-task classification model (VLMC) to classify ROIs and text lines through fusing text and image information. It is the first deep learning-based framework that allows generating POI information with different attributes (such as title, address, and tag) from the text lines of scene images and updating with high-performance models in the three-stage technique. In the experiments, we employ multiple STR data sets and annotated street view images for model training. The result shows that the deep learning-based framework can generate POI records from scene images with high accuracy (F1-score = 52.62%). Moreover, we find that the multi-modal VLMC model integrating the linguistic and visual embeddings has a higher accuracy in POI-generation than single-modal methods. We further use a trained framework to generate POI from Baidu Street View (BSV) images and Tencent Street View (TSV) images in Shenzhen, China, and ultimately obtain a long-term POI data set during 2013 – 2020 with 2,699,895 street view images. Of 815,616 records in the generated POI data set in 2020, 70.94% are covered by the existing Baidu POI data set of Shenzhen in 2013. This confirms the validity of the newly generated POI data set. These results demonstrate that the proposed deep-learning POI-generation framework and dataset can provide new insights for geographic data updating and urban scene understanding for fast growing cities. To facilitate future research, an implementation is made available at https://github.com/KampauCheung/scene-image-poi-generation.

Weather Prediction and Smart Notification System

In the recent period, many real-world applications and institutions generates a huge amount of data which is unstructured i.e., in the form of images containing data, receipts, invoices, forms, statements, contracts etc. This rich and detailed information presented in the text is of great significance in computer vision-based applications (driverless cars, assisting blind and visually impaired people, detecting labels and packages, automatic number plate recognition etc.). Recently, there has been a hike in the efforts, researches and progresses being done in this domain for its significance in data analysis and computer vision. Here has been a diversity of challenges in unstructured data like image sensor noise, different viewing angles, blur, lighting conditions, resolution, and non-planar object. Our objective for taking up this topic for research are i) to detect and recognize the text from the data ii) to handle diversity and variability of text in natural scene iii) to explore various datasets iv) to deal with various issues occurring in scene text detection. To tackle this problem, we propose a robust scene text detection and recognition method with adaptive text region representation using deep learning model open CV with EAST algorithm as detection pipeline and tesseract. The recurrent neural network-based adaptive text region representation is proposed for text region refinement, where a pair of boundary points are predicted each time step until no new points are found. In this way, text regions in an image are detected and represented with the adaptive number of boundary points.

2D Positional Embedding-based Transformer for Scene Text Recognition

Recent state-of-the-art scene text recognition methods are primarily based on Recurrent Neural Networks (RNNs), however, these methods require one-dimensional (1D) features and are not designed for recognizing irregular-text instances due to the loss of spatial information present in the original two-dimensional (2D) images. In this paper, we leverage a Transformer-based architecture for recognizing both regular and irregular text-in-the-wild images. The proposed method takes advantage of using a 2D positional encoder with the Transformer architecture to better preserve the spatial information of 2D image features than previous methods. The experiments on popular benchmarks, including the challenging COCO-Text dataset, demonstrate that the proposed scene text recognition method outperformed the state-of-the-art in most cases, especially on irregular-text recognition.

Improvement of the end-to-end scene text recognition method for “text-to-speech” conversion

Methods for text detection and recognition in images of natural scenes have become an active research topic in computer vision and have obtained encouraging achievements over several benchmarks. In this paper, we introduce a robust yet simple pipeline that produces accurate and fast text detection and recognition for the Uzbek language in natural scene images using a fully convolutional network and the Tesseract OCR engine. First, the text detection step quickly predicts text in random orientations in full-color images with a single fully convolutional neural network, discarding redundant intermediate stages. Then, the text recognition step recognizes the Uzbek language, including both the Latin and Cyrillic alphabets, using a trained Tesseract OCR engine. Finally, the recognized text can be pronounced using the Uzbek language text-to-speech synthesizer. The proposed method was tested on the ICDAR 2013, ICDAR 2015 and MSRA-TD500 datasets, and it showed an advantage in efficiently detecting and recognizing text from natural scene images for assisting the visually impaired.

Multi-branch guided attention network for irregular text recognition

Reading irregular text of arbitrary shape or low quality in natural scene images is a challenging task. Existing irregular scene text recognition methods mainly focus on irregular text with arbitrary shape, but rarely focus on irregular text of low quality. In this work, we propose a simple but effective method for recognizing irregular texts with arbitrary shape and low quality simultaneously. The proposed Multi-Branch guided Attention Network (MBAN) makes mutual guidance among multi-branch data in training, so as to learn invariant semantic representation between regular text images and the corresponding irregular images. Compared with the standard attention framework for text recognition, MBAN can significantly improve the performance of irregular text recognition while preserving similar performance for regular text recognition. In addition, regarding the attention drift problem encountered in standard attention network, MBAN can significantly improve the accuracy of alignment factors at each time step. We verify the effectiveness of MBAN in irregular text recognition and attention drift problem through extensive experiments. The performance of MBAN is shown to be comparable on regular datasets and superior on some irregular datasets with state-of-the-art methods.