Natural Scene Text Research Articles

Soft set-based MSER end-to-end system for occluded scene text detection, recognition and prediction

The presence of unpredictable occlusions on natural scene text is a significant challenge, exacerbating the difficulties already posed on text detection and recognition by the variability of such images. Addressing the need for a robust, consistently performing approach that can effectively address the above challenges, this paper presents a new Soft Set-based end-to-end system for text detection, recognition and prediction in occluded natural scene images. This is the first approach to integrate text detection, recognition and prediction, unlike existing systems developed for end-to-end text spotting (text detection and recognition) only. For candidate text components detection, the proposed combination of Soft Sets with Maximally Stable Extremal Regions (SS-MSER) improves text detection and spotting in natural scene images, irrespectively of the presence of arbitrarily orientated and shaped text, complex backgrounds and occlusion. Furthermore, a Graph Recurrent Neural Network is proposed for grouping candidate text components into text lines and for fitting accurate bounding boxes to each word. Finally, a Convolutional Recurrent Neural Network (CRNN) is proposed for the recognition of text and for predicting missing characters due to occlusion. Experimental results on a new occluded scene text dataset (OSTD) and on the most relevant benchmark natural scene text datasets demonstrate that the proposed system outperforms the state-of-the-art in text detection, recognition and prediction. The code and dataset are available at https://github.com/alloydas/Softset-MSER-Based-Occluded-Scene-Text-Spotting/blob/master/Soft_set_MSER.ipynb

A Multi-Scale Natural Scene Text Detection Method Based on Attention Feature Extraction and Cascade Feature Fusion.

Scene text detection is an important research field in computer vision, playing a crucial role in various application scenarios. However, existing scene text detection methods often fail to achieve satisfactory results when faced with text instances of different sizes, shapes, and complex backgrounds. To address the challenge of detecting diverse texts in natural scenes, this paper proposes a multi-scale natural scene text detection method based on attention feature extraction and cascaded feature fusion. This method combines global and local attention through an improved attention feature fusion module (DSAF) to capture text features of different scales, enhancing the network's perception of text regions and improving its feature extraction capabilities. Simultaneously, an improved cascaded feature fusion module (PFFM) is used to fully integrate the extracted feature maps, expanding the receptive field of features and enriching the expressive ability of the feature maps. Finally, to address the cascaded feature maps, a lightweight subspace attention module (SAM) is introduced to partition the concatenated feature maps into several sub-space feature maps, facilitating spatial information interaction among features of different scales. In this paper, comparative experiments are conducted on the ICDAR2015, Total-Text, and MSRA-TD500 datasets, and comparisons are made with some existing scene text detection methods. The results show that the proposed method achieves good performance in terms of accuracy, recall, and F-score, thus verifying its effectiveness and practicality.

SterAF: A Scene Text Recognizer with Appearance-Flow Rectification

As the demand for recognizing irregular text in natural scenes increases, people are increasingly realizing the value of such applications, such as license plate recognition systems, image search, handwriting recognition, and autonomous driving, which are profoundly changing our lives in the field of text recognition. Recent studies have shown that the recognition of curved text and perspective text has become an important challenge in the field of text recognition, and the correction of curved text is a key step to achieve accurate recognition. However, current methods use strained text image correction methods, resulting in poor recognition accuracy when recognizing curved text. Therefore, we propose an end-to-end framework called Scene Text Recognizer with Appearance-Flow rectification (SterAF), which includes a correction network and a recognition network. Specifically, the framework’s steps are as follows: first, the input text image is deformed through an appearance flow-based correction network to adaptively warp the text image, to prevent irregular and unnatural deformations of the text image. Second, a sequence-to-sequence recognition network predicts the sequence of characters in the corrected text image to accurately recognize the text in the image. Through subjective and objective experiments, our SterAF model has shown excellent performance in both qualitative and quantitative experiments.

TiTDet: A tiny text detector with scale-sensitive loss and effective fusion factor

The text in natural scenes is often smaller compared to artificially designed text. Due to the small proportion of pixels, low resolution, less semantic information, and susceptibility to complex scenes, tiny text detection often results in many missed detections. To address this issue, this paper draws inspiration from small object detection methods and proposes TiTDet, a detection algorithm more suitable for tiny text. Due to the small proportion of pixels, low resolution, less semantic information, and susceptibility to complex scenes, tiny text detection often results in many missed detections. To address this issue, this paper draws inspiration from small object detection methods and proposes TiTDet, a detection algorithm more suitable for tiny text. Firstly, this paper incorporates a context extraction module and an attention-guided module. These modules guide contextual information learning through a self attention mechanism, while eliminating the possible negative impact caused by redundant information. Regarding multi-scale feature fusion, this paper proposes a fine-grained effective fusion factor, making the fusion process emphasize small object learning more and highlight the feature expression of tiny texts. In terms of post-processing, this paper proposes a differentiable binarization module, incorporating the binarization process into model training. Leveraging the implicit information in the data to drive model improvement can enhance the post-processing effect. Lastly, this paper proposes a scale-sensitive loss, which can handle tiny texts more fairly, fully considering the positional relationship between the predicted and real regions, and better guiding the model training. This paper proves that TiTDet exhibits high sensitivity and accuracy in detecting tiny texts, achieving an 86.0% F1-score on ICDAR2015. The paper also compares the superiority of the method on CTW1500 and Total-Text.

Text Localization and Enhancement of Mobile Camera based Complex Natural Bilingual Text Scene Images

In this paper, we proposed a text localization and enhancement of the Mobile camera based complex natural scene images using fixed coordinates for removal of unwanted non text regions and making it localized. The morphological erosion technique is applied for the smoothness of foreground text regions. Here, the experimental working nature of the proposed method is first localized color input image is converted into a gray level, then a median filter is applied for the removal of noises. After that, the input image is further converted into binary image. In addition to this, morphological dilation and erosion operations also applied for connecting the broken pixels. Further, the connected component segmentation technique is applied to extract the foreground text regions from the binary input image. The real time input samples are collected by using mobile camera from the various categories of natural scenes of monolingual/bilingual text such as Stone, Wall, Poster and Iron-based. A total of 140 real time samples of natural scene text images are considered for the experimental setup. The experimental results shown that 132 real time sample images are localized and then obtained the accuracy of 94.24%. Further, bilingual text regions are enhanced using hybrid approach and obtain the segmentation accuracy of 83.17%. The novelty of the paper is working on the real time data set and it contains a mixture of printed and handwritten natural scene text images.

Pragmatic degradation learning for scene text image super-resolution with data-training strategy

Super-resolution of scene text images represents a formidable computational problem, marred by a myriad of intricate challenges. This paper focuses on the specific hurdles that have impeded significant advancements in this domain, and introduces the Higher-Order Degradation-Based Super-Resolution Network (HDSN) as a novel solution to address these intricate issues. The challenges in super-resolving scene text images are manifold. Firstly, the semantic ambiguity inherent to text in natural scenes often leads to degraded results, as standard super-resolution techniques struggle to preserve meaningful textual content. Additionally, the uncertainty surrounding font variability exacerbates this issue, as different fonts require distinct treatment for optimal super-resolution. Furthermore, scene text images often exhibit long trailing shadows, artifacts, and strong noise, rendering conventional methods inadequate in producing satisfactory results. To tackle these intricate challenges, we propose a pragmatic higher-order degradation modeling process. This process takes into account the nuanced characteristics of scene text images, including the diverse forms of noise such as Gaussian, Poisson, speckle, and JPEG compression noise, as well as varying levels of blurring. By meticulously considering these real-world scenarios, our approach significantly enhances the robustness and adaptability of super-resolution for scene text images. In addition to addressing these challenges, we recognize the issues arising from sparse datasets and the lack of corresponding paired images for training. To surmount this limitation, we introduce a text image pre-training strategy, which proves to be highly effective in improving recognition accuracy. The experimental results on TextZoom affirm the effectiveness of our approach, demonstrating substantial improvements over existing methods. Notably, our HDSN achieves average recognition rates of 67.2% on ASTER, 63.2% on MORAN, and 58.0% on CRNN, surpassing the performance of available approaches. Our source code is available at https://github.com/syyang2022/HDSN.

Natural scene text localization and detection using MSER and its variants: a comprehensive survey

Natural scene text localization and detection using MSER and its variants: a comprehensive survey

Text Detection Using Transformation Scaling Extension Algorithm in Natural Scene Images

In recent study efforts, the importance of text identification and recognition in images of natural scenes has been stressed more and more. Natural scene text contains an enormous amount of useful semantic data that can be applied in a variety of vision-related applications. The detection of shape-robust text confronts two major challenges: 1. A large number of traditional quadrangular bounding box-based detectors failed to identify text with irregular forms, making it difficult to include such text within perfect rectangles.2. Pixel-wise segmentation-based detectors sometimes struggle to identify closely positioned text examples from one another. Understanding the surroundings and extracting information from images of natural scenes depends heavily on the ability to detect and recognise text. Scene text can be aligned in a variety of ways, including vertical, curved, random, and horizontal alignments. This paper has created a novel method, the Transformation Scaling Extention Algorithm (TSEA), for text detection using a mask-scoring R-ConvNN (Region Convolutional Neural Network). This method works exceptionally well at accurately identifying text that is curved and text that has multiple orientations inside real-world input images. This study incorporates a mask-scoring R-ConvNN network framework to enhance the model's ability to score masks correctly for the observed occurrences. By providing more weight to accurate mask predictions, our scoring system eliminates inconsistencies between mask quality and score and enhances the effectiveness of instance segmentation. This paper also incorporates a Pyramid-based Text Proposal Network (PBTPN) and a Transformation Component Network (TCN) to enhance the feature extraction capabilities of the mask-scoring R-ConvNN for text identification and segmentation with the TSEA. Studies show that Pyramid Networks are especially effective in reducing false alarms caused by images with backgrounds that mimic text. On benchmark datasets ICDAR 2015, SCUT-CTW1500 containing multi-oriented and curved text, this method outperforms existing methods by conducting extensive testing across several scales and utilizing a single model. This study expands the field of vision-oriented applications by highlighting the growing significance of effectively locating and detecting text in natural situations.

Optical character recognition with different languages

Optical character recognition is the combination of optical technology and computer technology to identify text in an image and then recognize the text content in the image, providing individuals with a great deal of ease in their daily lives. Document text recognition, natural scene text recognition, bill text recognition, and ID card recognition have been used in daily life, but there are still many factors that lead to inaccurate identification and detection. Therefore, different texts, patterns or characters are suitable for different types of Optical character recognition. In this paper, we can learn about the Optical character recognition operation methods and find the similarities and differences through researching the technical routes and four different types of Optical character recognition. In addition, by comparing the Optical character recognition of several commonly used languages, the advantages and disadvantages of each method can be analysed.

IDBNet: Improved differentiable binarisation network for natural scene text detection

AbstractThe text in the natural scene can express rich semantic information, which helps people understand and analyse daily things. This paper focuses on the problems of discrete text spatial distribution and variable text geometric size in natural scenes with complex backgrounds and proposes an end‐to‐end natural scene text detection method based on DBNet. The authors first use IResNet as the backbone network, which does not increase network parameters while retaining more text features. Furthermore, a module with Transformer is introduced in the feature extraction stage to strengthen the correlation between high‐level feature pixels. Then, the authors add a spatial pyramid pooling structure in the end of feature extraction, which realises the combination of local and global features, enriches the expressive ability of feature maps, and alleviates the detection limitations caused by the geometric size of features. Finally, to better integrate the features of each level, a dual attention module is embedded after multi‐scale feature fusion. Extensive experiments on the MSRA‐TD500, CTW1500, ICDAR2015, and MLT2017 data set are conducted. The results showed that IDBNet can improve the average precision, recall, and F‐measure of a text compared with the state of art text detection methods and has higher predictive ability and practicability.

Natural scene text detection algorithm based on the regional proposal

Text detection in natural scenes is widely used in various fields. The combination of deep learning models and text detection improves the accuracy and efficiency of detection. Its algorithm is also endless, with continuous optimization and improvement. On this basis, some algorithms for text detection in natural scenes are summarized from the two aspects of the Faster R-CNN method and SSD method based on regional suggestion. These algorithms and experimental data are analyzed and compared to further summarize the challenges faced by text detection in natural scenes and the direction of improvement.

PCBSNet: A Pure Convolutional Bilateral Segmentation Network for Real-Time Natural Scene Text Detection

Scene text detection is a fundamental research work in the field of image processing and has extensive application value. Segmentation-based methods have time-consuming feature processing, while post-processing algorithms are excellent. Real-time semantic segmentation methods use lightweight backbone networks for feature extraction and aggregation but lack effective post-processing methods. The pure convolutional network improves model performance by changing key components. Combining the advantages of three types of methods, we propose a Pure Convolutional Bilateral Segmentation Network (PCBSNet) for real-time natural scene text detection. First, we constructed a bilateral feature extraction backbone network to significantly improve detection speed. The low extraction detail branch captures spatial information, while the efficient semantic extraction branch accurately captures semantic features through a series of micro designs. Second, we built an efficient attention aggregation module to guide the efficient and adaptive aggregation of features from the two branches. The fused feature map undergoes feature enhancement to obtain more accurate and reliable feature representation. Finally, we used differentiable binarization post-processing to construct text instance boundaries. To evaluate the effectiveness of the proposed model, we compared it with mainstream lightweight models on three datasets: ICDAR2015, MSRA-TD500, and CTW1500. The F-measure scores were 82.9%, 82.8%, and 78.9%, respectively, and the FPS were 59.1, 94.3, and 75.5 frames per second. We also conducted extensive ablation experiments on the ICDAR2015 dataset to validate the rationality of the proposed improvements. The obtained results indicate that the proposed model significantly improves inference speed while enhancing accuracy and demonstrates good competitiveness compared to other advanced detection methods. However, when faced with curved text, the detection performance of PCBSNet needs to be improved.

A light-weight natural scene text detection and recognition system

A light-weight natural scene text detection and recognition system

Lightweight Scene Text Recognition Based on Transformer.

Scene text recognition (STR) has been a hot research field in computer vision, aiming to recognize text in natural scenes using computers. Currently, attention-based encoder-decoder frameworks struggle to precisely align feature regions with the target object when dealing with complex and low-quality images, a phenomenon known as attention drift. Additionally, with the rise of Transformer, the increasing size of parameters results in higher computational costs. In order to solve the above problems, based on the latest research results of Vision Transformer (ViT), we utilize an additional position-enhancement branch to alleviate attention drift and dynamically fused position information with visual information to achieve better recognition accuracy. The experimental results demonstrate that our model achieves a 3% higher average recognition accuracy on the test set compared to the baseline. Meanwhile, our model maintains the advantage of a small number of parameters and fast inference speed, achieving a good balance between accuracy, speed, and computational load.

SANet-SI: A new Self-Attention-Network for Script Identification in scene images

Developing an automatic method for identifying scripts in natural scene text images is of great importance for improving performance of multilingual OCR. This paper presents a new Self-Attention Network (SANet-SI) for script identification in natural scene text images. The rationale behind proposing SANet-SI is that each script exhibits its own pattern because of different characteristics of scripts. To extract such observations, we explore self-attention-based CNN with a multi-scale feature extraction approach. The proposed multi-scale feature extraction involves local, global features extraction and fusion of both the features. Furthermore, to extract dominant features from the pool of features that contribute more for script identification, we explore Style-based Recalibration Module (SRM) in a new way. In addition, to improve the performance of the identification and reduce the model size, the proposed model uses the Global Average Pooling (GAP) layer, instead of Fully Connected(FC) layers in this work. The proposed model is evaluated on standard datasets, namely, RRC-MLT2017, SIW-13, and CVSI2015 to show effectiveness over state-of-the-art methods in terms of confusion matrix and classification rate. In addition, we also conducted experiments for Cross Dataset Validation to show that the proposed model is independent of the number of scripts and different datasets.

A Convolutional Recurrent Neural-Network-Based Machine Learning for Scene Text Recognition Application

Optical character recognition (OCR) is the process of acquiring text and layout information through analysis and recognition of text data image files. It is also a process to identify the geometric location and orientation of the texts and their symmetrical behavior. It usually consists of two steps: text detection and text recognition. Scene text recognition is a subfield of OCR that focuses on processing text in natural scenes, such as streets, billboards, license plates, etc. Unlike traditional document category photographs, it is a challenging task to use computer technology to locate and read text information in natural scenes. Imaging sequence recognition is a longstanding subject of research in the field of computer vision. Great progress has been made in this field; however, most models struggled to recognize text in images of complex scenes with high accuracy. This paper proposes a new pattern of text recognition based on the convolutional recurrent neural network (CRNN) as a solution to address this issue. It combines real-time scene text detection with differentiable binarization (DBNet) for text detection and segmentation, text direction classifier, and the Retinex algorithm for image enhancement. To evaluate the effectiveness of the proposed method, we performed experimental analysis of the proposed algorithm, and carried out simulation on complex scene image data based on existing literature data and also on several real datasets designed for a variety of nonstationary environments. Experimental results demonstrated that our proposed model performed better than the baseline methods on three benchmark datasets and achieved on-par performance with other approaches on existing datasets. This model can solve the problem that CRNN cannot identify text in complex and multi-oriented text scenes. Furthermore, it outperforms the original CRNN model with higher accuracy across a wider variety of application scenarios.

Location of Scene Text Based on Yolov7

Text is everywhere in daily life, and it carries rich and accurate information. Natural scene text detection technology can be widely used in various fields of life. Chinese is an important tool to carry culture. Therefore, it is of great significance to study natural scene Chinese text detection. For text location in Chinese scene, we use target detection method to train based on YOLOv7 model, and obtain effective detection results.

Dual Relation Network for Scene Text Recognition

Local visual and long-range contextual features yield two complementary cues for human reading text in natural scene. Existing scene text recognition methods mainly extract local features in low level features and then model long-range dependencies in high level, this sequential pipeline may be sub-optimal to construct complete and effective representation. Except for high level features, long-range contextual relation is of importance in low level features as well since it can help separate different characters based on the intervals between characters and thus enhance the features in character area. To address this issue, we develop a dual relation module to extract complementary features in a parallel manner for scene text recognition. Dual relation module consists of a local visual branch and a long-range contextual branch from low level to high level. The local visual branch employs a topological-aware operation to model intra-character characteristic and extract discriminative features of different characters. Meanwhile, the long-range contextual branch utilizes a simple but effective strategy to incorporate inter-character relations into feature maps. Our dual relation module is a plug-and-play block which can be easily incorporated into modern deep architectures. Experimental results demonstrate that our methods achieved top performance on several standard benchmarks. Code and models will become publicly available in the future.

Study on Deep Learning-Based Natural Scene Text Recognition

Aiming at the problem that the natural scene text recognition algorithm pays too much attention to the local character classification content and ignores the global information content of the entire text, a natural scene text recognition algorithm based on multi-network convergence and multi-head attention mechanism is proposed. Firstly, the algorithm uses a multi-network convergence structure to design multiple residual modules to capture contextual features and semantic features in visual features. Then, in the process of character prediction, a multi-head attention mechanism encoder is proposed, which stitches position information, visual features, context features and semantic features into a new feature space. Finally, the new feature space is reweighted by the self-attention mechanism, which improves the accuracy of predicting text information while paying attention to the connection between feature sequences. The recognition accuracy of SVT and ICDAR2015 on the regular and irregular text datasets reached 91.4% and 82.4%, respectively, which improved by about 1.8% and 2.4% compared with the current popular algorithms. Experimental results show that the model can make better use of position features, global semantic features and context features to more accurately identify text content, and improve the accuracy of the model.

A New Language-Independent Deep CNN for Scene Text Detection and Style Transfer in Social Media Images.

Due to the adverse effect of quality caused by different social media and arbitrary languages in natural scenes, detecting text from social media images and transferring its style is challenging. This paper presents a novel end-to-end model for text detection and text style transfer in social media images. The key notion of the proposed work is to find dominant information, such as fine details in the degraded images (social media images), and then restore the structure of character information. Therefore, we first introduce a novel idea of extracting gradients from the frequency domain of the input image to reduce the adverse effect of different social media, which outputs text candidate points. The text candidates are further connected into components and used for text detection via a UNet++ like network with an EfficientNet backbone (EffiUNet++). Then, to deal with the style transfer issue, we devise a generative model, which comprises a target encoder and style parameter networks (TESP-Net) to generate the target characters by leveraging the recognition results from the first stage. Specifically, a series of residual mapping and a position attention module are devised to improve the shape and structure of generated characters. The whole model is trained end-to-end so as to optimize the performance. Experiments on our social media dataset, benchmark datasets of natural scene text detection and text style transfer show that the proposed model outperforms the existing text detection and style transfer methods in multilingual and cross-language scenario.