Text In Natural Scene Images Research Articles

A Comprehensive Review on Text Detection and Recognition in Scene Images

Detecting and recognizing text in natural scene images and videos is vital for several real-world applications, such as in the analysis of Crime scene CCTV footage, sports videos, and autonomous driving, to name a few. Therefore, one can expect several challenges, namely, arbitrarily oriented and shaped Text detection and identification in movies and natural environments. Many methods have been developed in the past to address these challenges, including advanced deep-learning models and transformers. Due to several methods available in the literature, it is not so easy to understand the open challenges, applications, directions, scope, limitations, and weaknesses of the methods. Therefore, there is a need to write a survey/review to highlight and discuss the strengths and weaknesses of the developed methods. This survey/review presents different categories of work and discusses their importance, limitations, new challenges, applications, and, finally, directions such that readers can choose appropriate methods and directions to carry out research work in the field of text detection/recognition in the natural scene and videos.

An End-to-End Scene Text Recognition for Bilingual Text

Text localization and recognition from natural scene images has gained a lot of attention recently due to its crucial role in various applications, such as autonomous driving and intelligent navigation. However, two significant gaps exist in this area: (1) prior research has primarily focused on recognizing English text, whereas Arabic text has been underrepresented, and (2) most prior research has adopted separate approaches for scene text localization and recognition, as opposed to one integrated framework. To address these gaps, we propose a novel bilingual end-to-end approach that localizes and recognizes both Arabic and English text within a single natural scene image. Specifically, our approach utilizes pre-trained CNN models (ResNet and EfficientNetV2) with kernel representation for localization text and RNN models (LSTM and BiLSTM) with an attention mechanism for text recognition. In addition, the AraElectra Arabic language model was incorporated to enhance Arabic text recognition. Experimental results on the EvArest, ICDAR2017, and ICDAR2019 datasets demonstrated that our model not only achieves superior performance in recognizing horizontally oriented text but also in recognizing multi-oriented and curved Arabic and English text in natural scene images.

Deep Learning Techniques for Detecting and Segmenting Text in Natural Scene Images: Review

Text detection and segmentation in natural scene images is an active research problem in computer vision and document analysis. Unlike scanned documents, scene text exhibits significant diversity in appearance, orientation, scale, font, and lighting conditions. In this review, survey the current state-of-the-art in techniques and methodologies aimed at detecting and segmenting text regions from images of natural scenes are presented. Both traditional approaches using hand-crafted features as well as modern data-driven deep learning methods will be discussed. The review will analyze common datasets, evaluation protocols and metrics used for benchmarking. Limitations of existing methods and open challenges in handling multilingual text, curved text, and efficiency will be highlighted. Promising future directions towards robust and generalizable scene text extraction systems will be identified. In summary, the review will provide a comprehensive overview of the advances, remaining challenges and future opportunities in developing automated systems for detecting and segmenting text in unconstrained natural images.

Irregular Scene Text Detection Based on a Graph Convolutional Network.

Detecting irregular or arbitrary shape text in natural scene images is a challenging task that has recently attracted considerable attention from research communities. However, limited by the CNN receptive field, these methods cannot directly capture relations between distant component regions by local convolutional operators. In this paper, we propose a novel method that can effectively and robustly detect irregular text in natural scene images. First, we employ a fully convolutional network architecture based on VGG16_BN to generate text components via the estimated character center points, which can ensure a high text component detection recall rate and fewer noncharacter text components. Second, text line grouping is treated as a problem of inferring the adjacency relations of text components with a graph convolution network (GCN). Finally, to evaluate our algorithm, we compare it with other existing algorithms by performing experiments on three public datasets: ICDAR2013, CTW-1500 and MSRA-TD500. The results show that the proposed method handles irregular scene text well and that it achieves promising results on these three public datasets.

Deep-Learning-Based Complex Scene Text Detection Algorithm for Architectural Images

With the advent of smart cities, the text information in an image can be accurately located and recognized, and then applied to the fields of instant translation, image retrieval, card surface information recognition, and license plate recognition. Thus, people’s lives and work will become more convenient and comfortable. Owing to the varied orientations, angles, and shapes of text, identifying textual features from images is challenging. Therefore, we propose an improved EAST detector algorithm for detecting and recognizing slanted text in images. The proposed algorithm uses reinforcement learning to train a recurrent neural network controller. The optimal fully convolutional neural network structure is selected, and multi-scale features of text are extracted. After importing this information into the output module, the Generalized Intersection over Union algorithm is used to enhance the regression effect of the text bounding box. Next, the loss function is adjusted to ensure a balance between positive and negative sample classes before outputting the improved text detection results. Experimental results indicate that the proposed algorithm can address the problem of category homogenization and improve the low recall rate in target detection. When compared with other image detection algorithms, the proposed algorithm can better identify slanted text in natural scene images. Finally, its ability to recognize text in complex environments is also excellent.

Dominating set based arbitrary oriented bilingual scene text localization

<p>Localizing and recognizing arbitrarily oriented text in natural scene images is the biggest challenge. It is because scene texts are often erratic in shapes. This paper presents a simple and effective graph representational algorithm for detecting arbitrary-oriented text location to smoothen the text recognition process because of its high impact and simplicity of representation. An arbitrarily oriented text can be horizontal, vertical, perspective, curved (diagonal/off-diagonal), or even a combination. As a pre-processing step, image enhancement is performed in the frequency domain to improve the representation of images that are invariant to intensity. It is necessary to draw bounding boxes for each candidate character in the scene images to extract text regions. This step is carried out by utilizing the advantage of the region-based approach called maximally stable extremal regions. A typical problem with curved text localization is that non-text objects may occur within localized text regions. Our method is the first in the literature that searches for dominating sets to solve this problem. This dominating set method outperforms several traditional methods, including deep learning methods used for arbitrary text localization, on challenging datasets like 13<sup>th</sup> international conference on document analysis and recognition (ICDAR 2015), multi-script robust reading competition (MRRC), CurvedText 80 (CUTE80), and arbitrary text (ArT).</p>

Local Resultant Gradient Vector Difference and Inpainting for 3D Text Detection in the Wild

Three-dimensional (3D) text appearing in natural scene images is common due to 3D cameras and the capture of text from different angles, which presents new problems for text detection. This is because of the presence of depth information, shadows, and decorative characters in the images. In this work, we consider those images where 3D text appears with depth, as well as shadow information for text detection. We propose a novel method based on local resultant gradient vector difference (LRGVD), inpainting and a deep learning model for detecting 3D as well as two-dimensional (2D) texts in natural scene images. The boundary of components that are invariant to the above challenges is detected by exploring LRGVD. The LRGVD uses gradient magnitude and direction in a novel way for detecting the boundary of the components. Further, we propose an inpainting method in a new way for restoring the character background information using boundaries. For a given region and the input image, the inpainting method divides the whole image into planes and then propagates the values in the planes into the missing region based on posterior probabilities and neighboring information. This results in text regions with false positives. Then, the differential binarization network (DB-Net) is proposed for detecting text irrespective of orientation, background, 3D or 2D, etc. Experiments conducted on our 3D text images and standard datasets of natural scene text images, namely ICDAR 2019 MLT, ICDAR 2019 ArT, DAST1500, Total-Text and SCUT-CTW1500, show that the proposed method is effective in detecting 3D and 2D texts in the images.

Contour feature learning for locating text in natural scene images

Contour feature learning for locating text in natural scene images

A new method for detection and prediction of occluded text in natural scene images

Text detection from natural scene images is an active research area for computer vision, signal, and image processing because of several real-time applications such as driving vehicles automatically and tracing person behaviors during sports or marathon events. In these situations, there is a high probability of missing text information due to the occlusion of different objects/persons while capturing images. Unlike most of the existing methods, which focus only on text detection by ignoring the effect of missing texts, this work detects and predicts missing texts so that the performance of the OCR improves. The proposed method exploits the property of DCT for finding significant information in images by selecting multiple channels. For chosen DCT channels, the proposed method studies texture distribution based on statistical measurement to extract features. We propose to adopt Bayesian classifier for categorizing text pixels using extracted features. Then a deep learning model is proposed for eliminating false positives to improve text detection performance. Further, the proposed method employs a Natural Language Processing (NLP) model for predicting missing text information by using detected and recognition texts. Experimental results on our dataset, which contains texts occluded by objects, show that the proposed method is effective in predicting missing text information. To demonstrate the effectiveness and objectiveness of the proposed method, we also tested it on the standard datasets of natural scene images, namely, ICDAR 2017-MLT, Total-Text, and CTW1500.

Urdu text in natural scene images: a new dataset and preliminary text detection.

Text detection in natural scene images for content analysis is an interesting task. The research community has seen some great developments for English/Mandarin text detection. However, Urdu text extraction in natural scene images is a task not well addressed. In this work, firstly, a new dataset is introduced for Urdu text in natural scene images. The dataset comprises of 500 standalone images acquired from real scenes. Secondly, the channel enhanced Maximally Stable Extremal Region (MSER) method is applied to extract Urdu text regions as candidates in an image. Two-stage filtering mechanism is applied to eliminate non-candidate regions. In the first stage, text and noise are classified based on their geometric properties. In the second stage, a support vector machine classifier is trained to discard non-text candidate regions. After this, text candidate regions are linked using centroid-based vertical and horizontal distances. Text lines are further analyzed by a different classifier based on HOG features to remove non-text regions. Extensive experimentation is performed on the locally developed dataset to evaluate the performance. The experimental results show good performance on test set images. The dataset will be made available for research use. To the best of our knowledge, the work is the first of its kind for the Urdu language and would provide a good dataset for free research use and serve as a baseline performance on the task of Urdu text extraction.

Rule-based perspective rectification for Chinese text in natural scene images

Recognizing text with large perspective deformation is challenging, especially when the orientation of the text is arbitrary. A rule-based method is proposed in this paper to recover the fronto-parallel image for Chinese text in natural scenes. It relies on the fact that there are many horizontal and vertical strokes in Chinese text and they are accurately or approximately perpendicular to each other in non-italic type. A homography is utilized to describe the relationship between the scene text and its image. Horizontal and vertical strokes are extracted and selected to construct perspective quadrilaterals and the expected rectangles for calculating the homography. Experiments show that the proposed method could rectify the perspective deformation of Chinese text in any orientation effectively and it outperforms the state-of-the-art perspective rectification method based on vanishing point detection.

A comparative approach on detecting multi-lingual and multi-oriented text in natural scene images

Text helps to convey the intended message to users very accurately. Detecting text from natural scene images for quadrilateral-type and polygon-type datasets is the primary scope of this work. A regression-based method using modified You Only Look Once YOLOv4 network is used for quadrilateral-type datasets. Hyperparameters for training the network are optimized using the Genetic Algorithm which proves to be a suitable candidate than traditional methods. The Pixels-IoU (PIoU) loss is introduced to derive an accurate bounding box and it seems to be productive under various challenging scenarios with high aspect ratios and complex background. This yielded quick results for quadrilateral-type datasets but did not scale for arbitrarily-shaped and curved scene text. So the approach is changed to segmentation based for enhancing the results. This introduces binarization operation in a segmentation network to boost its detection accuracy for polygon-type datasets. The introduction of a new module DiffBiSeg (Differentiable Binarization in Segmentation network) facilitates post-processing and text detection performance by setting the thresholds flexibly for binarization in the segmentation network. The efficacy of both approaches is clearly seen in their respective experimental results.

Cursive-Text: A Comprehensive Dataset for End-to-End Urdu Text Recognition in Natural Scene Images.

Reading text in natural scene images is an active research area in the fields of computer vision and pattern recognition as text detection, text recognition and script identification are required. In this data article, a comprehensive dataset for Urdu text detection and recognition in natural scene images is presented and analysed. To develop the dataset, more than 2500 natural scene images were captured using a digital camera and a built-in mobile phone camera. Three separate datasets for isolated Urdu character images, cropped word images and end-to-end text spotting were developed. The isolated Urdu character and cropped word images dataset contain a much larger number of samples than existing Arabic natural scene text datasets. The Urdu text spotting dataset contains images with Urdu, English and Sindhi text instances. However, the focus has been given to the Urdu text instances. The ground truths for each image in the isolated character, cropped word or text spotting datasets are provided separately. The proposed datasets can be used to perform Urdu text detection and recognition or end-to-end recognition in natural scenes. These datasets can also be helpful to develop Arabic and Persian natural scene text detection and recognition systems, as Urdu is a derived language of these scripts and has many similar letters. The datasets can also be helpful to develop multi-language translation systems, which can facilitate foreign tourists to read and translate multilingual text in natural scene images. To evaluate the datasets, state-of-the-art machine learning and deep neural networks were used to build the text detection and recognition models, where the best classification accuracies are achieved. To the best of the authors’ knowledge, this is the first dataset proposed for Urdu text detection, recognition or end-to-end text recognition in natural scene images. The aim of this data article is to present a benchmark work in the field of document analysis and recognition.Computer ScienceComputer Vision and Pattern RecognitionTablesFiguresImagesText FilesUsing a digital camera with a 20 megapixels (MP) sensor, an iPhone with a 12 MP back camera and a Samsung mobile with a 16MP back camera.RawAnalyzedEnvironmental factors such as illuminations, blurring and lighting conditions were considered while capturing images. The focus was given to the text within an image.The images in the dataset were obtained from the advertisement banners, sign-boards along the road side and streets, shop name boards, text written on the passing vehicles and walls.The images provided in this dataset were collected in different cities of Sindh, Pakistan.Summarized data are hosted with the article.The datasets and their related files are hosted in a Mendeley public data repository.DOI: https://data.mendeley.com/datasets/k5fz57zd9z/1URL: http://dx.doi.org/10.17632/k5fz57zd9z.1

Text detection in natural scene images using morphological component analysis and Laplacian dictionary

Text in natural scene images usually carries abundant semantic information. However, due to variations of text and complexity of background, detecting text in scene images becomes a critical and challenging task. In this paper, we present a novel method to detect text from scene images. Firstly, we decompose scene images into background and text components using morphological component analysis &#x0028 MCA &#x0029 , which will reduce the adverse effects of complex backgrounds on the detection results. In order to improve the performance of image decomposition, two discriminative dictionaries of background and text are learned from the training samples. Moreover, Laplacian sparse regularization is introduced into our proposed dictionary learning method which improves discrimination of dictionary. Based on the text dictionary and the sparse-representation coefficients of text, we can construct the text component. After that, the text in the query image can be detected by applying certain heuristic rules. The results of experiments show the effectiveness of the proposed method.

Text Extraction and Recognition in Natural Scene Images using Contourlet Transform and PNN

Of late, the rapid development in the technology and multimedia capability in digital cameras and mobile devices has led to ever increasing number of images or multi-media data to the digital world. Particularly, in natural scene images, the text content provides explicit information to understand the semantics of images. Therefore, a system developed for extracting and recognizing texts accurately from natural scene images, in real-time, has significant relevance to numerous applications such as, assistive technology for people with vision impairment, tourist with language barrier, vehicle number plate detection, street signs, advertisement bill boards, robotics, etc. The extraction of the texts from natural scene images is a formidable task due to large variations in character fonts, styles, sizes, text orientations, presence of complex backgrounds and varying light conditions. The main focus of this research paper is to propose a novel hybrid approach for automatic detection, localization, extraction and recognition of text in natural scene images with cluttered background. Firstly, image regions with text are detected using edge features (GLCM) extracted from Contourlet transformed image and SVM (Support Vector Machine) classifier. Secondly, horizontal projection is applied on text regions for segmenting lines and vertical projection is applied on each text line for segmenting characters. The proposed method for text extraction has produced the precision, recall, F-Score and accuracy of 98.50%, 90.85.62%, 95.00%, and 89.90%, respectively. And, these results prove that the proposed method is efficient. Further, the so extracted characters are processed for recognition using contourlet transform and Probabilistic Neural Network (PNN) classifier. The computed features are moment invariants. Only the English script is considered for the experimentation. The proposed character recognition method has accuracy of 79.07%, which is higher in comparison to accuracy of 75.15% obtained by KNN (K-Nearest Neighbors) classifier

Real-time localization of multi-oriented text in natural scene images using a linear spatial filter

This paper proposes a multi-oriented text localization method in natural images suitable for real-time processing of high-definition video on portable and mobile devices. Our method is based on the connected components (CC) approach: first, CC are isolated by convolving a multi-scale pyramid with a specifically designed linear spatial filter followed by hysteresis thresholding. Next, non-textual CC are pruned employing a local classifier consisting of a cascade of multilayer perceptron (MLP) fed with increasingly extended feature vectors. The stroke width feature is estimated in linear time complexity by computing the maximal inscribed squares in the CC. Candidate CC and their neighbors are then checked using a more context aware neural network classifier that takes into account the target CC and their vicinity. Finally, text sequences are extracted in all pyramid levels and fused using dynamic programming. The main contribution of the work presented here is execution speed: the CPU-only parallel implementation of the proposed method is capable of processing 1080p HD video at nearly 30 frames per second on a standard laptop. Furthermore, when benchmarked on the ICDAR 2013 Robust Reading and on the ICDAR 2015 Incidental Scene Text data sets, our system performs more than twice faster than the state-of-the-art, while still delivering competitive results in terms of precision and recall.

Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition

Recognizing irregular text in natural scene images is challenging due to the large variance in text appearance, such as curvature, orientation and distortion. Most existing approaches rely heavily on sophisticated model designs and/or extra fine-grained annotations, which, to some extent, increase the difficulty in algorithm implementation and data collection. In this work, we propose an easy-to-implement strong baseline for irregular scene text recognition, using offthe-shelf neural network components and only word-level annotations. It is composed of a 31-layer ResNet, an LSTMbased encoder-decoder framework and a 2-dimensional attention module. Despite its simplicity, the proposed method is robust. It achieves state-of-the-art performance on irregular text recognition benchmarks and comparable results on regular text datasets. The code will be released.

Multi-level Fuzzy Based Renyi Entropy for Linguistic Classification of Texts in Natural Scene Images

This paper focuses on linguistic classification of scene texts in natural scene images. In this paper, an attempt is made to localize texts based on multi-level thresholding by fuzzy-based Renyi entropy. Complex natural scene images with diversified challenges are considered. A set of heuristic rules comprising geometric filters and stroke width transform govern the process of locating potential text regions. The scene images may contain more than one language, where text recognition by optical character recognition system becomes challenging. Manual intervention is needed to specify the language of each text. To overcome this hurdle, linguistic classification of text regions is suggested in this paper. The proposed method is validated using publicly available dataset—MSRA-TD500. Results show that fuzzy-based Renyi entropy thresholding is able to segment the foreground text from complex natural scene images. Geometric filters could capture the inherent uniformity of the text. Stroke width transform eliminates the non-text regions. The performance measures such as precision, recall and F-measures are 78%, 77% and 76%, respectively. This shows the ability of the algorithm to extract the text from the scenes. The geometric feature such as area and corner shows better variation in discriminating the linguistic texts. Further, the first three Hu moment features also contribute remarkable role in analyzing the shape of extracted text regions. The classifier based on support vector machine (SVM) yields classification accuracy of 85.45% in discriminating English and Chinese alphabets. Area under the ROC curve (AUC) is 0.851 for SVM classifier. The proposed methodology has proved its robustness against common degradations, such as uneven illumination, varying font characteristics and blurring effects. Experimental results show that our method achieves better performance in linguistic classification.

Scene word recognition from pieces to whole

Convolutional neural networks (CNNs) have had great success with regard to the object classification problem. For character classification, we found that training and testing using accurately segmented character regions with CNNs resulted in higher accuracy than when roughly segmented regions were used. Therefore, we expect to extract complete character regions from scene images. Text in natural scene images has an obvious contrast with its attachments. Many methods attempt to extract characters through different segmentation techniques. However, for blurred, occluded, and complex background cases, those methods may result in adjoined or over segmented characters. In this paper, we propose a scene word recognition model that integrates words from small pieces to entire after-cluster-based segmentation. The segmented connected components are classified as four types: background, individual character proposals, adjoined characters, and stroke proposals. Individual character proposals are directly inputted to a CNN that is trained using accurately segmented character images. The sliding window strategy is applied to adjoined character regions. Stroke proposals are considered as fragments of entire characters whose locations are estimated by a stroke spatial distribution system. Then, the estimated characters from adjoined characters and stroke proposals are classified by a CNN that is trained on roughly segmented character images. Finally, a lexicon-driven integration method is performed to obtain the final word recognition results. Compared to other word recognition methods, our method achieves a comparable performance on Street View Text and the ICDAR 2003 and ICDAR 2013 benchmark databases. Moreover, our method can deal with recognizing text images of occlusion and improperly segmented text images.

A Novel Dataset for English-Arabic Scene Text Recognition (EASTR)-42K and Its Evaluation Using Invariant Feature Extraction on Detected Extremal Regions

© 2019 IEEE. The recognition of text in natural scene images is a practical yet challenging task due to the large variations in backgrounds, textures, fonts, and illumination. English as a secondary language is extensively used in Gulf countries along with Arabic script. Therefore, this paper introduces English-Arabic scene text recognition 42K scene text image dataset. The dataset includes text images appeared in English and Arabic scripts while maintaining the prime focus on Arabic script. The dataset can be employed for the evaluation of text segmentation and recognition task. To provide an insight to other researchers, experiments have been carried out on the segmentation and classification of Arabic as well as English text and report error rates like 5.99% and 2.48%, respectively. This paper presents a novel technique by using adapted maximally stable extremal region (MSER) technique and extracts scale-invariant features from MSER detected region. To select discriminant and comprehensive features, the size of invariant features is restricted and considered those specific features which exist in the extremal region. The adapted MDLSTM network is presented to tackle the complexities of cursive scene text. The research on Arabic scene text is in its infancy, thus this paper presents benchmark work in the field of text analysis.