Text In Scene Images Research Articles

A New Contrastive Learning based Vision Transformer for Sentiment Analysis using Scene Text Images

A New Contrastive Learning based Vision Transformer for Sentiment Analysis using Scene Text Images

Self-supervised memory learning for scene text image super-resolution

Computerised recognition of low-resolution scene text images has been a persistent challenge. To improve the recognition performance, image quality enhancement via image super-resolution technology provides an intuitive solution. Typical deep learning-based scene text image super-resolution methods assume that the image quality degradation from high-resolution images to their corresponding low-resolution counterparts can be represented by mapping well-distributed samples, which limits their reconstruction performance in a practical text recognition system. For real-world scenarios this assumption typically does not hold since image degradations arise from multiple sources during image capture and processing. In this paper, to alleviate this problem, we propose a novel self-supervised end-to-end memory network model for scene text image super-resolution. In particular, after extracting enriched and finer representations from low-resolution text images via a spatial refinement block, we introduce a memory-based network to yield an improved super-resolution model that can handle complex degradation sources. Furthermore, to boost the effectiveness of our method, we design a multi-term loss to exploit textual structure information, where, in addition to the traditional reconstruction loss, we embed a character perceptual loss and a boundary enhancement loss. Extensive experiments on different datasets demonstrate that our proposed MNTSR method effectively improves the recognition accuracy for several scene text image recognition models and achieves state-of-the-art results. The source code is made available at https://github.com/xyzhu1/MNTSR.

Batch-transformer for scene text image super-resolution

Batch-transformer for scene text image super-resolution

A Hybrid Scene Text Script Identification Network for Regional Indian Languages

In this work, we introduce WAFFNet, an attention-centric feature fusion architecture tailored for word-level multi-lingual scene text script identification. Motivated by the limitations of traditional approaches that rely exclusively on feature-based methods or deep learning strategies, our approach amalgamates statistical and deep features to bridge the gap. At the core of WAFFNet, we utilized the merits of Local Binary Pattern—a prominent descriptor capturing low-level texture features with high-dimensional, semantically-rich convolutional features. This fusion is judiciously augmented by a spatial attention mechanism, ensuring targeted emphasis on semantically critical regions of the input image. To address the class imbalance problem in multi-class classification scenarios, we employed a weighted objective function. This not only regularizes the learning process but also addresses the class imbalance problem. The architectural integrity of WAFFNet is preserved through an end-to-end training paradigm, leveraging transfer learning to expedite convergence and optimize performance metrics. Considering the under-representation of regional Indian languages in current datasets, we meticulously curated IIITG-STLI2023, a comprehensive dataset encapsulating English alongside six under-represented Indian languages: Hindi, Kannada, Malayalam, Telugu, Bengali, and Manipuri. Rigorous evaluation of the IIITG-STLI2023, as well as the established MLe2e and SIW-13 datasets, underscores WAFFNet’s supremacy over both traditional feature-engineering approaches as well as state-of-the-art deep learning frameworks. Thus, the proposed WAFFNet framework offers a robust and effective solution for language identification in scene text images.

Box2Mask: Box-Supervised Instance Segmentation via Level-Set Evolution.

In contrast to fully supervised methods using pixel-wise mask labels, box-supervised instance segmentation takes advantage of simple box annotations, which has recently attracted increasing research attention. This paper presents a novel single-shot instance segmentation approach, namely Box2Mask, which integrates the classical level-set evolution model into deep neural network learning to achieve accurate mask prediction with only bounding box supervision. Specifically, both the input image and its deep features are employed to evolve the level-set curves implicitly, and a local consistency module based on a pixel affinity kernel is used to mine the local context and spatial relations. Two types of single-stage frameworks, i.e., CNN-based and transformer-based frameworks, are developed to empower the level-set evolution for box-supervised instance segmentation, and each framework consists of three essential components: instance-aware decoder, box-level matching assignment and level-set evolution. By minimizing the level-set energy function, the mask map of each instance can be iteratively optimized within its bounding box annotation. The experimental results on five challenging testbeds, covering general scenes, remote sensing, medical and scene text images, demonstrate the outstanding performance of our proposed Box2Mask approach for box-supervised instance segmentation. In particular, with the Swin-Transformer large backbone, our Box2Mask obtains 42.4% mask AP on COCO, which is on par with the recently developed fully mask-supervised methods.

Scene Text Image Superresolution Through Multiscale Interaction of Structural and Semantic Priors

Scene Text Image Superresolution Through Multiscale Interaction of Structural and Semantic Priors

Scene text image super-resolution using multi-scale convolutional neural network with skip connections

Scene text image super-resolution using multi-scale convolutional neural network with skip connections

SARN: Script-Aware Recognition Network for scene multilingual text recognition

Recent research in script-specific text recognition has achieved remarkable performance on scene text images. However, the challenges of multilingual text in real-world images have not been effectively addressed, thereby limiting the practical application. For multilingual text recognition, the prevailing single-head scheme that uses the same recognition head for all scripts suffers from the issues of enormous alphabet size and linguistic knowledge discrepancies. Meanwhile, the multi-head scheme, which assigns distinct heads for individual scripts, encounters error accumulation within the two-step inference process. In this paper, we explore the influence of script attribute information within existing methods on achieving accurate recognition. Correspondingly, we propose a script-aware recognition network encompassing script identification and text recognition, which leverages script information to make character features more discriminative in text recognition. In particular, we design the script-aware module to utilize script information at the local level, global level, and their combination, respectively. Furthermore, our method is extended to the multi-head scheme, utilizing mutual assistance between script and character features to improve the performance in both script identification and text recognition tasks. Extensive experiments demonstrate that our method obtains superior results in multilingual text recognition, and reveal the importance of script information in alleviating the issues of single-head and multi-head recognition scheme.

Text Image Inpainting via Global Structure-Guided Diffusion Models

Real-world text can be damaged by corrosion issues caused by environmental or human factors, which hinder the preservation of the complete styles of texts, e.g., texture and structure. These corrosion issues, such as graffiti signs and incomplete signatures, bring difficulties in understanding the texts, thereby posing significant challenges to downstream applications, e.g., scene text recognition and signature identification. Notably, current inpainting techniques often fail to adequately address this problem and have difficulties restoring accurate text images along with reasonable and consistent styles. Formulating this as an open problem of text image inpainting, this paper aims to build a benchmark to facilitate its study. In doing so, we establish two specific text inpainting datasets which contain scene text images and handwritten text images, respectively. Each of them includes images revamped by real-life and synthetic datasets, featuring pairs of original images, corrupted images, and other assistant information. On top of the datasets, we further develop a novel neural framework, Global Structure-guided Diffusion Model (GSDM), as a potential solution. Leveraging the global structure of the text as a prior, the proposed GSDM develops an efficient diffusion model to recover clean texts. The efficacy of our approach is demonstrated by thorough empirical study, including a substantial boost in both recognition accuracy and image quality. These findings not only highlight the effectiveness of our method but also underscore its potential to enhance the broader field of text image understanding and processing. Code and datasets are available at: https://github.com/blackprotoss/GSDM.

Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model

Recently, diffusion-based image generation methods are credited for their remarkable text-to-image generation capabilities, while still facing challenges in accurately generating multilingual scene text images. To tackle this problem, we propose Diff-Text, which is a training-free scene text generation framework for any language. Our model outputs a photo-realistic image given a text of any language along with a textual description of a scene. The model leverages rendered sketch images as priors, thus arousing the potential multilingual-generation ability of the pre-trained Stable Diffusion. Based on the observation from the influence of the cross-attention map on object placement in generated images, we propose a localized attention constraint into the cross-attention layer to address the unreasonable positioning problem of scene text. Additionally, we introduce contrastive image-level prompts to further refine the position of the textual region and achieve more accurate scene text generation. Experiments demonstrate that our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.

Text Localization and Enhancement of Mobile Camera based Complex Natural Bilingual Text Scene Images

In this paper, we proposed a text localization and enhancement of the Mobile camera based complex natural scene images using fixed coordinates for removal of unwanted non text regions and making it localized. The morphological erosion technique is applied for the smoothness of foreground text regions. Here, the experimental working nature of the proposed method is first localized color input image is converted into a gray level, then a median filter is applied for the removal of noises. After that, the input image is further converted into binary image. In addition to this, morphological dilation and erosion operations also applied for connecting the broken pixels. Further, the connected component segmentation technique is applied to extract the foreground text regions from the binary input image. The real time input samples are collected by using mobile camera from the various categories of natural scenes of monolingual/bilingual text such as Stone, Wall, Poster and Iron-based. A total of 140 real time samples of natural scene text images are considered for the experimental setup. The experimental results shown that 132 real time sample images are localized and then obtained the accuracy of 94.24%. Further, bilingual text regions are enhanced using hybrid approach and obtain the segmentation accuracy of 83.17%. The novelty of the paper is working on the real time data set and it contains a mixture of printed and handwritten natural scene text images.

Scene text image super-resolution via textual reasoning and multiscale cross-convolution

Scene text image super-resolution via textual reasoning and multiscale cross-convolution

Explicitly-Decoupled Text Transfer With Minimized Background Reconstruction for Scene Text Editing.

Scene text editing aims to replace the source text with the target text while preserving the original background. Its practical applications span various domains, such as data generation and privacy protection, highlighting its increasing importance in recent years. In this study, we propose a novel Scene Text Editing network with Explicitly-decoupled text transfer and Minimized background reconstruction, called STEEM. Unlike existing methods that usually fuse text style, text content, and background, our approach focuses on decoupling text style and content from the background and utilizes the minimized background reconstruction to reduce the impact of text replacement on the background. Specifically, the text-background separation module predicts the text mask of the scene text image, separating the source text from the background. Subsequently, the style-guided text transfer decoding module transfers the geometric and stylistic attributes of the source text to the content text, resulting in the target text. Next, the background and target text are combined to determine the minimal reconstruction area. Finally, the context-focused background reconstruction module is applied to the reconstruction area, producing the editing result. Furthermore, to ensure stable joint optimization of the four modules, a task-adaptive training optimization strategy has been devised. Experimental evaluations conducted on two popular datasets demonstrate the effectiveness of our approach. STEEM outperforms state-of-the-art methods, as evidenced by a reduction in the FID index from 29.48 to 24.67 and an increase in text recognition accuracy from 76.8% to 78.8%.

Pragmatic degradation learning for scene text image super-resolution with data-training strategy

Super-resolution of scene text images represents a formidable computational problem, marred by a myriad of intricate challenges. This paper focuses on the specific hurdles that have impeded significant advancements in this domain, and introduces the Higher-Order Degradation-Based Super-Resolution Network (HDSN) as a novel solution to address these intricate issues. The challenges in super-resolving scene text images are manifold. Firstly, the semantic ambiguity inherent to text in natural scenes often leads to degraded results, as standard super-resolution techniques struggle to preserve meaningful textual content. Additionally, the uncertainty surrounding font variability exacerbates this issue, as different fonts require distinct treatment for optimal super-resolution. Furthermore, scene text images often exhibit long trailing shadows, artifacts, and strong noise, rendering conventional methods inadequate in producing satisfactory results. To tackle these intricate challenges, we propose a pragmatic higher-order degradation modeling process. This process takes into account the nuanced characteristics of scene text images, including the diverse forms of noise such as Gaussian, Poisson, speckle, and JPEG compression noise, as well as varying levels of blurring. By meticulously considering these real-world scenarios, our approach significantly enhances the robustness and adaptability of super-resolution for scene text images. In addition to addressing these challenges, we recognize the issues arising from sparse datasets and the lack of corresponding paired images for training. To surmount this limitation, we introduce a text image pre-training strategy, which proves to be highly effective in improving recognition accuracy. The experimental results on TextZoom affirm the effectiveness of our approach, demonstrating substantial improvements over existing methods. Notably, our HDSN achieves average recognition rates of 67.2% on ASTER, 63.2% on MORAN, and 58.0% on CRNN, surpassing the performance of available approaches. Our source code is available at https://github.com/syyang2022/HDSN.

Buffer-text: Detecting arbitrary shaped text in natural scene image

Scene text detection has always been a research hotspot in computer vision and image understanding. With the development of deep learning, segmentation-based methods have achieved an exceptional effect in regular or curved text detection, but they cannot separate adjacent word-level texts. In this paper, we proposed a detector called Buffer-Text for the detection of irregular text in the natural scene image. First, the buffer region is proposed for bending text detection which widens the spatial distance between word-level texts. Then, a centerline-based polygon expansion algorithm is developed for the acquisition of the buffer region. After that, the scene text image is divided into different regions which are predicted by adopting the idea of multiclass semantic segmentation. To obtain effective segmentation results and solve the category imbalance problem, a Fully Convolutional Networks (FCN) with Spatial and Channel Squeeze & Excitation Block module is designed, and a loss function with adaptive weight updating is defined for the network. Ultimately, the post-processing including the total erosion and the single expansion is applied to eliminate the areas of noise in the segmented image and to separate the weak junctions in the word-level text. To verify the validity of the proposed method, several experiments were conducted on two curved text datasets, namely Total-Text and CTW1500, and the results indicated that the proposed method achieved significant accuracy in three statistical indicators (precision, recall, and F-score), particularly for the images with natural scenes and various text shapes.

Dynamic receptive field adaptation for scene text recognition

Scene text recognition methods of the Encoder–Decoder framework, generally assume that the proportion of characters in the same text instance are basically the same. However, this assumption does not always hold in the context of complex scene images. For adaptively revising the receptive field according to the different font in the scene text image, we propose a Dynamic Receptive Field Adaption Framework which consists of Memory Attention (MA) module and Dynamic Feature Adaptive (DFA) module. MA percepts the historical location information to adapt to the change of character position in the decoder. DFA selects the most distinguishing features from feature maps of different levels dynamically. Additionally, MA and DFA can be easily extended to the existing attention-based and transformer-based text recognition methods to improve their performance. With extension experiments on public benchmark datasets, including IIIT-5K, SVT, SVTP, CUTE80, RECTS, LSVT, and RCTW, our method has shown effectiveness and robustness.

SECANet: A structure‐enhanced attention network with dual‐domain contrastive learning for scene text image super‐resolution

AbstractIn this letter, we developed novel Structure Enhanced Channel Attention Network (SECANet) for scene text image super‐resolution (STISR). The newly proposed SECANet integrates a group of Structure‐Enhanced Attention Modules to focus more on both local and global structural features in the character regions of text images. Moreover, we elaborately formulate a Dual‐Domain Contrastive Learning framework that integrates one pixel‐level contrastive loss and the other semantic‐level contrastive loss to jointly optimize the SECANet for generating more visually pleasing yet better recognizable high‐quality SR images without introducing any additional prior generators in both the training and testing stages, showing promising computational efficiency. Experimental results on the Textzoom dataset indicate that our method can achieve both decent performance in super‐resolving more impressive scene text images from low‐resolution ones and better recognition accuracy than other competitors.

HAFE: A Hierarchical Awareness and Feature Enhancement Network for Scene Text Recognition

Scene text recognition (STR) is to obtain text information from natural text images. Compared to the optical character recognition, STR has attracted much attentions due to the variability of scene images. However, most STR methods tend to loss some spatial information due to only simply adding Resnet-45 to the front end of the encoder. To solve this problem, we propose a Hierarchical Awareness and Feature Enhancement (HAFE) network. Our proposed Hierarchical Awareness (HA) module can effectively extract the depth information of scene text images from multiple dimensions and then improve the anti-interference ability of the network. Besides, the proposed Feature Enhancement (FE) module can enhance the stability of the model through strengthening the global features, which is helpful to improve the accuracy of text recognition. On the whole, our HAFE network tends to strengthen the information in encoder, better learn the feature maps from different levels, and then improve the recognition accuracy of non-horizontal texts. Extensive comparative experiments on seven benchmark datasets show that the proposed method has the most advanced performance and outperforms most state-of-the-art ones. Our code will be released at website: https://github.com/HAFE in the future.

Few‐shot learning for word‐level scene text script identification

AbstractScript identification of text in scene images has attracted massive attention recently. However, the existing techniques primarily emphasize on scripts where data are available abundantly, such as English, European, or East Asian. Although these methods are robust in dealing with high‐resource data, how these techniques will work on low‐resource scripts has yet to be discovered. For example, in India, there is a disparity among the text scripts across the country's demographic. To bridge the research gap for resource‐constraint script identification, we present a few‐shot learning network called the TextScriptFSLNet. This network does not require huge training data while achieving state‐of‐the‐art performance on benchmark datasets. Our proposed method acts in accordance with a ‐way ‐shot paradigm by splitting the train set as support and query set, respectively. The support set learns representative knowledge of each class and creates its prototypes. We use multi‐kernel spatial attention fused 2‐layer convolutional neural network and averaging technique to generate the prototype of each class. This spatial attention aids in grasping important information in an image and enriches the feature representation. To the best of our knowledge, the proposed work is the first of its kind in the scene text understanding domain. Additionally, we created a dataset called Indic‐FSL2023 comprising 10 of the 22 officially recognized Indian scripts. The proposed method achieves the highest accuracy among the tested methods on the newly created Indic‐FSL2023. Experiments are also conducted on MLe2e to demonstrate its versatility. Furthermore, we also showed how our proposed model performed concerning illumination changes and blur on scene text script images.

Pixel-Level Degradation for Text Image Super-Resolution and Recognition

In the realm of image reconstruction, deep learning-based super-resolution (SR) has established itself as a prevalent technique, particularly in the domain of text image restoration. This study aims to address notable deficiencies in existing research, including constraints imposed by restricted datasets and challenges related to model generalization. Specifically, the goal is to enhance the super-resolution network’s reconstruction of scene text image super-resolution and utilize the generated degenerate dataset to alleviate issues associated with poor generalization due to the sparse scene text image super-resolution dataset. The methodology employed begins with the degradation of images from the MJSynth dataset, using a stochastic degradation process to create eight distinct degraded versions. Subsequently, a blank image is constructed, preserving identical dimensions to the low-resolution image, with each pixel sourced randomly from the corresponding points across the eight degraded images. Following several iterations of training via Finetune, the LR-HR method is applied to the TextZoom dataset. The pivotal metric for assessment is optical character recognition (OCR) accuracy, recognized for its fundamental role in gauging the pragmatic effectiveness of this approach. The experimental findings reveal a notable enhancement in OCR accuracy when compared to the TBSRN model, yielding improvements of 2.4%, 2.3%, and 4.8% on the TextZoom dataset. This innovative approach, founded on pixel-level degradation, not only exhibits commendable generalization capabilities but also demonstrates resilience in confronting the intricate challenges inherent to text image super-resolution.