Scene Text Research Articles

A Hybrid Scene Text Script Identification Network for Regional Indian Languages

In this work, we introduce WAFFNet, an attention-centric feature fusion architecture tailored for word-level multi-lingual scene text script identification. Motivated by the limitations of traditional approaches that rely exclusively on feature-based methods or deep learning strategies, our approach amalgamates statistical and deep features to bridge the gap. At the core of WAFFNet, we utilized the merits of Local Binary Pattern—a prominent descriptor capturing low-level texture features with high-dimensional, semantically-rich convolutional features. This fusion is judiciously augmented by a spatial attention mechanism, ensuring targeted emphasis on semantically critical regions of the input image. To address the class imbalance problem in multi-class classification scenarios, we employed a weighted objective function. This not only regularizes the learning process but also addresses the class imbalance problem. The architectural integrity of WAFFNet is preserved through an end-to-end training paradigm, leveraging transfer learning to expedite convergence and optimize performance metrics. Considering the under-representation of regional Indian languages in current datasets, we meticulously curated IIITG-STLI2023, a comprehensive dataset encapsulating English alongside six under-represented Indian languages: Hindi, Kannada, Malayalam, Telugu, Bengali, and Manipuri. Rigorous evaluation of the IIITG-STLI2023, as well as the established MLe2e and SIW-13 datasets, underscores WAFFNet’s supremacy over both traditional feature-engineering approaches as well as state-of-the-art deep learning frameworks. Thus, the proposed WAFFNet framework offers a robust and effective solution for language identification in scene text images.

FCCText: frequency-color complementary bistream structure for scene text detection

FCCText: frequency-color complementary bistream structure for scene text detection

Super-Resolution Virtual Scene Rendering Technology Based on Generalized Huber-MRF Image Modeling

The traditional rendering technology creates virtual scenes with insufficient fidelity, which are quite different from real scenes. To address this issue, a super-resolution virtual scene rendering technology based on generalized Huber-MRF image modeling has been studied. This study preprocesses the original image through three steps: graying, filtering, and enhancement. The generalized Huber-MRF is employed for super-resolution image restoration to enhance image clarity. Corner features are extracted from the super-resolution image, and the Delaunay triangular grid method is used to construct the image's 3D model. Texture and lighting conditions of the virtual scene are then set through texture mapping, shadow rendering, and other technologies to achieve realistic scene effects. The results indicate that, when applied, the research technology yields a relatively small chamfer distance in virtual scene modeling, suggesting that the design method preserves the details and shape information of the original image, reducing the difference between the virtual scene and the real scene and increasing the fidelity of the virtual scene. Furthermore, this method achieves maximum PSNR and SSIM values of 17.54 and 0.978, respectively, with an image preprocessing time of only 1.21 s and a CPU utilization rate of only 35.5%. This method demonstrates excellent performance across multiple aspects.

Deep learning-based modified-EAST scene text detector: insights from a novel multiscript dataset

Deep learning-based modified-EAST scene text detector: insights from a novel multiscript dataset

Trademark Text Recognition Combining SwinTransformer and Feature-Query Mechanisms

The task of trademark text recognition is a fundamental component of scene text recognition (STR), which currently faces a number of challenges, including the presence of unordered, irregular or curved text, as well as text that is distorted or rotated. In applications such as trademark infringement detection and analysis of brand effects, the diversification of artistic fonts in trademarks and the complexity of the product surfaces where the trademarks are located pose major challenges for relevant research. To tackle these issues, this paper proposes a novel recognition framework named SwinCornerTR, which aims to enhance the accuracy and robustness of trademark text recognition. Firstly, a novel feature-extraction network based on SwinTransformer with EFPN (enhanced feature pyramid network) is proposed. By incorporating SwinTransformer as the backbone, efficient capture of global information in trademark images is achieved through the self-attention mechanism and enhanced feature pyramid module, providing more accurate and expressive feature representations for subsequent text extraction. Then, during the encoding stage, a novel feature point-retrieval algorithm based on corner detection is designed. The OTSU-based fast corner detector is presented to generate a corner map, achieving efficient and accurate corner detection. Furthermore, in the encoding phase, a feature point-retrieval mechanism based on corner detection is introduced to achieve priority selection of key-point regions, eliminating character-to-character lines and suppressing background interference. Finally, we conducted extensive experiments on two open-access benchmark datasets, SVT and CUTE80, as well as a self-constructed trademark dataset, to assess the effectiveness of the proposed method. Our results showed that the proposed method achieved accuracies of 92.9%, 92.3% and 84.8%, respectively, on these datasets. These results demonstrate the effectiveness and robustness of the proposed method in the analysis of trademark data.

Scene text recognition: an Indic perspective

Scene text recognition: an Indic perspective

Correction: DC-PSENet: a novel scene text detection method integrating double ResNet-based and changed channels recursive feature pyramid

Correction: DC-PSENet: a novel scene text detection method integrating double ResNet-based and changed channels recursive feature pyramid

A novel ensemble deep network framework for scene text recognition

In recent years, scene text recognition (STR) has always been considered a sequence-to-sequence problem. Attention-based techniques have a greater potential for context-semantic modelling, but they tend to overfit inadequate training data. STR is one of the most important and difficult challenges in image-based sequence recognition. A novel framework ensemble deep network (EDN) is proposed, EDN comprises customized convolutional neural network (CNN), and deep autoencoder. Customized CNN is designed by introducing the optimal spatial transformation module for optimizing the input of irregular text to read for same size. Further, deep autoencoder is introduced with effective attention mechanism utilizing the inherent features. The proposed ensemble deep network-proposed system (EDN-PS) approach outperforms the existing state-of-art techniques for both irregular and regular scene-texts and upon further simulations, the proposed model generates better results for IIIT5K, ICDAR-13, ICDAR-15, and CUTE dataset in comparison with the existing system hence our proposed EDN-PS model outperforms the existing state-of-art methods.

Virtual home staging and relighting from a single panorama under natural illumination

Virtual staging technique can digitally showcase a variety of real-world scenes. However, relighting indoor scenes from a single image is challenging due to unknown scene geometry, material properties, and outdoor spatially-varying lighting. In this study, we use the High Dynamic Range (HDR) technique to capture an indoor panorama and its paired outdoor hemispherical photograph, and we develop a novel inverse rendering approach for scene relighting and editing. Our method consists of four key components: (1) panoramic furniture detection and removal, (2) automatic floor layout design, (3) global rendering with scene geometry, new furniture objects, and the real-time outdoor photograph, and (4) virtual staging with new camera position, outdoor illumination, scene texture, and electrical light. The results demonstrate that a single indoor panorama can be used to generate high-quality virtual scenes under new environmental conditions. Additionally, we contribute a new calibrated HDR (Cali-HDR) dataset that consists of 137 paired indoor and outdoor photographs. The animation for virtual rendered scenes is available here.

Box2Mask: Box-Supervised Instance Segmentation via Level-Set Evolution.

In contrast to fully supervised methods using pixel-wise mask labels, box-supervised instance segmentation takes advantage of simple box annotations, which has recently attracted increasing research attention. This paper presents a novel single-shot instance segmentation approach, namely Box2Mask, which integrates the classical level-set evolution model into deep neural network learning to achieve accurate mask prediction with only bounding box supervision. Specifically, both the input image and its deep features are employed to evolve the level-set curves implicitly, and a local consistency module based on a pixel affinity kernel is used to mine the local context and spatial relations. Two types of single-stage frameworks, i.e., CNN-based and transformer-based frameworks, are developed to empower the level-set evolution for box-supervised instance segmentation, and each framework consists of three essential components: instance-aware decoder, box-level matching assignment and level-set evolution. By minimizing the level-set energy function, the mask map of each instance can be iteratively optimized within its bounding box annotation. The experimental results on five challenging testbeds, covering general scenes, remote sensing, medical and scene text images, demonstrate the outstanding performance of our proposed Box2Mask approach for box-supervised instance segmentation. In particular, with the Swin-Transformer large backbone, our Box2Mask obtains 42.4% mask AP on COCO, which is on par with the recently developed fully mask-supervised methods.

Scene Text Image Superresolution Through Multiscale Interaction of Structural and Semantic Priors

Scene Text Image Superresolution Through Multiscale Interaction of Structural and Semantic Priors

GLaLT: Global-Local Attention-Augmented Light Transformer for Scene Text Recognition.

Recent years have witnessed the growing popularity of connectionist temporal classification (CTC) and attention mechanism in scene text recognition (STR). CTC-based methods consume less time with few computational burdens, while they are not as effective as attention-based methods. To retain computational efficiency and effectiveness, we propose the global-local attention-augmented light Transformer (GLaLT), which adopts a Transformer-based encoder-decoder structure to orchestrate CTC and attention mechanism. The encoder integrates the self-attention module with the convolution module to augment the attention, where the self-attention module pays more attention to capturing long-term global dependencies and the convolution module focuses on local context modeling. The decoder consists of two parallel modules: one is the Transformer-decoder-based attention module and the other is the CTC module. The first one is removed in the testing phase and can guide the second one to extract robust features in the training phase. Extensive experiments on standard benchmarks demonstrate that GLaLT achieves state-of-the-art performance for both regular and irregular STR. In terms of tradeoffs, the proposed GLaLT is at or near the frontiers for maximizing speed, accuracy, and computational efficiency at the same time.

(HTBNet)Arbitrary Shape Scene Text Detection with Binarization of Hyperbolic Tangent and Cross-Entropy.

The existing segmentation-based scene text detection methods mostly need complicated post-processing, and the post-processing operation is separated from the training process, which greatly reduces the detection performance. The previous method, DBNet, successfully simplified post-processing and integrated post-processing into a segmentation network. However, the training process of the model took a long time for 1200 epochs and the sensitivity to texts of various scales was lacking, leading to some text instances being missed. Considering the above two problems, we design the text detection Network with Binarization of Hyperbolic Tangent (HTBNet). First of all, we propose the Binarization of Hyperbolic Tangent (HTB), optimized along with which the segmentation network can expedite the initial convergent speed by reducing the number of epochs from 1200 to 600. Because features of different channels in the same scale feature map focus on the information of different regions in the image, to better represent the important features of all objects in the image, we devise the Multi-Scale Channel Attention (MSCA). Meanwhile, considering that multi-scale objects in the image cannot be simultaneously detected, we propose a novel module named Fused Module with Channel and Spatial (FMCS), which can fuse the multi-scale feature maps from channel and spatial dimensions. Finally, we adopt cross-entropy as the loss function, which measures the difference between predicted values and ground truths. The experimental results show that HTBNet, compared with lightweight models, has achieved competitive performance and speed on Total-Text (F-measure:86.0%, FPS:30) and MSRA-TD500 (F-measure:87.5%, FPS:30).

A Multi-Scale Natural Scene Text Detection Method Based on Attention Feature Extraction and Cascade Feature Fusion.

Scene text detection is an important research field in computer vision, playing a crucial role in various application scenarios. However, existing scene text detection methods often fail to achieve satisfactory results when faced with text instances of different sizes, shapes, and complex backgrounds. To address the challenge of detecting diverse texts in natural scenes, this paper proposes a multi-scale natural scene text detection method based on attention feature extraction and cascaded feature fusion. This method combines global and local attention through an improved attention feature fusion module (DSAF) to capture text features of different scales, enhancing the network's perception of text regions and improving its feature extraction capabilities. Simultaneously, an improved cascaded feature fusion module (PFFM) is used to fully integrate the extracted feature maps, expanding the receptive field of features and enriching the expressive ability of the feature maps. Finally, to address the cascaded feature maps, a lightweight subspace attention module (SAM) is introduced to partition the concatenated feature maps into several sub-space feature maps, facilitating spatial information interaction among features of different scales. In this paper, comparative experiments are conducted on the ICDAR2015, Total-Text, and MSRA-TD500 datasets, and comparisons are made with some existing scene text detection methods. The results show that the proposed method achieves good performance in terms of accuracy, recall, and F-score, thus verifying its effectiveness and practicality.

Visual and semantic guided scene text retrieval

Visual and semantic guided scene text retrieval

SterAF: A Scene Text Recognizer with Appearance-Flow Rectification

As the demand for recognizing irregular text in natural scenes increases, people are increasingly realizing the value of such applications, such as license plate recognition systems, image search, handwriting recognition, and autonomous driving, which are profoundly changing our lives in the field of text recognition. Recent studies have shown that the recognition of curved text and perspective text has become an important challenge in the field of text recognition, and the correction of curved text is a key step to achieve accurate recognition. However, current methods use strained text image correction methods, resulting in poor recognition accuracy when recognizing curved text. Therefore, we propose an end-to-end framework called Scene Text Recognizer with Appearance-Flow rectification (SterAF), which includes a correction network and a recognition network. Specifically, the framework’s steps are as follows: first, the input text image is deformed through an appearance flow-based correction network to adaptively warp the text image, to prevent irregular and unnatural deformations of the text image. Second, a sequence-to-sequence recognition network predicts the sequence of characters in the corrected text image to accurately recognize the text in the image. Through subjective and objective experiments, our SterAF model has shown excellent performance in both qualitative and quantitative experiments.

Deep Learning Techniques for Detecting and Segmenting Text in Natural Scene Images: Review

Text detection and segmentation in natural scene images is an active research problem in computer vision and document analysis. Unlike scanned documents, scene text exhibits significant diversity in appearance, orientation, scale, font, and lighting conditions. In this review, survey the current state-of-the-art in techniques and methodologies aimed at detecting and segmenting text regions from images of natural scenes are presented. Both traditional approaches using hand-crafted features as well as modern data-driven deep learning methods will be discussed. The review will analyze common datasets, evaluation protocols and metrics used for benchmarking. Limitations of existing methods and open challenges in handling multilingual text, curved text, and efficiency will be highlighted. Promising future directions towards robust and generalizable scene text extraction systems will be identified. In summary, the review will provide a comprehensive overview of the advances, remaining challenges and future opportunities in developing automated systems for detecting and segmenting text in unconstrained natural images.

Meta-learning integrated with Signed Distance Functions to Enhance Scene Text Segmentation

Meta-learning integrated with Signed Distance Functions to Enhance Scene Text Segmentation

Sign Tone: Two Way Sign Language Recognition and Multilingual Interpreter System Using Deep Learning

In this paper, a comparative experimental assessment of computer vision-based methods for sign language reclog- nation is conducted. By implementing the most recent deep neural network methods in this field, a thorough evaluation on multiple publicly available datasets is performed. The aim of the present study is to provide insights on sign language recognition, focusing on mapping non-segmented video streams to glosses. For this task, two new sequence training criteria, known from the fields of speech and scene text recognition, are introduced. Furthermore, a plethora of pretraining schemes is thoroughly discussed. Finally, a new RGB+D dataset for the Greek sign language is created. To the best of our knowledge, this is the first sign language dataset where three annotation levels are provided (individual gloss, sentence and spoken language) for the same set of video captures

Towards reduced-complexity scene text recognition (RCSTR) through a novel salient feature selection

Towards reduced-complexity scene text recognition (RCSTR) through a novel salient feature selection