Text Detection Tasks Research Articles

DPNet: Scene text detection based on dual perspective CNN-transformer.

With the continuous advancement of deep learning, research in scene text detection has evolved significantly. However, complex backgrounds and various text forms complicate the task of detecting text from images. CNN is a deep learning algorithm that automatically extracts features through convolution operation. In the task of scene text detection, it can capture local text features in images, but it lacks global attributes. In recent years, inspired by the application of transformers in the field of computer vision, it can capture the global information of images and describe them intuitively. Therefore, this paper proposes scene text detection based on dual perspective CNN-transformer. The channel enhanced self-attention module (CESAM) and spatial enhanced self-attention module (SESAM) proposed in this paper are integrated into the traditional ResNet backbone network. This integration effectively facilitates the learning of global contextual information and positional relationships of text, thereby alleviating the challenge of detecting small target text. Furthermore, this paper introduces a feature decoder designed to refine the effective text information within the feature map and enhance the perception of detailed information. Experiments show that the method proposed in this paper significantly improves the robustness of the model for different types of text detection. Compared to the baseline, it achieves performance improvements of 2.51% (83.81 vs. 81.3) on the Total-Text dataset, 1.87% (86.07 vs. 84.2) on the ICDAR 2015 dataset, and 3.63% (86.72 vs. 83.09) on the MSRA-TD500 dataset, while also demonstrating better visual effects.

Turning a CLIP Model Into a Scene Text Spotter.

We exploit the potential of the large-scale Contrastive Language-Image Pretraining (CLIP) model to enhance scene text detection and spotting tasks, transforming it into a robust backbone, FastTCM-CR50. This backbone utilizes visual prompt learning and cross-attention in CLIP to extract image and text-based prior knowledge. Using predefined and learnable prompts, FastTCM-CR50 introduces an instance-language matching process to enhance the synergy between image and text embeddings, thereby refining text regions. Our Bimodal Similarity Matching (BSM) module facilitates dynamic language prompt generation, enabling offline computations and improving performance. FastTCM-CR50 offers several advantages: 1) It can enhance existing text detectors and spotters, improving performance by an average of 1.6% and 1.5%, respectively. 2) It outperforms the previous TCM-CR50 backbone, yielding an average improvement of 0.2% and 0.55% in text detection and spotting tasks, along with a 47.1% increase in inference speed. 3) It showcases robust few-shot training capabilities. Utilizing only 10% of the supervised data, FastTCM-CR50 improves performance by an average of 26.5% and 4.7% for text detection and spotting tasks, respectively. 4) It consistently enhances performance on out-of-distribution text detection and spotting datasets, particularly the NightTime-ArT subset from ICDAR2019-ArT and the DOTA dataset for oriented object detection.

DPGS: Cross-cooperation guided dynamic points generation for scene text spotting

End-to-end text spotting aims to combine scene text detection and recognition into a unified framework. Dealing with the relationship between the two sub-tasks plays a pivotal role in designing effective spotters. While polygon or segmentation-based methods eliminate heuristic post-processing, they still face challenges such as background noise and high computational burden. In this study, we introduce DPGS, a coarse-to-fine learning framework that lets Dynamic Points Generation for text Spotting. DPGS simultaneously learns character representations for both detection and recognition tasks. Specifically, for each text instance, we represent the character sequence as ordered points and model them with learnable point queries. This approach progressively selects appropriate key points covering character and leverages group attention to associate similar information from different positions, improving detection accuracy. After passing through a single decoder, the point queries encode text semantics and locations, facilitating decoding to central line, boundary, script, and confidence of text through simple prediction heads. Additionally, we introduce an adaptive cooperative criterion to combine more useful feature knowledge, enhancing training efficiency. Extensive experiments show the superiority of our DPGS when handling scene text detection and recognition tasks. Compared to their respective top-1 methods, DPGS has significantly improved the average recognition accuracy by 3.7%, 1.9%, and 0.7% on the Total-Text, ICDAR15, and CTW1500 datasets, respectively.

A Historical Handwritten French Manuscripts Text Detection Method in Full Pages

Historical handwritten manuscripts pose challenges to automated recognition techniques due to their unique handwriting styles and cultural backgrounds. In order to solve the problems of complex text word misdetection, omission, and insufficient detection of wide-pitch curved text, this study proposes a high-precision text detection method based on improved YOLOv8s. Firstly, the Swin Transformer is used to replace C2f at the end of the backbone network to solve the shortcomings of fine-grained information loss and insufficient learning features in text word detection. Secondly, the Dysample (Dynamic Upsampling Operator) method is used to retain more detailed features of the target and overcome the shortcomings of information loss in traditional upsampling to realize the text detection task for dense targets. Then, the LSK (Large Selective Kernel) module is added to the detection head to dynamically adjust the feature extraction receptive field, which solves the cases of extreme aspect ratio words, unfocused small text, and complex shape text in text detection. Finally, in order to overcome the CIOU (Complete Intersection Over Union) loss in target box regression with unclear aspect ratio, insensitive to size change, and insufficient correlation between target coordinates, Gaussian Wasserstein Distance (GWD) is introduced to modify the regression loss to measure the similarity between the two bounding boxes in order to obtain high-quality bounding boxes. Compared with the State-of-the-Art methods, the proposed method achieves optimal performance in text detection, with the precision and mAP@0.5 reaching 86.3% and 82.4%, which are 8.1% and 6.7% higher than the original method, respectively. The advancement of each module is verified by ablation experiments. The experimental results show that the method proposed in this study can effectively realize complex text detection and provide a powerful technical means for historical manuscript reproduction.

A Method for Analyzing the Operating Data of Electric Energy Meters Based on Data Mining Analysis

Aiming at the problem of error estimation of smart meters in distribution network, a method of error estimation of smart meters based on particle swarm optimization convolutional neural network is proposed. This method establishes an intelligent energy meter error estimation model through data collection, data prediction, and preprocessing. To address the convergence issue in training, the interlayer distribution of weights is adjusted to improve training quality. This method fully utilizes template calibration information to transform indicator detection under complex conditions into simple and effective isometric segmentation, transforming label recognition from complex text detection and recognition tasks to simple and efficient binary detection tasks, with better robustness. The effectiveness and high robustness of the proposed method have been demonstrated through experimental verification.

Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?

Large language models (LLMs) such as ChatGPT are increasingly being used for various use cases, including text content generation at scale. Although detection methods for such AI-generated text exist already, we investigate ChatGPT's performance as a detector on such AI-generated text, inspired by works that use ChatGPT as a data labeler or annotator. We evaluate the zeroshot performance of ChatGPT in the task of human-written vs. AI-generated text detection, and perform experiments on publicly available datasets. We empirically investigate if ChatGPT is symmetrically effective in detecting AI-generated or human-written text. Our findings provide insight on how ChatGPT and similar LLMs may be leveraged in automated detection pipelines by simply focusing on solving a specific aspect of the problem and deriving the rest from that solution. All code and data is available at https://github.com/AmritaBh/ChatGPT-as-Detector.

UIT-MLReceipts: A Multilingual Benchmark for Detecting and Recognizing Key Information in Receipts

The 4.0 industrial evolution has paved the way for development potential and revolution in Vietnam. In this movement, digitization appears to be necessary to transform numerous traditional economic sectors. It will provide valuable digital data for many automation applications and decision-making processes. Particularly in the retail industry, data has long played a vital factor. Hence, digitizing documents such as receipts can help businesses in management and enterprise development. Nevertheless, the digital transformation process is still slow because of the shortage of cleaned datasets for this type of document. This paper introduces a new dataset named UIT-MLReceipts for extracting key information in receipts. The task includes two sub-tasks: Receipt Text Detection (RTD) and Receipt Text Recognition (RTR). We thoroughly evaluate current state-of-the-art Receipt Text Detection using Faster R-CNN, YOLOv3, YOLOF, and Faster R-CNN with Precise RoI-Pooling on our dataset. To evaluate the performance of Receipt Text Recognition, we experiment with two text recognition baselines: RobustScanner and SATRN. Experimental results indicate that the Faster R-CNN with Precise RoI-Pooling outperforms the competitors and achieves the best mean Average Precision (mAP) score at 51.6% in the Receipt Text Detection task. With the Receipt Text Recognition task, results show that SATRN performs better.

DSText V2: A comprehensive video text spotting dataset for dense and small text

Recently, video text detection, tracking, and recognition in natural scenes are becoming very popular in the computer vision community. However, most existing algorithms and benchmarks focus on common text cases (e.g.,normal size, density) and single scenario, while ignoring extreme video text challenges, i.e., dense and small text in various scenarios. In this paper, we establish a video text reading benchmark, named DSText V2, which focuses on Dense and Small text reading challenges in the video with various scenarios. Compared with the previous datasets, the proposed dataset mainly include three new challenges: (1) Dense video texts, a new challenge for video text spotters to track and read. (2) High-proportioned small texts, coupled with the blurriness and distortion in the video, will bring further challenges. (3) Various new scenarios, e.g., ‘Game’, ‘Sports’, etc. The proposed DSText V2 includes 140 video clips from 7 open scenarios, supporting three tasks, i.e., video text detection (Task 1), video text tracking (Task 2), and end-to-end video text spotting (Task 3).In this article, we describe detailed statistical information of the dataset, tasks, evaluation protocols, and the results summaries. Most importantly, a thorough investigation and analysis targeting three unique challenges derived from our dataset are provided, aiming to provide new insights. Moreover, we hope the benchmark will promise video text research in the community. DSText v2 is built upon DSText v1, which was previously introduced to organize the ICDAR 2023 competition for dense and small video text. The dataset website can be found at RRC or Zenodo.

CommuSpotter: Scene Text Spotting with Multi-Task Communication

Scene text spotting is a challenging multi-task modulation for locating and recognizing texts in complex scenes. Existing end-to-end text spotters generally adopt sequentially decoupled multi-tasks, consisting of text detection and text recognition modules. Although customized modules are designed to connect the tasks closely, there is no interaction among multiple tasks, resulting in compatible information loss for the overall text spotting. Moreover, the independent and sequential modulation is unidirectional, accumulating errors from early to later tasks. In this paper, we propose CommuSpotter, which enhances multi-task communication by explicitly and concurrently sharing compatible information in overall scene text spotting. To address task-specific inconsistencies, we propose a Conversation Mechanism (CM) to extract and exchange expertise in each specific task with others. Specifically, the detection task is rectified by the text recognition task to filter out duplicated results and false positives, while the text recognition task is corrected by the rectified text detection task to replenish missing characters and decrease non-text interruptions. Consequently, the communication compensates for interaction information and breaks the sequential pipeline of error propagation. In addition, we adopt text semantic segmentation in the text recognition task, which reduces the complex design of customized modules and corresponding extra annotations. Compared with state-of-the-art methods, experimental results show that our method achieves competitive results with computation efficiency.

A Non-Intrusive Automated Testing System for Internet of Vehicles App Based on Deep Learning

In the non-intrusive automated testing system for Internet of Vehicles (IoV) applications, automatic recognition of text and icons on vehicle central control screens is of paramount importance. However, the detection and recognition of content on vehicle central control screens are inherently complex. Additionally, during non-intrusive vehicle central control screen image testing, there is a deficiency of suitable datasets and detection methods. This deficiency renders information within vehicle application images difficult to be accurately extracted by the detection network. To address this problem, this study first constructs a dataset tailored for text detection and recognition on vehicle screens. This dataset encompasses a variety of vehicle central control images, enabling the generic text detection and recognition network to more effectively identify and interpret text within vehicle screens. Subsequently, this research proposes an enhanced Fully Convolutional Networks for Text Detection (FOTS) method for vehicle central control screen text detection and recognition. This method elevates the semantic expression capabilities of features by sharing vehicle central control screen text detection and recognition features. Furthermore, it improves multi-scale feature processing capabilities through the utilization of a feature transformation module. Validation through visual and quantitative experiments demonstrates that the proposed method can effectively accomplish text detection and recognition tasks on vehicle screens. This achievement bears significant implications for the field of automated testing in IoV applications.

A handwritten ancient text detector based on improved feature pyramid network

Text detection is the primary task for digitization of ancient books. Different from the common scene text detection tasks (ICDAR, TotalText, etc.), the texts in handwritten ancient documents are more densely distributed and generally small objects; at the same time, the layout structure is also more complex, with problems such as mixed arrangement of pictures and texts and high background noise, all of which pose challenges for detection. According to the characteristics of ancient book images, this paper proposes a new fusion structure based on Feature Pyramid Networks, and takes FCOS as the baseline model to form a new detector (named RFCOS). We enhance the detection capability for dense and small text instances by adding bottom-up fusion paths, cross-layer connections and weighted fusion. Meanwhile, the loss of high-level feature maps during fusion is reduced by new upsampling method and lateral connections. We verified the effectiveness of our RFCOS on the HWAD (Handwritten Ancient Books Dataset), a dataset containing samples in four languages - Yi, Chinese, Tibetan and Tangut, and verify the generalization of RFCOS on another public dataset MTHv2. The results show that RFCOS outperformed most of the existing text detectors in terms of precision, recall and F-measure.

Incorporating topic membership in review rating prediction from unstructured data: a gradient boosting approach

AbstractRating prediction is a crucial element of business analytics as it enables decision-makers to assess service performance based on expressive customer feedback. Enhancing rating score predictions and demand forecasting through incorporating performance features from verbatim text fields, particularly in service quality measurement and customer satisfaction modelling is a key objective in various areas of analytics. A range of methods has been identified in the literature for improving the predictability of customer feedback, including simple bag-of-words-based approaches and advanced supervised machine learning models, which are designed to work with response variables such as Likert-based rating scores. This paper presents a dynamic model that incorporates values from topic membership, an outcome variable from Latent Dirichlet Allocation, with sentiment analysis in an Extreme Gradient Boosting (XGBoost) model used for rating prediction. The results show that, by incorporating features from simple unsupervised machine learning approaches (LDA-based), an 86% prediction accuracy (AUC based) can be achieved on objective rating values. At the same time, a combination of polarity and single-topic membership can yield an even higher accuracy when compared with sentiment text detection tasks both at the document and sentence levels. This study carries significant practical implications since sentiment analysis tasks often require dictionary coverage and domain-specific adjustments depending on the task at hand. To further investigate this result, we used Shapley Additive Values to determine the additive predictability of topic membership values in combination with sentiment-based methods using a dataset of customer reviews from food delivery services.

CDText: Scene text detector based on context-aware deformable transformer

Scene text detection task aims to precisely locate text regions in natural scenes. However, the existing methods still face challenges in detecting arbitrary-shaped text, due to their limited feature representation capability. To alleviate this problem, we propose a scene text detector, i.e., CDText, based on structure of context-aware deformable transformer. Specifically, CDText firstly adopts different convolution kernel designs for feature extraction, which designs receptive fields with different size for multi-scale feature perception and fusion. Meanwhile, multi-head self-attention mechanism is used to strengthen the reasoning ability of CDText in a global sense, thus enhancing feature maps with abundant context information by extracting implicit relationship between multi-scale text features. Moreover, CDText designs a segmentation head to segment text instances of arbitrary shapes from rectangular detection boxes. Experiments show that CDText is superior to comparative methods in detection accuracy, achieving F-scores of 92.7, 81.9, and 82.9 on ICDAR2013, Total Text, and CTW-1500 datasets, respectively.

Efficient Neural Network for Text Recognition in Natural Scenes Based on End-to-End Multi-Scale Attention Mechanism

Text recognition in natural scenes has been a very challenging task in recent years, and rich text semantic information is of great significance for the understanding of a scene. However, text images in natural scenes often contain a lot of noise data, which leads to error detection. The problems of high error detection rate and low recognition accuracy have brought great challenges to the task of text recognition. To solve this problem, we propose a text recognition algorithm based on natural scenes. First, the task of text detection and recognition is completed in an end-to-end way in a framework, which can reduce the cumulative error prediction and calculation caused by cascading, and has higher real-time and faster speed. In addition, we integrate a multi-scale attention mechanism to obtain attention features of different scale feature maps. Finally, we use the efficient deep learning network (EE-ACNN), which combines a convolutional neural network (CNN) with an end-to-end algorithm and multi-scale attention to enrich the text features to be detected, expands its receptive field, produces good robustness to the effective natural text information, and improves the recognition performance. Through experiments on text data sets of natural scenes, the accuracy of this method reached 93.87%, which is nearly 0.96–1.02% higher than that of traditional methods, and which proves the feasibility of this method.

TextDC: Exploring Multidimensional Text Detection via a New Benchmark and Solution

Text detection has been significantly boosted by the development of deep neural networks but most existing methods focus on a single kind of text instance (i.e., overlaid text, layered text, scene text). In this paper, we expand the text detection task from a single dimension to multiple dimensions, thus providing multi-type text descriptions for the scene and content analysis of videos. Specifically, we establish a new task to detect and classify text instances simultaneously, termed TextDC. As far as we know, existing benchmarks cannot meet the requirements of the proposed task. To this end, we collect a large-scale text detection and classification dataset, named Text3C, which is annotated using multilingual labels, location information, and text categories. Together with the collected dataset, we introduce a multi-stage and strict evaluation metric, which penalizes detection approaches for missing text instances, false positive detection, inaccurate location boxes, and error text categories, developing a new benchmark for the proposed TextDC task. In addition, we extend several state-of-the-art detectors by modifying the prediction head to solve the new task. Then, a generalized text detection and classification framework is designed and formulated. Extensive experiments using the updated methods are conducted on the established benchmark to verify the solvability of the proposed task, the challenges of the dataset, and the effectiveness of the solution.

JMNET: Arbitrary-shaped scene text detection using multi-space perception

The arbitrary-shape scene text detection faces enormous challenges in accuracy and speed with increasing demand in the industrial sector. Although most studies have introduced segmentation and achieved remarkable performance in text detection tasks, researchers ignored the importance of detection efficiency. This paper proposes a new Joint Multi-Space Perception Network (JMNET) for efficient scene text detection to address this issue. Based on a lightweight feature extraction backbone, we put forward two novel modules, i.e., Scale Spatial Perception Module (SSPM) and Attention Spatial Perception Module (ASPM) to enhance the expression ability of text features with low computational complexity. Moreover, we propose an Unsupervised Embedding Spatial Perception Loss function (UnESP Loss) by introducing the Euclidean distance measurement between the embeddings to overcome the ambiguity of text instance boundary, such as a small line spacing. In this way, text embedding learning is not restricted by specific shapes, and detection robustness can be improved. Extensive experiments on four benchmarks, including the ICDAR2015, MSRA-TD500, CTW1500 and Totaltext, show that the proposed JMNET achieves competitive performance in terms of both accuracy and speed over the state-of-the-art methods. Particularly, our method can reach a remarkable F-measure of 85.5% at 52 FPS on Totaltext. Code is available at:https://github.com/sakura-910/JMNET.

CE-text: A context-Aware and embedded text detector in natural scene images

• A novel deep and context-aware CNN structure for accurate and fast text detection • Hierarchically channel wise attention scheme with channel wise and multilayer features • Adopts frequency-based deep compression method to build a lightweight text detector With the significant power of deep learning architectures, researchers have made much progress on effectiveness and efficiency of text detection in the past few years. However, due to the lack of consideration of unique characteristics of text components, directly applying deep learning models to perform text detection task is prone to result in low accuracy, especially producing false positive detection results. To ease this problem, we propose a lightweight and context-aware deep convolutional neural network (CNN) named as CE-Text, which appropriately encodes multi-level channel attention information to construct discriminative feature map for accurate and efficient text detection. To fit with low computation resource of embedded systems, we further transform CE-Text into a lighter version with a frequency based deep CNN compression method, which expands applicable scenarios of CE-Text into variant embedded systems. Experiments on several popular datasets show that CE-Text not only has achieved accurate text detection results in scene images, but also could run with fast performance in embedded systems.

Scene Uyghur Text Detection Based on Fine-Grained Feature Representation.

Scene text detection task aims to precisely localize text in natural environments. At present, the application scenarios of text detection topics have gradually shifted from plain document text to more complex natural scenarios. Objects with similar texture and text morphology in the complex background noise of natural scene images are prone to false recall and difficult to detect multi-scale texts, a multi-directional scene Uyghur text detection model based on fine-grained feature representation and spatial feature fusion is proposed, and feature extraction and feature fusion are improved to enhance the network’s ability to represent multi-scale features. In this method, the multiple groups of 3 × 3 convolutional feature groups that are connected like the hierarchical residual to build a residual network for feature extraction, which captures the feature details and increases the receptive field of the network to adapt to multi-scale text and long glued dimensional font detection and suppress false positives of text-like objects. Secondly, an adaptive multi-level feature map fusion strategy is adopted to overcome the inconsistency of information in multi-scale feature map fusion. The proposed model achieves 93.94% and 84.92% F-measure on the self-built Uyghur dataset and the ICDAR2015 dataset, respectively, which improves the accuracy of Uyghur text detection and suppresses false positives.

Overcoming Language Disparity in Online Content Classification with Multimodal Learning

Advances in Natural Language Processing (NLP) have revolutionized the way researchers and practitioners address crucial societal problems. Large language models are now the standard to develop state-of-the-art solutions for text detection and classification tasks. However, the development of advanced computational techniques and resources is disproportionately focused on the English language, sidelining a majority of the languages spoken globally. While existing research has developed better multilingual and monolingual language models to bridge this language disparity between English and non-English languages, we explore the promise of incorporating the information contained in images via multimodal machine learning. Our comparative analyses on three detection tasks focusing on crisis information, fake news, and emotion recognition, as well as five high-resource non-English languages, demonstrate that: (a) detection frameworks based on pre-trained large language models like BERT and multilingual-BERT systematically perform better on the English language compared against non-English languages, and (b) including images via multimodal learning bridges this performance gap. We situate our findings with respect to existing work on the pitfalls of large language models, and discuss their theoretical and practical implications.

Text detection and recognition based on a lensless imaging system.

Lensless cameras are characterized by several advantages (e.g., miniaturization, ease of manufacture, and low cost) as compared with conventional cameras. However, they have not been extensively employed due to their poor image clarity and low image resolution, especially for tasks that have high requirements on image quality and details such as text detection and text recognition. To address the problem, a framework of deep-learning-based pipeline structure was built to recognize text with three steps from raw data captured by employing lensless cameras. This pipeline structure consisted of the lensless imaging model U-Net, the text detection model connectionist text proposal network (CTPN), and the text recognition model convolutional recurrent neural network (CRNN). Compared with the method focusing only on image reconstruction, U-Net in the pipeline was able to supplement the imaging details by enhancing factors related to character categories in the reconstruction process, so the textual information can be more effectively detected and recognized by CTPN and CRNN with fewer artifacts and high-clarity reconstructed lensless images. By performing experiments on datasets of different complexities, the applicability to text detection and recognition on lensless cameras was verified. This study reasonably demonstrates text detection and recognition tasks in the lensless camera system, and develops a basic method for novel applications.