Natural Scene Images Research Articles

Multi‐lingual text detection and identification using agile convolutional neural network

AbstractMulti‐lingual scene text detection and identification is a challenging task in today's world due to the prevalence of many digitized multi‐lingual documents, images, and videos. A valuable method for detecting multi‐lingual text from natural scene images is proposed which uses the convolutional neural network, namely, You Only Look Once (YOLOv3) as the backbone. The proposed system is more agile than YOLOv3 with the introduction of atrous separable convolution (ASC). The multi‐scale prediction in YOLOv3 emphasizes the integration of global features of multi‐scale convolutional layers while it overlooks the blend of the multi‐scale local region features on the same convolutional layer. To overcome this, ASC is applied to efficiently compute dense local region feature maps, thereby reducing computation complexity substantially. Complete IoU loss, which is an accumulation of overlap area, distance, and aspect ratio, is introduced for enhanced accuracy in bounding box regression, wherein IoU designates the measure of overlap between the predicted and the ground truth bounding boxes. The experimental results show that the proposed system is efficacious in detecting multi‐lingual as well as English text from natural scene images.

Subtask Attention Based Object Detection in Remote Sensing Images

Object detection in remote sensing images (RSIs) is one of the basic tasks in the field of remote sensing image automatic interpretation. In recent years, the deep object detection frameworks of natural scene images (NSIs) have been introduced into object detection on RSIs, and the detection performance has improved significantly because of the powerful feature representation. However, there are still many challenges concerning the particularities of remote sensing objects. One of the main challenges is the missed detection of small objects which have less than five percent of the pixels of the big objects. Generally, the existing algorithms choose to deal with this problem by multi-scale feature fusion based on a feature pyramid. However, the benefits of this strategy are limited, considering that the location of small objects in the feature map will disappear when the detection task is processed at the end of the network. In this study, we propose a subtask attention network (StAN), which handles the detection task directly on the shallow layer of the network. First, StAN contains one shared feature branch and two subtask attention branches of a semantic auxiliary subtask and a detection subtask based on the multi-task attention network (MTAN). Second, the detection branch uses only low-level features considering small objects. Third, the attention map guidance mechanism is put forward to optimize the network for keeping the identification ability. Fourth, the multi-dimensional sampling module (MdS), global multi-view channel weights (GMulW) and target-guided pixel attention (TPA) are designed for further improvement of the detection accuracy in complex scenes. The experimental results on the NWPU VHR-10 dataset and DOTA dataset demonstrated that the proposed algorithm achieved the SOTA performance, and the missed detection of small objects decreased. On the other hand, ablation experiments also proved the effects of MdS, GMulW and TPA.

Visual memorability in the absence of semantic content

What determines how well people remember images? Most past research has explored properties of the people doing the remembering — such as their age, emotional state, or individual capacity. However, recent work has also characterized memorability — the likelihood of an image being remembered across observers. But what makes some images more memorable than others? Part of the answer must surely involve the meanings of the images, but here we ask whether this is the entire story: is there also purely visual memorability, driven not by semantic content but by perceptual features per se? We isolated visual memorability in an especially direct manner — by eliminating semantic content while retaining many visual properties. We did so by transforming a set of natural scene images using phase scrambling, and then testing memorability for both intact and scrambled images in independent samples. Across several experiments, observers saw sequences of images and responded anytime they saw a repeated image. We found reliable purely visual memorability at the temporal scales of both short-term memory (2–15 s) and longer-term memory (several minutes), and this could not be explained by the extent to which people could generate semantic labels for some scrambled images. Collectively, these results suggest that the memorability of images is a function not only of what they mean, but also of how they look in the first place.

Naturalness and aesthetics of colors – Preference for color compositions perceived as natural

What makes a colored image, e.g. an abstract painting or a landscape, look pleasing? We hypothesized that a preference for complex color compositions, such as paintings and images of natural scenes, might be related to how natural the colors are perceived. We tested this possibility with two experiments in which the degree of naturalness of images was manipulated by rotating their color gamut rigidly in the color space CIELAB. This changed just the hue composition, but preserved saturation and lightness. In the first experiment we obtained individual scaling curves for perceived naturalness and for preference as a function of the angle of gamut rotation for a small set of images. The naturalness and preference scaling curves were found to be largely similar and their maxima were close to the original image. In the second experiment, we tested whether this effect generalized to a larger set of images. We used a simultaneous 5AFC procedure where in each trial participants had to select the most natural or the most preferred image from five different rotations of the color gamut. The results confirmed the first experiment and showed that, in general, the images perceived as the more natural tend to be the ones that are preferred. Together these results show that perceived naturalness and preference are indeed perceptually closely related and may be driven by related mechanisms.

COCO-Search18 fixation dataset for predicting goal-directed attention control

Attention control is a basic behavioral process that has been studied for decades. The currently best models of attention control are deep networks trained on free-viewing behavior to predict bottom-up attention control – saliency. We introduce COCO-Search18, the first dataset of laboratory-quality goal-directed behavior large enough to train deep-network models. We collected eye-movement behavior from 10 people searching for each of 18 target-object categories in 6202 natural-scene images, yielding sim 300,000 search fixations. We thoroughly characterize COCO-Search18, and benchmark it using three machine-learning methods: a ResNet50 object detector, a ResNet50 trained on fixation-density maps, and an inverse-reinforcement-learning model trained on behavioral search scanpaths. Models were also trained/tested on images transformed to approximate a foveated retina, a fundamental biological constraint. These models, each having a different reliance on behavioral training, collectively comprise the new state-of-the-art in predicting goal-directed search fixations. Our expectation is that future work using COCO-Search18 will far surpass these initial efforts, finding applications in domains ranging from human-computer interactive systems that can anticipate a person’s intent and render assistance to the potentially early identification of attention-related clinical disorders (ADHD, PTSD, phobia) based on deviation from neurotypical fixation behavior.

Text detection and localization in scene images: a broad review

Nowadays, text detection and localization have gained much popularity in the field of text analysis systems as they pave the way for the number of real-time based applications like mobile transliteration technologies, assistive methods for visually impaired persons, etc. Text detection and localization techniques are used to find the position of text area in the image.This paper intends to present a broad review in this field as five-fold: (1) comparison of document images with scene images and applications of natural scene images, (2) significant and up-to-date traditional machine learning and deep learning-based approaches for the text detection and localization for different languages, (3) various publicly available benchmarked datasets, (4) comparative analysis for other benchmarked datasets and, (5) related challenges and future scope on the field. The paper summarises some of the potential ways in this field, which can serve as a useful reference for the researchers for future exploration of the area.

Multi-granularity Deep Local Representations for Irregular Scene Text Recognition

Recognizing irregular text from natural scene images is challenging due to the unconstrained appearance of text, such as curvature, orientation, and distortion. Recent recognition networks regard this task as a text sequence labeling problem and most networks capture the sequence only from a single-granularity visual representation, which to some extent limits the performance of recognition. In this article, we propose a hierarchical attention network to capture multi-granularity deep local representations for recognizing irregular scene text. It consists of several hierarchical attention blocks, and each block contains a Local Visual Representation Module (LVRM) and a Decoder Module (DM). Based on the hierarchical attention network, we propose a scene text recognition network. The extensive experiments show that our proposed network achieves the state-of-the-art performance on several benchmark datasets including IIIT-5K, SVT, CUTE, SVT-Perspective, and ICDAR datasets under shorter training time.

Towards Accurate Scene Text Detection with Bidirectional Feature Pyramid Network

Scene text detection, this task of detecting text from real images, is a hot research topic in the machine vision community. Most of the current research is based on an anchor box. These methods are complex in model design and time-consuming to train. In this paper, we propose a new Fully Convolutional One-Stage Object Detection (FCOS)-based text detection method that can robustly detect multioriented and multilingual text from natural scene images in a per pixel prediction approach. Our proposed text detector employs an anchor-free approach, unlike state-of-the-art text detectors that do not rely on a predefined anchor box. In order to enhance the feature representation ability of FCOS for text detection tasks, we apply the Bidirectional Feature Pyramid Network (BiFPN) as the backbone network, enhancing the model learning capacity and increasing the receptive field. We demonstrate the superior performance of our method on multioriented (ICDAR-2015, ICDAR-2017 MLT) and horizontal (ICDAR-2013) text detection benchmark tasks. Moreover, our method has an f-measure of 88.65 and 86.32 for the benchmark datasets ICDAR 2013 and ICDAR 2015, respectively, and 80.75 for the ICDAR-2017 MLT dataset.

Text detection and script identification in natural scene images using deep learning

The detection of text in an image and identification of its language are important tasks in optical character recognition. Such tasks are challenging, particularly in natural scene images. Previous studies have been conducted with a focus on convolutional neural networks for script identification. In other studies, fully convolutional networks (FCNs) have been used for model enhancement and not as classifiers. In this study, we use FCNs for both model enhancement and classification. The proposed methodology improves the Efficient and Accurate Scene Text Detector by adding new FCN branches for script identification. Moreover, whereas most end-to-end (e2e) methods train the text detection and script identification models separately, we propose two e2e methods for jointly training the models, namely, multi-channel mask (MCM) and multi-channel segmentation (MCS). The results show that the performance of an MCM is similar to that of other state-of-the-art methods, whereas MCS outperforms existing methods with recall values of 54.34% and 81.13%, when using the ICDAR MLT 2017 and MLe2e datasets, respectively.

Character-based handwritten text transcription with attention networks

The paper approaches the task of handwritten text recognition (HTR) with attentional encoder–decoder networks trained on sequences of characters, rather than words. We experiment on lines of text from popular handwriting datasets and compare different activation functions for the attention mechanism used for aligning image pixels and target characters. We find that softmax attention focuses heavily on individual characters, while sigmoid attention focuses on multiple characters at each step of the decoding. When the sequence alignment is one-to-one, softmax attention is able to learn a more precise alignment at each step of the decoding, whereas the alignment generated by sigmoid attention is much less precise. When a linear function is used to obtain attention weights, the model predicts a character by looking at the entire sequence of characters and performs poorly because it lacks a precise alignment between the source and target. Future research may explore HTR in natural scene images, since the model is capable of transcribing handwritten text without the need for producing segmentations or bounding boxes of text in images.

Sparse elastic net multi-label rank support vector machine with pinball loss and its applications

Multi-label rank support vector machine (RankSVM) is an effective technique to deal with multi-label classification problems, which has been widely used in various fields. However, it is sensitive to noise points and cannot delete redundant features for high dimensional problems. Therefore, to address the above two limitations, a sparse elastic net multi-label RankSVM with pinball loss (pin-ENR) is first proposed in this paper. On the one hand, pinball loss is employed to enhance the robustness. On the other hand, it adopts the sparse elastic net regularization, so that it can do variable selection. However, it still has challenges for large-scale problems with a huge number of features, samples, and labels. Therefore, motivated by the sparsity of pin-ENR, a safe simultaneous feature and label-pair elimination rule is further constructed for accelerating pin-ENR, which is termed as FLER-pin-ENR. Its main idea is to delete a large number of inactive features and label-pairs simultaneously before training without sacrificing accuracy. Numerical experiments on four synthetic and seven benchmark datasets demonstrate the feasibility and validity. Moreover, we apply our FLER-pin-ENR to the diagnosis of diabetes complications and the natural scene image classification problems, which further verifies the practicability of our proposed method.

Rule-based perspective rectification for Chinese text in natural scene images

Recognizing text with large perspective deformation is challenging, especially when the orientation of the text is arbitrary. A rule-based method is proposed in this paper to recover the fronto-parallel image for Chinese text in natural scenes. It relies on the fact that there are many horizontal and vertical strokes in Chinese text and they are accurately or approximately perpendicular to each other in non-italic type. A homography is utilized to describe the relationship between the scene text and its image. Horizontal and vertical strokes are extracted and selected to construct perspective quadrilaterals and the expected rectangles for calculating the homography. Experiments show that the proposed method could rectify the perspective deformation of Chinese text in any orientation effectively and it outperforms the state-of-the-art perspective rectification method based on vanishing point detection.

What is visible across the visual field?

It is sometimes claimed that because the resolution and sensitivity of visual perception are better in the fovea than in the periphery, peripheral vision cannot support the same kinds of colour and sharpness percepts as foveal vision. The fact that a scene nevertheless seems colourful and sharp throughout the visual field then poses a puzzle. In this study, I use a detailed model of human spatial vision to estimate the visibility of certain properties of natural scenes, including aspects of colourfulness, sharpness, and blurriness, across the visual field. The model is constructed to reproduce basic aspects of human contrast and colour sensitivity over a range of retinal eccentricities. I apply the model to colourful, complex natural scene images, and estimate the degree to which colour and edge information are present in the model’s representation of the scenes. I find that, aside from the intrinsic drift in the spatial scale of the representation, there are not large qualitative differences between foveal and peripheral representations of ‘colourfulness’ and ‘sharpness’.

Scale-Invariant Multidirectional License Plate Detection with the Network Combining Indirect and Direct Branches.

As the license plate is multiscale and multidirectional in the natural scene image, its detection is challenging in many applications. In this work, a novel network that combines indirect and direct branches is proposed for license plate detection in the wild. The indirect detection branch performs small-sized vehicle plate detection with high precision in a coarse-to-fine scheme using vehicle–plate relationships. The direct detection branch detects the license plate directly in the input image, reducing false negatives in the indirect detection branch due to the miss of vehicles’ detection. We propose a universal multidirectional license plate refinement method by localizing the four corners of the license plate. Finally, we construct an end-to-end trainable network for license plate detection by combining these two branches via post-processing operations. The network can effectively detect the small-sized license plate and localize the multidirectional license plate in real applications. To our knowledge, the proposed method is the first one that combines indirect and direct methods into an end-to-end network for license plate detection. Extensive experiments verify that our method outperforms the indirect methods and direct methods significantly.

Attractiveness in the Eyes: A Possibility of Positive Loop between Transient Pupil Constriction and Facial Attraction.

Contrary to the long-held belief of a close linkage between pupil dilation and attractiveness, we found an early and transient pupil constriction response when participants viewed an attractive face (and the effect of luminance/contrast was controlled). While human participants were making an attractiveness judgment on faces, their pupil constricted more for the more attractive (as-to-be-rated) faces. Further experiments showed that the effect of pupil constriction to attractiveness judgment extended to intrinsically esthetic visual objects such as natural scene images (as well as faces) but not to line-drawing geometric figures. When participants were asked to judge the roundness of faces, pupil constriction still correlated with their attractiveness but not the roundness rating score, indicating the automaticity of the pupil constriction to attractiveness. When pupillary responses were manipulated implicitly by relative background luminance changes (from the prestimulus screen), the facial attractiveness ratings were in accordance with the amount of pupil constriction, which could not be explained solely by simultaneous or sequential luminance contrast. The overall results suggest that pupil constriction not only reflects but, as a part of self-monitoring and attribution mechanisms, also possibly contributes to facial attractiveness implicitly.

Convolutional Neural Network for the Semantic Segmentation of Remote Sensing Images

In recent years, the success of deep learning in natural scene image processing boosted its application in the analysis of remote sensing images. In this paper, we applied Convolutional Neural Networks (CNN) on the semantic segmentation of remote sensing images. We improve the Encoder- Decoder CNN structure SegNet with index pooling and U-net to make them suitable for multi-targets semantic segmentation of remote sensing images. The results show that these two models have their own advantages and disadvantages on the segmentation of different objects. In addition, we propose an integrated algorithm that integrates these two models. Experimental results show that the presented integrated algorithm can exploite the advantages of both the models for multi-target segmentation and achieve a better segmentation compared to these two models.

Automatic Text Segmentation and Recognition in Natural Scene Images Using Msocr

Introduction: Segmentation and recognition of text from the scene image are a challenging task due to blurred, low-resolution and small sized image. Materials and Methods: Innovative methods have been proposed to address this problem and to recognize the text from the natural scene image. The acquired image is pre-processed by the YUV channel conversion technique and the Y channel image is converted to a gray scale image. Connected Component Based Text Segmentation Algorithm (CCBTSA) and MSER methods are used for segmentation and recognition of text using Optical Character Recognition (OCR). GLCM and FOS features are extracted from the segmented region. The Template matching algorithm is used to extract the text character from the bounding box of the segmented image. Results and Conclusion: Trained SVM classifier is used to classify the image containing text and non-text region. Performances are analyzed based on the recall rate, precision, accuracy, and Fmeasure. From the experimental results, the accuracy of the proposed classifier was obtained as 95%.

Estimation of tea leaf blight severity in natural scene images

Tea leaf blight (TLB) is a common tea disease seriously affecting the quality and yield of tea. An accurate estimation of TLB severity can be used to guide tea farmers to reasonably spray pesticides. This study proposes an estimation method for TLB severity in natural scene images and consists of four main steps: segmentation of the diseased leaves, area fitting of the diseased leaves, segmentation of the disease spots, and estimation of disease severity. Target leaves with TLB in the tea images are segmented by combining the U-Net network and fully connected conditional random field to reduce the influence of complex background. An ellipse restoration method is proposed to generate an elliptic mask to fit the full size of the occluded or damaged TLB leaves. The disease spot regions are segmented from the TLB leaves by a support vector machine classifier to calculate the Initial Disease Severity (IDS) index. The IDS index, color features, and texture features of the TLB leaves are inputted into the metric learning model to finally estimated disease severity. Experimental results show that the proposed method has higher estimation accuracy and stronger robustness against occluded and damaged TLB leaves compared with conventional convolution neural network methods and classical machine learning techniques.

Robust license plate detection and recognition with automatic rectification

We propose a robust license plate detection and recognition (LPDR) framework with automatic rectification. We explore the YOLOv2 object detector based on deep learning and train it to detect license plates (LPs) effectively. The LPs in natural scene images tend to be tilted and distorted because of the shooting angle or the geometric deformation of LPs. To solve the problem in which the LP tilt and distortion affect recognition accuracy, we introduce a spatial transformation network with thin-plate-spline transformation and propose a neural network called inverse compositional spatial transformer network-hierarchical spatial transformer network (ICSTN-CRNN). ICSTN-CRNN can automatically rectify and recognize LPs. Furthermore, we manually supplement the LP character annotations in PKUData. Our LPDR method achieves satisfactory results on three datasets, including Chinese City Parking Dataset, PKUData, and application-oriented license plate. Through a series of comparative experiments, we prove that our method is more accurate than other advanced methods.

Stopping Criterion during Rendering of Computer-Generated Images Based on SVD-Entropy.

The estimation of image quality and noise perception still remains an important issue in various image processing applications. It has also become a hot topic in the field of photo-realistic computer graphics where noise is inherent in the calculation process. Unlike natural-scene images, however, a reference image is not available for computer-generated images. Thus, classic methods to assess noise quantity and stopping criterion during the rendering process are not usable. This is particularly important in the case of global illumination methods based on stochastic techniques: They provide photo-realistic images which are, however, corrupted by stochastic noise. This noise can be reduced by increasing the number of paths, as proved by Monte Carlo theory, but the problem of finding the right number of paths that are required in order to ensure that human observers cannot perceive any noise is still open. Until now, the features taking part in the human evaluation of image quality and the remaining perceived noise are not precisely known. Synthetic image generation tends to be very expensive and the produced datasets are high-dimensional datasets. In that case, finding a stopping criterion using a learning framework is a challenging task. In this paper, a new method for characterizing computational noise for computer generated images is presented. The noise is represented by the entropy of the singular value decomposition of each block composing an image. These Singular Value Decomposition (SVD)-entropy values are then used as input to a recurrent neural network architecture model in order to extract image noise and in predicting a visual convergence threshold of different parts of any image. Thus a new no-reference image quality assessment is proposed using the relation between SVD-Entropy and perceptual quality, based on a sequence of distorted images. Experiments show that the proposed method, compared with experimental psycho-visual scores, demonstrates a good consistency between these scores and stopping criterion measures that we obtain.