Natural Scene Images Research Articles

Camera System Performance Derived from Natural Scenes

The Modulation Transfer Function (MTF) is a wellestablished measure of camera system performance, commonly employed to characterize optical and image capture systems. It is a measure based on Linear System Theory; thus, its use relies on the assumption that the system is linear and stationary. This is not the case with modern-day camera systems that incorporate non-linear image signal processes (ISP) to improve the output image. Nonlinearities result in variations in camera system performance, which are dependent upon the specific input signals. This paper discusses the development of a novel framework, designed to acquire MTFs directly from images of natural complex scenes, thus making the use of traditional test charts with set patterns redundant. The framework is based on extraction, characterization and classification of edges found within images of natural scenes. Scene derived performance measures aim to characterize non-linear image processes incorporated in modern cameras more faithfully. Further, they can produce ‘live’ performance measures, acquired directly from camera feeds.

Multi-Oriented Text Detection in Natural Scene Images Based on the Intersection of MSER With the Locally Binarized Image

Abstract The problem of text extraction is an interesting area of research in computer vision domain. In the recent years, emergence of various applications on smart hand-held devices such as translation of text from one language to another in real time, computerized aid for visually impaired, user navigation & traffic monitoring and driving assistance systems, has stimulated the renewed research interest in this domain. Retrieving text directly from natural scene images or videos is a challenging task due to variant patterns and orientations of scene text. Although various research investigations are available on horizontal oriented text detection in natural scene images, a little number of studies exist on text detection of multiple orientations. In this article, a robust method has been proposed for scene text detection having multiple orientations. The text in the images are horizontally, non-horizontally and curve oriented. The proposed method contains two main stages—maximally stable extremal region (MSER) computation and stroke width transformation (SWT). MSERs have been used to detect maximally stable extremal regions in the images. It is followed by implementing Canny edge detector for enhancement of edges in the images. To remove the non-text regions in the images, the combination of SWT image and geometric information has been used. The performance of the proposed method has been assessed on the IITR text datasets and it produces very encouraging results.

Mining discriminative patches for script identification in natural scene images

This paper focuses on script identification in natural scene images. Traditional CNNs (Convolution Neural Networks) cannot solve this problem perfectly for two reasons: one is the arbitrary aspect ratios of scene images which bring much difficulty to traditional CNNs with a fixed size image as the input. And the other is that some scripts with minor differences are easily confused because they share a subset of characters with the same shapes. We propose a novel approach combing Score CNN, Attention CNN and patches. Attention CNN is utilized to determine whether a patch is a discriminative patch and calculate the contribution weight of the discriminative patch to script identification of the whole image. Score CNN uses a discriminative patch as input and predict the score of each script type. Firstly patches with the same size are extracted from the scene images. Secondly these patches are used as inputs to Score CNN and Attention CNN to train two patch-level classifiers. Finally, the results of multiple discriminative patches extracted from the same image via the above two classifiers are fused to obtain the script type of this image. Using patches with the same size as inputs to CNN can avoid the problems caused by arbitrary aspect ratios of scene images. The trained classifiers can mine discriminative patches to accurately identify some confusing scripts. The experimental results show the good performance of our approach on four public datasets.

Confident Classification Using a Hybrid Between Deterministic and Probabilistic Convolutional Neural Networks

Traditional neural networks trained using point-based maximum likelihood estimation are deterministic models and have exhibited near-human performance in many image classification tasks. However, their insistence on representing network parameters with point-estimates renders them incapable of capturing all possible combinations of the weights; consequently, resulting in a biased predictor towards their initialisation. Most importantly, these deterministic networks are inherently unable to provide any uncertainty estimate for their prediction which is highly sought after in many critical application areas. On the other hand, Bayesian neural networks place a probability distribution on network weights and give a built-in regularisation effect making these models able to learn well from small datasets without overfitting. These networks provide a way of generating posterior distribution which can be used for model's uncertainty estimation. However, Bayesian estimation is computationally very expensive since it greatly widens the parameter space. This paper proposes a hybrid convolutional neural network which combines high accuracy of deterministic models with posterior distribution approximation of Bayesian neural networks. This hybrid architecture is validated on 13 publicly available benchmark classification datasets from a wide range of domains and different modalities like natural scene images, medical images, and time-series. Our results show that the proposed hybrid approach performs better than both deterministic and Bayesian methods in terms of classification accuracy and also provides an estimate of uncertainty for every prediction. We further employ this uncertainty to filter out unconfident predictions and achieve significant additional gain in accuracy for the remaining predictions.

Cluttered TextSpotter: An End-to-End Trainable Light-Weight Scene Text Spotter for Cluttered Environment

Scene text spotting aims at simultaneously localizing and recognizing text instances, symbols, and logos in natural scene images. Scene text detection and recognition approaches have received immense attention in computer vision research community. The presence of partial occlusion or truncation artifact due to the cluttered background of scene images creates an obstacle in perceiving the text instances, which makes the process of spotting very complex. In this paper, we propose a light-weight scene text spotter that can address the issue of cluttered environment of scene images. It is an end-to-end trainable deep neural network that uses local part information, global structural features, and context cue information of oriented region proposals for spotting text instances. It helps to localize in scene images with background clutters, where partially occluded text parts, truncation artifacts, and perspective distortions are present. We mitigate the problem of misclassification caused by inter-class interference by exploring inter-class separability and intra-class compactness. We also incorporate multi-language character segmentation and word-level recognition in a light-weight recognition module. We have used six publicly available benchmark datasets in different smart devices to illustrate the efficacy of the network.

Text detection in natural scene images using morphological component analysis and Laplacian dictionary

Text in natural scene images usually carries abundant semantic information. However, due to variations of text and complexity of background, detecting text in scene images becomes a critical and challenging task. In this paper, we present a novel method to detect text from scene images. Firstly, we decompose scene images into background and text components using morphological component analysis &#x0028 MCA &#x0029 , which will reduce the adverse effects of complex backgrounds on the detection results. In order to improve the performance of image decomposition, two discriminative dictionaries of background and text are learned from the training samples. Moreover, Laplacian sparse regularization is introduced into our proposed dictionary learning method which improves discrimination of dictionary. Based on the text dictionary and the sparse-representation coefficients of text, we can construct the text component. After that, the text in the query image can be detected by applying certain heuristic rules. The results of experiments show the effectiveness of the proposed method.

Urdu-Text Detection and Recognition in Natural Scene Images Using Deep Learning

Urdu text is a cursive script and belongs to a non-Latin family of other cursive scripts like Arabic, Chinese, and Hindi. Urdu text poses a challenge for detection/localization from natural scene images, and consequently recognition of individual ligatures in scene images. In this paper, a methodology is proposed that covers detection, orientation prediction, and recognition of Urdu ligatures in outdoor images. As a first step, the custom FasterRCNN algorithm has been used in conjunction with well-known CNNs like Squeezenet, Googlenet, Resnet18, and Resnet50 for detection and localization purposes for images of size $320\times 240$ pixels. For ligature Orientation prediction, a custom Regression Residual Neural Network (RRNN) is trained/tested on datasets containing randomly oriented ligatures. Recognition of ligatures was done using Two Stream Deep Neural Network (TSDNN). In our experiments, five-set of datasets, containing 4.2K and 51K Urdu-text-embedded synthetic images were generated using the CLE annotation text to evaluate different tasks of detection, orientation prediction, and recognition of ligatures. These synthetic images contain 132, and 1600 unique ligatures corresponding to 4.2K and 51K images respectively, with 32 variations of each ligature (4-backgrounds and font 8-color variations). Also, 1094 real-world images containing more than 12k Urdu characters were used for TSDNN’s evaluation. Finally, all four detectors were evaluated and used to compare them for their ability to detect/localize Urdu-text using average-precision (AP). Resnet50 features based FasterRCNN was found to be the winner detector with AP of.98. While Squeeznet, Googlenet, Resnet18 based detectors had testing AP of.65,.88, and.87 respectively. RRNN achieved and accuracy of 79% and 99% for 4k and 51K images respectively. Similarly, for characters classification in ligatures, TSDNN attained a partial sequence recognition rate of 94.90% and 95.20% for 4k and 51K images respectively. Similarly, a partial sequence recognition rate of 76.60% attained for real world-images.

A Database for Urdu Text Detection and Recognition in Natural Scene Images

Conventional solar cells are not economical and are recently too expensive to the manufacturers for extensive-scale electricity generation. Cost and efficiency is most vital factor in the accomplishment of any solar technology. In order to improve the conversion efficiency, the major research in third generation photovoltaic (PV) cells is directed toward retaining more sunlight using nanotechnology. Advancement in nanotechnology solar cell via quantum dots (QDs) could reduce the cost of PV cell and additionally enhance cell conversion efficiency. Silicon quantum dots (Si-QDs) are semiconductor nano crystals of nanometers dimension whose electron-holes are confined in all three spatial dimensions. Quantum dots have discrete electronic states. Quantum dots have capacity to change band gap with the adjustment in size of quantum dot. As the quantum dots size fluctuates over a wide range that demonstrates the variety of band gap so it will assimilate or discharge light. In this paper, the generic mathematical models of PV cell are adopted and then I-V and P-V characteristic curves are obtained from selected parameters using MATLAB software. The essential parameters are taken from datasheets. I-V and P-V characteristics curves are obtained for selected model. Silicon quantum dots have the tunable band gap that is added to conventional PV cell and obtain the I-V and P-V curves. After simulation, efficiency and power of Conventional PV cell to quantum dots based PV cell is compared. The property of quantum dots is used in extending the band gap of solar cells and increasing the maximum proportion of incident sunlight absorbed, hence improving efficiency.

Object Recognition Algorithm Based on an Improved Convolutional Neural Network

In order to accomplish the task of object recognition in natural scenes, a new object recognition algorithm based on an improved convolutional neural network (CNN) is proposed. First, candidate object windows are extracted from the original image. Then, candidate object windows are input into the improved CNN model to obtain deep features. Finally, the deep features are input into the Softmax and the confidence scores of classes are obtained. The candidate object window with the highest confidence score is selected as the object recognition result. Based on AlexNet, Inception V1 is introduced into the improved CNN and the fully connected layer is replaced by the average pooling layer, which widens the network and deepens the network at the same time. Experimental results show that the improved object recognition algorithm can obtain better recognition results in multiple natural scene images, and has a higher degree of accuracy than the classical algorithms in the field of object recognition.

Understanding and Predicting the Memorability of Outdoor Natural Scenes

Memorability measures how easily an image is to be memorized after glancing, which may contribute to designing magazine covers, tourism publicity materials, and so forth. Recent works have shed light on the visual features that make generic images, object images or face photographs memorable. However, these methods are not able to effectively predict the memorability of outdoor natural scene images. To overcome this shortcoming of previous works, in this paper, we provide an attempt to answer: “what exactly makes outdoor natural scenes memorable”. To this end, we first establish a large-scale outdoor natural scene image memorability (LNSIM) database, containing 2,632 outdoor natural scene images with their ground truth memorability scores and the multi-label scene category annotations. Then, similar to previous works, we mine our database to investigate how low-, middle- and high-level handcrafted features affect the memorability of outdoor natural scenes. In particular, we find that the high-level feature of scene category is rather correlated with outdoor natural scene memorability, and the deep features learnt by deep neural network (DNN) are also effective in predicting the memorability scores. Moreover, combining the deep features with the category feature can further boost the performance of memorability prediction. Therefore, we propose an end-to-end DNN based outdoor natural scene memorability (DeepNSM) predictor, which takes advantage of the learned category-related features. Then, the experimental results validate the effectiveness of our DeepNSM model, exceeding the state-of-the-art methods. Finally, we try to understand the reason of the good performance for our DeepNSM model, and also study the cases that our DeepNSM model succeeds or fails to accurately predict the memorability of outdoor natural scenes.

ORDNet: Capturing Omni-Range Dependencies for Scene Parsing.

Learning to capture dependencies between spatial positions is essential to many visual tasks, especially the dense labeling problems like scene parsing. Existing methods can effectively capture long-range dependencies with self-attention mechanism while short ones by local convolution. However, there is still much gap between long-range and short-range dependencies, which largely reduces the models' flexibility in application to diverse spatial scales and relationships in complicated natural scene images. To fill such a gap, we develop a Middle-Range (MR) branch to capture middle-range dependencies by restricting self-attention into local patches. Also, we observe that the spatial regions which have large correlations with others can be emphasized to exploit long-range dependencies more accurately, and thus propose a Reweighed Long-Range (RLR) branch. Based on the proposed MR and RLR branches, we build an Omni-Range Dependencies Network (ORDNet) which can effectively capture short-, middle- and long-range dependencies. Our ORDNet is able to extract more comprehensive context information and well adapt to complex spatial variance in scene images. Extensive experiments show that our proposed ORDNet outperforms previous state-of-the-art methods on three scene parsing benchmarks including PASCAL Context, COCO Stuff and ADE20K, demonstrating the superiority of capturing omni-range dependencies in deep models for scene parsing task.

Text Extraction and Recognition in Natural Scene Images using Contourlet Transform and PNN

Of late, the rapid development in the technology and multimedia capability in digital cameras and mobile devices has led to ever increasing number of images or multi-media data to the digital world. Particularly, in natural scene images, the text content provides explicit information to understand the semantics of images. Therefore, a system developed for extracting and recognizing texts accurately from natural scene images, in real-time, has significant relevance to numerous applications such as, assistive technology for people with vision impairment, tourist with language barrier, vehicle number plate detection, street signs, advertisement bill boards, robotics, etc. The extraction of the texts from natural scene images is a formidable task due to large variations in character fonts, styles, sizes, text orientations, presence of complex backgrounds and varying light conditions. The main focus of this research paper is to propose a novel hybrid approach for automatic detection, localization, extraction and recognition of text in natural scene images with cluttered background. Firstly, image regions with text are detected using edge features (GLCM) extracted from Contourlet transformed image and SVM (Support Vector Machine) classifier. Secondly, horizontal projection is applied on text regions for segmenting lines and vertical projection is applied on each text line for segmenting characters. The proposed method for text extraction has produced the precision, recall, F-Score and accuracy of 98.50%, 90.85.62%, 95.00%, and 89.90%, respectively. And, these results prove that the proposed method is efficient. Further, the so extracted characters are processed for recognition using contourlet transform and Probabilistic Neural Network (PNN) classifier. The computed features are moment invariants. Only the English script is considered for the experimentation. The proposed character recognition method has accuracy of 79.07%, which is higher in comparison to accuracy of 75.15% obtained by KNN (K-Nearest Neighbors) classifier

A Parallel Image Registration Algorithm Based on a Lattice Boltzmann Model

Image registration is a key pre-procedure for high level image processing. However, taking into consideration the complexity and accuracy of the algorithm, the image registration algorithm always has high time complexity. To speed up the registration algorithm, parallel computation is a relevant strategy. Parallelizing the algorithm by implementing Lattice Boltzmann method (LBM) seems a good candidate. In consequence, this paper proposes a novel parallel LBM based model (LB model) for image registration. The main idea of our method consists in simulating the convection diffusion equation through a LB model with an ad hoc collision term. By applying our method on computed tomography angiography images (CTA images), Magnet Resonance images (MR images), natural scene image and artificial images, our model proves to be faster than classical methods and achieves accurate registration. In the continuity of 2D image registration model, the LB model is extended to 3D volume registration providing excellent results in domain such as medical imaging. Our method can run on massively parallel architectures, ranging from embedded field programmable gate arrays (FPGAs) and digital signal processors (DSPs) up to graphics processing units (GPUs).

Widget Detection Network: widget detection in mobile screenshot with region-based attention networks

We propose an architecture to automatically detect widgets in mobile screenshots, considering only visual cues. Even though traditional object detection methods perform well on common objects in natural scene images, they are unable to deal with the screenshot images with complex widget layout. Therefore, we propose region-based Widget Detection Network (WDN), which introduces regularities in the screenshot images as the regularizations. First, we design a scale-aware attention structure to make the backbone network sensitive to widget scales so that the salient features of the interest regions could be captured. Second, a strategy of horizontal region generation is proposed to fully utilize the aligned property of widget arrangement, which generates all the region candidates in a horizontal line at once. Finally, a variant of online hard example mining is employed to alleviate the problem of imbalance samples, which explicitly restricts the ratio of foreground and background to achieve better balance. We conduct experiments on a proposed benchmark dataset. The quantitative results and qualitative analysis on the benchmark dataset show that WDN achieves impressive performance, which outperforms the common object detection methods in the widget detection task.

Image annotating tools for agricultural purpose - A requirements study

Images of natural scenes, like those relevant for agriculture, are characterised with a variety of forms of objects of interest and similarities between objects that one might want to discriminate. This introduces uncertainty to the analysis of such images. Requirements for an image annotation tool to be used in pattern recognition design for agriculture were discussed. A selection of open source annotating tools were presented. Advices how to use the software to handle uncertainty and missing functionalities were described.

CASR: a context-aware residual network for single-image super-resolution

With the significant power of deep learning architectures, researchers have made much progress on super-resolution in the past few years. However, due to low representational ability of feature maps extracted from nature scene images, directly applying deep learning architectures for super-resolution could result in poor visual effects. Essentially, unique characteristics like low-frequency information should be emphasized for better shape reconstruction, other than treated equally across different patches and channels. To ease this problem, we propose a lightweight context-aware deep residual network named as CASR network, which appropriately encodes channel and spatial attention information to construct context-aware feature map for single-image super-resolution. We firstly design a task-specified inception block with a novel structure of astrous filters and specially chosen kernel size to extract multi-level information from low-resolution images. Then, a Dual-Attention ResNet module is applied to capture context information by dually connecting spatial and channel attention schemes. With high representational ability of context-aware feature map, CASR can accurately and efficiently generate high-resolution images. Experiments on several popular datasets show the proposed method has achieved better visual improvements and superior efficiencies than most of the existing studies.

Delaunay triangulation based text detection from multi-view images of natural scene

Text detection in the wild is still considered as a challenging issue to the researchers because of its several real time applications like forensic application, where CCTV camera captures images at different angles of the same scene. Unlike the existing methods that consider a single view captured orthogonally for text detection, this paper considers multi-view (view-1 and view-2 of the same spot) of the same scene captured at different angles or different height distances for text detection. For each pair of the same scene, the proposed method extracts features that describe characteristics of text components based on Delaunay Triangulation (DT), namely corner points, area and cavity of the DT. The features of corresponding DT in view-1 and view-2 are compared through cosine distance measure to estimate the similarity between two components of respective view-1 and view-2. If the pair satisfies the similarity condition, the components are considered as Candidate Text Components (CTC). In other words, these are the common components for view-1 and view-2 that satisfy the similarity condition. From each CTC of view-1 and view-2, the proposed method finds nearest neighbor components to restore the components of the same text line based on estimating degree of similarly between CTC and neighbor components using Chi-square and cosine distance measures. Furthermore, the proposed method uses a recognition step to detect correct texts by comparing recognition results of view-1 and view-2. The same recognition step is used for removing false positives to improve the performance of the proposed method. Experimental results on our own dataset, which contains pair of images of different situations, and the standard datasets, namely, ICDAR 2013, MSRATD-500, CTW1500, Total-text, ICDAR 2017 MLT and COCO-text, show that the proposed method outperforms the existing methods.

An Efficient Technique for Scene Text Extraction from Videos

Text Detection from natural scene images and videos is imperative for applications in real world domain analysis. However the text detection process isperplexingbecause of exigent scenarios that the text exhibit. The information present in the video is either perceptual or it is either in semantic form. Amongst the different content that exists in the video, the text information is a major important content that describes more about the nature of the video. The text present in the video can be categorized into Caption text and Scene text. The caption text is the artificial text that is easy to detect while scene text are natural text which is difficult to identify. In this paper text extraction in natural images by edge based method is implemented. The algorithms are estimated with a set of images of natural scenes that differ alongside the scope of font size, illumination, scale and text direction. Precision, accuracy and recall rates are determined to evaluate the performance. The proposed system worked for all difficult scenarios of varied text and gave better results than the existing methods.

Multi-Scale Geospatial Object Detection Based on Shallow-Deep Feature Extraction

Multi-class detection in remote sensing images (RSIs) has garnered wide attention and introduced several service applications in many fields, including civil and military fields. However, several reasons make detection from aerial images very challenging and more difficult than nature scene images: Objects do not have a fixed size, often appear at very various scales and sometimes appear in dense groups, like vehicles and storage tanks, and have different surroundings or background areas. Furthermore, all of this makes the manual annotation of objects very complex and costly. The powerful effect of the feature extraction methods on object detection and the successes of deep convolutional neural networks (CNN) extract deep features more than traditional methods. This study introduced a novel network structure and designed a unique feature extraction which employs squeeze and excitation network (SENet) and residual network (ResNet) to obtain feature maps, named a shallow-deep feature extraction (SDFE), that improves the resolution and the localization at the same time. Furthermore, this novel model reduces the loss of dense groups and small objects, and provides higher and more stable detection accuracy which is not significantly affected by changing the value of the threshold of the intersection over union (IoU) and overcomes the difficulties of RSIs. Moreover, this study introduced strong evidence about the factors that affect the detection of RSIs. The proposed shallow-deep and multi-scale (SD-MS) method outperforms other approaches for the given ten classes of the NWPU VHR-10 dataset.

Color improves edge classification in human vision.

Despite the complexity of the visual world, humans rarely confuse variations in illumination, for example shadows, from variations in material properties, such as paint or stain. This ability to distinguish illumination from material edges is crucial for determining the spatial layout of objects and surfaces in natural scenes. In this study, we explore the role that color (chromatic) cues play in edge classification. We conducted a psychophysical experiment that required subjects to classify edges into illumination and material, in patches taken from images of natural scenes that either contained or did not contain color information. The edge images were of various sizes and were pre-classified into illumination and material, based on inspection of the edge in the context of the whole image from which the edge was extracted. Edge classification performance was found to be superior for the color compared to grayscale images, in keeping with color acting as a cue for edge classification. We defined machine observers sensitive to simple image properties and found that they too classified the edges better with color information, although they failed to capture the effect of image size observed in the psychophysical experiment. Our findings are consistent with previous work suggesting that color information facilitates the identification of material properties, transparency, shadows and the perception of shape-from-shading.