Complex Scene Images Research Articles

A High-Performance Approach for Irregular License Plate Recognition in Unconstrained Scenarios

This paper proposes a novel framework for locating and recognizing irregular license plates in real-world complex scene images. In the proposed framework, an efficient deep convolutional neural network (CNN) structure specially designed for keypoint estimation is first employed to predict the corner points of license plates. Then, based on the predicted corner points, perspective transformation is performed to align the detected license plates. Finally, a lightweight deep CNN structure based on the YOLO detector is designed to predict license plate characters. The character recognition network can predict license plate characters without depending on license plate layouts (i.e., license plates of single-line or double-line text). Experiment results on CCPD and AOLP datasets demonstrate that the proposed method obtains better recognition accuracy compared with previous methods. The proposed model also achieves impressive inference speed and can be deployed in real-time applications.

EMSCNet: Efficient Multisample Contrastive Network for Remote Sensing Image Scene Classification

Significant progress has been achieved in remote sensing image scene classification (RSISC) with the development of convolutional neural networks (CNNs) and vision transformers (ViT). However, high intra-class diversity and inter-class similarity are still enormous challenges for RSISC. Metric learning can effectively improve the discriminative ability of deep representations by constraining the distance between features. Previous metric learning methods only optimize the feature space representation through metric function, ignoring the information interaction between samples. For complex scene images, similarity and discriminative knowledge need to be summarized from the multiple positive and negative pairs. We propose a novel efficient multi-sample contrastive network (EMSCNet) to integrate knowledge from multiple samples. Specifically, we construct a dynamic dictionary with momentum updates to mine positive and negative pairs from the entire dataset. Then, the similarity and discriminative knowledge between samples are summarized by introducing a contrastive module. Finally, the knowledge of the contrastive module is transferred to the backbone classifier through knowledge distillation. The proposed contrastive module can be easily embedded into the training process of CNNs or ViT and removed during inference. Experimental results conducted on three datasets demonstrate the effectiveness of the proposed approach.

Improving Complex Scene Generation by Enhancing Multi-Scale Representations of GAN Discriminators

While recent advances of GAN models enabled photo-realistic synthesis of various object images, challenges still remain in modeling more complex image distributions such as scenes with multiple objects. The difficulty lies in the high structural complexity of scene images, where the discriminator carries a heavy burden in discriminating complex structural differences between real and fake scene images. Therefore, enhancing the discriminative capability of the discriminator could be one of the effective strategies to improve the generation performance of GAN models. In this paper, we explore ways to boost the discriminative capability by leveraging two recent paradigms on visual representation learning: self-supervised learning and transfer learning. As the first approach, we propose a self-supervised auxiliary task tailored to enhance the multi-scale representations of the discriminator. In the second approach, we further enhance the discriminator by utilizing pretrained representations from various scene understanding models. To fully utilize knowledge from multiple expert models, we propose a multi-scale feature ensemble to mix multi-sale representations. Empirical results on challenging scene datasets demonstrate that the proposed strategies significantly advance the generation performance, enabling diverse and photo-realistic synthesis of complex scene images.

Ship detection for complex scene images of space optical remote sensing

Ship detection for complex scene images of space optical remote sensing

Image Semantic Segmentation Fusion of Edge Detection and AFF Attention Mechanism

Deep learning has been widely used in various fields because of its accuracy and efficiency. At present, the improvement of image semantic segmentation accuracy has become the area of most concern. In terms of increasing accuracy, improved semantic segmentation models have attracted more attention. In this paper, a hybrid model is proposed to solve the problems of edge splitting and small objects disappearing from complex scene images. The hybrid model consists of three parts: (1) an improved HED network, (2) an improved PSP-Net, (3) an AFF attention mechanism. Continuous edges can be obtained by combining the improved HED network with an improved PSP-Net. The AFF attention mechanism can improve the segmentation effect of small target objects by enhancing its response recognition ability for specific semantic scenes. The experiments were carried out on Cityspaces, SIFT Flow, NYU-V2 and CamVid datasets, and the experimental results show that the segmentation accuracy of our method is improved by 2% for small target objects, and by 3% for scenes with complex object edges.

On Analog Silicon Photomultipliers in Standard 55-nm BCD Technology for LiDAR Applications

We present an analog silicon photomultiplier (SiPM) based on a standard 55-nm Bipolar-CMOS-DMOS (BCD) technology. The SiPM is composed of 16 × 16 single-photon avalanche diodes (SPADs) and measures 0.29 × 0.32 mm <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{2}$</tex-math></inline-formula> . Each SPAD cell is passively quenched by a monolithically integrated 3.3 V thick oxide transistor. The measured gain is 3.4 × 10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{5}$</tex-math></inline-formula> at 5 V excess bias voltage. The single-photon timing resolution (SPTR) is 185 ps and the multi-photon timing resolution (MPTR) is 120 ps at 3.3 V excess bias voltage. We integrate the SiPM into a co-axial light detection and ranging (LiDAR) system with a time-correlated single-photon counting (TCSPC) module in FPGA. The depth measurement up to 25 m achieves an accuracy of 2 cm and precision of 2 mm under the room ambient light condition. With co-axial scanning, the intensity and depth images of complex scenes with resolutions of 128 × 256 and 256 × 512 are demonstrated. The presented SiPM enables the development of cost-effective LiDAR system-on-chip (SoC) in the advanced technology.

HTC+ for SAR Ship Instance Segmentation

Existing instance segmentation models mostly pay less attention to the targeted characteristics of ships in synthetic aperture radar (SAR) images, which hinders further accuracy improvements, leading to poor segmentation performance in more complex SAR image scenes. To solve this problem, we propose a hybrid task cascade plus (HTC+) for better SAR ship instance segmentation. Aiming at the specific SAR ship task, seven techniques are proposed to ensure the excellent performance of HTC+ in more complex SAR image scenes, i.e., a multi-resolution feature extraction network (MRFEN), an enhanced feature pyramid net-work (EFPN), a semantic-guided anchor adaptive learning network (SGAALN), a context ROI extractor (CROIE), an enhanced mask interaction network (EMIN), a post-processing technique (PPT), and a hard sample mining training strategy (HSMTS). Results show that each of them offers an observable accuracy gain, and the instance segmentation performance in more complex SAR image scenes becomes better. On two public datasets SSDD and HRSID, HTC+ surpasses the other nine competitive models. It achieves 6.7% higher box AP and 5.0% higher mask AP than HTC on SSDD. These are 4.9% and 3.9% on HRSID.

Multitask Learning-Based for SAR Image Superpixel Generation

Most of the existing synthetic aperture radar (SAR) image superpixel generation methods are designed based on the raw SAR images or artificially designed features. However, such methods have the following limitations: (1) SAR images are severely affected by speckle noise, resulting in unstable pixel distance estimation. (2) Artificially designed features cannot be well-adapted to complex SAR image scenes, such as the building regions. Aiming to overcome these shortcomings, we propose a multitask learning-based superpixel generation network (ML-SGN) for SAR images. ML-SGN firstly utilizes a multitask feature extractor to extract deep features, and constructs a high-dimensional feature space containing intensity information, deep semantic informantion, and spatial information. Then, we define an effective pixel distance measure based on the high-dimensional feature space. In addition, we design a differentiable soft assignment operation instead of the non-differentiable nearest neighbor operation, so that the differentiable Simple Linear Iterative Clustering (SLIC) and multitask feature extractor can be combined into an end-to-end superpixel generation network. Comprehensive evaluations are performed on two real SAR images with different bands, which demonstrate that our proposed method outperforms other state-of-the-art methods.

Modulating human memory for complex scenes with artificially generated images

Visual memory schemas (VMS) are two-dimensional memorability maps that capture the most memorable regions of a given scene, predicting with a high degree of consistency human observer’s memory for the same images. These maps are hypothesized to correlate with a mental framework of knowledge employed by humans to encode visual memories. In this study, we develop a generative model we term ‘MEMGAN’ constrained by extracted visual memory schemas that generates completely new complex scene images that vary based on their degree of predicted memorability. The generated populations of high and low memorability images are then evaluated for their memorability using a human observer experiment. We gather VMS maps for these generated images from participants in the memory experiment and compare these with the intended target VMS maps. Following the evaluation of observers’ memory performance through both VMS-defined memorability and hit rate, we find significantly superior memory performance by human observers for the highly memorable generated images compared to poorly memorable. Implementing and testing a construct from cognitive science allows us to generate images whose memorability we can manipulate at will, as well as providing a tool for further study of mental schemas in humans.

Robust Registration Algorithm for Optical and SAR Images Based on Adjacent Self-Similarity Feature

Because optical and synthetic aperture radar (SAR) images are complementary, their registration has received extensive attention in joint applications. However, robust optical and SAR image registration is challenging due to substantial geometric and radiometric differences. To address this problem, we propose a fast and robust registration algorithm for optical and SAR images based on a novel feature type known as the adjacent self-similarity (ASS). The ASS feature of the pixelwise feature representation is defined to quickly and finely capture the structural features of the image. The ASS feature is extracted by using an optimized offset mean filtering method in a neighborhood of the unit pixel radius to accelerate and refine calculations and the local statistics weighted difference operation to suppress coherent speckles. Based on the ASS feature, we extract the minimum self-similarity map (SSM) and the index map, which are robust against radiometric differences and speckles. Then, based on the excellent characteristics of the two maps, we propose a feature detector based on suppressing the local nonmaximum on the minimum SSM and a novel feature descriptor based on calculating the distribution histogram of the index map in a log-polar grid. In addition, we design a rotation invariance enhancement method for the descriptor to improve the rotation invariance robustness of the algorithm. We conduct experiments with both synthetic and real image pairs. The registration results demonstrate that the proposed algorithm has good scale and rotation invariance, as well as good antinoise ability, and that the algorithm performs better than existing state-of-the-art algorithms in terms of registration robustness, accuracy, and efficiency. The registration results on two real optical and SAR image pairs with complex image scenes show the adaptability of the proposed algorithm. The source code of ASS is publicly available¹.

Highly similar and competing visual scenes lead to diminished object but not spatial detail in memory drawings

ABSTRACT Drawings of scenes made from memory can be highly detailed and spatially accurate, with little information not found in the observed stimuli. While prior work has focused on studying memory for distinct scenes, less is known about the specific detail recalled when episodes are highly similar and competing. Here, participants (N = 30) were asked to study and recall eight complex scene images using a drawing task. Importantly, four of these images were exemplars of different scene categories, while the other four images were from the same scene category. The resulting 213 drawings were judged by 1764 online scorers for a comprehensive set of measures, including scene and object diagnosticity, spatial information, and fixation and pen movement behaviour. We observed that competition in memory resulted in diminished object detail, with drawings and objects that were less diagnostic of their original image. However, repeated exemplars of a category did not result in differences in spatial memory accuracy, and there were no differences in fixations during study or pen movements during recall. These results reveal that while drawings for distinct categories of scenes can be highly detailed and accurate, drawings for scenes from repeated categories, creating competition in memory, show reduced object detail.

CSFF-Net: Scene Text Detection Based on Cross-Scale Feature Fusion

In the last years, methods for detecting text in real scenes have made significant progress with an increase in neural networks. However, due to the limitation of the receptive field of the central nervous system and the simple representation of text by using rectangular bounding boxes, the previous methods may be insufficient for working with more challenging instances of text. To solve this problem, this paper proposes a scene text detection network based on cross-scale feature fusion (CSFF-Net). The framework is based on the lightweight backbone network Resnet, and the feature learning is enhanced by embedding the depth weighted convolution module (DWCM) while retaining the original feature information extracted by CNN. At the same time, the 3D-Attention module is also introduced to merge the context information of adjacent areas, so as to refine the features in each spatial size. In addition, because the Feature Pyramid Network (FPN) cannot completely solve the interdependence problem by simple element-wise addition to process cross-layer information flow, this paper introduces a Cross-Level Feature Fusion Module (CLFFM) based on FPN, which is called Cross-Level Feature Pyramid Network (Cross-Level FPN). The proposed CLFFM can better handle cross-layer information flow and output detailed feature information, thus improving the accuracy of text region detection. Compared to the original network framework, the framework provides a more advanced performance in detecting text images of complex scenes, and extensive experiments on three challenging datasets validate the realizability of our approach.

Development of data driven adaptive edge detectors for image processing

In most vision processing activities, the early stage involves identifying the features in an image that provide cues to structure and properties of the object in the scene. Most common features in an image or in a scene arel edges. Edges arel significant local changes in intensity within anlimage. Most important goal of edge detection is to produce a line drawing from anlimage representing the scene. The significant features of an image such as line, curve and corners can be extracted from edges. During the stage of discovering and exploring the information from an image of that scene, edge detection is the most important and early-stage activity and as such it is prominent active area in image processing. Most popular edge detection algorithm such as Robert, Sobel, Canny, Prewitt and Laplacian of Gaussian (LoG), etc. are currently in use. This paper emphasis on an experimental study of limitations of conventional edge detectors and to devise a novel approach to resolve the conflicting issues i.e., limitations of these edge detectors in adaptive space utilizing novel methods such as Bi-dimensional Empirical Mode Decomposition (BEMD), Image Empirical Mode Decomposition (IEMD), Complete Ensemble Empirical Mode Decomposition (CEEMD) and Multivariate Decomposition techniques. Further, to study the performance of these modified edge detectors on the images of complex scenes which are of societal and agricultural importance.

A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos

Bridge inspection is an important step in preserving and rehabilitating transportation infrastructure for extending their service lives. The advancement of mobile robotic technology allows the rapid collection of a large amount of inspection video data. However, the data are mainly the images of complex scenes, wherein a bridge of various structural elements mix with a cluttered background. Assisting bridge inspectors in extracting structural elements of bridges from the big complex video data, and sorting them out by classes, will prepare inspectors for the element-wise inspection to determine the condition of bridges. This article is motivated to develop an assistive intelligence model for segmenting multiclass bridge elements from the inspection videos captured by an aerial inspection platform. With a small initial training dataset labeled by inspectors, a Mask Region-based Convolutional Neural Network pre-trained on a large public dataset was transferred to the new task of multiclass bridge element segmentation. Besides, the temporal coherence analysis attempts to recover false negatives and identify the weakness that the neural network can learn to improve. Furthermore, a semi-supervised self-training method was developed to engage experienced inspectors in refining the network iteratively. Quantitative and qualitative results from evaluating the developed deep neural network demonstrate that the proposed method can utilize a small amount of time and guidance from experienced inspectors (3.58 h for labeling 66 images) to build the network of excellent performance (91.8% precision, 93.6% recall, and 92.7% f1-score). Importantly, the article illustrates an approach to leveraging the domain knowledge and experiences of bridge professionals into computational intelligence models to efficiently adapt the models to varied bridges in the National Bridge Inventory.

What is visible across the visual field?

It is sometimes claimed that because the resolution and sensitivity of visual perception are better in the fovea than in the periphery, peripheral vision cannot support the same kinds of colour and sharpness percepts as foveal vision. The fact that a scene nevertheless seems colourful and sharp throughout the visual field then poses a puzzle. In this study, I use a detailed model of human spatial vision to estimate the visibility of certain properties of natural scenes, including aspects of colourfulness, sharpness, and blurriness, across the visual field. The model is constructed to reproduce basic aspects of human contrast and colour sensitivity over a range of retinal eccentricities. I apply the model to colourful, complex natural scene images, and estimate the degree to which colour and edge information are present in the model’s representation of the scenes. I find that, aside from the intrinsic drift in the spatial scale of the representation, there are not large qualitative differences between foveal and peripheral representations of ‘colourfulness’ and ‘sharpness’.

A Comparison of Convolutional Neural Network Architectures for Automated Detection and Identification of Waterfowl in Complex Environments

Waterfowl monitoring is an important task for understanding waterfowl distribution and habitats. Surveying approaches using hyper-spatial airborne imagery, collected by small unoccupied aerial systems (sUAS), hold potential to overcome the limitations of traditional methods while improving count efficiency and reliability. Difficulties obtaining waterfowl counts, particularly in complex image scenes, from the high quantity of imagery required hinders deployment of large-scale surveys. In this paper, we test Convolutional Neural Networks (CNNs) to understand their potential and how they behave across different versions of our waterfowl dataset. Three CNN architectures (YOLO, Retinanet and Faster R-CNN) were trained on 3 hierarchical levels: waterfowl detection (True / False), waterfowl type (3 classes), and waterfowl species (8 classes). The architectures generally performed well, and results indicate that automated waterfowl detection in complex environments, and therefore enumeration, is feasible using current technology. Waterfowl identification in complex environments was not successful using the available training data, but we propose steps that might enhance the results.

Hier R-CNN: Instance-Level Human Parts Detection and A New Benchmark.

Detecting human parts at instance-level is an essential prerequisite for the analysis of human keypoints, actions, and attributes. Nonetheless, there is a lack of a large-scale, rich-annotated dataset for human parts detection. We fill in the gap by proposing COCO Human Parts. The proposed dataset is based on the COCO 2017, which is the first instance-level human parts dataset, and contains images of complex scenes and high diversity. For reflecting the diversity of human body in natural scenes, we annotate human parts with (a) location in terms of a bounding-box, (b) various type including face, head, hand, and foot, (c) subordinate relationship between person and human parts, (d) fine-grained classification into right-hand/left-hand and left-foot/right-foot. A lot of higher-level applications and studies can be founded upon COCO Human Parts, such as gesture recognition, face/hand keypoint detection, visual actions, human-object interactions, and virtual reality. There are a total of 268,030 person instances from the 66,808 images, and 2.83 parts per person instance. We provide a statistical analysis of the accuracy of our annotations. In addition, we propose a strong baseline for detecting human parts at instance-level over this dataset in an end-to-end manner, call Hier(archy) R-CNN. It is a simple but effective extension of Mask R-CNN, which can detect human parts of each person instance and predict the subordinate relationship between them. Codes and dataset are publicly available (https://github.com/soeaver/Hier-R-CNN).

Detecting Text in Scene and Traffic Guide Panels With Attention Anchor Mechanism

Text detection in complex scene images is a challenging task for intelligent transportation. Recently, anchor mechanisms are widely utilized in scene text detection tasks. However, in existing methods, anchors are generally predefined empirically, degrading robustness to complex scenarios with various sizes and orientation variations. In this paper, we propose a novel Attention Anchor Mechanism (AAM), especially targeting at predicting appropriate anchors for each pixel. To be concrete, we regard a series of predefined anchors as basic anchors and utilize an attention model to predict weights corresponding to basic anchors. Consequently, the weighted sum of basic anchors in each pixel can obtain a predicted anchor. In this way, the gap between the predicted anchors and the corresponding ground truth boxes could be narrowed, making the network easier to regress. For facilitating the design of basic anchors, we adopt a dimension-decomposition mechanism to predict width, height, and angle of anchors, respectively. Extensive experiments on several public datasets demonstrate that our method achieves state-of-the-art performance.

A Comprehensive Pipeline for Complex Text-to-Image Synthesis

Synthesizing a complex scene image with multiple objects and background according to text description is a challenging problem. It needs to solve several difficult tasks across the fields of natural language processing and computer vision. We model it as a combination of semantic entity recognition, object retrieval and recombination, and objects’ status optimization. To reach a satisfactory result, we propose a comprehensive pipeline to convert the input text to its visual counterpart. The pipeline includes text processing, foreground objects and background scene retrieval, image synthesis using constrained MCMC, and post-processing. Firstly, we roughly divide the objects parsed from the input text into foreground objects and background scenes. Secondly, we retrieve the required foreground objects from the foreground object dataset segmented from Microsoft COCO dataset, and retrieve an appropriate background scene image from the background image dataset extracted from the Internet. Thirdly, in order to ensure the rationality of foreground objects’ positions and sizes in the image synthesis step, we design a cost function and use the Markov Chain Monte Carlo (MCMC) method as the optimizer to solve this constrained layout problem. Finally, to make the image look natural and harmonious, we further use Poisson-based and relighting-based methods to blend foreground objects and background scene image in the post-processing step. The synthesized results and comparison results based on Microsoft COCO dataset prove that our method outperforms some of the state-of-the-art methods based on generative adversarial networks (GANs) in visual quality of generated scene images.

Camera System Performance Derived from Natural Scenes

The Modulation Transfer Function (MTF) is a wellestablished measure of camera system performance, commonly employed to characterize optical and image capture systems. It is a measure based on Linear System Theory; thus, its use relies on the assumption that the system is linear and stationary. This is not the case with modern-day camera systems that incorporate non-linear image signal processes (ISP) to improve the output image. Nonlinearities result in variations in camera system performance, which are dependent upon the specific input signals. This paper discusses the development of a novel framework, designed to acquire MTFs directly from images of natural complex scenes, thus making the use of traditional test charts with set patterns redundant. The framework is based on extraction, characterization and classification of edges found within images of natural scenes. Scene derived performance measures aim to characterize non-linear image processes incorporated in modern cameras more faithfully. Further, they can produce ‘live’ performance measures, acquired directly from camera feeds.