Local Image Regions Research Articles

A Partial Differential Equation-Based Image Restoration Method in Environmental Art Design

With the rapid development of networks and the emergence of various devices, images have become the main form of information transmission in real life. Image restoration, as an important branch of image processing, can be applied to real-life situations such as pixel loss in image transmission or network prone to packet loss. However, existing image restoration algorithms have disadvantages such as fuzzy restoration effect and slow speed; to solve such problems, this paper adopts a dual discriminator model based on generative adversarial networks, which effectively improves the restoration accuracy by adding local discriminators to track the information of local missing regions of images. However, the model is not optimistic in generating reasonable semantic information, and for this reason, a partial differential equation-based image restoration model is proposed. A classifier and a feature extraction network are added to the dual discriminator model to provide category, style, and content loss constraints to the generative network, respectively. To address the training instability problem of discriminator design, spectral normalization is introduced to the discriminator design. Extensive experiments are conducted on a data dataset of partial differential equations, and the results show that the partial differential equation-based image restoration model provides significant improvements in image restoration over previous methods and that image restoration techniques are exceptionally important in the application of environmental art design.

Multi-level adversarial network for domain adaptive semantic segmentation

Recent progresses in domain adaptive semantic segmentation demonstrate the effectiveness of adversarial learning (AL) in unsupervised domain adaptation. However, most adversarial learning based methods align source and target distributions at a global image level but neglect the inconsistency around local image regions. This paper presents a novel multi-level adversarial network (MLAN) that aims to address inter-domain inconsistency at both global image level and local region level optimally. MLAN has two novel designs, namely, region-level adversarial learning (RL-AL) and co-regularized adversarial learning (CR-AL). Specifically, RL-AL models prototypical regional context-relations explicitly in the feature space of a labelled source domain and transfers them to an unlabelled target domain via adversarial learning. CR-AL fuses region-level AL and image-level AL optimally via mutual regularization. In addition, we design a multi-level consistency map that can guide domain adaptation in both input space (i.e., image-to-image translation) and output space (i.e., self-training) effectively. Extensive experiments show that MLAN outperforms the state-of-the-art with a large margin consistently across multiple datasets.

TGSL-Dependent Feature Selection for Boosting the Visual Sentiment Classification

The automatic recognition of the emotions in still images is inherently more challenging than other visual recognition tasks, such as scene recognition, object classification and semantic image classification, as it involves a higher level of abstraction in the human cognition perspective. Symmetry can be found in many objects in the nature and can be used for many purposes such as object detection and recognition. Furthermore, rotating and flipping of the image is employed based on symmetry for training the classifier for the most accurate classification. Hence, there is a need to handle effectively large intra-class variance, scalability and subjectivity during recognition, and it is inherently ambiguous as an image can evoke multiple emotions. To address these issues, many of the existing works focus on improving the image representations. It is motivated by the observation that both global distributions and local image regions carry massive sentiments. In this research, three different pre-trained architectural models are implemented, and the classification performance of binary sentiment classification is examined on five widely-used effective datasets. Moreover, the features from the pre-trained models are selected optimally using the proposed Teaching Gaining Sharing Learning (TGSL) algorithm, which is the major contribution of the research. Extensive experiment results on the five datasets demonstrate that the proposed Visual sentiment analysis based on the TGSL algorithm with data augmentation achieved an improved performance compared to all other conventional techniques. The proposed framework uses the pre-trained model and never utilized any hand-crafted features, boosting the mean accuracy, sensitivity, and specificity to 99.11%, 99.31%, and 99.22%, respectively, for abstract dataset.

Scene-Specific Dark Channel Prior for Single Image Fog Removal

Extensive presence of fog in outdoor images severely alters the scene appearance and hence reduces the visibility. Image processing based defogging algorithms are used to restore the details and colour in a single foggy image. Performances of the previous defogging approaches are considerably low since they fail to consider the image-specific cues. In this paper, a novel and simple defogging approach is proposed based on the estimation of depth map by considering the density of fog in local image regions. The proposed approach uses the scene-specific depth map information to compute the dark channel and transmission. The quality of recovered image is further improved by a post-processing technique. Experimental evaluation performed on FRIDA and FRIDA2 benchmark datasets demonstrates the proposed defogging framework outperforms state-of-the-art approaches. The code and the results of this work are open-sourced for reproducibility (https://github.com/RPRO5/Defogging).

Learning sequentially diversified representations for fine-grained categorization

Learning representation carrying rich local information is essential for recognizing fine-grained objects. Existing methods to this task resort to multi-stage frameworks to capture fine-grained information. However, they usually require multiple forward passes of the backbone network, resulting in efficiency deterioration. In this paper, we propose Sequentially Diversified Networks (SDNs) that enrich representation by promoting their diversity while maintaining the extraction efficiency. Specifically, we construct multiple lightweight sub-networks to model mutually different scales of discriminative patterns. The design of these sub-networks follows the sequentially diversified constraint, encouraging them to be varied in spatial attention. By inserting these sub-networks into a single backbone network, SDNs enable information interaction among local regions of the fine-grained image. In this way, SDNs jointly promote diversity in terms of scale and spatial attention in the one-stage pipeline, thereby facilitating the learning of diversified representation efficiently. We evaluate our proposed method on three challenging datasets, namely CUB-200-2011, Stanford-Cars, and FGVC-Aircraft. Experiments demonstrate its effectiveness in learning diversified information. Moreover, our method achieves state-of-the-art performance, only requiring a single forward pass of the backbone network, which reduces inference time noticeably.

Attention-guided image captioning with adaptive global and local feature fusion

Although attention mechanisms are exploited widely in encoder-decoder neural network-based image captioning framework, the relation between the selection of salient image regions and the supervision of spatial information on local and global representation learning was overlooked, thereby degrading captioning performance. Consequently, we propose an image captioning scheme based on adaptive spatial information attention (ASIA), extracting a sequence of spatial information of salient objects in a local image region or an entire image. Specifically, in the encoding stage, we extract the object-level visual features of salient objects and their spatial bounding-box. We obtain the global feature maps of an entire image, which are fused with local features and the fused features are fed into the LSTM-based language decoder. In the decoding stage, our adaptive attention mechanism dynamically selects the corresponding image regions specified by an image description. Extensive experiments conducted on two datasets demonstrate the effectiveness of the proposed method.

Selective Region-based Photo Color Adjustment for Graphic Designs

When adding a photo onto a graphic design, professional graphic designers often adjust its colors based on some target colors obtained from the brand or product to make the entire design more memorable to audiences and establish a consistent brand identity. However, adjusting the colors of a photo in the context of a graphic design is a difficult task, with two major challenges: (1) Locality: The color is often adjusted locally to preserve the semantics and atmosphere of the original image; and (2) Naturalness: The modified region needs to be carefully chosen and recolored to obtain a semantically valid and visually natural result. To address these challenges, we propose a learning-based approach to photo color adjustment for graphic designs, which maps an input photo along with the target colors to a recolored result. Our method decomposes the color adjustment process into two successive stages: modifiable region selection and target color propagation. The first stage aims to solve the core, challenging problem of which local image region(s) should be adjusted, which requires not only a common sense of colors appearing in our visual world but also understanding of subtle visual design heuristics. To this end, we capitalize on both natural photos and graphic designs to train a region selection network, which detects the most likely regions to be adjusted to the target colors. The second stage trains a recoloring network to naturally propagate the target colors in the detected regions. Through extensive experiments and a user study, we demonstrate the effectiveness of our selective region-based photo recoloring framework.

Face Recognition Based on Statistical Texture Features

Facial recognition has attracted the attention of researchers and has been one of the most prominent topics in the fields of image processing and pattern recognition since 1990. This resulted in a very large number of recognition methods and techniques with the aim of increasing the accuracy and robustness of existing systems. Many techniques have been developed to address the challenges and reliable recognition systems have been reached but require considerable processing time, suffer from high memory consumption and are relatively complex. The focus of this paper is on extracting subset of descriptors (less correlated and less calculations) from the co-occurrence matrix with the goal of enhancing the performance of Haralick’s descriptors. Improvements are achieved by adding the image pre-processing and selecting the proper method according to the database problem and by extracting features from image local regions.

Photoelectric Sensor Circuit and Image Segmentation Method for Radar System

In this exploration, based on the principle and system parameters of laser three-dimensional (3D) radar imaging technology, the corresponding photoelectric sensor circuit scheme is formulated. The sense circuit of avalanche photon diode (APD) converts the signal through the transresistance amplifier circuit. Then, LMH6629 is selected as a precision amplifier with low input noise voltage and low input error current. The capacitance is used as a compensation element to compensate the phase. For the power supply scheme, choosing the mode of switching power supply and LDO to work together can improve the efficiency of power supply and reduce the output of current ripple. At the same time, semantic segmentation is carried out for the obtained photoelectric images. Based on the traditional spatial pyramid pooling algorithm, the fusion of mean intersection over union and cross information entropy loss function is introduced to improve the weight of local image region. In the experiment, Multisim software is used to simulate the circuit. The APD reverse bias voltage is set to 90 V, and the multiplication coefficient is 98.7. The feedback resistance, bandwidth, phase compensation capacitance and other parameters are further calculated. It is found that there is obvious self-excited phenomenon in the output waveform of the transresistance amplifier without phase compensation capacitor. When the feedback capacitance reaches 0.8 pF, the oscillation phenomenon is obviously reduced; further calculation shows that the bandwidth of transresistance amplifier is 230 MHz, and the noise of APD power supply is mainly caused by BUCK switching power supply switch when the bottom noise of oscilloscope is ignored. However, the noise is suppressed under the action of the back-end LDO device; after the loss function is introduced, the contour of the photoelectric image is preserved completely, and then the more accurate segmentation results are obtained.

Landscape Similarity Analysis Using Texture Encoded Deep-Learning Features on Unclassified Remote Sensing Imagery

Convolutional neural networks (CNNs) are known for their ability to learn shape and texture descriptors useful for object detection, pattern recognition, and classification problems. Deeper layer filters of CNN generally learn global image information vital for whole-scene or object discrimination. In landscape pattern comparison, however, dense localized information encoded in shallow layers can contain discriminative information for characterizing changes across image local regions but are often lost in the deeper and non-spatial fully connected layers. Such localized features hold potential for identifying, as well as characterizing, process–pattern change across space and time. In this paper, we propose a simple yet effective texture-based CNN (Tex-CNN) via a feature concatenation framework which results in capturing and learning texture descriptors. The traditional CNN architecture was adopted as a baseline for assessing the performance of Tex-CNN. We utilized 75% and 25% of the image data for model training and validation, respectively. To test the models’ generalization, we used a separate set of imagery from the Aerial Imagery Dataset (AID) and Sentinel-2 for model development and independent validation. The classical CNN and the Tex-CNN classification accuracies in the AID were 91.67% and 96.33%, respectively. Tex-CNN accuracy was either on par with or outcompeted state-of-the-art methods. Independent validation on Sentinel-2 data had good performance for most scene types but had difficulty discriminating farm scenes, likely due to geometric generalization of discriminative features at the coarser scale. In both datasets, the Tex-CNN outperformed the classical CNN architecture. Using the Tex-CNN, gradient-based spatial attention maps (feature maps) which contain discriminative pattern information are extracted and subsequently employed for mapping landscape similarity. To enhance the discriminative capacity of the feature maps, we further perform spatial filtering, using PCA and select eigen maps with the top eigen values. We show that CNN feature maps provide descriptors capable of characterizing and quantifying landscape (dis)similarity. Using the feature maps histogram of oriented gradient vectors and computing their Earth Movers Distances, our method effectively identified similar landscape types with over 60% of target-reference scene comparisons showing smaller Earth Movers Distance (EMD) (e.g., 0.01), while different landscape types tended to show large EMD (e.g., 0.05) in the benchmark AID. We hope this proposal will inspire further research into the use of CNN layer feature maps in landscape similarity assessment, as well as in change detection.

CycleSegNet: Object Co-Segmentation With Cycle Refinement and Region Correspondence.

Image co-segmentation is an active computer vision task that aims to segment the common objects from a set of images. Recently, researchers design various learning-based algorithms to undertake the co-segmentation task. The main difficulty in this task is how to effectively transfer information between images to make conditional predictions. In this paper, we present CycleSegNet, a novel framework for the co-segmentation task. Our network design has two key components: a region correspondence module which is the basic operation for exchanging information between local image regions, and a cycle refinement module, which utilizes ConvLSTMs to progressively update image representations and exchange information in a cycle and iterative manner. Extensive experiments demonstrate that our proposed method significantly outperforms the state-of-the-art methods on four popular benchmark datasets - PASCAL VOC dataset, MSRC dataset, Internet dataset, and iCoseg dataset, by 2.6%, 7.7%, 2.2%, and 2.9%, respectively.

Efficient Pixel-Wise SVD Required for Image Processing Using the Color Line Feature

In color image processing based on mathematical optimization, a color-line image feature has been considered and many methods that give good results have been proposed. A color-line is a linear color distribution (correlation line) observed in a local image region, and is numerically represented by the sparsity of a data matrix generated from the neighboring pixel values. However, the calculation requires a lot of processing time because each data matrix is processed by inverse calculation or singular value decomposition (SVD) with some operations on the decomposed singular values. In this paper, in order to address this problem, we propose a method that can effectively compute SVD for each data matrix. Using the experimental knowledge that matrices obtained from neighboring regions (each centered at an adjacent pixel) are similar to each other, we intentionally design the SVD by using an iterative method (Arnoldi iteration), and propagate the converged singular vectors at a pixel to the next pixel as the initial vectors of the iteration. This propagation can drastically reduce the number of iterations required for convergence. Additionally, the singular values and vectors are obtained in descending order, which is advantageous when a matrix is reconstructed after reducing small singular values, so we can truncate the calculation when a singular value becomes lower than a threshold value. To show the effectiveness, we apply the proposed method to a denoising method, arranged structure-tensor total variation (ASTV), and show that the processing time is shortened by 95% compared to the naive method without losing numerical accuracy.

Attribute-Aligned Domain-Invariant Feature Learning for Unsupervised Domain Adaptation Person Re-Identification

Domain invariance and discrimination of learned features as two crucial factors affect the performance of unsupervised domain adaptation (UDA) person re-identification (Re-ID). Person attributes (such as “backpack”, “boots”, “handbag”, etc) remaining unchanged across multiple domains have been used as mid-level visual-semantic information in UDA person Re-ID. As two main challenges, both misalignment of attribute-related regions across multiple images and domain shift between source and target domains affect the learning of domain-invariant features (DIF). To address the above two challenges, this article proposes to take advantage of the stability of person attributes and the complementarity of person attributes and the corresponding low-level visual features to guide the learning of discriminative DIF. Specifically, the proposed solution contains the generation of latent attribute-correlated visual features (GLAVF), DIF learning under the guidance of person attributes, and the alignment of person attributes corresponding to the local regions of pedestrian images. Due to the gap between person attributes and visual features, person attributes are first converted into latent attribute-correlated visual features (LAVF) without any specific domain information in GLAVF, and then LAVF are used as the substitutions of person attributes to guide the learning of DIF. To enhance the discrimination of learned features, the proposed solution mainly explores the alignment between person attributes and corresponding local regions, and the alignment of the same person attributes across multiple pedestrian images. A fully connected layer is used to achieve the above two types of alignment in the proposed framework, which reduces the adverse impacts of inference information and ensures the semantic consistency between person attributes and corresponding local regions across multiple pedestrian images. The effectiveness of the proposed solution is confirmed on four existing datasets by comparative experiments.

Joint Constrained Least-Square Regression With Deep Convolutional Feature for Palmprint Recognition

Various palmprint recognition methods have been proposed and applied in security, particularly authentication. However, improving the performance of palmprint recognition with insufficient training samples per person is still a challenging task. The undersampling problem limits the application and popularization of palmprint recognition. In this article, by regularly sampling different local regions of the palmprint image, we learn complete and discriminative convolution features by using deep convolutional neural networks (DCNNs). With this powerful palmprint description, a joint constrained least-square regression (JCLSR) framework, which performs representation for each local region of the same palmprint image requiring all regular local regions of the palmprint image to have similar projected target matrices, is presented to exploit the commonality of different patches. The proposed method can well solve the undersampling classification problem in palmprint recognition. Experiments were conducted on the IITD, CASIA, noisy IITD, and PolyU multispectral palmprint databases. It can be seen from the experimental results that the proposed JCLSR consistently outperformed the classical palmprint recognition methods and some subspace learning-based methods for palmprint recognition.

Towards Optical Imaging for Spine Tracking without Markers in Navigated Spine Surgery.

Surgical navigation systems are increasingly used for complex spine procedures to avoid neurovascular injuries and minimize the risk for reoperations. Accurate patient tracking is one of the prerequisites for optimal motion compensation and navigation. Most current optical tracking systems use dynamic reference frames (DRFs) attached to the spine, for patient movement tracking. However, the spine itself is subject to intrinsic movements which can impact the accuracy of the navigation system. In this study, we aimed to detect the actual patient spine features in different image views captured by optical cameras, in an augmented reality surgical navigation (ARSN) system. Using optical images from open spinal surgery cases, acquired by two gray-scale cameras, spinal landmarks were identified and matched in different camera views. A computer vision framework was created for preprocessing of the spine images, detecting and matching local invariant image regions. We compared four feature detection algorithms, Speeded Up Robust Feature (SURF), Maximal Stable Extremal Region (MSER), Features from Accelerated Segment Test (FAST), and Oriented FAST and Rotated BRIEF (ORB) to elucidate the best approach. The framework was validated in 23 patients and the 3D triangulation error of the matched features was < mm. Thus, the findings indicate that spine feature detection can be used for accurate tracking in navigated surgery.

Structural Toughness Under Noise: An Efficient No-Reference Image Distortion Assessment for Blur and Noise

In image denoising and reconstruction problems, it is useful to monotonically quantify the distortions in images representing the same scene. For this purpose, we propose a training-free no-reference image distortion assessment method for both blur and noise. The method is based on the observation that the structural similarity between an input image and its shifted-and-noised copy is related to the levels of blur and noise distorting the input image. Computing a noised copy would require a random number generation, but assuming virtual noise independent from the input image, we derived the method that does not require computing actual noised copy. In our experiments of assessing the singly/multiply distorted images representing the same scene, the proposed method generally showed better monotonicity than competing state-of-the-art methods. The proposed method runs about 80 times faster than those competing methods and it can assess local regions of the input image, which makes it useful in spatially adaptive denoising applications.

Fast and Consistent Matching for Landmark-Based Place Recognition

Severe condition changes make visual place recognition a difficult task for robotics applications. Landmark-based methods that combine local image regions and features from a convolution neural network (CNN) have shown state-of-the-art performance. This paper focuses on improving the performance of landmark-based methods for place recognition. To save time in landmark matching, a novel method is presented to extract landmarks based directly on the lower layer of CNN features instead of using object proposals or superpixels. The centers of landmarks are projected onto the feature maps of the higher layer of the CNN to generate compact descriptors. We then extract potential images through similarities computed by direct matching. In the refinement stage, we exploit the Markov random field (MRF), combined with Bayes filters, to perform robust matchings. This stage integrates the spatial information of landmarks and introduces a new method for similarity measurements. We present extensive experiments using challenging datasets and compare the results to state-of-the-art methods to evaluate our approach. The results show that our approach ensures a remarkable performance under severe condition changes and significantly improves efficiency.

2DKD: a toolkit for content-based local image search

BackgroundDirect comparison of 2D images is computationally inefficient due to the need for translation, rotation, and scaling of the images to evaluate their similarity. In many biological applications, such as digital pathology and cryo-EM, often identifying specific local regions of images is of particular interest. Therefore, finding invariant descriptors that can efficiently retrieve local image patches or subimages becomes necessary.ResultsWe present a software package called Two-Dimensional Krawtchouk Descriptors that allows to perform local subimage search in 2D images. The new toolkit uses only a small number of invariant descriptors per image for efficient local image retrieval. This enables querying an image and comparing similar patterns locally across a potentially large database. We show that these descriptors appear to be useful for searching local patterns or small particles in images and demonstrate some test cases that can be helpful for both assembly software developers and their users.ConclusionsLocal image comparison and subimage search can prove cumbersome in both computational complexity and runtime, due to factors such as the rotation, scaling, and translation of the object in question. By using the 2DKD toolkit, relatively few descriptors are developed to describe a given image, and this can be achieved with minimal memory usage.

Construction Algorithm for Adaptive Morphological Structuring Elements Based on the Neighborhood Gray Difference Changing Vector Field and Relative Density

Structuring elements of fixed shape and size are used in most conventional mathematical morphology operations, which makes the border of image targets shift, produces new image artifacts and loses small image objects due to the diversity and complexity of the image targets. In this paper, a new construction algorithm for adaptive structuring elements is proposed based on the neighborhood gray difference changing vector field and relative density. The proposed structuring element is able to adaptively change shape according to the gray and edge characteristics of an image. This algorithm involves first incorporating the gray difference changing vector field to smooth the local image region and make the gray level within the image target more uniform and then defining a border degree function based on relative density to determine whether the center pixel of the local image region is a border pixel. The adaptive structuring element is composed of all the strong border pixels found in a local image region. Dilation and erosion operations and other derivative operations are proposed with this new adaptive structuring element based on conventional morphology operation principles. The experimental results show that this proposed algorithm is able to effectively suppress the shifting effect of the image target borders while accurately locating the border of the image target region. Additionally, other effective image information is retained and image distortion is reduced while weakening the image details.

Multi-class Image Classification Using Deep Learning Algorithm

Classifying images is a complex problem in the field of computer vision. The deep learning algorithm is a computerized model simulates the human brain functions and operations. Training the deep learning model is a costly process in machine resources and time. Investigating the performance of the deep learning algorithm is mostly needed. The convolutional neural network (CNN) is most commonly used to build a structure of the deep learning models. In this paper convolutional neural network (CNN) model pre-trained on Image-Net is used for classification of images of the PASCAL VOC 2007 data-set. The transfer learning approach is used to improve the performance of the deep learning CNN model where classification works fairly well with the smallest amount of computation time and fewer machine resources. The behavior of the Deep learning CNN model is studied and the performance has been measured. The obtained results are compared with the obtained test results from the Super-vector coding of local image descriptors method, SVM method, and Region Ranking SVM method, which tested with the PASCAL VOC 2007 data-set. The final results evaluate the deep learning algorithm as a state-of-the-art method for an image classification task.