Feature Image Registration Research Articles

Background Instance-Based Copy-Paste Data Augmentation for Object Detection

In supervised deep learning object detection, the quantity of object information and annotation quality in a dataset affect model performance. To augment object detection datasets while maintaining contextual information between objects and backgrounds, we proposed a Background Instance-Based Copy-Paste (BIB-Copy-Paste) data augmentation model. We devised a method to generate background pseudo-labels for all object classes by calculating the similarity between object background features and image region features in Euclidean space. The background classifier, trained with these pseudo-labels, can guide copy-pasting to ensure contextual relevance. Several supervised object detectors were evaluated on the PASCAL VOC 2012 dataset, achieving a 1.1% average improvement in mean average precision. Ablation experiments with the BlitzNet object detector on the PASCAL VOC 2012 dataset showed an improvement of mAP by 1.19% using the proposed method, compared to a 0.18% improvement with random copy-paste. Images from the MS COCO dataset containing objects of the same classes as in PASCAL VOC 2012 were also selected for object pasting experiments. The contextual relevance of pasted objects demonstrated our model’s effectiveness and transferability between datasets with same class of objects.

The Correlation between Macroscopic Image and Object Properties with Bubble Size in Flotation

This paper studies the correlation between different macroscopic features of image regions and object properties with the Sauter diameter (D32) of bubble size in flotation. Bubbles were sampled from the collection zone of a two-dimensional flotation cell using a McGill Bubble Size Analyzer, and photographed bubbles were processed using image analysis. The Sauter mean diameters were obtained under different experimental conditions using a semiautomated methodology, in which non-identifiable bubbles were manually characterized to estimate the bubble size distribution. For the same processed images, different image properties from their binary representation were studied in terms of their correlation with D32. The median and variability of the shadow percentage, aspect ratio, power spectral density, perimeter, equivalent diameters, solidity, and circularity, among other image or object properties, were studied. These properties were then related to the measured D32 values, from which four predictors were chosen to obtain a multivariable model that adequately described the Sauter diameter. After removing abnormal gas dispersion conditions, the multivariable linear model was able to represent D32 values (99 datasets) for superficial gas rates in the range of 0.4–2.5 cm/s, for four types of frothers and surfactant concentrations ranging from 0 to 32 ppm. The model was tested with 72 independent datasets, showing the generalizability of the results. Thus, the approach proved to be applicable at the laboratory scale for D32 = 1.3–6.7 mm.

RFE-SRN: Image-text similarity reasoning network based on regional feature enhancement

Image-text matching aims to make a connection between visual and natural information. Some of the current methods have made great progress by using global alignment between images and sentences and local alignment between the image region and its corresponding word. However, the importance of the correlation between global alignment and local alignment is ignored to some extends. Therefore, in this paper, we propose a new region feature enhancement-based image text similarity inference network. Firstly, the image region feature is enhanced by graph convolutional neural network, which is used to find the correlation among different image region features with the production of features’ sematic relationship. Secondly, we propose a vector-based similarity representation to describe local and global alignment with a more comprehensive way. Finally, a graph convolution neural network is introduced to construct a similarity graph for propagating correlation between local alignment and global alignment to every part. By testing on the MSCOCO and Flickr30k dataset, our proposed method shows great accuracy performance and competitiveness.

Double-Stream Position Learning Transformer Network for Image Captioning

Image captioning has made significant achievement through developing feature extractor and model architecture. Recently, the image region features extracted by object detector prevail in most existing models. However, region features are criticized for the lacking of background and full contextual information. This problem can be remedied by providing some complementary visual information from patch features. In this paper, we propose a Double-Stream Position Learning Transformer Network (DSPLTN) which exploits the advantages of region features and patch features. Specifically, the region-stream encoder utilizes a Transformer encoder with Relative Position Learning (RPL) module to enhance the representations of region features through modeling the relationships between regions and positions respectively. As for the patch-stream encoder, we introduce convolutional neural network into the vanilla Transformer encoder and propose a novel Convolutional Position Learning (CPL) module to encode the position relationships between patches. CPL improves the ability of relationship modeling by combining the position and visual content of patches. Incorporating CPL into the Transformer encoder can synthesize the benefits of convolution in local relation modeling and self-attention in global feature fusion, thereby compensating for the information loss caused by the flattening operation of 2D feature maps to 1D patches. Furthermore, an Adaptive Fusion Attention (AFA) mechanism is proposed to balance the contribution of enhanced region and patch features. Extensive experiments on MSCOCO demonstrate the effectiveness of the double-stream encoder and CPL, and show the superior performance of DSPLTN.

Attention-Aligned Transformer for Image Captioning

Recently, attention-based image captioning models, which are expected to ground correct image regions for proper word generations, have achieved remarkable performance. However, some researchers have argued “deviated focus” problem of existing attention mechanisms in determining the effective and influential image features. In this paper, we present A2 - an attention-aligned Transformer for image captioning, which guides attention learning in a perturbation-based self-supervised manner, without any annotation overhead. Specifically, we add mask operation on image regions through a learnable network to estimate the true function in ultimate description generation. We hypothesize that the necessary image region features, where small disturbance causes an obvious performance degradation, deserve more attention weight. Then, we propose four aligned strategies to use this information to refine attention weight distribution. Under such a pattern, image regions are attended correctly with the output words. Extensive experiments conducted on the MS COCO dataset demonstrate that the proposed A2 Transformer consistently outperforms baselines in both automatic metrics and human evaluation. Trained models and code for reproducing the experiments are publicly available.

Conditional Feature Learning Based Transformer for Text-Based Person Search.

Text-based person search aims at retrieving the target person in an image gallery using a descriptive sentence of that person. The core of this task is to calculate a similarity score between the pedestrian image and description, which requires inferring the complex latent correspondence between image sub-regions and textual phrases at different scales. Transformer is an intuitive way to model the complex alignment by its self-attention mechanism. Most previous Transformer-based methods simply concatenate image region features and text features as input and learn a cross-modal representation in a brute force manner. Such weakly supervised learning approaches fail to explicitly build alignment between image region features and text features, causing an inferior feature distribution. In this paper, we present CFLT, Conditional Feature Learning based Transformer. It maps the sub-regions and phrases into a unified latent space and explicitly aligns them by constructing conditional embeddings where the feature of data from one modality is dynamically adjusted based on the data from the other modality. The output of our CFLT is a set of similarity scores for each sub-region or phrase rather than a cross-modal representation. Furthermore, we propose a simple and effective multi-modal re-ranking method named Re-ranking scheme by Visual Conditional Feature (RVCF). Benefit from the visual conditional feature and better feature distribution in our CFLT, the proposed RVCF achieves significant performance improvement. Experimental results show that our CFLT outperforms the state-of-the-art methods by 7.03% in terms of top-1 accuracy and 5.01% in terms of top-5 accuracy on the text-based person search dataset.

Latent Space Semantic Supervision Based on Knowledge Distillation for Cross-Modal Retrieval.

As an important field in information retrieval, fine-grained cross-modal retrieval has received great attentions from researchers. Existing fine-grained cross-modal retrieval methods made several improvements in capturing the fine-grained interplay between vision and language, failing to consider the fine-grained correspondences between the features in the image latent space and the text latent space respectively, which may lead to inaccurate inference of intra-modal relations or false alignment of cross-modal information. Considering that object detection can get the fine-grained correspondences of image region features and the corresponding semantic features, this paper proposed a novel latent space semantic supervision model based on knowledge distillation (L3S-KD), which trains classifiers supervised by the fine-grained correspondences obtained from an object detection model by using knowledge distillation for image latent space fine-grained alignment, and by the labels of objects and attributes for text latent space fine-grained alignment. Compared with existing fine-grained correspondence matching methods, L3S-KD can learn more accurate semantic similarities for local fragments in image-text pairs. Extensive experiments on MS-COCO and Flickr30K datasets demonstrate that the L3S-KD model consistently outperforms state-of-the-art methods for image-text matching.

VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning

It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated in the novel object captioning challenge (nocaps). In this challenge, no additional image-caption training data, other than COCO Captions, is allowed for model training. Thus, conventional Vision-Language Pre-training (VLP) methods cannot be applied. This paper presents VIsual VOcabulary pre-training (VIVO) that performs pre-training in the absence of caption annotations. By breaking the dependency of paired image-caption training data in VLP, VIVO can leverage large amounts of paired image-tag data to learn a visual vocabulary. This is done by pre-training a multi-layer Transformer model that learns to align image-level tags with their corresponding image region features. To address the unordered nature of image tags, VIVO uses a Hungarian matching loss with masked tag prediction to conduct pre-training. We validate the effectiveness of VIVO by fine-tuning the pre-trained model for image captioning. In addition, we perform an analysis of the visual-text alignment inferred by our model. The results show that our model can not only generate fluent image captions that describe novel objects, but also identify the locations of these objects. Our single model has achieved new state-of-the-art results on nocaps and surpassed the human CIDEr score.

A Novel Region-Based Image Registration Method for Multisource Remote Sensing Images Via CNN

The comprehensive utilization of images from various satellite sensors can significantly increase the performance of remote sensing applications and has, therefore, attracted extensive research attention. One of the essential challenges that research encounters comes from multisource image registration. This article proposes a novel region-based image registration method for multisource images. The proposed method exploits the region features of input images, which provide more consistent and common information of the multisource data. The image region features are extracted based on image semantic segmentation using the deep convolutional neural network approach. The final registration result is a pixel-level output corresponding to the input images. The proposed registration scheme overcomes the limits of traditional feature extraction methods (e.g., point feature) adopted in previous registration schemes. Results indicate that the proposed method has good performance for the multisource remote sensing image registration and can serve as a building block for the fusion of multisource images.

Covariance descriptors on a Gaussian manifold and their application to image set classification

Covariance descriptors (CovDs) for image set classification have been widely studied recently. Different from the conventional CovDs, which describe similarities between pixels at different locations, we focus more on similarities between regions that convey more comprehensive information. In this paper, we extract pixel-wise features of image regions and represent them by Gaussian models. We extend the conventional covariance computation onto a special type of Riemannian manifold, namely a Gaussian manifold, so that it is applicable to our image set data representation provided in terms of Gaussian models. We present two methods to calculate a Riemannian local difference vector on the Gaussian manifold (RieLDV-G) and generate our proposed Riemannian covariance descriptors (RieCovDs) using the resulting RieLDV-G. By measuring the recognition accuracy achieved on benchmarking datasets, we demonstrate experimentally the superior performance of our proposed RieCovDs descriptors, as compared with state-of-the-art methods. (The code is available at:https://github.com/Kai-Xuan/RiemannianCovDs)

Selective residual learning for Visual Question Answering

Visual Question Answering (VQA) aims to reason an answer, given a textual question and image pair. VQA methods are required to learn the relationship between image region features. These methods have the limitation of inefficient learning that can produce a performance drop. It is because current intra-relationship methods are trying to learn all the intra-relationships, regardless of their importance. In this paper, a novel self-attention based VQA module named Selective Residual learning (SelRes) is proposed. SelRes processes the residual learning selectively in self-attention networks. It measures the importance of the input vectors by the attention map and limits residual learning, except in the selected regions which related to the correct answer. Selective masking is also proposed, which can ensure that the selection in SelRes is preserved in the multi-stack structure of the VQA network. Our full model achieves new state-of-the-art performances on both from-scratch and fine-tuning models.

A Multitemporal Remote Sensing Image Registration Method Based on Water Bodies for the Lake-Rich Region

Accurate multitemporal remotely sensed image registration is essential for water body change detection and analysis of lake dynamics. However, lake-rich regions, such as Siberia, are mostly dominated by rivers and lakes, with few stable geometry features such as those typically used as control points for image registration. Lakes in Arctic regions are generally not static; their shorelines tend to expand and shrink seasonally, and may change substantially between different years, making it difficult to find consistent features for image registration. Consequently, traditional image-to-image registration methods, and even many sophisticated registration algorithms, rarely achieve accurate geometric correction in Arctic regions due to a lack of sufficient control points. In this article, we proposed a summary lake spatial attribute, the inferred deepest point of the lake (DPL), as a feature that is relatively insensitive to lake area changes, and therefore useful for registration of multitemporal images. The central focus of the DPL estimation algorithm is the identification of the largest inner circle (LIC) of the lake polygon. First, the internal Voronoi diagram of a lake polygon is calculated by the ``divide-and-conquer” method. The medial axis (MA) is then calculated by Voronoi diagram simplification, and finally, the LIC center is obtained by computing the distance from the MA intersection to all polygon edges. The approach was used to register HJ-1 and Landsat multispectral scanner (MSS) images in Siberia, where water bodies dominate the landscape and change significantly over time. The proposed method found a large number of control points from the extracted water bodies. Subpixel registration accuracy of 0.62 pixels (18.5 m) and 0.33 pixels (19.6 m) root mean square error (RMSE) was obtained for the HJ-1 and MSS images, respectively. In comparison, the alternative method of using lake centroids, only achieved 0.75 pixels (22.4 m) and 0.49 pixels (29.3 m) RMSE. This registration accuracy improvement is potentially important for large-scale regional cartography and change detection applications.

A soft image representation approach by exploiting local neighborhood structure of self-organizing map (SOM)

When images are described with visual words based on vector quantization of low-level color, texture, and edge-related visual features of image regions, it is usually referred as "bag-of-visual words (BoVW)"-based presentation. Although it has proved to be effective for image representation similar to document representation in text retrieval, the hard image encoding approach based on one-to-one mapping of regions to visual words is not expressive enough to characterize the image contents with higher level semantics and prone to quantization error. Each word is considered independent of all the words in this model. However, it is found that the words are related and their similarity of occurrence in documents can reflect the underlying semantic relations between them. To consider this, a soft image representation scheme is proposed by spreading each region's membership values through a local fuzzy membership function in a neighborhood to all the words in a codebook generated by self-organizing map (SOM). The topology preserving property of the SOM map is exploited to generate a local membership function. A systematic evaluation of retrieval results of the proposed soft representation on two different image (natural photographic and medical) collections has shown significant improvement in precision at different recall levels when compared to different low-level and "BoVW"-based feature that consider only probability of occurrence (or presence/absence) of a word.

Learning Natural Colors for Image Recoloring

AbstractWe present a data‐driven method for automatically recoloring a photo to enhance its appearance or change a viewer's emotional response to it. A compact representation called a RegionNet summarizes color and geometric features of image regions, and geometric relationships between them. Correlations between color property distributions and geometric features of regions are learned from a database of well‐colored photos. A probabilistic factor graph model is used to summarize distributions of color properties and generate an overall probability distribution for color suggestions. Given a new input image, we can generate multiple recolored results which unlike previous automatic results, are both natural and artistic, and compatible with their spatial arrangements.

Photo Retrieval Based on Spatial Layout with Hardware Acceleration for Mobile Devices

A new photo retrieval system for mobile devices is proposed. The system can be used to search for photos with similar spatial layouts efficiently, and it adopts an image segmentation algorithm that extracts features of image regions based on K-Means clustering. Since K-Means is computationally intensive for real-time applications and prone to generate clustering results with local optima, parallel hardware architectures are designed to meet the real-time requirement of the retrieval process. Experiments show that the proposed algorithm in the photo retrieval system obtains better mean average precision than other methods, and it is tested with image recognition problems. The robustness of the algorithm is also evaluated with noise and image blurring. Besides, the proposed K-Means hardware can provide a trade-off between the execution time and the retrieval performance on the software and hardware cosimulation platform. The contribution of this work is twofold. The first is the development of a photo retrieval framework for mobile devices, where a new texture feature is employed in the algorithm to enhance the retrieval performance. The other is the integration of the K-Means hardware accelerator and the photo retrieval system. The hardware architecture is analyzed, and the specifications are compared with previous works.

Multiple source imagery and linear features for detection of urban expansion in Aqaba City, Jordan

Aqaba is one of the most strategic cities in Jordan and the entire region, as it is the only seaport for Jordan and has a special economic zone as the only window to global markets. The main purpose of this study is to detect urban development in Aqaba region by detecting and registering linear features in images with various geometric and radiometric properties taken at different times. This article used linear features for image registration that were chosen since they can be reliably extracted from imagery with significantly different geometric and radiometric properties. The modified iterated Hough transform (MIHT) is used as the matching strategy for automatically deriving an estimate of the parameters involved in the transformation function relating the images to be registered as well as the correspondence between conjugate lines. Derived edges from the registered images are used as the basis for change detection. The utilization of edges is motivated by the fact that they are invariant with respect to possible radiometric differences between the images in question. Linear features extraction, feature matching, image registration and pixel–pixel subtraction have been implemented using SPOT, Landsat, Ikonos and aerial photographs that have different radiometric, spatial and temporal resolutions. It has been shown that linear features (straight-line segments) have high semantics and can be reliably extracted from the images. These linear features can be used for accurate co-registration as an essential prerequisite for a reliable change detection procedure. For the purpose of change detection, image–image registration is more crucial than image–ground registration, where corresponding features in images are registered with respect to each other regardless of the associated absolute errors. The results illustrate that using edges as the base for change detection in urban areas is efficient and reliable.

Pulmonary nodule feature extraction based on short CT image series

Concerning the limitation of using single CT image to detect the lung nodule, short image series consisting of a few sequential CT image was used in nodule's auto-detection in this paper. Meanwhile, the image corresponding to the Region of Interest (ROI) was taken as a surface of some 2-d function. Then the new features, different from traditional image region features, were extracted, which depicted the surface's shape and its variances in short image series. At last, the effectiveness of the extracted features is proved by using the Support Vector Machine (SVM).

Interactive image registration tool for positioning verification in head and neck radiotherapy

This paper presents the development of a computer-aided image management tool, named CAIMT, used for patient setup error measurement in radiation treatment planning. The CAIMT incorporates user interaction to facilitate contour detection from both portal film and simulation film. The detected contours were then used as features for image registration through application of the generalized Hough transform (GHT). Positioning error was measured when the optimal registration was achieved. The CAIMT has been applied to register the image pairs of a rando TM phantom and five real subjects, in order to evaluate its reliability and stability. The promising results suggest that the CAIMT can assist radiotherapists to align the simulation film and the portal film precisely and steadily and therefore adjust patient positioning to the optimum in a more efficient way.