Query Image Research Articles

Power amplifier circuit defect detection based on improved Patch SVDD

AbstractIn the manufacturing process of power amplifier (PA), various defects on the circuit surface will seriously affect the circuit performance and its operation. To solve the above problems, this article proposes a circuit defect detection method based on improved patch‐based support vector data description algorithm (Patch SVDD), which uses contrastive learning to enhance the feature extraction ability of neural network. An image calibration algorithm is used for calibration preprocessing to enhance the robustness of the model. The Euclidean distance between a query image patch and its nearest normal patch is defined to be an anomaly score. For verification, three defects in the PA circuit, including components missing, solder contamination and component orientation dislocation, were detected. The experimental results show that, compared with conventional method, the area under receiver operating characteristic (AUROC) of the improved anomaly detection model increased from 90.1% to 95.1%, which improves the detection accuracy effectively.

PatchMix Augmentation to Identify Causal Features in Few-shot Learning.

The task of Few-shot learning (FSL) aims to transfer the knowledge learned from base categories with sufficient labelled data to novel categories with scarce known information. It is currently an important research question and has great practical values in the real-world applications. Despite extensive previous efforts are made on few-shot learning tasks, we emphasize that most existing methods did not take into account the distributional shift caused by sample selection bias in the FSL scenario. Such a selection bias can induce spurious correlation between the semantic causal features, that are causally and semantically related to the class label, and the other non-causal features. Critically, the former ones should be invariant across changes in distributions, highly related to the classes of interest, and thus well generalizable to novel classes, while the latter ones are not stable to changes in the distribution. To resolve this problem, we propose a novel data augmentation strategy dubbed as PatchMix that can break this spurious dependency by replacing the patch-level information and supervision of the query images with random gallery images from different classes from the query ones. We theoretically show that such an augmentation mechanism, different from existing ones, is able to identify the causal features. To further make these features to be discriminative enough for classification, we propose Correlation-guided Reconstruction (CGR) and Hardness-Aware module for instance discrimination and easier discrimination between similar classes. Moreover, such a framework can be adapted to the unsupervised FSL scenario. The utility of our method is demonstrated on the state-of-the-art results consistently achieved on several benchmarks including miniImageNet, tieredImageNet, CIFAR-FS , CUB, Cars, Places and Plantae, in all settings of single-domain, cross-domain and unsupervised FSL. By studying the intra-variance property of learned features and visualizing the learned features, we further quantitatively and qualitatively show that such a promising result is due to the effectiveness in learning causal features.

ISimLoc: Visual Global Localization for Previously Unseen Environments With Simulated Images

The camera is an attractive device for use in beyond visual line of sight drone operation since cameras are low in size, weight, power, and cost. However, state-of-the-art visual localization algorithms have trouble matching visual data that have significantly different appearances due to changes in illumination or viewpoint. This article presents iSimLoc, a learning-based global relocalization approach that is robust to appearance and viewpoint differences. The features learned by iSimLoc's place recognition network can be utilized to match query images to reference images of a different stylistic domain and viewpoint. In addition, our hierarchical global relocalization module searches in a coarse-to-fine manner, allowing iSimLoc to perform fast and accurate pose estimation. We evaluate our method on a dataset with appearance variations and a dataset that focuses on demonstrating large-scale matching over a long flight over complex terrain. iSimLoc achieves 88.7% and 83.8% successful retrieval rates on our two datasets, with 1.5 s inference time, compared to 45.8% and 39.7% using the next best method. These results demonstrate robust localization in a range of environments and conditions.

Enhanced descriptive captioning model for histopathological patches

AbstractThe interpretation of medical images into a natural language is a developing field of artificial intelligence (AI) called image captioning. This field integrates two branches of artificial intelligence which are computer vision and natural language processing. This is a challenging topic that goes beyond object recognition, segmentation, and classification since it demands an understanding of the relationships between various components in an image and how these objects function as visual representations. The content-based image retrieval (CBIR) uses an image captioning model to generate captions for the user query image. The common architecture of medical image captioning systems consists mainly of an image feature extractor subsystem followed by a caption generation lingual subsystem. We aim in this paper to build an optimized model for histopathological captions of stomach adenocarcinoma endoscopic biopsy specimens. For the image feature extraction subsystem, we did two evaluations; first, we tested 5 different vision models (VGG, ResNet, PVT, SWIN-Large, and ConvNEXT-Large) using (LSTM, RNN, and bidirectional-RNN) and then compare the vision models with (LSTM-without augmentation, LSTM-with augmentation and BioLinkBERT-Large as an embedding layer-with augmentation) to find the accurate one. Second, we tested 3 different concatenations of pairs of vision models (SWIN-Large, PVT_v2_b5, and ConvNEXT-Large) to get among them the most expressive extracted feature vector of the image. For the caption generation lingual subsystem, we tested a pre-trained language embedding model which is BioLinkBERT-Large compared to LSTM in both evaluations, to select from them the most accurate model. Our experiments showed that building a captioning system that uses a concatenation of the two models ConvNEXT-Large and PVT_v2_b5 as an image feature extractor, combined with the BioLinkBERT-Large language embedding model produces the best results among the other combinations.

A Large-scale Virtual Dataset and Egocentric Localization for Disaster Responses.

With the increasing social demands of disaster response, methods of visual observation for rescue and safety have become increasingly important. However, because of the shortage of datasets for disaster scenarios, there has been little progress in computer vision and robotics in this field. With this in mind, we present the first large-scale synthetic dataset of egocentric viewpoints for disaster scenarios. We simulate pre- and post-disaster cases with drastic changes in appearance, such as buildings on fire and earthquakes. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for the semantic label, depth in metric scale, optical flow with sub-pixel precision, and surface normal as well as their corresponding camera poses. To create realistic disaster scenes, we manually augment the effects with 3D models using physically-based graphics tools. We train various state-of-the-art methods to perform computer vision tasks using our dataset, evaluate how well these methods recognize the disaster situations, and produce reliable results of virtual scenes as well as real-world images. We also present a convolutional neural network-based egocentric localization method that is robust to drastic appearance changes, such as the texture changes in a fire, and layout changes from a collapse. To address these key challenges, we propose a new model that learns a shape-based representation by training on stylized images, and incorporate the dominant planes of query images as approximate scene coordinates. We evaluate the proposed method using various scenes including a simulated disaster dataset to demonstrate the effectiveness of our method when confronted with significant changes in scene layout. Experimental results show that our method provides reliable camera pose predictions despite vastly changed conditions.

Attentional prototype inference for few-shot segmentation

This paper aims to address few-shot segmentation. While existing prototype-based methods have achieved considerable success, they suffer from uncertainty and ambiguity caused by limited labeled examples. In this work, we propose attentional prototype inference (API), a probabilistic latent variable framework for few-shot segmentation. We define a global latent variable to represent the prototype of each object category, which we model as a probabilistic distribution. The probabilistic modeling of the prototype enhances the model’s generalization ability by handling the inherent uncertainty caused by limited data and intra-class variations of objects. To further enhance the model, we introduce a local latent variable to represent the attention map of each query image, which enables the model to attend to foreground objects while suppressing the background. The optimization of the proposed model is formulated as a variational Bayesian inference problem, which is established by amortized inference networks. We conduct extensive experiments on four benchmarks, where our proposal obtains at least competitive and often better performance than state-of-the-art prototype-based methods. We also provide comprehensive analyses and ablation studies to gain insight into the effectiveness of our method for few-shot segmentation.

Learned Data-aware Image Representations of Line Charts for Similarity Search

Finding line-chart images similar to a given line-chart image query is a common task in data exploration and image query systems, e.g. finding similar trends in stock markets or medical Electroencephalography images. The state-of-the-art approaches consider either data-level similarity (when the underlying data is present) or image-level similarity (when the underlying data is absent). In this paper, we study the scenario that during query time, only line-chart images are available. Our goal is to train a neural network that can turn these line-chart images into representations that are aware of the data used to generate these line charts, so as to learn better representations. Our key idea is that we can collect both data and line-chart images to learn such a neural network (at training step), while during query (or inference) time, we support the case that only line-chart images are provided. To this end, we present LineNet, a Vision Transformer-based Triplet Autoencoder model to learn data-aware image representations of line charts for similarity search. We design a novel pseudo labels selection mechanism to guide LineNet to capture both data-aware and image-level similarity of line charts. We further propose a diversified training samples selection strategy to optimize the learning process and improve the performance. We conduct both quantitative evaluation and case studies, showing that LineNet significantly outperforms the state-of-the-art methods for searching similar line-chart images.

Efficient improvement of classification accuracy via selective test-time augmentation

Test-time augmentation (TTA) is typically used in image classification tasks to improve the classification accuracy of convolutional neural networks (CNNs). In the inference phase, TTA aggregates the predictions of a CNN over multiple random transformations of a query image to obtain the final prediction. However, TTA incurs a high computational cost owing to its effectiveness. In this study, we propose a selective-TTA method for reducing computational costs while maintaining the high classification accuracy. Given a query image, we quantify the predictive uncertainty of the CNN on the image to determine whether TTA should be performed. We perform TTA to obtain an enhanced prediction only when the uncertainty is high. Experimental results on benchmark datasets demonstrated that the proposed method significantly reduced average computational costs while attaining comparable or higher classification accuracy than conventional full-TTA. When the proportion of performing TTA was set to 10%, the computational cost was reduced by 90% with no performance degradation on most datasets.

Learning global image representation with generalized‐mean pooling and smoothed average precision for large‐scale CBIR

AbstractContent‐based image retrieval (CBIR) is the problem of searching for items in an image database that are similar to the query image. Most of the existing image retrieval methods are trained based on metric learning loss functions (e.g. contrastive loss or triplet loss), however, which require the use of hard sample mining strategies (HMS) to better train the model. The HMS implies that picking out hard positive or negative samples increases the complexity of model training and requires a large amount of additional training time. To address this issue, lessons from recent work are leveraged on representation learning and a model called GS is proposed that combines the state‐of‐the‐art Generalized‐Mean (GeM) pooling and the smoothed average precision (AP). The entire network can be learned end‐to‐end by approximating the non‐differentiable AP function to a differentiable one‐without mining hard samples, only image‐level annotations. A model named GSA is also presented which achieves excellent retrieval performance jointly trained by two various loss functions. Experimental results validate the effectiveness of the proposed approach and demonstrate the competitive performance on a common standard image retrieval dataset (Revisited Oxford and Paris).

ConCoNet: Class-agnostic counting with positive and negative exemplars

Class-agnostic counting is usually phrased as a matching problem between a user-defined exemplar patch and a query image. The count is derived based on the number of objects similar to the exemplar patch. However, defining a target class using only positive exemplar patches inevitably miscounts unintended objects that are visually alike to the exemplar. In this paper, we propose to include negative exemplars that define what not to count. This allows the model to calibrate its notion of what is similar based on both positive and negative exemplars. It effectively disentangles visually similar negatives, leading to a more discriminative definition of the target object. We designed our method such that it can be incorporated with other class-agnostic counting models. Moreover, application-wise, our model can be used into a semi-automatic labeling tool to simplify the job of the annotator

Gaze-Dependent Image Re-Ranking Technique for Enhancing Content-Based Image Retrieval

Content-based image retrieval (CBIR) aims to find desired images similar to the image input by the user, and it is extensively used in the real world. Conventional CBIR methods do not consider user preferences since they only determine retrieval results by referring to the degree of resemblance or likeness between the query and potential candidate images. Because of the above reason, a “semantic gap” appears, as the model may not accurately understand the potential intention that a user has included in the query image. In this article, we propose a re-ranking method for CBIR that considers a user’s gaze trace as interactive information to help the model predict the user’s inherent attention. The proposed method uses the user’s gaze trace corresponding to the image obtained from the initial retrieval as the user’s preference information. We introduce image captioning to effectively express the relationship between images and gaze information by generating image captions based on the gaze trace. As a result, we can transform the coordinate data into a text format and explicitly express the semantic information of the images. Finally, image retrieval is performed again using the generated gaze-dependent image captions to obtain images that align more accurately with the user’s preferences or interests. The experimental results on an open image dataset with corresponding gaze traces and human-generated descriptions demonstrate the efficacy or efficiency of the proposed method. Our method considers visual information as the user’s feedback to achieve user-oriented image retrieval.

A cross-view geo-localization method guided by relation-aware global attention

Cross-view geo-localization mainly exploits query images to match images from the same geographical location from different platforms. Most existing methods fail to adequately consider the effect of image structural information on cross-view geo-localization, resulting in the extracted features can not fully characterize the image, which affects the localization accuracy. Based on this, this paper proposes a cross-view geo-localization method guided by relation-aware global attention, which can capture the rich global structural information by perfectly integrating attention mechanism and feature extraction network, thus improving the representation ability of features. Meanwhile, considering the important role of semantic and context information in geo-localization, a joint training structure with parallel global branch and local branch is designed to fully mine multi-scale context features for image matching, which can further improve the accuracy of cross-view geo-localization. The quantitative and qualitative experimental results on University-1652, CVUSA, and CVACT datasets show that the algorithm in this paper outperforms other advanced methods in recall accuracy (Recall) and image retrieval average precision (AP).

Semantic-guided de-attention with sharpened triplet marginal loss for visual place recognition

Thanks to Earth-level Street View images from Google Maps, a visual image geo-localization can estimate the coarse location of a query image with a visual place recognition process. However, this can get very challenging when non-static objects change with time, severely degrading image retrieval accuracy. We address the problem of city-scale visual place recognition in complex urban environments crowded with non-static clutters. To this end, we first analyze what clutters degrade similarity matching between the query and database images. Second, we design a self-supervised trainable de-attention module that prevents the network from focusing on non-static objects in an input image. In addition, we propose a novel triplet marginal loss called sharpened triplet marginal loss to make feature descriptors more discriminative. Lastly, due to the lack of geo-tagged public datasets with a high density of non-static objects, we propose a clutter augmentation method to evaluate our approach. The experimental results show that our model has notably improved over the existing attention methods in geo-localization tasks on the public benchmark datasets and on their augmented versions with high population and traffic. Our code is available at https://github.com/ccsmm78/deattention_with_stml_for_vpr.

Query-based image tagging model using ensemble learning with enhanced artificial bee colony optimization

Digital images make up most multimedia data and are analysed in computer vision applications. Daily uploads of millions of pictures to Internet archives such as satellite image repositories complicate multimedia content and image graphs. As feature vectors, content based image retrieval (CBIR) and image classification models represent high-level image viewpoints. Observing photos recognizes objects and evaluates their significance for image enhancement. To access the visual information of big datasets, efficiently retrieve and query picture graphs. The artificial bee colony (ABC) algorithm is inspired by the foraging behaviour of honeybee swarms. ABC is susceptible to laziness in convergence and local optimums, just like other optimization methods. This study created an enhanced ABC (EABC) model to enhance precision. This study presents query-based image tagging model using ensemble learning with EABC (QbITM-ELEABC) for CBIR for appropriately tagging images based on the query image. We examine a number of convolutional neural network (CNNs) with varying topologies that can be trained on the dataset with varying degrees of similarity. As representations, each network extracts class probability vectors from images. The final image representation is created by combining the ensemble's class probability vectors with image.

Quaternion-Valued Correlation Learning for Few-Shot Semantic Segmentation

Few-shot segmentation (FSS) aims to segment unseen classes given only a few annotated samples. Encouraging progress has been made for FSS by leveraging semantic features learned from base classes with sufficient training samples to represent novel classes. The correlation-based methods lack the ability to consider interaction of the two subspace matching scores due to the inherent nature of the real-valued 2D convolutions. In this paper, we introduce a quaternion perspective on correlation learning and propose a novel Quaternion-valued Correlation Learning Network (QCLNet), with the aim to alleviate the computational burden of high-dimensional correlation tensor and explore internal latent interaction between query and support images by leveraging operations defined by the established quaternion algebra. Specifically, our QCLNet is formulated as a hyper-complex valued network and represents correlation tensors in the quaternion domain, which uses quaternion-valued convolution to explore the external relations of query subspace when considering the hidden relationship of the support sub-dimension in the quaternion space. Extensive experiments on the PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>i</i></sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i>i</i></sup> datasets demonstrate that our method outperforms the existing state-of-the-art methods effectively.

Cross domain 2D-3D descriptor matching for unconstrained 6-DOF pose estimation

This paper presents a novel approach for cross-domain descriptor matching between 2D and 3D modalities. The 2D-3D matching is applied to localize 2D images in 3D point clouds. Direct cross-domain matching allows our technique to localize images in any type of 3D point cloud without any constraints on the nature or mechanism by which it is obtained. We propose a learning based framework, called Desc-Matcher, to directly match features between the two modalities. A dataset of 2D and 3D features with corresponding locations in images and point clouds is generated to train the Desc-Matcher. To estimate the pose of an image in any 3D cloud, keypoints and feature descriptors are extracted from the query image and the point cloud. The trained Desc-Matcher is then used to match the features from the image and the point cloud. A robust pose estimator is used to predict the location and orientation of the query image from the corresponding positions of the matched 2D and 3D features. We carried out an extensive evaluation of the proposed method for indoor and outdoor scenarios and with different types of point clouds to verify the feasibility of our approach. Experimental results show that the proposed approach can reliably estimate the 6-DOF poses of query cameras in any type of 3D point cloud with high precision. We achieved average median errors of 1.09cm/0.27∘ and 19cm/0.39∘ on the Stanford and Cambridge datasets, respectively.

Knowledge transduction for cross-domain few-shot learning

Cross-Domain Few-Shot Learning (CDFSL) aims to classify new categories from new domains with few samples. It confronts a greater domain shift than Few-Shot Learning (FSL). Based on the transfer learning framework, we propose a Knowledge Transduction method (KT) to alleviate domain shift and achieve few-shot recognition. First, a feature adaptation module based on feed-forward attention is constructed to learn domain-adapted features. The feature adaptation module weakens domain shift by transducing knowledge from an auxiliary dataset to the new dataset. Second, a feature transduction module based on deep sparse representation is developed to gather class semantics from limited support images. The feature transduction module transduces knowledge from support images to query images for few-shot recognition. In addition, a stochastic image augmentation method is proposed for FSL to train a more generalized model through consistency representation learning. Our method achieves competitive accuracy on four CDFSL datasets and four FSL datasets compared to state-of-the-art methods. The source code is available at https://github.com/XDUpfLi/KT.

Co-attention enabled content-based image retrieval

Content-based image retrieval (CBIR) aims to provide the most similar images to a given query. Feature extraction plays an essential role in retrieval performance within a CBIR pipeline. Current CBIR studies would either uniformly extract feature information from the input image and use it directly or employ some trainable spatial weighting module which is then used for similarity comparison between pairs of query and candidate matching images. These spatial weighting modules are normally query non-sensitive and only based on the knowledge learned during the training stage. They may focus towards incorrect regions, especially when the target image is not salient or is surrounded by distractors. This paper proposes an efficient query sensitive co-attention11“Co-attention” in this paper refers to spatial attention conditioned on the query content. mechanism for large-scale CBIR tasks. In order to reduce the extra computation cost required by the query sensitivity to the co-attention mechanism, the proposed method employs clustering of the selected local features. Experimental results indicate that the co-attention maps can provide the best retrieval results on benchmark datasets under challenging situations, such as having completely different image acquisition conditions between the query and its match image.

CLoc: Confident Initial Estimation of Long-Term Visual Localization Using a Few Sequential Images in Large-Scale Spaces

Estimation of six degree-of-freedom camera pose with images is a fundamental task for various applications, such as autonomous driving and augmented reality. The challenges in visual localization stem primarily from the scale of spaces and the changes that can happen during the long-term intervals between when the database and query images are captured. Although most sequence-based visual localization (SVL) methods rely on the outputs of one-shot visual localization (OVL) for the initial estimation of the camera pose, OVL may yield incorrect poses because of these limitations. In this study, we aim to achieve a confident initial estimation using a few sequential images. Accordingly, we propose a pipeline called CLoc that narrows down possible solution spaces with hypotheses generated by associating pose candidates of keyframes and their relative poses to determine the most probable pose of the latest keyframe. Moreover, we create publicly available visual localization datasets including three building-scale indoor spaces. The datasets provide long-term challenges and sequential query images simultaneously, enabling the evaluation of the OVL and SVL methods in large-scale spaces. We evaluate CLoc using our datasets and a city-scale outdoor dataset and compare it with two state-of-the-art OVL methods and three SVL methods. By properly leveraging sequential data, CLoc can outperform other methods and achieves 100% accuracy in the [2m, 10°] evaluation thresholds for all datasets.

PIHA: Detection method using perceptual image hashing against query-based adversarial attacks

As a representative adversarial example generation method in a black-box scenario, query-based adversarial attack has recently been highlighted due to its high attack success rate and feasibility. To provide the robustness against query-based adversarial attacks, the state-of-the-art defense methods use similarity of queries. Among them, two state-of-the-art defense methods have shown the good performance in the number of detected attack queries: (1) Stateful Detection; and (2) Blacklight. In this paper, we first propose two simple but effective mitigation strategies(i.e., parallelization and brightness adjustment) to show how to mitigate the robustness of the state-of-the-art defense methods. Next, we propose a new defense method, which uses the concept of perceptual hashing. Given a query image, the proposed defense method generates a hash sequence using a perceptual image hashing scheme, called PIHA, and compares the hash sequence with those of previous queries to detect query-based adversarial attacks. Here, a hash sequence has invariance to small perturbations and color changes when detecting query-based adversarial attacks. From the experimental results under various conditions, we show that the proposed defense method provides good robustness against well-known query-based adversarial attacks without or with mitigation strategies applied.