Query Features Research Articles

A vehicle classification model based on deep active learning

Vehicle classification is widely used in intelligent transportation and smart cities. However, the good performance of vehicle classification often depends on a large number of labeling data, which leads to the high cost of manual labeling. How to use a small number of labeled samples with rich features to achieve good classification performance is a challenge in vehicle classification. To solve these problems, we propose a deep active learning framework called Feature Fusion Spatial Pyramid Pooling and Re-parameterization Visual Geometry Group (FFSPP-RepVGG). The framework is mainly divided into two parts: query strategy and feature extraction module. Specifically, the query strategy uses Feature Fusion (FF) to calculate the loss, thus defining uncertainty and being able to select more valuable samples for labeling training. The feature extraction module uses Spatial Pyramid Pooling and Re-parameterization Visual Geometry Group (SPP-RepVGG) model, which can extract more reliable image features. It can reduce the cost of data labeling and have better performance at the same time. The experimental results on the BIT-Vehicle data set and the car-10 data set show that the FFSPP-RepVGG framework has superior performance than that of the comparison models.

Hash Indexing-Based Image Matching for 3D Reconstruction

Image matching is a basic task in three-dimensional reconstruction, which, in recent years, has attracted extensive attention in academic and industrial circles. However, when dealing with large-scale image datasets, these methods have low accuracy and slow speeds. To improve the effectiveness of modern image matching methods, this paper proposes an image matching method for 3D reconstruction. The proposed method can obtain high matching accuracy through hash index in a very short amount of time. The core of hash matching includes two parts: creating the hash table and hash index. The former is used to encode local feature descriptors into hash codes, and the latter is used to search candidates for query feature points. In addition, the proposed method is extremely robust to image scaling and transformation by using various verifications. A comprehensive experiment was carried out using several challenging datasets to evaluate the performance of hash matching. Experimental results show that the HashMatch presents excellent results compared to the state-of-the-art methods in both computational efficiency and matching accuracy.

Accurate few-shot object counting with Hough matching feature enhancement.

Given some exemplars, few-shot object counting aims to count the corresponding class objects in query images. However, when there are many target objects or background interference in the query image, some target objects may have occlusion and overlap, which causes a decrease in counting accuracy. To overcome the problem, we propose a novel Hough matching feature enhancement network. First, we extract the image feature with a fixed convolutional network and refine it through local self-attention. And we design an exemplar feature aggregation module to enhance the commonality of the exemplar feature. Then, we build a Hough space to vote for candidate object regions. The Hough matching outputs reliable similarity maps between exemplars and the query image. Finally, we augment the query feature with exemplar features according to the similarity maps, and we use a cascade structure to further enhance the query feature. Experiment results on FSC-147 show that our network performs best compared to the existing methods, and the mean absolute counting error on the test set improves from 14.32 to 12.74. Ablation experiments demonstrate that Hough matching helps to achieve more accurate counting compared with previous matching methods.

MSFFAL: Few-Shot Object Detection via Multi-Scale Feature Fusion and Attentive Learning

Few-shot object detection (FSOD) is proposed to solve the application problem of traditional detectors in scenarios lacking training samples. The meta-learning methods have attracted the researchers' attention for their excellent generalization performance. They usually select the same class of support features according to the query labels to weight the query features. However, the model cannot possess the ability of active identification only by using the same category support features, and feature selection causes difficulties in the testing process without labels. The single-scale feature of the model also leads to poor performance in small object detection. In addition, the hard samples in the support branch impact the backbone's representation of the support features, thus impacting the feature weighting process. To overcome these problems, we propose a multi-scale feature fusion and attentive learning (MSFFAL) framework for few-shot object detection. We first design the backbone with multi-scale feature fusion and channel attention mechanism to improve the model's detection accuracy on small objects and the representation of hard support samples. Based on this, we propose an attention loss to replace the feature weighting module. The loss allows the model to consistently represent the objects of the same category in the two branches and realizes the active recognition of the model. The model no longer depends on query labels to select features when testing, optimizing the model testing process. The experiments show that MSFFAL outperforms the state-of-the-art (SOTA) by 0.7-7.8% on the Pascal VOC and exhibits 1.61 times the result of the baseline model in MS COCO's small objects detection.

Query-Guided Prototype Learning with Decoder Alignment and Dynamic Fusion in Few-Shot Segmentation

Few-shot segmentation aims to segment objects belonging to a specific class under the guidance of a few annotated examples. Most existing approaches follow the prototype learning paradigm and generate category prototypes by squeezing masked feature maps extracted from images in the support set. These support prototypes may lead to inaccurate predictions when directly compared with features extracted from the query set due to the considerable distribution discrepancy between support and query features. We propose a query-guided prototype learning architecture to address this problem from two aspects: (i) We propose a cross-alignment loss for training the segmentation decoder. This loss function will help the decoder improve its robustness against the distribution discrepancy between support and query features. (ii) We build a dynamic fusion module to strengthen the original support prototype with another prototype extracted from query features. Experiments show that our method achieves promising results compared to previous prototype learning methods on PASCAL-5 i and COCO-20 i datasets.

CrossDet++: Growing Crossline Representation for Object Detection

In object detection, precise object representation is a key factor to successfully classify and locate objects of an image. Existing methods usually use rectangular anchor boxes or a set of points to represent objects. However, these methods either introduce background noise or miss the continuous appearance information inside the object, and thus cause incorrect detection results. In this paper, we propose a novel anchor-free object detection network, called CrossDet++, which uses a set of growing crosslines along horizontal and vertical axes as object representations. An object can be flexibly represented as crosslines in different combinations, which inspires us to select the expressive crossline to effectively reduce the interference of noise. Meanwhile, the crossline representation takes into account the continuous adjacent object information, which is useful to enhance the discriminability of object features and find the object boundaries. Based on the learned crosslines, we propose an axis-query crossline growing module to adaptively capture features of crosslines and query surrounding pixels related to the line features for subsequent growing of crosslines. Their growing offsets and scales can be supervised by a decoupled regression mechanism, which limits the regression target to a specific direction for decreasing the optimization difficulty. During the training, we design a semantic-guided label assignment to emphasize the importance of crossline targets with higher semantic richness, further improving the detection performance. The experiment results demonstrate the effectiveness of our proposed method. Code can be available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/QiuHeqian/CrossDet</uri> .

Adaptive-Attention Completing Network for Remote Sensing Image

The reconstruction of missing pixels is essential for remote sensing images, as they often suffer from problems such as covering, dead pixels, and scan line corrector (SLC)-off. Image inpainting techniques can solve these problems, as they can generate realistic content for the unknown regions of an image based on the known regions. Recently, convolutional neural network (CNN)-based inpainting methods have integrated the attention mechanism to improve inpainting performance, as they can capture long-range dependencies and adapt to inputs in a flexible manner. However, to obtain the attention map for each feature, they compute the similarities between the feature and the entire feature map, which may introduce noise from irrelevant features. To address this problem, we propose a novel adaptive attention (Ada-attention) that uses an offset position subnet to adaptively select the most relevant keys and values based on self-attention. This enables the attention to be focused on essential features and model more informative dependencies on the global range. Ada-attention first employs an offset subnet to predict offset position maps on the query feature map; then, it samples the most relevant features from the input feature map based on the offset position; next, it computes key and value maps for self-attention using the sampled features; finally, using the query, key and value maps, the self-attention outputs the reconstructed feature map. Based on Ada-attention, we customized a u-shaped adaptive-attention completing network (AACNet) to reconstruct missing regions. Experimental results on several digital remote sensing and natural image datasets, using two image inpainting models and two remote sensing image reconstruction approaches, demonstrate that the proposed AACNet achieves a good quantitative performance and good visual restoration results with regard to object integrity, texture/edge detail, and structural consistency. Ablation studies indicate that Ada-attention outperforms self-attention in terms of PSNR by 0.66%, SSIM by 0.74%, and MAE by 3.9%, and can focus on valuable global features using the adaptive offset subnet. Additionally, our approach has also been successfully applied to remove real clouds in remote sensing images, generating credible content for cloudy regions.

CUR Transformer: A Convolutional Unbiased Regional Transformer for Image Denoising

Image denoising is a fundamental problem in computer vision and multimedia computation. Non-local filters are effective for image denoising. But existing deep learning methods that use non-local computation structures are mostly designed for high-level tasks, and global self-attention is usually adopted. For the task of image denoising, they have high computational complexity and have a lot of redundant computation of uncorrelated pixels. To solve this problem and combine the marvelous advantages of non-local filter and deep learning, we propose a Convolutional Unbiased Regional (CUR) transformer. Based on the prior that, for each pixel, its similar pixels are usually spatially close, our insights are that (1) we partition the image into non-overlapped windows and perform regional self-attention to reduce the search range of each pixel, and (2) we encourage pixels across different windows to communicate with each other. Based on our insights, the CUR transformer is cascaded by a series of convolutional regional self-attention (CRSA) blocks with U-style short connections. In each CRSA block, we use convolutional layers to extract the query, key, and value features, namely Q , K , and V , of the input feature. Then, we partition the Q , K , and V features into local non-overlapped windows and perform regional self-attention within each window to obtain the output feature of this CRSA block. Among different CRSA blocks, we perform the unbiased window partition by changing the partition positions of the windows. Experimental results show that the CUR transformer outperforms the state-of-the-art methods significantly on four low-level vision tasks, including real and synthetic image denoising, JPEG compression artifact reduction, and low-light image enhancement.

Cross-scale sampling transformer for semantic image segmentation

In increasingly complex scenes, multi-scale information fusion becomes more and more critical for semantic image segmentation. Various methods are proposed to model multi-scale information, such as local to global, but this is not enough for the scene changes more and more, and the image resolution becomes larger and larger. Cross-Scale Sampling Transformer is proposed in this paper. We first propose that each scale feature is sparsely sampled at one time, and all other features are fused, which is different from all previous methods. Specifically, the Channel Information Augmentation module is first proposed to enhance query feature features, highlight part of the response to sampling points and enhance image features. Next, the Multi-Scale Feature Enhancement module performs a one-time fusion of full-scale features, and each feature can obtain information about other scale features. In addition, the Cross-Scale Fusion module is used for cross-scale fusion of query feature and full-scale feature. Finally, the above three modules constitute our Cross-Scale Sampling Transformer(CSSFormer). We evaluate our CSSFormer on four challenging semantic segmentation benchmarks, including PASCAL Context, ADE20K, COCO-Stuff 10K, and Cityscapes, achieving 59.95%, 55.48%, 50.92%, and 84.72% mIoU, respectively, outperform the state-of-the-art.

Content-Based Image Retrieval System using Color Moment and Bag of Visual Words with Local Binary Pattern

Researchers have recently focused their attention on Content-Based Image Retrieval (CBIR). It has emerged as one of the most fascinating areas in image processing and computer vision. With CBIR, the most comparable pictures that match the query image are pulled from an image database. As a result, it necessitates feature extraction (Local / Global) and similarity calculation. This paper uses a CBIR technique to determine the images that best match the image query by utilizing both global and local image features. A color moment is used for global features to describe the complete image. Local Binary Pattern (LBP) as a local feature, on the other hand, extracts interest points by building a Bag of Visual Words (BoVW). The distance between the query and database image features is computed using the Euclidean distance. Precision and recall are computed on the Corel-1K dataset to assess the retrieval performance.

Development And Evaluation Of The Webgis Application To Support Volcanic Hazard Mitigation In The Southern Flank Of Merapi Volcano, Sleman Regency, Yogyakarta Province, Indonesia

Merapi Volcano is one of the active volcanoes in Indonesia, which is located in the Central Java and Yogyakarta Province. The eruption of Merapi Volcano is a threat to people living on the slopes of Merapi, especially on its denselypopulated southern flank. The purpose of this study was to build a webGIS to support volcanic hazard mitigation regarding Merapi Volcano and evaluate the webGIS system for determining the community’s perception. This research was the first to produce a product that is used by government agencies related to volcanic disaster mitigation. webGIS development was carried out using an open source platform. System evaluation was carried out using usability testing. The samples were obtained using systematic the random sampling method of respondents who lived in the villages on the southern flank of Merapi volcano. webGIS was built using LeafletJS and QGIS, combined with spatial data about the evacuation locations, health facilities, evacuation routes, government offices, educational facilities and worship facilities, with a basemap obtained from Openstreet Map and Google Satellite. WebGIS was equipped with a database query feature to make it easier for users to find geographical information. The usability testing results showed that as many as 83% of the respondents were very satisfied with the appearance and information of webGIS, while as many as 82% were very satisfied with the navigation offered via the webGIS display.

Coarse Mask Guided Interactive Object Segmentation.

Interactive object segmentation aims to produce object masks with user interactions, such as clicks, bounding boxes, and scribbles. Click point is the most popular interactive cue for its efficiency, and related deep learning methods have attracted lots of interest in recent years. Most works encode click points as gaussian maps and concatenate them with images as the model's input. However, the spatial and semantic information of gaussian maps would be noised through multiple convolution layers and won't be fully exploited by top layers for mask prediction. To pass click information to top layers exactly and efficiently, we propose a coarse mask guided model (CMG) which predicts coarse masks with a coarse module to guide the object mask prediction. Specifically, the coarse module encodes user clicks as query features and enriches their semantic information with backbone features through transformer layers, coarse masks are generated based on the enriched query feature and fed into CMG's decoder. Benefiting from the efficiency of transformer, CMG's coarse module and decoder module are lightweight and computationally efficient, making the interaction process more smooth. Experiments on several segmentation benchmarks demonstrate the effectiveness of our method, and we get new state-of-the-art results compared with previous works.

Multi-Modal Cross-Domain Alignment Network for Video Moment Retrieval

As an increasingly popular task in multimedia information retrieval, video moment retrieval (VMR) aims to localize the target moment from an untrimmed video according to a given language query. Most previous methods depend heavily on numerous manual annotations ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , moment boundaries), which are extremely expensive to acquire in practice. In addition, due to the domain gap between different datasets, directly applying these pre-trained models to an unseen domain leads to a significant performance drop. In this paper, we focus on a novel task: cross-domain VMR, where fully-annotated datasets are available in one domain (“source domain”), but the domain of interest (“target domain”) only contains unannotated datasets. As far as we know, we present the first study on cross-domain VMR. To address this new task, we propose a novel Multi-Modal Cross-Domain Alignment (MMCDA) network to transfer the annotation knowledge from the source domain to the target domain. However, due to the domain discrepancy between the source and target domains and the semantic gap between videos and queries, directly applying trained models to the target domain generally leads to a performance drop. To solve this problem, we develop three novel modules: (i) a domain alignment module is designed to align the feature distributions between different domains of each modality; (ii) a cross-modal alignment module aims to map both video and query features into a joint embedding space and to align the feature distributions between different modalities in the target domain; (iii) a specific alignment module tries to obtain the fine-grained similarity between a specific frame and the given query for optimal localization. By jointly training these three modules, our MMCDA can learn domain-invariant and semantic-aligned cross-modal representations. Extensive experiments on three challenging benchmarks (ActivityNet Captions, Charades-STA and TACoS) illustrate that our cross-domain method MMCDA outperforms all state-of-the-art single-domain methods. Impressively, MMCDA raises the performance by more than 7% in representative cases, which demonstrates its effectiveness.

Few-Shot Multi-Agent Perception with Ranking-Based Feature Learning.

In this article, we focus on performing few-shot learning (FSL) under multi-agent scenarios in which participating agents only have scarce labeled data and need to collaborate to predict labels of query observations. We aim at designing a coordination and learning framework in which multiple agents, such as drones and robots, can collectively perceive the environment accurately and efficiently under limited communication and computation conditions. We propose a metric-based multi-agent FSL framework which has three main components: an efficient communication mechanism that propagates compact and fine-grained query feature maps from query agents to support agents; an asymmetric attention mechanism that computes region-level attention weights between query and support feature maps; and a metric-learning module which calculates the image-level relevance between query and support data fast and accurately. Furthermore, we propose a specially designed ranking-based feature learning module, which can fully utilize the order information of training data by maximizing the inter-class distance, while minimizing the intra-class distance explicitly. We perform extensive numerical studies and demonstrate that our approach can achieve significantly improved accuracy in visual and acoustic perception tasks such as face identification, semantic segmentation, and sound genre recognition, consistently outperforming the state-of-the-art baselines by 5%-20%.

Unseen Object Few-Shot Semantic Segmentation for Robotic Grasping

Semantic segmentation is a popular technology enabling robots to perceive and interact with the environment sufficiently. However, unseen objects in the new environment make it challenging to obtain accurate segmentation results. To solve this problem, we apply few-shot semantic segmentation in robot perception and propose a complete semantic grasping framework that integrates few-shot semantic segmentation and grasp pose detection for the first time. Our method can quickly identify the unseen target and obtain accurate segmentation results with only a few labeled support images. Meanwhile, our proposed multi-scale prototype extraction module and self-attention-guided feature learning module can generate representative prototypes and query features. Furthermore, we set up a few-shot semantic segmentation benchmark for robotic environments named GraspNet-10 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{i}$</tex-math></inline-formula> . Experimental results on the dataset show that our method outperforms previous state-of-the-art methods by a large margin. We also conduct real robotic grasping experiments to demonstrate the feasibility and effectiveness of our pipeline.

APANet: Adaptive Prototypes Alignment Network for Few-Shot Semantic Segmentation

Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. However, this framework suffers from biased classification due to incomplete feature comparisons. To address this issue, we present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes and thus construct complete sample pairs for learning semantic alignment with query features. The complementary features learning manner effectively enriches feature comparison and helps yield an unbiased segmentation model in the few-shot setting. It is implemented with a two-branch end-to-end network (i.e., a class-specific branch and a class-agnostic branch), which generates prototypes and then combines query features to perform comparisons. In addition, the proposed class-agnostic branch is simple yet effective. In practice, it can adaptively generate multiple class-agnostic prototypes for query images and learn feature alignment in a self-contrastive manner. Extensive experiments on PASCAL-5$^i$ and COCO-20$^i$ demonstrate the superiority of our method. At no expense of inference efficiency, our model achieves state-of-the-art results in both 1-shot and 5-shot settings for semantic segmentation.

Methodology for Constructing a Knowledgebase for Plant Gene Regulation Information.

The amount of biological data is growing at a rapid pace as many high-throughput omics technologies and data pipelines are developed. This is resulting in the growth of databases for DNA and protein sequences, gene expression, protein accumulation, structural, and localization information. The diversity and multi-omics nature of such bioinformatic data requires well-designed databases for flexible organization and presentation. Besides general-purpose online bioinformatic databases, users need narrowly focused online databases to quickly access a meaningful collection of related data for their research. Here, we describe the methodology used to implement a plant gene regulatory knowledgebase, with data, query, and tool features, as well as the ability to expand to accommodate future datasets. We exemplify this methodology for the GRASSIUS knowledgebase, but it is applicable to developing and updating similar plant gene regulatory knowledgebases. GRASSIUS organizes and presents gene regulatory data from grass species with a central focus on maize (Zea mays). The main class of data presented include not only the families of transcription factors (TFs) and co-regulators (CRs) but also protein-DNA interaction data, where available.

Transformer-Based Efficient Salient Instance Segmentation Networks With Orientative Query

Salient instance segmentation (SIS) can be consid-ered as the next generation task for saliency detection com-munity. As a new challenging task, most of the existing state-of-the-art methods are built on the mainstream Mask R-CNN architecture. However, this mechanism relies heavily on hand-designed anchors and NMS post-processing. In this paper, we provide a one stage SIS framework with transformers, termed Orientative Query Transformer (OQTR). To leverage long-range dependencies of transformer, a cross fusion module is designed to efficiently fuse the global features in encoder and salient query features for salient mask prediction. Furthermore, derived from the center prior in traditional saliency models, we propose an orientative query that is considered as the initial salient object query to accelerate convergence. In addition, to mitigate the issue of lacking large-scale dataset with salient instance labels, we collect a new SIS dataset (SIS10K) containing over 10K images elaborately annotated with both object- and instance-level labels to promote the community. Without any post-processing, our end-to-end OQTR framework significantly surpass the top-1 RDPNet by an average of 13.1% AP scores across all three challenging datasets, showing a strong performance of the proposed OQTR. The code and the dataset proposed in this work are available at: https://github.com/ssecv/OQTR.

Generalized Ridge Regression-Based Channelwise Feature Map Weighted Reconstruction Network for Fine-Grained Few-Shot Ship Classification

Fine-grained ship classification (FGSCR) has many applications in military and civilian fields. In recent years, deep learning has been widely used for classification tasks, and its success is inseparable from that of big data. However, ship images are valuable, with only a few images of a specific category being obtained, leading to the fine-grained few-shot ship classification problem. In addition, feature map channels contain distinct characteristics and discriminative details, which significantly influence FGSCR. Intuitively, channels with distinct characteristics should be assigned larger weights for classification, but most few-shot learning methods treat the channels equally. Therefore, we propose a generalized ridge-regression-based channelwise feature map weighted reconstruction network to address these issues. First, we reconstruct the query feature map by assigning different weights to the support feature map channels using the generalized ridge regression method. The channels with large discriminative details contribute more toward reconstruction. Second, we propose a support channel weight module to calculate the channel weight matrix used in the generalized ridge regression method. Finally, based on the reconstructed query feature map, we can calculate the reconstruction error. The reconstruction error is adopted as the distance metric. Our proposed method achieves excellent performance on the fine-grained ship, bird, aircraft, and WHU-RS19 datasets compared with other representative few-shot learning methods. Considering the limited studies on the fine-grained few-shot ship classification problem, we believe that our work is of great significance.

Multi-Scale and Multi-Match for Few-Shot Plant Disease Image Semantic Segmentation

Currently, deep convolutional neural networks have achieved great achievements in semantic segmentation tasks, but existing methods all require a large number of annotated images for training and do not have good scalability for new objects. Therefore, few-shot semantic segmentation methods that can identify new objects with only one or a few annotated images are gradually gaining attention. However, the current few-shot segmentation methods cannot segment plant diseases well. Based on this situation, a few-shot plant disease semantic segmentation model with multi-scale and multi-prototypes match (MPM) is proposed. This method generates multiple prototypes and multiple query feature maps, and then the relationships between prototypes and query feature maps are established. Specifically, the support feature and query feature are first extracted from the high-scale layers of the feature extraction network; subsequently, masked average pooling is used for the support feature to generate prototypes for a similarity match with the query feature. At the same time, we also fuse low-scale features and high-scale features to generate another support feature and query feature that mix detailed features, and then a new prototype is generated through masked average pooling to establish a relationship with the query feature of this scale. Subsequently, in order to solve the shortcoming of traditional cosine similarity and lack of spatial distance awareness, a CES (cosine euclidean similarity) module is designed to establish the relationship between prototypes and query feature maps. To verify the superiority of our method, experiments are conducted on our constructed PDID-5i dataset, and the mIoU is 40.5%, which is 1.7% higher than that of the original network.