Query Features Research Articles

Few-Shot Object Detection with Local Feature Enhancement and Feature Interrelation

Few-shot object detection (FSOD) aims at designing models that can accurately detect targets of novel classes in a scarce data regime. Existing research has improved detection performance with meta-learning-based models. However, existing methods continue to exhibit certain imperfections: (1) Only the interacting global features of query and support images lead to ignoring local critical features in the imprecise localization of objects from new categories. (2) Convolutional neural networks (CNNs) encounter difficulty in learning diverse pose features from exceedingly limited labeled samples of unseen classes. (3) Local context information is not fully utilized in a global attention mechanism, which means the attention modules need to be improved. As a result, the detection performance of novel-class objects is compromised. To overcome these challenges, a few-shot object detection network is proposed with a local feature enhancement module and an intrinsic feature transformation module. In this paper, a local feature enhancement module (LFEM) is designed to raise the importance of intrinsic features of the novel-class samples. In addition, an Intrinsic Feature Transform Module (IFTM) is explored to enhance the feature representation of novel-class samples, which enriches the feature space of novel classes. Finally, a more effective cross-attention module, called Global Cross-Attention Network (GCAN), which fully aggregates local and global context information between query and support images, is proposed in this paper. The crucial features of novel-class objects are extracted effectively by our model before the feature fusion between query images and support images. Our proposed method increases, on average, the detection performance by 0.93 (nAP) in comparison with previous models on the PASCAL VOC FSOD benchmark dataset. Extensive experiments demonstrate the effectiveness of our modules under various experimental settings.

CLIP-Driven Prototype Network for Few-Shot Semantic Segmentation.

Recent research has shown that visual-text pretrained models perform well in traditional vision tasks. CLIP, as the most influential work, has garnered significant attention from researchers. Thanks to its excellent visual representation capabilities, many recent studies have used CLIP for pixel-level tasks. We explore the potential abilities of CLIP in the field of few-shot segmentation. The current mainstream approach is to utilize support and query features to generate class prototypes and then use the prototype features to match image features. We propose a new method that utilizes CLIP to extract text features for a specific class. These text features are then used as training samples to participate in the model's training process. The addition of text features enables model to extract features that contain richer semantic information, thus making it easier to capture potential class information. To better match the query image features, we also propose a new prototype generation method that incorporates multi-modal fusion features of text and images in the prototype generation process. Adaptive query prototypes were generated by combining foreground and background information from the images with the multi-modal support prototype, thereby allowing for a better matching of image features and improved segmentation accuracy. We provide a new perspective to the task of few-shot segmentation in multi-modal scenarios. Experiments demonstrate that our proposed method achieves excellent results on two common datasets, PASCAL-5i and COCO-20i.

Prototype expansion and feature calibration for few-shot point cloud semantic segmentation

Few-shot point cloud semantic segmentation has attracted increasing attention due to its applicability to real-world scenarios in which each novel class contains only a few labelled data. Although existing methods have made great progress, a great challenge still remains: inaccurate decision boundaries are induced by the scarcity of labelled data. In this paper, a Prototype Expansion and Feature Calibration (PEFC) method is proposed, which fully exploits the information contained in query data to increase the amounts of class prototypes and calibrate these prototypes. To this end, a Prototype Expansion Module (PEM) is designed, which clusters query data into pseudo prototypes and expands the set of class prototypes with pseudo prototypes that have a high similarity to the class prototypes. Then, a Feature Calibration Module (FCM) is designed to calibrate these class prototypes, which is achieved by enabling bidirectional information flow between class prototypes and query features in a task-aware manner. Extensive experiments on two benchmark datasets show that the proposed PEFC outperforms the state-of-the-art methods by a large margin in all experimental settings.

Robust Prototypical Few-Shot Organ Segmentation with Regularized Neural-ODEs.

Despite the tremendous progress made by deep learning models in image semantic segmentation, they typically require large annotated examples, and increasing attention is being diverted to problem settings like Few-Shot Learning (FSL) where only a small amount of annotation is needed for generalisation to novel classes. This is especially seen in medical domains where dense pixel-level annotations are expensive to obtain. In this paper, we propose Regularized Prototypical Neural Ordinary Differential Equation (R-PNODE), a method that leverages intrinsic properties of Neural-ODEs, assisted and enhanced by additional cluster and consistency losses to perform Few-Shot Segmentation (FSS) of organs. R-PNODE constrains support and query features from the same classes to lie closer in the representation space thereby improving the performance over the existing Convolutional Neural Network (CNN) based FSS methods. We further demonstrate that while many existing Deep CNN-based methods tend to be extremely vulnerable to adversarial attacks, R-PNODE exhibits increased adversarial robustness for a wide array of these attacks. We experiment with three publicly available multi-organ segmentation datasets in both in-domain and cross-domain FSS settings to demonstrate the efficacy of our method. In addition, we perform experiments with seven commonly used adversarial attacks in various settings to demonstrate R-PNODE's robustness. R-PNODE outperforms the baselines for FSS by significant margins and also shows superior performance for a wide array of attacks varying in intensity and design.

Snipper: A Spatiotemporal Transformer for Simultaneous Multi-Person 3D Pose Estimation Tracking and Forecasting on a Video Snippet

Multi-person pose understanding from RGB videos involves three complex tasks: pose estimation, tracking and motion forecasting. Intuitively, accurate multi-person pose estimation facilitates robust tracking, and robust tracking builds crucial history for correct motion forecasting. Most existing works either focus on a single task or employ multi-stage approaches to solving multiple tasks separately, which tends to make sub-optimal decision at each stage and also fail to exploit correlations among the three tasks. In this paper, we propose Snipper, a unified framework to perform multi-person 3D pose estimation, tracking, and motion forecasting simultaneously in a single stage. We propose an efficient yet powerful deformable attention mechanism to aggregate spatiotemporal information from the video snippet. Building upon this deformable attention, a video transformer is learned to encode the spatiotemporal features from the multi-frame snippet and to decode informative pose features for multi-person pose queries. Finally, these pose queries are regressed to predict multi-person pose trajectories and future motions in a single shot. In the experiments, we show the effectiveness of Snipper on three challenging public datasets where our generic model rivals specialized state-of-art baselines for pose estimation, tracking, and forecasting. Code is available at https://github.com/JimmyZou/Snipper.

Learning Spatial-Frequency Transformer for Visual Object Tracking

Recently, some researchers have begun to adopt the Transformer to combine or replace the widely used ResNet as their new backbone network. As the Transformer captures the long-range relations between pixels well using the self-attention scheme, which complements the issues caused by the limited receptive field of CNN. Although their trackers work well in regular scenarios, they simply flatten the 2D features into a sequence to better match the Transformer. We believe these operations ignore the spatial prior of the target object, which may lead to sub-optimal results only. In addition, many works demonstrate that self-attention is actually a low-pass filter, which is independent of input features or keys/queries. That is to say, it may suppress the high-frequency component of the input features and preserve or even amplify the low-frequency information. To handle these issues, in this paper, we propose a unified Spatial-Frequency Transformer that models the Gaussian spatial Prior and High-frequency emphasis Attention (GPHA) simultaneously. To be specific, Gaussian spatial prior is generated using dual Multi-Layer Perceptrons (MLPs) and injected into the similarity matrix produced by multiplying Query and Key features in self-attention. The output will be fed into a softmax layer and then decomposed into two components, i.e., the direct and high-frequency signal. The low- and high-pass branches are rescaled and combined to achieve all-pass, therefore, the high-frequency features will be protected well in stacked self-attention layers. We further integrate the Spatial-Frequency Transformer into the Siamese tracking framework and propose a novel tracking algorithm termed SFTransT. The cross-scale fusion based SwinTransformer is adopted as the backbone, and also a multi-head cross-attention module is used to boost the interaction between search and template features. The output will be fed into the tracking head for target localization. Extensive experiments on short-term and long-term tracking benchmarks all demonstrate the effectiveness of our proposed framework. Source code will be released at https://github.com/Tchuanm/SFTransT.git.

Collaborative 3D Object Detection for Autonomous Vehicles via Learnable Communications

3D object detection from LiDAR point cloud is a challenging task in autonomous driving systems. Collaborative perception can incorporate information from spatially diverse sensors and provide significant benefits for accurate 3D object detection from point clouds. In this work, we consider that the autonomous vehicle uses local point cloud data and combines information from neighboring infrastructures through wireless links for cooperative 3D object detection. However, information sharing among vehicles and infrastructures in predefined communication schemes may result in communication congestion and/or bring limited performance improvement. To this end, we propose a novel collaborative 3D object detection framework using an encoder-decoder network architecture and an attention-based learnable communications scheme. It consists of three components: a feature encoder network that maps point clouds into feature maps; an attention-based communication module that propagates compact and fine-grained query feature maps from the vehicle to support infrastructures, and optimizes attention weights between query and key to refine support feature maps; a region proposal network that fuses local feature maps and weighted support feature maps for 3D object detection. We evaluate the performance of the proposed framework on CARLA-3D, a new dataset that we synthesized using CARLA for 3D cooperative object detection. Experimental results and bandwidth consumption analysis show that the proposed collaborative 3D object detection framework achieves a better detection performance and communication bandwidth trade-off than five baseline 3D object detection models under different detection difficulties.

Selective Query Processing: A Risk-Sensitive Selection of Search Configurations

In information retrieval systems, search parameters are optimized to ensure high effectiveness based on a set of past searches, and these optimized parameters are then used as the search configuration for all subsequent queries. A better approach, however, would be to adapt the parameters to fit the query at hand. Selective query expansion is one such an approach, in which the system decides automatically whether or not to expand the query, resulting in two possible search configurations. This approach was extended recently to include many other parameters, leading to many possible search configurations where the system automatically selects the best configuration on a per-query basis. One problem with this approach is the system training, which requires evaluation of each training query with every possible configuration. In real-world systems, so many parameters and possible values must be evaluated that this approach is impractical, especially when the system must be updated frequently, as is the case for commercial search engines. In general, the more configurations, the greater the effectiveness when configuration selection is appropriate but also the greater the risk of decreasing effectiveness in the case of an inappropriate configuration selection. To determine the ideal configurations to be used for each query in real-world systems, we have developed a method in which a limited number of possible configurations are pre-selected, then used in a meta-search engine that decides the best search configuration for each query. We define a risk-sensitive approach for configuration pre-selection that considers the risk-reward tradeoff between the number of configurations kept and system effectiveness. We define two alternative risk functions to apply to different goals. For final configuration selection, the decision is based on query feature similarities. We compare two alternative risk functions on two query types (ad hoc and diversity) and compare these to more sophisticated machine learning based methods. We find that a relatively small number of configurations (20) selected by our risk-sensitive model is sufficient to obtain results close to the best achievable results for each query. Effectiveness is increased by about 15% according to the P@10 and nDCG@10 evaluation metrics when compared to traditional grid search using a single configuration and by about 20% when compared to learning to rank documents. Our risk-sensitive approach works for both diversity- and ad hoc oriented searches. Moreover, the similarity-based selection method outperforms the more sophisticated approaches. Thus, we demonstrate the feasibility of developing per-query information retrieval systems, which will guide future research in this direction.

Lite-FENet: Lightweight multi-scale feature enrichment network for few-shot segmentation

Current methods for few-shot segmentation focus on extracting information from support and query targets, however, most of these methods not only suffer from high model complexity but also fail to capture long-range interactions between query features and support features. To efficiently reduce the information loss due to the appearance inconsistency in spatial between support and query objects and make full use of the limited features extracted from support and query samples, we present a lightweight multi-scale feature enrichment network named Lite-FENet that enhances multi-scale feature interactions and ensures the discrimination with the final features under low computational cost. Specifically, we design a lightweight and efficient spatial interaction module for the Lite-FENet to achieve the contextual information transfer between different spatial scales over long distances. The repeated interactions generate a descriptive feature, and show superior robustness for pixel-level prediction, especially when there is a gap between different instances. Experimental results show that the proposed method achieves 63.5% and 64.7% MIoU on the PASCAL-5i dataset, and 39.5% and 43.1% MIoU on the COCO-20i datasets under 1-shot and 5-shot settings with only 6.7M learnable parameters, respectively, which significantly exceeds the results of current methods, while ensuring a far smaller model size. Code is available at: https://github.com/Sunbaoquan/Lite-FENet.

MillenniumDB: An Open-Source Graph Database System

ABSTRACT In this systems paper, we present MillenniumDB: a novel graph database engine that is modular, persistent, and open source. MillenniumDB is based on a graph data model, which we call domain graphs, that provides a simple abstraction upon which a variety of popular graph models can be supported, thus providing a flexible data management engine for diverse types of knowledge graph. The engine itself is founded on a combination of tried and tested techniques from relational data management, state-of-the-art algorithms for worst-case-optimal joins, as well as graph-specific algorithms for evaluating path queries. In this paper, we present the main design principles underlying MillenniumDB, describing the abstract graph model and query semantics supported, the concrete data model and query syntax implemented, as well as the storage, indexing, query planning and query evaluation techniques used. We evaluate MillenniumDB over real-world data and queries from the Wikidata knowledge graph, where we find that it outperforms other popular persistent graph database engines (including both enterprise and open source alternatives) that support similar query features.

DLIRIR : Deep learning based improved Reverse Image Retrieval

Deep feature search has transpired as a sub-problem of large-scale Reverse Image Retrieval (RIR) algorithms in computer vision. In light of recent progress, transforming deep features into binary codes and employing various search strategies to delineate similar images has gained popularity. However, these binary codes cannot accurately define the similarity between two images, which is essential for image retrieval. Moreover, when retrieving k closest images, the existing RIR approaches show a drop in precision value as k increases. In this paper, we put forward a method, Deep Learning based Improved Reverse Image Retrieval (DLIRIR), which solves the RIR problem by utilizing a deep neural network, clustering methodology, and dimensionality reduction technique. Motivated by the need to prevent information loss in converting deep features to binary codes and reduce search time, we cluster image features and select some features from each cluster called Representative Points (RPs). These RPs are then compared to features of an image (query) to retrieve images similar to it from the database. The Euclidean distances of the query feature and the RPs indicate similarity between the query and the cluster of images represented by that RP. We reduce the number of comparisons by selecting as few RPs as possible. We evaluate the performance of DLIRIR by comparing it with other state-of-the-art algorithms in the RIR domain. Our method achieves a precision greater than 99% in benchmark datasets Caltech 256, Corel 10k, and Adaptiope, while a precision between 80%–85% on large size datasets Cifar 100, Tiny ImageNet, and VGGFace2.

Query-support semantic correlation mining for few-shot segmentation

Given one or several annotation samples, few-shot segmentation has the capability to complete the segmentation of unseen target objects, thereby reducing the need for extensive data annotation. However, due to the scarcity and diversity of data, it becomes challenging to employ a few support samples effectively for guiding the segmentation of one query sample. Most existing approaches are committed to excavating support samples’ information while ignoring the crucial relationship between query and support samples. To address this limitation and capture richer mutual information, we propose a query-support semantic correlation mining module that serves as an elaborate guide for subsequent dense matching. Specifically, two kinds of query guide maps are generated that leverage global and local similarities to learn the comprehensive correlation between multi-scale support and query features. By incorporating this module, the network can better execute dense matching and mitigate the possibility of misclassification effectively. In addition, a dual-attention module is introduced to obtain a discriminating initial prototype representation to avert prototype bias. Furthermore, our method generalizes well in the k-shot setting. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that the proposed method is potent and successful, which yields competitive segmentation results with state-of-the-art methods.

Mining Opinions on a Prominent Health Insurance Provider from Social Media Microblog: Affective Model and Contextual Analysis Approach

Social media plays a significant role in enhancing communication among organizations, communities, and individuals. Besides being a mode of communication, the data generated from these interactions can also be leveraged to assess the performance of an institution or organization. People may evaluate public companies based on the opinions of their users. However, user-supplied information is brief and written in natural language. In addition to being brief, the process of sending messages or engaging in other social media interactions contains a great deal of context information. This multiplicity of context can be utilized to conduct a more in-depth analysis of user opinion. This study presents a new approach to opinion mining for social media microblogging data by applying an affective model and contextual analyses. The affective model is applied for sentiment analysis to measure the degree of each adjective from user opinion by evaluating adjectives according to their varying levels of pleasure and arousal. The contextual analysis in this paper is modeled based on topic, user, adjective, and personal characteristics. The contextual analysis has four main features: (1) Temporal keyword sentiment context, (2) Temporal user sentiment context, (3) User impression context, and (4) Temporal user character context. Our affective model outperformed 75.6% the accuracy and 74.98% of F1-score, rather than SVM. In the experiment, the contextual analysis performed graph visualization of output results for each query feature for future development. Feature one to four successfully processes the query to produce a visualization graph.

Multi-Scale Similarity Guidance Few-Shot Network for Ship Segmentation in SAR Images

Target detection and segmentation in synthetic aperture radar (SAR) images are vital steps for many remote sensing applications. In the era of data-driven deep learning, this task is extremely challenging due to the limited labeled data. Few-shot learning has the ability to learn quickly from a few samples with supervised information. Inspired by this, a few-shot learning framework named MSG-FN is proposed to solve the segmentation of ship targets in heterologous SAR images with few annotated samples. The proposed MSG-FN adopts a dual-branch network consisting of a support branch and a query branch. The support branch is used to extract features with an encoder, and the query branch uses a U-shaped encoder–decoder structure to segment the target in the query image. The encoder of each branch is composed of well-designed residual blocks combined with filter response normalization to capture robust and domain-independent features. A multi-scale similarity guidance module is proposed to improve the scale adaptability of detection by applying hand-on-hand guidance of support features to query features of various scales. In addition, a SAR dataset named SARShip-4i is built to evaluate the proposed MSG-FN, and the experimental results show that the proposed method achieves superior segmentation results compared with the state-of-the-art.

Few-Shot Object Detection via Variational Feature Aggregation

As few-shot object detectors are often trained with abundant base samples and fine-tuned on few-shot novel examples, the learned models are usually biased to base classes and sensitive to the variance of novel examples. To address this issue, we propose a meta-learning framework with two novel feature aggregation schemes. More precisely, we first present a Class-Agnostic Aggregation (CAA) method, where the query and support features can be aggregated regardless of their categories. The interactions between different classes encourage class-agnostic representations and reduce confusion between base and novel classes. Based on the CAA, we then propose a Variational Feature Aggregation (VFA) method, which encodes support examples into class-level support features for robust feature aggregation. We use a variational autoencoder to estimate class distributions and sample variational features from distributions that are more robust to the variance of support examples. Besides, we decouple classification and regression tasks so that VFA is performed on the classification branch without affecting object localization. Extensive experiments on PASCAL VOC and COCO demonstrate that our method significantly outperforms a strong baseline (up to 16%) and previous state-of-the-art methods (4% in average).

MillenniumDB: An Open-Source Graph Database System

Abstract In this systems paper, we present MillenniumDB: a novel graph database engine that is modular, persistent, and open source. MillenniumDB is based on a graph data model, which we call domain graphs, that provides a simple abstraction upon which a variety of popular graph models can be supported, thus providing a flexible data management engine for diverse types of knowledge graph. The engine itself is founded on a combination of tried and tested techniques from relational data management, state-of-the-art algorithms for worst-case-optimal joins, as well as graph-specific algorithms for evaluating path queries. In this paper, we present the main design principles underlying MillenniumDB, describing the abstract graph model and query semantics supported, the concrete data model and query syntax implemented, as well as the storage, indexing, query planning and query evaluation techniques used. We evaluate MillenniumDB over real-world data and queries from the Wikidata knowledge graph, where we find that it outperforms other popular persistent graph database engines (including both enterprise and open source alternatives) that support similar query features.

Learning Graph Databases: Neo4j an overview

As the internet is growing day by day, the amount of data being generated is huge. This data includes structured data and unstructured data. The data along with its relationship with other data makes the most powerful and meaningful information. Maximum data exists in the form of the relationship between different or same objects and the noticeable thing is the relationship between the data is more important than the data itself. These relationships are handled efficiently by Relational databases that store data having structures and which have several records. The important point to be noted here is that these Relational database management Systems use tables with normalization concept. If the amount of data in such tables is huge, then handling such a large amount of data with its relationships is a tedious task. Here, Graph Databases come into picture. Entities and their relationships in relational databases will be reflected with nodes and relationships in graph databases. Graph databases provide very simple data model than databases with Online Transaction Processing systems. Graph databases provide features such as transactional integrity and operational availability. This paper introduces the idea of graph database systems in conjunction with Neo4j encompassing its query features, consistency, transactions, availability, and scaling.

Real-Time Detection of Crops with Dense Planting Using Deep Learning at Seedling Stage

Crop seedlings are similar in appearance to weeds, making crop detection extremely difficult. To solve the problem of detecting crop seedlings in complex field environments, a seedling dataset with four crops was constructed in this study. The single leaf labeling method was proposed as an alternative to conventional labeling approaches to improve the detection accuracy for dense planting crops. Second, a seedling detection network based on YOLOv5 and a transformer mechanism was proposed, and the effects of three features (query, key and value) in the transformer mechanism on the detection accuracy were explored in detail. Finally, the seedling detection network was optimized into a lightweight network. The experimental results show that application of the single leaf labeling method could improve the mAP0.5 of the model by 1.2% and effectively solve the problem of missed detection. By adding the transformer mechanism module, the mAP0.5 was improved by 1.5%, enhancing the detection capability of the model for dense and obscured targets. In the end, this study found that query features had the least impact on the transformer mechanism, and the optimized model improved the computation speed by 23 ms·frame−1 on the intelligent computing platform Jetson TX2, providing a theoretical basis and technical support for real-time seedling management.

Multi-stage distribution correction: A promising data augmentation method for few-shot fault diagnosis

Benefiting from the excellent capability of data processing, deep learning-based methods have been well applied in fault diagnosis. However, these methods may perform poorly due to lack of labeled data for training in real-world applications In few-shot learning settings, these methods may be trapped in overfitting since the data distribution estimated from a small number of labeled samples may be overly biased, which cannot cover the ground-truth data distribution. Given this problem, we propose a data augmentation method named Multi-Stage Distribution Correction (MSDC) for few-shot fault diagnosis. The proposed method can be divided into four training stages. In the first stage, unlabeled query features are clustered into several groups via an unsupervised Fuzzy C-means algorithm. Secondly, according to cosine similarity, the few labeled support features are assigned to the cluster with the highest similarity. Then, the Gaussian statistics of each cluster are extracted to correct the distributions of support features. Specifically, Our proposed method assumes that each dimension of the feature representation from the same class follows Gaussian distribution. New labeled features are generated by sampling from the corrected distributions. These generated features, along with the labeled support features, are used to train the task-specific classifier in the last stage. By joint training of the four stages, the scale of the labeled training dataset is effectively expanded, and the classification performance of the proposed method can be enhanced. The experimental results reveal that the proposed method outperforms the selected comparative methods in two case studies.

Grep: A Graph Learning Based Database Partitioning System

Database partitioning is a fundamental but challenging task in distributed databases, which selects specific columns as a partitioning key for each table and uses the partitioning key to allocate the table data into different compute nodes in order to maximize the performance. However, this problem is NP-hard and existing distributed databases require users to manually specify the partitioning keys, which may cause potential performance degradation. Although reinforcement learning based methods have been proposed, they have several limitations. First, they do not capture the complex data distributions and query access patterns, and thus involve high computation cost across different compute nodes to answer a query. Second, they involve an expensive step to repetitively partition the data into different compute nodes in order to train a learned key-selection model, which is a waste of time and resources. To address these limitations, we propose a practical learned database partitioning system Grep. We first adopt a graph model to encode data and query features, where vertices are columns, edges are query relations, and the weights of columns are computed based on the localized graph structures (e.g., data diversity, joined columns). We then utilize graph neural networks to embed the partitioning factors into embedding vectors in order to capture the data and query correlations. Next we propose a key-selection model to select appropriate partitioning keys based on the graph model. Finally, we propose an evaluation model to estimate the partitioning performance without actually partitioning the database. We have implemented Grep in a commercial distributed database, and experiments show the effectiveness of our system (e.g., 68% higher throughput for 30K queries in a real banking scenario).