Query Vector Research Articles

Prototype-Based Semantic Segmentation.

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

Application of End-to-End Perception Framework Based on Boosted DETR in UAV Inspection of Overhead Transmission Lines

As crucial predecessor tasks for fault detection and transmission line inspection, insulators, anti-vibration hammers, and arc sag detection are critical jobs. Due to the complexity of the high-voltage transmission line environment and other factors, target detection work on transmission lines remains challenging. A method for high-voltage transmission line inspection based on DETR (TLI-DETR) is proposed to detect insulators, anti-vibration hammers, and arc sag. This model achieves a better balance in terms of speed and accuracy than previous methods. Due to environmental interference such as mountainous forests, rivers, and lakes, this paper uses the Improved Multi-Scale Retinex with Color Restoration (IMSRCR) algorithm to make edge extraction more robust with less noise interference. Based on the TLI-DETR’s feature extraction network, we introduce the edge and semantic information by Momentum Comparison (MoCo) to boost the model’s feature extraction ability for small targets. The different shooting angles and distances of drones result in the target images taking up small proportions and impeding each other. Consequently, the statistical profiling of the area and aspect ratio of transmission line targets captured by UAV generate target query vectors with prior information to enable the model to adapt to the detection needs of transmission line targets more accurately and effectively improve the detection accuracy of small targets. The experimental results show that this method has excellent performance in high-voltage transmission line detection, achieving up to 91.65% accuracy and a 55FPS detection speed, which provides a technical basis for the online detection of transmission line targets.

Sea–Land Segmentation of Remote-Sensing Images with Prompt Mask-Attention

Remote-sensing technology has gradually become one of the most important ways to extract sea–land boundaries due to its large scale, high efficiency, and low cost. However, sea–land segmentation (SLS) is still a challenging problem because of data diversity and inconsistency, “different objects with the same spectrum” or “the same object with different spectra”, and noise and interference problems, etc. In this paper, a new sea–land segmentation method (PMFormer) for remote-sensing images is proposed. The contributions are mainly two points. First, based on Mask2Former architecture, we introduce the prompt mask by normalized difference water index (NDWI) of the target image and prompt encoder architecture. The prompt mask provides more reasonable constraints for attention so that the segmentation errors are alleviated in small region boundaries and small branches, which are caused by insufficiency of prior information by large data diversity or inconsistency. Second, for the large intra-class difference problem in the foreground–background segmentation in sea–land scenes, we use deep clustering to simplify the query vectors and make them more suitable for binary segmentation. Then, traditional NDWI and eight other deep-learning methods are thoroughly compared with the proposed PMFormer on three open sea–land datasets. The efficiency of the proposed method is confirmed, after the quantitative analysis, qualitative analysis, time consumption, error distribution, etc. are presented by detailed contrast experiments.

Indonesian-English Textual Similarity Detection Using Universal Sentence Encoder (USE) and Facebook AI Similarity Search (FAISS)

The tremendous development in Natural Language Processing (NLP) has enabled the detection of bilingual and multilingual textual similarity. One of the main challenges of the Textual Similarity Detection (TSD) system lies in learning effective text representation. The research focuses on identifying similar texts between Indonesian and English across a broad range of semantic similarity spectrums. The primary challenge is generating English and Indonesian dense vector representation, a.k.a. embeddings that share a single vector space. Through trial and error, the research proposes using the Universal Sentence Encoder (USE) model to construct bilingual embeddings and FAISS to index the bilingual dataset. The comparison between query vectors and index vectors is done using two approaches: the heuristic comparison with Euclidian distance and a clustering algorithm, Approximate Nearest Neighbors (ANN). The system is tested with four different semantic granularities, two text granularities, and evaluation metrics with a cutoff value of k={2,10}. Four semantic granularities used are highly similar or near duplicate, Semantic Entailment (SE), Topically Related (TR), and Out of Topic (OOT), while the text granularities take on the sentence and paragraph levels. The experimental results demonstrate that the proposed system successfully ranks similar texts in different languages within the top ten. It has been proven by the highest F1@2 score of 0.96 for the near duplicate category on the sentence level. Unlike the near-duplicate category, the highest F1 scores of 0.77 and 0.89 are shown by the SE and TR categories, respectively. The experiment results also show a high correlation between text and semantic granularity.

Decoupled Cross-Modal Transformer for Referring Video Object Segmentation.

Referring video object segmentation (R-VOS) is a fundamental vision-language task which aims to segment the target referred by language expression in all video frames. Existing query-based R-VOS methods have conducted in-depth exploration of the interaction and alignment between visual and linguistic features but fail to transfer the information of the two modalities to the query vector with balanced intensities. Furthermore, most of the traditional approaches suffer from severe information loss in the process of multi-scale feature fusion, resulting in inaccurate segmentation. In this paper, we propose DCT, an end-to-end decoupled cross-modal transformer for referring video object segmentation, to better utilize multi-modal and multi-scale information. Specifically, we first design a Language-Guided Visual Enhancement Module (LGVE) to transmit discriminative linguistic information to visual features of all levels, performing an initial filtering of irrelevant background regions. Then, we propose a decoupled transformer decoder, using a set of object queries to gather entity-related information from both visual and linguistic features independently, mitigating the attention bias caused by feature size differences. Finally, the Cross-layer Feature Pyramid Network (CFPN) is introduced to preserve more visual details by establishing direct cross-layer communication. Extensive experiments have been carried out on A2D-Sentences, JHMDB-Sentences and Ref-Youtube-VOS. The results show that DCT achieves competitive segmentation accuracy compared with the state-of-the-art methods.

Discriminative Feature Enhancement Network for few-shot classification and beyond

Few-shot classification aims to recognize query samples from novel classes given scarce labeled data, which remains a challenging problem in machine learning. This paper proposes a Discriminative Feature Enhancement Network (DFENet) to distinguish the discriminative feature of a novel category with the following characteristics: (1) a Cross-Modal Guidance Module (CMG-Module) is proposed to enrich the query vectors via leveraging label information from the additional modality; (2) a Neural-Decoding based Attention Module (NDA-Module) further explores the relationship between the query and support samples, and the attention weights depend on contribution of the support to the reconstructed query. The main intuition of the NDA-Module is to deepen our understanding of the self-attention mechanism from a neural decoding perspective, emphasizing that the best reconstruction can be used as a universal guideline; (3) a flexible triplet loss is designed to incorporate the semantic context among all classes and distinguish samples from different classes. The experimental results on two few-shot classification datasets show that our DFENet can learn a more discriminative representation for novel classes progressively. We also test the proposed method on image retrieval and facial expression recognition tasks, demonstrating consistent improvement.

HGCTNet: Handcrafted Feature-Guided CNN and Transformer Network for Wearable Cuffless Blood Pressure Measurement.

Biosignals collected by wearable devices, such as electrocardiogram and photoplethysmogram, exhibit redundancy and global temporal dependencies, posing a challenge in extracting discriminative features for blood pressure (BP) estimation. To address this challenge, we propose HGCTNet, a handcrafted feature-guided CNN and transformer network for cuffless BP measurement based on wearable devices. By leveraging convolutional operations and self-attention mechanisms, we design a CNN-Transformer hybrid architecture to learn features from biosignals that capture both local information and global temporal dependencies. Then, we introduce a handcrafted feature-guided attention module that utilizes handcrafted features extracted from biosignals as query vectors to eliminate redundant information within the learned features. Finally, we design a feature fusion module that integrates the learned features, handcrafted features, and demographics to enhance model performance. We validate our approach using two large wearable BP datasets: the CAS-BP dataset and the Aurora-BP dataset. Experimental results demonstrate that HGCTNet achieves an estimation error of 0.9 ± 6.5 mmHg for diastolic BP (DBP) and 0.7 ± 8.3 mmHg for systolic BP (SBP) on the CAS-BP dataset. On the Aurora-BP dataset, the corresponding errors are -0.4 ± 7.0 mmHg for DBP and -0.4 ± 8.6 mmHg for SBP. Compared to the current state-of-the-art approaches, HGCTNet reduces the mean absolute error of SBP estimation by 10.68% on the CAS-BP dataset and 9.84% on the Aurora-BP dataset. These results highlight the potential of HGCTNet in improving the performance of wearable cuffless BP measurements.

Extracting Key-phrase Embedding using Deep Average Network and Maximal Marginal Relevance to Enhance Information Retrieval

Background: Automatic keyphrase extraction (AKE) is essential to many NLP and information retrieval tasks. Extracting high-quality key phrases is difficult due to technological advancements and the exponential growth ‎of ‎textual data and digital sources. Unsupervised keyphrase extraction with cheap computing cost that relies ‎on ‎heuristic notions of phrase importance such as embedding similarities but their development necessitates in-depth subject expertise. Materials and Methods: This paper presents a method to obtain a semantic understanding of the query ‎and index documents by using ‎the embedding technique(universal Sentence ‎encoder (USE) ) while keeping the most informative ‎using Maximal Marginal Relevance (MMR) and then scoring(an ‎inverted index) the most documents relevant to the query ‎vector to improve the ‎performance of IR systems.‎ Results: The proposed retrieval model implement on the (Fire2011) dataset. The final ‎stage was evaluating the results of the baseline and the results (indexing and ranking) by using mean average precision (MAP). The ‎result of the baseline was 0.61, while the result ‎inverted index was 0.6277519 .‎ Conclusions: In this paper, we have discussed document retrieval using keep key phrases that ‎have ‏informativeness ‎properties by ‎using maximal ‎marginal ‎relevance, since if we extract a fixed number of top ‎keyphrases, ‎redundancy ‎hinders the diversification of the ‎extracted keyphrases.

Adaptive weighted multi-view subspace clustering method for recognizing urban functions from multi-source social sensing data

ABSTRACT Multi-source social sensing data provide new opportunities to identify urban functions from the perspective of human activity. The information embedded in multi-source data typically needs to be fused to obtain a comprehensive view of urban functions. Although multi-view clustering has been successfully used to fuse multi-source social sensing data, the adaptive determination of fusion weights for high-dimensional and noisy multi-source social sensing data remains challenging. Therefore, this study proposes an adaptive weighted multi-view subspace clustering (AWMSC) method. First, we use two neural networks to map multi-source data into a common latent representation and multiple specific latent representations, which serve as the query vector and input vectors of the attention mechanism, respectively. Then, the weight of each type of data is calculated based on the attention mechanism. Finally, the specific latent representations of the multi-source data are weighted and fused into a shared subspace representation, which is used as the input of the spectral clustering algorithm to obtain clustering results. AWMSC is applied to identify urban functional zones in Beijing using bus transactions, taxi trajectories, and points of interest datasets. The results show that AWMSC outperforms the typical single-view, weighted-average, and representative multi-view methods. AWMSC can obtain a comprehensive understanding of urban functional zones which may help government departments make more accurate strategic decisions.

DBGCN

Given the extensive use of online learning in educational settings, Knowledge Tracing (KT) is becoming increasingly essential. KT primarily aims to predict a student's future knowledge acquisition based on their past learning activities, thus enhancing the efficiency of student learning. However, the effective acquisition of dynamic and evolving student representations from their historical records presents a formidable challenge. This paper introduces a Knowledge Tracing methodology predicated on Dynamic Broadth Graph Convolutional Networks (DBGCN). DBGCN leverages the mechanisms of breadth graph convolutional networks to proficiently acquire representations of questions and knowledge points from dynamically constructed topological graphs. It employs student state information as an attention query vector to augment student representations, thereby partially mitigating the challenge of capturing the dynamic shifts in user states. The effectiveness of our proposed DBGCN method has been demonstrated through extensive experimentation.

Deep code search efficiency based on clustering

AbstractThe deep‐learning based code search model mainly takes accuracy as the only target for judging the performance of the model, ignoring the efficiency of code search. This article proposes a clustering‐based code search model (C‐DCS). C‐DCS uses the K‐Means to divide the code vector base into K clusters and obtains the center vectors of K clusters. While searching, C‐DCS first matches the query vector with the K center vectors to get the best matching center vector. After matching the center vector, C‐DCS matches the query vector with code vectors in the cluster corresponding to the best matching center vector one by one and then gets the best matching code snippet vector. To verify the efficiency of C‐DCS in the code search task, experimental analysis was built on a large dataset. The experimental results showed that C‐DCS saves 92.2% of the search time compared to the baseline model while remaining the accuracy. In the experimental evaluation section, we optimized the K‐Means algorithm to improve the code search efficiency of C‐DCS further, reducing the search time to 93.8% of the baseline model. Hence, C‐DCS reduces the code search time greatly with not affecting the accuracy, improving the efficiency of software development.

SeRF: Segment Graph for Range-Filtering Approximate Nearest Neighbor Search

Effective vector representation models, e.g., word2vec and node2vec, embed real-world objects such as images and documents in high dimensional vector space. In the meanwhile, the objects are often associated with attributes such as timestamps and prices. Many scenarios need to jointly query the vector representations of the objects together with their attributes. These queries can be formalized as range-filtering approximate nearest neighbor search (ANNS) queries. Specifically, given a collection of data vectors, each associated with an attribute value whose domain has a total order. The range-filtering ANNS consists of a query range and a query vector. It finds the approximate nearest neighbors of the query vector among all the data vectors whose attribute values fall in the query range. Existing approaches suffer from a rapidly degrading query performance when the query range width shifts. The query performance can be optimized by a solution that builds an ANNS index for every possible query range; however, the index time and index size become prohibitive -- the number of query ranges is quadratic to the number n of data vectors. To overcome these challenges, for the query range contains all attribute values smaller than a user-provided threshold, we design a structure called the segment graph whose index time and size are the same as a single ANNS index, yet can losslessly compress the n ANNS indexes, reducing the indexing cost by a factor of Ω(n). To handle general range queries, we propose a 2D segment graph with average-case index size O(n log n) to compress n segment graphs, breaking the quadratic barrier. Extensive experiments conducted on real-world datasets show that our proposed structures outperformed existing methods significantly; our index also exhibits superior scalability.

Black-Box Boundary Attack Based on Gradient Optimization

Deep neural networks have gained extensive applications in computer vision, demonstrating significant success in fundamental research tasks such as image classification. However, the robustness of these networks faces severe challenges in the presence of adversarial attacks. In real-world scenarios, addressing hard-label attacks often requires the execution of tens of thousands of queries. To combat these challenges, the Black-Box Boundary Attack leveraging Gradient Optimization (GOBA) has been introduced. This method employs a binary search strategy to acquire an initial adversarial example with significant perturbation. The Monte Carlo algorithm is utilized to estimate the gradient of the sample, facilitating iterative movement along the estimated gradient and the direction of the malicious label. Moreover, query vectors positively correlated with the gradient are extracted to construct a sampling space with an optimal scale, thereby enhancing the efficiency of the Monte Carlo algorithm. Experimental evaluations were conducted using the HSJA, QEBA, and NLBA attack methodologies on the ImageNet, CelebA, and MNIST datasets, respectively. The results indicate that, under the constraint of 3 k query times, the GOBA, compared to other methods, can, on average, reduce perturbation (L2 distance) by 55.74% and simultaneously increase the attack success rate by an average of 13.78%.

Simple Conditional Spatial Query Mask Deformable Detection Transformer: A Detection Approach for Multi-Style Strokes of Chinese Characters

In the Chinese character writing task performed by robotic arms, the stroke category and position information should be extracted through object detection. Detection algorithms based on predefined anchor frames have difficulty resolving the differences among the many different styles of Chinese character strokes. Deformable detection transformer (deformable DETR) algorithms without predefined anchor frames result in some invalid sampling points with no contribution to the feature update of the current reference point due to the random sampling of sampling points in the deformable attention module. These processes cause a reduction in the speed of the vector learning stroke features in the detection head. In view of this problem, a new detection method for multi-style strokes of Chinese characters, called the simple conditional spatial query mask deformable DETR (SCSQ-MDD), is proposed in this paper. Firstly, a mask prediction layer is jointly determined using the shallow feature map of the Chinese character image and the query vector of the transformer encoder, which is used to filter the points with actual contributions and resample the points without contributions to address the randomness of the correlation calculation among the reference points. Secondly, by separating the content query and spatial query of the transformer decoder, the dependence of the prediction task on the content embedding is relaxed. Finally, the detection model without predefined anchor frames based on the SCSQ-MDD is constructed. Experiments are conducted using a multi-style Chinese character stroke dataset to evaluate the performance of the SCSQ-MDD. The mean average precision (mAP) value is improved by 3.8% and the mean average recall (mAR) value is improved by 1.1% compared with the deformable DETR in the testing stage, illustrating the effectiveness of the proposed method.

A SWOT-Based Framework for Personalized Ranking of IoT Service Providers With Generalized Fuzzy Data for Sustainable Transport in Urban Regions

Sustainable transport in cities has been gaining a lot of attraction recently and a core focus of engineering management. Internet of Things (IoT) is seen as a widely accepted technology that promotes sustainability through the interconnection of diverse computing sources for solving environmental problems. Previous studies on IoT have discussed interesting factors toward its adoption, but selecting a suitable IoT service provider (IoTSP) is an open challenge due to a diverse set of factors in practice. Driven by the challenge, in this article, a generalized fuzzy-based decision model is put forward for IoTSP selection, which is the prime objective of the study. Initially, a strength, weakness, opportunity, threat (SWOT) analysis is adopted to identify the crucial challenges in IoT adoption. Later, the relative significance of these challenges is calculated by adopting the regret/rejoice approach. Due to uncertainty, certain rating information of IoTSPs is missing that are rationally imputed by proposing an analytical approach. Rating matrices from experts are transformed into opinion vectors, and a prioritization algorithm is developed with query vector for rational personalized ordering of IoTSPs. Data for the study are acquired via questionnaire, which is filled by experts. The efficacy of the developed model is exemplified by using a real case study of IoTSP selection for pollution management in Chennai. Concerning the findings, mobility, security, and connectivity are the most vital factors for IoTSP selection. Results show that the proposed model is a viable tool for IoTSP selection and it is robust, unique, and stable compared to its counterparts.

Semi-supervised inverted file index approach for approximate nearest neighbor search

This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc.

Utilizing cognitive signals generated during human reading to enhance keyphrase extraction from microblogs

Microblogging platforms have seen exponential growth, leading to an abundance of user-generated content. The challenge now is to efficiently extract crucial information from this vast and dispersed text data. It also serves as the goal of our research on Automatic Keyphrase Extraction (AKE) for microblog. Eye-tracking signals, that reflect users' tendency to prioritize certain words while reading, have been employed to enhance AKE performance from microblogs. However, relying solely on eye-tracking has its limitations owing to constraints in physiological mechanism support, acquisition techniques, and feature decoding. Consequently, we propose the integration of electroencephalogram (EEG) signals with eye-tracking signals to improve microblogs-based AKE, thereby overcoming the aforementioned limitations. Our first step is identifying specific features present in cognitive signals generated during human reading. We selected EEG signals (8 features) and eye-tracking signals (17 features) from the cognitive language processing corpus ZUCO, to examine the efficacy when they are combined with the microblogs-based AKE. To avoid cognitive signal distortion by certain model structures, we introduced these signals at the inputs of the soft attention layer and at the query vectors of the self-attention layer. For evaluation, we performed several AKE tests on microblogs with various combinations of cognitive signals. The results demonstrate a consistent enhancement in the performance of AKE due to cognitive signals generated during human reading, regardless of different feature combinations and models. Specifically, EEG signals exhibited the most significant improvement. However, combining EEG signals with eye-tracking signals yielded results that fell between the performance levels of the two signal types, indicating that their integration might have some synergistic effects. Further investigation is needed to understand the underlying mechanisms responsible for this outcome. The code and dataset for this paper can be accessed at https://github.com/yan-xinyi/AKE.

Adaptive2Former: Enhancing Chromosome Instance Segmentation with Adaptive Query Decoder

Chromosome instance segmentation plays a crucial role in chromosomal karyotype analysis. However, the overlapping of chromosome instances and their individual morphological differences make accurate chromosome instance segmentation a challenging task. Especially in handling overlapping chromosome instances, traditional segmentation methods tend to confuse instances with one another. To solve these problems, this paper proposes an innovative method named Adaptive2Former. It builds upon our novel devised Adaptive Query Decoder (AQD) module to enhance segmentation precision. The AQD effectively utilizes the [ cls ] token from the backbone network to dynamically generate adaptive query vectors instead of using fixed queries. This new design leverages the semantic information inside the input images, producing representations more conducive to subsequent segmentation module, thereby improving the model’s segmentation performance. Experiments conducted on our dataset demonstrate that the proposed Adaptive2Former significantly enhances the performance of chromosome instance segmentation compared to Mask2Former and other existing models, achieving results of 97.65% mAP75 and 0.21 Dice Loss.

Predicting Pedestrian Trajectories with Deep Adversarial Networks Considering Motion and Spatial Information

This paper proposes a novel prediction model termed the social and spatial attentive generative adversarial network (SSA-GAN). The SSA-GAN framework utilizes a generative approach, where the generator employs social attention mechanisms to accurately model social interactions among pedestrians. Unlike previous methodologies, our model utilizes comprehensive motion features as query vectors, significantly enhancing predictive performance. Additionally, spatial attention is integrated to encapsulate the interactions between pedestrians and their spatial context through semantic spatial features. Moreover, we present a novel approach for generating simulated multi-trajectory datasets using the CARLA simulator. This method circumvents the limitations inherent in existing public datasets such as UCY and ETH, particularly when evaluating multi-trajectory metrics. Our experimental findings substantiate the efficacy of the proposed SSA-GAN model in capturing the nuances of pedestrian interactions and providing accurate multimodal trajectory predictions.

Sketch-based semantic retrieval of medical images

The volume of medical images stored in hospitals is rapidly increasing; however, the utilization of these accumulated medical images remains limited. Existing content-based medical image retrieval (CBMIR) systems typically require example images, leading to practical limitations, such as the lack of customizable, fine-grained image retrieval, the inability to search without example images, and difficulty in retrieving rare cases. In this paper, we introduce a sketch-based medical image retrieval (SBMIR) system that enables users to find images of interest without the need for example images. The key concept is feature decomposition of medical images, which allows the entire feature of a medical image to be decomposed into and reconstructed from normal and abnormal features. Building on this concept, our SBMIR system provides an easy-to-use two-step graphical user interface: users first select a template image to specify a normal feature and then draw a semantic sketch of the disease on the template image to represent an abnormal feature. The system integrates both types of input to construct a query vector and retrieves reference images. For evaluation, ten healthcare professionals participated in a user test using two datasets. Consequently, our SBMIR system enabled users to overcome previous challenges, including image retrieval based on fine-grained image characteristics, image retrieval without example images, and image retrieval for rare cases. Our SBMIR system provides on-demand, customizable medical image retrieval, thereby expanding the utility of medical image databases.