Text Query Research Articles

Minebot: Chatbot to Respond to Text Queries Pertaining to Various Acts, Rules, And Regulations Applicable to Mining Industries

Minebot: Chatbot to Respond to Text Queries Pertaining to Various Acts, Rules, And Regulations Applicable to Mining Industries

Leaf Senescence Database v5.0: A Comprehensive Repository for Facilitating Plant Senescence Research

Through an extensive literature survey, we have upgraded the Leaf Senescence Database (LSD v5.0; https://ngdc.cncb.ac.cn/lsd/), a curated repository of comprehensive senescence-associated genes (SAGs) and their corresponding mutants. Since its inception in 2010, LSD undergoes frequent updates to encompass the latest advances in leaf senescence research and its current version comprises a high-quality collection of 31,740 SAGs and 1,209 mutants from 148 species, which were manually searched based on robust experimental evidence and further categorized according to their functions in leaf senescence. Furthermore, LSD was greatly enriched with comprehensive annotations for the SAGs through meticulous curation using both manual and computational methods. In addition, it was equipped with user-friendly web interfaces that facilitate text queries, BLAST searches, and convenient download of SAG sequences for localized analysis. Users can effortlessly navigate the database to access a plethora of information, including literature references, mutants, phenotypes, multi-omics data, miRNA interactions, homologs in other plants, and cross-links to various databases. Taken together, the upgraded version of LSD stands as the most comprehensive and informative plant senescence-related database to date, incorporating the largest collection of SAGs and thus bearing great utility for a wide range of studies related to plant senescence.

Searching the Genetic Programming Bibliography

Since February 2024 the genetic programming bibliography has supported free text queries to find GP papers via http://gpbib.cs.ucl.ac.uk/gp-search/

Hierarchical matching and reasoning for multi-query image retrieval

As a promising field, Multi-Query Image Retrieval (MQIR) aims at searching for the semantically relevant image given multiple region-specific text queries. Existing works mainly focus on a single-level similarity between image regions and text queries, which neglect the hierarchical guidance of multi-level similarities and result in incomplete alignments. Besides, the high-level semantic correlations that intrinsically connect different region–query pairs are rarely considered. To address above limitations, we propose a novel Hierarchical Matching and Reasoning Network (HMRN) for MQIR. It disentangles MQIR into three hierarchical semantic representations, which is responsible to capture fine-grained local details, contextual global scopes, and high-level inherent correlations. HMRN consists of two modules: Scalar-based Matching (SM) module and Vector-based Reasoning (VR) module. Specifically, the SM module characterizes the multi-level alignment similarity, which consists of a fine-grained local-level similarity and a context-aware global-level similarity. Afterwards, the VR module is developed to excavate the potential semantic correlations among multiple region–query pairs, which further explores the high-level reasoning similarity. Finally, these three-level similarities are aggregated into a joint similarity space to form the ultimate similarity. Extensive experiments on the benchmark dataset demonstrate that our HMRN substantially surpasses the current state-of-the-art methods. For instance, compared with the existing best method Drill-down, the metric R@1 in the last round is improved by 23.4%. Our source codes will be released at https://github.com/LZH-053/HMRN.

End-to-end Visual Grounding Based on Query Text Guidance and Multi-stage Reasoning

<p>Visual grounding locates target objects or areas in the image based on natural language expression. Most current methods extract visual features and text embeddings independently, and then carry out complex fusion reasoning to locate target objects mentioned in the query text. However, such independently extracted visual features often contain many features that are irrelevant to the query text or misleading, thus affecting the subsequent multimodal fusion module, and deteriorating target localization. This study introduces a combined network model based on the transformer architecture, which realizes more accurate visual grounding by using query text to guide visual feature generation and multi-stage fusion reasoning. Specifically, the visual feature generation module reduces the interferences of irrelevant features and generates visual features related to query text through the guidance of query text features. The multi-stage fused reasoning module uses the relevant visual features obtained by the visual feature generation module and the query text embeddings for multi-stage interactive reasoning, further infers the correlation between the target image and the query text, so as to achieve the accurate localization of the object described by the query text. The effectiveness of the proposed model is experimentally verified on five public datasets and the model outperforms state-of-the-art methods. It achieves an improvement of 1.04%, 2.23%, 1.00% and +2.51% over the previous state-of-the-art methods in terms of the top-1 accuracy on TestA and TestB of the RefCOCO and RefCOCO+ datasets, respectively.</p> <p> </p>

Deep learning-based information retrieval with normalized dominant feature subset and weighted vector model

Multimedia data, which includes textual information, is employed in a variety of practical computer vision applications. More than a million new records are added to social media and news sites every day, and the text content they contain has gotten increasingly complex. Finding a meaningful text record in an archive might be challenging for computer vision researchers. Most image searches still employ the tried and true language-based techniques of query text and metadata. Substantial work has been done in the past two decades on content-based text retrieval and analysis that still has limitations. The importance of feature extraction in search engines is often overlooked. Web and product search engines, recommendation systems, and question-answering activities frequently leverage these features. Extracting high-quality machine learning features from large text volumes is a challenge for many open-source software packages. Creating an effective feature set manually is a time-consuming process, but with deep learning, new actual feature demos from training data are analyzed. As a novel feature extraction method, deep learning has made great strides in text mining. Automatically training a deep learning model with the most pertinent text attributes requires massive datasets with millions of variables. In this research, a Normalized Dominant Feature Subset with Weighted Vector Model (NDFS-WVM) is proposed that is used for feature extraction and selection for information retrieval from big data using natural language processing models. The suggested model outperforms the conventional models in terms of text retrieval. The proposed model achieves 98.6% accuracy in information retrieval.

Improving Text-Based Person Retrieval by Excavating All-Round Information Beyond Color.

Text-based person retrieval is the process of searching a massive visual resource library for images of a particular pedestrian, based on a textual query. Existing approaches often suffer from a problem of color (CLR) over-reliance, which can result in a suboptimal person retrieval performance by distracting the model from other important visual cues such as texture and structure information. To handle this problem, we propose a novel framework to Excavate All-round Information Beyond Color for the task of text-based person retrieval, which is therefore termed EAIBC. The EAIBC architecture includes four branches, namely an RGB branch, a grayscale (GRS) branch, a high-frequency (HFQ) branch, and a CLR branch. Furthermore, we introduce a mutual learning (ML) mechanism to facilitate communication and learning among the branches, enabling them to take full advantage of all-round information in an effective and balanced manner. We evaluate the proposed method on three benchmark datasets, including CUHK-PEDES, ICFG-PEDES, and RSTPReid. The experimental results demonstrate that EAIBC significantly outperforms existing methods and achieves state-of-the-art (SOTA) performance in supervised, weakly supervised, and cross-domain settings.

Symbolic Music Generation From Graph-Learning-Based Preference Modeling and Textual Queries

Symbolic Music Generation From Graph-Learning-Based Preference Modeling and Textual Queries

Image-Specific Information Suppression and Implicit Local Alignment for Text-Based Person Search.

Text-based person search (TBPS) is a challenging task that aims to search pedestrian images with the same identity from an image gallery given a query text. In recent years, TBPS has made remarkable progress, and state-of-the-art (SOTA) methods achieve superior performance by learning local fine-grained correspondence between images and texts. However, most existing methods rely on explicitly generated local parts to model fine-grained correspondence between modalities, which is unreliable due to the lack of contextual information or the potential introduction of noise. Moreover, the existing methods seldom consider the information inequality problem between modalities caused by image-specific information. To address these limitations, we propose an efficient joint multilevel alignment network (MANet) for TBPS, which can learn aligned image/text feature representations between modalities at multiple levels, and realize fast and effective person search. Specifically, we first design an image-specific information suppression (ISS) module, which suppresses image background and environmental factors by relation-guided localization (RGL) and channel attention filtration (CAF), respectively. This module effectively alleviates the information inequality problem and realizes the alignment of information volume between images and texts. Second, we propose an implicit local alignment (ILA) module to adaptively aggregate all pixel/word features of image/text to a set of modality-shared semantic topic centers and implicitly learn the local fine-grained correspondence between modalities without additional supervision and cross-modal interactions. Also, a global alignment (GA) is introduced as a supplement to the local perspective. The cooperation of global and local alignment modules enables better semantic alignment between modalities. Extensive experiments on multiple databases demonstrate the effectiveness and superiority of our MANet.

CoVR-2: Automatic Data Construction for Composed Video Retrieval.

Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, comprising image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs, while also expanding the scope of the task to include composed video retrieval (CoVR). To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. Applying this methodology to the extensive WebVid2M collection, we automatically construct our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we introduce a new benchmark for CoVR with a manually annotated evaluation set, along with baseline results. We further validate that our methodology is equally applicable to image-caption pairs, by generating 3.3 million CoIR training triplets using the Conceptual Captions dataset. Our model builds on BLIP-2 pretraining, adapting it to composed video (or image) retrieval, and incorporates an additional caption retrieval loss to exploit extra supervision beyond the triplet, which is possible since captions are readily available for our training data by design. We provide extensive ablations to analyze the design choices on our new CoVR benchmark. Our experiments also demonstrate that training a CoVR model on our datasets effectively transfers to CoIR, leading to improved state-of-the-art performance in the zero-shot setup on the CIRR, FashionIQ, and CIRCO benchmarks. Our code, datasets, and models are publicly available at https://imagine.enpc.fr/~ventural/covr.

Improving First-stage Retrieval of Point-of-interest Search by Pre-training Models

Point-of-interest (POI) search is important for location-based services, such as navigation and online ride-hailing service. The goal of POI search is to find the most relevant destinations from a large-scale POI database given a text query. To improve the effectiveness and efficiency of POI search, most existing approaches are based on a multi-stage pipeline that consists of an efficiency-oriented retrieval stage and one or more effectiveness-oriented re-rank stages. In this article, we focus on the first efficiency-oriented retrieval stage of the POI search. We first identify the limitations of existing first-stage POI retrieval models in capturing the semantic-geography relationship and modeling the fine-grained geographical context information. Then, we propose a Geo-Enhanced Dense Retrieval framework for POI search to alleviate the above problems. Specifically, the proposed framework leverages the capacity of pre-trained language models (e.g., BERT) and designs a pre-training approach to better model the semantic match between the query prefix and POIs. With the POI collection, we first perform a token-level pre-training task based on a geographical-sensitive masked language prediction and design two retrieval-oriented pre-training tasks that link the address of each POI to its name and geo-location. With the user behavior logs collected from an online POI search system, we design two additional pre-training tasks based on users’ query reformulation behavior and the transitions between POIs. We also utilize a late-interaction network structure to model the fine-grained interactions between the text and geographical context information within an acceptable query latency. Extensive experiments on the real-world datasets collected from the Didichuxing application demonstrate that the proposed framework can achieve superior retrieval performance over existing first-stage POI retrieval methods.

Sparse graph matching network for temporal language localization in videos

Temporal language localization in videos aims to retrieve the moment that best matches the text description in the untrimmed video using the query text. Existing methods using graph convolutional networks have been effective in feature representation and cross-modal interaction, but the existing methods do not consider the sparsity constraint of the graph when constructing the graph structure, which can easily cause an increase in computational cost and introduce redundant connections that may adversely affect the accuracy of the results. Therefore, we propose a novel sparse graph matching network for temporal language localization in videos. Specifically, we use graph convolutional networks to learn video features and dynamically construct video graph with constraints on sparsity and connectivity; the complementarity between the sequential context and the syntactic structure of text is used to model the semantic features of text. For cross-modal interaction, we design a sparse graph matching method based on affinity matrix to match video and text graphs, and align cross-modal semantic features. Finally, by fusing the features of the two modalities, candidate moments are generated, and their confidence scores are calculated to locate the moment matching the query. Experimental results on the public datasets TACoS, ActivityNet Caption, and QVHighlights demonstrate the superiority of our method compared to state-of-the-art methods.

Towards Unsupervised Referring Expression Comprehension with Visual Semantic Parsing

Referring Expression Comprehension (REC) is a task that involves grounding a specific object in an image based on a given referring query in the form of bounding boxes. Existing fully-supervised or weakly-supervised REC methods rely on either manually annotated regional coordinates or query texts. In this paper, we propose an unsupervised training paradigm for the REC task that does not require any manual annotated data. Specifically, we introduce a Visual-Semantic-Parsing-based Unsupervised Referring Expression Comprehension framework (VUREC), which leverages a Visual Semantic Parser (VSP) as its core module to recognize the rich semantic information in images and construct pseudo-region-query pairs as the training supervision for REC. The VSP comprises a Scene Graph Parser (SGP) and a Visual Concept Detector (VCD) that can detect the locations, categories, attributes of objects, and visual relationships among them in images. Furthermore, we present a Referring Expression Reasoning (RER) model that utilizes a Multi-Modal Cascade Attention Decoder (MCAD) for fine-grained multi-modality fusion and regresses the four coordinates of the referential object directly. The experimental results on three benchmark datasets of Refcoco, Refcoco+ and Refcocog demonstrate the effectiveness of our proposed method.

Multi-modal interaction with transformers: bridging robots and human with natural language

AbstractThe language-guided visual robotic grasping task focuses on enabling robots to grasp objects based on human language instructions. However, real-world human-robot collaboration tasks often involve situations with ambiguous language instructions and complex scenarios. These challenges arise in the understanding of linguistic queries, discrimination of key concepts in visual and language information, and generation of executable grasping configurations for the robot’s end-effector. To overcome these challenges, we propose a novel multi-modal transformer-based framework in this study, which assists robots in localizing spatial interactions of objects using text queries and visual sensing. This framework facilitates object grasping in accordance with human instructions. Our developed framework consists of two main components. First, a visual-linguistic transformer encoder is employed to model multi-modal interactions for objects referred to in the text. Second, the framework performs joint spatial localization and grasping. Extensive ablation studies have been conducted on multiple datasets to evaluate the advantages of each component in our model. Additionally, physical experiments have been performed with natural language-driven human-robot interactions on a physical robot to validate the practicality of our approach.

WITHDRAWN: Efficient fragmentation and allocation on clustering in distributed environment (FACE)

Distributed data processing systems require a reliable storage structure that ensures consistent, high-scale, and scalable data access for all users. Data fragmentation and subsequent allocation are important aspects of a flexible distributed database system. Existing techniques may result in significant data loss while performing data allocation and fragmentation. To overcome these issues a novel fragmentation and Allocation on Clustering in Distributed Environment (FACE) is proposed for reliable storage with scalable data access for all users. Initially, the user query is given to distributed database system (DDBS), then the query is fed to the query parser. The query parser parses and converts a textual query into a query object. After parsing the query data, it is passed to the query optimizer, which optimizes the query to access the data more efficiently. In the next step, the query is fragmented using dual (horizontal and vertical) fuzzy C means clustering. In Dual fuzzy C means clustering, the vertical fragmentation focuses on clustering attribute values that are frequently accessed together, and horizontal fragmentation focuses on grouping records that meet data selection criteria. After fragmentation, the query is allocated using sequential rule mining, in which queries occur in sequential order. The allocated query is optimized using the Efficient Fragment Security (EFS) algorithm to normalize the parameters to achieve better results. The performance metrics of the proposed techniques are calculated based on Final Prediction Error (FPE), and Execution time. The proposed FACE achieves a lower execution time of the 1.56 s. The result shows that the proposed FACE decreases the overall execution time lower than the 4.70 s, 4.46 s, 3.3 Sec, and 3.14 s in MAPA, DMA, BBE, and RATS-HM respectively.

Fully Transformer-Equipped Architecture for end-to-end Referring Video Object Segmentation

Referring Video Object Segmentation (RVOS) requires segmenting the object in video referred by a natural language query. Existing methods mainly rely on sophisticated pipelines to tackle such cross-modal task, and do not explicitly model the object-level spatial context which plays an important role in locating the referred object. Therefore, we propose an end-to-end RVOS framework completely built upon transformers, termed Fully Transformer-Equipped Architecture (FTEA), which treats the RVOS task as a mask sequence learning problem and regards all the objects in video as candidate objects. Given a video clip with a text query, the visual–textual features are yielded by encoder, while the corresponding pixel-level and word-level features are aligned in terms of semantic similarity. To capture the object-level spatial context, we have developed the Stacked Transformer, which individually characterizes the visual appearance of each candidate object, whose feature map is decoded to the binary mask sequence in order directly. Finally, the model finds the best matching between mask sequence and text query. In addition, to diversify the generated masks for candidate objects, we impose a diversity loss on the model for capturing more accurate mask of the referred object. Empirical studies have shown the superiority of the proposed method on three benchmarks, e.g., FETA achieves 45.1% and 38.7% in terms of mAP on A2D Sentences (3782 videos) and J-HMDB Sentences (928 videos), respectively; it achieves 56.6% in terms of J&F on Ref-YouTube-VOS (3975 videos and 7451 objects). Particularly, compared to the best candidate method, it has a gain of 2.1% and 3.2% in terms of P@0.5 on the former two, respectively, while it has a gain of 2.9% in terms of J on the latter one.

PMG—Pyramidal Multi-Granular Matching for Text-Based Person Re-Identification

Given a textual query, text-based person re-identification is supposed to search for the targeted pedestrian images from a large-scale visual database. Due to the inherent heterogeneity between different modalities, it is challenging to measure the cross-modal affinity between visual and textual data. Existing works typically employ single-granular methods to extract local features and align image regions with relevant words/phrases. Nevertheless, the limited robustness of single-granular methods cannot adapt to the imprecision and variances of visual and textual features, which are usually influenced by the background clutter, position transformation, posture diversity, and occlusion in surveillance videos, thereby leading to the deterioration of cross-modal matching accuracy. In this paper, we propose a Pyramidal Multi-Granular matching network (PMG) that incorporates a gradual transition process between the coarsest global information and the finest local information by a coarse-to-fine pyramidal method for multi-granular cross-modal features extraction and affinities learning. For each body part of a pedestrian, PMG is adequate in ensuring the integrity of local information while minimizing the surrounding interference signals at a certain scale and can adapt to capture discriminative signals of different body parts and achieve semantically alignment between image strips with relevant textual descriptions, thus suppressing the variances of feature extraction and improving the robustness of feature matching. Comprehensive experiments are conducted on the CUHK-PEDES and RSTPReid datasets to validate the effectiveness of the proposed method and results show that PMG outperforms state-of-the-art (SOTA) methods significantly and yields competitive accuracy of cross-modal retrieval.

Query-Guided Refinement and Dynamic Spans Network for Video Highlight Detection and Temporal Grounding in Online Information Systems

With the surge in online video content, finding highlights and key video segments have garnered widespread attention. Given a textual query, video highlight detection (HD) and temporal grounding (TG) aim to predict frame-wise saliency scores from a video while concurrently locating all relevant spans. Despite recent progress in DETR-based works, these methods crudely fuse different inputs in the encoder, which limits effective cross-modal interaction. To solve this challenge, the authors design QD-Net (query-guided refinement and dynamic spans network) tailored for HD&TG. Specifically, they propose a query-guided refinement module to decouple the feature encoding from the interaction process. Furthermore, they present a dynamic span decoder that leverages learnable 2D spans as decoder queries, which accelerates training convergence for TG. On QVHighlights dataset, the proposed QD-Net achieves 61.87 HD-HIT@1 and 61.88 TG-mAP@0.5, yielding a significant improvement of +1.88 and +8.05, respectively, compared to the state-of-the-art method.

Webly-supervised semantic segmentation via curriculum learning

In this paper, we propose a weakly supervised semantic segmentation method by directly learning from web images, which are crawled from the Internet by using text queries, without any explicit user annotation or even data filtering. With the goal of handling the massive amount of noisy labels in web images, we design a three-stage approach for weakly-supervised semantic segmentation based on curriculum learning. We first generate pixel-level masks for the training images via a popular weakly-supervised semantic segmentation framework. Then, we consider the noise of the web data in two ways. At the image-level, the complexity of data is measured using its distribution density in a classification feature space. At the pixel-level, the complexity of the mask is evaluated by exploiting the relationship between the saliency map and those segmented images in an unsupervised manner. The key insight to this design is that, common and simple object patterns in images should be salient with both the saliency detector and weakly supervised DCNNs, where they should be sparse with high regional consistency between them. This allows for an efficient implementation of curriculum learning from noisy web images. Experiments on the popular PASCAL VOC 2012 benchmark show that we achieve very competitive performance with scores of 64.0% mIoU using our pure web dataset, which contains noisy, single-label images. We further improve the performance to 69.7% mIoU by using the CurriculumWebSegNet fine-tuned on the PASCAL VOC dataset, which has more precise multi-label supervision.

Image Retrieval Through Free-Form Query using Intelligent Text Processing

Image Retrieval is the process of retrieving images from the image/multimedia databases. Retrieval of images are carried out with various types of queries, free-form query is a text-query that consists of single or multiple keywords and/or concepts or descriptions of images with or without the inclusion of wild-card characters and/or punctuations. This work aims to handle image retrieval based on free-form text queries. Simple & complex queries of conceptual descriptions of images are explored and an intelligent processing system with free-form queries based on the Bag-of-Words model is modified and built for natural scene images and on Diverse Social Images using the Damerau-Levenshtein edit distance measure. The efficacy of the proposed system is evaluated by testing 1500 free-form text queries and has resulted in a recall accuracy of 91.3% on natural scene images (of Wang/Corel database) and 100% on Diverse Social Images (of DIV400 dataset). These results show that the system proposed has produced satisfactory performance compared to published results such as the harmonic mean of precision and recall (i.e. F1-Score) of 76.70% & 63.32% at retrieval of 20 images etc in reported works.