Retrieval Datasets Research Articles

Robust multimodal discrete hashing for cross-modal similarity search

Hashing technology improves the search efficiency and reduces the storage space of data. However, building an effective modal with unsupervised cross modal retrieval and generating efficient binary code is still a challenging task, considering of some issues needed to be further discussed and researched for unsupervised multimodal hashing. Most of the existing methods ignore the discrete restriction, and manually or experientially determine the weights of each modality. These limitations may significantly reduce the retrieval accuracy of unsupervised cross-modal hashing methods. To solve these problems, we propose a robust hash modal that can efficiently learn binary code by employing a flexible and noise-resistant ℓ2,1-loss with nonlinear kernel embedding. In addition, we introduce an intermediate state mapping that facilitate later modal optimization to measure the loss between the hash codes and the intermediate states. Experiments on several public multimedia retrieval datasets validate the superiority of the proposed method from various aspects.

Towards Flexible Retrieval, Integration and Analysis of JSON Data Sets through Fuzzy Sets: A Case Study

How to exploit the incredible variety of JSON data sets currently available on the Internet, for example, on Open Data portals? The traditional approach would require getting them from the portals, then storing them into some JSON document store and integrating them within the document store. However, once data are integrated, the lack of a query language that provides flexible querying capabilities could prevent analysts from successfully completing their analysis. In this paper, we show how the J-CO Framework, a novel framework that we developed at the University of Bergamo (Italy) to manage large collections of JSON documents, is a unique and innovative tool that provides analysts with querying capabilities based on fuzzy sets over JSON data sets. Its query language, called J-CO-QL, is continuously evolving to increase potential applications; the most recent extensions give analysts the capability to retrieve data sets directly from web portals as well as constructs to apply fuzzy set theory to JSON documents and to provide analysts with the capability to perform imprecise queries on documents by means of flexible soft conditions. This paper presents a practical case study in which real data sets are retrieved, integrated and analyzed to effectively show the unique and innovative capabilities of the J-CO Framework.

Knowledge discovery and visualisation framework using machine learning for music information retrieval from broadcast radio data

Music radio data is currently underutilised in radio program management. Software tools that listen to and analyse music airplay are in many markets nonexistent, limited, or unaffordable. In this paper we present a novel knowledge discovery and visualisation framework for broadcast radio, ZeitMetric. The ZeitMetric framework uses machine learning and music information retrieval techniques to label radio audio automatically for knowledge discovery.The framework incorporates a novel music dataset collection technique (MusiGrab) to leverage online music services for ground-truth data, as well as a novel knowledge visualisation and presentation technique based on self-organizing maps (ZeitViz). The framework is compared to what little literature relating to this topic exists, and a set of requirements for a high-quality broadcast radio knowledge discovery is developed. MusiGrab specifically is compared to an existing static music information retrieval dataset and shown to offer superior results in this context. Future research directions using and extending the framework are also discussed. On acceptance of the paper, code for a use-case of the MusiGrab dataset collection technique will be released on GitHub.

A Test Collection for Dataset Retrieval in Biodiversity Research

Searching for scientific datasets is a prominent task in scholars' daily research practice. A variety of data publishers, archives and data portals offer search applications that allow the discovery of datasets. The evaluation of such dataset retrieval systems requires proper test collections, including questions that reflect real world information needs of scholars, a set of datasets and human judgements assessing the relevance of the datasets to the questions in the benchmark corpus. Unfortunately, only very few test collections exist for a dataset search. In this paper, we introduce the BEF-China test collection, the very first test collection for dataset retrieval in biodiversity research, a research field with an increasing demand in data discovery services. The test collection consists of 14 questions, a corpus of 372 datasets from the BEF-China project and binary relevance judgements provided by a biodiversity expert.

A decision support system for automating document retrieval and citation screening

The systematic literature review (SLR) process includes several steps to collect secondary data and analyze it to answer research questions. In this context, the document retrieval and primary study selection steps are heavily intertwined and known for their repetitiveness, high human workload, and difficulty identifying all relevant literature. This study aims to reduce human workload and error of the document retrieval and primary study selection processes using a decision support system (DSS). An open-source DSS is proposed that supports the document retrieval step, dataset preprocessing, and citation classification. The DSS is domain-independent, as it has proven to carefully select an article’s relevance based solely on the title and abstract. These features can be consistently retrieved from scientific database APIs. Additionally, the DSS is designed to run in the cloud without any required programming knowledge for reviewers. A Multi-Channel CNN architecture is implemented to support the citation screening process. With the provided DSS, reviewers can fill in their search strategy and manually label only a subset of the citations. The remaining unlabeled citations are automatically classified and sorted based on probability. It was shown that for four out of five review datasets, the DSS's use achieved significant workload savings of at least 10%. The cross-validation results show that the system provides consistent results up to 88.3% of work saved during citation screening. In two cases, our model yielded a better performance over the benchmark review datasets. As such, the proposed approach can assist the development of systematic literature reviews independent of the domain. The proposed DSS is effective and can substantially decrease the document retrieval and citation screening steps' workload and error rate.

A Unified Pretraining Framework for Passage Ranking and Expansion

Pretrained language models have recently advanced a wide range of natural language processing tasks. Nowadays, the application of pretrained language models to IR tasks has also achieved impressive results. Typical methods either directly apply a pretrained model to improve the re-ranking stage, or use it to conduct passage expansion and term weighting for first-stage retrieval. We observe that the passage ranking and passage expansion tasks share certain inherent relations, and can benefit from each other. Therefore, in this paper, we propose a general pretraining framework to enhance both tasks with Unified Encoder-Decoder networks (UED). The overall ranking framework consists of two parts in a cascade manner: (1) passage expansion with a pretraining-based query generation method; (2) re-ranking of passage candidates from a traditional retrieval method with a pretrained transformer encoder. Both the two parts are based on the same pretrained UED model, where we jointly train the passage ranking and query generation tasks for further improving the full ranking pipeline. An extensive set of experiments have been conducted on two large-scale passage retrieval datasets to demonstrate the state-of-the-art results of the proposed framework in both the first-stage retrieval and the final re-ranking. In addition, we successfully deploy the framework to our online production system, which can stably serve industrial applications with a request volume of up to 100 QPS in less than 300ms.

Deep Graph Matching and Searching for Semantic Code Retrieval

Code retrieval is to find the code snippet from a large corpus of source code repositories that highly matches the query of natural language description. Recent work mainly uses natural language processing techniques to process both query texts (i.e., human natural language) and code snippets (i.e., machine programming language), however, neglecting the deep structured features of query texts and source codes, both of which contain rich semantic information. In this article, we propose an end-to-end deep graph matching and searching (DGMS) model based on graph neural networks for the task of semantic code retrieval. To this end, we first represent both natural language query texts and programming language code snippets with the unified graph-structured data, and then use the proposed graph matching and searching model to retrieve the best matching code snippet. In particular, DGMS not only captures more structural information for individual query texts or code snippets, but also learns the fine-grained similarity between them by cross-attention based semantic matching operations. We evaluate the proposed DGMS model on two public code retrieval datasets with two representative programming languages (i.e., Java and Python). Experiment results demonstrate that DGMS significantly outperforms state-of-the-art baseline models by a large margin on both datasets. Moreover, our extensive ablation studies systematically investigate and illustrate the impact of each part of DGMS.

Dual-path CNN with Max Gated block for text-based person re-identification

Text-based person re-identification (Re-id) is an important task in video surveillance, which consists of retrieving the corresponding person's image given a textual description from a large gallery of images. It is difficult to directly match visual contents with the textual descriptions due to the modality heterogeneity. On the one hand, the textual embedding are not discriminative enough, which originates from the high abstraction of the textual descriptions. One the other hand, Global average pooling (GAP) is commonly utilized to extract more general or smoothed features implicitly but ignores salient local features, which are more important for the cross-modal matching problem. With that in mind, a novel Dual-path CNN with Max Gated block (DCMG) is proposed to extract discriminative word embedding and make visual-textual association concern more on remarkable features of both modalities. The proposed framework is based on two deep residual CNNs jointly optimized with cross-modal projection matching (CMPM) loss and cross-modal projection classification (CMPC) loss to embed the two modalities into a joint feature space. First, the pre-trained language model, BERT, is combined with the convolutional neural network (CNN) to learn better word embedding in the text-to-image matching domain. Second, the global Max pooling (GMP) layer is applied to make the visual textual features focus more on the salient part. To further alleviate the noise of the maxed-pooled features, the gated block (GB) is proposed to produce an attention map that focuses on meaningful features of both modalities. Finally, extensive experiments are conducted on the benchmark dataset, CUHK-PEDES, in which our approach achieves the rank-1 score of 55.81% and outperforms the state-of-the-art method by 1.3%. We also evaluate our method on two generic retrieval datasets (Flickr30K, Oxford-102 Flowers) and obtain the competitive performance. Code is available at https://github.com/voriarty/Dual-path-CNN-with-Max-Gated-block-for-Text-Based-Person-Re-identification

Toward Improving Image Retrieval via Global Saliency Weighted Feature

For full description of images’ semantic information, image retrieval tasks are increasingly using deep convolution features trained by neural networks. However, to form a compact feature representation, the obtained convolutional features must be further aggregated in image retrieval. The quality of aggregation affects retrieval performance. In order to obtain better image descriptors for image retrieval, we propose two modules in our method. The first module is named generalized regional maximum activation of convolutions (GR-MAC), which pays more attention to global information at multiple scales. The second module is called saliency joint weighting, which uses nonparametric saliency weighting and channel weighting to focus feature maps more on the salient region without discarding overall information. Finally, we fuse the two modules to obtain more representative image feature descriptors that not only consider the global information of the feature map but also highlight the salient region. We conducted experiments on multiple widely used retrieval data sets such as roxford5k to verify the effectiveness of our method. The experimental results prove that our method is more accurate than the state-of-the-art methods.

Dataset search in biodiversity research: Do metadata in data repositories reflect scholarly information needs?

The increasing amount of publicly available research data provides the opportunity to link and integrate data in order to create and prove novel hypotheses, to repeat experiments or to compare recent data to data collected at a different time or place. However, recent studies have shown that retrieving relevant data for data reuse is a time-consuming task in daily research practice. In this study, we explore what hampers dataset retrieval in biodiversity research, a field that produces a large amount of heterogeneous data. In particular, we focus on scholarly search interests and metadata, the primary source of data in a dataset retrieval system. We show that existing metadata currently poorly reflect information needs and therefore are the biggest obstacle in retrieving relevant data. Our findings indicate that for data seekers in the biodiversity domain environments, materials and chemicals, species, biological and chemical processes, locations, data parameters and data types are important information categories. These interests are well covered in metadata elements of domain-specific standards. However, instead of utilizing these standards, large data repositories tend to use metadata standards with domain-independent metadata fields that cover search interests only to some extent. A second problem are arbitrary keywords utilized in descriptive fields such as title, description or subject. Keywords support scholars in a full text search only if the provided terms syntactically match or their semantic relationship to terms used in a user query is known.

Learning deep part-aware embedding for person retrieval

Person retrieval is an important vision task, aiming at matching the images of the same person under various camera views. The key challenge of person retrieval lies in the large intra-class variations among the person images. Therefore, how to learn discriminative feature representations becomes the core problem. In this paper, we propose a deep part-aware representation learning method for person retrieval. First, an improved triplet loss is introduced such that the global feature representations from the same identity are closely clustered. Meanwhile, a localization branch is proposed to automatically localize those discriminative person-wise parts or regions, only using identity labels in a weakly supervised manner. Via the learning simultaneously guided by the global branch and the localization branch, the proposed method can further improve the performance for person retrieval. Through an extensive set of ablation studies, we verify that the localization branch and the improved triplet loss each contributes to the performance boosts of the proposed method. Our model obtains superior (or comparable) performance compared to state-of-the-art methods for person retrieval on the four public person retrieval datasets. On the CUHK03-labeled dataset, for instance, the performance increases from 73.0% mAP and 77.9% rank-1 accuracy to 80.8% (+7.8%) mAP and 83.9% (+6.0%) rank-1 accuracy.

A novel benchmark dataset of color steel sheds for remote sensing image retrieval

Benchmark datasets play an important role in evaluating remote sensing image retrieval (RSIR) methods. The current datasets cover many scene categories, but omit an important scene of color steel sheds, which are widely distributed with a large number on the earth’s surface. Therefore, we propose a new benchmark dataset of color steel sheds (CSS) from Google map imagery for RSIR and share it open access in our V-RSIR system. The new dataset has rich intra-class and inter-class diversity, and is composed of blue, red and white color steel sheds with the total number of 2407 remote sensing images. We conduct evaluation experiments on the new dataset by using ten low/mid feature-based and ten deep learning feature-based methods. Experimental results indicate that the dataset is effective for evaluating RSIR methods and using the dataset can construct an effective retrieval model for color steel sheds. Besides, we have experimentally demonstrated that color constancy does affect retrieval performance on our CSS dataset. Furthermore, some experiments of merging the CSS dataset with the PatternNet, VGoogle and NWPU45 datasets are also conducted. Experimental results demonstrate that our dataset can be used as a complement to other retrieval datasets. Furthermore, these experimental results can be used as baseline for future applications on RSIR.

FhVLAD: Fine-grained quantization and encoding high-order descriptor statistics for scalable image retrieval

We are interested in the encoding of local descriptors of an image (e.g. SIFT) to design a compact representation vector and thereby address scalable image retrieval. We revisit the implicit design choices in the popular vector of locally aggregated descriptors (VLAD), which aggregates the residuals of descriptors to the codewords. VLAD’s use of a coarse codebook and first-order descriptor statistics in residual computation results in less discriminative residuals. To address this problem, we propose a division of codebook feature space using a novel fine-grained quantization strategy. After quantization, we embed the resulting residuals with high-order statistics of descriptor distribution. Experiments on three challenging image retrieval datasets (INRIA Holidays, UKBench, Oxford 5k) confirm the improved discriminative power of our novel encoding method called FhVLAD. We observe superior accuracy to baseline and competitive performance to state-of-the-art techniques with a limited increase in dimension.

Zero-shot Cross-modal Retrieval by Assembling AutoEncoder and Generative Adversarial Network

Conventional cross-modal retrieval models mainly assume the same scope of the classes for both the training set and the testing set. This assumption limits their extensibility on zero-shot cross-modal retrieval (ZS-CMR), where the testing set consists of unseen classes that are disjoint with seen classes in the training set. The ZS-CMR task is more challenging due to the heterogeneous distributions of different modalities and the semantic inconsistency between seen and unseen classes. A few of recently proposed approaches are inspired by zero-shot learning to estimate the distribution underlying multimodal data by generative models and make the knowledge transfer from seen classes to unseen classes by leveraging class embeddings. However, directly borrowing the idea from zero-shot learning (ZSL) is not fully adaptive to the retrieval task, since the core of the retrieval task is learning the common space. To address the above issues, we propose a novel approach named Assembling AutoEncoder and Generative Adversarial Network (AAEGAN), which combines the strength of AutoEncoder (AE) and Generative Adversarial Network (GAN), to jointly incorporate common latent space learning, knowledge transfer, and feature synthesis for ZS-CMR. Besides, instead of utilizing class embeddings as common space, the AAEGAN approach maps all multimodal data into a learned latent space with the distribution alignment via three coupled AEs. We empirically show the remarkable improvement for ZS-CMR task and establish the state-of-the-art or competitive performance on four image-text retrieval datasets.

Improving vehicle re‐identification using CNN latent spaces: Metrics comparison and track‐to‐track extension

This paper addresses the problem of vehicle re-identification using distance comparison of images in CNN latent spaces.Firstly, we study the impact of the distance metrics, comparing performances obtained with different metrics: the minimal Euclidean distance (MED), the minimal cosine distance (MCD), and the residue of the sparse coding reconstruction (RSCR). These metrics are applied using features extracted from five different CNN architectures, namely ResNet18, AlexNet, VGG16, InceptionV3 and DenseNet201. We use the specific vehicle re-identification dataset VeRi to fine-tune these CNNs and evaluate results. In overall, independently of the CNN used, MCD outperforms MED, commonly used in the literature. These results are confirmed on other vehicle retrieval datasets. Secondly, we extend the state-of-the-art image-to-track process (I2TP) to a track-to-track process (T2TP). The three distance metrics are extended to measure distance between tracks, enabling T2TP. We compared T2TP with I2TP using the same CNN models. Results show that T2TP outperforms I2TP for MCD and RSCR. T2TP combining DenseNet201 and MCD-based metrics exhibits the best performances, outperforming the state-of-the-art I2TP-based models. Finally, experiments highlight two main results: i) the impact of metric choice in vehicle re-identification, and ii) T2TP improves the performances compared to I2TP, especially when coupled with MCD-based metrics.

Predictive immunogenetic markers in COVID-19

Immunorelevant genes are among the most probable modulators of coronavirus disease 2019 (COVID-19) progression and prognosis. However, in the few months of the pandemic, data generated on host genetics has been scarce. The present study retrieved data sets of HLA-B alleles, KIR genes and functional single nucleotide polymorphisms (SNPs) in cytokines related to COVID-19 cytokine storm from two publicly available databases: Allele Frequency Net Database and Ensembl, and correlated these frequency data with Case Fatality Rate (CFR) and Daily Death Rates (DDR) across countries. Correlations of eight HLA-B alleles and polymorphisms in three cytokine genes (IL6, IL10, and IL12B) were observed and were mainly associated with DDR. Additionally, HLA-B correlations suggest that differences in allele affinities to SARS-CoV-2 peptides are also associated with DDR. These results may provide rationale for future host genetic marker surveys on COVID-19.

Automatic Detection of Genetics and Genomics of Eye Disease Using Deep Assimilation Learning Algorithm.

Diabetic retinopathy (DR) is one of the most prevalent genetic diseases in human and it is caused by damage to the blood vessels in the eye retina. If it is undetected and untreated at right time, it can lead to vision loss. There are many medical imaging and processing technologies to improve the diagnostic process of DR to overcome the lack of human experts. In the existing image processing methods, there are issues such as lack of noise removal, improper clustering segmentation and less classification accuracy. This can be accomplished by automatic diagnosis of DR using advanced image processing method. The cotton wool spot (CWS), hard exudates (HE) contains a common manifestation of many diseases in retina including DR and acquired immunodeficiency syndrome. In the present work, super iterative clustering algorithm (SICA) is proposed to identify the CWS, HE on retinal image. Feature-based medical image retrieval (FBMIR) datasets are utilized for this purpose. Noises present on the images and histogram-filtering technique is used to convert red, green, and blue (RGB) images into a perfect greyscale image without noise. After pre-processing, SICA is used to identify the CWS, HE detection on retinal images and eliminates unnecessary areas of interest. In the third stage, after detecting CWS and HE, various statistical features are extracted for further classification using deep assimilation learning algorithm (DALA). The performance of DALA technique is examined with various classification parameters like recall, precision, and F-measure. Finally, the false classification ratios are computed to compare the performance of the trained networks. The proposed method produces accurate detection of affected regions with an accuracy ratio of 98.5% and it is higher than the other conventional methods. This method may improve the accuracy of automatic detection and classification of eye diseases.

Duplicate Frame Video Forgery Detection Using Siamese-based RNN

Video and image data is the most important and widely used format of communication today. It is used as evidence and authenticated proof in different domains such as law enforcement, forensic studies, journalism, and others. With the increase of video applications and data, the problem of forgery in video and images has also originated. Although a lot of work has been done on image forgery, video forensic is still a challenging area. Videos are manipulated in many ways. Frame insertion, deletion, and frame duplication are a few of the major challenges. Moreover, in the perspective of duplicated frames, frame rate variation and loop detection are also key issues. Identification of forged duplication frames for large videos with variant frame rates in real-time is not applicable due to computational limitations, lack of generalization, and low-performance accuracy. This research has investigated the problem of frame duplication with varied frame rates using a deep learning approach. A novel deep learning framework consisting of Inflated 3D (I3D) and Siamese-based Recurrent Neural Network (RNN) is proposed to resolve the aforementioned issues. The first step in the proposed framework is to extract the features and convert videos into frames. I3D network receives an original and a forged video to detect frame-to-frame duplication. Then multiple frames are merged to create a sequence. This sequence is passed to Siamese-based RNN which is used for the sequence to sequence forgery detection in video. Media Forensic Challenge (MFC) is a relatively new dataset with various frame rates, and a huge volume of videos. MFC and Image Retrieval and Analysis Tool (VIRAT) datasets are used for training and validation of the proposed model. The accuracy of the proposed method with the VIRAT dataset is 86.6% and with the MFC dataset 93%. The comparative analysis with state-of-the-art approaches has shown the robustness of the proposed approach.

Local Semantic Structure Captured and Instance Discriminated by Unsupervised Hashing

Recently, unsupervised Hashing has attracted much attention in the machine learning and information retrieval communities, due to its low storage and high search efficiency. Most of existing unsupervised Hashing methods rely on the local semantic structure of the data as the guiding information, requiring to preserve such semantic structure in the Hamming space. Thus, how to precisely represent the local structure of the data and Hashing code s becomes the key point to success. This study proposes a novel Hashing method based on self-supervised learning. Specifically, it is proposed to utilize the contrast learning to acquire a compact and accurate feature representation for each sample, and then a semantic structure matrix can be constructed for representing the similarity between samples. Meanwhile, a new loss function is proposed to preserve the semantic information and improve the discriminative ability in the Hamming space, by the spirit of the instance discrimination method proposed recently. The proposed framework is end-to-end trainable. Extensive experiments on two large-scale image retrieval data sets show that the proposed method can significantly outperform current state-of-the-art methods.

A New Algorithm for Sketch‐Based Fashion Image Retrieval Based on Cross‐Domain Transformation

Due to the rise of e‐commerce platforms, online shopping has become a trend. However, the current mainstream retrieval methods are still limited to using text or exemplar images as input. For huge commodity databases, it remains a long‐standing unsolved problem for users to find the interested products quickly. Different from the traditional text‐based and exemplar‐based image retrieval techniques, sketch‐based image retrieval (SBIR) provides a more intuitive and natural way for users to specify their search need. Due to the large cross‐domain discrepancy between the free‐hand sketch and fashion images, retrieving fashion images by sketches is a significantly challenging task. In this work, we propose a new algorithm for sketch‐based fashion image retrieval based on cross‐domain transformation. In our approach, the sketch and photo are first transformed into the same domain. Then, the sketch domain similarity and the photo domain similarity are calculated, respectively, and fused to improve the retrieval accuracy of fashion images. Moreover, the existing fashion image datasets mostly contain photos only and rarely contain the sketch‐photo pairs. Thus, we contribute a fine‐grained sketch‐based fashion image retrieval dataset, which includes 36,074 sketch‐photo pairs. Specifically, when retrieving on our Fashion Image dataset, the accuracy of our model ranks the correct match at the top‐1 which is 96.6%, 92.1%, 91.0%, and 90.5% for clothes, pants, skirts, and shoes, respectively. Extensive experiments conducted on our dataset and two fine‐grained instance‐level datasets, i.e., QMUL‐shoes and QMUL‐chairs, show that our model has achieved a better performance than other existing methods.