Retrieval Tasks Research Articles

Whole slide image representation in bone marrow cytology

One of the goals of AI-based computational pathology is to generate compact representations of whole slide images (WSIs) that capture the essential information needed for diagnosis. While such approaches have been applied to histopathology, few applications have been reported in cytology. Bone marrow aspirate cytology is the basis for key clinical decisions in hematology. However, visual inspection of aspirate specimens is a tedious and complex process subject to variation in interpretation, and hematopathology expertise is scarce. The ability to generate a compact representation of an aspirate specimen may form the basis for clinical decision-support tools in hematology. In this study, we leverage our previously published end-to-end AI-based system for counting and classifying cells from bone marrow aspirate WSIs, which enables the direct use of individual cells as inputs rather than WSI patches. We then construct bags of individual cell features from each WSI, and apply multiple instance learning to extract their vector representations. To evaluate the quality of our representations, we conducted WSI retrieval and classification tasks. Our results show that we achieved a mAP@10 of 0.58 ±0.02 in WSI-level image retrieval, surpassing the random-retrieval baseline of 0.39 ±0.1. Furthermore, we predicted five diagnostic labels for individual aspirate WSIs with a weighted-average F1 score of 0.57 ±0.03 using a k-nearest-neighbors (k-NN) model, outperforming guessing using empirical class prior probabilities (0.26 ±0.02). We present the first example of exploring trainable mechanisms to generate compact, slide-level representations in bone marrow cytology with deep learning. This method has the potential to summarize complex semantic information in WSIs toward improved diagnostics in hematology, and may eventually support AI-assisted computational pathology approaches.

Towards Human–Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework

The multimodality nature of web data has necessitated complex multimodal information retrieval for a wide range of web applications. Deep neural networks (DNNs) have been widely employed to extract semantic features from raw samples to improve retrieval accuracy. In addition, hashing is widely used to improve computational and storage efficiency. As such, deep hashing frameworks have been applied for multimodal retrieval tasks. However, there is still a great recognitive gap between primate brain structure-inspired DNNs and humans. On computer vision tasks, well-crafted DNN models can be easily defeated by invisible small attacks, and this phenomenon indicates a large recognition gap between DNN models and humans. Recently, adversarial defense methods have been shown to improve the human–machine recognition alignment in several classification tasks. However, the robustness problem on the retrieval tasks, especially on the deep hashing-based multimodal retrieval models, is still not well studied. Therefore, in this article, we present an adversarially robust training mechanism to improve model robustness for the purpose of human–machine recognition alignment on retrieval tasks. Through extensive experimental results on several social multimodal retrieval benchmarks, we show that the robust training hashing framework proposed can mitigate the recognition gap on retrieval tasks. Our study highlights the necessity of robustness enhancement on deep hashing models.

Data-driven Morphological Feature Perception of Single Neuron with Graph Neural Network.

Clarifying the morphological characteristics of neurons can promote the understanding of brain function. However, traditional morphometrics fail to capture the modeling of each point in reconstructed neurons, leading to limited ability to distinguish massive nerve fibers and restricted application scenarios. To address these challenges, we propose MorphoGNN, a single neuron morphological embedding based on a graph neural network in this study. MorphoGNN learns the point-level structure information of reconstructed nerve fibers by considering their nearest neighbors on each hidden layer. This enables MorphoGNN to capture the lower-dimensional representation of a single neuron through an end-to-end model. In order to meet the requirements of various tasks, both supervised and self-supervised training strategies are designed to learn the characteristics that fit artificial semantics or the morphological patterns of neurons, respectively. We quantitatively compare our embeddings with other features in neuron classification and retrieval tasks and demonstrate cutting-edge performance. Additionally, we introduce our embeddings to the task of reconstruction quality classification and neuron clustering, where they can help detect reconstruction errors and obtain similar subtyping results to existing work. Furthermore, our method can be handily combined with other modal features, such as microscopic image features and traditional morphometrics. Ablation and robustness tests are also conducted to analyze the impact of several network components and low-quality reconstructed neurons on the performance of our method. The code is available at https://github.com/fun0515/MorphoGNN.

Free association in a neural network.

Free association among words is a fundamental and ubiquitous memory task. Although distributed semantics (DS) models can predict the association between pairs of words, and semantic network (SN) models can describe transition probabilities in free association data, there have been few attempts to apply established cognitive process models of memory search to free association data. Thus, researchers are currently unable to explain the dynamics of free association using memory mechanisms known to be at play in other retrieval tasks, such as free recall from lists. We address this issue using a popular neural network model of free recall, the context maintenance and retrieval (CMR) model, which we fit using stochastic gradient descent on a large data set of free association norms. Special cases of CMR mimic existing DS and SN models of free association, and we find that CMR outperforms these models on out-of-sample free association data. We also show that training CMR on free association data generates improved predictions for free recall from lists, demonstrating the value of free association for the study of many different types of memory phenomena. Overall, our analysis provides a new account of the dynamics of free association, predicts free association with increased accuracy, integrates theories of free association with established models of memory, and shows how large data sets and neural network training methods can be used to model complex cognitive processes that operate over thousands of representations. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Reading Comprehension in Information Retrieval (RCIR) for Personalized Results

ABSTRACTRecent research on personalized retrieval technology has been actively conducted to meet the needs of users for seeking adequate information. To refine the retrieval, researchers are considering user behavior patterns in a variety of ways. In this study, we use eye‐tracking metadata to predict users' levels of comprehension as textual evidence for IR processes. Furthermore, we incorporated eye‐tracking metadata on the Automated Readability Index (ARI), a readability assessment tool of an English text. Our research is largely divided into two tasks: i) comprehension evaluation task (CET) and ii) comprehension‐based retrieval task (CRT). In the CET task, for predicting the comprehension level, we applied various regression models. Among them, the Voting regressor demonstrated the highest performance with a Spearman's 𝜌 of 0.68. In the CRT task, we incorporated the level of comprehension predicted in the CET task and ARI score into the ranking results. We derived a sentenceBERT to find the relevant text for a query and the Normalized Discounted Cumulative Gain (nDCG) for evaluating the CRT task. The nDCG score for Comprehension Level only and that with ARI together were 0.65 and 0.78, respectively. Thus, applying ARI resulted in a higher nDCG score compared to comprehension level only.

Dynamic causal modeling of reorganization of memory and language networks in temporal lobe epilepsy.

To evaluate the alterations of language and memory functions using dynamic causal modeling, in order to identify the epileptogenic hemisphere in temporal lobe epilepsy (TLE). Twenty-two patients with left TLE and 13 patients with right TLE underwent functional magnetic resonance imaging (fMRI) during four memory and four language mapping tasks. Dynamic causal modeling (DCM) was employed on fMRI data to examine effective directional connectivity in memory and language networks and the alterations in people with TLE compared to healthy individuals. DCM analysis suggested that TLE can influence the memory network more widely compared to the language network. For memory mapping, it demonstrated overall hyperconnectivity from the left hemisphere to the other cranial regions in the picture encoding, and from the right hemisphere to the other cranial regions in the word encoding tasks. On the contrary, overall hypoconnectivity was seen from the brain hemisphere contralateral to the seizure onset in the retrieval tasks. DCM analysis further manifested hypoconnectivity between the brain's hemispheres in the language network in patients with TLE compared to controls. The CANTAB® neuropsychological test revealed a negative correlation for the left TLE and a positive correlation for the right TLE cohorts for the connections extracted by DCM that were significantly different between the left and right TLE cohorts. In this study, dynamic causal modeling evidenced the reorganization of language and memory networks in TLE that can be used for a better understanding of the effects of TLE on the brain's cognitive functions.

A practical investigation of cultivating students’ art innovation ability based on word vector model

Abstract In this paper, a multi-branch deep network architecture (JESF-MBF) based on joint embedded semantic features is proposed. In order to construct the structural consistency between visual image space and label semantic space, as well as to ensure the correspondence between sketch or image data and labels, the JESF-MBF model uses cross-entropy-based multivariate classification loss and cosine distance-based semantic similarity loss for model optimization. The optimized model improves the ability of label semantic information representation and image visual information representation, alleviates the discrepancy of data domain distribution between sketch and image, and constructs the mapping relationship between visual embedding space and semantic embedding space, so the sketch-based zero-sample image retrieval task is realized based on the intermediate medium. In the comparative model analysis, the average evaluation index of JESF-MBF is 0.412, better than that of Doodle2search at 0.344. In terms of the teaching effectiveness of the teaching platform constructed in this paper, the average time spent for the experimental class to complete each subject work is 9.4 class periods, while the average time required for the experimental class is 19.7 class periods. The results show that the art teaching platform based on the JESF-MBF model is very effective in terms of performance as well as improving the art level.

Multimodal video retrieval with CLIP: a user study

Recent machine learning advances demonstrate the effectiveness of zero-shot models trained on large amounts of data collected from the internet. Among these, CLIP (Contrastive Language-Image Pre-training) has been introduced as a multimodal model with high accuracy on a number of different tasks and domains. However, the unconstrained nature of the model begs the question whether it can be deployed in open-domain real-word applications effectively in front of non-technical users. In this paper, we evaluate whether CLIP can be used for multimodal video retrieval in a real-world environment. For this purpose, we implemented impa, an efficient shot-based retrieval system powered by CLIP. We additionally implemented advanced query functionality in a unified graphical user interface to facilitate an intuitive and efficient usage of CLIP for video retrieval tasks. Finally, we empirically evaluated our retrieval system by performing a user study with video editing professionals and journalists working in the TV news media industry. After having the participants solve open-domain video retrieval tasks, we collected data via questionnaires, interviews, and UI interaction logs. Our evaluation focused on the perceived intuitiveness of retrieval using natural language, retrieval accuracy, and how users interacted with the system’s UI. We found that our advanced features yield higher task accuracy, user ratings, and more efficient queries. Overall, our results show the importance of designing intuitive and efficient user interfaces to be able to deploy large models such as CLIP effectively in real-world scenarios.

Toward Best Practices for Training Multilingual Dense Retrieval Models

Dense retrieval models using a transformer-based bi-encoder architecture have emerged as an active area of research. In this article, we focus on the task of monolingual retrieval in a variety of typologically diverse languages using such an architecture. Although recent work with multilingual transformers demonstrates that they exhibit strong cross-lingual generalization capabilities, there remain many open research questions, which we tackle here. Our study is organized as a “best practices” guide for training multilingual dense retrieval models, broken down into three main scenarios: when a multilingual transformer is available, but training data in the form of relevance judgments are not available in the language and domain of interest (“have model, no data”); when both models and training data are available (“have model and data”); and when training data are available but not models (“have data, no model”). In considering these scenarios, we gain a better understanding of the role of multi-stage fine-tuning, the strength of cross-lingual transfer under various conditions, the usefulness of out-of-language data, and the advantages of multilingual vs. monolingual transformers. Our recommendations offer a guide for practitioners building search applications, particularly for low-resource languages, and while our work leaves open a number of research questions, we provide a solid foundation for future work.

Defining Semantically Close Words of Kazakh Language with Distributed System Apache Spark

This work focuses on determining semantically close words and using semantic similarity in general in order to improve performance in information retrieval tasks. The semantic similarity of words is an important task with many applications from information retrieval to spell checking or even document clustering and classification. Although, in languages with rich linguistic resources, the methods and tools for this task are well established, some languages do not have such tools. The first step in our experiment is to represent the words in a collection in a vector form and then define the semantic similarity of the terms using a vector similarity method. In order to tame the complexity of the task, which relies on the number of word (and, consequently, of the vector) pairs that have to be combined in order to define the semantically closest word pairs, A distributed method that runs on Apache Spark is designed to reduce the calculation time by running comparison tasks in parallel. Three alternative implementations are proposed and tested using a list of target words and seeking the most semantically similar words from a lexicon for each one of them. In a second step, we employ pre-trained multilingual sentence transformers to capture the content semantics at a sentence level and a vector-based semantic index to accelerate the searches. The code is written in MapReduce, and the experiments and results show that the proposed methods can provide an interesting solution for finding similar words or texts in the Kazakh language.

Multi-Scale Feature Fusion Based on PVTv2 for Deep Hash Remote Sensing Image Retrieval

For high-resolution remote sensing image retrieval tasks, single-scale features cannot fully express the complexity of the image information. Due to the large volume of remote sensing images, retrieval requires extensive memory and time. Hence, the problem of how to organically fuse multi-scale features and enhance retrieval efficiency is yet to be resolved. We propose an end-to-end deep hash remote sensing image retrieval model (PVTA_MSF) by fusing multi-scale features based on the Pyramid Vision Transformer network (PVTv2). We construct the multi-scale feature fusion module (MSF) by using a global attention mechanism and a multi-head self-attention mechanism to reduce background interference and enhance the representation capability of image features. Deformable convolution is introduced to address the challenge posed by varying target orientations. Moreover, an intra-class similarity (ICS) loss is proposed to enhance the discriminative capability of the hash feature by minimizing the distance among images of the same category. The experimental results show that, compared with other state-of-the-art methods, the proposed hash feature could yield an excellent representation of remote sensing images and improve remote sensing image retrieval accuracy. The proposed hash feature can gain an increase of 4.2% and 1.6% in terms of mAP on the UC Merced and NWPU-RESISC45 datasets, respectively, in comparison with other methods.

How Arabic diglossia affects the performances on phonological working memory tasks: Research evidence among first graders

The effect of bilingualism on verbal learning and memory was explored in different studies. Different researchers assume that the Arabic diglossia, represents a case of bilingualism in the lingual context. Hence, the current study aimed to investigate the impact of diglossia in Arabic on the phonological working memory among beginner readers. Forty-one Arabic first graders (M = 7.13, SD = .73) were administered three tasks of phonological working memory in two versions (i.e., spoken and standard language); Two tasks were designed to test verbal retrieval and one task was designed to test remembering of instructions. The participants showed significant diglossic differences between spoken and standard stimuli in verbal retrieval tasks while no such significant differences appeared in remembering of instructions’ task, especially, when the processing demands increased. In addition, the findings may shed light on the importance of developing research tools and tasks with a higher level of sensitivity in order to examine the diglossic effect on memory functions in general and verbal working memory in particular. The results were discussed considering the impact of the Arabic diglossia on cognitive and memory processing skills.

“Do this instead” – Robots that Adequately Respond to Corrected Instructions

Natural language instructions are effective at tasking autonomous robots and for teaching them new knowledge quickly. Yet, human instructors are not perfect and are likely to make mistakes at times, and will correct themselves when they notice errors in their own instructions. In this paper, we introduce a complete system for robot behaviors to handle such corrections, during both task instruction and action execution. We then demonstrate its operation in an integrated cognitive robotic architecture through spoken language in two tasks: a navigation and retrieval task and a meal assembly task. Verbal corrections occur before, during, and after verbally taught sequences of tasks, demonstrating that the proposed methods enable fast corrections not only of the semantics generated from the instructions, but also of overt robot behavior in a manner shown to be reasonable when compared to human behavior and expectations.

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

In recent years, there has been a growing interest in remote sensing image–text cross-modal retrieval due to the rapid development of space information technology and the significant increase in the volume of remote sensing image data. Remote sensing images have unique characteristics that make the cross-modal retrieval task challenging. Firstly, the semantics of remote sensing images are fine-grained, meaning they can be divided into multiple basic units of semantic expression. Different combinations of basic units of semantic expression can generate diverse text descriptions. Additionally, these images exhibit variations in resolution, color, and perspective. To address these challenges, this paper proposes a multi-task guided fusion encoder (MTGFE) based on the multimodal fusion encoding method, the progressiveness of which has been proved in the cross-modal retrieval of natural images. By jointly training the model with three tasks: image–text matching (ITM), masked language modeling (MLM), and the newly introduced multi-view joint representations contrast (MVJRC), we enhance its capability to capture fine-grained correlations between remote sensing images and texts. Specifically, the MVJRC task is designed to improve the model’s consistency in joint representation expression and fine-grained correlation, particularly for remote sensing images with significant differences in resolution, color, and angle. Furthermore, to address the computational complexity associated with large-scale fusion models and improve retrieval efficiency, this paper proposes a retrieval filtering method, which achieves higher retrieval efficiency while minimizing accuracy loss. Extensive experiments were conducted on four public datasets to evaluate the proposed method, and the results validate its effectiveness.

Lexical Access Restrictions after the Age of 80.

During the fourth age (80+ years), cognitive difficulties increase. Although language seems to resist the advancement of age, an older person without pathological developments in cognition may exhibit deficits in lexical access. This study examines the restrictions on lexical access in people aged 80 and older in word recognition and retrieval modalities through four lexical tasks. The effect of aging on response time and accuracy was measured using recognition (lexical decision/naming/priming) and retrieval (picture naming) tasks. A fourth age group (>80) and two third age groups (60-69/70-79) were compared according to lexical access modality and type of task employed through linear regression models. People aged 80 and older exhibit a strong lexical access constraint, as they are slower and less accurate in recognizing and retrieving words than both third age groups. These restrictions are more profound for the word retrieval modality, especially in the picture naming task. Impaired fluid intelligence and internode transmission deficits during advanced aging could further reduce the ability to recognize and/or retrieve words, having an impact on access speed and accuracy. Furthermore, the idea that crystallized intelligence could strengthen the accuracy of lexical access during aging is supported, specifically in word recognition modality.

Self-supervised Multi-view Learning via Auto-encoding 3D Transformations

3D object representation learning is a fundamental challenge in computer vision to infer about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based methods have performed best so far. However, feature learning of multiple views in existing methods is mostly performed in a supervised fashion, which often requires a large amount of data labels with high costs. In contrast, self-supervised learning aims to learn multi-view feature representations without involving labeled data. To this end, we propose a novel self-supervised framework to learn Multi-View Transformation Equivariant Representations (MV-TER), exploring the equivariant transformations of a 3D object and its projected multiple views that we derive. Specifically, we perform a 3D transformation on a 3D object and obtain multiple views before and after the transformation via projection. Then, we train a representation encoding module to capture the intrinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after the transformation. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks and show the generalization to real-world datasets. The code is available at https://github.com/gyshgx868/mvter .

A self-supervised language model selection strategy for biomedical question answering

Large neural-based Pre-trained Language Models (PLM) have recently gained much attention due to their noteworthy performance in many downstream Information Retrieval (IR) and Natural Language Processing (NLP) tasks. PLMs can be categorized as either general-purpose, which are trained on resources such as large-scale Web corpora, and domain-specific which are trained on in-domain or mixed-domain corpora. While domain-specific PLMs have shown promising performance on domain-specific tasks, they are significantly more computationally expensive compared to general-purpose PLMs as they have to be either retrained or trained from scratch. The objective of our work in this paper is to explore whether it would be possible to leverage general-purpose PLMs to show competitive performance to domain-specific PLMs without the need for expensive retraining of the PLMs for domain-specific tasks. By focusing specifically on the recent BioASQ Biomedical Question Answering task, we show how different general-purpose PLMs show synergistic behaviour in terms of performance, which can lead to overall notable performance improvement when used in tandem with each other. More concretely, given a set of general-purpose PLMs, we propose a self-supervised method for training a classifier that systematically selects the PLM that is most likely to answer the question correctly on a per-input basis. We show that through such a selection strategy, the performance of general-purpose PLMs can become competitive with domain-specific PLMs while remaining computationally light since there is no need to retrain the large language model itself. We run experiments on the BioASQ dataset, which is a large-scale biomedical question-answering benchmark. We show that utilizing our proposed selection strategy can show statistically significant performance improvements on general-purpose language models with an average of 16.7% when using only lighter models such as DistilBERT and DistilRoBERTa, as well as 14.2% improvement when using relatively larger models such as BERT and RoBERTa and so, their performance become competitive with domain-specific large language models such as PubMedBERT.

Robust online hashing with label semantic enhancement for cross-modal retrieval

Online hashing technology has attracted extensive attention owing to its effectiveness and efficiency in processing large-scale streaming data. However, there are still some limitations: (1) In practical applications, the observed labels of multimedia data are obtained through manual annotation, which may inevitably introduce some noises into labels. This may lead to retrieval performance degradation when the noisy labels are directly applied to retrieval tasks. (2) The potential semantic correlation of multi-labels cannot be fully explored. To overcome these limitations, in this paper, we propose robust online hashing with label semantic enhancement (ROHLSE). Specifically, ROHLSE seeks to recover the clean labels from the provided noisy labels by imposing low-rank and sparse constraints. Meanwhile, it employs the representation of samples in the feature space to predict the labels via the dependency between sample instances and labels. To efficiently handle streaming data, ROHLSE preserves the similarity between new data, and establishes the semantic relationships between new and old data through chunk similarity, simultaneously. Furthermore, ROHLSE can fully utilize the semantic correlations between multiple labels of each instance. Extensive experiments are conducted on three benchmark datasets to demonstrate the superiority of the proposed ROHLSE approach.

Describing and understanding the time course of the property listing task.

To study linguistically coded concepts, researchers often resort to the Property Listing Task (PLT). In a PLT, participants are asked to list properties that describe a concept (e.g., for DOG, subjects may list "is a pet", "has four legs", etc.). When PLT data is collected for many concepts, researchers obtain Conceptual Properties Norms (CPNs), which are used to study semantic content and as a source of control variables. Though the PLT and CPNs are widely used across psychology, only recently a model that describes the listing course of a PLT has been developed and validated. That original model describes the listing course using order of production of properties. Here we go a step beyond and validate the model using response times (RT), i.e., the time from cue onset to property listing. Our results show that RT data exhibits the same regularities observed in the previous model, but now we can also analyze the time course, i.e., dynamics of the PLT. As such, the RT validated model may be applied to study several similar memory retrieval tasks, such as the Free Listing Task, Verbal Fluidity Task, and to research related cognitive processes. To illustrate those kinds of analyses, we present a brief example of the difference in PLT's dynamics between listing properties for abstract versus concrete concepts, which shows that the model may be fruitfully applied to study concepts.

A Purely Entity-Based Semantic Search Approach for Document Retrieval

Over the past decade, knowledge bases (KB) have been increasingly utilized to complete and enrich the representation of queries and documents in order to improve the document retrieval task. Although many approaches have used KB for such purposes, the problem of how to effectively leverage entity-based representation still needs to be resolved. This paper proposes a Purely Entity-based Semantic Search Approach for Information Retrieval (PESS4IR) as a novel solution. The approach includes (i) its own entity linking method and (ii) an inverted indexing method, and for document retrieval and ranking, (iii) an appropriate ranking method is designed to take advantage of all the strengths of the approach. We report the findings on the performance of our approach, which is tested by queries annotated by two known entity linking tools, REL and DBpedia-Spotlight. The experiments are performed on the standard TREC 2004 Robust and MSMARCO collections. By using the REL method on the Robust collection, for the queries whose terms are all annotated and whose average annotation scores are greater than or equal to 0.75, our approach achieves the maximum nDCG@5 score (1.00). Also, it is shown that using PESS4IR alongside another document retrieval method would improve performance, unless that method alone achieves the maximum nDCG@5 score for those highly annotated queries.