Information Retrieval Research Articles

Privacy preserving large language models: ChatGPT case study based vision and framework

AbstractThe generative Artificial Intelligence (AI) tools based on Large Language Models (LLMs) use billions of parameters to extensively analyse large datasets and extract critical information such as context, specific details, identifying information, use this information in the training process, and generate responses for the requested queries. The extracted data also contain sensitive information, seriously threatening user privacy and reluctance to use such tools. This article proposes the conceptual model called PrivChatGPT, a privacy‐preserving model for LLMs consisting of two main components, that is, preserving user privacy during the data curation/pre‐processing and preserving private context and the private training process for large‐scale data. To demonstrate the applicability of PrivChatGPT, it is shown how a private mechanism could be integrated into the existing model for training LLMs to protect user privacy; specifically, differential privacy and private training using Reinforcement Learning (RL) were employed. The privacy level probabilities are associated with the document contents, including the private contextual information, and with metadata, which is used to evaluate the disclosure probability loss for an individual's private information. The privacy loss is measured and the measure of uncertainty or randomness is evaluated using entropy once differential privacy is applied. It recursively evaluates the level of privacy guarantees and the uncertainty of public databases and resources during each update when new information is added for training purposes. To critically evaluate the use of differential privacy for private LLMs, other mechanisms were hypothetically compared such as Blockchain, private information retrieval, randomisation, obfuscation, anonymisation, and the use of Tor for various performance measures such as the model performance and accuracy, computational complexity, privacy vs. utility, training latency, vulnerability to attacks, and resource consumption. It is concluded that differential privacy, randomisation, and obfuscation can impact the training models' utility and performance; conversely, using Tor, Blockchain, and Private Information Retrieval (PIR) may introduce additional computational complexity and high training latency. It is believed that the proposed model could be used as a benchmark for privacy‐preserving LLMs for generative AI tools.

Leveraging Deep Learning for Personalized Book Recommendations: A Big Data Algorithm Combining Capsule Networks and Attention Mechanisms

In the era of big data, personalized book recommendations have become crucial for enhancing user satisfaction and improving information retrieval efficiency. This study addresses the limitations of existing book recommendation algorithms by proposing a novel Hybrid Book Recommendation Algorithm Considering Different Preferences (HBRACDP). Our approach integrates Capsule Networks and Self-Attention Mechanisms to model both short-term and long-term user borrowing preferences. We construct separate models for these preferences and combine them using a controllable multi-interest network with label attention. Experimental results on the Goodreads dataset demonstrate the superiority of HBRACDP, achieving an accuracy of 0.984, recall of 0.987, and F1 score of 0.988 in ablation tests. In practical scenarios with 1000 students, HBRACDP significantly outperformed traditional algorithms, with a recommendation accuracy of 97.89% and an error rate of only 0.08%. This study provides new insights for developing more accurate and efficient big data recommendation systems in library services and beyond.

Fixation durations on familiar items are longer due to attenuation of exploration.

Previous studies have shown that fixations on familiar stimuli tend to be longer than on unfamiliar stimuli, theorized to be a result of retrieval of information from memory. We hypothesize that extended fixations are due to a lesser need to explore an already familiar stimulus. Participant's gaze was tracked as they tried to encode or retrieve a familiar face displayed either alone or alongside other unfamiliar faces. Regardless of the memory task (encoding\retrieval), longer fixation durations were observed when a single familiar face was presented alone, and not when presented among unfamiliar ones. Thus, fixations were not prolonged when it was possible to explore other, unfamiliar stimuli. We conclude that prolonged fixations on familiar stimuli reflect a lesser need to explore an already familiar percept. The results underscore how memory representations influence active sensing, yielding fresh insights into efficient deployment of attention resources. We conclude that fixation durations could be used in applied memory detection tests, preferably together with other measures and when the familiar stimulus is presented alone.

AI assistance in enterprise UX design workflows: enhancing design brief creation for designers

The study explores the impact of AI tools on the daily tasks of designers in corporate environments, with a focus on the creation and evaluation processes of design briefs. Given ChatGPT’s advanced natural language processing capabilities and its potential to meet the complex communication and analysis needs of design work, this tool was selected to investigate its application in designers’ workflows. Through expert interviews, experimental testing, and third-party expert evaluations, we collected and analyzed data to understand the impact of AI on work processes. The findings indicate that AI tools significantly enhance both operational experience and subjective perceptions across most tasks. Additionally, the study provides a visual comparison of the testing process through a user experience map, highlighting AI’s positive influence on work efficiency, information retrieval, verification, analysis, communication, and decision-making. However, challenges remain in ensuring information authenticity, protecting content copyright, and maintaining professional identity. The primary objective is to gain a comprehensive understanding of the current state of AI application in business contexts and its impact on designers’ roles. By analyzing real-world feedback, the research aims to identify the strengths and weaknesses of AI solutions in enterprises and offer practical recommendations. The study underscores the importance of integrating AI thinking into workflows and adopting a human-centric approach for the future development of corporate work environments.

Enhancement of the Performance of Large Language Models in Diabetes Education through Retrieval-Augmented Generation: Comparative Study.

Large language models (LLMs) demonstrated advanced performance in processing clinical information. However, commercially available LLMs lack specialized medical knowledge and remain susceptible to generating inaccurate information. Given the need for self-management in diabetes, patients commonly seek information online. We introduce the Retrieval-augmented Information System for Enhancement (RISE) framework and evaluate its performance in enhancing LLMs to provide accurate responses to diabetes-related inquiries. This study aimed to evaluate the potential of the RISE framework, an information retrieval and augmentation tool, to improve the LLM's performance to accurately and safely respond to diabetes-related inquiries. The RISE, an innovative retrieval augmentation framework, comprises 4 steps: rewriting query, information retrieval, summarization, and execution. Using a set of 43 common diabetes-related questions, we evaluated 3 base LLMs (GPT-4, Anthropic Claude 2, Google Bard) and their RISE-enhanced versions respectively. Assessments were conducted by clinicians for accuracy and comprehensiveness and by patients for understandability. The integration of RISE significantly improved the accuracy and comprehensiveness of responses from all 3 base LLMs. On average, the percentage of accurate responses increased by 12% (15/129) with RISE. Specifically, the rates of accurate responses increased by 7% (3/43) for GPT-4, 19% (8/43) for Claude 2, and 9% (4/43) for Google Bard. The framework also enhanced response comprehensiveness, with mean scores improving by 0.44 (SD 0.10). Understandability was also enhanced by 0.19 (SD 0.13) on average. Data collection was conducted from September 30, 2023 to February 5, 2024. The RISE significantly improves LLMs' performance in responding to diabetes-related inquiries, enhancing accuracy, comprehensiveness, and understandability. These improvements have crucial implications for RISE's future role in patient education and chronic illness self-management, which contributes to relieving medical resource pressures and raising public awareness of medical knowledge.

Designing a Tactile Document UI for 2D Refreshable Tactile Displays: Towards Accessible Document Layouts for Blind People

Understanding document layouts is vital for enhancing document exploration and information retrieval for sighted individuals. However, for blind and visually impaired people, it becomes challenging to have access to layout information using typical assistive technologies such as screen readers. In this paper, we examine the potential benefits of presenting documents on two-dimensional (2D) refreshable tactile displays. These displays enable the tactile perception of 2D data, offering the advantage of dynamic and interactive functionality. Despite their potential, the development of user interfaces (UIs) for such displays has not advanced significantly. Thus, we propose a design of an intelligent tactile user interface (TUI), incorporating touch and audio feedback to represent documents in a tactile format. Our exploratory study for evaluating this approach revealed satisfaction from participants with the experience of directly viewing documents in their true form, rather than relying on screen-reading interpretations. Additionally, participants offered recommendations for incorporating additional features and refining the approach in future iterations. To facilitate further research and development, we have made our dataset and models publicly available.

Knowledge Graph Modeling-Driven Large Language Model Operating System (LLM OS) for Task Automation in Process Engineering Problem-Solving

We present the Process Engineering Operations Assistant (PEOA), an AI-driven framework designed to solve complex problems in the chemical and process industries. The framework employs a modular architecture orchestrated by a meta-agent, which serves as the central coordinator, managing an action generator and instruction-tuned small-scale language models (expert models). The action generator decomposes complex problems into sub-tasks and identifies suitable expert models to execute each, delivering precise solutions for multi-step problem-solving. Key techniques include advanced knowledge modeling using property graphs for improved information retrieval, facilitating more accurate and contextually relevant solutions. Additionally, the framework utilizes a teacher-student transfer-learning approach with GPT-4 (Omni) to fine-tune the action generator and expert models for domain adaptation, alongside an iterative problem-solving mechanism with sophisticated error handling. Custom datasets were developed to evaluate the framework against leading proprietary language models on various engineering tasks. The results demonstrate the framework’s effectiveness in automating calculations, accelerating prototyping, and providing AI-augmented decision support for industrial processes, marking a significant advancement in process engineering capabilities.

Advancements in deep learning for visual object tracking

Abstract. Since contemporary information-retrieval systems rely heavily on the content of titles and abstracts to identify relevant articles in literature searches, great care should be taken in constructing both. This comprehensive review delves into the transformative impact of deep learning on the domain of visual object tracking. Since the inception of AlexNet in 2012, deep learning has revolutionized feature extraction, leading to significant advancements in tracking accuracy and robustness. The article explores the integration of deep learning with various tracking algorithms, including deep correlation filters, classification-based approaches, Siamese networks, gradient-based methods, and the innovative application of Transformer architectures. Moreover, the role of tracking datasets in fostering algorithm development and innovation is highlighted, with an emphasis on the expansion in scale, diversity, and annotation quality. Furthermore, the article also examines the multifaceted evaluation metrics for tracking algorithms, encompassing precision, robustness, efficiency, generalization, and real-time capabilities. Looking ahead, the review outlines future research directions, such as algorithm optimization for lightweight and accelerated performance, enhancing generalizability, leveraging multimodal data fusion, and refining Transformer models for improved temporal information processing. The challenges of long-term tracking and the growing importance of algorithm interpretability and transparency are also discussed. In summary, the article underscores the promising trajectory of deep learning in visual object tracking, with ongoing research poised to make tracking technologies smarter, more efficient, and robust, catering to a wide array of practical applications and environments.

Retrieve–Revise–Refine: A novel framework for retrieval of concise entailing legal article set

The retrieval of entailing legal article sets aims to identify a concise set of legal articles that holds an entailment relationship with a legal query or its negation. Unlike traditional information retrieval that focuses on relevance ranking, this task demands conciseness. However, prior research has inadequately addressed this need by employing traditional methods. To bridge this gap, we propose a three-stage Retrieve–Revise–Refine framework which explicitly addresses the need for conciseness by utilizing both small and large language models (LMs) in distinct yet complementary roles. Empirical evaluations on the COLIEE 2022 and 2023 datasets demonstrate that our framework significantly enhances performance, achieving absolute increases in the macro F2 score by 3.17% and 4.24% over previous state-of-the-art methods, respectively. Specifically, our Retrieve stage, employing various tailored fine-tuning strategies for small LMs, achieved a recall rate exceeding 0.90 in the top-5 results alone—ensuring comprehensive coverage of entailing articles. In the subsequent Revise stage, large LMs narrow this set, improving precision while sacrificing minimal coverage. The Refine stage further enhances precision by leveraging specialized insights from small LMs, resulting in a relative improvement of up to 19.15% in the number of concise article sets retrieved compared to previous methods. Our framework offers a promising direction for further research on specialized methods for retrieving concise sets of entailing legal articles, thereby more effectively meeting the task’s demands.

A Review of Hybrid Search Architectures: Integrating Keyword and Semantic Search to Optimize Query Relevance

Abstract—The advent of semantic search alongside the tra- ditional robustness of keyword search systems has established an intriguing dichotomy within the landscape of information retrieval. While keyword-based methods excel in terms of speci- ficity, semantic models offer contextual richness and intent aware- ness. However, each paradigm faces challenges in harnessing the strengths of the other. This paper investigates a hybrid search model that integrates both keyword and semantic search techniques, aiming to optimize relevance, interpretability, and user experience in complex information-seeking scenarios. We present an innovative hybrid architecture, conduct a comprehen- sive evaluation of its performance, and discuss its implications for information retrieval in specialized domains such as healthcare and education. Index Terms—Hybrid Search, Keyword Search, Semantic Search, Information Retrieval, Natural Language Processing, Search Architectures

Improvement of Optimal Human Resource Allocation Algorithm Based on Deep Latent Semantic Model

With the advances of technology, the banking industry faces the challenge of processing large-scale, heterogeneous human resource data. Traditional methods have difficulties in providing efficient and accurate analysis and prediction. This paper proposes an optimized human resource allocation algorithm based on deep latent Semantic Model (DLSM), which improves the accuracy and efficiency of data analysis by integrating deep neural network (DNN) and transfer learning technology. The experimental results show that the DLSM model performs well in text classification and information retrieval tasks, with the accuracy, accuracy, recall and F1 scores improving by 7.5%, 5.5%, 6.0% and 5.8%, respectively. This study confirms the excellent performance of DLSM model in HR data processing, provides an efficient and scientifically based solution for recruitment optimization in the banking industry, and enhances the competitiveness of enterprises.

EFL Students' Perceptions on the Integration of AI in Fostering Critical Thinking Skills

A developing discussion over the effects of Artificial Intelligence (AI) on students' critical thinking skills has emerged with their introduction into the classroom. Even if AI can provide several advantages, some worries depending too much on these technologies could impede the growth of critical thinking abilities. This research aims to analyze how students perceive the influence of AI on their critical thinking abilities. This research is descriptive qualitative research that involves collecting data from direct interviews. The respondents are 15 (fifteen) students from the English Literature study program, Letter Faculty, Sawerigading University Makassar. The research found that students perceive the following impact of AI use on their critical thinking abilities: 1) Hindering the development of critical thinking skills. 2) Assisting in the retrieval of information, data analysis, and the resolution of everyday issues. 3) Causing dependence on technology. 4) Helping to evaluate the information quickly; 5) Exploring alternative ideas; and 6) Improving reasoning ability. Overall, we need to balance the use of AI while prioritizing students' critical thinking. It is important to encourage students to use AI responsibly and judiciously, while also emphasizing the importance of critically evaluating the information provided by AI systems.

When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10-20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

Opportunities for retrieval and tool augmented large language models in scientific facilities

AbstractUpgrades to advanced scientific user facilities such as next-generation x-ray light sources, nanoscience centers, and neutron facilities are revolutionizing our understanding of materials across the spectrum of the physical sciences, from life sciences to microelectronics. However, these facility and instrument upgrades come with a significant increase in complexity. Driven by more exacting scientific needs, instruments and experiments become more intricate each year. This increased operational complexity makes it ever more challenging for domain scientists to design experiments that effectively leverage the capabilities of and operate on these advanced instruments. Large language models (LLMs) can perform complex information retrieval, assist in knowledge-intensive tasks across applications, and provide guidance on tool usage. Using x-ray light sources, leadership computing, and nanoscience centers as representative examples, we describe preliminary experiments with a Context-Aware Language Model for Science (CALMS) to assist scientists with instrument operations and complex experimentation. With the ability to retrieve relevant information from facility documentation, CALMS can answer simple questions on scientific capabilities and other operational procedures. With the ability to interface with software tools and experimental hardware, CALMS can conversationally operate scientific instruments. By making information more accessible and acting on user needs, LLMs could expand and diversify scientific facilities’ users and accelerate scientific output.

Biomedical Information Retrieval with Positive-Unlabeled Learning and Knowledge Graphs

The rapid growth of biomedical publications has presented significant challenges in the field of information retrieval. Most existing work focuses on document retrieval given explicit queries. However, in real applications such as curated biomedical database maintenance, explicit queries are missing. In this paper, we propose a two-step model for biomedical information retrieval in the case that only a small set of example documents is available without explicit queries. Initially, we extract keywords from the observed documents using large pre-trained language models and biomedical knowledge graphs. These keywords are then enriched with domain-specific entities. Information retrieval techniques can subsequently use the collected entities to rank the documents. Following this, we introduce an iterative Positive-Unlabeled learning method to classify all unlabeled documents. Experiments conducted on the PubMed dataset demonstrate that the proposed technique outperforms the state-of-the-art positive-unlabeled learning methods. The results underscore the effectiveness of integrating large language models and biomedical knowledge graphs in improving zero-shot information retrieval performance in the biomedical domain.

Online and Offline Evaluation in Search Clarification

The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to employ assessment approaches that encompass both real-time feedback from users (online evaluation) and the characteristics of clarification questions evaluated through human assessment (offline evaluation). However, the relationship between online and offline evaluations has been debated in information retrieval. This study aims to investigate how this discordance holds in search clarification. We use user engagement as ground truth and employ several offline labels to investigate to what extent the offline ranked lists of clarification resemble the ideal ranked lists based on online user engagement. Contrary to the current understanding that offline evaluations fall short of supporting online evaluations, we indicate that when identifying the most engaging clarification questions from the user’s perspective, online and offline evaluations correspond with each other. We show that the query length does not influence the relationship between online and offline evaluations, and reducing uncertainty in online evaluation strengthens this relationship. We illustrate that an engaging clarification needs to excel from multiple perspectives, and SERP quality and characteristics of the clarification are equally important. We also investigate if human labels can enhance the performance of Large Language Models (LLMs) and Learning-to-Rank (LTR) models in identifying the most engaging clarification questions from the user’s perspective by incorporating offline evaluations as input features. Our results indicate that LTR models do not perform better than individual offline labels. However, GPT, an LLM, emerges as the standout performer, surpassing all LTR models and offline labels.

How user language affects conflict fatality estimates in ChatGPT

OpenAI’s ChatGPT language model has gained popularity as a powerful tool for problem-solving and information retrieval. However, concerns arise about the reproduction of biases present in the language-specific training data. In this study, we address this issue in the context of the Israeli–Palestinian and Turkish–Kurdish conflicts. Using GPT-3.5, we employed an automated query procedure to inquire about casualties in specific airstrikes, in both Hebrew and Arabic for the former conflict and Turkish and Kurdish for the latter. Our analysis reveals that GPT-3.5 provides 34 ± 11% lower fatality estimates when queried in the language of the attacker than in the language of the targeted group. Evasive answers denying the existence of such attacks further increase the discrepancy. A simplified analysis on the current GPT-4 model shows the same trends. To explain the origin of the bias, we conducted a systematic media content analysis of Arabic news sources. The media analysis suggests that the large-language model fails to link specific attacks to the corresponding fatality numbers reported in the Arabic news. Due to its reliance on co-occurring words, the large-language model may provide death tolls from different attacks with greater news impact or cumulative death counts that are prevalent in the training data. Given that large-language models may shape information dissemination in the future, the language bias identified in our study has the potential to amplify existing biases along linguistic dyads and contribute to information bubbles.

The effect of gamified project-based learning with AIGC in information literacy education

ABSTRACT The emergence of artificial intelligence has necessitated a transformation in pedagogical methodologies. This study evaluated the integration of gamified project-based learning (GPBL) and artificial intelligence-generated content (AIGC) tools in enhancing information literacy education. The primary objectives were to explore the relationship among students’ learning experiences, information literacy, creativity, and engagement in an integrated learning environment. Using an explanatory sequential design with surveys and interviews, first-year university students (n = 36) from the School of Art and Design participated. Results indicated that students perceived moderately high levels of engagement(3.72) and creativity(4.11) during the GPBL activities, with positive emotions contributing significantly to motivation. Moreover, information literacy skills have substantially improved, such as in information retrieval, using AIGC tools, and understanding the ethical aspects of AI technology. A process model illustrated how positive emotions, motivation, creativity, engagement, and information literacy were linked in a GPBL-incorporating AI setting.

Influence and collaboration in library and information science research: A university perspective

Objective. The objective of this study was to examine the influence and collaboration patterns within library and information science (LIS) research, with a particular focus on the contributions of universities over time. Design/Methodology/Approach. This study used bibliometrics and LIS network analysis to examine 14,517 articles indexed in Scopus from 1954 to 2023. The Bibliometrix R-package was used to identify publication trends, influential institutions, authors, and collaboration networks in the LIS field. Findings. The key findings indicated that Wuhan University was the foremost institution in LIS research, with notable contributions from Chinese scholars who were identified as the most influential authors in the field. The keywords “information science” and “information retrieval” emerged as common thematic areas, reflecting core research interests. The content analysis also revealed that LIS research had robust connections with fields such as physics, computer science, and information technology, underscoring an interdisciplinary trend. Originality/Value. This study offered original insights into the evolving landscape of LIS research, identifying both leadership in academic output and potential gaps in current research coverage. It underscored the necessity for future research to expand the scope of its database and refine keyword selection for more comprehensive representation. The findings contributed to the strategic development of LIS research and the fostering of international collaboration.

Exploring fonts as retrieval cues in text-based learning

A growing body of research focuses on optimizing retrieval processes to help learners recall information from memory. The context-dependent memory effect suggests that information retrieval is more effective when the retrieval context matches the learning context. This phenomenon has been investigated in a variety of study materials and samples. However, there is a research gap in which this effect has not yet been investigated with font. Therefore, the font may act as a contextual retrieval cue, making it easier for learners to recall the presented information. In this context, the present experiment assumes that a match between the font used during learning (encoding phase) and that used during testing (decoding phase) improves retrieval processes and thus learning performance, while a mismatch has a negative effect. To test this assumption, 130 university students were randomly assigned to one condition of a two (font of learning texts: Haettenschweiler vs. Times New Roman) × two (font of learning questions: Haettenschweiler vs. Times New Roman) between-subjects design. Contrary to expectations, using the same font for encoding and decoding did not improve learning performance. Furthermore, learners did not report higher levels of mental effort when they had to work on learning questions presented in the same font as the learning texts. Although the font of the learning text is the most salient feature, it does not seem to serve as an effective retrieval cue in text-based learning.