Language Interface Research Articles

Zero-shot Video Grounding with Pseudo Query Lookup and Verification.

Video grounding, the process of identifying a specific moment in an untrimmed video based on a natural language query, has become a popular topic in video understanding. However, fully supervised learning approaches for video grounding that require large amounts of annotated data can be expensive and time-consuming. Recently, zero-shot video grounding (ZS-VG) methods that leverage pre-trained object detectors and language models to generate pseudo-supervision for training video grounding models have been developed. However, these approaches have limitations in recognizing diverse categories and capturing specific dynamics and interactions in the video context. To tackle these challenges, we introduce a novel two-stage ZS-VG framework called Lookup-and-Verification (LoVe), which treats the pseudo-query generation procedure as a video-to-concept retrieval problem. Our approach allows for the extraction of diverse concepts from an open-concept pool and employs a verification process to ensure the relevance of the retrieved concepts to the objects or events of interest in the video proposals. Comprehensive experimental results on the Charades-STA, ActivityNet-Captions, and DiDeMo datasets demonstrate the effectiveness of the LoVe framework.

ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language.

The use of natural language interfaces (NLIs) to create charts is becoming increasingly popular due to the intuitiveness of natural language interactions. One key challenge in this approach is to accurately capture user intents and transform them to proper chart specifications. This obstructs the wide use of NLI in chart generation, as users' natural language inputs are generally abstract (i.e., ambiguous or under-specified), without a clear specification of visual encodings. Recently, pre-trained large language models (LLMs) have exhibited superior performance in understanding and generating natural language, demonstrating great potential for downstream tasks. Inspired by this major trend, we propose ChartGPT, generating charts from abstract natural language inputs. However, LLMs are struggling to address complex logic problems. To enable the model to accurately specify the complex parameters and perform operations in chart generation, we decompose the generation process into a step-by-step reasoning pipeline, so that the model only needs to reason a single and specific sub-task during each run. Moreover, LLMs are pre-trained on general datasets, which might be biased for the task of chart generation. To provide adequate visualization knowledge, we create a dataset consisting of abstract utterances and charts and improve model performance through fine-tuning. We further design an interactive interface for ChartGPT that allows users to check and modify the intermediate outputs of each step. The effectiveness of the proposed system is evaluated through quantitative evaluations and a user study.

Analysis of the Possibilities of Carrying Out Attacks on the Functions of Transferring Control to Operating System Console Using Active Intelligence Methods

This paper researches the possibility of conducting attacks on the console control transfer functions using active intelligence methods. The research employs a comprehensive approach involving ports scanning, directories searching, parameters modifying, and credentials searching based on a user dictionary. Additionally, the study involves the development of a software tool designed to detect vulnerabilities in network nodes. The software developed within the framework of this study is delivered in the form of two modules, the first module contains the main program with the mechanisms implemented in it to indicate the possibility of attacks, with an emphasis on current web applications and services. Checking for the possibility of an attack for any network node is that first a list of requests is compiled, the purpose of which is to identify weaknesses of a web application running on the server, and a list of expected responses from the server is also compiled for these requests. After that request goes to the server, the program waits for responses from the server, and if the expected responses from the compiled list coincide with the actual ones, then this fact signals the possibility of an attack on the web application. The second module stores localization dictionaries responsible for the presence of two interface languages in the program – Russian and English. The result of this work is the MaxNetScanner2022 software, which identifies the possibility of an attack on the system control transfer functions using active intelligence tools.

Single-Frame Supervision for Spatio-Temporal Video Grounding.

Spatio-Temporal Video Grounding (STVG) aims at localizing the spatio-temporal tube of a specific object in an untrimmed video given a free-form natural language query. As the annotation of tubes is labor intensive, researchers are motivated to explore weakly supervised approaches in recent works, which usually results in significant performance degradation. To achieve a less expensive STVG method with acceptable accuracy, this work investigates the "single-frame supervision" paradigm that requires a single frame labeled with a bounding box within the temporal boundary of the fully supervised counterpart as the supervisory signal. Based on the characteristics of the STVG problem, we propose a Two-Stage Multiple Instance Learning (T-SMILE) method, which creates pseudo labels by expanding the annotated frame to its contextual frames, thereby establishing a fully-supervised problem to facilitate further model training. The innovations of the proposed method are three-folded, including 1) utilizing multiple instance learning to dynamically select instances in positive bags for the recognition of starting and ending timestamps, 2) learning highly discriminative query features by incorporating spatial prior constraints in cross-attention, and 3) designing a curriculum learning-based strategy that iterative assigns dynamic weights to spatial and temporal branches, thereby gradually adapting to the learning branch with larger difficulty. To facilitate future research on this task, we also contribute a large-scale benchmark containing 12,469 videos on complex scenes with single-frame annotation. The extensive experiments on two benchmarks demonstrate that T-SMILE significantly outperforms all weakly-supervised methods. Remarkably, it also performs better than some fully-supervised methods associated with much more annotation labor costs. The dataset and codes are available at https://github.com/qumengxue/T-SMILE.

Deriving inferences through natural language from structured datasets for asset lifecycle management

Industry 4.0 is expected to revolutionize the manufacturing world in terms of improved decision-making leading to increased value extraction from assets. Yet, since its inception, its adoption seems to have not scaled with time as expected. One of the bottlenecks can be due to the knowledge disparity between the Industry 4.0 technology developer and the domain expert/end user, which can lead to higher resources, cost, and time requirements for scalable adoption. Modern day Large Language Models (LLMs) have the ability to process large datasets and can serve as an intermediatory tool for domain end-users. Thus, in this paper, an Industrial-GPT tool is developed to translate natural language queries into meaningful inferences about asset-related data. The tool is evaluated using an exemplary dataset, representing a manufacturing industry with four production lines, multiple assets, readings and inferences from multiple sensors. The experimental results show that reasonable insight may be extracted from the dataset, using prompt engineering, while fine-tuning the model to adapt to business logic can help achieve better and faster inferences. Future research directions are proposed that can be addressed while developing Industrial-GPT-based tools and reducing the development cycle for Industry 4.0 technologies.

Part of speech-based semantic similarity for RDF Predicate Selection : A benchmarking approach

RDF predicate selection and RDF subject identification are two key steps for effectively processing the natural language queries by the semantic question answering systems. For subject identification, techniques such as Named Entity Recognition are used. For the relatively complex task of predicate selection, Natural Language Processing techniques can be utilized along with knowledge bases to select the semantically matching relationships. Semantic features can be extracted from queries utilizing phrase embeddings of the transformer models. In this paper firstly, a novel Predicate Selection approach on RDF Knowledge Bases has been proposed and implemented utilizing BERT-based phrase embeddings and custom logic for key part-of-speech token combinations. Secondly, the implemented prototype is benchmarked over combinations of key part of speech tokens and stop words for predicate selection accuracy over YAGO Date Facts, demonstrating the effectiveness of our approach. Benchmarking results also highlight the importance of verb stop words for the task of predicate selection.

Evaluation of a Voice-Enabled Autonomous Camera Control System for the da Vinci Surgical Robot

Robotic surgery involves significant task switching between tool control and camera control, which can be a source of distraction and error. This study evaluated the performance of a voice-enabled autonomous camera control system compared to a human-operated camera for the da Vinci surgical robot. Twenty subjects performed a series of tasks that required them to instruct the camera to move to specific locations to complete the tasks. The subjects performed the tasks (1) using an automated camera system that could be tailored based on keywords; and (2) directing a human camera operator using voice commands. The data were analyzed using task completion measures and the NASA Task Load Index (TLX) human performance metrics. The human-operated camera control method was able to outperform an automated algorithm in terms of task completion (6.96 vs. 7.71 correct insertions; p-value = 0.044). However, subjective feedback suggests that a voice-enabled autonomous camera control system is comparable to a human-operated camera control system. Based on the subjects’ feedback, thirteen out of the twenty subjects preferred the voice-enabled autonomous camera control system including the surgeon. This study is a step towards a more natural language interface for surgical robotics as these systems become better partners during surgery.

A Fuzzification-Based Processing of Natural Language Query Using Database

Abstract: The requirement for natural language interfaces to database has turn out to be more and more acute as increasingly more people get entry to data from internet browsers. But NLI (Natural Language Interface) is only usable if they map natural language inquiries to sql queries effectively Natural Language processing is turning into one of the maximum energetic areas in Human-pc interaction. It is a department of AI (artificial Intelligence) which incorporates in records retrieval, gadget translation and Language evaluation. The intention of NLP (Natural Language Processing) is to enable communique between human beings and computers without requiring to memorization of complicated commands and techniques. In other words NLP (natural Language Processing) is strategies that may make the pc apprehend the languages evidently utilized by humans. The main motive of Natural Language query Processing is for an English sentence to be interpreted by way of the pc and suitable action taken. Asking questions to databases in Natural language is a totally handy and easy approach of statistics access, in particular for casual customers who do not recognize complicated database question languages which includes square.

CLUE: Contrastive language-guided learning for referring video object segmentation

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics.

Semi-supervised inverted file index approach for approximate nearest neighbor search

This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc.

Generative AI in Management – Today and Tomorrow

Theoretical background: The rapid and exponential technological advancements have far-reaching impacts on management information systems, management practices, and human life. The promising outcomes in Artificial Intelligence and cutting-edge research on semantic networks and natural language processing have motivated the authors to envision the future of management technology. Purpose of the article: Our paper focuses on the new communication facilities and artificial intelligence models used to process management-type queries in natural language. Research methods: The article discusses recently developed technologies, proposed by Google and Microsoft, notably Google Bard and Bing integrated with ChatGPT-4. Both chatbots use Generative AI methods and large language models to understand domain-based queries and generate answers. Main findings: The practical and social implications of new models in management practice are discussed. To illustrate the qualities and weaknesses of the features of new technologies, four examples of management decision-making are discussed. The case studies also show differences between these two technologies. Finally, the paper concludes by summarizing the expectations and limitations of Generative AI applications in management. The paper is one of the first publications describing and demonstrating the idea of interfaces in natural language in business-oriented applications.

Survey of Code Search Based on Deep Learning

Code writing is repetitive and predictable, inspiring us to develop various code intelligence techniques. This survey focuses on code search, that is, to retrieve code that matches a given natural language query by effectively capturing the semantic similarity between the query and code. Deep learning, being able to extract complex semantics information, has achieved great success in this field. Recently, various deep learning methods, such as graph neural networks and pretraining models, have been applied to code search with significant progress. Deep learning is now the leading paradigm for code search. In this survey, we provide a comprehensive overview of deep learning-based code search. We review the existing deep learning-based code search framework that maps query/code to vectors and measures their similarity. Furthermore, we propose a new taxonomy to illustrate the state-of-the-art deep learning-based code search in a three-step process: query semantics modeling, code semantics modeling, and matching modeling, which involves the deep learning model training. Finally, we suggest potential avenues for future research in this promising field.

KAPE: k NN-based Performance Testing for Deep Code Search

Code search is a common yet important activity of software developers. An efficient code search model can largely facilitate the development process and improve the programming quality. Given the superb performance of learning the contextual representations, deep learning models, especially pre-trained language models, have been widely explored for the code search task. However, studies mainly focus on proposing new architectures for ever-better performance on designed test sets but ignore the performance on unseen test data where only natural language queries are available. The same problem in other domains, e.g., CV and NLP, is usually solved by test input selection that uses a subset of the unseen set to reduce the labeling effort. However, approaches from other domains are not directly applicable and still require labeling effort. In this article, we propose the k NN-b a sed p erformance t e sting ( KAPE ) to efficiently solve the problem without manually matching code snippets to test queries. The main idea is to use semantically similar training data to perform the evaluation. Extensive experiments on six programming language datasets, three state-of-the-art pre-trained models, and seven baseline methods demonstrate that KAPE can effectively assess the model performance (e.g., CodeBERT achieves MRR 0.5795 on JavaScript) with a slight difference (e.g., 0.0261).

DEVELOPMENT OF AN AI-POWERED EMOTIONALLY INTELLIGENT, INTERACTIVE AGENT PROTOTYPE TO SUPPORT INFORMAL CAREGIVERS

Abstract Long-term informal caregiving for older adults with dementia is exhausting and poses risks to caregivers’ health outcomes. Most informal caregivers are not well prepared for their new role, facing various challenges and receiving limited support. With the advent of artificial intelligence (AI) and information and communication technologies (ICTs), assistive technology is increasingly useful for supporting caregivers. However, limited research assesses the potential for AI to facilitate a privacy-preserving computation environment to provide proactive assistance to caregivers. This project aims to fill this gap by leveraging the tremendous potential of Multimodal Machine Learning and Large Language Models (LLMs) to develop a prototype of an AI-powered emotionally intelligent, interactive agent for informal caregivers. We introduce an effective context-driven instruction tuning technique that enables LLMs to finetune various caregiving-related tasks (tips generation, answering questions, elaborating a specific training module) with instructions. The project will also deliver a first-of-its-kind custom dataset augmented with manualized instructions for relevant caregiving tasks, which is used for finetuning several state-of-the-art LLM models. Our intelligent interactive agent prototype currently has the ability to evaluate caregivers’ unique affective states at query times to provide personalized assistance, addressing various types of caregivers’ situation-specific queries (e.g., managing agitation, wandering, or medications). The prototype is capable of understanding and responding to the user’s natural language queries and designing short, machine-driven persuasive conversations on a pre-defined set of topics. Our prototype has the potential to provide individualized and contextualized support for caregiving tasks which in turn may greatly reduce caregiver burden.

Design and Implementation of Chatbot using Python

Chatbots have become increasingly popular in recent years, with applications spanning customer support, virtual assistants, and more. This research paper explores the design and implementation of a chatbot using Python. We delve into various aspects of chatbot development, including natural language processing (NLP), dialogue management, and user interface integration. The paper discusses key Python libraries and tools for building a chatbot and presents a case study demonstrating the creation of a simple chatbot.

Traduire en se décentrant : exemples d’écueils rencontrés par des étudiants en français de l’économie et de la finance

This article focuses on the relationship between language and culture in the context of the translation of journalistic texts in economics and finance. We can assume that any text or discourse, whether literary or specialized, contains a number of culturemes, that is, elements at the interface of language and culture, typical of the given language, which pose translation difficulties. For the purposes of the article, translations made by students of Romance philology were analyzed and selected problems with the translation of culturemes, including the problem of adopting an appropriate deictic perspective, were highlighted. Another issue raised in the article is the hybrid nature of French media texts in economics and finance, which exhibit a large number of features typical of literary texts. The conclusion emphasizes the fact that students should be sensitized to those layers of text and discourse that are at the interface between language and culture. These layers must be properly identified, decoded and translated from the language of the source-culture to the language of the target culture.

Status of patent archives in Asian continent: a vivid picture from OpenDOAR

Purpose This study aims to highlight the many distinguishing characteristics of open-access repositories that archive “Patents” in the Directory of Open-Access Repositories (OpenDOAR) provided by Asian nations. Design/methodology/approach The OpenDOAR was chosen as a data collection tool that provides a quality-assured list of repositories indexed globally. The data was extracted on 28 March 2023. Findings The study found that only eight Asian countries contributed to open-access repositories on OpenDOAR, with China being the highest contributor. These countries mainly focused on institutional repositories, primarily using DSpace and English as the main language interface. Web 2.0 tools, especially RSS and Atom, were commonly used, along with some presence of social media platforms on the sites, although to a lesser extent. While many repositories followed the OAI-PMH protocol, a considerable portion did not adopt open-access policies. Originality/value To the best of the authors’ knowledge, this study is the first one that brings to light the different features of repositories archiving one of the important content types, i.e. “Patents” in the OpenDOAR by Asian countries.

Is your search query well-formed? A natural query understanding for patent prior art search

Recent advances in Deep Learning based prior art search has enabled the development of easy-to-use prior art search engines that accept natural language search queries and provide improved search performance. However, unlike conventional keyword-based techniques where the results are readily interpreted by the presence of queried keywords, Deep Learning based techniques act like a black box. As a result, it is difficult for users to articulate their information in order to obtain optimal results. In this paper, we share insights on query well-formedness from extensive experimentation with PQAI,11https://projectpq.ai/. an open source Deep Learning based prior art search engine. We study the effects of various query parameters such as grammar, specificity, and verbosity on the search results and show that ill-formed queries containing grammatical errors, non-essential content, and broad terminology adversely affect the relevance of search results. We also develop a number of Machine Learning models, viz. Grammatical Error Detection Model (GEDM), Query Specificity Model (QSM), and Query Verbosity Model (QVM), to identify and mitigate commonly encountered issues with ill-formed queries. The data, survey forms, and code relating to this work will be released to the community22https://github.com/Renuk9390/Query_formedness_PQAI.. Towards future breakthroughs, critical areas of query understanding in prior art search for advancing research are given in the end.

The power and potentials of Flexible Query Answering Systems: A critical and comprehensive analysis

The popularity of chatbots, such as ChatGPT, has brought research attention to question answering systems, capable to generate natural language answers to user’s natural language queries. However, also in other kinds of systems, flexibility of querying, including but also going beyond the use of natural language, is an important feature. With this consideration in mind the paper presents a critical and comprehensive analysis of recent developments, trends and challenges of Flexible Query Answering Systems (FQASs). Flexible query answering is a multidisciplinary research field that is not limited to question answering in natural language, but comprises other query forms and interaction modalities, which aim to provide powerful means and techniques for better reflecting human preferences and intentions to retrieve relevant information. It adopts methods at the crossroad of several disciplines among which Information Retrieval (IR), databases, knowledge based systems, knowledge and data engineering, Natural Language Processing (NLP) and the semantic web may be mentioned. The analysis principles are inspired by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) framework, characterized by a top-down process, starting with relevant keywords for the topic of interest to retrieve relevant articles from meta-sources And complementing these articles with other relevant articles from seed sources Identified by a bottom-up process. to mine the retrieved publication data a network analysis is performed Which allows to present in a synthetic way intrinsic topics of the selected publications. issues dealt with are related to query answering methods Both model-based and data-driven (the latter based on either machine learning or deep learning) And to their needs for explainability and fairness to deal with big data Notably by taking into account data veracity. conclusions point out trends and challenges to help better shaping the future of the FQAS field.

Dative Doubling in Non-Mandatory Contexts in European Spanish

Clitic doubling (CD) is the co-appearance in the same sentence of the clitic and a correlative syntagma in the canonical position of the object. Apart from obligatory contexts, CD of the indirect object (IO) is found with variable frequency in Romance languages and even in different varieties of the same language, most likely because it is a phenomenon of internal/external language interface. The objective of this work is to determine the frequency of CD in non-obligatory contexts of recipient and location IO in peninsular Spanish, and to analyse its features according to the referential hierarchy used for the diachronic evolution of the phenomenon. For this purpose, we extracted data from two open access corpora of interviews (COREC and PRESEEA) from different regions that are (or are not) areas of historical contact with other languages. The results show a significant extension of doubling in contexts where this is optional and the neutralisation of features that previously predicted CD of IOs. Nevertheless, there are geographical differences in peninsular Spanish in terms of frequency, definiteness, specificity, the influence of the cliticization of the direct object, and the accessibility of the IO referents in the minds of the speakers.