Language Interface Research Articles

Rapid: Zero-shot Domain Adaptation for Code Search with Pre-trained Models

Code search, which refers to the process of identifying the most relevant code snippets for a given natural language query, plays a crucial role in software maintenance. However, current approaches heavily rely on labeled data for training, which results in performance decreases when confronted with cross-domain scenarios including domain-specific or project-specific situations. This decline can be attributed to their limited ability to effectively capture the semantics associated with such scenarios. To tackle the aforementioned problem, we propose a ze R o-shot dom A in ada P tion with pre-tra I ned mo D els framework for code search named RAPID. The framework first generates synthetic data by pseudo labeling, then trains the CodeBERT with sampled synthetic data. To avoid the influence of noisy synthetic data and enhance the model performance, we propose a mixture sampling strategy to obtain hard negative samples during training. Specifically, the mixture sampling strategy considers both relevancy and diversity to select the data that are hard to be distinguished by the models. To validate the effectiveness of our approach in zero-shot settings, we conduct extensive experiments and find that RAPID outperforms the CoCoSoDa and UniXcoder model by an average of 15.7% and 10%, respectively, as measured by the MRR metric. When trained on full data, our approach results in an average improvement of 7.5% under the MRR metric using CodeBERT. We observe that as the model’s performance in zero-shot tasks improves, the impact of hard negatives diminishes. Our observation also indicates that fine-tuning CodeT5 for generating pseudo labels can enhance the performance of the code search model, and using only 100-shot samples can yield comparable results to the supervised baseline. Furthermore, we evaluate the effectiveness of RAPID in real-world code search tasks in three GitHub projects through both human and automated assessments. Our findings reveal RAPID exhibits superior performance, e.g., an average improvement of 18% under the MRR metric over the top-performing model.

HFD: Hierarchical feature decoupling for SQL generation from text

Text-to-SQL, a computational linguistics task, seeks to facilitate the conversion of natural language queries into SQL queries. Recent methodologies have leveraged the concept of slot-filling in conjunction with predetermined SQL templates to effectively bridge the semantic gap between natural language questions and structured database queries, achieving commendable performance by harnessing the power of multi-task learning. However, employing identical features across diverse tasks is an ill-suited practice, fraught with inherent drawbacks. Firstly, based on our observation, there are clear boundaries in the natural language corresponding to SELECT and WHERE clauses. Secondly, the exclusive features integral to each subtask are inadequately emphasized and underutilized, thereby hampering the acquisition of discriminative features for each specific subtask. In an endeavor to rectify these issues, the present work introduces an innovative approach: the hierarchical feature decoupling model for SQL query generation from natural language. This novel approach involves the deliberate separation of features pertaining to subtasks within both SELECT and WHERE clauses, further dissociating these features at the subtask level to foster better model performance. Empirical results derived from experiments conducted on the WikiSQL benchmark dataset reveal the superiority of the proposed approach over several state-of-the-art baseline methods in the context of text-to-SQL query generation.

Multi-Criteria Decision-Making Framework with Fuzzy Queries for Multimedia Data Fusion

Multimedia Internet of Things (MIoT) is widely explored in many smart applications for connectivity with wireless communication. Such networks are not like ordinary networks because it has to collect a massive amount of data and are further forwarded to processing systems. As MIoT is very limited in terms of resources for healthcare, smart homes, etc., therefore, energy efficiency with reliable data transmission is a significant research challenge. As smart applications rely on bounded constraints, therefore duplicate and unnecessary data transmission should be minimized. In addition, the timely delivery of data in crucial circumstances has a significant impact on any proposed system. Consequently, this paper presents a fuzzy logic-based edge computing framework to provide cooperative decision-making while avoiding inefficient use of the sensing power of smart devices. The proposed framework can be applied to critical applications to improve response time and processing cost. It consists of the following two functional components: Firstly, it provides the automated routing process with a natural language interface at the sink node. Secondly, to ensure reasonable performance, it also transmits semantic data between sensors using fuzzy queries and security. According to the performance evaluation, the proposed framework significantly outperformed related studies in terms of energy consumption, packet overhead, network throughput, and end-to-end delay.

TRANS-VQA: Fully Transformer-Based Image Question-Answering Model Using Question-guided Vision Attention

Understanding multiple modalities and relating them is an easy task for humans. But for machines, this is a stimulating task. One such multi-modal reasoning task is Visual question answering which demands the machine to produce an answer for the natural language query asked based on the given image. Although plenty of work is done in this field, there is still a challenge of improving the answer prediction ability of the model and breaching human accuracy. A novel model for answering image-based questions based on a transformer has been proposed. The proposed model is a fully Transformer-based architecture that utilizes the power of a transformer for extracting language features as well as for performing joint understanding of question and image features. The proposed VQA model utilizes F-RCNN for image feature extraction. The retrieved language features and object-level image features are fed to a decoder inspired by the Bi-Directional Encoder Representation Transformer - BERT architecture that learns jointly the image characteristics directed by the question characteristics and rich representations of the image features are obtained. Extensive experimentation has been carried out to observe the effect of various hyperparameters on the performance of the model. The experimental results demonstrate that the model’s ability to predict the answer increases with the increase in the number of layers in the transformer’s encoder and decoder. The proposed model improves upon the previous models and is highly scalable due to the introduction of the BERT. Our best model reports 72.31% accuracy on the test-standard split of the VQAv2 dataset.

FAVOR-GPT: a generative natural language interface to whole genome variant functional annotations.

Functional Annotation of genomic Variants Online Resources (FAVOR) offers multi-faceted, whole genome variant functional annotations, which is essential for Whole Genome and Exome Sequencing (WGS/WES) analysis and the functional prioritization of disease-associated variants. A versatile chatbot designed to facilitate informative interpretation and interactive, user-centric summary of the whole genome variant functional annotation data in the FAVOR database is needed. We have developed FAVOR-GPT, a generative natural language interface powered by integrating large language models (LLMs) and FAVOR. It is developed based on the Retrieval Augmented Generation (RAG) approach, and complements the original FAVOR portal, enhancing usability for users, especially those without specialized expertise. FAVOR-GPT simplifies raw annotations by providing interpretable explanations and result summaries in response to the user's prompt. It shows high accuracy when cross-referencing with the FAVOR database, underscoring the robustness of the retrieval framework. Researchers can access FAVOR-GPT at FAVOR's main website (https://favor.genohub.org).

Improving Semantic Analysis in Visualization with Meta Network Representation and Parsing Algorithm

This article aims to advance semantic analysis models, particularly in visualization, by proposing a novel semantic representation method utilizing the semantic Meta Network (MNet). MNet is a complex framework comprising semantic elements, internal and external relationships, and feature attributes, defined hierarchically through recursive processes, aiming to depict the comprehensive semantic space from phrase-level components to complete texts. The methodology involves the development of a general construction algorithm for MNet, encompassing meta relationships, tree structures, and network structures, and a Parsing method for specific semantic analysis problems, including a bottom-up specification-based MNet semantic dependency tree construction algorithm and a network construction algorithm tailored for natural language interface parsing. Empirical experiments confirm the effectiveness of these algorithms, particularly in parsing natural language control interface instructions in Supervisory Control and Data Acquisition (SCADA) systems, bridging specific semantic analysis problems with the general construction and parsing processes of MNet, accounting for internal semantics concerning language unit structures and foreign language meanings in the linguistic context, thereby contributing significantly to the field of natural language semantic analysis.

PEPhub: a database, web interface, and API for editing, sharing, and validating biological sample metadata.

As biological data increase, we need additional infrastructure to share them and promote interoperability. While major effort has been put into sharing data, relatively less emphasis is placed on sharing metadata. Yet, sharing metadata is also important and in some ways has a wider scope than sharing data themselves. Here, we present PEPhub, an approach to improve sharing and interoperability of biological metadata. PEPhub provides an API, natural-language search, and user-friendly web-based sharing and editing of sample metadata tables. We used PEPhub to process more than 100,000 published biological research projects and index them with fast semantic natural-language search. PEPhub thus provides a fast and user-friendly way to finding existing biological research data or to share new data. https://pephub.databio.org.

Logistic Model for Predicting Coliform Growth in Wastewater Effluent From the Chambo River (Ecuador)

Purpose: The objective of this work is to apply a predictive model of bacterial growth to determine the concentration of total coliforms, under controlled laboratory conditions, in wastewater from the city of Riobamba that flows into the Chambo River (Ecuador). Theoretical Framework: Wastewater pollution is a global problem that affects water quality and public health. Population growth has a drastic impact on the contamination of water bodies, particularly in Ecuador, since most of the effluents are not adequately treated, which becomes a public health and environmental problem. For this reason, over the years, total and fecal coliforms have been used as indicators of water quality. Design/Methodology/Approach: The study used a mathematical modeling approach of a logistic and deterministic type, and calculated through an executable command designed in the integrated development environment (IDE) with the Visual C# programming language (logic) and the XAML design language (interface). Findings: The results achieved are 1) the theoretical construction of a mathematical model (logistic model) starting from a differential equation and the taking of three experimental measurements, 2) the computer construction of the bacterial growth model and 3) it has been demonstrated that there is agreement between the values obtained from the model of total coliform growth and the experimental results with a margin of error of less than 1%. Research, Practical & Social Implications: This information can be used by stakeholders to determine actions that could improve current conditions. Originality/Value: The result of this study emphasizes that the model proposed can reproduce the growth of coliforms up to a maximum growth point, it is necessary to continue with the development of the same in the stationary phase of bacterial growth and then in the decay phase.

Logistic Model for Predicting Coliform Growth in Wastewater Effluent From the Chambo River (Ecuador)

Purpose: The objective of this work is to apply a predictive model of bacterial growth to determine the concentration of total coliforms, under controlled laboratory conditions, in wastewater from the city of Riobamba that flows into the Chambo River (Ecuador). Theoretical Framework: Wastewater pollution is a global problem that affects water quality and public health. Population growth has a drastic impact on the contamination of water bodies, particularly in Ecuador, since most of the effluents are not adequately treated, which becomes a public health and environmental problem. For this reason, over the years, total and fecal coliforms have been used as indicators of water quality. Design/Methodology/Approach: The study used a mathematical modeling approach of a logistic and deterministic type, and calculated through an executable command designed in the integrated development environment (IDE) with the Visual C# programming language (logic) and the XAML design language (interface). Findings: The results achieved are 1) the theoretical construction of a mathematical model (logistic model) starting from a differential equation and the taking of three experimental measurements, 2) the computer construction of the bacterial growth model and 3) it has been demonstrated that there is agreement between the values obtained from the model of total coliform growth and the experimental results with a margin of error of less than 1%. Research, Practical & Social Implications: This information can be used by stakeholders to determine actions that could improve current conditions. Originality/Value: The result of this study emphasizes that the model proposed can reproduce the growth of coliforms up to a maximum growth point, it is necessary to continue with the development of the same in the stationary phase of bacterial growth and then in the decay phase.

Zero-shot Video Grounding with Pseudo Query Lookup and Verification.

Video grounding, the process of identifying a specific moment in an untrimmed video based on a natural language query, has become a popular topic in video understanding. However, fully supervised learning approaches for video grounding that require large amounts of annotated data can be expensive and time-consuming. Recently, zero-shot video grounding (ZS-VG) methods that leverage pre-trained object detectors and language models to generate pseudo-supervision for training video grounding models have been developed. However, these approaches have limitations in recognizing diverse categories and capturing specific dynamics and interactions in the video context. To tackle these challenges, we introduce a novel two-stage ZS-VG framework called Lookup-and-Verification (LoVe), which treats the pseudo-query generation procedure as a video-to-concept retrieval problem. Our approach allows for the extraction of diverse concepts from an open-concept pool and employs a verification process to ensure the relevance of the retrieved concepts to the objects or events of interest in the video proposals. Comprehensive experimental results on the Charades-STA, ActivityNet-Captions, and DiDeMo datasets demonstrate the effectiveness of the LoVe framework.

ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language.

The use of natural language interfaces (NLIs) to create charts is becoming increasingly popular due to the intuitiveness of natural language interactions. One key challenge in this approach is to accurately capture user intents and transform them to proper chart specifications. This obstructs the wide use of NLI in chart generation, as users' natural language inputs are generally abstract (i.e., ambiguous or under-specified), without a clear specification of visual encodings. Recently, pre-trained large language models (LLMs) have exhibited superior performance in understanding and generating natural language, demonstrating great potential for downstream tasks. Inspired by this major trend, we propose ChartGPT, generating charts from abstract natural language inputs. However, LLMs are struggling to address complex logic problems. To enable the model to accurately specify the complex parameters and perform operations in chart generation, we decompose the generation process into a step-by-step reasoning pipeline, so that the model only needs to reason a single and specific sub-task during each run. Moreover, LLMs are pre-trained on general datasets, which might be biased for the task of chart generation. To provide adequate visualization knowledge, we create a dataset consisting of abstract utterances and charts and improve model performance through fine-tuning. We further design an interactive interface for ChartGPT that allows users to check and modify the intermediate outputs of each step. The effectiveness of the proposed system is evaluated through quantitative evaluations and a user study.

Analysis of the Possibilities of Carrying Out Attacks on the Functions of Transferring Control to Operating System Console Using Active Intelligence Methods

This paper researches the possibility of conducting attacks on the console control transfer functions using active intelligence methods. The research employs a comprehensive approach involving ports scanning, directories searching, parameters modifying, and credentials searching based on a user dictionary. Additionally, the study involves the development of a software tool designed to detect vulnerabilities in network nodes. The software developed within the framework of this study is delivered in the form of two modules, the first module contains the main program with the mechanisms implemented in it to indicate the possibility of attacks, with an emphasis on current web applications and services. Checking for the possibility of an attack for any network node is that first a list of requests is compiled, the purpose of which is to identify weaknesses of a web application running on the server, and a list of expected responses from the server is also compiled for these requests. After that request goes to the server, the program waits for responses from the server, and if the expected responses from the compiled list coincide with the actual ones, then this fact signals the possibility of an attack on the web application. The second module stores localization dictionaries responsible for the presence of two interface languages in the program – Russian and English. The result of this work is the MaxNetScanner2022 software, which identifies the possibility of an attack on the system control transfer functions using active intelligence tools.

Single-Frame Supervision for Spatio-Temporal Video Grounding.

Spatio-Temporal Video Grounding (STVG) aims at localizing the spatio-temporal tube of a specific object in an untrimmed video given a free-form natural language query. As the annotation of tubes is labor intensive, researchers are motivated to explore weakly supervised approaches in recent works, which usually results in significant performance degradation. To achieve a less expensive STVG method with acceptable accuracy, this work investigates the "single-frame supervision" paradigm that requires a single frame labeled with a bounding box within the temporal boundary of the fully supervised counterpart as the supervisory signal. Based on the characteristics of the STVG problem, we propose a Two-Stage Multiple Instance Learning (T-SMILE) method, which creates pseudo labels by expanding the annotated frame to its contextual frames, thereby establishing a fully-supervised problem to facilitate further model training. The innovations of the proposed method are three-folded, including 1) utilizing multiple instance learning to dynamically select instances in positive bags for the recognition of starting and ending timestamps, 2) learning highly discriminative query features by incorporating spatial prior constraints in cross-attention, and 3) designing a curriculum learning-based strategy that iterative assigns dynamic weights to spatial and temporal branches, thereby gradually adapting to the learning branch with larger difficulty. To facilitate future research on this task, we also contribute a large-scale benchmark containing 12,469 videos on complex scenes with single-frame annotation. The extensive experiments on two benchmarks demonstrate that T-SMILE significantly outperforms all weakly-supervised methods. Remarkably, it also performs better than some fully-supervised methods associated with much more annotation labor costs. The dataset and codes are available at https://github.com/qumengxue/T-SMILE.

Deriving inferences through natural language from structured datasets for asset lifecycle management

Industry 4.0 is expected to revolutionize the manufacturing world in terms of improved decision-making leading to increased value extraction from assets. Yet, since its inception, its adoption seems to have not scaled with time as expected. One of the bottlenecks can be due to the knowledge disparity between the Industry 4.0 technology developer and the domain expert/end user, which can lead to higher resources, cost, and time requirements for scalable adoption. Modern day Large Language Models (LLMs) have the ability to process large datasets and can serve as an intermediatory tool for domain end-users. Thus, in this paper, an Industrial-GPT tool is developed to translate natural language queries into meaningful inferences about asset-related data. The tool is evaluated using an exemplary dataset, representing a manufacturing industry with four production lines, multiple assets, readings and inferences from multiple sensors. The experimental results show that reasonable insight may be extracted from the dataset, using prompt engineering, while fine-tuning the model to adapt to business logic can help achieve better and faster inferences. Future research directions are proposed that can be addressed while developing Industrial-GPT-based tools and reducing the development cycle for Industry 4.0 technologies.

Part of speech-based semantic similarity for RDF Predicate Selection : A benchmarking approach

RDF predicate selection and RDF subject identification are two key steps for effectively processing the natural language queries by the semantic question answering systems. For subject identification, techniques such as Named Entity Recognition are used. For the relatively complex task of predicate selection, Natural Language Processing techniques can be utilized along with knowledge bases to select the semantically matching relationships. Semantic features can be extracted from queries utilizing phrase embeddings of the transformer models. In this paper firstly, a novel Predicate Selection approach on RDF Knowledge Bases has been proposed and implemented utilizing BERT-based phrase embeddings and custom logic for key part-of-speech token combinations. Secondly, the implemented prototype is benchmarked over combinations of key part of speech tokens and stop words for predicate selection accuracy over YAGO Date Facts, demonstrating the effectiveness of our approach. Benchmarking results also highlight the importance of verb stop words for the task of predicate selection.

Evaluation of a Voice-Enabled Autonomous Camera Control System for the da Vinci Surgical Robot

Robotic surgery involves significant task switching between tool control and camera control, which can be a source of distraction and error. This study evaluated the performance of a voice-enabled autonomous camera control system compared to a human-operated camera for the da Vinci surgical robot. Twenty subjects performed a series of tasks that required them to instruct the camera to move to specific locations to complete the tasks. The subjects performed the tasks (1) using an automated camera system that could be tailored based on keywords; and (2) directing a human camera operator using voice commands. The data were analyzed using task completion measures and the NASA Task Load Index (TLX) human performance metrics. The human-operated camera control method was able to outperform an automated algorithm in terms of task completion (6.96 vs. 7.71 correct insertions; p-value = 0.044). However, subjective feedback suggests that a voice-enabled autonomous camera control system is comparable to a human-operated camera control system. Based on the subjects’ feedback, thirteen out of the twenty subjects preferred the voice-enabled autonomous camera control system including the surgeon. This study is a step towards a more natural language interface for surgical robotics as these systems become better partners during surgery.

A Fuzzification-Based Processing of Natural Language Query Using Database

Abstract: The requirement for natural language interfaces to database has turn out to be more and more acute as increasingly more people get entry to data from internet browsers. But NLI (Natural Language Interface) is only usable if they map natural language inquiries to sql queries effectively Natural Language processing is turning into one of the maximum energetic areas in Human-pc interaction. It is a department of AI (artificial Intelligence) which incorporates in records retrieval, gadget translation and Language evaluation. The intention of NLP (Natural Language Processing) is to enable communique between human beings and computers without requiring to memorization of complicated commands and techniques. In other words NLP (natural Language Processing) is strategies that may make the pc apprehend the languages evidently utilized by humans. The main motive of Natural Language query Processing is for an English sentence to be interpreted by way of the pc and suitable action taken. Asking questions to databases in Natural language is a totally handy and easy approach of statistics access, in particular for casual customers who do not recognize complicated database question languages which includes square.

CLUE: Contrastive language-guided learning for referring video object segmentation

Referring video object segmentation (R-VOS), the task of separating the object described by a natural language query from the video frames, has become increasingly critical with recent advances in multi-modal understanding. Existing approaches are mainly visual-dominant in both representation-learning and decision-making process, and are less sensitive to fine-grained clues in text description. To address this, we propose a language-guided contrastive learning and data augmentation framework to enhance the model sensitivity to the fine-grained textual clues (i.e., color, location, subject) in the text that relate heavily to the video information. By substituting key information of the original sentences and paraphrasing them with a text-based generation model, our approach conducts contrastive learning through automatically building diverse and fluent contrastive samples. We further enhance the multi-modal alignment with a sparse attention mechanism, which can find the most relevant video information by optimal transport. Experiments on a large-scale R-VOS benchmark show that our method significantly improves strong Transformer-based baselines, and further analysis demonstrates the better ability of our model in identifying textual semantics.

Semi-supervised inverted file index approach for approximate nearest neighbor search

This paper introduces a novel modification to the Inverted File (IVF) index approach for approximate nearest neighbor search, incorporating supervised learning techniques to enhance the efficacy of intermediate clustering and achieve more balanced cluster sizes. The proposed method involves creating clusters using a neural network by solving a task to classify query vectors into the same bucket as their corresponding nearest neighbor vectors in the original dataset. When combined with minimizing the standard deviation of the bucket sizes, the indexing process becomes more efficient and accurate during the approximate nearest neighbor search. Through empirical evaluation on a test dataset, we demonstrate that the proposed semi-supervised IVF index approach outperforms the industry-standard IVF implementation with fixed parameters, including the total number of clusters and the number of clusters allocated to queries. This novel approach has promising implications for enhancing nearest-neighbor search efficiency in high-dimensional datasets across various applications, including information retrieval, natural language search, recommendation systems, etc.

Generative AI in Management – Today and Tomorrow

Theoretical background: The rapid and exponential technological advancements have far-reaching impacts on management information systems, management practices, and human life. The promising outcomes in Artificial Intelligence and cutting-edge research on semantic networks and natural language processing have motivated the authors to envision the future of management technology. Purpose of the article: Our paper focuses on the new communication facilities and artificial intelligence models used to process management-type queries in natural language. Research methods: The article discusses recently developed technologies, proposed by Google and Microsoft, notably Google Bard and Bing integrated with ChatGPT-4. Both chatbots use Generative AI methods and large language models to understand domain-based queries and generate answers. Main findings: The practical and social implications of new models in management practice are discussed. To illustrate the qualities and weaknesses of the features of new technologies, four examples of management decision-making are discussed. The case studies also show differences between these two technologies. Finally, the paper concludes by summarizing the expectations and limitations of Generative AI applications in management. The paper is one of the first publications describing and demonstrating the idea of interfaces in natural language in business-oriented applications.