Language Interface Research Articles

When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs

The use of machine learning (ML) models to assess and score textual data has become increasingly pervasive in an array of contexts including natural language processing, information retrieval, search and recommendation, and credibility assessment of online content. A significant disruption at the intersection of ML and text are text-generating large-language models (LLMs) such as generative pre-trained transformers (GPTs). We empirically assess the differences in how ML-based scoring models trained on human content assess the quality of content generated by humans versus GPTs. To do so, we propose an analysis framework that encompasses essay scoring ML-models, human and ML-generated essays, and a statistical model that parsimoniously considers the impact of type of respondent, prompt genre, and the ML model used for assessment model. A rich testbed is utilized that encompasses 18,460 human-generated and GPT-based essays. Results of our benchmark analysis reveal that LLMs and transformer pretrained language models (PLMs) more accurately score human essay quality as compared to CNN/RNN and feature-based ML methods. Interestingly, we find that LLMs and transformer PLMs tend to score GPT-generated text 10-20% higher on average, relative to human-authored documents. Conversely, traditional deep learning and feature-based ML models score human text considerably higher. Further analysis reveals that even though the LLMs and transformer PLMs are exclusively fine-tuned on human text, they more prominently attend to certain tokens appearing only in GPT-generated text, possibly (in part) due to familiarity/overlap in pre-training. Our framework and results have implications for text classification settings where automated scoring of text is likely to be disrupted by generative AI.

Under the lens: Learning management system (LMS) platforms and DEIA components

Many global higher educational institutions speak of accessibility, diversity, and inclusivity for students in terms of assignments, documents and syllabi, and assessments, but most in academia have not fully addressed the need for true accessibility in the learning management system (LMS) environment. The chosen LMS must be able to translate course content into meaningful lexicon for all students, including those who are multilingual, and/or have learning differences, yet it is not yet inherently capable of doing so. Faculty and higher educational leaders throughout the halls of higher learning do not have a universal, mandatory, standardized means of being trained in LMS best practices. Selections of a primary language interface can be made, but content uploaded by American English professors, for example, cannot be automatically translated into another language by a LMS. Many issues arise with how to provide alternative text for images in the testing environment without revealing the assessment’s answers, yet providing enough context with which to select the appropriate answer choice. We will explore the delicate balance of using alt text, scripting, translation, and other existing accessibility, diversity, and inclusivity tools within LMS platforms, address the need for global, universal LMS best practices training for academics, and delve into other issues that inhibit student success within the online learning environment.

Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey

Natural Language Interfaces for Tabular Data Querying and Visualization: A Survey

SOX-10 and Melan-A Immunostaining in Areas of Focal Acantholytic Dyskeratosis and Epidermolytic Hyperkeratosis Within Dysplastic Nevi Biopsies: An Observational Study.

Focal acantholytic dyskeratosis (FAD) and epidermolytic hyperkeratosis (EHK) are common incidental epidermal histologic findings within dysplastic nevi biopsies. We evaluate whether areas of FAD and EHK within dysplastic nevi biopsies stain with immunostains used to characterize melanocytic neoplasms. In this case series, a natural language search of histopathology reports from our institution in the past year (2020-2021) identified dysplastic nevus biopsies with concurrent FAD and/or EHK. Tissue samples were examined for positive melanocytic immunostaining with SOX-10 and Melan-A in areas of FAD and EHK. Out of 32 biopsies, 20 of 26 FAD specimens (76.9%) and 2 of 6 EHK specimens (33.3%) showed unexpected suprabasal layer staining with a melanocytic marker that did not correspond to definitively identified melanocytes on the H&E-stained sections. The immunohistochemical staining of FAD and EHK was observed in 2 forms: nonspecific background staining or "true" staining (ie, seemed nuclear on SOX-10 or cytoplasmic on Melan-A). This pilot examination provides evidence that areas of incidental FAD within dysplastic nevi biopsies demonstrate unexpected suprabasal layer staining with melanocytic markers. When dermatopathologists evaluate melanocytic neoplasms with melanocytic markers, it is possible the presence of incidental FAD could lead to over diagnosing pagetoid scatter within these lesions. This study is a proof of concept with mild to moderately dysplastic nevi that do not typically incur the use of melanocytic stains; however, the implication of this unexpected staining pattern would be important when using melanocytic markers on borderline melanocytic neoplasms that have incidental FAD. Close correlation with H&E is imperative to prevent misinterpretation in these cases.

Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions

AbstractNatural language and visualization are two complementary modalities of human communication that play a crucial role in conveying information effectively. While visualizations help people discover trends, patterns and anomalies in data, natural language descriptions help explain these insights. Thus, combining text with visualizations is a prevalent technique for effectively delivering the core message of the data. Given the rise of natural language generation (NLG), there is a growing interest in automatically creating natural language descriptions for visualizations, which can be used as chart captions, answering questions about charts or telling data‐driven stories. In this survey, we systematically review the state of the art on NLG for visualizations and introduce a taxonomy of the problem. The NLG tasks fall within the domain of natural language interfaces (NLIs) for visualization, an area that has garnered significant attention from both the research community and industry. To narrow down the scope of the survey, we primarily concentrate on the research works that focus on text generation for visualizations. To characterize the NLG problem and the design space of proposed solutions, we pose five Wh‐questions, why and how NLG tasks are performed for visualizations, what the task inputs and outputs are, as well as where and when the generated texts are integrated with visualizations. We categorize the solutions used in the surveyed papers based on these ‘five Wh‐questions’. Finally, we discuss the key challenges and potential avenues for future research in this domain.

GraphPPL.jl: A Probabilistic Programming Language for Graphical Models

This paper presents GraphPPL.jl, a novel probabilistic programming language designed for graphical models. GraphPPL.jl uniquely represents probabilistic models as factor graphs. A notable feature of GraphPPL.jl is its model nesting capability, which facilitates the creation of modular graphical models and significantly simplifies the development of large (hierarchical) graphical models. Furthermore, GraphPPL.jl offers a plugin system to incorporate inference-specific information into the graph, allowing integration with various well-known inference engines. To demonstrate this, GraphPPL.jl includes a flexible plugin to define a Constrained Bethe Free Energy minimization process, also known as variational inference. In particular, the Constrained Bethe Free Energy defined by GraphPPL.jl serves as a potential inference framework for numerous well-known inference backends, making it a versatile tool for diverse applications. This paper details the design and implementation of GraphPPL.jl, highlighting its power, expressiveness, and user-friendliness. It also emphasizes the clear separation between model definition and inference while providing developers with extensibility and customization options. This establishes GraphPPL.jl as a high-level user interface language that allows users to create complex graphical models without being burdened with the complexity of inference while allowing backend developers to easily adopt GraphPPL.jl as their frontend language.

Intelligent Hazard Assessment of Mangrove Degradation

Mangrove ecosystems face multiple threats from human activities and natural disasters, posing a threat to their existence. The results of the automated system-cognitive analysis implemented in the intellectual system Eidos and allowing to assess the danger of degradation of mangrove forests are presented. Factors that have a negative and positive impact on the well-being of mangrove ecosystems are taken into account, and the strength and direction of the influence of each threat are determined. The results of the study can be used by coastal protection funds in many countries of the world due to the fact that the Eidos system is available free of charge on the Internet, and it is possible to change the user interface language to another most common language in the world. The application of the new research methodology will provide valuable information on the risk of degradation of mangrove forests in the current conditions of their growth in order to take timely measures for their conservation. By understanding the threats that the mangrove ecosystem faces, effective management practices can be applied that contribute to the conservation and sustainable development of this vital coastal resource

Effective Hard Negative Mining for Contrastive Learning-based Code Search

Background. Code search aims to find the most relevant code snippet in a large codebase based on a given natural language query. An accurate code search engine can increase code reuse and improve programming efficiency. The focus of code search is how to represent the semantic similarity of code and query. With the development of code pre-trained models, the pattern of using numeric feature vectors (embeddings) to represent code semantics and using vector distance to represent semantic similarity has replaced traditional string matching methods. The quality of semantic representations is critical to the effectiveness of downstream tasks such as code search. Currently, the state-of-the-art (SOTA) learning method uses the contrastive learning paradigm. The objective of contrastive learning is to maximize the similarity between matching code and query (positive samples) and minimize the similarity between mismatched pairs (negative samples). To increase the reusing of negative samples, prior contrastive learning approaches use a large queue (memory bank) to store embeddings. Problem. However, there is still a lot of room for improvement in using negative examples for code search: ① Due to the random selection of negative samples, semantic representations learned by existing models cannot distinguish similar codes well. ② Since semantic vectors in the memory bank are reused from previous inference results and then directly used for loss function calculation without gradient descent, the model cannot effectively learn the negative sample semantic information. Method. To solve the above problems, we propose a contrastive learning code search model with hard negative mining called CoCoHaNeRe: ❶ To enable the model to distinguish similar codes, we introduce hard negative examples into contrastive training, which are negative examples in the codebase that are most similar to positive examples. As a result, hard negative examples are most likely to make the model make mistakes. ❷ To improve the learning efficiency of negative samples during training, we add all hard negative examples to the model's gradient descent process. Result. To verify the effectiveness of CoCoHaNeRe, we conducted experiments on large code search datasets with six programming languages, as well as similar retrieval tasks code clone detection and code question answering. Experimental results show that our model achieves SOTA performance. In the code search task, the average MRR score of CoCoHaNeRe exceeds CodeBERT, GraphCodeBERT, and UniXcoder by 11.25%, 8.13%, and 7.38%, respectively. It has also made great progress in code clone detection and code question answering. In addition, our method performs well in different programming languages and code pre-training models. Furthermore, qualitative analysis shows that our model effectively distinguishes high-order semantic differences between similar codes.

Understanding with the body? Testing the role of verb relative embodiment across tasks at the interface of language and memory

Multiple representation accounts of conceptual knowledge argue that information associated with sensory-motor experience, in addition to pure linguistic information, contributes to word processing. A number of issues, however, remain under-investigated, including the extent to which these dimensions affect verb processing (rather than nouns), especially in languages other than English, and their role across different tasks along the language-memory continuum. Here, we collected ratings for a verb-specific dimension linked to bodily experience (relative embodiment, RE) for 647 Italian verbs and we tested its effects in three tasks differently modulating semantic activation and memory processes (i.e., lexical decision, grammatical decision, and memory recognition). Our results showed reliable influences of RE during lexical decision and memory recognition, but not in grammatical decision, possibly due to the Italian morphological richness. The cross-task analysis showed that RE effects were substantially higher in memory recognition compared to lexical decision, indicating that semantic and episodic processes interact at the interface of language and memory. Overall, results support the flexible and context-dependent role of sensory-motor and bodily-related experience during verb processing, pointing also to language-specific factors and implications for the organization of declarative memory.

LLM Based Chatbot for Farm-to-Fork Blockchain Traceability Platform

Blockchain technology has been used with great effect in farm-to-fork traceability projects. However, this technology has a steep learning curve when it comes to its user interface. To minimize this difficulty, we created a solution based on a Large Language Model (LLM) conversational agent. Our implementation, starting with an existing knowledge base that is prepared and processed with an embedding model to be stored in a vector database, follows a Retrieval-Augmented Generation (RAG) approach. Other non-textual media like images and videos are aggregated with the embeddings to enrich the user experience. User queries are combined with a proximity search in the vector database and feed into an LLM that considers the conversation history with the user in its replies. Given the asynchronous nature of these models, we implemented a similarly asynchronous scheme using Server-Sent Events that deliver the models’ replies to a UI that supports multimodal media types such as images and videos by providing the visualization of these resources. The end solution allows users to interact with advanced technologies using a natural language interface; this in turn empowers food traceability projects to overcome their natural difficulty in engaging early adopters.

ChatGeoAI: Enabling Geospatial Analysis for Public through Natural Language, with Large Language Models

Large Language Models (LLMs) such as GPT, BART, and Gemini stand at the forefront of Generative Artificial Intelligence, showcasing remarkable prowess in natural language comprehension and task execution. This paper proposes a novel framework developed on the foundation of Llama 2, aiming to bridge the gap between natural language queries and executable code for geospatial analyses within the PyQGIS environment. It empowers non-expert users to leverage GIS technology without requiring deep knowledge of geospatial programming or tools. Through cutting-edge Natural Language Processing (NLP) techniques, including tailored entity recognition and ontology mapping, the framework accurately interprets user intents and translates them into specific GIS operations. Integration of geospatial ontologies enriches semantic comprehension, ensuring precise alignment between user descriptions, geospatial datasets, and geospatial analysis tasks. A code generation module empowered by Llama 2 converts these interpretations into PyQGIS scripts, enabling the execution of geospatial analysis and results visualization. Rigorous testing across a spectrum of geospatial analysis tasks, with incremental complexity, evaluates the framework and the performance of such a system, with LLM at its core. The proposed system demonstrates proficiency in handling various geometries, spatial relationships, and attribute queries, enabling accurate and efficient analysis of spatial datasets. Moreover, it offers robust error-handling mechanisms and supports tasks related to map styling, visualization, and data manipulation. However, it has some limitations, such as occasional struggles with ambiguous attribute names and aliases, which leads to potential inaccuracies in the filtering and retrieval of features. Despite these limitations, the system presents a promising solution for applications integrating LLMs into GIS and offers a flexible and user-friendly approach to geospatial analysis.

Aberrant CD45 Immunoreactivity in Neuroendocrine Neoplasms: A Diagnostic Pitfall-Report of 10 Specimens and Clinical Recommendations.

Leukocyte common antigen (LCA), or CD45, is classically thought of as a leukocyte-exclusive protein, and as such, CD45 immunohistochemistry (IHC) is often used as a key differentiator between non-Hodgkin lymphomas (NHLs) and morphologically similar neuroendocrine neoplasms (NENs). Herein, we report our experience regarding aberrant CD45 immunoreactivity in a series of NENs. A natural language search was used to retrieve desired archival patient files. All prior NENs which had a positive neuroendocrine diagnosis or IHC results (synaptophysin, chromogranin, CD56, and/or neuron-specific enolase), as well as CD45 staining performed, were reviewed for possible CD45 positivity (n = 686). Among these 686 NENs, 10 were aberrantly positive for CD45 staining. CD45 showed nuclear, cytoplasmic, and/or membranous staining in tumor cells. The significance of such staining is unclear. Albeit for a minority of patients, pathologists should be aware that NENs may aberrantly stain with CD45 and thereby pose a diagnostic pitfall. Therefore, broadening routine IHC panels is recommended to differentiate NENs more clearly from NHLs.

Framework for LLM applications in manufacturing

In the era of Industry 4.0, the proliferation of data within manufacturing environments has presented both unprecedented opportunities and challenges. This paper introduces a framework that capitalizes on the capabilities of Large Language Models (LLMs) to revolutionize data integration and decision-making processes in manufacturing systems. Addressing the critical need for efficient data management, our framework streamlines the consolidation, processing, and generation of responses to essential inquiries, thus enhancing manufacturers’ capabilities to extract valuable insights. The focus of this paper is twofold. First to establish a framework for the use of LLM applications in manufacturing settings. Secondly, to provide an overview of the manufacturing connection between data, AI, and chat-bots, while also addressing a few pain points identified from the manufacturing literature. The paper then introduces FILLIS (Factory Integrated Logic and Language Interface System), a Large Language Model assistant, through a compelling case study. FILLIS showcases remarkable versatility, excelling in tasks ranging from elucidating machine operations to language translation. The study underscores FILLIS’s proficiency in handling specific contexts, answering questions from uploaded documents with precision. However, inherent limitations surface in tasks involving mathematical operations, emphasizing the need for external agents in specific scenarios. This pivotal opportunity is explored in the proposed framework as it advocates for integrating external agents alongside LLMs, creating a more versatile and comprehensive assistant tool. The findings of this paper and proposed framework position LLMs as transformative tools for intelligent data processing.

Building a Natural Language Interface for FHIR Clinical Terminology Server.

While Fast Healthcare Interoperability Resources (FHIR) clinical terminology server enables quick and easy search and retrieval of coded medical data, it still has some drawbacks. When searching, any typographical errors, variations in word forms, or deviations in word sequence might lead to incorrect search outcomes. For retrieval, queries to the server must strictly follow the FHIR application programming interface format, which requires users to know the syntax and remember the attribute codes they wish to retrieve. To improve its functionalities, a natural language interface was built, that harnesses the capabilities of two preeminent large language models, along with other cutting-edge technologies such as speech-to-text conversion, vector semantic searching, and conversational artificial intelligence. Preliminary evaluation shows promising results in building a natural language interface for the FHIR clinical terminology system.

Zero-shot visual grounding via coarse-to-fine representation learning

Visual grounding (VG) locates target objects in visual scenes by understanding given natural language queries. Current methods for VG mainly focus on grounding referring expressions or noun phrases covered in the labeled training samples. Despite their grounding prowess, these approaches struggle in grounding novel query–image pairs excluded from the training data. This shortage is usually caused by the deficiency of discriminative representation learning both in images and queries. To address these issues, we propose a one-stage coarse-to-fine framework for zero-shot VG to ground novel query-image samples. Specifically, in the coarse stage, we mine the global context information in the visual features and query embeddings by employing a multi-head self-attention block, strengthening the intra-modality relations in the visual and textual features. In the fine stage, we first learn the query-aware visual representations based on the acquired global context information via a multi-modal relation-enhanced transformer block, which explores the informative information by modeling the cross-modal interaction among visual and textual domains. We further excavate target-oriented discriminative representations from the acquired query-aware visual representations by a noun phrase-guided multi-modal interaction network, which augments the interaction between target-related phrases and the obtained query-aware visual representations to enhance the distinction of target regions, enhancing the subsequent referred target grounding and generalizing. In order to validate the proposed approach, we implement extensive experiments and ablation studies on public benchmark datasets, including RefCOCO, RefCOCO+, RefCOCOg, Flickr30K Entity, Flickr-Split-0 and Flickr-Split-1. Experimental results demonstrate that our approach substantially improves the grounding accuracy and achieves new state-of-the-art performance under single-stage training and testing.

Triangulating evidence in health sciences with Annotated Semantic Queries.

Integrating information from data sources representing different study designs has the potential to strengthen evidence in population health research. However, this concept of evidence "triangulation" presents a number of challenges for systematically identifying and integrating relevant information. These include the harmonization of heterogenous evidence with common semantic concepts and properties, as well as the priortization of the retrieved evidence for triangulation with the question of interest. We present Annotated Semantic Queries (ASQ), a natural language query interface to the integrated biomedical entities and epidemiological evidence in EpiGraphDB, which enables users to extract "claims" from a piece of unstructured text, and then investigate the evidence that could either support, contradict the claims, or offer additional information to the query. This approach has the potential to support the rapid review of preprints, grant applications, conference abstracts, and articles submitted for peer review. ASQ implements strategies to harmonize biomedical entities in different taxonomies and evidence from different sources, to facilitate evidence triangulation and interpretation. ASQ is openly available at https://asq.epigraphdb.org and its source code is available at https://github.com/mrcieu/epigraphdb-asq under GPL-3.0 license.

Event-Oriented State Alignment Network for Weakly Supervised Temporal Language Grounding.

Weakly supervised temporal language grounding (TLG) aims to locate events in untrimmed videos based on natural language queries without temporal annotations, necessitating a deep understanding of semantic context across both video and text modalities. Existing methods often focus on simple correlations between query phrases and isolated video segments, neglecting the event-oriented semantic coherence and consistency required for accurate temporal grounding. This can lead to misleading results due to partial frame correlations. To address these limitations, we propose the Event-oriented State Alignment Network (ESAN), which constructs "start-event-end" semantic state sets for both textual and video data. ESAN employs relative entropy for cross-modal alignment through knowledge distillation from pre-trained large models, thereby enhancing semantic coherence within each modality and ensuring consistency across modalities. Our approach leverages vision-language models to extract static frame semantics and large language models to capture dynamic semantic changes, facilitating a more comprehensive understanding of events. Experiments conducted on two benchmark datasets demonstrate that ESAN significantly outperforms existing methods. By reducing false high correlations and improving the overall performance, our method effectively addresses the challenges posed by previous approaches. These advancements highlight the potential of ESAN to improve the precision and reliability of temporal language grounding tasks.

Developing a Generative AI-Powered Chatbot for Analyzing MAUDE Database.

This paper presents a chatbot that simplifies accessing and understanding the open-access records of adverse events related to medical devices in the MAUDE database. The chatbot is powered by generative AI technology, enabling count and search queries. The chatbot uses the openFDA API and GPT-4 model to interpret users' natural language queries, generate appropriate API calls, and summarize adverse event reports. The chatbot also provides a downloadable link to the original reports. The model's performance in generating accurate API calls was assessed and improved by training it with few-shot examples of query-URL pairs. Additionally, the quality of content-based summaries was evaluated by human expert ratings. This initiative is a significant step towards making patient safety data accessible, replicable, and easily manageable by a broader range of researchers.

VAIV bio-discovery service using transformer model and retrieval augmented generation

BackgroundThere has been a considerable advancement in AI technologies like LLM and machine learning to support biomedical knowledge discovery.Main bodyWe propose a novel biomedical neural search service called ‘VAIV Bio-Discovery’, which supports enhanced knowledge discovery and document search on unstructured text such as PubMed. It mainly handles with information related to chemical compound/drugs, gene/proteins, diseases, and their interactions (chemical compounds/drugs-proteins/gene including drugs-targets, drug-drug, and drug-disease). To provide comprehensive knowledge, the system offers four search options: basic search, entity and interaction search, and natural language search. We employ T5slim_dec, which adapts the autoregressive generation task of the T5 (text-to-text transfer transformer) to the interaction extraction task by removing the self-attention layer in the decoder block. It also assists in interpreting research findings by summarizing the retrieved search results for a given natural language query with Retrieval Augmented Generation (RAG). The search engine is built with a hybrid method that combines neural search with the probabilistic search, BM25.ConclusionAs a result, our system can better understand the context, semantics and relationships between terms within the document, enhancing search accuracy. This research contributes to the rapidly evolving biomedical field by introducing a new service to access and discover relevant knowledge.

Considering Socially Scalable Human-Robot Interfaces

Collaborative robots are becoming increasingly present in everyday life, with applications ranging food and parcel delivery, security, and more. They can offer great value propositions for organizations and consumers. However, most people lack knowledge of how to interact with robots, and many robots themselves necessitate formal training which can be inaccessible to many and thus not societally scalable. Further, there is a lack of existing work investigating interfaces designs that can support non-dyadic interactions consisting of two or more individuals interacting with a robot sequentially and simultaneously; such interactions will be common in real-world usage. This research explores the efficacy of natural language interfaces for human-robot interaction through a human experiment and post-experiment survey. Results show that the natural language interface can afford teams enhanced capabilities to share robot control and avoid errors relative to other interfaces, while also increasing user perceptions towards overall interaction.