Language Modeling Framework Research Articles

GeoLLM: A specialized large language model framework for intelligent geotechnical design

Large language models (LLMs) have achieved remarkable success in various industrial and research fields, enhancing work efficiency by assisting machines in comprehending human language. In geotechnical design where extensive repetitive cross-checking of design codes consumes considerable time and labour, the utilization of LLMs to enhance design procedures has not been explored before. The challenge is to ensure that LLMs accurately comprehend professional geotechnical information from text and execute mathematical calculations correctly. This study makes the first attempt at developing a specialized LLM framework, GeoLLM, integrated with an innovative prompt engineering strategy to extract professional information from text and enable accurate mathematical calculations. GeoLLM is applied to the design of single piles involving bearing capacity and settlement calculations. The results reveal that GeoLLM exhibits excellent performance in single pile cases. Additionally, compared with LLMs of varying architectures and sizes, commercial LLMs with over 100 billion parameters presented outstanding comprehensive capacities, while those with 1.8 ∼ 72 billion parameters degraded relatively. These findings indicate the promising capacity of GeoLLM to address professional tasks in geotechnical design.

Conversing with business process-aware large language models: the BPLLM framework

AbstractTraditionally, process-aware Decision Support Systems (DSSs) have been enhanced with AI functionalities to facilitate quick and informed decision-making. In this context, AI-Augmented Business Process Management Systems have emerged as innovative human-centric information systems, blending flexibility, autonomy, and conversational capability. Large Language Models (LLMs) have significantly boosted such systems, showcasing remarkable natural language processing capabilities across various tasks. Despite the potential of LLMs to support human decisions in business contexts, empirical validations of their effectiveness for process-aware decision support are scarce in the literature. In this paper, we propose the Business Process Large Language Model (BPLLM) framework, a novel approach for enacting actionable conversations with human workers. BPLLM couples Retrieval-Augmented Generation with fine-tuning, to enrich process-specific knowledge. Additionally, a process-aware chunking approach is incorporated to enhance the BPLLM pipeline. We evaluated the approach in various experimental scenarios to assess its ability to generate accurate and contextually relevant answers to users’ questions. The empirical study shows the promising performance of the framework in identifying the presence of particular activities and sequence flows within the considered process model, offering insights into its potential for enhancing process-aware DSSs.

High-Altitude-UAV-Relayed Satellite D2D Communications for 6G IoT Network

High-altitude UAVs (HA-UAVs) have emerged as vital components in 6G communication infrastructures, providing stable relays for telecommunications services above terrestrial and aerial disturbances. This paper explores the multifaceted roles of HA-UAVs in remote sensing, data relay, and telecommunication network enhancement. A Large Language Model (LLM) framework is introduced that dynamically predicts optimal HA-UAV connectivity for IoT devices, enhancing network performance and adaptability. The study emphasizes HA-UAVs’ operational efficiency, broad coverage, and potential to transform global communications, particularly in remote and underserved areas. Our proposed satellite-HA-UAV-IoT architecture with LLM optimization demonstrated substantial improvements, including a 25% increase in network throughput (from 20 Mbps to 25 Mbps at a 20 km distance), a 40% reduction in latency (from 25 ms to 15 ms), and a 28% enhancement in energy efficiency (from 0.25 μJ/bit to 0.18 μJ/bit), significantly advancing the performance and adaptability of next-generation IoT networks. These advancements pave the way for unprecedented connectivity and set the stage for future communication technologies.

A link prediction method for Chinese financial event knowledge graph based on graph attention networks and convolutional neural networks

Finance is a knowledge-intensive domain in nature, with its data containing a significant amount of interconnected information. Constructing a financial knowledge graph is an important application for transforming financial text/web content into machine-readable data. However, the complexity of Chinese financial knowledge and the dynamic and evolving nature of Chinese financial data often lead to incomplete knowledge graphs. To address this challenge, we propose a novel link prediction method for Chinese financial event knowledge graph based on Graph Attention Networks and Convolutional Neural Networks. Our method begins with the construction of the foundational Chinese financial event knowledge graph using a relational triple extraction module integrated with a large language model framework, along with a Prompting with Iterative Verification (PiVe) module for validation. To enhance the completeness of the knowledge graph, we introduce an encoder-decoder framework, where a graph attention network with joint embeddings of financial event entities and relations acts as the encoder, while a Convolutional Knowledge Base embedding model (ConvKB) serves as the decoder. This framework effectively aggregates crucial neighbor information and captures global relationships among entity and relation embeddings. Extensive comparative experiments demonstrate the utility and accuracy of this method, ultimately enabling the effective completion of Chinese financial event knowledge graphs.

GEML: a graph-enhanced pre-trained language model framework for text classification via mutual learning

GEML: a graph-enhanced pre-trained language model framework for text classification via mutual learning

Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology

Although augmenting large language models (LLMs) with knowledge bases may improve medical domain-specific performance, practical methods are needed for local implementation of LLMs that address privacy concerns and enhance accessibility for health care professionals. To develop an accurate, cost-effective local implementation of an LLM to mitigate privacy concerns and support their practical deployment in health care settings. ChatZOC (Sun Yat-Sen University Zhongshan Ophthalmology Center), a retrieval-augmented LLM framework, was developed by enhancing a baseline LLM with a comprehensive ophthalmic dataset and evaluation framework (CODE), which includes over 30 000 pieces of ophthalmic knowledge. This LLM was benchmarked against 10 representative LLMs, including GPT-4 and GPT-3.5 Turbo (OpenAI), across 300 clinical questions in ophthalmology. The evaluation, involving a panel of medical experts and biomedical researchers, focused on accuracy, utility, and safety. A double-masked approach was used to try to minimize bias assessment across all models. The study used a comprehensive knowledge base derived from ophthalmic clinical practice, without directly involving clinical patients. LLM response to clinical questions. Accuracy, utility, and safety of LLMs in responding to clinical questions. The baseline model achieved a human ranking score of 0.48. The retrieval-augmented LLM had a score of 0.60, a difference of 0.12 (95% CI, 0.02-0.22; P = .02) from baseline and not different from GPT-4 with a score of 0.61 (difference = 0.01; 95% CI, -0.11 to 0.13; P = .89). For scientific consensus, the retrieval-augmented LLM was 84.0% compared with the baseline model of 46.5% (difference = 37.5%; 95% CI, 29.0%-46.0%; P < .001) and not different from GPT-4 with a value of 79.2% (difference = 4.8%; 95% CI, -0.3% to 10.0%; P = .06). Results of this quality improvement study suggest that the integration of high-quality knowledge bases improved the LLM's performance in medical domains. This study highlights the transformative potential of augmented LLMs in clinical practice by providing reliable, safe, and practical clinical information. Further research is needed to explore the broader application of such frameworks in the real world.

MeLM, a generative pretrained language modeling framework that solves forward and inverse mechanics problems

We report a flexible multi-modal mechanics language model, MeLM, applied to solve various nonlinear forward and inverse problems, that can deal with a set of instructions, numbers and microstructure data. The framework is applied to various examples including bio-inspired hierarchical honeycomb design, carbon nanotube mechanics, and protein unfolding. In spite of the adaptable nature of the model-which allows us to easily incorporate diverse materials, scales, and mechanical features-it performs well across disparate forward and inverse tasks. Based on an autoregressive attention-model, MeLM effectively represents a large multi-particle system consisting of hundreds of millions of neurons, where the interaction potentials are discovered through graph-forming self-attention mechanisms that are then used to identify relationships from emergent structures, while taking advantage of synergies discovered in the training data. We show that the model can solve complex degenerate mechanics design problems and determine novel material architectures across a range of hierarchical levels, providing an avenue for materials discovery and analysis. To illustrate the use case for broader possibilities, we outline a human-machine interactive MechGPT model, here trained on a set of 1,103 Wikipedia articles related to mechanics, showing how the general framework can be used not only to solve forward and inverse problems but in addition, for complex language tasks like summarization, generation of new research concepts, and knowledge extraction. Looking beyond the demonstrations reported in this paper, we discuss other opportunities in applied mechanics and general considerations about the use of large language models in modeling, design, and analysis that can span a broad spectrum of material properties from mechanical, thermal, optical, to electronic.

A Large Language Model Framework to Uncover Underreporting in Traffic Crashes

A Large Language Model Framework to Uncover Underreporting in Traffic Crashes

TOPIC MODELING BY USING LDA

Topic modelling plays a significant Information retrieval showed good performance on a wide variety of tasks over the years. It has wide research in machine learning and text mining. The combination of the probability mixture model and the term model presents the topic model. They are the language modelling framework of information retrieval. The present topic modelling methods are probabilistic latent semantic analysis and latent dirichlet allocation. Two main faults of topic modelling are First, common or popular words are different topics, often causing ambivalence to understand the topic. Second, single words have to be presented correctly. Actual problems of topic modelling are Efficiency and scaling are discussed and compared in different types of Topic modelling. The review of article is about how topic modelling for information retrieval is done using LDA(Latent Dirichlet Allocation).

Impact of Sentence Representation Matching in Neural Machine Translation

Most neural machine translation models are implemented as a conditional language model framework composed of encoder and decoder models. This framework learns complex and long-distant dependencies, but its deep structure causes inefficiency in training. Matching vector representations of source and target sentences improves the inefficiency by shortening the depth from parameters to costs and generalizes NMTs with a different perspective to cross-entropy loss. In this paper, we propose matching methods to derive the cost based on constant word-embedding vectors of source and target sentences. To find the best method, we analyze the impact of the methods with varying structures, distance metrics, and model capacity in a French to English translation task. An optimally configured method is applied to English translation tasks from and to French, Spanish, and German. In the tasks, the method showed performance improvement by 3.23 BLEU at maximum, with an improvement of 0.71 on average. We evaluated the robustness of this method to various embedding distributions and models, such as conventional gated structures and transformer networks, and empirical results showed that it has a higher chance to improve performance in those models.

Quantum Natural Language Processing on Near-Term Quantum Computers

In this work, we describe a full-stack pipeline for natural language processing on near-term quantum computers, aka QNLP. The language-modelling framework we employ is that of compositional distributional semantics (DisCoCat), which extends and complements the compositional structure of pregroup grammars. Within this model, the grammatical reduction of a sentence is interpreted as a diagram, encoding a specific interaction of words according to the grammar. It is this interaction which, together with a specific choice of word embedding, realises the meaning (or "semantics") of a sentence. Building on the formal quantum-like nature of such interactions, we present a method for mapping DisCoCat diagrams to quantum circuits. Our methodology is compatible both with NISQ devices and with established Quantum Machine Learning techniques, paving the way to near-term applications of quantum technology to natural language processing.

Hierarchical Concept-Driven Language Model

For guiding natural language generation, many semantic-driven methods have been proposed. While clearly improving the performance of the end-to-end training task, these existing semantic-driven methods still have clear limitations: for example, (i) they only utilize shallow semantic signals (e.g., from topic models) with only a single stochastic hidden layer in their data generation process, which suffer easily from noise (especially adapted for short-text etc.) and lack of interpretation; (ii) they ignore the sentence order and document context, as they treat each document as a bag of sentences, and fail to capture the long-distance dependencies and global semantic meaning of a document. To overcome these problems, we propose a novel semantic-driven language modeling framework, which is a method to learn a Hierarchical Language Model and a Recurrent Conceptualization-enhanced Gamma Belief Network, simultaneously. For scalable inference, we develop the auto-encoding Variational Recurrent Inference, allowing efficient end-to-end training and simultaneously capturing global semantics from a text corpus. Especially, this article introduces concept information derived from high-quality lexical knowledge graph Probase, which leverages strong interpretability and anti-nose capability for the proposed model. Moreover, the proposed model captures not only intra-sentence word dependencies, but also temporal transitions between sentences and inter-sentence concept dependence. Experiments conducted on several NLP tasks validate the superiority of the proposed approach, which could effectively infer meaningful hierarchical concept structure of document and hierarchical multi-scale structures of sequences, even compared with latest state-of-the-art Transformer-based models.

A MapReduce implementation of posterior probability clustering and relevance models for recommendation

Relevance-Based Language Models are a formal probabilistic approach for explicitly introducing the concept of relevance in the Statistical Language Modelling framework. Recently, they have been determined as a very effective way of computing recommendations. When combining this new recommendation approach with Posterior Probabilistic Clustering for computing neighbourhoods, the item ranking is further improved, radically surpassing rating prediction recommendation techniques. Nevertheless, in the current landscape where the number of recommendation scenarios reaching the big data scale is increasing day after day, high figures of effectiveness are not enough. In this paper, we address one urging and common need of recommendation systems which is algorithm scalability. Particularly, we adapted those highly effective algorithms to the functional MapReduce paradigm, that has been previously proved as an adequate tool for enabling recommenders scalability. We evaluated the performance of our approach under realistic circumstances, showing a good scalability behaviour on the number of nodes in the MapReduce cluster. Additionally, as a result of being able to execute our algorithms distributively, we can show measures in a much bigger collection supporting the results presented on the seminal paper.

Enhancing Text Using Emotion Detected from EEG Signals

Often people might not be able to express themselves properly on social media, like not being able to think of appropriate words representative of their emotional state. In this paper, we propose an end to end system which aims to enhance user-input sentence according to his/her current emotional state. It works by a) detecting the emotion of the user and b) enhancing the input sentence by inserting emotive words to make the sentence more representative of the emotional state of the user. The emotional state of the user is recognized by analyzing the Electroencephalogram (EEG) signals from the brain. For text enhancement, we modify the words corresponding to the detected emotion using correlation finder scheme. Next, the verification of the sentence correctness has been performed using Long Short Term Memory (LSTM) Networks based Language Modeling framework. An accuracy of 74.95% has been recorded for the classification of five emotional states in a dataset of 25 participants using EEG signals. Similarly, promising results have been obtained for the task text enhancement and overall end-to-end system. To the best of our knowledge, this work is the first of its kind to enhance text according to the emotional state detected by EEG brainwaves. The system also releases an individual from thinking and typing words, which might be a complicated procedure sometimes.

A language model-based framework for multi-publisher content-based recommender systems

The rapid growth of the Web has increased the difficulty of finding the information that can address the users’ information needs. A number of recommendation approaches have been developed to tackle this problem. The increase in the number of data providers has necessitated the development of multi-publisher recommender systems; systems that include more than one item/data provider. In such environments, preserving the privacy of both publishers and subscribers is a key and challenging point. In this paper, we propose a multi-publisher framework for recommender systems based on a client–server architecture, which preserves the privacy of both data providers and subscribers. We develop our framework as a content-based filtering system using the statistical language modeling framework. We also introduce AUTO, a simple yet effective threshold optimization algorithm, to find a dissemination threshold for making acceptance and rejection decisions for new published documents. We further propose a language model sketching technique to reduce the network traffic between servers and clients in the proposed framework. Extensive experiments using the TREC-9 Filtering Track and the CLEF 2008-09 INFILE Track collections indicate the effectiveness of the proposed models in both single- and multi-publisher settings.

A Position-Aware Language Modeling Framework for Extractive Broadcast News Speech Summarization

Extractive summarization, a process that automatically picks exemplary sentences from a text (or spoken) document with the goal of concisely conveying key information therein, has seen a surge of attention from scholars and practitioners recently. Using a language modeling (LM) approach for sentence selection has been proven effective for performing unsupervised extractive summarization. However, one of the major difficulties facing the LM approach is to model sentences and estimate their parameters more accurately for each text (or spoken) document. We extend this line of research and make the following contributions in this work. First, we propose a position-aware language modeling framework using various granularities of position-specific information to better estimate the sentence models involved in the summarization process. Second, we explore disparate ways to integrate the positional cues into relevance models through a pseudo-relevance feedback procedure. Third, we extensively evaluate various models originated from our proposed framework and several well-established unsupervised methods. Empirical evaluation conducted on a broadcast news summarization task further demonstrates performance merits of the proposed summarization methods.

Parallel Sentiment Analysis with Storm

In the era of big data, huge volumes of data are generated from online social networks, sensor networks, mobile devices, and organizations’ enterprise systems. This phenomenon provides organizations with unprecedented opportunities to tap into big data to mine valuable business intelligence. However, traditional business analytics methods may not be able to cope with the flood of big data. The main contribution of this paper is the illustration of the development of a novel big data analytics framework named ASMF that leverages a probabilistic language model to analyze the consumer sentiments embedded in hundreds of millions of online product reviews. In particular, an inference model is embedded into the classical language modeling framework to enhance the prediction of consumer sentiments. The practical implication of our research work is that organizations can apply our big data analytics framework to analyze consumers’ product preferences, and hence develop more effective marketing and production strategies.

A hybrid input-type recurrent neural network for LVCSR language modeling

Substantial amounts of resources are usually required to robustly develop a language model for an open vocabulary speech recognition system as out-of-vocabulary (OOV) words can hurt recognition accuracy. In this work, we applied a hybrid lexicon of word and sub-word units to resolve the problem of OOV words in a resource-efficient way. As sub-lexical units can be combined to form new words, a compact set of hybrid vocabulary can be used while still maintaining a low OOV rate. For Thai, a syllable-based unit called pseudo-morpheme (PM) was chosen as a sub-word unit. To also benefit from different levels of linguistic information embedded in different input types, a hybrid recurrent neural network language model (RNNLM) framework is proposed. An RNNLM can model not only information from multiple-type input units through a hybrid input vector of words and PMs, but can also capture long context history through recurrent connections. Several hybrid input representations were also explored to optimize both recognition accuracy and computational time. The hybrid LM has shown to be both resource-efficient and well-performed on two Thai LVCSR tasks: broadcast news transcription and speech-to-speech translation. The proposed hybrid lexicon can constitute an open vocabulary for Thai LVCSR as it can greatly reduce the OOV rate to less than 1 % while using only 42 % of the vocabulary size of the word-based lexicon. In terms of recognition performance, the best proposed hybrid RNNLM, which uses a mixed word-PM input, obtained 1.54 % relative WER reduction when compared with a conventional word-based RNNLM. In terms of computational time, the best hybrid RNNLM has the lowest training and decoding time among all RNNLMs including the word-based RNNLM. The overall relative reduction on WER of the proposed hybrid RNNLM over a traditional n-gram model is 6.91 %.

Exploring the use of unsupervised query modeling techniques for speech recognition and summarization

Statistical language modeling (LM) that intends to quantify the acceptability of a given piece of text has long been an interesting yet challenging research area. In particular, language modeling for information retrieval (IR) has enjoyed remarkable empirical success; one emerging stream of the LM approach for IR is to employ the pseudo-relevance feedback process to enhance the representation of an input query so as to improve retrieval effectiveness. This paper presents a continuation of such a general line of research and the major contributions are three-fold. First, we propose a principled framework which can unify the relationships among several widely-cited query modeling formulations. Second, on top of this successfully developed framework, two extensions have been proposed. On one hand, we present an extended query modeling formulation by incorporating critical query-specific information cues to guide the model estimation. On the other hand, a word-based relevance modeling has also been leveraged to overcome the obstacle of time-consuming model estimation when the framework is being utilized for practical applications. In addition, we further adopt and formalize such a framework to the speech recognition and summarization tasks. A series of experiments reveal the empirical potential of such an LM framework and the performance merits of the deduced models on these two tasks.

Mining document, concept, and term associations for effective biomedical retrieval: introducing MeSH-enhanced retrieval models

Manually assigned subject terms, such as Medical Subject Headings (MeSH) in the health domain, describe the concepts or topics of a document. Existing information retrieval models do not take full advantage of such information. In this paper, we propose two MeSH-enhanced (ME) retrieval models that integrate the concept layer (i.e. MeSH) into the language modeling framework to improve retrieval performance. The new models quantify associations between documents and their assigned concepts to construct conceptual representations for the documents, and mine associations between concepts and terms to construct generative concept models. The two ME models reconstruct two essential estimation processes of the relevance model (Lavrenko and Croft 2001) by incorporating the document-concept and the concept-term associations. More specifically, in Model 1, language models of the pseudo-feedback documents are enriched by their assigned concepts. In Model 2, concepts that are related to users' queries are first identified, and then used to reweight the pseudo-feedback documents according to the document-concept associations. Experiments carried out on two standard test collections show that the ME models outperformed the query likelihood model, the relevance model (RM3), and an earlier ME model. A detailed case analysis provides insight into how and why the new models improve/worsen retrieval performance. Implications and limitations of the study are discussed. This study provides new ways to formally incorporate semantic annotations, such as subject terms, into retrieval models. The findings of this study suggest that integrating the concept layer into retrieval models can further improve the performance over the current state-of-the-art models.