Large Text Datasets Research Articles

Evaluation of a Large Language Model on the American Academy of Pediatrics' PREP Emergency Medicine Question Bank.

Large language models (LLMs), including ChatGPT (Chat Generative Pretrained Transformer), a popular, publicly available LLM, represent an important innovation in the application of artificial intelligence. These systems generate relevant content by identifying patterns in large text datasets based on user input across various topics. We sought to evaluate the performance of ChatGPT in practice test questions designed to assess knowledge competency for pediatric emergency medicine (PEM). We evaluated the performance of ChatGPT for PEM board certification using a popular question bank used for board certification in PEM published between 2022 and 2024. Clinicians assessed performance of ChatGPT by inputting prompts and recording the software's responses, asking each question over 3 separate iterations. We calculated correct answer percentages (defined as correct in at least 2/3 iterations) and assessed for agreement between the iterations using Fleiss' κ. We included 215 questions over the 3 study years. ChatGPT responded correctly to 161 of PREP EM questions over 3 years (74.5%; 95% confidence interval, 68.5%-80.5%), which was similar within each study year (75.0%, 71.8%, and 77.8% for study years 2022, 2023, and 2024, respectively). Among correct responses, most were answered correctly on all 3 iterations (137/161, 85.1%). Performance varied by topic, with the highest scores in research and medical specialties and lower in procedures and toxicology. Fleiss' κ across the 3 iterations was 0.71, indicating substantial agreement. ChatGPT provided correct answers to PEM responses in three-quarters of cases, over the recommended minimum of 65% provided by the question publisher for passing. Responses by ChatGPT included detailed explanations, suggesting potential use for medical education. We identified limitations in specific topics and image interpretation. These results demonstrate opportunities for LLMs to enhance both the education and clinical practice of PEM.

Pemanfaatan Natural Language Processing Untuk Pengecekan Ejaan Sesuai KBBI

Spelling mistakes in the Indonesian language often occur due to lack of knowledge of spelling rules, the influence of foreign languages, and the use of informal language. This research aims to develop a Natural Language Processing (NLP) based spelling checking system that refers to the Dictionary of Indonesian Language (KBBI). The system is expected to be able to perform automatic spelling verification and correction with high accuracy. The method used is qualitative with an analytic descriptive approach, involving data collection from the KBBI and processing using Python. The research stages include collection, data cleaning, tokenization, and data analysis. The system was trained using a large Indonesian text dataset. The results showed the system achieved a spelling check accuracy of 81.64%, more accurate, easy to use, flexible, and adaptive than conventional systems. The number of words in the text affects the checking time, with longer text taking longer.

Dynamic Question Generation using NER with various Feature Extraction and NLP Techniques

The purpose of the Automatic Question Generator is to produce novel questions from the given text that are both linguistically natural, semantically precise, and syntactically coherent. Unlike activities such as summarization and paraphrase, replies play a vital role in question-answering tasks. The creation of multiple-choice questions involves the use of distractions of high quality and the formulation of successful questions. This technology enables instructors to generate multiple-choice assessment questions including both right answers and distractors. An instructor may promptly evaluate a student's understanding of the material. This method enables students to assess their own comprehension level of the topic matter. This technique is beneficial for generating assessment papers for evaluation in the educational industry. Teachers may develop questions about their topics by duplicating or transferring one or more paragraphs. In this paper we proposed an dynamic question generation using Named Entity Recognition (NER), Natural Language Processing (NLP) based approach with conventional learning techniques. The proposed NLP-NER based approach is applicable on heterogeneous as well as large text dataset. However, a compilation of inquiries is generated, derived from the texts that were identified as noteworthy or informative. Various methods of inquiry often yield questions that are primarily focused on gathering factual information, such as inquiries about individuals, time, location, reasons, and details. A natural language processing programme facilitates the comprehension of language and spoken communication by machines. The approach dissects the material into its constituent elements, deciphers the meaning of the language, selects the appropriate actions, and ultimately delivers the content to the user in a comprehensible language.

Text Summarization Based on Semantic Similarity

In the contemporary information age, the sheer volume of textual data poses a significant challenge for efficient comprehension and utilization. This project endeavors to address this challenge by developing a Text Summarization System grounded in semantic similarities. The primary goal is to create a robust and intuitive tool that extracts key information from large textual datasets, offering users a concise and meaningful summary. The proposed system employs advanced Natural Language Processing (NLP) techniques to analyze the semantic relationships within the text. Rather than relying solely on syntactic structures, the model identifies and leverages semantic similarities, such as shared concepts, themes, and contextual relationships, to distill the essential content. This approach enhances the summarization process by ensuring that the generated summaries reflect a deeper understanding of the underlying semantics, thereby capturing the core meaning of the text. Throughout the development of this project, the B.Tech student will delve into the intricacies of semantic analysis, exploring techniques to recognize and prioritize key concepts. The system's effectiveness will be evaluated through rigorous testing on diverse textual datasets, assessing its ability to generate coherent and relevant summaries across various domains. This project not only contributes to the field of NLP but also has practical applications in information retrieval, document summarization, and content curation. By providing an innovative solution to the challenges of information overload, the Text Summarization System based on semantic similarities offers a valuable tool for enhancing efficiency in information processing and decision-making. Index terms Text Summarization, Semantic Similarities, Natural Language Processing (NLP), Semantic Analysis, Information Retrieval, Document Summarization, Content Curation, Information Overload, Decision Making, Textual Data Analysis, Key Concept Recognition, Conceptual Relationships, Syntactic Structures, Semantic Understanding, Textual Datasets Evalution.

Enabling action crossmodality for a pretrained large language model

Natural language processing and vision tasks have recently seen large improvements through the rise of Transformer architectures. The high-performing large language models (LLMs) benefit from large textual datasets that are numerously available online. However, action and bidirectional action-language tasks are less developed, as these require more specific and labeled data. Therefore, we aim at enabling these robotic action capabilities for a pretrained LLM, while maintaining high efficiency with regards to the required training time and data size. To achieve this, we split up a Transformer-based LLM and insert a multimodal architecture into it. Specifically, we split a pretrained T5 LLM between its encoder and decoder parts, to insert a crossmodal Transformer component of a Paired Transformed Autoencoders (PTAE) bidirectional action-language model. The experiments are conducted on a new dataset, consisting of unimodal language translation and crossmodal bidirectional action-language translation. The natural language capabilities of the original T5 are re-established efficiently by training the crossmodal Transformer, which requires only one 5.7 millionth of the T5 model’s original training data. Furthermore, the new model, called CrossT5, achieves high accuracy for the vision- and language-guided robotic action tasks. By design, the CrossT5 agent acts robustly when tested with language commands not included in the dataset. The results demonstrate that this novel approach is successful in combining the advanced linguistic capabilities of LLMs with the low-level robotic control skills of vision-action models. The code is available at this URL: https://github.com/samsoneko/CrossT5.

A systematic evaluation of text mining methods for short texts: Mapping individuals’ internal states from online posts

Short texts generated by individuals in online environments can provide social and behavioral scientists with rich insights into these individuals’ internal states. Trained manual coders can reliably interpret expressions of such internal states in text. However, manual coding imposes restrictions on the number of texts that can be analyzed, limiting our ability to extract insights from large-scale textual data. We evaluate the performance of several automatic text analysis methods in approximating trained human coders’ evaluations across four coding tasks encompassing expressions of motives, norms, emotions, and stances. Our findings suggest that commonly used dictionaries, although performing well in identifying infrequent categories, generate false positives too frequently compared to other methods. We show that large language models trained on manually coded data yield the highest performance across all case studies. However, there are also instances where simpler methods show almost equal performance. Additionally, we evaluate the effectiveness of cutting-edge generative language models like GPT-4 in coding texts for internal states with the help of short instructions (so-called zero-shot classification). While promising, these models fall short of the performance of models trained on manually analyzed data. We discuss the strengths and weaknesses of various models and explore the trade-offs between model complexity and performance in different applications. Our work informs social and behavioral scientists of the challenges associated with text mining of large textual datasets, while providing best-practice recommendations.

Enhancing Document Clustering with Hybrid Recurrent Neural Networks and Autoencoders: A Robust Approach for Effective Semantic Organization of Large Textual Datasets

This research presents an innovative document clustering method that uses recurrent neural networks (RNNs) and autoencoders. RNNs capture sequential dependencies while autoencoders improve feature representation. The hybrid model, tested on different datasets (20-Newsgroup, Reuters, BBC Sports), outperforms traditional clustering, revealing semantic relationships and robustness to noise. Preprocessing includes denoising techniques (stemming, lemmatization, tokenization, stopword removal) to ensure a refined data set. Evaluation metrics (adjusted randomness evaluation, normalized mutual information evaluation, completeness evaluation, homogeneity evaluation, V-measure, accuracy) validate the effectiveness of the model and provide a powerful solution for organizing and understanding large text datasets.

Taking the Road Less Travelled: How Corpus‐Assisted Discourse Studies Can Enrich Qualitative Explorations of Large Textual Datasets

AbstractHow might interpretivist qualitative researchers tackle large data sets consisting of millions or even billions of words? Corpus‐assisted discourse studies (CADS) is the approach we explore here. Specifically designed for the analysis of voluminous textual data, it offers a recognized empirical approach for making sense of such data. But it does so within an epistemology that understands language to be central in shaping our understanding of the world around us, so that CADS can assist researchers in revealing the social dynamics of the text – including the ideology and power that is latent in many such corpora. Bringing together the training of applied linguists and a management scholar, we discuss the background to CADS and its differences from text‐mining approaches such as topic modelling, which have been more widely used in management studies to date. Focusing on the needs of people who are new to the approach, we then offer a worked example to show CADS’ potential in exploring a management‐related corpus. Our paper concludes with a discussion of the strengths and weaknesses of the approach and its potential for future discursively orientated management research – especially in the context of the rise of ‘big data’.

Investigating the Key Aspects of a Smart City through Topic Modeling and Thematic Analysis

In recent years, the emergence of the smart city concept has garnered attention as a promising innovation aimed at addressing the multifactorial challenges arising from the concurrent trends of urban population growth and the climate crisis. In this study, we delve into the multifaceted dimensions of the smart city paradigm to unveil its underlying structure, employing a combination of quantitative and qualitative techniques. To achieve this, we collected textual data from three sources: scientific publication abstracts, news blog posts, and social media entries. For the analysis of this textual data, we introduce an innovative semi-automated methodology that integrates topic modeling and thematic analysis. Our findings highlight the intricate nature of the smart city domain, which necessitates examination from three perspectives: applications, technology, and socio-economic perspective. Through our analysis, we identified ten distinct aspects of the smart city paradigm, encompassing mobility, energy, infrastructure, environment, IoT, data, business, planning and administration, security, and people. When comparing the outcomes across the three diverse datasets, we noted a relative lack of attention within the scientific community towards certain aspects, notably in the realm of business, as well as themes relevant to citizens’ everyday lives, such as food, shopping, and green spaces. This work reveals the underlying thematic structure of the smart city concept to help researchers, practitioners, and public administrators participate effectively in smart city transformation initiatives. Furthermore, it introduces a novel data-driven method for conducting thematic analysis on large text datasets.

Exploring Key Issues in Cybersecurity Data Breaches: Analyzing Data Breach Litigation with ML-Based Text Analytics

While data breaches are a frequent and universal phenomenon, the characteristics and dimensions of data breaches are unexplored. In this novel exploratory research, we apply machine learning (ML) and text analytics to a comprehensive collection of data breach litigation cases to extract insights from the narratives contained within these cases. Our analysis shows stakeholders (e.g., litigants) are concerned about major topics related to identity theft, hacker, negligence, FCRA (Fair Credit Reporting Act), cybersecurity, insurance, phone device, TCPA (Telephone Consumer Protection Act), credit card, merchant, privacy, and others. The topics fall into four major clusters: “phone scams”, “cybersecurity”, “identity theft”, and “business data breach”. By utilizing ML, text analytics, and descriptive data visualizations, our study serves as a foundational piece for comprehensively analyzing large textual datasets. The findings hold significant implications for both researchers and practitioners in cybersecurity, especially those grappling with the challenges of data breaches.

Applying machine-learning to rapidly analyze large qualitative text datasets to inform the COVID-19 pandemic response: comparing human and machine-assisted topic analysis techniques.

Machine-assisted topic analysis (MATA) uses artificial intelligence methods to help qualitative researchers analyze large datasets. This is useful for researchers to rapidly update healthcare interventions during changing healthcare contexts, such as a pandemic. We examined the potential to support healthcare interventions by comparing MATA with "human-only" thematic analysis techniques on the same dataset (1,472 user responses from a COVID-19 behavioral intervention). In MATA, an unsupervised topic-modeling approach identified latent topics in the text, from which researchers identified broad themes. In human-only codebook analysis, researchers developed an initial codebook based on previous research that was applied to the dataset by the team, who met regularly to discuss and refine the codes. Formal triangulation using a "convergence coding matrix" compared findings between methods, categorizing them as "agreement", "complementary", "dissonant", or "silent". Human analysis took much longer than MATA (147.5 vs. 40 h). Both methods identified key themes about what users found helpful and unhelpful. Formal triangulation showed both sets of findings were highly similar. The formal triangulation showed high similarity between the findings. All MATA codes were classified as in agreement or complementary to the human themes. When findings differed slightly, this was due to human researcher interpretations or nuance from human-only analysis. Results produced by MATA were similar to human-only thematic analysis, with substantial time savings. For simple analyses that do not require an in-depth or subtle understanding of the data, MATA is a useful tool that can support qualitative researchers to interpret and analyze large datasets quickly. This approach can support intervention development and implementation, such as enabling rapid optimization during public health emergencies.

Leveraging Summary Guidance on Medical Report Summarization.

This study presents three deidentified large medical text datasets, named DISCHARGE, ECHO and RADIOLOGY, which contain 50 K, 16 K and 378 K pairs of report and summary that are derived from MIMIC-III, respectively. We implement convincing baselines of automated abstractive summarization on the created datasets with pre-trained encoder-decoder language models, including BERT2BERT, BERTShare, RoBERTaShare, Pegasus, ProphetNet, T5-large, BART and GSUM. Further, based on the BART model, we leverage the sampled summaries from the training set as prior knowledge guidance, for encoding additional contextual representations of the guidance with the encoder and enhancing the decoding representations in the decoder. The experimental results confirm the improvement of ROUGE scores and BERTScore made by the proposed method.

Verbs Are Associated With Agency

Abstract. Agency and communion comprise the big two of social perception, with agency in general denoting goal orientation and communion orientation toward others. Both dimensions can be coded and detected in language through agentic and communal words, however, only agency through lexical classes, namely verbs. This research examines whether the link between semantic and lexical agency occurs in natural language, as past research was limited to pseudowords or single words. Using existing databases, Study 1 found that people rated verbs as more agentic than nouns. Using large textual datasets, Study 2 found verb use to be positively related to agentic – but not communal word – use. These findings contribute to the identification of agency in language and indicate its importance in social cognition.

Intelligent analysis system of college students' employment and entrepreneurship situation: Big data and artificial intelligence-driven approach

This study proposes an analysis plan for college students' employment and entrepreneurship based on big data and artificial intelligence, aiming to provide an important basis for the development of employment guidance activities. To realize feature analysis and useful information extraction of large-scale text information data, this study adopts a big data mining algorithm based on fuzzy hierarchical clustering analysis and semantic similarity correlation feature extraction. The algorithm uses generalization mapping to construct a semantic concept tree, and a fuzzy analytic hierarchy process to judge the semantic similarity and relevance of big data. This study demonstrates the efficacy of a big data mining algorithm through experimental verification. Two public datasets were used to validate the proposed method. The experimental results revealed that the method could efficiently process and analyze large textual datasets, leading to superior data classification performance.

Dialect classification using acoustic and linguistic features in Arabic speech

<span lang="EN-US">Speech dialects refer to linguistic and pronunciation variations in the speech of the same language. Automatic dialect classification requires considerable acoustic and linguistic differences between different dialect categories of speech. This paper proposes a classification model composed of a combination of classifiers for the Arabic dialects by utilizing both the acoustic and linguistic features of spontaneous speech. The acoustic classification comprises of an ensemble of classifiers focusing on different frequency ranges within the short-term spectral features, as well as a classifier utilizing the ‘i-vector’, whilst the linguistic classifiers use features extracted by transformer models pre-trained on large Arabic text datasets. It has been shown that the proposed fusion of multiple classifiers achieves a classification accuracy of 82.44% for the identification task of five Arabic dialects. This represents the highest accuracy reported on the dataset, despite the relative simplicity of the proposed model, and has shown its applicability and relevance for dialect identification tasks. </span>

Deep Learning based Text Abstraction

Abstract: Text abstraction plays a vital role in extracting crucial information from large textual datasets. With the advancements in deep learning, the application of transformer models has shown great potential in the field of text abstraction. This technical paper presents a comprehensive study on "Deep Learning-Based Text Abstraction" with a specific focus on utilizing transformer models for effective abstraction. The paper begins by providing an overview of text abstraction and its significance in handling extensive amounts of textual data while preserving essential information. It then delves into the fundamentals of transformer models, explaining their architecture and mechanisms, particularly attention mechanisms that enable capturing contextual relationships within text. The main contribution of this paper lies in the implementation of text abstraction using transformer models. It discusses the utilization of pre-trained transformer models and their adaptation for text abstraction tasks. The paper elaborates on techniques like fine-tuning and transfer learning to optimize the transformer models for text abstraction. Furthermore, the paper presents a detailed experimental setup, including the selection of datasets, evaluation metrics, and training procedures. It discusses the performance evaluation of the implemented transformer-based text abstraction system, comparing it against existing techniques and benchmark datasets. The results and analysis section showcases the effectiveness and efficiency of the proposed approach, highlighting improvements achieved in terms of extraction accuracy, summarization quality, and computational efficiency. Finally, the paper concludes by outlining potential areas for future research and development in deep learning-based text abstraction using transformer models. Overall, this technical paper provides a comprehensive study on deep learning-based text abstraction, with a focus on transformer models. It serves as a valuable resource for researchers and practitioners interested in leveraging transformers for efficient and accurate text abstraction, opening avenues for advancements in natural language processing and information extraction.

Automatized spatio-temporal detection of drought impacts from newspaper articles using natural language processing and machine learning

Droughts are expected to increase both in terms of frequency and magnitude across Europe. Despite the multitude of adverse effects these disasters impose on social-ecological systems, most impact assessments are constrained to single event and/or single sector analyses. Furthermore, existing longitudinal multi-sectoral datasets are limited in spatiotemporal homogeneity and scope, resulting in fragmented datasets. To address this gap, we propose a novel method for the automatized detection of drought impacts based on newspaper articles. We employ natural language processing (NLP) and machine learning to identify different socio-economic impacts (e.g. agriculture, forestry, livestock, fires) and their geographic and temporal scope from 40,000 newspaper articles reporting about droughts in Germany between 2000 and 2021. Our method is able to track impacts over long time periods, allowing us to assess how drought impacts evolve. Accuracy levels of 92–96% per impact class were obtained for the automatic classification of the impacts when evaluated on a human-annotated dataset. Furthermore, our resulting impact dataset can replicate both temporal and spatial trends when validated against independent impact and hazard data. Overall, the proposed approach advances current research as it (1) requires a significantly lower workload than conventional impact assessment methods, (2) allows addressing large text datasets, (3) reduces subjectivity and human bias, (4) is generalizable to other hazard types as well as text corpora, and (5) achieves sufficient levels of accuracy. The findings highlight the applicability of NLP and machine learning to create comprehensive longitudinal impact datasets.

N/A Modern Approaches in Natural Language Processing

Natural Language Processing (NLP) is one of the major branches in the emerging field of Artificial Intelligence (AI). Classical approaches in this area were mostly based on parsing and information extraction techniques, which suffered from great difficulty when dealing with very large textual datasets available in practical applications. This issue can potentially be addressed with the recent advancement of the Deep Learning (DL) techniques, which are naturally assuming very large datasets for training. In fact, NLP research has witnessed a remarkable achievement with the introduction of Word Embedding techniques, which allows a document to be represented meaningfully as a matrix, on which major DL models like CNN or RNN can be deployed effectively to accomplish common NLP tasks. Gradually, NLP scholars keep developing specific models for their areas, notably attention-enhanced BiLSTM, Transformer and BERT. The births of those models have introduced a new wave of modern approaches which frequently report new breaking results and open much novel research directions. The aim of this paper is to give readers a roadmap of those modern approaches in NLP, including their ideas, theories and applications. This would hopefully offer a solid background for further research in this area.

Central Kurdish Automatic Speech Recognition using Deep Learning

Automatic Speech Recognition (ASR) as an interesting field of speech processing, is nowadays utilized in real applications which are implemented using various techniques. Amongst them, the artificial neural network is the most popular one. Increasing the performance and making these systems robust to noise are among the current challenges. This paper addresses the development of an ASR system for the Central Kurdish language (CKB) using a transfer learning of Deep Neural Networks (DNN). The combination of Mel-Frequency Cepstral Coefficients (MFCCs) for extracting features of speech signals, Long Short-Term Memory (LSTM) with Connectionist Temporal Classification (CTC) output layer is used to create an Acoustic Model (AM) on the AsoSoft CKB speech dataset. Also, we have used the N-gram language model on the collected large text dataset which includes about 300 million tokens. The text corpus is also used to extract a dynamic lexicon model that contains over 2.5 million CKB words. The obtained results show that the use of a DNN improves the results compared to classical statistics modules. The proposed method achieves a 0.22%-word error rate by combining transfer learning and language model adaptation. This result is superior to the best-reported result for the CKB.

Content Deduplication with Granularity Tweak Based on Base and Deviation for Large Text Dataset

The concept of storage optimization has evolved as one of the hottest research projects in big data which brings out better solutions such as data compression which almost converges towards the deduplication technique. Deduplication is a technique that finds and eliminates duplicate content by storing only the unique copies of data whose efficiency is being qualified based on the amount of duplicate content that they hideout from the data source. The deduplication technique is a well-established storage optimization technique, so in the due course of time, various tweaks have been provided for its betterment, but it quite has some limitations that it cannot determine the tiny changes that occur among similar contents, and the chunks which are generated by segmenting and hashing the data are more sensitive to changes which produce a new chunk for every small change which ruins the concept of storage optimization, so to tackle this, content deduplication with granularity tweak (CDGT) in the Hadoop architecture has been proposed for large text datasets. The CDGT aims to improve the efficiency of deduplication by utilizing the Reed Solomon technique. This pumps out more duplicate content by verifying both intracontent and intercontent as consequence performance enhancements are met, and this system incorporates cluster-based indexing to reduce the time involved in data management activities.