Large-scale Natural Language Research Articles

West African Pidgin: World Language Against the Grain

West African Pidgin (“Pidgin”) is a cluster of related, mutually intelligible, restructured Englishes with up to 140 million speakers in Nigeria, Cameroon, Ghana, Sierra Leone, Equatorial Guinea, and The Gambia. Spoken by just few thousand people two centuries ago, “modernisation” and “shallow social entrenchment” have driven the transformation of Pidgin into a “super-central” world language. Demographic growth, migration, the expansion of West African cultural industries and economies, and people-to-people contacts are likely to expand Pidgin further. Already the largest language of West Africa, Pidgin may be spoken by 400 million people by 2100. The rise of Pidgin goes against the grain. World languages like English, French, Chinese, or Arabic mostly spread through colonisation, elite engineering, and state intervention. The trajectory of Pidgin, therefore, holds great potential for exploring the dynamics of large-scale natural language evolution in the twenty-first century.

A Comparative Analysis of Chinese-English Translation Quality Based on ChatGPT: A Case Study of Chinese Characteristic Words

The release of ChatGPT, a large-scale natural language interactive program, at the end of 2022 has once again garnered worldwide attention to artificial intelligence technology. As a deep learning-based natural language processing model, ChatGPT demonstrates unique advantages in the field of translation that are unparalleled by previous machine translation software. Chinese characteristic words, which lack direct equivalents in other cultures or countries, have always posed a challenge in Chinese-English translation, but they are also an essential part of effectively conveying the Chinese story. This article selected some Chinese characteristic words mentioned in the Chinese Government Work Report as translation materials and asked ChatGPT with completing tasks such as translating and polishing the translation according to instructions. By comparing the official Chinese-English translation version and traditional machine translation version, this article analyzed the translation quality of ChatGPT in C-E translation from the perspectives of syntactic structure, coherence, and domain adjustment, with the aim of summarizing machine-assisted translation methods and translation strategies for Chinese characteristic words.

Improving short text classification with augmented data using GPT-3

Abstract GPT-3 is a large-scale natural language model developed by OpenAI that can perform many different tasks, including topic classification. Although researchers claim that it requires only a small number of in-context examples to learn a task, in practice GPT-3 requires these training examples to be either of exceptional quality or a higher quantity than easily created by hand. To address this issue, this study teaches GPT-3 to classify whether a question is related to data science by augmenting a small training set with additional examples generated by GPT-3 itself. This study compares two augmented classifiers: the Classification Endpoint with an increased training set size and the Completion Endpoint with an augmented prompt optimized using a genetic algorithm. We find that data augmentation significantly increases the accuracy of both classifiers, and that the embedding-based Classification Endpoint achieves the best accuracy of about 76%, compared to human accuracy of 85%. In this way, giving large language models like GPT-3 the ability to propose their own training examples can improve short text classification performance.

Effects of Target Age and Genre on Morphological Complexity in Children’s Reading Material

ABSTRACT Purpose Morphological regularities are an important feature of the English writing system, and exposure to written morphology may be key in the development of skilled word recognition. Our aim was to investigate children’s experiences of written morphology by analyzing a large-scale corpus of children’s reading materials spanning a target age range from 5 to 14 years. Method Analysis was based on the Oxford Children’s Reading Corpus. We examined frequency distributions of derived and compound words by target age and genre, as well as type and token frequencies for individual derivational suffixes. Results We found that the proportion of morphologically complex words – and derived words particularly – increased in line with target age, and that nonfiction contained more complex words than fiction. Frequencies of individual suffixes also varied by target age and genre, with Germanic forms more common in fiction and texts for younger children, and Latinate forms more common in nonfiction and texts for older children. Conclusion These findings provide a comprehensive picture of how children’s experience with written morphology changes over the course of reading development. We discuss these findings in the context of developmental changes in morphological processing, and the benefits and limitations of using large-scale natural language datasets.

Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults.

The clinical narrative in electronic health records (EHRs) carries valuable information for predictive analytics; however, its free-text form is difficult to mine and analyze for clinical decision support (CDS). Large-scale clinical natural language processing (NLP) pipelines have focused on data warehouse applications for retrospective research efforts. There remains a paucity of evidence for implementing NLP pipelines at the bedside for health care delivery. We aimed to detail a hospital-wide, operational pipeline to implement a real-time NLP-driven CDS tool and describe a protocol for an implementation framework with a user-centered design of the CDS tool. The pipeline integrated a previously trained open-source convolutional neural network model for screening opioid misuse that leveraged EHR notes mapped to standardized medical vocabularies in the Unified Medical Language System. A sample of 100 adult encounters were reviewed by a physician informaticist for silent testing of the deep learning algorithm before deployment. An end user interview survey was developed to examine the user acceptability of a best practice alert (BPA) to provide the screening results with recommendations. The planned implementation also included a human-centered design with user feedback on the BPA, an implementation framework with cost-effectiveness, and a noninferiority patient outcome analysis plan. The pipeline was a reproducible workflow with a shared pseudocode for a cloud service to ingest, process, and store clinical notes as Health Level 7 messages from a major EHR vendor in an elastic cloud computing environment. Feature engineering of the notes used an open-source NLP engine, and the features were fed into the deep learning algorithm, with the results returned as a BPA in the EHR. On-site silent testing of the deep learning algorithm demonstrated a sensitivity of 93% (95% CI 66%-99%) and specificity of 92% (95% CI 84%-96%), similar to published validation studies. Before deployment, approvals were received across hospital committees for inpatient operations. Five interviews were conducted; they informed the development of an educational flyer and further modified the BPA to exclude certain patients and allow the refusal of recommendations. The longest delay in pipeline development was because of cybersecurity approvals, especially because of the exchange of protected health information between the Microsoft (Microsoft Corp) and Epic (Epic Systems Corp) cloud vendors. In silent testing, the resultant pipeline provided a BPA to the bedside within minutes of a provider entering a note in the EHR. The components of the real-time NLP pipeline were detailed with open-source tools and pseudocode for other health systems to benchmark. The deployment of medical artificial intelligence systems in routine clinical care presents an important yet unfulfilled opportunity, and our protocol aimed to close the gap in the implementation of artificial intelligence-driven CDS. ClinicalTrials.gov NCT05745480; https://www.clinicaltrials.gov/ct2/show/NCT05745480.

EDITORIAL: ChatGPT, the Blade in Scientific Writing

One impact of the industrial revolution is the development of computer systems, including the birth of ChatGPT. ChatGPT (Generative Pre-trained Transformer) is a large-scale natural language processing model developed by OpenAI to generate text similar to human language. The model is trained on massive textual data and can produce diverse text output, including chatbot responses, article summaries, concept explanations, and more. The ability of ChatGPT influences the way of writing, including scientific writing.

Large-scale photonic natural language processing

Modern machine-learning applications require huge artificial networks demanding computational power and memory. Light-based platforms promise ultrafast and energy-efficient hardware, which may help realize next-generation data processing devices. However, current photonic networks are limited by the number of input-output nodes that can be processed in a single shot. This restricted network capacity prevents their application to relevant large-scale problems such as natural language processing. Here, we realize a photonic processor for supervised learning with a capacity exceeding 1.5 × 10 10 optical nodes, more than one order of magnitude larger than any previous implementation, which enables photonic large-scale text encoding and classification. By exploiting the full three-dimensional structure of the optical field propagating in free space, we overcome the interpolation threshold and reach the over-parameterized region of machine learning, a condition that allows high-performance sentiment analysis with a minimal fraction of training points. Our results provide a novel solution to scale up light-driven computing and open the route to photonic natural language processing.

Debiasing NLU Models via Causal Intervention and Counterfactual Reasoning

Recent studies have shown that strong Natural Language Understanding (NLU) models are prone to relying on annotation biases of the datasets as a shortcut, which goes against the underlying mechanisms of the task of interest. To reduce such biases, several recent works introduce debiasing methods to regularize the training process of targeted NLU models. In this paper, we provide a new perspective with causal inference to find out the bias. On one hand, we show that there is an unobserved confounder for the natural language utterances and their respective classes, leading to spurious correlations from training data. To remove such confounder, the backdoor adjustment with causal intervention is utilized to find the true causal effect, which makes the training process fundamentally different from the traditional likelihood estimation. On the other hand, in inference process, we formulate the bias as the direct causal effect and remove it by pursuing the indirect causal effect with counterfactual reasoning. We conduct experiments on large-scale natural language inference and fact verification benchmarks, evaluating on bias sensitive datasets that are specifically designed to assess the robustness of models against known biases in the training data. Experimental results show that our proposed debiasing framework outperforms previous state-of-the-art debiasing methods while maintaining the original in-distribution performance.

An Accurate Matching Query Method of Natural Language Knowledge Graph Based on Hierarchical Graph Topological Sequence

In recent years, although the application of knowledge graph in natural language processing has made some progress, there are still some key problems to be solved, especially the matching query problem in natural language knowledge graph. Since the basic data model of knowledge graph is graph, the matching query problem in natural language knowledge graph is usually transformed into graph matching query problem. However, at present, the traditional graph matching technology applied in knowledge graph consumes too much time and has low query efficiency, which cannot meet the needs of users for large-scale natural language knowledge graph query. Based on the full analysis of the defects of the traditional graph matching technology applied in the knowledge graph, according to the characteristics of the natural language knowledge graph, in order to improve the query efficiency, we propose an accurate matching query method of graph hierarchical topological sequence based on the graph model of knowledge graph. Through experiment analysis, compared with the traditional graph model matching algorithm applied in knowledge graph query, this method can quickly filter the unqualified knowledge graph candidate sets, effectively reduce the number of knowledge graph candidate sets, and make it have more advantages in matching efficiency and time performance. In addition, compared with two algorithms of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GIndex</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FG-Index</i> , this method has better performance in index construction time, average size of candidate set and average running time of online update.

A commentary of GPT-3 in MIT Technology Review 2021

Through the development of large-scale natural language models with writing and dialogue capabilities, artificial intelligence (AI) has taken a significant stride towards better natural language understanding (NLU) and human-computer interaction (HCI). As of today, the GPT-3 model, developed by OpenAI, is the language model with the most parameters, the largest scale, and the strongest capabilities. Using a large amount of Internet text data and thousands of books for model training, GPT-3 can imitate the natural language patterns of humans nearly perfectly. This language model is extremely realistic and is considered the most impressive model as of today.Despite its powerful modeling and description capabilities, there are significant issues and limitations. First and foremost, the GPT-3 model does not understand writing (natural language generation) well and sometimes generates uncontrollable content. Secondly, training the GPT-3 model requires a large amount of computing power, data, and capital investment, and releases significant carbon dioxide emissions. Developing similar models is only possible in laboratories with adequate resources. Furthermore, as the GPT-3 model is trained with Internet text data rife with error messages and prejudices, it often produces chapters and paragraphs with biased content similar to the training data.①①Original source in Chinese: M. Zhang, J. Li, GPT-3, Bulletin of National Natural Science Foundation of China. 35 (3) (2021) 403-406.

Unknown Oriented Programming: Mathematical Continuation

Natural language semantic engineering problems are faced with unknown input and intensive knowledge challenges. In order to adapt to the featuresof natural language semantic engineering, the AI programinglanguage needs to be expanded mathematically: 1) Using many ways to improve the spatial distribution and coverage of instances; 2) Keeping different abstract function versions running at the same time; 3) Providing a large numberof knowledge configuration files and supporting functions to deal with intensive knowledge problems; 4) Using the most possibilitypriority call to solve the problem of multiple running branchestraversal. This paper introduces the unknown oriented programming ideas, basic strategy formulation,language design and simulation running examples. It provides a new method for the incremental research and development of large-scale natural language semantic engineeringapplication. Finally, this paper summarizes the full text and puts forward the further research direction.

Predicting leadership perception with large-scale natural language data

We present a computational method for predicting, and identifying the correlates of, leadership perceptions for prominent individuals. Our approach proxies knowledge representations for these individuals using high-dimensional semantic vectors derived from large-scale news media datasets. It then applies machine learning techniques to build a model that maps these vectors onto participant ratings of leadership effectiveness. This method greatly outperforms other approaches and achieves accuracy rates comparable to human participants in predicting leadership effectiveness judgments. Crucially, it relies on attributes and associations identified by established theories of leadership perception—notably implicit leadership theories—as guiding lay leadership perception. Thus, our model appears to have learnt the same implicit leadership cues as our human participants. In addition, we show that our approach can be used to not only predict leadership effectiveness judgments, but also to identify dimensions that people associate with effective leadership, as well as quantify the extent of this association for each dimension. We illustrate the broad applicability of our method by using it to predict leadership perceptions for over 6000 individuals in the public sphere, and to algorithmically uncover the particular traits, concepts, and attributes that people most strongly associate with effective leaders.

Diverse climate actors show limited coordination in a large-scale text analysis of strategy documents

Networks of non-state actors and subnational governments have proliferated since the Paris Agreement formally recognized their contributions to global climate change governance. Understanding the ways these actors are taking action and how they align with each other and national governments is critical given the need for coordinated actions to achieve ambitious global climate goals. Here, we present a large analysis (n = 9,326), applying large-scale natural language processing methods and social network analysis to the climate strategy documents of countries, regions, cities and companies. We find that climate mitigation in employee travel and office operations, green building standards, and municipal and citizen actions are common themes in climate actions across companies and city and regional governments, whereas approaches to setting targets in specific sectors and emissions scopes are more diverse. We also find links between the strategies of regions and countries, whereas companies are disconnected. Gaps in climate action for most actors include adaptation and consumption/supply-chain emission reduction efforts. We suggest that although actors may appear to be self-organizing and allocating climate actions in a mutually beneficial and synergistic way, there may also be missed opportunities for deeper coordination that could result in more ambitious action.

Changes in Gender Stereotypes Over Time: A Computational Analysis

We combined established psychological measures with techniques in machine learning to measure changes in gender stereotypes over the course of the 20th century as expressed in large-scale historical natural language data. Although our analysis replicated robust gender biases previously documented in the literature, we found that the strength of these biases has diminished over time. This appears to be driven by changes in gender biases for stereotypically feminine traits (rather than stereotypically masculine traits) and changes in gender biases for personality-related traits (rather than physical traits). Our results illustrate the dynamic nature of stereotypes and show how recent advances in data science can be used to provide a long-term historical analysis of core psychological variables. In terms of practice, these findings may, albeit cautiously, suggest that women and men can be less constrained by prescriptions of feminine traits. Additional online materials for this article are available on PWQ’s website at 10.1177/0361684320977178

What are We Talking About? Natural Language Processing in Organizations

Text is a rich and naturally-occurring source of data throughout all organizations, but it is not trivial to analyze - it is high-dimensional, sparse, and context-dependent. Each presenter focuses on building scaleable and interpretable models of how people communicate inside organizations. Across all four examples in this symposium, natural language processing is employed to directly observe the contents of organizational communication, and suggest simple conversational interventions that might support the broader goals of the organization. The topic of each of the presentations concerns an interpersonal communication challenge of central interest to OB scholars, such as feedback, team work, and reputation management. Each presentation offers a novel application from a suite of modern advances in natural language processing, including open-source software implementations that can be used to replicate our methods in other domains. Each work considers the managerial implications of natural language processing by drawing on a distinct theoretical framework for the goals of communication in each domain. This allows our work, as a whole, to outline a scope of what is possible in future applications of these natural language processing in organizational research. Corporate Seers: Discursive Prescience and Firm Stock Returns Presenter: Paul Vicinanza; Stanford Graduate School of Business Predicting Leadership Perceptions with Representations from Large-Scale Natural Language Data Presenter: Sudeep Bhatia; U. of Pennsylvania Emotional Accommodation: Women and the Give-and-Take of Emotions in the Workplace Presenter: Sanaz Mobasseri; Boston U. Questrom School of Business Concreteness, Concretely: Advice for Advisors Presenter: Michael Yeomans; Harvard Business School

Is your document novel? Let attention guide you. An attention-based model for document-level novelty detection

AbstractDetecting, whether a document contains sufficient new information to be deemed as novel, is of immense significance in this age of data duplication. Existing techniques for document-level novelty detection mostly perform at the lexical level and are unable to address the semantic-level redundancy. These techniques usually rely on handcrafted features extracted from the documents in a rule-based or traditional feature-based machine learning setup. Here, we present an effective approach based on neural attention mechanism to detect document-level novelty without any manual feature engineering. We contend that the simple alignment of texts between the source and target document(s) could identify the state of novelty of a target document. Our deep neural architecture elicits inference knowledge from a large-scale natural language inference dataset, which proves crucial to the novelty detection task. Our approach is effective and outperforms the standard baselines and recent work on document-level novelty detection by a margin of $\sim$ 3% in terms of accuracy.

Feature-Frequency–Adaptive On-line Training for Fast and Accurate Natural Language Processing

Training speed and accuracy are two major concerns of large-scale natural language processing systems. Typically, we need to make a tradeoff between speed and accuracy. It is trivial to improve the training speed via sacrificing accuracy or to improve the accuracy via sacrificing speed. Nevertheless, it is nontrivial to improve the training speed and the accuracy at the same time, which is the target of this work. To reach this target, we present a new training method, feature-frequency–adaptive on-line training, for fast and accurate training of natural language processing systems. It is based on the core idea that higher frequency features should have a learning rate that decays faster. Theoretical analysis shows that the proposed method is convergent with a fast convergence rate. Experiments are conducted based on well-known benchmark tasks, including named entity recognition, word segmentation, phrase chunking, and sentiment analysis. These tasks consist of three structured classification tasks and one non-structured classification task, with binary features and real-valued features, respectively. Experimental results demonstrate that the proposed method is faster and at the same time more accurate than existing methods, achieving state-of-the-art scores on the tasks with different characteristics.

Online Classification Using a Voted RDA Method

We propose a voted dual averaging method for on- line classification problems with explicit regularization. This method employs the update rule of the regularized dual averaging (RDA) method proposed by Xiao, but only on the subsequence of training examples where a classification error is made. We derive a bound on the number of mistakes made by this method on the training set, as well as its generalization error rate. We also intro- duce the concept of relative strength of regularization, and show how it affects the mistake bound and gener- alization performance. We examine the method using l1-regularization on a large-scale natural language pro- cessing task, and obtained state-of-the-art classification performance with fairly sparse models.

DeepQA Jeopardy! Gamification: A Machine-Learning Perspective

DeepQA is a large-scale natural language processing (NLP) question-and-answer system that responds across a breadth of structured and unstructured data, from hundreds of analytics that are combined with over 50 models, trained through machine learning. After the 2011 historic milestone of defeating the two best human players in the Jeopardy! game show, the technology behind IBM Watson, DeepQA, is undergoing gamification into real-world business problems. Gamifying a business domain for Watson is a composite of functional, content, and training adaptation for nongame play. During domain gamification for medical, financial, government, or any other business, each system change affects the machine-learning process. As opposed to the original Watson Jeopardy!, whose class distribution of positive-to-negative labels is 1:100, in adaptation the computed training instances, question-and-answer pairs transformed into true-false labels, result in a very low positive-to-negative ratio of 1:100 000. Such initial extreme class imbalance during domain gamification poses a big challenge for the Watson machine-learning pipelines. The combination of ingested corpus sets, question-and-answer pairs, configuration settings, and NLP algorithms contribute toward the challenging data state. We propose several data engineering techniques, such as answer key vetting and expansion, source ingestion, oversampling classes, and question set modifications to increase the computed true labels. In addition, algorithm engineering, such as an implementation of the Newton-Raphson logistic regression with a regularization term, relaxes the constraints of class imbalance during training adaptation. We conclude by empirically demonstrating that data and algorithm engineering are complementary and indispensable to overcome the challenges in this first Watson gamification for real-world business problems.

E-Science–towards the cloud: infrastructures, applications and research

The 2011 e-Science All Hands Meeting (AHM) marked the tenth annual gathering of technologists and scientists first brought together by the UK e-Science Programme—the £250 million research venture funded by the UK Research Councils. In the decade since the founding of this programme in 2001, e-science research has evolved from fundamental research focused on computational grids into work involving research from a wide variety of domains, such as bioinformatics, computing, astronomy, physics and medicine. The AHM community is now multinational, interdisciplinary and dynamic, and has recently moved towards addressing the issues and challenges raised by the increasing emergence of cloud computing in daily life. This continuing popularity and success has been highlighted by AHM 2011, hosted at the University of York by the EPSRC White Rose Grid e-Science Centre and attended by over 200 delegates. This Theme Issue provides a snapshot of the community's vibrant diversity, featuring the best papers presented at the conference, peer-reviewed for this issue, written by researchers from many different countries and backgrounds. The first paper in this issue, ‘The application of cloud computing to scientific workflows: a study of cost and performance’ by Berriman et al. [1], discusses a key factor in the use of cloud computing—is it economically beneficial for scientists to use a third-party cloud provider? Following this, Turilli et al. [2], writing in ‘Flexible services for the support of research’, focus on the adoption of the cloud by higher education institutions and address both flexible on-demand access to storage resources and scalability across a heterogeneous set of cloud infrastructures. Turner et al. [3] write of their experiences in developing secure virtual research environments to integrate multidisciplinary research in ‘Secure data sharing across portals: experiences from OneVRE’. Next, Tablan et al. [4] present ‘GATECloud.net: a platform for large-scale, open-source text processing on the cloud’, a paper discussing the authors' cloud-based platform's ability to perform large-scale data-intensive natural language processing experiments by harnessing the computing power of the Amazon Cloud. Additionally, the paper includes a cost–benefit analysis and usage evaluation. Moving away from the cloud, Yang et al. [5] discuss philosophical issues concerning data quality, identify actual user needs on data quality, review current standards and specifications, and propose an integrated model for data quality in the field of Earth observation in ‘An integrated view of data quality in Earth observation’. Service orientation is another interest of the AHM community, and Weeks et al. [6] present a developmental overview of the CARMEN virtual laboratory platform for integrating scientific code through a service-oriented infrastructure in ‘The CARMEN software as a service infrastructure’. In ‘Enhancing research publications using Rich Interactive Narratives’, Takeda et al. [7] describe a mechanism that provides an interactive framework for research publications in the fields of archaeology and chemistry. Following this, Colling et al. [8] discuss the highly important work of the UK's gridPP project in providing the grid infrastructure needed by experiments at the Large Hadron Collider at CERN in the paper ‘Processing LHC data in the UK’. Returning to the cloud, the work undertaken within the RAPPORT project to investigate the running of a group of multi-domain scientific applications on cloud resources is presented by Cohen et al. [9] in ‘RAPPORT: running scientific high-performance computing applications on the cloud’. Djemame et al. [10] then present their approach to risk management as applied to cloud computing, with a particular emphasis on legal issues, in the paper ‘Legal issues in clouds: towards a risk inventory’. In ‘Developing cloud applications using e-Science Central’, Hiden et al. [11] present an overview of the e-Science Central (e-SC) cloud processing system and its application to a number of e-science projects. Finally, Suresh et al. [12] discuss how cloud computing has the potential for enabling a new technique for using queuing theory to meet specified response-time targets against fluctuating event arrival rates in their paper ‘Scalable and responsive event processing in the cloud’. As can be seen from the above descriptions, the variety, scope and standard of submissions to AHM 2011 was very high; in order to help focus the conference, we are indebted to the Conference Programme Committee, featuring 36 international experts and leaders in their respective fields. Owing to space constraints, many high-quality pieces of research presented at the conference cannot be published in this Theme Issue, but we are confident that the whole body of work presented at the conference will help to continue to drive e-science and digital research both in the UK and abroad.