Text Mining Model Research Articles

Constructing and analyzing domain-specific language model for financial text mining

The application of natural language processing (NLP) to financial fields is advancing with an increase in the number of available financial documents. Transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) have been successful in NLP in recent years. These cutting-edge models have been adapted to the financial domain by applying financial corpora to existing pre-trained models and by pre-training with the financial corpora from scratch. In Japanese, by contrast, financial terminology cannot be applied from a general vocabulary without further processing. In this study, we construct language models suitable for the financial domain. Furthermore, we compare methods for adapting language models to the financial domain, such as pre-training methods and vocabulary adaptation. We confirm that the adaptation of a pre-training corpus and tokenizer vocabulary based on a corpus of financial text is effective in several downstream financial tasks. No significant difference is observed between pre-training with the financial corpus and continuous pre-training from the general language model with the financial corpus. We have released our source code and pre-trained models.

A data-driven optimization model for the workover rig scheduling problem: Case study in an oil company

After completion, oil wells often require intervention services to increase productivity, correct oil flow losses, and solve mechanical failures. These interventions, known as workovers, are made using oil rigs, an expensive and scarce resource. The workover rig scheduling problem (WRSP) comprises deciding which wells demanding workovers will be attended to, which rigs will serve them, and when the operations must be performed, minimizing the rig fleet costs and the oil production loss associated with the workover delay. This study presents a data-driven optimization methodology for the WRSP using text mining and regression models to predict the duration of the workover activities and a mixed-integer linear programming model to obtain the solutions for the model. A sensitivity analysis is performed using simulation to measure the impact of the regression error in the solution.

An Integrated Topic Modelling and Graph Neural Network for Improving Cross-lingual Text Classification

In recent years, along with the dramatic developments of deep learning in the natural language processing (NLP) domain, notable multilingual pre-trained language techniques have been proposed. These recent multilingual text analysis and mining models have demonstrated state-of-the-art performance in several primitive NLP tasks, including cross-lingual text classification (CLC). However, these recent multilingual pre-trained language models still suffer limitations regarding their adaptation for specific task-driven fine-tuning in the context of low-resource languages. Moreover, they also encounter problems related to the capability of preserving the global semantic (e.g., topic, etc.) and long-range relationships between words to better fine-tune and effectively handle the cross-lingual text classification task. To meet these challenges, in this article, we propose a novel topic-driven multi-typed text graph attention–based representation learning method for dealing with the cross-lingual text classification problem called TG-CTC. In the proposed TG-CTC model, we utilize a novel fused topic-driven multi-typed text graph representation to jointly learn the rich-schematic structural and global semantic information of texts to effectively handle the CLC task. More specifically, we integrate the heterogeneous text graph attention network with the neural topic modelling approach to enrich the semantic information of learned textual representations in the context of multiple languages. Extensive experiments in benchmark multilingual datasets showed the effectiveness of the proposed TG-CTC model compared with the contemporary state-of-the-art baselines.

Augmented Reality and Virtual Reality in Education: Public Perspectives, Sentiments, Attitudes, and Discourses

This study aims to understand the public’s perspectives, sentiments, attitudes, and discourses regarding the adoption, integration, and use of augmented reality and virtual reality in education and in general by analyzing social media data. Due to its nature, Twitter was the selected platform. Over 17 million tweets were retrieved from January 2010 to December 2020 and four datasets were created. Two of them referred to the general use of these technologies and two to their educational use. The data was analyzed using text mining, sentiment analysis (e.g., polarity and emotion detection), and topic modeling methods. TextBlob, Word-Emotion Association Lexicon (EmoLex), Valence Aware Dictionary for Sentiment Reasoning (VADER), and Latent Dirichlet Allocation (LDA) were some of the tools used. Based on the results, the majority of the public were positively disposed toward the general and the educational use of both augmented reality and virtual reality and mostly expressed positive emotions (e.g., anticipation, trust, and joy) when referring to them. In total, 11 topics emerged that were related to education, new technologies, digital and social media use, marketing and advertising, the industrial domain, the health domain, gaming, fitness and exercising, devices, the travel and tourism domain, and software development kits. The educational benefits of augmented reality and virtual reality, their ability to enrich both teaching and learning activities, and their role as effective educational means were evident.

Discovering ethnic minority business research directions using text mining and topic modelling

PurposeThis study aims to unveil within the current academic literature the principal directions in the ethnic entrepreneurship and small business marketing research context.Design/methodology/approachAn automated literature analysis procedure was undertaken, attempting to cover all literature published on the subject since 1962. A total of 188 articles were analysed using text mining and topic modelling.FindingsThe results show a lack of framing of ethnic entrepreneurship literature outside the narrower scope of migration. Some core themes were found (e.g. network, diversity) around which several other themes orbit, including both related issues to the ethnic factor (e.g. barriers and minorities) and managerial issues (e.g. marketing and production).Originality/valueEthnic minority business and small business marketing research has seen a growing number of publications. However, a careful review of existing work is missing.

What we talk about when we talk about EEMs: using text mining and topic modeling to understand building energy efficiency measures (1836-RP)

Energy Efficiency Measures (EEMs) play a central role throughout the building energy efficiency industry, and lists of EEMs therefore exist in a variety of resources. However, each of these use different conventions for describing and organizing measures, which presents a major challenge for aggregating information across these resources. The goal of this study is to discover trends in how existing resources describe and organize EEMs using topic modeling and other text mining methods. A unique dataset of 3,490 EEMs from 16 different documents was compiled and analyzed using frequency analysis, part of speech tagging, and topic modeling. The results showed three major trends. First, a typical EEM contains six words and is phrased in verb-noun format, although these characteristics varied widely. Second, there are words and bigrams commonly used across many EEMs, and these include action words, specific building components, broader building systems, and descriptor terms. Third, there are thematic similarities between the EEM lists, which in some cases highlight the ways in which these lists are derived from one another. These findings provide insight into the nature of EEMs and can be used as the basis for developing a standardized system for organizing and describing EEMs.

Forecasting Financial Time Series Using Deep Learning Networks: Evidence from Long-Short Term Memory and Gated Recurrent Unit

The ability to predict the stock market and analyze market trends is invaluable to researchers and anyone interested in investing. However, this task is a challenging problem due to a large number of parameters and unpredictable noise that may affect the stock price. To overcome this issue, researchers have employed numerous approaches such as Moving Average (MA), Support Vector Machine (SVM), and Neural Networks. With technological advances, deep learning methods have become popular in processing time-series data. In this paper, we compare two recently introduced deep learning models, namely a Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), in forecasting daily movements of the Standard & Poor (S&P 500) index using the daily closing price of this index from 14/5/1991 to 14/5/2021. Results show that both models are effective and accurate in stock market prediction. In this case study, the mean squared error (MSE) and mean absolute error (MAE) for the GRU model are slightly lower than the LSTM model; hence, GRU outperformed the LSTM model despite its simpler structure. The results of this study are applicable in various instances where it is challenging to identify patterns among large volumes of unstructured data, such as medical data analysis, text mining, and financial time series modeling.

Understanding Philosophies of Higher Education between Countries in China’s Belt and Road Initiative: Analysis of University Mottos Based on Natural Language Processing Technology

University mottos have been demonstrated to convey educational philosophies, but few studies have investigated the educational philosophies reflected by university mottos from countries along the “One Belt, One Road” (OBOR) route, with applying the analytical approaches like text mining and natural language processing (NLP). This present investigation constructed a database of university mottos from 1,535 universities in 61 countries along the OBOR route and applied NLP technology to explore the similarities and differences of educational philosophies among the universities along the OBOR route. Methods of NLP used in this article included text mining, cluster analysis, and Latent Dirichlet Allocation (LDA) topic model. The social network analysis (SNA) model was used to further analyze the interconnection and exchange of educational philosophies expressed through university mottos. Based on the five identified topic categories, we concluded that countries along the OBOR route have the same basic educational philosophy for higher education but have different emphases due to cultural differences, such as the different emphases on virtue education, collectivism, and individualism. Education for sustainability is another focus of higher education philosophy reflected by the topic categories of university mottos, which is consistent with the United Nations Sustainable Development Goals, for example, quality education. In addition to demonstrating the BRI impact on educational philosophies of countries along the OBOR route, this present investigation could also contribute to the growing body of knowledge about international cooperation and communication of higher education under the framework of BRI, for a globalized, shared and sustainable education in the post-COVID era.

Capturing the Complexity of COVID-19 Research: Trend Analysis in the First Two Years of the Pandemic Using a Bayesian Probabilistic Model and Machine Learning Tools

Publications about COVID-19 have occurred practically since the first outbreak. Therefore, studying the evolution of the scientific publications on COVID-19 can provide us with information on current research trends and can help researchers and policymakers to form a structured view of the existing evidence base of COVID-19 and provide new research directions. This growth rate was so impressive that the need for updated information and research tools become essential to mitigate the spread of the virus. Therefore, traditional bibliographic research procedures, such as systematic reviews and meta-analyses, become time-consuming and limited in focus. This study aims to study the scientific literature on COVID-19 that has been published since its inception and to map the evolution of research in the time range between February 2020 and January 2022. The search was carried out in PubMed extracting topics using text mining and latent Dirichlet allocation modeling and a trend analysis was performed to analyze the temporal variations in research for each topic. We also study the distribution of these topics between countries and journals. 126,334 peer-reviewed articles and 16 research topics were identified. The countries with the highest number of scientific publications were the United States of America, China, Italy, United Kingdom, and India, respectively. Regarding the distribution of the number of publications by journal, we found that of the 7040 sources Int. J. Environ. Res. Public Health, PLoS ONE, and Sci. Rep., were the ones that led the publications on COVID-19. We discovered a growing tendency for eight topics (Prevention, Telemedicine, Vaccine immunity, Machine learning, Academic parameters, Risk factors and morbidity and mortality, Information synthesis methods, and Mental health), a falling trend for five of them (Epidemiology, COVID-19 pathology complications, Diagnostic test, Etiopathogenesis, and Political and health factors), and the rest varied throughout time with no discernible patterns (Therapeutics, Pharmacological and therapeutic target, and Repercussion health services).

Quantitative risk assessment of railway intrusions with text mining and fuzzy Rule-Based Bow-Tie model

With the increasing traveling speed of railway transportation, rail right-of-way intrusions can cause high-consequence accidents and pose severe challenges to railway safety. Although intrusion detection technologies have been widely studied and applied, they can only support in-event inspection and post-event control. In the current complex environment, there is a critical need to analyze the causal chain of railway intrusions and mitigate safety risks before or during the risk evolution process. This paper developed a novel methodological framework on the cause-consequence model based on the text mining techniques and fuzzy bow-tie modeling to systematically investigate the railway intrusion risks. In order to mine both critical factors and their interrelationships, a lexical co-occurrence analysis was carried out on a customized corpus of intrusion accident recordings. Then structured bow-tie diagrams were developed based on the networks generated by unstructured data. To overcome the data uncertainty issue, this paper utilized cause-consequence-based probabilistic analysis and fuzzy theory to quantify the risks involving the occurrence probability of top events and outcomes in terms of expert judgements. The application of the proposed bow-tie model was demonstrated based on the case of the Hualien Derailment accident. The findings based on the bow-tie model and historical accidents in this research have systematically summarized basic events and causal chains. Ultimately, they can be utilized by researchers and practitioners both to identify the critical risk factors and enhance railway safety via proactive and reactive measures.

A text mining and topic modeling based bibliometric exploration of information science research

This study investigates the evolution of information science research based on bibliometric analysis and semantic mining. The study discusses the value and application of metadata tagging and topic modeling. Forty-two thousand seven hundred thirty-eight articles were extracted from Clarivate Analytic's Web of Science Core Collection 2010-2020. This study was divided into two phases. Firstly, bibliometric analyzes were performed with VOSviewer. Secondly, the topic identification and evolution trends of information science research were conducted through the topic modeling approach latent dirichlet allocation (LDA) is often used to extract themes from a corpus, and the topic model was a representation of a collection of documents that is simplified using topic-modeling-toolkit (TMT). The top 10 core topics (tags) were information research design, information health-based, model data public, study information studies, analysis effect implications, knowledge support web, data research, social research study, study media information, and research impact time for the studied period. Not only does topic modeling assist in identifying popular topics or related areas within a researcher's area, but it may be used to discover emerging topics or areas of study throughout time.

An automatic descriptors recognizer customized for materials science literature

Materials science literature contains domain knowledge about numerous descriptors, which play a critical role in data-driven materials design. However, automatically extracting descriptors from literature remains challenging. Here, we develop an automatic descriptors recognizer based on natural language processing (NLP) to mine latent descriptors, which consists of a conditional data augmentation model incorporating materials domain knowledge (cDA-DK), coarse- and fine-grained descriptors subrecognizers (CGDR and FGDR). cDA-DK conducts augmenting training data of text mining model, which can significantly reduce the cost of manually labeling and enhance the robustness of its model. On this basis, CGDR recognizes coarse-grained descriptor entities automatically, and FGDR performs screening of fine-grained descriptors related to specific materials design. Following this, the activation energy of NASICON-type solid electrolytes, which is influenced by complicated descriptors, is taken as an example to demonstrate the potential utility of our recognizer. CGDR extracts 106896 descriptor entities from 1808 relevant articles with an accuracy (F1) of 0.87. Furthermore, with features from 408 descriptors screened by FGDR, six activation energy prediction models are constructed to perform experiments, achieving an optimal prediction performance (R2) of 0.96. This work provides important insight towards the understanding of structure-activity relationships, thus promoting materials design and discovery.

Health Information Technology During the COVID-19 Epidemic: A Review via Text Mining.

Due to the prevalence of the COVID-19 epidemic in all countries of the world, the need to apply health information technology is of great importance. hence, the study has identified the role of health information technology during the period of the COVID-19 epidemic. The present research is a review study by employing text mining techniques. Therefore, 941 published documents related to health information technology's role during the COVID-19 epidemic were extracted by keyword searching in the Web of Science database. In order to analyze the data and implement the text mining and topic modeling algorithms, Python programming language was applied. The results indicated that the highest number of publications related to the role of health information technology in the period of the COVID-19 epidemic was respectively on the following topics: "Models and smart systems," "Telemedicine," "Health care," "Health information technology," "Evidence-based medicine," "Big data and Statistic analysis." Health information technology has been extensively used during the COVID-19 epidemic. Therefore, different communities can apply these technologies, considering the conditions and facilities to manage the COVID-19 epidemic better.

Construction of 3D Model of College Public Physical Education Class Based on Text Mining Model

The responsibility and mission of higher education is to cultivate high-level talents needed for the development of society. The contemporary society has undergone great changes. The modern society characterized by the globalization of the knowledge economy has put forward social demands for higher education, and the development of science and technology with information and biotechnology as the core has put forward social demands on higher public physical education. Higher public physical education should creatively set new functions according to the development needs of the globalization of the knowledge economy and must cultivate people around the world, so that college courses can be adapted to the development needs of information technology, so as to meet the challenges of new technological revolutions. This research uses the literature research method, information research method, and observation method, uses text mining technology, and takes some ordinary colleges and universities as the survey objects. There are many problems in public physical teaching, and the corresponding feasible countermeasures and suggestions are put forward.

Discovering Thematically Coherent Biomedical Documents Using Contextualized Bidirectional Encoder Representations from Transformers-Based Clustering.

The increasing expansion of biomedical documents has increased the number of natural language textual resources related to the current applications. Meanwhile, there has been a great interest in extracting useful information from meaningful coherent groupings of textual content documents in the last decade. However, it is challenging to discover informative representations and define relevant articles from the rapidly growing biomedical literature due to the unsupervised nature of document clustering. Moreover, empirical investigations demonstrated that traditional text clustering methods produce unsatisfactory results in terms of non-contextualized vector space representations because that neglect the semantic relationship between biomedical texts. Recently, pre-trained language models have emerged as successful in a wide range of natural language processing applications. In this paper, we propose the Gaussian Mixture Model-based efficient clustering framework that incorporates substantially pre-trained (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining) BioBERT domain-specific language representations to enhance the clustering accuracy. Our proposed framework consists of main three phases. First, classic text pre-processing techniques are used biomedical document data, which crawled from the PubMed repository. Second, representative vectors are extracted from a pre-trained BioBERT language model for biomedical text mining. Third, we employ the Gaussian Mixture Model as a clustering algorithm, which allows us to assign labels for each biomedical document. In order to prove the efficiency of our proposed model, we conducted a comprehensive experimental analysis utilizing several clustering algorithms while combining diverse embedding techniques. Consequently, the experimental results show that the proposed model outperforms the benchmark models by reaching performance measures of Fowlkes mallows score, silhouette coefficient, adjusted rand index, Davies-Bouldin score of 0.7817, 0.3765, 0.4478, 1.6849, respectively. We expect the outcomes of this study will assist domain specialists in comprehending thematically cohesive documents in the healthcare field.

Text mining for identification of biological entities related to antibiotic resistant organisms.

Antimicrobial resistance is a significant public health problem worldwide. In recent years, the scientific community has been intensifying efforts to combat this problem; many experiments have been developed, and many articles are published in this area. However, the growing volume of biological literature increases the difficulty of the biocuration process due to the cost and time required. Modern text mining tools with the adoption of artificial intelligence technology are helpful to assist in the evolution of research. In this article, we propose a text mining model capable of identifying and ranking prioritizing scientific articles in the context of antimicrobial resistance. We retrieved scientific articles from the PubMed database, adopted machine learning techniques to generate the vector representation of the retrieved scientific articles, and identified their similarity with the context. As a result of this process, we obtained a dataset labeled “Relevant” and “Irrelevant” and used this dataset to implement one supervised learning algorithm to classify new records. The model’s overall performance reached 90% accuracy and the f-measure (harmonic mean between the metrics) reached 82% accuracy for positive class and 93% for negative class, showing quality in the identification of scientific articles relevant to the context. The dataset, scripts and models are available at https://github.com/engbiopct/TextMiningAMR.

Reflecting the trends in the academic landscape of special education using probabilistic dynamic topic modeling

PurposeSpecial education is the education segment that deals with the students facing hurdles in the traditional education system. Research data have evolved in the domain of special education due to scientific advances. The present study aims to employ text mining to extract the latent patterns from the scientific data.Design/methodology/approachThis study examined the 12,781 Scopus-indexed titles, abstracts and keywords published from 1987 to 2021 through an integrated text-mining and topic modeling approach. It combines dynamic topic models with highly cited reviews of this domain. It facilitates the extraction of topic clusters and communities in the topic network.FindingsThis methodology discovered children’s communication and speech using gaming techniques, mental retardation, cost effect on infant birth, involvement of special education children and their families, assistive technology information for special education, syndrome epilepsy and the impact of group study on skill development peers or self as the hottest topic of research in this domain. In addition to finding research hotspots, it further explores annual topic proportion trends, topic correlations and intertopic research areas.Originality/valueThe results provide a comprehensive summary of the popularity of research topics in special education in the past 34 years, and the results can provide useful insights and implications, and it could be used as a guide for contributors in special education form a structured view of past research and plan future research directions.

MatSciBERT: A materials domain language model for text mining and information extraction

A large amount of materials science knowledge is generated and stored as text published in peer-reviewed scientific literature. While recent developments in natural language processing, such as Bidirectional Encoder Representations from Transformers (BERT) models, provide promising information extraction tools, these models may yield suboptimal results when applied on materials domain since they are not trained in materials science specific notations and jargons. Here, we present a materials-aware language model, namely, MatSciBERT, trained on a large corpus of peer-reviewed materials science publications. We show that MatSciBERT outperforms SciBERT, a language model trained on science corpus, and establish state-of-the-art results on three downstream tasks, named entity recognition, relation classification, and abstract classification. We make the pre-trained weights of MatSciBERT publicly accessible for accelerated materials discovery and information extraction from materials science texts.

Research Paper on Fake Online Reviews Detection using Semi-supervised and Supervised learning

Abstract: Today's business and commerce are heavily influenced by online reviews. Most online product purchase decisions are based on customer reviews. As a result, opportunistic individuals or groups seek to shake product reviews in their favor. Fake online reviews have a significant impact on the efficiency of online consumers, merchants and e-commerce markets. Despite academic efforts to study fake reviews, there remains a need for research that can systematically analyze and summarize their causes and consequences. This task provides a semi-supervised and supervised text mining model for detecting fake web reviews and comparing their effectiveness to hotel review datasets. Keywords: Semi-Supervised. Supervised, Detection, Fake Review, Marketing

Service Design From Staffing to Outsourcing

The term outsourcing has become a conventional means of describing anything associated with the transaction of services that enables client organisations to blur core activities and thereby reduce their internal workforce and costs. The main objective of this study is confirming a gap in detailed and spe-cific reviews of formats and economic transactions through non-standard forms of employment, namely in a service design model from Staffing to Outsourcing. The literature review was performed using text mining and topic modelling techniques to group relevant topics and decreases the likelihood of human bias, while bringing robustness to the analysis. The results are reflected in a conceptual state of the art diagram that will serve as a basis to new discussions.