Pre-trained BERT Model Research Articles

Knowledge Graph Completion for High-Speed Railway Turnout Switch Machine Maintenance Based on the Multi-Level KBGC Model

The incompleteness of the existing knowledge graphs in the railway domain creates information gaps, impacting their quality and effectiveness in the operation and maintenance of high-speed railway turnout switch machines. To address this, we propose a multi-layer model (KBGC) that combines KG-BERT, graph attention network (GAT), and Convolutional Embedding Network (ConvE) for knowledge graph completion in railway maintenance. KG-BERT fine-tunes a pre-trained BERT model to extract deep semantic features from entities and relationships, converting them into graph structures. GAT captures key structural relationships between nodes using an attention mechanism, producing enriched semantic and structural embeddings. Finally, ConvE reshapes and convolves these embeddings to learn complex entity interactions, enabling accurate link prediction. Extensive experiments on the HRTOM dataset, containing triplet data from high-speed railway turnout switch machines, demonstrate the model’s effectiveness, achieving an MRR of 50.8% and a Hits@10 of 60.7%. These findings show that the KBGC model significantly improves knowledge graph completion, aiding railway maintenance personnel in decision making and preventive maintenance, and providing new tools for railway maintenance applications.

A Novel Fusion of Machine Learning Methods for Enhancing Named Entity Recognition in Indonesian Language Text

One of the important implementations in machine learning is Named Entity Recognition (NER), which is used to process text and extract entities such as people, organizations, laws, religions, and locations. NER for the Indonesian language still faces significant challenges due to the lack of high-quality labelled datasets, which limits the development of more advanced models. To address this issue, we utilized several pre-trained BERT models (bert-base-uncased, indobenchmark/indobert-base-p1, indolem/indobert-base-uncased) and datasets (NERGRIT-IndoNLU, NERGRIT-Corpus, NERUGM, and NERUI). This study proposes a novel fusion approach by integrating deep learning architectures such as CNN, Bi-LSTM, Bi-GRU, and CRF to detect 19 entities. This approach enhances BERT’s sequence modelling and feature extraction capabilities, while CRF improves entity prediction by enforcing global word-sequence constraints. Experimental results demonstrate that the fusion approach outperforms previous methods. On the bert-base-uncased dataset, accuracy reached 94.75%, while indobenchmark/indobert-base-p1 achieved 95.75%, and indolem/indobert-base-uncased achieved 95.85%. This study emphasizes the effectiveness of combining deep learning architectures with pre-trained transformers to improve NER performance in the Indonesian language. The proposed methodology offers significant advancements in entity extraction for languages with limited datasets, such as Indonesian.

Preservation of emotional context in tweet embeddings on social networking sites

In communication, emotional information is crucial, yet its preservation in tweet embeddings remains a challenge. This study aims to address this gap by exploring three distinct methods for generating embedding vectors of tweets: word2vec models, pre-trained BERT models, and fine-tuned BERT models. We conducted an analysis to assess the degree to which emotional information is conserved in the resulting embedding vectors. Our findings indicate that the fine-tuned BERT model exhibits a higher level of preservation of emotional information compared to other methods. These results underscore the importance of utilizing advanced natural language processing techniques for preserving emotional context in text data, with potential implications for enhancing sentiment analysis and understanding human communication in social media contexts.

Emotions, behaviors and places: Mapping sentiments with behaviors in Japanese tweets

Emotions are the basis of human behavior and an important part of spatial constructions. Despite numerous efforts to leverage spatio-temporal big data from social media to capture public sentiments, there is a lack of understanding of how individuals' behaviors play a role in people's sentiments at places they visit. Thus, this study aims to advance our understanding of the complex relationships between sentiments, behaviors, and places. Through a case study in Japan in 2021, we analyzed one year of countrywide geotagged social media by fine-tuning a pre-trained Japanese BERT model (accuracy: 74 %). The results emphasize the importance of understanding user emotions by considering people's behaviors, even within similar types of locations. Our research framework would be beneficial for urban planners and policymakers to enhance urban spaces.

TourismNER: A Tourism Named Entity Recognition method based on entity boundary joint prediction

Tourism named entity recognition is indispensable in tourism information extraction, and plays a crucial role in constructing tourism knowledge map and enhancing tourism knowledge quiz system. The difficulty of tourism named entity recognition lies in its complex nested structure, and the lengthy entity naming length. To address these existing problems, we propose a tourism named entity recognition model that jointly predicts entity boundaries, adopting a training strategy of data preprocessing to enhance the model’s ability for tourism named entity boundary recognition, while our model introduces a pre-trained Bert model as well as BiLSTM coding to enhance the representation of the model’s contexts, and uses a combined predictor of Biaffine and MLP to enhance the model’s recognition performance for boundaries, as well as introducing label smoothing cross entropy to smooth the target labels during the training process. Experiments are conducted on three datasets with different granularities. From the analysis of the experimental results, it can be seen that the named entity recognition method achieves higher accuracy and F1 value compared with the optimal baseline model, and also proves the effectiveness and generality of the modeling method proposed in this paper.

Named Entity Recognition Using BERT

Abstract: In our current era where technology keeps increasing, where data driven applications are reign supreme, Mamed Entity Recognition (NER) emerges as an important tool, aiding in better understanding of text and powering various natural language processing tasks downstream. This paper takes into the use of pre trained BERT model for NER tasks, Skipping the need for further adjustments. Our aim is to boost the accuracy and speed of NER systems by leveraging BERT’s pre trained representations. Our approach involves leveraging built in capabilities to spot key entity labels like names, dates, and people, streaming the NER process while achieving competitive results. This research takes a step forward in maximizing the potential of pre trained models for NER applications ultimately making text analysis easier and more efficient in today’s digital world.

Enhancing Chinese–Braille translation: A two-part approach with token prediction and segmentation labeling

Visually assistive systems for the visually impaired play a pivotal role in enhancing the quality of life for the visually impaired. Assistive technologies for the visually impaired have undergone a remarkable transformation with the advent of deep learning and sophisticated assistive devices. In particular, the paper utilizes the latest machine translation models and techniques to accomplish the Chinese–Braille translation task, providing convenience for visually impaired individuals. The Traditional end-to-end Chinese–Braille translation approach incorporates Braille dots and Braille word segmentation symbols as tokens within the model’s vocabulary. However, our findings reveal that Braille word segmentation is significantly more complex than Braille dot prediction. The paper proposes a novel Two-Part Loss (TPL) method that treats these tasks distinctly, leading to significant accuracy improvements. To enhance translation performance further, we introduce a BERT-Enhanced Segmentation Transformer (BEST) method. BEST leverages knowledge distillation techniques to transfer knowledge from a pre-trained BERT model to the translate model, mitigating its limitations in word segmentation. Additionally, soft label distillation is employed to improve overall efficacy further. The TPL approach achieves an average BLEU score improvement of 1.16 and 5.42 for Transformer and GPT models on four datasets, respectively. In addition, The work presents a two-stage deep learning-based translation approach that outperforms traditional multi-step and end-to-end methods. The proposed two-stage translation method achieves an average BLEU score improvement of 0.85 across four datasets.

Mapping vaccine names in clinical trials to vaccine ontology using cascaded fine-tuned domain-specific language models

BackgroundVaccines have revolutionized public health by providing protection against infectious diseases. They stimulate the immune system and generate memory cells to defend against targeted diseases. Clinical trials evaluate vaccine performance, including dosage, administration routes, and potential side effects. ClinicalTrials.gov is a valuable repository of clinical trial information, but the vaccine data in them lacks standardization, leading to challenges in automatic concept mapping, vaccine-related knowledge development, evidence-based decision-making, and vaccine surveillance.ResultsIn this study, we developed a cascaded framework that capitalized on multiple domain knowledge sources, including clinical trials, the Unified Medical Language System (UMLS), and the Vaccine Ontology (VO), to enhance the performance of domain-specific language models for automated mapping of VO from clinical trials. The Vaccine Ontology (VO) is a community-based ontology that was developed to promote vaccine data standardization, integration, and computer-assisted reasoning. Our methodology involved extracting and annotating data from various sources. We then performed pre-training on the PubMedBERT model, leading to the development of CTPubMedBERT. Subsequently, we enhanced CTPubMedBERT by incorporating SAPBERT, which was pretrained using the UMLS, resulting in CTPubMedBERT + SAPBERT. Further refinement was accomplished through fine-tuning using the Vaccine Ontology corpus and vaccine data from clinical trials, yielding the CTPubMedBERT + SAPBERT + VO model. Finally, we utilized a collection of pre-trained models, along with the weighted rule-based ensemble approach, to normalize the vaccine corpus and improve the accuracy of the process. The ranking process in concept normalization involves prioritizing and ordering potential concepts to identify the most suitable match for a given context. We conducted a ranking of the Top 10 concepts, and our experimental results demonstrate that our proposed cascaded framework consistently outperformed existing effective baselines on vaccine mapping, achieving 71.8% on top 1 candidate’s accuracy and 90.0% on top 10 candidate’s accuracy.ConclusionThis study provides a detailed insight into a cascaded framework of fine-tuned domain-specific language models improving mapping of VO from clinical trials. By effectively leveraging domain-specific information and applying weighted rule-based ensembles of different pre-trained BERT models, our framework can significantly enhance the mapping of VO from clinical trials.

An aspect sentiment analysis model with Aspect Gated Convolution and Dual-Feature Filtering layers

Aspect level sentiment analysis is a basic task to determine the sentiment bias based on the contextual information near the aspect words. Some sentences contain many confusing feature words due to incomplete structure or high complexity in sentiment prediction, which can easily lead to the problem that the model pays too much attention to unimportant features. Furthermore, the role of aspect-related sentiment features is not significant in the feature extraction process. The capsule network, due to its complex network structure, leads to capsule detachment when too much information is found in the dataset. Too much computational resources are easily consumed during dynamic routing, which in turn affects the model's judgement of the polarity of aspect-related sentiment. To address these issues, we propose a capsule network (AGCDFF-Caps) with aspect-gated convolution and Dual-Feature Filtering layers for aspect-level sentiment analysis. A pre-trained BERT model is used to generate a serialised representation of the text, and the semantic representation in the contextual text is enhanced by the self-attention mechanism and bi-GRU. An aspect-gating based convolutional neural network is constructed to selectively extract contextual sentiment information about aspects and discard irrelevant information. A capsule network is then used to learn the multispatial semantic features of the aspect and the text. In particular, we incorporate a Dual-Feature Filtering network structure into the capsule network structure to strengthen the interaction between the particular aspect and the context from global and local perspectives, filtering the redundant information in local semantics and global semantics. The opinion feature representations that can more accurately express the emotional tendency of the aspect are obtained. Experimental results on SemEval2014 and Twitter datasets show that the proposed hybrid network structure has superior classification performance compared to 12 advanced baseline methods.

DepressionEmo: A novel dataset for multilabel classification of depression emotions

Emotions are integral to human social interactions, with diverse responses elicited by various situational contexts. Particularly, the prevalence of negative emotional states has been correlated with negative outcomes for mental health, necessitating a comprehensive analysis of their occurrence and impact on individuals. In this paper, we introduce a novel dataset named DepressionEmo designed to detect 8 emotions associated with depression by 6037 examples of long Reddit user posts. This dataset was created through a majority vote over inputs by zero-shot classifications from pre-trained models and validating the quality by annotators and ChatGPT, exhibiting an acceptable level of inter-rater reliability between annotators. The correlation between emotions, and linguistic analysis are conducted on DepressionEmo. Besides, we provide several text classification methods classified into two groups: machine learning methods such as SVM, XGBoost, and LightGBM; and deep learning methods such as BERT, BART, GAN-BERT, and T5. Despite achieving the same F1 Macro score of 0.76 as BART, the pretrained BERT model, bert-base-uncased, stands out as the most efficient model in our experiments due to its lower number of parameters. Across all emotions, the highest F1 Macro value is achieved by suicide intent, indicating a certain value of our dataset in identifying emotions in individuals with depression symptoms through text analysis. The curated dataset is publicly available at: https://github.com/abuBakarSiddiqurRahman/DepressionEmo.

Transfer learning across different chemical domains: virtual screening of organic materials with deep learning models pretrained on small molecule and chemical reaction data

Machine learning is becoming a preferred method for the virtual screening of organic materials due to its cost-effectiveness over traditional computationally demanding techniques. However, the scarcity of labeled data for organic materials poses a significant challenge for training advanced machine learning models. This study showcases the potential of utilizing databases of drug-like small molecules and chemical reactions to pretrain the BERT model, enhancing its performance in the virtual screening of organic materials. By fine-tuning the BERT models with data from five virtual screening tasks, the version pretrained with the USPTO–SMILES dataset achieved R2 scores exceeding 0.94 for three tasks and over 0.81 for two others. This performance surpasses that of models pretrained on the small molecule or organic materials databases and outperforms three traditional machine learning models trained directly on virtual screening data. The success of the USPTO–SMILES pretrained BERT model can be attributed to the diverse array of organic building blocks in the USPTO database, offering a broader exploration of the chemical space. The study further suggests that accessing a reaction database with a wider range of reactions than the USPTO could further enhance model performance. Overall, this research validates the feasibility of applying transfer learning across different chemical domains for the efficient virtual screening of organic materials.Scientific contributionThis study verifies the feasibility of applying transfer learning to large language models in different chemical fields to help organic materials perform virtual screening. Through the comparison of transfer learning from different chemical fields to a variety of organic material molecules, the high precision virtual screening of organic materials is realized.

Impact of COVID-19 Vaccine Persuasion Strategies on Social Endorsement and Public Response on Chinese Social Media

ABSTRACT Persuading the public to get vaccinated against infectious diseases is critical and carries profound implications for preparing for future pandemics. This study examined whether and how persuasion strategies employed in pro-vaccine messages affect social endorsement and audience stance toward the COVID-19 vaccine expressed in comments on Chinese social media. Through manual coding and pre-trained BERT model, we analyzed 1,500 Weibo posts focused on COVID-19 vaccination persuasion and 238,201 associated comments. Results showed that medical experts succeeded in eliciting heightened social endorsement and receiving more pro-vaccine comments. Posts that employed negative emotional appeal were less likely to be liked or receive pro-vaccine comments. Besides, vaccine persuasion messages presented in a narrative format or emphasizing vaccine efficacy garnered significantly more likes but did not significantly receive more pro-vaccine comments. Discussing domestic issues or employing joy appeal received more pro-vaccine comments. These results offer valuable insights for health practitioners and communicators, highlighting more effective persuasion strategies for engaging citizens in vaccine-related discussions on social media. This study underscores the importance of leveraging persuasion tactics on social media to foster vaccination uptake and better prepare us for handling future pandemics.

A Multi-solution Study on GDPR AI-enabled Completeness Checking of DPAs

Specifying legal requirements for software systems to ensure their compliance with the applicable regulations is a major concern of requirements engineering. Personal data which is collected by an organization is often shared with other organizations to perform certain processing activities. In such cases, the General Data Protection Regulation (GDPR) requires issuing a data processing agreement (DPA) which regulates the processing and further ensures that personal data remains protected. Violating GDPR can lead to huge fines reaching to billions of Euros. Software systems involving personal data processing must adhere to the legal obligations stipulated both at a general level in GDPR as well as the obligations outlined in DPAs highlighting specific business. In other words, a DPA is yet another source from which requirements engineers can elicit legal requirements. However, the DPA must be complete according to GDPR to ensure that the elicited requirements cover the complete set of obligations. Therefore, checking the completeness of DPAs is a prerequisite step towards developing a compliant system. Analyzing DPAs with respect to GDPR entirely manually is time consuming and requires adequate legal expertise. In this paper, we propose an automation strategy that addresses the completeness checking of DPAs against GDPR provisions as a text classification problem. Specifically, we pursue ten alternative solutions which are enabled by different technologies, namely traditional machine learning, deep learning, language modeling, and few-shot learning. The goal of our work is to empirically examine how these different technologies fare in the legal domain. We computed F2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$_2$$\\end{document} score on a set of 30 real DPAs. Our evaluation shows that best-performing solutions yield F2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$_2$$\\end{document} score of 86.7% and 89.7% are based on pre-trained BERT and RoBERTa language models. Our analysis further shows that other alternative solutions based on deep learning (e.g., BiLSTM) and few-shot learning (e.g., SetFit) can achieve comparable accuracy, yet are more efficient to develop.

GraphPyRec: A novel graph-based approach for fine-grained Python code recommendation

Artificial intelligence has been widely applied in software engineering areas such as code recommendation. Significant progress has been made in code recommendation for static languages in recent years, but it remains challenging for dynamic languages like Python as accurately determining data flows before runtime is difficult. This limitation hinders data flow analysis, affecting the performance of code recommendation methods that rely on code analysis. In this study, a graph-based Python recommendation approach (GraphPyRec) is proposed by converting source code into a graph representation that captures both semantic and dynamic information. Nodes represent semantic information, with unique rules defined for various code statements. Edges depict control flow and data flow, utilizing a child-sibling-like process and a dedicated algorithm for data transfer extraction. Alongside the graph, a bag of words is created to include essential names, and a pre-trained BERT model transforms it into vectors. These vectors are integrated into a Gated Graph Neural Network (GGNN) process of the code recommendation model, enhancing its effectiveness and accuracy. To validate the proposed method, we crawled over a million lines of code from GitHub. Experimental results show that GraphPyRec outperforms existing mainstream Python code recommendation methods, achieving Top-1, 5, and 10 accuracy rates of 68.52%, 88.92%, and 94.05%, respectively, along with a Mean Reciprocal Rank (MRR) of 0.772.

Machine Learning and Deep Learning Sentiment Analysis Models: Case Study on the SENT-COVID Corpus of Tweets in Mexican Spanish

This article presents a comprehensive evaluation of traditional machine learning and deep learning models in analyzing sentiment trends within the SENT-COVID Twitter corpus, curated during the COVID-19 pandemic. The corpus, filtered by COVID-19 related keywords and manually annotated for polarity, is a pivotal resource for conducting sentiment analysis experiments. Our study investigates various approaches, including classic vector-based systems such as word2vec, doc2vec, and diverse phrase modeling techniques, alongside Spanish pre-trained BERT models. We assess the performance of readily available sentiment analysis libraries for Python users, including TextBlob, VADER, and Pysentimiento. Additionally, we implement and evaluate traditional classification algorithms such as Logistic Regression, Naive Bayes, Support Vector Machines, and simple neural networks like Multilayer Perceptron. Throughout the research, we explore different dimensionality reduction techniques. This methodology enables a precise comparison among classification methods, with BETO-uncased achieving the highest accuracy of 0.73 on the test set. Our findings underscore the efficacy and applicability of traditional machine learning and deep learning models in analyzing sentiment trends within the context of low-resource Spanish language scenarios and emerging topics like COVID-19.

A topic relevance-aware click model for web search

To better understand users’ behavior patterns in web search, numerous click models are proposed to extract the implicit interaction feedback. Most existing click models are heavily based on the implicit information to model user behaviors, ignoring the impact of explicit information between queries and documents in search sessions. In this paper, we fully consider the topic relevance between queries and documents in search sessions and propose a novel topic relevance-aware click model (TRA-CM) for web search. TRA-CM consists of a relevance estimator and an examination predictor. The relevance estimator consists of a topic relevance predictor and a click context encoder. In the topic relevance predictor, we utilize the pre-trained BERT model to model the content information of queries and documents in search sessions. Meanwhile, we use transformer to encode users’ click behaviors in the click context encoder. We further apply a two-stage fusion strategy to obtain the final relevance scores. The examination predictor estimates the examination probability of each document. We further utilize learnable filters to attenuate log noise and obtain purer input features in both relevance estimator and examination predictor, and investigate different combination functions to integrate relevance scores and examination probabilities into click prediction. Extensive experiment results on two real-world session datasets prove that TRA-CM outperforms existing click models in both click prediction and relevance estimation tasks.

EDUVQA – Visual Question Answering: An Educational Perspective

Increasing applications of artificial intelligence in the field of education have changed the way school children learn various concepts. Educational Visual Question Answering or EDUVQA is one such application that allows students to interact directly with images, ask educational questions, and get the correct answer. Two major challenges faced by educational VQA are the lack of availability of domain-specific datasets and often it requires referring to the external knowledge bases to answer open-domain questions. We propose a novel EDUVQA model developed especially for educational purposes and introduce our own EDUVQA dataset. The dataset consists of four categories of images - animals, plants, fruits, and vegetables. The majority of the currently used techniques focus on the extraction of picture and question characteristics in order to discover the joint feature embeddings via multimodal fusion or attention mechanisms. We propose a different method that aims to better utilize the semantic knowledge present in images. Our approach entails building an EDUVQA dataset using educational images, where each data point is made up of an image, a question that corresponds to it, a valid response, and a fact that supports it. The fact is created in the form of <S,V,O> triplet where ‘s’ denotes a subject, ‘v’ a verb, and ‘o’ an object. First, an SVO detector model is trained on EDUVQA dataset capable of predicting the Subject, Verb, and Object present in the image-question pair. Using this <S,V,O> triplet, the most relevant facts from our fact base are extracted. The final answer is predicted using these extracted facts, image, and question attributes. The image features are extricated using pretrained ResNet and question features using a pre-trained BERT model. We have optimized and improved on the current methodologies that use a Relation-based approach and built our SVO-detector model that outperforms current models by 10%.

SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation

Large language Models (LLMs), though growing exceedingly powerful, comprises of orders of magnitude less neurons and synapses than the human brain. However, it requires significantly more power/energy to operate. In this work, we propose a novel bio-inspired spiking language model (LM) which aims to reduce the computational cost of conventional LMs by drawing motivation from the synaptic information flow in the brain. In this paper, we demonstrate a framework that leverages the average spiking rate of neurons at equilibrium to train a neuromorphic spiking LM using implicit differentiation technique, thereby overcoming the non-differentiability problem of spiking neural network (SNN) based algorithms without using any type of surrogate gradient. The steady-state convergence of the spiking neurons also allows us to design a spiking attention mechanism, which is critical in developing a scalable spiking LM. Moreover, the convergence of average spiking rate of neurons at equilibrium is utilized to develop a novel ANN-SNN knowledge distillation based technique wherein we use a pre-trained BERT model as “teacher” to train our “student” spiking architecture. While the primary architecture proposed in this paper is motivated by BERT, the technique can be potentially extended to different kinds of LLMs. Our work is the first one to demonstrate the performance of an operational spiking LM architecture on multiple different tasks in the GLUE benchmark. Our implementation source code is available at https://github.com/NeuroCompLab-psu/SpikingBERT.

Multilevel Attention Network with Semi-supervised Domain Adaptation for Drug-Target Prediction

Prediction of drug-target interactions (DTIs) is a crucial step in drug discovery, and deep learning methods have shown great promise on various DTI datasets. However, existing approaches still face several challenges, including limited labeled data, hidden bias issue, and a lack of generalization ability to out-of-domain data. These challenges hinder the model's capacity to learn truly informative interaction features, leading to shortcut learning and inferior predictive performance on novel drug-target pairs. To address these issues, we propose MlanDTI, a semi-supervised domain adaptive multilevel attention network (Mlan) for DTI prediction. We utilize two pre-trained BERT models to acquire bidirectional representations enriched with information from unlabeled data. Then, we introduce a multilevel attention mechanism, enabling the model to learn domain-invariant DTIs at different hierarchical levels. Moreover, we present a simple yet effective semi-supervised pseudo-labeling method to further enhance our model's predictive ability in cross-domain scenarios. Experiments on four datasets show that MlanDTI achieves state-of-the-art performances over other methods under intra-domain settings and outperforms all other approaches under cross-domain settings. The source code is available at https://github.com/CMACH508/MlanDTI.

CourseKG: An Educational Knowledge Graph Based on Course Information for Precision Teaching

With the rapid development of advanced technologies, such as artificial intelligence and deep learning, educational informatization has entered a new era. However, the explosion of information has brought numerous challenges. Knowledge graphs, as a crucial component of artificial intelligence, can contribute to the quality of teaching. This study proposes an educational knowledge graph based on course information named CourseKG for precision teaching. Precision teaching seeks to individualize the curriculum for each learner and optimize learning efficiency. CourseKG aims to establish a correct and comprehensive curriculum knowledge system and promote personalized learning paths. CourseKG can address the issue that current general-purpose knowledge graphs are not suitable for the education field. Particularly, this study proposes a framework for educational entity recognition based on the pre-trained BERT model. This framework captures relevant information in the educational domain using the BERT model and combines it with the BiGRU and multi-head self-attention mechanism to extract multi-scale and multi-level global dependency relationships. In addition, the CRF is used for character-label decoding. Further, a relationship extraction method based on the BERT model, which integrates sentence features and educational entities and estimates the similarity between knowledge pairs using cosine similarity, is proposed. The proposed CourseKG is verified by experiments using real-world C programming course data. The experimental results demonstrate the effectiveness of CourseKG. Finally, the results show that the proposed CourseKG can significantly enhance the precision teaching quality and realize multi-directional adaptation among teachers, courses, and students.