Data Natural Language Processing Research Articles

Development of a Knowledge Base for Construction Risk Assessments Using BERT and Graph Models

As a significant percentage of disasters and fatal accidents still occur in the construction sector, it is legally obligatory to conduct workplace risk assessments to avoid accidents and enhance safety. Identifying harmful and hazardous elements is crucial to discern the distinctive characteristics of potential accidents. However, conventional risk-assessment approaches, which rely on the skills and experience of safety managers, may overlook important factors, leading to inconsistencies in the procedures employed across different sites. Such unstructured safety knowledge reduces accessibility and utility, increases reliance on individual skills, and renders information management inefficient. Recently, the focus has shifted from efficient data storage to obtaining valuable knowledge tailored to specific use-cases. Knowledge-graph-based systems integrate and manage the relationships between knowledge entities, thereby enhancing the development of knowledge bases. Research on automatically extracting and managing predefined knowledge from various forms of data through natural language processing (NLP) is ongoing. This study proposes a novel method that uses NLP and graph models to automatically extract predefined knowledge from unstructured construction data and build an entity-relationship-based risk-assessment knowledge base. We developed an entity-name recognition and keyword-extraction engine that defines the core knowledge related to construction safety and risk assessments. This engine can automatically extract predefined knowledge from unstructured data by learning from NLP data. The extracted risk-assessment knowledsge was used to create a knowledge base, and its efficiency and effectiveness were validated through comparisons with existing methods. The results of this study are significant because they lay the foundation for an automatic knowledge-management system for construction safety and risk assessment, offering both practical and academic contributions to the field of construction safety.

Natural language processing‐based deep transfer learning model across diverse tabular datasets for bond strength prediction of composite bars in concrete

AbstractAs conventional machine learning models often struggle with scarcity and structural variation of training data, this paper proposes a novel regression transfer learning framework called transferable tabular regressor (TransTabRegressor) to address this challenge. The TransTabRegressor integrates natural language processing (NLP) for feature encoding, transformer for enhanced feature representation, and deep learning (DL) for robust modeling, facilitating effective transfer learning across tabular datasets using reducing input parameters. By leveraging the NLP data processor, the framework embeds both parameter names and values, enabling it to recognize and adapt to different expressions of similar parameters. For instance, the bond strength of fiber‐reinforced polymer (FRP) bars embedded in ultra‐high‐performance concrete (UHPC) is critical for ensuring the integrity of FRP‐UHPC structures. While pullout tests are widely adopted for their simplicity to generate substantial data, beam tests provide a closer approximation to actual stress conditions but are more complex thus resulting in limited data size. As a verification, the framework is applied to predict the bond strength of FRP bars embedded in UHPC using limited beam test data. A pre‐trained model is first established using 479 pieces of pullout test data. Subsequently, two transfer learning models are developed by fine‐tuning on 115 pieces of beam test data, where 66 correspond to concrete splitting failure and 49 correspond to pullout failure. For comparative analysis, XGBoost and neural network models are directly trained on the beam test data. Evaluation results demonstrate that the transfer learning models achieve significantly improved prediction accuracy and generalization capability. This study significantly highlights the effectiveness of the proposed TransTabRegressor in handling data scarcity and variability in input parameters across various engineering applications.

Developing and testing a framework for coding general practitioners’ free-text diagnoses in electronic medical records - a reliability study for generating training data in natural language processing

BackgroundDiagnoses entered by general practitioners into electronic medical records have great potential for research and practice, but unfortunately, diagnoses are often in uncoded format, making them of little use. Natural language processing (NLP) could assist in coding free-text diagnoses, but NLP models require local training data to unlock their potential. The aim of this study was to develop a framework of research-relevant diagnostic codes, to test the framework using free-text diagnoses from a Swiss primary care database and to generate training data for NLP modelling.MethodsThe framework of diagnostic codes was developed based on input from local stakeholders and consideration of epidemiological data. After pre-testing, the framework contained 105 diagnostic codes, which were then applied by two raters who independently coded randomly drawn lines of free text (LoFT) from diagnosis lists extracted from the electronic medical records of 3000 patients of 27 general practitioners. Coding frequency and mean occurrence rates (n and %) and inter-rater reliability (IRR) of coding were calculated using Cohen’s kappa (Κ).ResultsThe sample consisted of 26,980 LoFT and in 56.3% no code could be assigned because it was not a specific diagnosis. The most common diagnostic codes were, ‘dorsopathies’ (3.9%, a code covering all types of back problems, including non-specific lower back pain, scoliosis, and others) and ‘other diseases of the circulatory system’ (3.1%). Raters were in almost perfect agreement (Κ ≥ 0.81) for 69 of the 105 diagnostic codes, and 28 codes showed a substantial agreement (K between 0.61 and 0.80). Both high coding frequency and almost perfect agreement were found in 37 codes, including codes that are particularly difficult to identify from components of the electronic medical record, such as musculoskeletal conditions, cancer or tobacco use.ConclusionThe coding framework was characterised by a subset of very frequent and highly reliable diagnostic codes, which will be the most valuable targets for training NLP models for automated disease classification based on free-text diagnoses from Swiss general practice.

Layer-Weighted Attention and Ascending Feature Selection: An Approach for Seriousness Level Prediction Using the FDA Adverse Event Reporting System

In this study, we introduce a novel combination of layer-static-weighted attention and ascending feature selection techniques to predict the seriousness level of adverse drug events using the Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS). We utilized natural language processing (NLP) to analyze the terms in the active substance field, in addition to considering demographic and event information such as patient sex, healthcare provider qualification, and drug characterization. Our ascending feature selection method, which progressively incorporates additional features based on their importance, demonstrated continuous enhancements in prediction performance. Simultaneously, we employed a layer-static-weighted attention technique, which dynamically adjusts the model’s focus between natural language processing (NLP) and demographic features. This technique achieved its best performance at a balanced weight of 50%, yielding an average test accuracy of 74.56% and CV ROC score of 0.83 when 4000 features were included, indicating a compelling advantage to include a larger volume of meaningful features. By integrating these methodologies, we constructed a robust model capable of effectively predicting seriousness levels, offering significant potential for improving pharmacovigilance and enhancing drug safety monitoring. The results underscore the value of NLP and demographic data in predicting drug event seriousness and demonstrate the effectiveness of our combined techniques. We encourage further research to refine these methods and evaluate their application to other clinical datasets.

Bridging information gaps in menopause status classification through natural language processing.

To use natural language processing (NLP) of clinical notes to augment existing structured electronic health record (EHR) data for classification of a patient's menopausal status. A rule-based NLP system was designed to capture evidence of a patient's menopause status including dates of a patient's last menstrual period, reproductive surgeries, and postmenopause diagnosis as well as their use of birth control and menstrual interruptions. NLP-derived output was used in combination with structured EHR data to classify a patient's menopausal status. NLP processing and patient classification were performed on a cohort of 307512 female Veterans receiving healthcare at the US Department of Veterans Affairs (VA). NLP was validated at 99.6% precision. Including the NLP-derived data into a menopause phenotype increased the number of patients with data relevant to their menopausal status by 118%. Using structured codes alone, 81173 (27.0%) are able to be classified as postmenopausal or premenopausal. However, with the inclusion of NLP, this number increased 167804 (54.6%) patients. The premenopausal category grew by 532.7% with the inclusion of NLP data. By employing NLP, it became possible to identify documented data elements that predate VA care, originate outside VA networks, or have no corresponding structured field in the VA EHR that would be otherwise inaccessible for further analysis. NLP can be used to identify concepts relevant to a patient's menopausal status in clinical notes. Adding NLP-derived data to an algorithm classifying a patient's menopausal status significantly increases the number of patients classified using EHR data, ultimately enabling more detailed assessments of the impact of menopause on health outcomes.

Takween: A Comprehensive Arabic Collection and Annotation Tool

In the rapidly evolving field of Natural Language Processing (NLP), the availability of labeled data is crucial for developing and improving machine learning models. Labeled data provides the essential context and structure required for training algorithms to accurately interpret and process human language. However, obtaining labeled data, especially for languages with complex structures like Arabic, presents significant challenges. This paper introduces ”Takween,” a novel tool designed to meet the need for high-quality labeled data in Arabic NLP. Takween leverages social media and Wikipedia as data sources to facilitate the collection of large-scale Arabic text data. The tool enables users to annotate and label the collected data, with functionalities for reviewing annotations and visualizing the labeled data to ensure quality. Additionally, Takween incorporates active learning techniques, which allow for automatically selecting data samples for annotation based on model uncertainty. By integrating data collection, annotation, review, and visualization features with active learning capabilities, Takween offers a comprehensive solution for acquiring high-quality labeled data for Arabic NLP tasks.

Quantitative fusion of NLP, fMRI, and EEG data : A mathematical model for decoding semantic processing in the brain

This research introduces a novel mathematical model designed to integrate Natural Language Processing (NLP), functional Magnetic Resonance Imaging (fMRI), and Electroencephalography (EEG) data, aiming to decode the complex neural mechanisms of semantic processing in the human brain. By leveraging the complementary strengths of each modality—NLP’s linguistic analysis, fMRI’s spatial resolution, and EEG’s temporal precision—the model provides a groundbreaking approach to understanding how semantic information is processed across different brain regions and over time. The core of the proposed model is a dynamic, multi-layered framework that utilizes advanced statistical methods and machine learning algorithms. At its foundation, the model employs vector space representations from NLP to quantify semantic similarity and contextuality in language. These representations are then mapped onto neural activation patterns captured by fMRI and EEG, using a series of transformation matrices that are optimized through machine learning techniques. The model uniquely incorporates time-series analysis to account for the temporal dynamics of EEG data, while spatial patterns from fMRI data are analyzed through convolutional neural networks, ensuring a comprehensive integration of multimodal neuroimaging data. Key to proposed approach is the application of Bayesian inference methods to fuse these diverse data sources, allowing for the probabilistic modeling of semantic processing pathways in the brain. This enables the prediction of neural responses to linguistic stimuli with unprecedented accuracy and detail. Theoretical implications of our model suggest significant advances in understanding the neural basis of language comprehension, offering new insights into the dynamic interplay between linguistic structures and neural processes.

Implementing NLP techniques and data for Europe financial market forecasting

The large military conflict between Russia and Ukraine has consequences for the global financial market. The European region is one of the hardest hits, which experienced a high energy cost and devaluation of the currency. This paper investigates the Employment Expectations Indicator (EEI) and Economic Sentiment Indicator (ESI) from the business consumer survey index to reflect the reaction of the consumers of the European market by implementing Natural Language Processing (NLP) techniques and machine learning. In the object, the monetary policy decisions from European Central Bank are considered the data set for forecasting the Index. In the result, all nine models from scikit-learn contributed a great job in both categories of Accuracy, Precision, and F-score and Recall with a level of 90% as average. It represents that the NLP techniques effectively forecast the future value of two European business and consumer indexes. However, the limit of the dataset and model mostly related to the performance of NLP techniques is working for forecasting but less related to the expected values of the EEI and ESI indices. Future research would focus more on forecasting the expected value of the two indexes and reflect the consumers' reactions in the markets.

A large-scale evaluation of NLP-derived chemical-gene/protein relationships from the scientific literature: Implications for knowledge graph construction.

One area of active research is the use of natural language processing (NLP) to mine biomedical texts for sets of triples (subject-predicate-object) for knowledge graph (KG) construction. While statistical methods to mine co-occurrences of entities within sentences are relatively robust, accurate relationship extraction is more challenging. Herein, we evaluate the Global Network of Biomedical Relationships (GNBR), a dataset that uses distributional semantics to model relationships between biomedical entities. The focus of our paper is an evaluation of a subset of the GNBR data; the relationships between chemicals and genes/proteins. We use Evotec's structured 'Nexus' database of >2.76M chemical-protein interactions as a ground truth to compare with GNBRs relationships and find a micro-averaged precision-recall area under the curve (AUC) of 0.50 and a micro-averaged receiver operating characteristic (ROC) curve AUC of 0.71 across the relationship classes 'inhibits', 'binding', 'agonism' and 'antagonism', when a comparison is made on a sentence-by-sentence basis. We conclude that, even though these micro-average scores are modest, using a high threshold on certain relationship classes like 'inhibits' could yield high fidelity triples that are not reported in structured datasets. We discuss how different methods of processing GNBR data, and the factuality of triples could affect the accuracy of NLP data incorporated into knowledge graphs. We provide a GNBR-Nexus(ChEMBL-subset) merged datafile that contains over 20,000 sentences where a protein/gene-chemical co-occur and includes both the GNBR relationship scores as well as the ChEMBL (manually curated) relationships (e.g., 'agonist', 'inhibitor') -this can be accessed at https://doi.org/10.5281/zenodo.8136752. We envisage this being used to aid curation efforts by the drug discovery community.

Optimizing Clinical Trial Eligibility Design Using Natural Language Processing Models and Real-World Data: Algorithm Development and Validation.

Clinical trials are vital for developing new therapies but can also delay drug development. Efficient trial data management, optimized trial protocol, and accurate patient identification are critical for reducing trial timelines. Natural language processing (NLP) has the potential to achieve these objectives. This study aims to assess the feasibility of using data-driven approaches to optimize clinical trial protocol design and identify eligible patients. This involves creating a comprehensive eligibility criteria knowledge base integrated within electronic health records using deep learning-based NLP techniques. We obtained data of 3281 industry-sponsored phase 2 or 3 interventional clinical trials recruiting patients with non-small cell lung cancer, prostate cancer, breast cancer, multiple myeloma, ulcerative colitis, and Crohn disease from ClinicalTrials.gov, spanning the period between 2013 and 2020. A customized bidirectional long short-term memory- and conditional random field-based NLP pipeline was used to extract all eligibility criteria attributes and convert hypernym concepts into computable hyponyms along with their corresponding values. To illustrate the simulation of clinical trial design for optimization purposes, we selected a subset of patients with non-small cell lung cancer (n=2775), curated from the Mount Sinai Health System, as a pilot study. We manually annotated the clinical trial eligibility corpus (485/3281, 14.78% trials) and constructed an eligibility criteria-specific ontology. Our customized NLP pipeline, developed based on the eligibility criteria-specific ontology that we created through manual annotation, achieved high precision (0.91, range 0.67-1.00) and recall (0.79, range 0.50-1) scores, as well as a high F1-score (0.83, range 0.67-1), enabling the efficient extraction of granular criteria entities and relevant attributes from 3281 clinical trials. A standardized eligibility criteria knowledge base, compatible with electronic health records, was developed by transforming hypernym concepts into machine-interpretable hyponyms along with their corresponding values. In addition, an interface prototype demonstrated the practicality of leveraging real-world data for optimizing clinical trial protocols and identifying eligible patients. Our customized NLP pipeline successfully generated a standardized eligibility criteria knowledge base by transforming hypernym criteria into machine-readable hyponyms along with their corresponding values. A prototype interface integrating real-world patient information allows us to assess the impact of each eligibility criterion on the number of patients eligible for the trial. Leveraging NLP and real-world data in a data-driven approach holds promise for streamlining the overall clinical trial process, optimizing processes, and improving efficiency in patient identification.

SRL-ACO: A text augmentation framework based on semantic role labeling and ant colony optimization

The process of creating high-quality labeled data is crucial for training machine-learning models, but it can be a time-consuming and labor-intensive process. Moreover, manual annotation by human annotators can lead to varying degrees of competency, training, and experience, which can result in inconsistent labeling and arbitrary standards. To address these challenges, researchers have been exploring automated methods for enhancing training and testing datasets. This paper proposes SRL-ACO, a novel text augmentation framework that leverages Semantic Role Labeling (SRL) and Ant Colony Optimization (ACO) techniques to generate additional training data for natural language processing (NLP) models. The framework uses SRL to identify the semantic roles of words in a sentence and ACO to generate new sentences that preserve these roles. SRL-ACO can enhance the accuracy of NLP models by generating additional data without requiring manual data annotation. The paper presents experimental results demonstrating the effectiveness of SRL-ACO on seven text classification datasets for sentiment analysis, toxic text detection and sarcasm identification. The results show that SRL-ACO improves the performance of a classifier on different NLP tasks. These results demonstrate that SRL-ACO has the potential to enhance the quality and quantity of training data for various NLP tasks.

SURVEY ON COMPREHENSIVE TECHNIQUES OF TEXT DATA AUGMENTATION

In the world of Data Science and machine learning everything starts with data and how well one can use it to get the desired output. But what can be done when enough data is not present. Obviously, overall result will get impacted because of scarcity of data. Hence, to resolve this problem, data augmentation comes into picture which helps to increase dataset size by augmenting it. Data augmentation is an assortment of techniques that facilitates to automatically generate high quality data with the help of existing data. In the field of Natural Language Processing (NLP) is difficult to augment the text due to huge complexity of language. The process of augmenting text data is more challenging and not as straightforward as one might expect. In this study, the way of doing NLP data augmentation and libraries available for it is explored. Data augmentation techniques, comparison and features of the python libraries which can be used for data augmentation is discussed. It will help researchers and data scientist to decide which library to use for their job.

A hybrid system to understand the relations between assessments and plans in progress notes

Objective:The paper presents a novel solution to the 2022 National NLP Clinical Challenges (n2c2) Track 3, which aims to predict the relations between assessment and plan subsections in progress notes. Methods:Our approach goes beyond standard transformer models and incorporates external information such as medical ontology and order information to comprehend the semantics of progress notes. We fine-tuned transformers to understand the textual data and incorporated medical ontology concepts and their relationships to enhance the model’s accuracy. We also captured order information that regular transformers cannot by taking into account the position of the assessment and plan subsections in progress notes. Results:Our submission earned third place in the challenge phase with a macro-F1 score of 0.811. After refining our pipeline further, we achieved a macro-F1 of 0.826, outperforming the top-performing system during the challenge phase. Conclusion:Our approach, which combines fine-tuned transformers, medical ontology, and order information, outperformed other systems in predicting the relationships between assessment and plan subsections in progress notes. This highlights the importance of incorporating external information beyond textual data in natural language processing (NLP) tasks related to medical documentation. Our work could potentially improve the efficiency and accuracy of progress note analysis.

An integrated implicit user preference mining approach for uncertain conceptual design decision-making: A pipeline inspection trolley design case study

User preference mining is one of the most crucial activities in the conceptual design process, aiming to gain customer attentions at the early design stage. Since user preference is usually ambiguous, existing methods of preference extraction and assessment strongly rely on design experience and hard to analyze the implicit design preferences at semantic level. In addition, the abstraction of the knowledge representation prevents a reasonable judgment of the scheme’s potential value. To fill these gaps, by utilizing natural language processing (NLP) technology and semantic network, an integrated implicit design preference (DP) mining approach is proposed to support the uncertain conceptual design decision-making. The proposed approach has three parts: first, the design knowledge semantic network (DKSN) is constructed by using NLP and patent data, which provides a new retrieval mode for function solving; second, TF–IDF model is introduced to extract the DPs, a similarity matrix between the principle solution (PS) retrieved from DKSN and design characteristic (DC) is constructed using word vector model, which can mine the potential satisfaction of various DPs in each PS; and third, based on the satisfaction matrix of PSs, a normalized comprehensive value matrix of the schemes is constructed, and the optimal scheme is obtained using rough VIKOR (VIseKriterijumska Optimizacija I Kompromisno Resenje). A case study of pipeline inspection trolley design is used to validate the proposed approach, sensitivity analysis shows that the proposed model truly captures the scheme’s intention. Additionally, the constructed DKSN has positive implications for assisting designers in exploring the design space.

User Feedback on the Use of a Natural Language Processing Application to Screen for Suicide Risk in the Emergency Department.

Suicide is the 10th leading cause of death in the USA and globally. Despite decades of research, the ability to predict who will die by suicide is still no better than 50%. Traditional screening instruments have helped identify risk factors for suicide, but they have not provided accurate predictive power for reducing death rates. Over the past decade, natural language processing (NLP), a form of machine learning (ML), has been used to identify suicide risk by analyzing language data. Recent work has demonstrated the successful integration of a suicide risk screening interview to collect language data for NLP analysis from patients in two emergency departments (ED) of a large healthcare system. Results indicated that ML/NLP models performed well identifying patients that came to the ED for suicide risk. However, little is known about the clinician's perspective of how a qualitative brief interview suicide risk screening tool to collect language data for NLP integrates into an ED workflow. This report highlights the feedback and observations of patient experiences obtained from clinicians using brief suicide screening interviews. The investigator used an open-ended, narrative interview approach to inquire about the qualitative interview process. Three overarching themes were identified: behavioral health workflow, clinical implications of interview probes, and integration of an application into provider patient experience. Results suggest a brief, qualitative interview method was feasible, person-centered, and useful as a suicide risk detection approach.

Abstract P068: A hybrid modelling approach for abstracting CT imaging indications by integrating natural language processing from radiology reports with structured data from electronic health records

Abstract Background: Real-world evidence (RWE) studies for surveillance patterns following lung cancer (LC) diagnosis can inform optimizing recommendations on surveillance and practice. One major obstacle in RWE studies for LC surveillance is the lack of radiologic imaging indication for surveillance vs. other reasons (e.g., symptoms). To enable RWE studies for surveillance to detect second primary lung cancer among LC survivors, we developed a hybrid modelling approach that integrates structured data from electronic health records (EHRs) with natural language processing (NLP) from radiology reports for abstracting computed tomography (CT) imaging indications in LC survivors. Methods: We manually reviewed and abstracted CT imaging indications, i.e., surveillance vs. others (e.g., symptoms and metastatic disease follow-up) to create a gold standard from 200 randomly selected radiology reports among 1,952 LC patients (i) who were diagnosed in 2000-2017 at Stanford Health Care (SHC) and (ii) survived ≧5 years after the diagnosis. We abstracted medically relevant key-phrases using the part-of-speech grammar and PageRank algorithms. Hierarchical clustering identified context-specific key-phrase clusters as follows: “surveillance”, “stable”, “nodule”, “symptom”, and “metastasis”. The text-based radiology reports were vectorized to generate NLP features using phrase occurrence frequencies. The structured variables from EHRs included: (i) diagnosis of lung diseases or chest symptoms in previous 6 months, (ii) ordering provider-type (oncology vs. others [e.g. emergency and internal medicine]), and (iii) time from previous CT (≧6 months). A hybrid model was then fitted using logistic regression including both structured and NLP features and validated using a 10-fold cross-validation. The model’s performance was compared to those solely based on NLP or structured data. Results: The dataset of 200 radiology reports included 141 LC survivors (49% White, 72% adenocarcinoma). The proposed hybrid model showed high discrimination (AUC: 0.92), outperforming those based solely on NLP (AUC: 0.88) or structured data (AUC: 0.87). The proposed model demonstrated higher sensitivity (SN: 0.73) and specificity (SP: 0.96) versus those solely based on NLP (SN: 0.53; SP: 0.96) or structured data (SN: 0.53; SP: 0.90). The hybrid model showed that the following variables were positively associated with a higher likelihood that the given CT imaging indication is “surveillance”: (i) a longer time interval (≧6 months) from the previous CT (odds ratio [OR]: 4.63; p=0.01) and key-phrases of (ii) “nodule” (OR: 1.55; p=0.004) and (iii) “stable” (OR: 1.37; p=0.03). On the other hand, the following were negatively associated with the likelihood of surveillance: the key-phrases of “symptom” (OR: 0.17; p=0.02) and “metastasis” (OR: 0.26; p=0.02). Conclusion: A hybrid modeling approach combining text-based NLP and structured EHRs has the potential for abstracting CT imaging indications for LC surveillance. Future directions include validation using other EHR systems and extension using larger data. Citation Format: Aparajita Khan, Julie Wu, Eunji Choi, Anna Graber-Naidich, Solomon Henry, Heather A. Wakelee, Allison W. Kurian, Su-Ying Liang, Ann Leung, Curtis Langlotz, Leah M. Backhus, Summer S. Han. A hybrid modelling approach for abstracting CT imaging indications by integrating natural language processing from radiology reports with structured data from electronic health records. [abstract]. In: Proceedings of the AACR Special Conference: Precision Prevention, Early Detection, and Interception of Cancer; 2022 Nov 17-19; Austin, TX. Philadelphia (PA): AACR; Can Prev Res 2023;16(1 Suppl): Abstract nr P068.

ViA: A Novel Vision-Transformer Accelerator Based on FPGA

Since Google proposed Transformer in 2017, it has made significant natural language processing (NLP) development. However, the increasing cost is a large amount of calculation and parameters. Previous researchers designed and proposed some accelerator structures for transformer models in field-programmable gate array (FPGA) to deal with NLP tasks efficiently. Now, the development of Transformer has also affected computer vision (CV) and has rapidly surpassed convolution neural networks (CNNs) in various image tasks. And there are apparent differences between the image data used in CV and the sequence data in NLP. The details in the models contained with transformer units in these two fields are also different. The difference in terms of data brings about the problem of the locality. The difference in the model structure brings about the problem of path dependence, which is not noticed in the existing related accelerator design. Therefore, in this work, we propose the ViA, a novel vision transformer (ViT) accelerator architecture based on FPGA, to execute the transformer application efficiently and avoid the cost of these challenges. By analyzing the data structure in the ViT, we design an appropriate partition strategy to reduce the impact of data locality in the image and improve the efficiency of computation and memory access. Meanwhile, by observing the computing flow of the ViT, we use the half-layer mapping and throughput analysis to reduce the impact of path dependence caused by the shortcut mechanism and fully utilize hardware resources to execute the Transformer efficiently. Based on optimization strategies, we design two reuse processing engines with the internal stream, different from the previous overlap or stream design patterns. In the stage of the experiment, we implement the ViA architecture in Xilinx Alveo U50 FPGA and finally achieved ~5.2 times improvement of energy efficiency compared with NVIDIA Tesla V100, and 4–10 times improvement of performance compared with related accelerators based on FPGA, that obtained nearly 309.6 GOP/s computing performance in the peek.

Model Based Approach to Extract Health Information from Textual Data

In current nuclear power plants (NPPs) a large amount of condition-based data is being generated and stored to assess and monitor component health and performance. The format of this data can be either numeric (e.g., pump vibration data) or textual (e.g., condition report which assess component health). While assessing component health from numeric data can be performed with a large variety of methods, the extraction of information from textual data still remains a challenge. Natural language processing (NLP) methods are starting to be deployed in current NPPs mainly to filter out incident reports (IRs) that are not safety related by employing supervised machine learning methods. However, these methods do not really provide the quantitative information that might be contained in IRs. This paper presents an approach to extract information from textual data (e.g., from IRs, maintenance reports) that is based on NLP data analytics methods coupled with model based system engineer (MBSE) models. NLP methods are employed to perform syntactic and semantic analyses. Syntactic analysis analyzes the grammatical structure of a sentence; such analysis includes: part of speech (POS) tagging (i.e., identification of grammatic elements of each string - e.g., nouns, verbs), named entity recognition (i.e., identification of text entities - e.g., names, dates, events), and relation extraction (e.g., coreference resolution). On the other hand, semantic analysis is designed to analyze the logic structure of a sentence. Through a specific set of rules, our methods can identify whether a sentence contains health information of a component (e.g., degraded performance, anomaly behavior) or the causal relationship between two events (i.e., a cause-effect pair). An innovative element of our approach is that semantic analysis relies on MBSE models to identify links between textual elements. MBSE are diagrams designed to represent system and component dependencies (from both a form and functional point of view). In our approach, MBSE models emulate system engineer knowledge about component/system architecture. This paper presents in detail how the integration of NLP methods and MBSE models is performed. Few analysis examples focusing on centrifugal pumps will be presented.

Natural Language Processing to Improve Prediction of Incident Atrial Fibrillation Using Electronic Health Records.

BackgroundModels predicting atrial fibrillation (AF) risk, such as Cohorts for Heart and Aging Research in Genomic Epidemiology AF (CHARGE‐AF), have not performed as well in electronic health records. Natural language processing (NLP) may improve models by using narrative electronic health record text.Methods and ResultsFrom a primary care network, we included patients aged ≥65 years with visits between 2003 and 2013 in development (n=32 960) and internal validation cohorts (n=13 992). An external validation cohort from a separate network from 2015 to 2020 included 39 051 patients. Model features were defined using electronic health record codified data and narrative data with NLP. We developed 2 models to predict 5‐year AF incidence using (1) codified+NLP data and (2) codified data only and evaluated model performance. The analysis included 2839 incident AF cases in the development cohort and 1057 and 2226 cases in internal and external validation cohorts, respectively. The C‐statistic was greater (P<0.001) in codified+NLP model (0.744 [95% CI, 0.735–0.753]) compared with codified‐only (0.730 [95% CI, 0.720–0.739]) in the development cohort. In internal validation, the C‐statistic of codified+NLP was modestly higher (0.735 [95% CI, 0.720–0.749]) compared with codified‐only (0.729 [95% CI, 0.715–0.744]; P=0.06) and CHARGE‐AF (0.717 [95% CI, 0.703–0.731]; P=0.002). Codified+NLP and codified‐only were well calibrated, whereas CHARGE‐AF underestimated AF risk. In external validation, the C‐statistic of codified+NLP (0.750 [95% CI, 0.740–0.760]) remained higher (P<0.001) than codified‐only (0.738 [95% CI, 0.727–0.748]) and CHARGE‐AF (0.735 [95% CI, 0.725–0.746]).ConclusionsEstimation of 5‐year risk of AF can be modestly improved using NLP to incorporate narrative electronic health record data.

NLP-based platform as a service: a brief review

Natural language processing (NLP) refers to the field of study that focuses on the interactions between human language and computers. It has recently gained much attention for analyzing human language computationally and has spread its applications for various tasks such as machine translation, information extraction, summarization, question answering, and others. With the rapid growth of cloud computing services, merging NLP in the cloud is a significant benefit. It allows researchers to conduct NLP-related experiments on large amounts of data handled by big data techniques while harnessing the cloud’s vast, on-demand computing power. However, it has not sufficiently spread its tools and applications as a service in the cloud and there is little literature available that discusses the scope of interdisciplinary work. NLP, cloud Computing, and big data are vast domains and contain their challenges and potentials. By overcoming those challenges and integrating these fields, great potential for NLP and its applications can be unleashed. This paper presents a survey of NLP in cloud computing with a key focus on the comparison of cloud-based NLP services, challenges of NLP and big data while emphasizing the necessity of viable cloud-based NLP services. In the first part of this paper, an overview of NLP is presented by discussing different levels of NLP and components of natural language generation (NLG), followed by the applications of NLP. In the second part, the concept of cloud computing is discussed that highlights the architectural layers and deployment models of cloud computing and cloud-hosted NLP services. In the third part, the field of big data in the cloud is discussed with an emphasis on NLP. Furthermore, information extraction via NLP techniques within big data is introduced.