Transformer-based Models Research Articles

Image restoration through few-mode fiber using a new comprehensive attention model

Few-mode fibers (FMF) has unique advantages in single fiber imaging (SFI) in special applications. However, the limited number of transmission modes restricts the information carriers, hence the high-quality image restoration is a major challenge. A deep learning network with new comprehensive attention, called few-mode fiber Swin Transformer (FFST), is developed based on Swin Transformer V2 to realize successful image restoration with a FMF on handwriting letters and numbers. This network is proved outperforming nine widely used convolutional neural networks or Transformer-based models with four evaluation criteria. In addition, single-fiber-multi-location (SFML) experiment is conducted to investigate the effect of different view on imaging feasibility, the result of which shows that SFML can improve the restoration quality despite one object corresponds to multiple different speckle images, indicating that multi-mapping relationships has constructive effect in single FMF imaging. This paper provides a method for imaging in very narrow cavities.

Sentiment analysis of student evaluation feedback using transformer-based language models

Sentiment analysis of student evaluation feedback using transformer-based language models

Transformer-Based Model Predictive Control: Trajectory Optimization via Sequence Modeling

Transformer-Based Model Predictive Control: Trajectory Optimization via Sequence Modeling

The Role of Sentiment Analysis in Machine Translation- A Review on the Example of ChatGPT

In today’s globalization, machine translation technology has become an indispensable bridge for cross-lingual communication. The integration of sentiment analysis makes machine translation not only able to convey information, but also able to accurately express and understand the emotional color of the original text, which is of great significance in the fields of cross-cultural communication, customer service, and social media interaction. This study overviews the development of machine translation technology, from early rule-based models to modern statistical models to Transformer-based neural network models, with a special focus on the integrated application of ChatGPT in sentiment analysis and machine translation. First, this study introduces the background and research significance of machine translation, and discusses the role of sentiment analysis in improving translation quality and user satisfaction. Subsequently, the historical development of machine translation technology is reviewed in detail, and the basic concepts and methodologies of sentiment analysis and its applications in different fields are elaborated. On this basis, this study analyzes in depth the technical architecture, language model and interaction capability of ChatGPT, as well as its specific application and performance in sentiment analysis. Further, this study explores the importance of sentiment analysis in machine translation, including sentiment recognition, translation strategies for sentiment expression, and the impact of sentiment analysis on improving translation quality and cultural adaptability. Meanwhile, the advantages and limitations of ChatGPT in sentiment analysis are assessed, and future research directions are proposed, including technology convergence, construction of datasets and evaluation criteria, and ethical and bias issues. Finally, this study summarizes the practical application value of sentiment analysis in machine translation and looks forward to the potential areas of future research, aiming to provide reference and inspiration for the further development of machine translation and sentiment analysis.

A flexible, calibrated transformer-based risk model for prediction of clinical outcomes in patients with heart failure; an open cohort study using linked UK EHR

Abstract Background Numerous risk scores, both uni- and multi-variable, have been developed to conduct prognostication for a range of clinical outcomes in heart failure (HF) patients [1,2]. However, these models provide only modest discrimination and often depend on complex test data (e.g., ejection fraction) collected outside of routine clinical care [2]. Furthermore, transparency in reporting of key individual-level metrics (e.g., precision, recall) is notably lacking. Purpose We developed and validated a novel deep learning model, the second version of the Transformer-based Risk assessment survival model (TRisk2), for 36-month prediction of all-cause mortality, cardiovascular-related mortality, fatal and non-fatal cardiovascular events, and renal outcomes in patients with HF using routine, linked UK electronic health records (EHR) data. Methods An open cohort of 405 thousand patients with HF between 40 and 90 years of age was identified using linked EHR from 1,063 and 355 English general practices, which were used for TRisk2 model development and external validation, respectively. Comparison was conducted against MAGGIC [2], modified for use on routine linked EHR. Additional analyses compared discriminatory performance in other age groups, by sex, and by baseline disease. All analyses were repeated for prediction of cardiovascular-related mortality, fatal and non-fatal cardiovascular events, and renal outcomes. Results TRisk2 demonstrated superior discrimination with Concordance index (C-index) of 0.828; 95% confidence interval (CI): 0.824 to 0.832 for all-cause mortality prediction outperforming MAGGIC. The proposed model’s performance was found to be stable across stratified analyses by sex, age, and baseline disease status. Both models were overall well-calibrated, and TRisk2 demonstrated greater net benefit than MAGGIC across reasonable decision boundaries in decision curve analyses. At exemplar risk threshold of 50% for 36-month prediction, TRisk2 outperformed MAGGIC by 0.08 for both, precision and recall and identified 10% more events. At the same threshold for 12-month prediction, TRisk2 outperformed MAGGIC by 0.14 and 0.26 precision and recall respectively and captured 70% more events. The well-calibrated TRisk2 similarly outperformed MAGGIC in prediction of cardiovascular-related mortality, fatal and non-fatal cardiovascular events, and renal outcomes. Conclusion Utilising solely routine EHR from primary and secondary care, TRisk2 enabled more accurate prediction than MAGGIC with 10-20% gain in C-index across the four outcome investigations. Integration of TRisk2 into routine clinical care could simultaneously eliminate the reliance of complex tests and improve prognostication of key clinical outcomes, thereby refining management of HF.Models' discrimination on four outcomes

Refined selection of individuals for preventive cardiovascular disease treatment with a transformer-based risk model

Abstract Background Although statistical models have been commonly used to identify patients at risk of cardiovascular disease (CVD) for preventive therapy, these models tend to over-recommend therapy [1]. Moreover, in populations with pre-existing diseases, the current approach is to indiscriminately treat all individuals, as modelling in this context is currently inadequate. Aims To develop and validate a novel deep learning model, the Transformer-based Risk assessment survival model (TRisk) model, for prediction of 10-year risk of CVD in both the general population and individuals with diabetes. Methods An open cohort of 3 million adults aged 25 to 84 years free of CVD and statin use was identified using linked primary and secondary electronic health records from 291 and 98 general practices in England, which were used for TRisk model development and external validation, respectively. Comparison against two commonly recommended statistical models (QRISK3 [2] and SCORE2 [3]) and a deep learning derivation of QRISK3 was conducted. Additional analyses compared discriminatory performance in other age groups, by sex, and across categories of socioeconomic status. Similar analyses were conducted in a subgroup of 59 thousand patients with diabetes. Results TRisk demonstrated superior discrimination (C index in the general population: 0.910; 95% confidence interval [CI]: 0.906 to 0.913). TRisk’s performance was found to be less sensitive to population age range than the benchmark models and outperformed other models also in all stratified analyses. All models were overall well-calibrated. In decision curve analyses, TRisk demonstrated greater net benefit than benchmark models across the range of relevant thresholds. In impact analyses, at the widely recommended 10% risk threshold and the higher 15% threshold for the general population, TRisk reduced both the total number of patients classified at high risk (by 27% and 39% respectively) and the number of false negatives as compared with currently recommended strategies. TRisk similarly outperformed other models in patients with diabetes. Compared with the widely recommended treat-all policy approach for patients with diabetes, TRisk at a 10% risk threshold would lead to deselection of 24% of individuals with a small fraction of false negatives (0.2% of cohort). Conclusion TRisk enabled a more targeted selection of individuals at risk of CVD compared to benchmark statistical and deep learning models, in both the general population and patients with diabetes. Incorporation of TRisk into routine clinical care would allow a reduction in the number of treatment-eligible patients in general population and subgroup with diabetes by approximately one-third and one-quarter respectively while preventing at least as many events as with currently adopted approaches.Models' discrimination on both cohortsImpact analyses for both cohorts

ETFT: Equiangular Tight Frame Transformer for Imbalanced Semantic Segmentation

Semantic segmentation often suffers from class imbalance, where the label ratio for each class in the dataset is not uniform. Recent studies have addressed the issue of class imbalance in semantic segmentation by leveraging the neural collapse phenomenon in conjunction with an Equiangular Tight Frame (ETF). While the use of ETF aids in enhancing the discriminability of minor classes, class correlation is another crucial factor that must be taken into account. However, managing the balance between class correlation and discrimination through neural collapse remains challenging, as these properties inherently conflict with one another. Moreover, this control is established during the training stage, resulting in a fixed classifier. There is no guarantee that this classifier will consistently perform well with different input images. To address this problem, we propose an Equiangular Tight Frame Transformer (ETFT), a transformer-based model that jointly processes the features and classifier using ETF structure, and dynamically generates the classifier as a function of the input for imbalanced semantic segmentation. Specifically, the classifier initialized with the ETF structure is jointly processed with the input patch tokens during the attention process. As a result, the transformed patch tokens, aided by the ETF structure, achieve discriminability between classes while preserving contextual correlation. The classifier, initially structured as an ETF, is adjusted to incorporate the correlation information, benefiting from the attention mechanism. Furthermore, the learned classifier is combined with the fixed ETF classifier, leveraging the advantages of both. Extensive experiments demonstrate that the proposed method outperforms state-of-the-art methods for imbalanced semantic segmentation on both the ADE20K and Cityscapes datasets.

Identifying heart failure dynamics using multi-point electrocardiograms and deep learning

Abstract Background Heart failure (HF) hospitalizations are related to poor survival outcomes, emphasizing the need for early detection and intervention. Recent studies have reported the potential of deep learning algorithms in identifying heart failure through electrocardiogram (ECG) with notable accuracy. However, evidence supporting their utility in the ongoing monitoring and follow-up of HF patients remains uncertain. Advanced diagnostic tools that can effectively track heart failure progression or improvement over time are needed. Purpose To address this challenge, we developed a deep learning model analyzing 12-lead ECG waveforms at two time points, aiming to detect HF deterioration or improvement, potentially revolutionizing HF management. Methods This study analyzed 32,045 ECGs from 6,892 unique adult patients (aged 18 years and older) visiting the Department of Cardiology at Kobe University Hospital. The participants were randomly assigned to training (4,284 patients), validation (1,034 patients), and test (1,034 patients) datasets. ElecTransformer was developed to classify HF status into deteriorated, improved, and no change classes based on ECG waveform signals at two different time points and to estimate brain natriuretic peptide (BNP) levels. Performance metrics, such as the area under the receiver operating characteristic curve (AUROC) and accuracy, were calculated, and attention mapping via gradient-weighted class activation mapping was utilized to interpret the model's decision-making ability. Results The patients had an average age of 65 years (±15.4 years) and BNP levels averaged 71.5 pg/mL (25.8 to 185.2 pg/mL). For HF status classification, ElecTransformer achieved an AUROC of 0.845 and an accuracy of 0.821, improving to an AUROC of 0.885 and an accuracy of 0.833 with baseline BNP integration. The ElecTransformer accurately estimated BNP levels from ECG data, evidenced by a correlation coefficient of 0.798. Averaged attention mapping showed that the QRS complex was the most important feature for determining model outputs with lesser effects seen for P waves and T waves variance Conclusions Harnessing a Transformer-based deep learning model to analyze multiple ECGs markedly improves the accuracy in detecting changes in heart failure status. This method offers a valuable non-invasive option for monitoring heart failure progression, potentially leading to better patient management and earlier intervention in heart failure treatment.Architecture and Performance of ModelsAveraged attention mapping

ABTrans: A Transformer-based Model for Predicting Interaction between Anti-Aβ Antibodies and Peptides.

Antibodies against Aβ peptide have been recently approved to treat Alzheimer's disease, underscoring the importance of understanding their interactions for developing more potent treatments. Here we investigated the interaction between anti-Aβ antibodies and various peptides using a deep learning model. Our model, ABTrans, was trained on dodecapeptide sequences from phage display experiments and known anti-Aβ antibody sequences sourced from public sources. It classified the binding ability between anti-Aβ antibodies and dodecapeptides into four levels: not binding, weak binding, medium binding, and strong binding, achieving an accuracy of 0.83. Using ABTrans, we examined the cross-reaction of anti-Aβ antibodies with other human amyloidogenic proteins, revealing that Aducanumab and Donanemab exhibited the least cross-reactivity. Additionally, we systematically screened interactions between eleven selected anti-Aβ antibodies and all human proteins to identify potential off-target candidates.

Addressing the antibody germline bias and its effect on language models for improved antibody design.

The versatile binding properties of antibodies have made them an extremely important class of biotherapeutics. However, therapeutic antibody development is a complex, expensive and time-consuming task, with the final antibody needing to not only have strong and specific binding, but also be minimally impacted by developability issues. The success of transformer-based language models in protein sequence space and the availability of vast amounts of antibody sequences, has led to the development of many antibody-specific language models to help guide antibody design. Antibody diversity primarily arises from V(D)J recombination, mutations within the CDRs, and/or from a few non-germline mutations outside the CDRs. Consequently, a significant portion of the variable domain of all natural antibody sequences remains germline. This affects the pre-training of antibody-specific language models, where this facet of the sequence data introduces a prevailing bias towards germline residues. This poses a challenge, as mutations away from the germline are often vital for generating specific and potent binding to a target, meaning that language models need be able to suggest key mutations away from germline. In this study, we explore the implications of the germline bias, examining its impact on both general-protein and antibody-specific language models. We develop and train a series of new antibody-specific language models optimised for predicting non-germline residues. We then compare our final model, AbLang-2, with current models and show how it suggests a diverse set of valid mutations with high cumulative probability. AbLang-2 is trained on both unpaired and paired data, and is freely available at https://github.com/oxpig/AbLang2.git. Supplementary data are available at Journal Name online.

Cryptocurrency Transaction Fraud Detection Based on Imbalanced Classification With Interpretable Analysis

This study introduces an interpretable imbalanced data classification method for detecting cryptocurrency transaction fraud. We address data imbalance using SMOTE oversampling and data augmentation through contrastive learning. Next, we introduce a Transformer-based deep learning model that learns sample relevance. The model undergoes pre-training with a contrastive loss and fine-tuning through Bayesian optimization to effectively extract high-dimensional, higher-order, and fraud-related features. We employ a SHAP-based interpreter along with attention scores to elucidate the role of various transaction features in fraud detection. Comparative results demonstrate the model's remarkable recall performance in identifying cryptocurrency transaction fraud. Furthermore, it achieves an excellent F1 value, striking a balance between accuracy and recall. This research not only enriches financial fraud detection but also enhances cryptocurrency transaction security, promotes market development, and contributes to economic stability and social security.

Topic aware probing: From sentence length prediction to idiom identification how reliant are neural language models on topic?

Abstract Transformer-based neural language models achieve state-of-the-art performance on various natural language processing tasks. However, an open question is the extent to which these models rely on word-order/syntactic or word co-occurrence/topic-based information when processing natural language. This work contributes to this debate by addressing the question of whether these models primarily use topic as a signal, by exploring the relationship between Transformer-based models’ (BERT and RoBERTa’s) performance on a range of probing tasks in English, from simple lexical tasks such as sentence length prediction to complex semantic tasks such as idiom token identification, and the sensitivity of these tasks to the topic information. To this end, we propose a novel probing method which we call topic-aware probing. Our initial results indicate that Transformer-based models encode both topic and non-topic information in their intermediate layers, but also that the facility of these models to distinguish idiomatic usage is primarily based on their ability to identify and encode topic. Furthermore, our analysis of these models’ performance on other standard probing tasks suggests that tasks that are relatively insensitive to the topic information are also tasks that are relatively difficult for these models.

Point-MPP: Point Cloud Self-Supervised Learning From Masked Position Prediction.

Masked autoencoding has gained momentum for improving fine-tuning performance in many downstream tasks. However, it tends to focus on low-level reconstruction details, lacking high-level semantics and resulting in weak transfer capability. This article presents a novel jigsaw puzzle solver inspired by the idea that predicting the positions of disordered point cloud patches provides more semantic information, similar to how children learn by solving jigsaw puzzles. Our method adopts the mask-then-predict paradigm, erasing the positions of selected point patches rather than their contents. We first partition input point clouds into irregular patches and randomly erase the positions of some patches. Then, a Transformer-based model is used to learn high-level semantic features and regress the positions of the masked patches. This approach forces the model to focus on learning transfer-robust semantics while paying less attention to low-level details. To tie the predictions within the encoding space, we further introduce a consistency constraint on their latent representations to encourage the encoded features to contain more semantic cues. We demonstrate that a standard Transformer backbone with our pretraining scheme can capture discriminative point cloud semantic information. Furthermore, extensive experiments indicate that our method outperforms the previous best competitor across six popular downstream vision tasks, achieving new state-of-the-art performance. Codes will be available at https://git.openi.org.cn/OpenPointCloud/Point-MPP.

Aggressiveness classification of clear cell renal cell carcinoma using registration-independent radiology-pathology correlation learning.

Renal cell carcinoma (RCC) is a common cancer that varies in clinical behavior. Clear cell RCC (ccRCC) is the most common RCC subtype, with both aggressive and indolent manifestations. Indolent ccRCC is often low-grade without necrosis and can be monitored without treatment. Aggressive ccRCC is often high-grade and can cause metastasis and death if not promptly detected and treated. While most RCCs are detected on computed tomography (CT) scans, aggressiveness classification is based on pathology images acquired from invasive biopsy orsurgery. CT imaging-based aggressiveness classification would be an important clinical advance, as it would facilitate non-invasive risk stratification and treatment planning. Here, we present a novel machine learning method, Correlated Feature Aggregation By Region (CorrFABR), for CT-based aggressiveness classification ofccRCC. CorrFABR is a multimodal fusion algorithm that learns from radiology and pathology images, and clinical variables in a clinically-relevant manner. CorrFABR leverages registration-independent radiology (CT) and pathology image correlations using features from vision transformer-based foundation models to facilitate aggressiveness assessment on CT images. CorrFABR consists of three main steps: (a) Feature aggregation where region-level features are extracted from radiology and pathology images at widely varying image resolutions, (b) Fusion where radiology features correlated with pathology features (pathology-informed CT biomarkers) are learned, and (c) Classification where the learned pathology-informed CT biomarkers, together with clinical variables of tumor diameter, gender, and age, are used to distinguish aggressive from indolent ccRCC using multi-layer perceptron-based classifiers. Pathology images are only required in the first two steps of CorrFABR, and are not required in the prediction module. Therefore, CorrFABR integrates information from CT images, pathology images, and clinical variables during training, but for inference, it relies solely on CT images and clinical variables, ensuring its clinical applicability. CorrFABR was trained with heterogenous, publicly-available data from 298 ccRCC tumors (136 indolent tumors, 162 aggressive tumors) in a five-fold cross-validation setup and evaluated on an independent test set of 74 tumors with a balanced distribution of aggressive and indolent tumors. Ablation studies were performed to test the utility of each component ofCorrFABR. CorrFABR outperformed the other classification methods, achieving an ROC-AUC (area under the curve) of 0.855±0.0005 (95% confidence interval: 0.775, 0.947), F1-score of 0.793±0.029, sensitivity of 0.741±0.058, and specificity of 0.876±0.032 in classifying ccRCC as aggressive or indolent subtypes. It was found that pathology-informed CT biomarkers learned through registration-independent correlation learning improves classification performance over using CT features alone, irrespective of the kind of features or the classification model used. Tumor diameter, gender, and age provide complementary clinical information, and integrating pathology-informed CT biomarkers with these clinical variables further improvesperformance. CorrFABR provides a novel method for CT-based aggressiveness classification of ccRCC by enabling the identification of pathology-informed CT biomarkers, and integrating them with clinical variables. CorrFABR enables learning of these pathology-informed CT biomarkers through a novel registration-independent correlation learning module that considers unaligned radiology and pathology images at widely varying imageresolutions.

Advanced neural network-based model for predicting court decisions on child custody

Predicting court rulings has gained attention over the past years. The court rulings are among the most important documents in all legal systems, profoundly impacting the lives of the children in case of divorce or separation. It is evident from literature that Natural language processing (NLP) and machine learning (ML) are widely used in the prediction of court rulings. In general, the court decisions comprise several pages and require a lot of space. In addition, extracting valuable information and predicting legal decisions task is difficult. Moreover, the legal system’s complexity and massive litigation make this problem more serious. Thus to solve this issue, we propose a new neural network-based model for predicting court decisions on child custody. Our proposed model efficiently performs an efficient search from a massive court decisions database and accurately identifies specific ones that especially deal with copyright claims. More specially, our proposed model performs a careful analysis of court decisions, especially on child custody, and pinpoints the plaintiff’s custody request, the court’s ruling, and the pivotal arguments. The working mechanism of our proposed model is performed in two phases. In the first phase, the isolation of pertinent sentences within the court ruling encapsulates the essence of the proceedings performed. In the second phase, these documents were annotated independently by using two legal professionals. In this phase, NLP and transformer-based models were employed and thus processed 3,000 annotated court rulings. We have used a massive dataset for the training and refining of our proposed model. The novelty of the proposed model is the integration of bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (Bi_LSTM). The traditional methods are primarily based on support vector machines (SVM), and logistic regression. We have performed a comparison with the state-of-the-art model. The efficient results indicate that our proposed model efficiently navigates the complex terrain of legal language and court decision structures. The efficiency of the proposed model is measured in terms of the F1 score. The achieved results show that scores range from 0.66 to 0.93 and Kappa indices from 0.57 to 0.80 across the board. The performance is achieved at times surpassing the inter-annotator agreement, underscoring the model’s adeptness at extracting and understanding nuanced legal concepts. The efficient results proved the potential of the proposed neural network model, particularly those based on transformers, to effectively discern and categorize key elements within legal texts, even amidst the intricacies of judicial language and the layered complexity of appellate rulings.

Anomaly detection in smart manufacturing: An Adaptive Adversarial Transformer-based model

In Industry 5.0, smart manufacturing brings additional intricacies and novel data processing challenges. Given the evolving nature of manufacturing processes and the inherent complexity of data, including noise and missing entries, achieving accurate anomaly detection becomes even more intricate. Conventional methods often miss nuanced anomalies, especially when dealing with high-dimensional, multivariate, non-stationary data. These data types are typical of smart manufacturing environments. Hence, many recent approaches have embraced deep learning to confront these challenges, making use of diverse attention mechanisms to acquire data representations. However, in manufacturing, where the dynamics of time series data change over time, methods relying solely on pointwise or pairwise representations often fall short. Thus, ensuring product quality and operational integrity calls for even more advanced methodologies. The deficiency lies in the capability of state-of-the-art models to effectively capture abnormal patterns while considering both local and global contextual information. This challenge is compounded by the rarity of anomalies, making it exceedingly challenging to establish substantial associations between individual abnormal points and the entire time series. To tackle these challenges, we introduce the “Adaptive Adversarial Transformer” as a novel deep learning technique that combines Transformer architecture with an anomaly attention mechanism and Adversarial Learning. Our Model effectively captures intricate temporal patterns, distinguishes normal and anomalous behaviors, and dynamically adjusts thresholds to align with the evolving dynamics of time-series data. Empirical validation on four benchmark datasets and three real-world manufacturing datasets demonstrates our model’s effectiveness compared to the state-of-the-art, as evidenced by the F1-Score.

A survey of recent approaches to form understanding in scanned documents

This paper presents a comprehensive survey of over 100 research works on the topic of form understanding in the context of scanned documents. We delve into recent advancements and breakthroughs in the field, with particular focus on transformer-based models, which have been shown to improve performance in form understanding tasks by up to 25% in accuracy compared to traditional methods. Our research methodology involves an in-depth analysis of popular documents and trends over the last decade, including 15 state-of-the-art models and 10 benchmark datasets. By examining these works, we offer novel insights into the evolution of this domain. Specifically, we highlight how transformers have revolutionized form-understanding techniques by enhancing the ability to process noisy scanned documents with significant improvements in OCR accuracy. Furthermore, we present an overview of the most relevant datasets, such as FUNSD, CORD, and SROIE, which serve as benchmarks for evaluating the performance of the models. By comparing the capabilities of these models and reporting an average improvement of 10–15% in key form extraction tasks, we aim to provide researchers and practitioners with useful guidance in selecting the most suitable solutions for their form understanding applications.

Health Care Language Models and Their Fine-Tuning for Information Extraction: Scoping Review.

In response to the intricate language, specialized terminology outside everyday life, and the frequent presence of abbreviations and acronyms inherent in health care text data, domain adaptation techniques have emerged as crucial to transformer-based models. This refinement in the knowledge of the language models (LMs) allows for a better understanding of the medical textual data, which results in an improvement in medical downstream tasks, such as information extraction (IE). We have identified a gap in the literature regarding health care LMs. Therefore, this study presents a scoping literature review investigating domain adaptation methods for transformers in health care, differentiating between English and non-English languages, focusing on Portuguese. Most specifically, we investigated the development of health care LMs, with the aim of comparing Portuguese with other more developed languages to guide the path of a non-English-language with fewer resources. This study aimed to research health care IE models, regardless of language, to understand the efficacy of transformers and what are the medical entities most commonly extracted. This scoping review was conducted using the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) methodology on Scopus and Web of Science Core Collection databases. Only studies that mentioned the creation of health care LMs or health care IE models were included, while large language models (LLMs) were excluded. The latest were not included since we wanted to research LMs and not LLMs, which are architecturally different and have distinct purposes. Our search query retrieved 137 studies, 60 of which met the inclusion criteria, and none of them were systematic literature reviews. English and Chinese are the languages with the most health care LMs developed. These languages already have disease-specific LMs, while others only have general-health care LMs. European Portuguese does not have any public health care LM and should take examples from other languages to develop, first, general-health care LMs and then, in an advanced phase, disease-specific LMs. Regarding IE models, transformers were the most commonly used method, and named entity recognition was the most popular topic, with only a few studies mentioning Assertion Status or addressing medical lexical problems. The most extracted entities were diagnosis, posology, and symptoms. The findings indicate that domain adaptation is beneficial, achieving better results in downstream tasks. Our analysis allowed us to understand that the use of transformers is more developed for the English and Chinese languages. European Portuguese lacks relevant studies and should draw examples from other non-English languages to develop these models and drive progress in AI. Health care professionals could benefit from highlighting medically relevant information and optimizing the reading of the textual data, or this information could be used to create patient medical timelines, allowing for profiling.

JST and rJST: joint estimation of sentiment and topics in textual data using a semi-supervised approach

ABSTRACT This paper demonstrates the performance of the Joint Sentiment Topic model (JST) and the reversed Joint Sentiment Topic model (rJST) in measuring sentiment in political speeches, comparing them against a set of popular methods for sentiment analysis: widely used off-the-shelf sentiment dictionaries; an embeddings-enhanced dictionary approach; Latent Semantic Scaling, a semi-supervised approach; and a zero-shot transformer-based approach using a large language model (GPT-4). The findings reveal JST’s superiority over all non-transformer-based approaches in predicting human-coded sentiment in multiple languages and its ability to replicate known sentiment trends in legislative speech. rJST, meanwhile, provides valuable topic-specific sentiment estimates, responsive to political dynamics and significant events. Both models are, however, outperformed by transformer-based models like GPT-4. Additionally, the paper introduces the ’sentitopics’ R-package, designed to facilitate the use of JST and rJST in computational text analysis workflows. This package is compatible with popular text analysis tools, making the models accessible for applied researchers in communication science.

Multi-scale dual-channel feature embedding decoder for biomedical image segmentation

Background and Objective:Attaining global context along with local dependencies is of paramount importance for achieving highly accurate segmentation of objects from image frames and is challenging while developing deep learning-based biomedical image segmentation. Several transformer-based models have been proposed to handle this issue in biomedical image segmentation. Despite this, segmentation accuracy remains an ongoing challenge, as these models often fall short of the target range due to their limited capacity to capture critical local and global contexts. However, the quadratic computational complexity is the main limitation of these models. Moreover, a large dataset is required to train those models. Methods:In this paper, we propose a novel multi-scale dual-channel decoder to mitigate this issue. The complete segmentation model uses two parallel encoders and a dual-channel decoder. The encoders are based on convolutional networks, which capture the features of the input images at multiple levels and scales. The decoder comprises a hierarchy of Attention-gated Swin Transformers with a fine-tuning strategy. The hierarchical Attention-gated Swin Transformers implements a multi-scale, multi-level feature embedding strategy that captures short and long-range dependencies and leverages the necessary features without increasing computational load. At the final stage of the decoder, a fine-tuning strategy is implemented that refines the features to keep the rich features and reduce the possibility of over-segmentation. Results:The proposed model is evaluated on publicly available LiTS, 3DIRCADb, and spleen datasets obtained from Medical Segmentation Decathlon. The model is also evaluated on a private dataset from Medical College Kolkata, India. We observe that the proposed model outperforms the state-of-the-art models in liver tumor and spleen segmentation in terms of evaluation metrics at a comparative computational cost. Conclusion:The novel dual-channel decoder embeds multi-scale features and creates a representation of both short and long-range contexts efficiently. It also refines the features at the final stage to select only necessary features. As a result, we achieve better segmentation performance than the state-of-the-art models.