Transformer-based Architecture Research Articles

Multi-label text classification via secondary use of large clinical real-world data sets.

Procedural coding presents a taxing challenge for clinicians. However, recent advances in natural language processing offer a promising avenue for developing applications that assist clinicians, thereby alleviating their administrative burdens. This study seeks to create an application capable of predicting procedure codes by analysing clinicians' operative notes, aiming to streamline their workflow and enhance efficiency. We downstreamed an existing and a native German medical BERT model in a secondary use scenario, utilizing already coded surgery notes to model the coding procedure as a multi-label classification task. In comparison to the transformer-based architecture, we were levering the non-contextual model fastText, a convolutional neural network, a support vector machine and logistic regression for a comparative analysis of possible coding performance. About 350,000 notes were used for model adaption. By considering the top five suggested procedure codes from medBERT.de, surgeryBERT.at, fastText, a convolutional neural network, a support vector machine and a logistic regression, the mean average precision achieved was 0.880, 0.867, 0.870, 0.851, 0.870and0.805 respectively. Support vector machines performed better for surgery reports with a sequence length greater than 512, achieving a mean average precision of 0.872 in comparison to 0.840 for fastText, 0.837 for medBERT.de and 0.820 for surgeryBERT.at. A prototypical front-end application for coding support was additionally implemented. The problem of predicting procedure codes from a given operative report can be successfully modelled as a multi-label classification task, with a promising performance. Support vector machines as a classical machine learning method outperformed the non-contextual fastText approach. FastText with less demanding hardware resources has reached a similar performance to BERT-based models and has shown to be more suitable for explaining the predictions efficiently.

A unified framework for unsupervised action learning via global-to-local motion transformer

Human action recognition remains challenging due to the inherent complexity arising from the combination of diverse granularity of semantics, ranging from the local motion of body joints to high-level relationships across multiple people. To learn this multi-level characteristic of human action in an unsupervised manner, we propose a novel pretraining strategy along with a transformer-based model architecture named GL-Transformer++. Prior methods in unsupervised action recognition or unsupervised group activity recognition (GAR) have shown limitations, often focusing solely on capturing a partial scope of the action, such as the local movements of each individual or the broader context of the overall motion. To tackle this problem, we introduce a novel pretraining strategy named multi-interval pose displacement prediction (MPDP) that enables the model to learn the diverse extents of the action. In the architectural aspect, we incorporate the global and local attention (GLA) mechanism within the transformer blocks to learn local dynamics between joints, global context of each individual, as well as high-level interpersonal relationships in both spatial and temporal manner. In fact, the proposed method is a unified approach that demonstrates efficacy in both action recognition and GAR. Particularly, our method presents a new and strong baseline, surpassing the current SOTA GAR method by significant margins: 29.6% in Volleyball and 60.3% and 59.9% on the xsub and xset settings of the Mutual NTU dataset, respectively.

MST-m6A: A Novel Multi-Scale Transformer-based Framework for Accurate Prediction of m6A Modification Sites Across Diverse Cellular Contexts

N6-methyladenosine (m6A) modification, a prevalent epigenetic mark in eukaryotic cells, is crucial in regulating gene expression and RNA metabolism. Accurately identifying m6A modification sites is essential for understanding their functions within biological processes and the intricate mechanisms that regulate them. Recent advances in high-throughput sequencing technologies have enabled the generation of extensive datasets characterizing m6A modification sites at single-nucleotide resolution, leading to the development of computational methods for identifying m6A RNA modification sites. However, most current methods focus on specific cell lines, limiting their generalizability and practical application across diverse biological contexts. To address the limitation, we propose MST-m6A, a novel approach for identifying m6A modification sites with higher accuracy across various cell lines and tissues. MST-m6A utilizes a multi-scale transformer-based architecture, employing dual k-mer tokenization to capture rich feature representations and global contextual information from RNA sequences at multiple levels of granularity. These representations are then effectively combined using a channel fusion mechanism and further processed by a convolutional neural network to enhance prediction accuracy. Rigorous validation demonstrates that MST-m6A significantly outperforms conventional machine learning models, deep learning models, and state-of-the-art predictors. We anticipate that the high precision and cross-cell-type adaptability of MST-m6A will provide valuable insights into m6A biology and facilitate advancements in related fields. The proposed approach is available at https://github.com/cbbl-skku-org/MST-m6A/ for prediction and reproducibility purposes.

Low Resource Summarization using Pre-trained Language Models

With the advent of Deep Learning-based Artificial Neural Network models, Natural Language Processing (NLP) has witnessed significant improvements in textual data processing in terms of its efficiency and accuracy. However, the research is mostly restricted to high-resource languages such as English, and low-resource languages still suffer from a lack of available resources in terms of training datasets as well as models with even baseline evaluation results. Considering the limited availability of resources for low-resource languages, we propose a methodology for adapting self-attentive transformer-based architecture models (mBERT, mT5) for low-resource summarization, supplemented by the construction of a new baseline dataset (76.5k article, summary pairs) in a low-resource language, Urdu. Choosing news (a publicly available source) as the application domain has the potential to make the proposed methodology useful for reproducing in other languages with limited resources. Our adapted summarization model urT5 with up to 44.78% reduction in size as compared to mT5 can capture contextual information of the low-resource language effectively with an evaluation score (up to 46.35 ROUGE-1, 77 BERTScore) on par with state-of-the-art models in the high-resource language of English (PEGASUS: 47.21, BART: 45.14 on XSUM Dataset) . The proposed method provided a baseline approach toward extractive as well as abstractive summarization with competitive evaluation results in a limited resource setup.

A hybrid transformer and attention based recurrent neural network for robust and interpretable sentiment analysis of tweets

Sentiment analysis is a pivotal tool in understanding public opinion, consumer behavior, and social trends, underpinning applications ranging from market research to political analysis. However, existing sentiment analysis models frequently encounter challenges related to linguistic diversity, model generalizability, explainability, and limited availability of labeled datasets. To address these shortcomings, we propose the Transformer and Attention-based Bidirectional LSTM for Sentiment Analysis (TRABSA) model, a novel hybrid sentiment analysis framework that integrates transformer-based architecture, attention mechanism, and recurrent neural networks like BiLSTM. The TRABSA model leverages the powerful RoBERTa-based transformer model for initial feature extraction, capturing complex linguistic nuances from a vast corpus of tweets. This is followed by an attention mechanism that highlights the most informative parts of the text, enhancing the model’s focus on critical sentiment-bearing elements. Finally, the BiLSTM networks process these refined features, capturing temporal dependencies and improving the overall sentiment classification into positive, neutral, and negative classes. Leveraging the latest RoBERTa-based transformer model trained on a vast corpus of 124M tweets, our research bridges existing gaps in sentiment analysis benchmarks, ensuring state-of-the-art accuracy and relevance. Furthermore, we contribute to data diversity by augmenting existing datasets with 411,885 tweets from 32 English-speaking countries and 7,500 tweets from various US states. This study also compares six word-embedding techniques, identifying the most robust preprocessing and embedding methodologies crucial for accurate sentiment analysis and model performance. We meticulously label tweets into positive, neutral, and negative classes using three distinct lexicon-based approaches and select the best one, ensuring optimal sentiment analysis outcomes and model efficacy. Here, we demonstrate that the TRABSA model outperforms the current seven traditional machine learning models, four stacking models, and four hybrid deep learning models, yielding notable gain in accuracy (94%) and effectiveness with a macro average precision of 94%, recall of 93%, and F1-score of 94%. Our further evaluation involves two extended and four external datasets, demonstrating the model’s consistent superiority, robustness, and generalizability across diverse contexts and datasets. Finally, by conducting a thorough study with SHAP and LIME explainable visualization approaches, we offer insights into the interpretability of the TRABSA model, improving comprehension and confidence in the model’s predictions. Our study results make it easier to analyze how citizens respond to resources and events during pandemics since they are integrated into a decision-support system. Applications of this system provide essential assistance for efficient pandemic management, such as resource planning, crowd control, policy formation, vaccination tactics, and quick reaction programs.

Computational screening of umami tastants using deep learning.

Umami, a fundamental human taste modality, refers to the savory flavors in meats and broths, often associated with monosodium glutamate and protein richness. With limited knowledge of umami molecules, the food industry seeks efficient approaches for identifying novel tastants. In this study, we have devised a virtual screening pipeline for identifying highly potent umami tastants from large molecular databases. We curated the most extensive classification dataset containing 439 umami and 428 non-umami molecules and trained a transformer-based architecture to differentiate between the two classes, achieving 93% accuracy. Additionally, we built a neural network model for predicting the potency of umami compounds, the first effort of its kind. The classification and potency prediction models were combined with similarity analysis and toxicity screening to build an end-to-end virtual framework for the rational discovery of novel tastants. We applied this framework to the FooDB database containing around 70,000 molecules as an illustrative use case for screening potent umami compounds. The screened molecules were validated using molecular docking with the umami taste receptor. This study demonstrates the potential of data-driven methods in discovering new tastants from structural and chemical features of molecules and proposes an efficient implementation for industrial applications.

Deep transformer-based architecture for the recognition of mathematical equations from real-world math problems

Identifying mathematical equations from real-world math problems presents a unique and challenging task within the field of Natural Language Processing (NLP). It has a wide range of applications in various areas, such as academics, digital content design, and the development of automatic or interactive learning systems. However, the accurate understanding of these equations still needs to be enhanced due to the intrinsic complexity of mathematical symbols and various structural formats. Additionally, the unique syntax, diverse symbols, and complex structure of mathematical equations present significant obstacles that traditional NLP methods and Optical Character Recognition (OCR) systems need help to overcome. In this research, we utilize deep transformer architecture to recognize mathematical equations, and we have utilized our novel dataset with 3433 distinct observations. This dataset, which we have collected to include a diverse range of mathematical equations, is used to predict six (z=x+y, z=x−y, z=xy, z=x/y, y=x! and y=x) basic mathematical equations. We applied different transformer-based architectures, such as BERT, ELECTRA, XLNet, RoBERTa, and DistilBERT, and BERT performs best with 99.80% accuracy. To the best of our knowledge, this is the first NLP work in any language where we recognize the equation from the mathematical text.

Intelligent fatigue damage tracking and prognostics of composite structures utilizing raw images via interpretable deep learning

In recent years, prognostics gained attention in various industries by optimizing maintenance, boosting operational efficiency, and preventing costly downtime. Central to prognostics is the Remaining Useful Life (RUL), representing the critical time before system failure. Deep learning advancements facilitate RUL forecasting by extracting features from diverse data formats such as time series, images, or sequences thereof, in one, two, or three dimensions, respectively. Yet, predicting RUL from image sequences often relies heavily on resource-intensive techniques like digital image correlation, complicating data acquisition. To address challenges with high-dimensional data and unreliable models, this study introduces ISTRUST, an innovative Transformer-based architecture. ISTRUST (Interpretable Spatiotemporal TRansformer for Understanding STructures) tackles the dual challenges posed by high-dimensional data and the black-box nature of existing models. Leveraging Transformers’ attention mechanism, ISTRUST breaks down the spatiotemporal domain, effectively realizing interpretable RUL predictions under uncertainty using only sparse raw image sequences as input. Evaluated on fatigue-loaded composite samples showcasing crack propagation, ISTRUST interprets the relation between cracks and RUL via the attention mechanism. The results substantiate its capacity to interpret and clarify instances in which predictions may exhibit variability in accuracy. Through the attention mechanism, a strong correlation between the model’s spatiotemporal focus and the RUL predictions is established, making it, to the best of our knowledge, the first model to provide interpretable stochastic RUL predictions directly from sequential images of this nature.

Subject-Agnostic Transformer-Based Neural Speech Decoding from Surface and Depth Electrode Signals.

This study investigates speech decoding from neural signals captured by intracranial electrodes. Most prior works can only work with electrodes on a 2D grid (i.e., Electrocorticographic or ECoG array) and data from a single patient. We aim to design a deep-learning model architecture that can accommodate both surface (ECoG) and depth (stereotactic EEG or sEEG) electrodes. The architecture should allow training on data from multiple participants with large variability in electrode placements and the trained model should perform well on participants unseen during training. We propose a novel transformer-based model architecture named SwinTW that can work with arbitrarily positioned electrodes by leveraging their 3D locations on the cortex rather than their positions on a 2D grid. We train subject-specific models using data from a single participant and multi-patient models exploiting data from multiple participants. The subject-specific models using only low-density 8x8 ECoG data achieved high decoding Pearson Correlation Coefficient with ground truth spectrogram (PCC=0.817), over N=43 participants, outperforming our prior convolutional ResNet model and the 3D Swin transformer model. Incorporating additional strip, depth, and grid electrodes available in each participant (N=39) led to further improvement (PCC=0.838). For participants with only sEEG electrodes (N=9), subject-specific models still enjoy comparable performance with an average PCC=0.798. The multi-subject models achieved high performance on unseen participants, with an average PCC=0.765 in leave-one-out cross-validation. The proposed SwinTW decoder enables future speech neuroprostheses to utilize any electrode placement that is clinically optimal or feasible for a particular participant, including using only depth electrodes, which are more routinely implanted in chronic neurosurgical procedures. Importantly, the generalizability of the multi-patient models suggests that such a model can be applied to new patients that do not have paired acoustic and neural data, providing an advance in neuroprostheses for people with speech disability, where acoustic-neural training data is not feasible.

Time series forecasting using transformer neural network

Time series forecasting is a vital component of data science, giving essential insights that help decision-makers to predict future trends across a number of sectors. This paper focuses on projecting complicated stock market price dynamics, weather data variations, and hourly traffic occupancy rates. The Informer-Based Time Series Forecasting (IBTSF) model, which employs a transformer-based architecture to capture long-term relationships and give probabilistic predictions, is proposed.This model is assessed against Long Short-Term Memory (LSTM), Temporal Convolutional Network (TCN) and Gated Recurrent Unit (GRU) model to benchmark its performance. Simulation findings demonstrate that the IBTSF model considerably outperforms other models across all datasets. For traffic data, the IBTSF model achieves a Mean Squared Error (MSE) of 10.1 and a Mean Absolute Scaled Error (MASE) of 1.69, compared to other model's larger mistakes. In stock data forecasting, the IBTSF model reports an MSE of 8.04 and a MASE of 1.53, again outperforming other models. Similarly, using meteorological data, the IBTSF model obtains an MSE of 86.12 and a MASE of 0.7, whereas other models fall short. These results underline the IBTSF model's resilience and accuracy in capturing subtle patterns and relationships within time series data, giving precise insights that benefit decision-making processes.

TKSTAGNet: A Top-K Spatio-Temporal Attention Gating Network for air pollution prediction

Accurate air pollution prediction is of great significance for human health, travel, and environmental management. As an emerging artificial intelligence technology, Transformer has already shown tremendous potential in air pollution prediction. Unfortunately, existing Transformer-based methods are challenging to accurately predict air pollution due to two limitations: (1) they neglect irrelevant spatial and temporal neighbors (e.g., neighboring sensors and timesteps) when modeling spatio-temporal correlations, and (2) they lack an adaptive mechanism to mine different effects caused by various external factors (e.g., meteorological conditions). Jointly taking these limitations into account, we propose a novel Transformer-based architecture for air pollution prediction, named Top-K Spatio-Temporal Attention Gating Network (TKSTAGNet). In this architecture, to effectively extract spatio-temporal correlations, we first design a TopKpooling Spatial Attention (TopK-SA) and a TopKpooling Temporal Attention (TopK-TA). In TopK-SA, the TopKpooling mechanism is introduced into spatial attention to filter out irrelevant spatial neighbors, accurately capturing spatial dependencies. In TopK-TA, the TopKpooling mechanism is integrated with temporal attention to identify and exclude irrelevant temporal neighbors, precisely extracting temporal dependencies. To adaptively fuse various external factors, a gating fusion module is then proposed. We conduct experiments on three real-world air pollution datasets from urban regions in Beijing, Shanghai, and Chongqing, respectively. Experimental results demonstrate that TKSTAGNet outperforms state-of-the-art baselines in prediction precision and fitting the variation trends of air pollution.

A Semi-Supervised Transformer-Based Deep Learning Framework for Automated Tooth Segmentation and Identification on Panoramic Radiographs.

Automated tooth segmentation and identification on dental radiographs are crucial steps in establishing digital dental workflows. While deep learning networks have been developed for these tasks, their performance has been inferior in partially edentulous individuals. This study proposes a novel semi-supervised Transformer-based framework (SemiTNet), specifically designed to improve tooth segmentation and identification performance on panoramic radiographs, particularly in partially edentulous cases, and establish an open-source dataset to serve as a unified benchmark. A total of 16,317 panoramic radiographs (1589 labeled and 14,728 unlabeled images) were collected from various datasets to create a large-scale dataset (TSI15k). The labeled images were divided into training and test sets at a 7:1 ratio, while the unlabeled images were used for semi-supervised learning. The SemiTNet was developed using a semi-supervised learning method with a label-guided teacher-student knowledge distillation strategy, incorporating a Transformer-based architecture. The performance of SemiTNet was evaluated on the test set using the intersection over union (IoU), Dice coefficient, precision, recall, and F1 score, and compared with five state-of-the-art networks. Paired t-tests were performed to compare the evaluation metrics between SemiTNet and the other networks. SemiTNet outperformed other networks, achieving the highest accuracy for tooth segmentation and identification, while requiring minimal model size. SemiTNet's performance was near-perfect for fully dentate individuals (all metrics over 99.69%) and excellent for partially edentulous individuals (all metrics over 93%). In edentulous cases, SemiTNet obtained statistically significantly higher tooth identification performance than all other networks. The proposed SemiTNet outperformed previous high-complexity, state-of-the-art networks, particularly in partially edentulous cases. The established open-source TSI15k dataset could serve as a unified benchmark for future studies.

Mammography classification with multi-view deep learning techniques: Investigating graph and transformer-based architectures

The potential and promise of deep learning systems to provide an independent assessment and relieve radiologists’ burden in screening mammography have been recognized in several studies. However, the low cancer prevalence, the need to process high-resolution images, and the need to combine information from multiple views and scales still pose technical challenges. Multi-view architectures that combine information from the four mammographic views to produce an exam-level classification score are a promising approach to the automated processing of screening mammography. However, training such architectures from exam-level labels, without relying on pixel-level supervision, requires very large datasets and may result in suboptimal accuracy. Emerging architectures such as Visual Transformers (ViT) and graph-based architectures can potentially integrate ipsi-lateral and contra-lateral breast views better than traditional convolutional neural networks, thanks to their stronger ability of modeling long-range dependencies. In this paper, we extensively evaluate novel transformer-based and graph-based architectures against state-of-the-art multi-view convolutional neural networks, trained in a weakly-supervised setting on a middle-scale dataset, both in terms of performance and interpretability. Extensive experiments on the CSAW dataset suggest that, while transformer-based architecture outperform other architectures, different inductive biases lead to complementary strengths and weaknesses, as each architecture is sensitive to different signs and mammographic features. Hence, an ensemble of different architectures should be preferred over a winner-takes-all approach to achieve more accurate and robust results. Overall, the findings highlight the potential of a wide range of multi-view architectures for breast cancer classification, even in datasets of relatively modest size, although the detection of small lesions remains challenging without pixel-wise supervision or ad-hoc networks.

SwinShadow: Shifted Window for Ambiguous Adjacent Shadow Detection

Shadow detection is a fundamental and challenging task in many computer vision applications. Intuitively, most shadows come from the occlusion of light by the object itself, resulting in the object and its shadow being contiguous (referred to as the adjacent shadow in this paper). In this case, when the color of the object is similar to that of the shadow, existing methods struggle to achieve accurate detection. To address this problem, we present SwinShadow, a transformer-based architecture that fully utilizes the powerful shifted window mechanism for detecting adjacent shadows. The mechanism operates in two steps. Initially, it applies local self-attention within a single window, enabling the network to focus on local details. Subsequently, it shifts the attention windows to facilitate inter-window attention, enabling the capture of a broader range of adjacent information. These combined steps significantly improve the network's capacity to distinguish shadows from nearby objects. And the whole process can be divided into three parts: encoder, decoder, and feature integration. During encoding, we adopt Swin Transformer to acquire hierarchical features. Then during decoding, for shallow layers, we propose a deep supervision (DS) module to suppress the false positives and boost the representation capability of shadow features for subsequent processing, while for deep layers, we leverage a double attention (DA) module to integrate local and shifted window in one stage to achieve a larger receptive field and enhance the continuity of information. Ultimately, a new multi-level aggregation (MLA) mechanism is applied to fuse the decoded features for mask prediction. Extensive experiments on three shadow detection benchmark datasets, SBU, UCF, and ISTD, demonstrate that our network achieves good performance in terms of balance error rate (BER). The source code and results are now publicly available at https://github.com/harrytea/SwinShadow .

Wave runup and total water level observations from time series imagery at several sites with varying nearshore morphologies

Coastal imaging systems have been developed to measure wave runup and total water level (TWL) at the shoreline, which is a key metric for assessing coastal flooding and erosion. However, extracting quantitative measurements from coastal images has typically been done through the laborious task of hand-digitization of wave runup timestacks. Timestacks are images created by sampling a cross-shore array of pixels from an image through time as waves propagate towards and run up a beach. We utilize over 7000 hand-digitized timestacks from six diverse locations to train and validate machine learning models to automate the process of TWL extraction. Using these data, we evaluate two deep learning model architectures for the task of runup detection. One is based on a fully convolutional architecture trained from scratch, and the other is a transformer-based architecture trained using transfer learning. The deep learning models provide a probability of each pixel being either wet or dry. When contoured at the 50% level (equal chance of being wet or dry), the deep learning models more accurately identified TWL maxima than minima at all sites. This resulted in accurate predictions of 2% exceedance runup, but under predictions of significant swash and over predictions of wave setup. Improved agreement with the complete TWL time series was obtained through post-processing by utilizing the wet/dry probability of each pixel to weight the contouring toward lower dryness probabilities for runup minima (maxima agreed well with observations without tuning). Overall, a transformer-based model using transfer learning provided the best agreement with wave runup statistics, including a) the 2% exceedance runup, b) significant swash, and c) wave setup at the shoreline. For a random subset of images, the model was found to be within the uncertainty range of hand-digitization. The relative success of the transfer learning model suggests that fine-tuning a large model has advantages compared to training a smaller model from scratch. Models provide per-pixel probabilistic estimates in less than 10 s per timestack on a single computational unit, versus the more than 5 min required for hand-digitization. The model is therefore well-suited for near real-time applications, allowing for the development of early warning systems for difficult to forecast events. Real-time wave runup and total water level observations can also be incorporated into coastal hazards forecasts for data assimilation and continual model validation and improvement.

Swin transformer-based fork architecture for automated breast tumor classification

Breast cancer constitutes a prevalent and escalating health concern globally. Additionally, the significance of early diagnosis for effective treatment cannot be overstated. Ultrasound imaging (UI) emerges as a cost-effective means for early diagnosis. However, the interpretation of user interface images can be challenging, leading to the development of computer-aided diagnostic systems. In this article, a network architecture named Swin Transformer-based Fork Network (SW-ForkNet) is developed for breast tumor classification. Utilizing the DenseNet121 backbone, SW-ForkNet amalgamates spatial, semantic, and long-context features. The inclusion of the spatial Squeeze-and-Excitation (sSE) block is instrumental for spatial details, while the Swin Transformer structure is employed for capturing global long-context features. Moreover, to optimize the acquisition of these features, a connection is established in the middle layer of the DenseNet121 architecture, feeding both sSE and Swin Transformers. In the network architecture’s output, three distinct feature groups are vectorized, combined, and processed to obtain the final feature map. Finally, a prediction is generated by applying the softmax classifier to this concluding feature map. Experimental evaluations conducted on three datasets (BUSI, GDPH, and SYSUCC) demonstrate the superior performance of SW-ForkNet, showcasing high accuracy (Acc) and F1-Scores (F1S) compared to existing methods and state-of-the-art models. Notable achievements include 93.12 % Acc and 92.27 % F1s for BUSI, 96.15 % Acc and 96.04 % F1S for GDPH, and 94.88 % Acc and 94.03 % F1S for SYSUCC. Consequently, the proposed SW-ForkNet model seems to present itself as a novel and effective structure in breast tumor classification.

A Fully Automated Pipeline Using Swin Transformers for Deep Learning-Based Blood Segmentation on Head Computed Tomography Scans After Aneurysmal Subarachnoid Hemorrhage

Accurate volumetric assessment of spontaneous aneurysmal subarachnoid hemorrhage (aSAH) is a labor-intensive task performed with current manual and semiautomatic methods that might be relevant for its clinical and prognostic implications. In the present research, we sought to develop and validate an artificial intelligence-driven, fully automated blood segmentation tool for subarachnoid hemorrhage (SAH) patients via noncontrast computed tomography (NCCT) scans employing a transformer-based Swin-UNETR architecture. We retrospectively analyzed NCCT scans from patients with confirmed aSAH utilizing the Swin-UNETR for segmentation. The performance of the proposed method was evaluated against manually segmented ground truth data using metrics such as Dice score, intersection over union, volumetric similarity index , symmetric average surface distance , sensitivity, and specificity. A validation cohort from an external institution was included to test the generalizability of the model. The model demonstrated high accuracy with robust performance metrics across the internal and external validation cohorts. Notably, it achieved high Dice coefficient (0.873±0.097), intersection over union (0.810±0.092), volumetric similarity index (0.840±0.131), sensitivity (0.821±0.217), and specificity (0.996±0.004) values and a low symmetric average surface distance (1.866±2.910), suggesting proficiency in segmenting blood in SAH patients. The model's efficiency was reflected in its processing speed, indicating potential for real-time applications. Our Swin UNETR-based model offers significant advances in the automated segmentation of blood in SAH patients on NCCT images. Despite the computational demands, the model operates effectively on standard hardware with a user-friendly interface, facilitating broader clinical adoption. Further validation across diverse datasets is warranted to confirm its clinical reliability.

A Kernel Attention-based Transformer Model for Survival Prediction of Heart Disease Patients.

Survival analysis is employed to scrutinize time-to-event data, with emphasis on comprehending the duration until the occurrence of a specific event. In this article, we introduce two novel survival prediction models: CosAttnSurv and CosAttnSurv DyACT. CosAttnSurv model leverages transformer-based architecture and a softmax-free kernel attention mechanism for survival prediction. Our second model, CosAttnSurv DyACT, enhances CosAttnSurv with Dynamic Adaptive Computation Time (DyACT) control, optimizing computation efficiency. The proposed models are validated using two public clinical datasets related to heart disease patients. When compared to other state-of-the-art models, our models demonstrated an enhanced discriminative and calibration performance. Furthermore, in comparison to other transformer architecture-based models, our proposed models demonstrate comparable performance while exhibiting significant reduction in both time and memory requirements. Overall, our models offer significant advancements in the field of survival analysis and emphasize the importance of computationally effective time-based predictions, with promising implications for medical decision-making and patient care.

Computational Argumentation-based Chatbots: A Survey

Chatbots are conversational software applications designed to interact dialectically with users for a plethora of different purposes. Surprisingly, these colloquial agents have only recently been coupled with computational models of arguments (i.e. computational argumentation), whose aim is to formalise, in a machine-readable format, the ordinary exchange of information that characterises human communications. Chatbots may employ argumentation with different degrees and in a variety of manners. The present survey sifts through the literature to review papers concerning this kind of argumentation-based bot, drawing conclusions about the benefits and drawbacks that this approach entails in comparison with standard chatbots, while also envisaging possible future development and integration with the Transformer-based architecture and state-of-the-art Large Language models.

Learning to Improve Operational Efficiency from Pose Error Estimation in Robotic Pollination

Autonomous pollination robots have been widely discussed in recent years. However, the accurate estimation of flower poses in complex agricultural environments remains a challenge. To this end, this work proposes the implementation of a transformer-based architecture to learn the translational and rotational errors between the pollination robot’s end effector and the target object with the aim of enhancing robotic pollination efficiency in cross-breeding tasks. The contributions are as follows: (1) We have developed a transformer architecture model, equipped with two feedforward neural networks that directly regress the translational and rotational errors between the robot’s end effector and the pollination target. (2) Additionally, we have designed a regression loss function that is guided by the translational and rotational errors between the robot’s end effector and the pollination targets. This enables the robot arm to rapidly and accurately identify the pollination target from the current position. (3) Furthermore, we have designed a strategy to readily acquire a substantial number of training samples from eye-in-hand observation, which can be utilized as inputs for the model. Meanwhile, the translational and rotational errors identified in the end-manipulator Cartesian coordinate system are designated as loss targets simultaneously. This helps to optimize the training of the model. We conducted experiments on a realistic robotic pollination system. The results demonstrate that the proposed method outperforms the state-of-the-art method, in terms of both accuracy and efficiency.