Transformer Model Research Articles

From facebook posts to news headlines: using transformer models to predict post-disaster impact on mass media content

From facebook posts to news headlines: using transformer models to predict post-disaster impact on mass media content

Stance Detection in the Context of Fake News—A New Approach

Online social networks (OSNs) are inundated with an enormous daily influx of news shared by users worldwide. Information can originate from any OSN user and quickly spread, making the task of fact-checking news both time-consuming and resource-intensive. To address this challenge, researchers are exploring machine learning techniques to automate fake news detection. This paper specifically focuses on detecting the stance of content producers—whether they support or oppose the subject of the content. Our study aims to develop and evaluate advanced text-mining models that leverage pre-trained language models enhanced with meta features derived from headlines and article bodies. We sought to determine whether incorporating the cosine distance feature could improve model prediction accuracy. After analyzing and assessing several previous competition entries, we identified three key tasks for achieving high accuracy: (1) a multi-stage approach that integrates classical and neural network classifiers, (2) the extraction of additional text-based meta features from headline and article body columns, and (3) the utilization of recent pre-trained embeddings and transformer models.

V-LTCS: Backbone exploration for Multimodal Misogynous Meme detection

Memes have become a fundamental part of online communication and humour, reflecting and shaping the culture of today’s digital age. The amplified Meme culture is inadvertently endorsing and propagating casual Misogyny. This study proposes V-LTCS (Vision- Language Transformer Combination Search), a framework that encompasses all possible combinations of the most fitting Text (i.e. BERT, ALBERT, and XLM-R) and Vision (i.e. Swin, ConvNeXt, and ViT) Transformer Models to determine the backbone architecture for identifying Memes that contains misogynistic contents. All feasible Vision-Language Transformer Model combinations obtained from the recognized optimal Text and Vision Transformer Models are evaluated on two (smaller and larger) datasets using varied standard metrics (viz. Accuracy, Precision, Recall, and F1-Score). The BERT-ViT combinational Transformer Model demonstrated its efficiency on both datasets, validating its ability to serve as a backbone architecture for all subsequent efforts to recognize Multimodal Misogynous Memes.

Unsupervised few shot learning architecture for diagnosis of periodontal disease in dental panoramic radiographs.

In the domain of medical imaging, the advent of deep learning has marked a significant progression, particularly in the nuanced area of periodontal disease diagnosis. This study specifically targets the prevalent issue of scarce labeled data in medical imaging. We introduce a novel unsupervised few-shot learning algorithm, meticulously crafted for classifying periodontal diseases using a limited collection of dental panoramic radiographs. Our method leverages UNet architecture for generating regions of interest (RoI) from radiographs, which are then processed through a Convolutional Variational Autoencoder (CVAE). This approach is pivotal in extracting critical latent features, subsequently clustered using an advanced algorithm. This clustering is key in our methodology, enabling the assignment of labels to images indicative of periodontal diseases, thus circumventing the challenges posed by limited datasets. Our validation process, involving a comparative analysis with traditional supervised learning and standard autoencoder-based clustering, demonstrates a marked improvement in both diagnostic accuracy and efficiency. For three real-world validation datasets, our UNet-CVAE architecture achieved up to average 14% higher accuracy compared to state-of-the-art supervised models including the vision transformer model when trained with 100 labeled images. This study not only highlights the capability of unsupervised learning in overcoming data limitations but also sets a new benchmark for diagnostic methodologies in medical AI, potentially transforming practices in data-constrained scenarios.

Fusing Transformer and diffusion for high-resolution prediction of daylight illuminance and glare based on sparse ceiling-mounted input

Accurate prediction of workplane daylight illuminance and eye-height glare is crucial for lighting control. Existing studies used machine learning to predict illuminance at predetermined locations based on indoor sensors, but they may encounter challenges in scenarios 1) with flexible seating arrangements, 2) with dynamic shading devices, and 3) requiring the prediction of glare. To address these challenges, we proposed a novel method fusing Transformer and Diffusion models, with the input being data collected from sparse ceiling-mounted illuminance sensors, and the outputs being high-resolution workplane illuminance and glare. The model works well for rooms without and with dynamic roller shades. For the former, the mean absolute errors for illuminance below 3000 lx and Daylight Glare Index (DGI) are only 20.77 lx and 0.20, and the error rates in detecting illuminance <500 lx and DGI>22 are only 0.85 % and 5.55 %. For the more complicated latter case, the aforementioned four numbers are 34.78 lx, 0.59, 2.47 % and 23.13 %. The model significantly outperforms the linear and the ANN models, particularly in glare prediction. The influence of sensor number and placement strategy on model performance was also revealed. The model can potentially enhance lighting control, especially in cases with dynamic shading, with flexible seating arrangements, and where glare is of interest.

Epileptic seizure prediction via multidimensional transformer and recurrent neural network fusion

BackgroundEpilepsy is a prevalent neurological disorder in which seizures cause recurrent episodes of unconsciousness or muscle convulsions, seriously affecting the patient’s work, quality of life, and health and safety. Timely prediction of seizures is critical for patients to take appropriate therapeutic measures. Accurate prediction of seizures remains a challenge due to the complex and variable nature of EEG signals. The study proposes an epileptic seizure model based on a multidimensional Transformer with recurrent neural network(LSTM-GRU) fusion for seizure classification of EEG signals.MethodologyFirstly, a short-time Fourier transform was employed in the extraction of time-frequency features from EEG signals. Second, the extracted time-frequency features are learned using the Multidimensional Transformer model. Then, LSTM and GRU are then used for further learning of the time and frequency characteristics of the EEG signals. Next, the output features of LSTM and GRU are spliced and categorized using the gating mechanism. Subsequently, seizure prediction is conducted.ResultsThe model was tested on two datasets: the Bonn EEG dataset and the CHB-MIT dataset. On the CHB-MIT dataset, the average sensitivity and average specificity of the model were 98.24% and 97.27%, respectively. On the Bonn dataset, the model obtained about 99% and about 98% accuracy on the binary classification task and the tertiary upper classification task, respectively.ConclusionThe findings of the experimental investigation demonstrate that our model is capable of exploiting the temporal and frequency characteristics present within EEG signals.

Deep learning-based construction and demolition plastic waste classification by resin type using RGB images

The construction and demolition sector generates a substantial portion of Australia's total waste, with plastics being a key recyclable component. The perceived financial impracticality of sorting and separating waste, coupled with the simplicity of landfilling processes often contribute to mixed material loads sent directly to landfills. Therefore, developing a commercially feasible system that can accurately separate the generated waste is imperative. This paper presents a comprehensive study on using RGB images for deep learning-based construction and demolition plastic waste classification by resin type. A large and specialised dataset of end-of-life plastic waste images is gathered. This dataset comprises four commonly used plastic types in construction projects—ABS, HDPE, PS, and PVC. Leveraging Transfer Learning with models pre-trained on ImageNet, highly accurate models tailored to this classification task are developed in this paper. Advanced Convolutional Neural Network and Vision Transformer-based models, including ResNet, ResNeXt, RegNet, and Swin Transformer, are trained and evaluated on this dataset. Another contribution of this work is Knowledge Distillation from a large, computationally intensive, and accurate model to enhance the accuracy of fast and compact models specifically designed for deployment on edge devices. This study applies Knowledge Distillation by using the output class probabilities of the large, computationally intensive Swin Transformer model to enhance the accuracy of the fast and lightweight MobileNetV3 models. The results demonstrate that RGB images offer a practical alternative to other costly and complex systems for effective plastic identification, due to their availability, low cost, ease of use, simple setups, and robustness to variations in operational conditions.

CTBViT: A novel ViT for tuberculosis classification with efficient block and randomized classifier

Background and ObjectiveDespite the impressive achievements of Transformer models in tuberculosis (TB) analysis, there are still some shortcomings, such as the high computational cost and overfitting problem. MethodsTo address these issues, this paper proposes a novel CTBViT model to detect TB in CT and X-ray images. In the traditional transformer block, all the tokens are taken into account for self-attention. However, many regions in the images are irrelevant to the classification task. Therefore, we propose the Patch Reduction Block (PRB) to reduce computation by evaluating each token then dropping less important tokens and retaining only important tokens for subsequent processing. Additionally, we propose the randomized classifier where all the parameters in the input layer are randomly set and do not change during training. This design effectively avoids the overfitting problem. ResultsTo fairly validate our model, we conducted experiments on both private and public datasets and the CTBViT surpassed the classification performance of other state-of-the-art methods. ConclusionsThe results reveal the effectiveness and efficiency of the proposed CTBViT in TB classification, which can be a useful tool in clinical diagnosis.

Research on fault diagnosis for three-phase rectifier device in direct current transmission system based on continuous wavelet transform and vision transformer

Abstract As a core component of photovoltaic power generation systems, the three-phase rectifier device plays a crucial role, and its failure can potentially reduce energy conversion efficiency and output quality. Presently, the performance of fault time-domain signal diagnosis methods based on three-phase rectifier circuits is challenging to enhance. This paper proposes a novel fault detection method for three-phase rectifier devices based on Vision Transformer, referred to as CWT-ViT, to address this issue. This method transforms time-domain fault signals into images through Continuous Wavelet Transform (CWT), subsequently inputting these images into a Vision Transformer model. Relying on its powerful self-attention mechanism and fully connected layers, it realizes the extraction and learning of rectifier device image features. A Simulink simulation model of a three-phase bridge controllable rectifier circuit is established for fault injection to collect fault signals. Fault diagnosis experiments demonstrate that the proposed diagnostic method achieves a prediction accuracy of 98.6%, maintaining a relatively high precision level. In comparison to four excellent classification models currently available: AlexNet, RepVGG, ResNet, and GoogLeNet, the proposed method demonstrates superior diagnostic performance. Additionally, this paper conducts ablation experiments to meticulously analyze the impact of each module in the fault diagnosis process. This research achieves more precise and efficient fault diagnosis in photovoltaic power generation systems, thereby reducing downtime and maintenance costs for actual equipment and enhancing the stability of photovoltaic power generation systems. This research provides an innovative, intelligent solution for the intelligent operations and maintenance of photovoltaic power generation.

Phase unwrapping via fully exploiting global and local spatial dependencies

Phase unwrapping (PU) is the process of extracting the authentic phase image from its noisy wrapped measurements, playing a crucial role in scientific imaging techniques. PU requires solving a challenging non-linear ill-posed problem, particularly in the presence of noticeable noise. In recent years, deep learning has emerged as a promising approach for PU. Inspired by the success of convolutional neural networks (CNNs) in image restoration, many existing works trained CNNs for PU. However, due to the locality of convolutional kernels, CNNs are not efficient in capturing global spatial dependencies, a critical cue for PU. As an alternate, recent studies employed recurrent neural networks (RNNs) defined on handcrafted pixel paths. Nonetheless, a limited number of pre-defined pixel paths cannot fully exploit global spatial dependencies existing in complex phase structures. In this paper, we introduce a vision transformer (ViT) model that effectively captures both global and local spatial dependencies using a hierarchical structure with a multi-scale process. The proposed ViT model employs a series of global transformer blocks to capture global spatial dependencies at the roughest scale. The resulting global features are used to guide a set of local transformer blocks to analyze local spatial dependencies in a coarse-to-fine progressive manner for unwrapping. Extensive experiments show that, our proposed ViT model produces higher-quality unwrapped phases over existing CNN/RNN-based methods, while maintaining a lightweight nature.

Extracting Sentence Embeddings from Pretrained Transformer Models

Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates sufficiently capture and represent the underlying meaning? After providing a comprehensive review of existing sentence embedding extraction and refinement methods, we thoroughly test different combinations and our original extensions of the most promising ones on pretrained models. Namely, given 110 M parameters, BERT’s hidden representations from multiple layers, and many tokens, we try diverse ways to extract optimal sentence embeddings. We test various token aggregation and representation post-processing techniques. We also test multiple ways of using a general Wikitext dataset to complement BERT’s sentence embeddings. All methods are tested on eight Semantic Textual Similarity (STS), six short text clustering, and twelve classification tasks. We also evaluate our representation-shaping techniques on other static models, including random token representations. Proposed representation extraction methods improve the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks, almost reach the performance of BERT-derived representations. Our work shows that the representation-shaping techniques significantly improve sentence embeddings extracted from BERT-based and simple baseline models.

Comparative Analysis of Machine Learning Algorithms with SMOTE and Boosting Techniques in Accuracy Improvement

This research explores and enhances accuracy in sentiment classification related to Indonesia's Capital City relocation by combining Naive Bayes (NB), Random Forest (RF), SMOTE, and XGBoost. The study addresses challenges of unbalanced data and complexity in social media sentiment analysis. The combination of RF with SMOTE achieved the highest accuracy at 91.25%, demonstrating SMOTE's effectiveness in balancing the dataset and improving minority class detection. While adding XGBoost slightly reduced accuracy (90.92%), it increased the NB model's accuracy from 77.45% to 85.97% when combined with SMOTE. RF alone reached 87.46% and improved to 88.78% with XGBoost. The study underscores the importance of selecting and combining techniques to maximize sentiment prediction accuracy. Future research could explore deep learning or transformer models for even better results, offering deeper insights into public sentiment and aiding effective policy strategy development.

Phoenixes at LLMs4OL 2024 Tasks A, B, and C: Retrieval Augmented Generation for Ontology Learning

Large language models (LLMs) showed great capabilities in ontology learning (OL) where they automatically extract knowledge from text. In this paper, we proposed a Retrieval Augmented Generation (RAG) formulation for three different tasks of ontology learning defined in the LLMs4OL Challenge at ISWC 2024. For task A - term typing - we considered terms as a query and encoded the query through the Query Encoder model for searching through knowledge base embedding of types embeddings obtained through Context Encoder. Next, using Zero-Shot Prompt template we asked LLM to determine what types are appropriate for a given term within the term typing task. Similarly, for Task B, we calculated the similarity matrix using an encoder-based transformer model, and by applying the similarity threshold we considered only similar pairs to query LLM to identify whatever pairs have the "is-a" relation between a given type and in a case of having the relationships which one is "parent" and which one is "child". In final, for Task C -- non-taxonomic relationship extraction -- we combined both approaches for Task A and B, where first using Task B formulation, child-parents are identified then using Task A, we assigned them an appropriate relationship. For the LLMs4OL challenge, we experimented with the proposed framework over 5 subtasks of Task A, all subtasks of Task B, and one subtask of Task C using Mistral-7B LLM.

A Deep Learning Application of Capsule Endoscopic Gastric Structure Recognition Based on a Transformer Model.

Gastric structure recognition systems have become increasingly necessary for the accurate diagnosis of gastric lesions in capsule endoscopy. Deep learning, especially using transformer models, has shown great potential in the recognition of gastrointestinal (GI) images according to self-attention. This study aims to establish an identification model of capsule endoscopy gastric structures to improve the clinical applicability of deep learning to endoscopic image recognition. A total of 3343 wireless capsule endoscopy videos collected at Nanfang Hospital between 2011 and 2021 were used for unsupervised pretraining, while 2433 were for training and 118 were for validation. Fifteen upper GI structures were selected for quantifying the examination quality. We also conducted a comparison of the classification performance between the artificial intelligence model and endoscopists by the accuracy, sensitivity, specificity, and positive and negative predictive values. The transformer-based AI model reached a relatively high level of diagnostic accuracy in gastric structure recognition. Regarding the performance of identifying 15 upper GI structures, the AI model achieved a macroaverage accuracy of 99.6% (95% CI: 99.5-99.7), a macroaverage sensitivity of 96.4% (95% CI: 95.3-97.5), and a macroaverage specificity of 99.8% (95% CI: 99.7-99.9) and achieved a high level of interobserver agreement with endoscopists. The transformer-based AI model can accurately evaluate the gastric structure information of capsule endoscopy with the same performance as that of endoscopists, which will provide tremendous help for doctors in making a diagnosis from a large number of images and improve the efficiency of examination.

Artificial intelligence-based power market price prediction in smart renewable energy systems: Combining prophet and transformer models

With the increasing integration of smart renewable energy systems and power electronic converters, electricity market price prediction is particularly important. It is not only crucial for the interests of power suppliers and market regulators but also plays a key role in ensuring the reliable and flexible operation of the power system, particularly during extreme weather events or abnormal conditions. This study develops a hybrid time series forecasting model that combines Prophet and Transformer, which takes advantage of deep learning to provide a new solution for electricity market price forecasting. By introducing the Stacking optimization strategy, this study improves the accuracy and stability of electricity market price sequence prediction. In addition, the study tries to integrate traditional time series forecasting methods (such as the Prophet model) with deep learning models (such as the Transformer model), aiming to make full use of their respective advantages to achieve more accurate and stable predictions. Through experimental evaluation on four electricity market data sets, this study finds that the hybrid forecast model exhibits significant performance improvements in enhancing the accuracy and stability of electricity market price predictions. This method not only provides a more accurate tool for power market price prediction, but also provides solid technical support for the efficient operation and sustainable development of smart renewable energy systems. Experimental results also show that the combination of deep learning models with traditional time series methods and the introduction of Stacking strategies is crucial to improving the performance of power market price prediction, and also helps us better understand and design smart renewable energy systems, price and energy management strategies, thereby providing an effective method for achieving efficient and reliable power and energy transmission.

Classification and Recognition of Lung Sounds Using Artificial Intelligence and Machine Learning: A Literature Review

This review explores the latest advances in artificial intelligence (AI) and machine learning (ML) for the identification and classification of lung sounds. The article provides a historical overview from the invention of the electronic stethoscope to the auscultation of lung sounds, emphasizing the importance of the rapid diagnosis of lung diseases in the post-COVID-19 era. The review classifies lung sounds, including wheezes and stridors, and explores their pathological relevance. In addition, the article deeply explores feature extraction strategies, measurement methods, and multiple advanced machine learning models for classification, such as deep residual networks (ResNets), convolutional neural networks combined with long short-term memory networks (CNN–LSTM), and transformer models (transformer). The article discusses the problems of insufficient data and replicating human expert experience and proposes future research directions, including improved data utilization, enhanced feature extraction, and classification using spectrograms. Finally, the article emphasizes the expanding role of AI and ML in lung sound diagnosis and their potential for further development in this field.

MFT: A novel memory flow transformer efficient intrusion detection method

Intrusion detection is a critical field in network security research that is devoted to detecting malicious traffic or attacks on networks. Even with the advances in today's Internet environment, a lot of intrusion detection techniques still fail to take into account the long-term characteristics present in network data, which results in a high false alarm rate. Some researchers have tried to address this problem by using the traditional transformer model; however, it is not very effective when dealing with complex relationships and the subtle classification requirements of large amounts of sequential data. This work presents a novel solution called the memory flow transformer (MFT) in response to the limitations of the conventional transformer model. By utilizing a carefully designed memory flow structure, MFT transcends traditional limitations and makes it possible to obtain complex long-term features from network traffic. This innovation enables the model to identify deep connections at a finer level between a wide variety of network traffic data. Extensive experiments were carried out on the complex CICIDS 2017 and NSL-KDD datasets to validate the effectiveness of the MFT model. The results were outstanding, demonstrating MFT's powerful detection abilities. With regard to performance metrics like accuracy, F1 score, false alarm rate, and training time, MFT is superior to current state-of-the-art approaches. Network security is greatly strengthened by MFT, which provides practitioners in the intrusion detection field with novel and effective techniques.

A novel hybrid model for multi-step-ahead forecasting of wind speed based on univariate data feature enhancement

Reliable multistep ahead wind speed forecasting (MAWSF) is critical for the energy management of wind farms and the long-term maintenance of wind power systems. However, relying on inherent meteorological features such as temperature and atmospheric pressure often fails to meet the deep learning model’s feature requirements for accurate wind speed forecasting (WSF). This paper introduces a hybrid multistep forecasting model that constructs a univariate wind speed feature enhancement framework, combining random forest (RF) and Transformer models for WSF. Initially, the hybrid enhancement framework decomposes the univariate wind speed data and extracts time-series features, effectively mining the latent features information. Subsequently, the RF feature selector filters out significant features contributing to WSF and eliminates redundant features to provide stable features data. Finally, Transformer model is utilized for both short-term and long-term MAWSF. This study conducted MAWSF on data with sampling intervals of 20 min, 30 min and 1 h. The results indicate that, compared to existing state-of-the-art models, the hybrid model in MAWSF tasks reduces the dependency of models on inherent meteorological features, achieving more accurate forecasting and faster computation speeds. Ultimately, the proposed model can provide reliable technical support for energy management and maintenance guidance in real-world wind farms.

Advanced streamflow forecasting for Central European Rivers: The Cutting-Edge Kolmogorov-Arnold networks compared to Transformers

Accurate streamflow forecasting is crucial for effective water resource management, flood mitigation, and maintaining ecological balance, especially in Central Europe’s major rivers. This study investigates the performance of two advanced deep learning algorithms—Kolmogorov-Arnold Network (KAN) and Transformers—using long-term hydrological data from four key rivers: the Rhine, Danube, Elbe, and Oder. These rivers, characterized by varying flow regimes and significant human and climatic impacts, serve as vital arteries for both commerce and ecosystems.The dataset comprises historical daily streamflow data, alongside derived input parameters such as moving averages and flow rate changes, used to predict short-term (1 to 7 days) river discharges. The KAN model, leveraging learnable spline-based activation functions, was developed to enhance accuracy and interpretability in capturing complex hydrological patterns. In contrast, the Transformer model uses advanced attention mechanisms, which excel in handling sequential data with long-range dependencies.Results show that the KAN model significantly outperforms the Transformer in short-term forecasts (up to 3 days). For 1-day forecasts, the KAN achieved R2 values of 0.975 for the Rhine, 0.956 for the Danube, 0.992 for the Elbe, and 0.969 for the Oder, with MAPE values ranging from 3.20 % to 8.09 %. At the 3-day horizon, the KAN’s R2 values remained high, demonstrating its robustness. However, as the forecasting horizon extends to 7 days, the performance of both models converges.These findings underscore the methodological novelty of KAN, which provides enhanced short-term predictive capabilities compared to existing methods, offering valuable insights for improving water resource management strategies in complex hydrological settings.

Predicting temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by artificial neural networks

Predicting temporal evacuation travel time in staircases between adjacent floors is crucial for dynamic evacuation guidance to improve vertical evacuation efficiency during emergencies in super high-rise buildings. However, the prediction methods used in previous studies mainly focus on the total evacuation time in buildings and cannot predict the temporal evacuation travel time in every single part of the total evacuation route. This study proposes a novel method to predict temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by using artificial neural networks. Firstly, evacuation simulation data of a 71-storey office building are analyzed for feature engineering and dataset building. Secondly, three types of artificial neural networks, including convolutional neural networks, RNN-based model (the traditional recurrent neural networks, long-short term memory networks, and bidirectional long-short term memory networks), and Attention-based model (Attention model and Transformer model) are extended to predict the temporal evacuation travel time in staircases between adjacent floors based on the simulation data. Finally, the prediction results by different artificial neural networks are compared. The results show that the artificial neural networks can provide accurate prediction values for the temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings.