Transformer Model Research Articles

EfficientUNetViT: Efficient Breast Tumor Segmentation Utilizing UNet Architecture and Pretrained Vision Transformer.

This study introduces a sophisticated neural network structure for segmenting breast tumors. It achieves this by combining a pretrained Vision Transformer (ViT) model with a UNet framework. The UNet architecture, commonly employed for biomedical image segmentation, is further enhanced with depthwise separable convolutional blocks to decrease computational complexity and parameter count, resulting in better efficiency and less overfitting. The ViT, renowned for its robust feature extraction capabilities utilizing self-attention processes, efficiently captures the overall context within images, surpassing the performance of conventional convolutional networks. By using a pretrained ViT as the encoder in our UNet model, we take advantage of its extensive feature representations acquired from extensive datasets, resulting in a major enhancement in the model's ability to generalize and train efficiently. The suggested model has exceptional performance in segmenting breast cancers from medical images, highlighting the advantages of integrating transformer-based encoders with efficient UNet topologies. This hybrid methodology emphasizes the capabilities of transformers in the field of medical image processing and establishes a new standard for accuracy and efficiency in activities related to tumor segmentation.

Pre-Trained Language Model Ensemble for Arabic Fake News Detection

Fake news detection (FND) remains a challenge due to its vast and varied sources, especially on social media platforms. While numerous attempts have been made by academia and the industry to develop fake news detection systems, research on Arabic content remains limited. This study investigates transformer-based language models for Arabic FND. While transformer-based models have shown promising performance in various natural language processing tasks, they often struggle with tasks involving complex linguistic patterns and cultural contexts, resulting in unreliable performance and misclassification problems. To overcome these challenges, we investigated an ensemble of transformer-based models. We experimented with five Arabic transformer models: AraBERT, MARBERT, AraELECTRA, AraGPT2, and ARBERT. Various ensemble approaches, including a weighted-average ensemble, hard voting, and soft voting, were evaluated to determine the most effective techniques for boosting learning models and improving prediction accuracies. The results of this study demonstrate the effectiveness of ensemble models in significantly boosting the baseline model performance. An important finding is that ensemble models achieved excellent performance on the Arabic Multisource Fake News Detection (AMFND) dataset, reaching an F1 score of 94% using weighted averages. Moreover, changing the number of models in the ensemble has a slight effect on the performance. These key findings contribute to the advancement of fake news detection in Arabic, offering valuable insights for both academia and the industry

Utilization of transformer model in multimodal data fusion learning: Cross-modal knowledge transfer in the new generation learning space

In response to the difficulties in integrating multimodal data and insufficient model generalization ability in traditional cross-modal knowledge transfer, this article used the Transformer model to explore it in the new generation learning space. Firstly, the article analyzed the processing methods of data and models in cross-modal knowledge transfer, and explored the application of Transformer models in the learning space. This model used natural language processing to represent and extract textual features, Mel Frequency Cepstral Coefficients (MFCCs) to represent and extract audio features, and Faster R-CNN (Faster Region-based Convolutional Neural Network) to represent and extract image features. The article also discussed the implementation process of the Transformer model functionality. The experiment used data from four datasets, including Quora Question Pairs, to test the performance of the model’s cross-modal knowledge transfer through intelligent question answering and task analysis. In single type data testing, the accuracy and recall of the model in this article were better than the comparison model in the three types of data. The highest accuracy and recall in the test set were 91% and 93%, respectively. In the most challenging multimodal intelligent question answering test, the speech-image question answering method achieved an accuracy rate of 89% in answering open questions, indicating that the model had good multimodal data fusion ability. In the analysis experiment of 6 homework prone knowledge points on images with text annotations, the induction accuracy reached 85%, indicating that the model had strong generalization ability. The experimental results showed that the Transformer model had good cross-modal knowledge transfer performance, providing a reference for subsequent research on cross-modal knowledge transfer in the new generation learning space.

A Full-Process, Fine-Grained, and Quantitative Rehabilitation Assessment Platform Enabled by On-Skin Sensors and Multi-Task Gait Transformer Model.

Rehabilitation of patients with lower limb movement disorders is a gradual process, which requires full-process assessments to guide the implementation of rehabilitation plans. However, the current methods can only complete the assessment in one stage and lack objective and quantitative assessment strategies. Here, a full-process, fine-grained, and quantitative rehabilitation assessments platform (RAP) supported by on-skin sensors and a multi-task gait transformer (MG-former) model for patients with lower limb movement disorders is developed. The signal quality and sensitivity of on-skin sensor is improved by the synthesis of high-performance triboelectric material and structure design. The MG-former model can simultaneously perform multiple tasks including binary classification, multiclassification, and regression, corresponding to assessment of fall risk, walking ability, and rehabilitation progress, covering the whole rehabilitation cycle. The RAP can assess the walking ability of 23 hemiplegic patients, which has highly consistent results with the scores by the experienced physician. Furthermore, the MG-former model outputs fine-grained assessment results when performing regression task to track slight progress of patients that cannot be captured by conventional scales, facilitating adjustment of rehabilitation plans. This work provides an objective and quantitative platform, which is instructive for physicians and patients to implement effective strategy throughout the whole rehabilitation process.

Siamese based few-shot learning lightweight transformer model for coagulant and disinfectant dosage simultaneous regulation

Deep learning (DL) has emerged as a transformative and promising approach to address inefficient resource management due to imprecise chemical dosing in conventional manual-dependent drinking water treatment. However, the scarcity of high-quality data and limited computing resources of embedded devices hinder the development and deployment of state-of-the-art (SOTA) DL models. Herein, a novel lightweight Transformer (LighT) model integrated with the Siamese structure was proposed for simultaneous regulation of coagulant dosage (FeCl3 and PAC) and disinfectant dosage (NaClO) and compared with other attention-based DL models. The results demonstrate that LighT showed robust few-shot learning effectiveness and outperformed other models in multi-output prediction (R2 values of 0.99 and RMSE of 0.20 mg/L, 0.16 mg/L and 0.45 mg/L, respectively). The LighT model also exhibited enhanced cost-effectiveness with a 79.74 % compression in model parameters and a 36.14 % decrease in inference time. The LighT-based chemical dosage prediction program was sustained for one month of operation, which can save chemical dosage by 15.81 % while maintaining satisfactory product water quality, indicating the potential of LighT for economizing chemical dosing strategy in drinking water treatment.

Lightweight vision image transformer (LViT) model for skin cancer disease classification

Lightweight vision image transformer (LViT) model for skin cancer disease classification

Review of Existing Tools for Software Implementation of Digital Twins in the Power Industry

Digital twin technology is an important tool for the digitalization of the power industry. A digital twin is a concept that allows for the creation of virtual copies of real objects that can be used for technical state analysis, predictive analysis, and optimization of the operation of power systems and their components. Digital twins are used to address different issues, including the management of equipment reliability and efficiency, integration of renewable energy sources, and increased flexibility and adaptability of power grids. Digital twins can be developed with the use of specialized software solutions for designing, prototyping, developing, deploying, and supporting. The existing diversity of software requires systematization for a well-informed choice of digital twin’s development tool. It is necessary to take into account the technical characteristics of power systems and their elements (equipment of power plants, substations and power grids of power systems, mini- and microgrids). The reviews are dedicated to tools for creating digital twins in the power industry. The usage of Digital Twin Definition Language for the description data of electromagnetic, thermal, and hydrodynamic models of a power transformer is presented.

CALIPSO-based aerosol extinction profile estimation from MODIS and MERRA-2 data using a hybrid model of Transformer and CNN

Acquiring aerosol vertical distribution information is crucial to accurately quantify the aerosol radiation effect on climate and understand the environmental pollution mechanism of the atmosphere. Passive remote sensing has shown its capability to gain large-scale, high spatiotemporal resolution aerosol vertical information such as aerosol layer height (ALH). However, it is still challenging to extract detailed aerosol vertical distribution information, e.g., aerosol extinction profile (AEP), from passive observations. To fill this gap, this study proposed a hybrid model of Transformer and convolutional neural network (CNN) to estimate the AEP from passive multispectral remote sensing (MODIS) measurements with the aid of three-dimensional reanalysis data (MERRA-2). Specifically, the model is learned to estimate the AEP, which is called AproNet, by using the active space-borne lidar (CALIPSO) data as supervised information. Besides, we design a shape invariant loss (SIL) to better capture the shape characteristics of the AEP and incorporate an auxiliary scene awareness loss (SAL) to enhance the model's generalization capacity and physical reliability outside the CALIPSO orbit. The extensive quantitative experiments show that the AEPs estimated by the proposed model agree well with the CALIPSO measurements with an overall performance of IOA=0.821, R=0.800, MAE= 0.014, and RMSE= 0.041, respectively. Qualitative comparisons also demonstrate the model's reliability in estimating the aerosol three-dimensional spatial distribution. Independent year test and comparisons with ground-based lidar measurements further indicate the robustness of the proposed model despite some degradation in performance. However, the incompleteness and uncertainty of the CALIOP products limited the performance of the proposed model to some extent. In the future, the model needs to be further physically constrained and strengthened with more data sources to improve reliability. In general, this study paves the way for acquiring aerosol extinction profiles with high spatiotemporal resolution over a large geographical space.

Highly accurate assembly polishing with DeepPolisher.

Accurate genome assemblies are essential for biological research, but even the highest quality assemblies retain errors caused by the technologies used to construct them. Base-level errors are typically fixed with an additional polishing step that uses reads aligned to the draft assembly to identify necessary edits. However, current methods struggle to find a balance between over-and under-polishing. Here, we present an encoder-only transformer model for assembly polishing called DeepPolisher, which predicts corrections to the underlying sequence using Pacbio HiFi read alignments to a diploid assembly. Our pipeline introduces a method, PHARAOH (Phasing Reads in Areas Of Homozygosity), which uses ultra-long ONT data to ensure alignments are accurately phased and to correctly introduce heterozygous edits in falsely homozygous regions. We demonstrate that the DeepPolisher pipeline can reduce assembly errors by half, with a greater than 70% reduction in indel errors. We have applied our DeepPolisher-based pipeline to 180 assemblies from the next Human Pangenome Reference Consortium (HPRC) data release, producing an average predicted Quality Value (QV) improvement of 3.4 (54% error reduction) for the majority of the genome.

Stock price prediction using combined GARCH-AI models

The non-linear and non-stationary nature of financial time series data poses significant challenges for standalone statistical and neural network methods. While predictive modeling in finance often focuses on volatility, there is a notable lack of research on predicting actual stock prices, particularly in the African market. This study addresses this gap by utilizing Airtel stock data from Yahoo Finance, spanning June 28, 2019, to May 8, 2024. The research employs the GARCH model to extract statistical properties, which are then combined with historical prices and fed into LSTM, GRU, and Transformer models leading to GARCH-LSTM, GARCH-GRU, GARCH-Transfomer hybrid models. These hybrid models are benchmarked against standalone LSTM, GRU and Transfomer models using RMSE, MAE, MAPE, and R-squared metrics. Results indicate that hybrid models, especially GARCH-LSTM, significantly outperform standalone models. This integration of GARCH with advanced AI models offers a more robust framework for stock price prediction, enhancing accuracy and reliability in forecasting future prices.

Error correction method based on deep learning for improving the accuracy of conceptual rainfall-runoff model

Due to the complex runoff and concentration situation, flood forecasting for small and medium-sized catchments is very difficult. To improve the accuracy of flood forecasting, this study constructs a distributed model for flood forecasting based on the Xainanjiang (XAJ) model and the North China (NC) model respectively, and takes the deep learning model including LSTM and transformer to compare. Taking the Podi Basin and Shibazi Basin as study cases. LSTM, Transformer, and LSTPencoder models were used to correct the error of distributed models, and the differential evolution (DE) algorithm was used to optimize the model parameters. Taking the observed rainfall and distributed model results as input, the residuals of the simulation flow are fitted to improve the accuracy of the distributed model. The research results show that the NC model performs better than the XAJ, LSTM, and transformer models. Compared with the XAJ model, the average Nash-Sutcliffe coefficient of the NC model for the two catchments increased by 25.7 % and 5.87 % respectively. The performance of the NC+LSTPencoder model is better than other models. Compared with the NC model, the average Nash-Sutcliffe coefficient of the NC+LSTPencoder model for two catchments increased by 89.7 % and 1.12 % respectively, and the Root Mean Square Errors (RMSE) of the two catchments reduced by 79.1 % and 63.4 % respectively. The model proposed in this paper has strong correction ability, which has important significance for improving the accuracy of flood forecasts.

Automatic Question Answering From Large ESG Reports

ESG reports contain crucial information about the corporations' environmental impact, social responsibilities, and governance. Many compliance audits rely on answering questions based on these reports. Moreover, for tasks such as Scope 3 GHG emissions estimation, a company needs to look at the ESG reports of all its typically numerous suppliers to tabulate its own compliance reports. Manually finding specific information from these large documents is immensely time-consuming. This paper presents the first system that automatically answers questions from an ESG report, using advanced machine learning and natural language processing. The proposed system also locates and highlights a cropped screenshot from the report providing the answer. The authors devise two methods for inferring the textual answer, one based on a transformer model pre-trained for extractive question answering and another using a large language model. The task-agnostic method overcomes the challenge of the lengthiness of ESG reports in a cost-effective manner.

Making use of manufacturing process variations: Machine learning approaches for efficient medical and biological study-based image compression and lossless transmission

This research paper explores advanced machine learning techniques for compressing and transmitting medical and biological study-based images without loss of critical diagnostic information. We investigate deep learning architectures including autoencoders, generative adversarial networks (GANs) and transformer models optimized for various medical and biological imaging modalities. Our proposed hybrid compression pipeline combines semantic segmentation, region-adaptive encoding, and learned post-processing to achieve state-of-the-art compression ratios while preserving clinically relevant features. Extensive experiments on large-scale datasets of X-rays, CT scans, MRI scans and microscopy images demonstrate the efficacy of our approach in terms of compression performance, reconstruction quality, and computational efficiency. We also present a novel blockchain-based system for secure and lossless transmission of the compressed medical and biological data. Our findings indicate that machine learning-driven compression can enable more efficient storage and sharing of medical and biological images in resource-constrained healthcare and research environments.

A Deep Reinforcement Learning Method Based on a Transformer Model for the Flexible Job Shop Scheduling Problem

The flexible job shop scheduling problem (FJSSP), which can significantly enhance production efficiency, is a mathematical optimization problem widely applied in modern manufacturing industries. However, due to its NP-hard nature, finding an optimal solution for all scenarios within a reasonable time frame faces serious challenges. This paper proposes a solution that transforms the FJSSP into a Markov Decision Process (MDP) and employs deep reinforcement learning (DRL) techniques for resolution. First, we represent the state features of the scheduling environment using seven feature vectors and utilize a transformer encoder as a feature extraction module to effectively capture the relationships between state features and enhance representation capability. Second, based on the features of the jobs and machines, we design 16 composite dispatching rules from multiple dimensions, including the job completion rate, processing time, waiting time, and manufacturing resource utilization, to achieve flexible and efficient scheduling decisions. Furthermore, we project an intuitive and dense reward function with the objective of minimizing the total idle time of machines. Finally, to verify the performance and feasibility of the algorithm, we evaluate the proposed policy model on the Brandimarte, Hurink, and Dauzere datasets. Our experimental results demonstrate that the proposed framework consistently outperforms traditional dispatching rules, surpasses metaheuristic methods on larger-scale instances, and exceeds the performance of existing DRL-based scheduling methods across most datasets.

Application of three Transformer neural networks for short-term photovoltaic power prediction: A case study

In order to solve the potential safety hazards caused by the fluctuation of photovoltaic (PV) power generation, it is necessary to predict it in advance and take countermeasures as soon as possible. Based on the three models of vanilla Transformer, Informer, and Autoformer, this paper considers three prediction scenarios: zero-cost prediction, low-cost prediction, and high-cost prediction, and realizes the power prediction under two prediction horizons of 4 h and 24 h for a matrix of a centralized PV power station in Hubei Province, China. The results of some configurations meet the industry-recommended metric requirements, and the overall performance of the vanilla Transformer is better than Informer and Autoformer. After comparing the three models and different prediction intervals, and considering the practical industrial demand, this paper gives recommended configurations for both 4 h and 24 h predictions. The practical rolling prediction performance of the recommended configurations demonstrates the applicability and flexibility of the proposed methods.

Do firms listen to the ESG voices of minority investors? Evidence from China

This study examines whether and how the expression of ESG(Environmental, Social, and Governance) concerns by minority investors in their communication with firms affects firms' ESG performance. We employ the bidirectional encoder representation from transformer (BERT) model to identify ESG statements in questions asked by minority investors and construct sentence-level ESG concern indicators. The findings indicate that the more ESG-related questions investors ask in their interactions with firms, the better the firms' future ESG performance. This effect is more pronounced when investors' ESG-related questions are more negative, when managers show greater willingness to learn from investor interactions, when listed firms emphasize investor relations management, and when listed firms are held by ESG funds. Additional analyses reveal that pressure from the market and regulation may explain why listed companies listen to the ESG voice of minority investors. In addition, we find that minority investor voices do not lead to opportunistic greenwashing when driving improvements in firms' ESG performance. Overall, our findings suggest that minority investors promote corporate sustainability through active voice.

A Framework for Agricultural Intelligent Analysis Based on a Visual Language Large Model

Smart agriculture has become an inevitable trend in the development of modern agriculture, especially promoted by the continuous progress of large language models like chat generative pre-trained transformer (ChatGPT) and general language model (ChatGLM). Although these large models perform well in general knowledge learning, they still have certain limitations and errors when facing agricultural professional knowledge about crop disease identification, growth stage judgment, and so on. Agricultural data involves images and texts and other modalities, which play an important role in agricultural production and management. In order to better learn the characteristics of different modal data in agriculture, realize cross-modal data fusion, and thus understand complex application scenarios, we propose a framework AgriVLM that uses a large amount of agricultural data to fine-tune the visual language model to analyze agricultural data. It can fuse multimodal data and provide more comprehensive agricultural decision support. Specifically, it utilizes Q-former as a bridge between an image encoder and a language model to achieve a cross-modal fusion of agricultural images and text data. Then, we apply a Low-Rank adaptive to fine-tune the language model to achieve an alignment between agricultural image features and a pre-trained language model. The experimental results prove that AgriVLM demonstrates great performance in crop disease recognition and growth stage recognition, with recognition accuracy exceeding 90%, demonstrating its capability to analyze different modalities of agricultural data.

Efficient topic identification for urgent MOOC Forum posts using BERTopic and traditional topic modeling techniques

AbstractMOOC platforms provide a means of communication through forums, allowing learners to express their difficulties and challenges while studying various courses. Within these forums, some posts require urgent attention from instructors. Failing to respond promptly to these posts can contribute to higher dropout rates and lower course completion rates. While existing research primarily focuses on identifying urgent posts through various classification techniques, it has not adequately addressed the underlying reasons behind them. This research aims to delve into these reasons and assess the extent to which they vary. By understanding the root causes of urgency, instructors can effectively address these issues and provide appropriate support and solutions. BERTopic utilizes the advanced language capabilities of transformer models and represents an advanced approach in topic modeling. In this study, a comparison was conducted to evaluate the performance of BERTopic in topic modeling on MOOCs discussion forums, alongside traditional topic models such as LDA, LSI, and NMF. The experimental results revealed that the NMF and BERTopic models outperformed the other models. Specifically, the NMF model demonstrated superior performance when a lower number of topics was required, whereas the BERTopic model excelled in generating topics with higher coherence when a larger number of topics was needed.The results considering all urgent posts from the dataset were as follows: Optimal number of topics is 6 for NMF and 50 for BERTopic; coherence scores is 0.66 for NMF and 0.616 for BERTopic; and IRBO scores is 1 for both models. This highlights the BERTopic model capability to distinguish and extract diverse topics comprehensively and coherently, aiding in the identification of various reasons behind MOOC Forum posts.

Chemical Graph-Based Transformer Models for Yield Prediction of High-Throughput Cross-Coupling Reaction Datasets.

The chemical reaction yield is an important factor to determine the reaction conditions. Recently, many data-driven models for yield prediction using high-throughput experimentation datasets have been reported. In this study, we propose a neural network architecture based on the chemical graphs of the reaction components to predict the reaction yield. The proposed model is the sequential combination of a message-passing neural network and a transformer encoder (MPNN-Transformer). The reaction components are converted to molecular matrices by the first network, followed by the interplay of the reaction components in the second network after adding the embeddings of the compound roles in the chemical reaction. The predictive ability of the proposed models was compared with state-of-the-art yield prediction models using two high-throughput experimental datasets: the Buchwald-Hartwig cross-coupling (BHC) and Suzuki-Miyaura cross-coupling (SMC) reaction datasets. Overall, the MPNN-Transformer models showed high prediction accuracy for the BHC reaction datasets and some of the extrapolation-oriented SMC reaction datasets. These models also performed well when the training dataset size was relatively large. Furthermore, analyzing the poorly predicted reactions for the BHC reaction dataset revealed a limitation of the data-driven yield prediction approach based on the chemical structural similarity.

Indirect prediction of the 3D printability of polysaccharide gels using multiple machine learning (ML) models

In this paper, the capabilities of NIR spectroscopy and LF-NMR data were compared for rapidly predicting the rheological properties of polysaccharide gels and assessing their printability. Seven machine learning (ML) models were established for rheological property prediction based on partial least squares regression (PLSR), support vector regression (SVR), back propagation artificial neural network (BPANN), one-dimensional convolutional neural network (1D CNN), recurrent neural network (RNN), long short-term memory (LSTM), and Transformer. The results showed that among the seven models, the SVR, BPANN, and 1D CNN models based on NIR spectroscopy effectively predicted the rheological parameters of polysaccharide gels, with the highest R2 in the prediction set reaching 0.9796 and the highest RPD reaching 7.0708. For most polysaccharide gels, using the LF-NMR relaxation time distribution curves provided better predictions of rheological properties than using transverse relaxation time and peak area. Among the seven models, the PLSR, SVR, 1D CNN, and Transformer models effectively predicted the rheological characteristics based on LF-NMR parameters, with the highest R2 in the prediction set reaching 0.9869 and the highest RPD reaching 8.7220. This study successfully established a prediction system for the rheological behaviors and 3D printing performance of polysaccharide gels using NIR spectroscopy and LF-NMR data combined with ML methods, achieving an intelligent assessment of the 3D printing behavior of polysaccharide gels.