Fine-tuning Pre-trained Models Research Articles

The application of explainable artificial intelligence methods to models for automatic creativity assessment

ObjectiveThe study is devoted to comparing various models based on Artificial Intelligence to determine the level of creativity based on drawings performed using the Urban test, as well as analyzing the results of applying explainable artificial intelligence methods to a trained model to identify the most relevant features in drawings that influence the model’s prediction.MethodsThe dataset is represented by a set of 1,823 scanned forms of drawings of participants performed according to the Urban test. The test results of each participant were assessed by an expert. Preprocessed images were used for fine-tuning pre-trained models such as MobileNet, ResNet18, AlexNet, DenseNet, ResNext, EfficientNet, ViT with additional linear layers to predict the participant’s score. Visualization of the areas that are of greatest importance from the point of view of the model was carried out using the Gradient-weighted Class Activation Mapping (Grad-CAM) method.ResultsTrained models based on MobileNet showed the highest prediction accuracy rate of 76%. The results of the application of explainable artificial intelligence demonstrated areas of interest that correlated with the criteria for expert assessment according to the Urban test. Analysis of erroneous predictions of the model in terms of interpretation of areas of interest made it possible to clarify the features of the drawing on which the model relies, contrary to the expert.ConclusionThe study demonstrated the possibility of using neural network methods for automated diagnosis of the level of creativity according to the Urban test based on the respondents’ drawings. The application of explainable artificial intelligence methods to the trained model demonstrated the compliance of the identified activation zones with the rules of expert assessment according to the Urban test.

Disambiguating Clinical Abbreviations by One-to-All Classification: Algorithm Development and Validation Study.

Electronic medical records store extensive patient data and serve as a comprehensive repository, including textual medical records like surgical and imaging reports. Their utility in clinical decision support systems is substantial, but the widespread use of ambiguous and unstandardized abbreviations in clinical documents poses challenges for natural language processing in clinical decision support systems. Efficient abbreviation disambiguation methods are needed for effective information extraction. This study aims to enhance the one-to-all (OTA) framework for clinical abbreviation expansion, which uses a single model to predict multiple abbreviation meanings. The objective is to improve OTA by developing context-candidate pairs and optimizing word embeddings in Bidirectional Encoder Representations From Transformers (BERT), evaluating the model's efficacy in expanding clinical abbreviations using real data. Three datasets were used: Medical Subject Headings Word Sense Disambiguation, University of Minnesota, and Chia-Yi Christian Hospital from Ditmanson Medical Foundation Chia-Yi Christian Hospital. Texts containing polysemous abbreviations were preprocessed and formatted for BERT. The study involved fine-tuning pretrained models, ClinicalBERT and BlueBERT, generating dataset pairs for training and testing based on Huang et al's method. BlueBERT achieved macro- and microaccuracies of 95.41% and 95.16%, respectively, on the Medical Subject Headings Word Sense Disambiguation dataset. It improved macroaccuracy by 0.54%-1.53% compared to two baselines, long short-term memory and deepBioWSD with random embedding. On the University of Minnesota dataset, BlueBERT recorded macro- and microaccuracies of 98.40% and 98.22%, respectively. Against the baselines of Word2Vec + support vector machine and BioWordVec + support vector machine, BlueBERT demonstrated a macroaccuracy improvement of 2.61%-4.13%. This research preliminarily validated the effectiveness of the OTA method for abbreviation disambiguation in medical texts, demonstrating the potential to enhance both clinical staff efficiency and research effectiveness.

StochCA: A novel approach for exploiting pretrained models with cross-attention

Utilizing large-scale pretrained models is a well-known strategy to enhance performance on various target tasks. It is typically achieved through fine-tuning pretrained models on target tasks. However, naï ve fine-tuning may not fully leverage knowledge embedded in pretrained models. In this study, we introduce a novel fine-tuning method, called stochastic cross-attention (StochCA), specific to Transformer architectures. This method modifies the Transformer’s self-attention mechanism to selectively utilize knowledge from pretrained models during fine-tuning. Specifically, in each block, instead of self-attention, cross-attention is performed stochastically according to the predefined probability, where keys and values are extracted from the corresponding block of a pretrained model. By doing so, queries and channel-mixing multi-layer perceptron layers of a target model are fine-tuned to target tasks to learn how to effectively exploit rich representations of pretrained models. To verify the effectiveness of StochCA, extensive experiments are conducted on benchmarks in the areas of transfer learning and domain generalization, where the exploitation of pretrained models is critical. Our experimental results show the superiority of StochCA over state-of-the-art approaches in both areas. Furthermore, we demonstrate that StochCA is complementary to existing approaches, i.e., it can be combined with them to further improve performance. We release the code at https://github.com/daintlab/stochastic_cross_attention.

The Superiority of Fine-tuning over Full-training for the Efficient Diagnosis of COPD from CXR Images

This research investigates the use of deep learning for diagnosing lung diseases like Chronic Obstructive Pulmonary Disease (COPD) using Chest X-rays (CXR). The study compares the impact of deep learning on improving these diagnoses by comparing the performance of models trained from the scratch with those enhanced through fine-tuning established architectures like InceptionV3, ResNet50, and VGG-19. The study revealed that fine-tuning pre-trained models offers significant benefits: faster convergence, improved stability, and increased accuracy. Data augmentation techniques were found to be particularly useful when dealing with limited or unbalanced datasets. A custom CNN model, Iyke-Net, showed promising results when fine-tuned. Interestingly, it was observed that models using grayscale images outperformed those using colour images in disease classification, suggesting that colour information might be less critical than previously thought for certain diagnostic procedures. The study emphasizes the importance of balancing model complexity with computational efficiency and diagnostic accuracy. It advocates for refining existing deep learning models for COPD diagnosis from CXR images, paving the way for further innovations in AI-enhanced medical diagnostics.

AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs

Fine-tuning pre-trained models has recently yielded remarkable performance gains in graph neural networks (GNNs). In addition to pre-training techniques, inspired by the latest work in the natural language fields, more recent work has shifted towards applying effective fine-tuning approaches, such as parameter-efficient fine-tuning (PEFT). However, given the substantial differences between GNNs and transformer-based models, applying such approaches directly to GNNs proved to be less effective. In this paper, we present a comprehensive comparison of PEFT techniques for GNNs and propose a novel PEFT method specifically designed for GNNs, called AdapterGNN. AdapterGNN preserves the knowledge of the large pre-trained model and leverages highly expressive adapters for GNNs, which can adapt to downstream tasks effectively with only a few parameters, while also improving the model's generalization ability. Extensive experiments show that AdapterGNN achieves higher performance than other PEFT methods and is the only one consistently surpassing full fine-tuning (outperforming it by 1.6% and 5.7% in the chemistry and biology domains respectively, with only 5% and 4% of its parameters tuned) with lower generalization gaps. Moreover, we empirically show that a larger GNN model can have a worse generalization ability, which differs from the trend observed in large transformer-based models. Building upon this, we provide a theoretical justification for PEFT can improve generalization of GNNs by applying generalization bounds. Our code is available at https://github.com/Lucius-lsr/AdapterGNN.

Trade-Offs in Fine-Tuned Diffusion Models between Accuracy and Interpretability

Recent advancements in diffusion models have significantly impacted the trajectory of generative machine learning re-search, with many adopting the strategy of fine-tuning pre-trained models using domain-specific text-to-image datasets. Notably, this method has been readily employed for medical applications, such as X-ray image synthesis, leveraging the plethora of associated radiology reports. Yet, a prevailing concern is the lack of assurance on whether these models genuinely comprehend their generated content. With the evolution of text conditional image generation, these models have grown potent enough to facilitate object localization scrutiny. Our research underscores this advancement in the critical realm of medical imaging, emphasizing the crucial role of interpretability. We further unravel a consequential trade-off between image fidelity – as gauged by conventional metrics – and model interpretability in generative diffusion models. Specifically, the adoption of learnable text encoders when fine-tuning results in diminished interpretability. Our in-depth exploration uncovers the underlying factors responsible for this divergence. Consequently, we present a set of design principles for the development of truly interpretable generative models. Code is available at https://github.com/MischaD/chest-distillation.

APPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained Models

APPT: Boosting Automated Patch Correctness Prediction via Fine-Tuning Pre-Trained Models

Effective Approach for Fine-Tuning Pre-Trained Models for the Extraction of Texts From Source Codes

This study introduces SR-Text, a robust approach leveraging pre-trained models like BERT and T5 for enhanced text extraction from source codes. Addressing the limitations of traditional manual summarization, our methodology focuses on fine-tuning these models to better understand and generate contextual summaries, thus overcoming challenges such as long-term dependency and dataset quality issues. We conduct a detailed analysis of programming language syntax and semantics to develop syntax-aware text retrieval techniques, significantly boosting the accuracy and relevance of the texts extracted. The paper also explores a hybrid approach that integrates statistical machine learning with rule-based methods, enhancing the robustness and adaptability of our text extraction processes across diverse coding styles and languages. Empirical results from a meticulously curated dataset demonstrate marked improvements in performance metrics: precision increased by 15%, recall by 20%, and an F1 score enhancement of 18%. These improvements underscore the effectiveness of using advanced machine learning models in software engineering tasks. This research not only paves the way for future work in multilingual code summarization but also discusses broader implications for automated software analysis tools, proposing directions for future research to further refine and expand this methodology.

Comparative analysis of BERT and FastText representations on crowdfunding campaign success prediction.

Crowdfunding has become a popular financing method, attracting investors, businesses, and entrepreneurs. However, many campaigns fail to secure funding, making it crucial to reduce participation risks using artificial intelligence (AI). This study investigates the effectiveness of advanced AI techniques in predicting the success of crowdfunding campaigns on Kickstarter by analyzing campaign blurbs. We compare the performance of two widely used text representation models, bidirectional encoder representations from transformers (BERT) and FastText, in conjunction with long-short term memory (LSTM) and gradient boosting machine (GBM) classifiers. Our analysis involves preprocessing campaign blurbs, extracting features using BERT and FastText, and evaluating the predictive performance of these features with LSTM and GBM models. All experimental results show that BERT representations significantly outperform FastText, with the highest accuracy of 0.745 achieved using a fine-tuned BERT model combined with LSTM. These findings highlight the importance of using deep contextual embeddings and the benefits of fine-tuning pre-trained models for domain-specific applications. The results are benchmarked against existing methods, demonstrating the superiority of our approach. This study provides valuable insights for improving predictive models in the crowdfunding domain, offering practical implications for campaign creators and investors.

Public Perception of ChatGPT and Transfer Learning for Tweets Sentiment Analysis Using Wolfram Mathematica

Understanding public opinion on ChatGPT is crucial for recognizing its strengths and areas of concern. By utilizing natural language processing (NLP), this study delves into tweets regarding ChatGPT to determine temporal patterns, content features, and topic modeling and perform a sentiment analysis. Analyzing a dataset of 500,000 tweets, our research shifts from conventional data science tools like Python and R to exploit Wolfram Mathematica’s robust capabilities. Additionally, with the aim of solving the problem of ignoring semantic information in the LDA model feature extraction, a synergistic methodology entwining LDA, GloVe embeddings, and K-Nearest Neighbors (KNN) clustering is proposed to categorize topics within ChatGPT-related tweets. This comprehensive strategy ensures semantic, syntactic, and topical congruence within classified groups by utilizing the strengths of probabilistic modeling, semantic embeddings, and similarity-based clustering. While built-in sentiment classifiers often fall short in accuracy, we introduce four transfer learning techniques from the Wolfram Neural Net Repository to address this gap. Two of these techniques involve transferring static word embeddings, “GloVe” and “ConceptNet”, which are further processed using an LSTM layer. The remaining techniques center on fine-tuning pre-trained models using scantily annotated data; one refines embeddings from language models (ELMo), while the other fine-tunes bidirectional encoder representations from transformers (BERT). Our experiments on the dataset underscore the effectiveness of the four methods for the sentiment analysis of tweets. This investigation augments our comprehension of user sentiment towards ChatGPT and emphasizes the continued significance of exploration in this domain. Furthermore, this work serves as a pivotal reference for scholars who are accustomed to using Wolfram Mathematica in other research domains, aiding their efforts in text analytics on social media platforms.

Automatic labeling of Parkinson's Disease gait videos with weak supervision.

Motor dysfunction in Parkinson's Disease (PD) patients is typically assessed by clinicians employing the Movement Disorder Society's Unified Parkinson's Disease Rating Scale (MDS-UPDRS). Such comprehensive clinical assessments are time-consuming, expensive, semi-subjective, and may potentially result in conflicting labels across different raters. To address this problem, we propose an automatic, objective, and weakly-supervised method for labeling PD patients' gait videos. The proposed method accepts videos of patients and classifies their gait scores as normal (Gait score in MDS-UPDRS = 0) or PD (MDS-UPDRS ≥ 1). Unlike previous work, the proposed method does not require a priori MDS-UPDRS ratings for training, utilizing only domain-specific knowledge obtained from neurologists. We propose several labeling functions that classify patients' gait and use a generative model to learn the accuracy of each labeling function in a self-supervised manner. Since results depended upon the estimated values of the patients' 3D poses, and existing pre-trained 3D pose estimators did not yield accurate results, we propose a weakly-supervised 3D human pose estimation method for fine-tuning pre-trained models in a clinical setting. Using leave-one-out evaluations, the proposed method obtains an accuracy of 89% on a dataset of 29 PD subjects - a significant improvement compared to previous work by 7%-10% depending upon the dataset. The method obtained state-of-the-art results on the Human3.6M dataset. Our results suggest that the use of labeling functions may provide a robust means to interpret and classify patient-oriented videos involving motor tasks.

Interpretability Study of Transfer Learning for Emotion Classification

Transfer learning is a powerful technique for improving the performance of deep neural networks on a target task with limited labeled data. However, it is not always clear why pre-trained models on unrelated tasks, such as ImageNet, can be effective for emotion recognition tasks. This paper evaluates the performance of transfer learning for facial expression recognition tasks and investigates the influence of network depth on performance using three different typical pre-trained models: VGG16, MobileNet, and ResNet50. The layers of the pre-trained models are frozen, and their output layer is substituted with a new output layer that corresponds to the seven-class emotion classes in the target dataset. The study demonstrates that transfer learning with pre-trained models, even though with limited labelled dataset regarding facial expressions, can be an effective approach for improving the performance of deep neural networks on emotion recognition tasks. The findings provide insights into the interpretability of transfer learning for facial expression recognition tasks and highlight the importance of fine-tuning pre-trained models for optimal performance. Additionally, the study also offers ideas for future model design in facial expression recognition tasks.

Prompt-Based Tuning of Transformer Models for Multi-Center Medical Image Segmentation of Head and Neck Cancer.

Medical image segmentation is a vital healthcare endeavor requiring precise and efficient models for appropriate diagnosis and treatment. Vision transformer (ViT)-based segmentation models have shown great performance in accomplishing this task. However, to build a powerful backbone, the self-attention block of ViT requires large-scale pre-training data. The present method of modifying pre-trained models entails updating all or some of the backbone parameters. This paper proposes a novel fine-tuning strategy for adapting a pretrained transformer-based segmentation model on data from a new medical center. This method introduces a small number of learnable parameters, termed prompts, into the input space (less than 1% of model parameters) while keeping the rest of the model parameters frozen. Extensive studies employing data from new unseen medical centers show that the prompt-based fine-tuning of medical segmentation models provides excellent performance regarding the new-center data with a negligible drop regarding the old centers. Additionally, our strategy delivers great accuracy with minimum re-training on new-center data, significantly decreasing the computational and time costs of fine-tuning pre-trained models. Our source code will be made publicly available.

Acoustic Emission Source Localization using Deep Transfer Learning and Finite Element Modeling–based Knowledge Transfer

This paper presents a novel data-driven approach to localize two types of acoustic emission sources in an aluminum plate, namely a Hsu-Nielsen source, which simulates a crack-like source, and steel ball impacts of varying diameters acting as the impact source. While deep neural networks have shown promise in previous studies, achieving high accuracy requires a large amount of training data, which may not always be feasible. To address this challenge, we investigated the applicability of transfer learning to address the issue of limited training data. Our approach involves transferring knowledge learned from numerical modeling to the experimental domain to localize nine different source locations. In the process, we evaluated six deep learning architectures using tenfold cross-validation and demonstrated the potential of transfer learning for efficient acoustic emission source localization, even with limited experimental data. This study contributes to the growing demand for running deep learning models with limited capacity and training time and highlights the promise of transfer learning methods such as fine-tuning pretrained models on large semi-related datasets.

On the Effectiveness of Parameter-Efficient Fine-Tuning

Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task. Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts. However, such kind of methods is still not well understood. Some natural questions arise: How does the parameter sparsity lead to promising performance? Why is the model more stable than the fully fine-tuned models? How to choose the tunable parameters? In this paper, we first categorize the existing methods into random approaches, rule-based approaches, and projection-based approaches based on how they choose which parameters to tune. Then, we show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. We indicate that the sparsity is actually imposing a regularization on the original model by controlling the upper bound of the stability. Such stability leads to better generalization capability which has been empirically observed in a lot of recent research works. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters. Currently, the random and rule-based methods do not utilize task-specific data information while the projection-based approaches suffer from the projection discontinuity problem. To better choose the tunable parameters, we propose a novel Second-order Approximation Method (SAM) which approximates the original problem with an analytically solvable optimization function. The tunable parameters are determined by directly optimizing the approximation function. We conduct extensive experiments on several tasks. The experimental results show that our proposed SAM model outperforms many strong baseline models and it also verifies our theoretical analysis. The source code of this paper can be obtained from https://github.com/fuzihaofzh/AnalyzeParameterEff\/icientFinetune .

Math Function Recognition with Fine-Tuning Pre-Trained Models

A Mathematical Function Recognition (MFR) is an important research direction for efficient downstream math tasks such as information retrieval, knowledge extraction, and question answering. The aim of this task is to identify and classify mathematical function into a predefined set of function. However, the lack of annotated data is the bottleneck in the development of an MFR automated model. We begin this paper by describing our approach to creating a labelled dataset for MFR. Then, to identify five categories of mathematical functions, we fine-tuned a set of common pre-trained models: BERT base-cased, BERT baseuncased, DistilBERT-cased, and DistilBERT-uncased. As a result, our contributions in this paper include: (1) an annotated MFR dataset that future researchers can use; and (2) SOTA results obtained by finetuning pre-trained models for the MFR task. Our experiments demonstrate that the proposed approach achieved a high-quality recognition, with an F1 score of 96.80% on a held-out test set provided by DistilBERT-cased model.

Knowledge-guided multi-granularity GCN for ABSA

Aspect-based sentiment analysis aims to determine sentiment polarities toward specific aspect terms within the same sentence or document. Most recent studies adopted attention-based neural network models to implicitly connect aspect terms with context words. However, these studies were limited by insufficient interaction between aspect terms and opinion words, leading to poor performance on robustness test sets. In addition, we have found that robustness test sets create new sentences that interfere with the original information of a sentence, which often makes the text too long and leads to the problem of long-distance dependence. Simultaneously, these new sentences produce more non-target aspect terms, misleading the model because of the lack of relevant knowledge guidance. This study proposes a knowledge guided multi-granularity graph convolutional neural network (KMGCN) to solve these problems. The multi-granularity attention mechanism is designed to enhance the interaction between aspect terms and opinion words. To address the long-distance dependence, KMGCN uses a graph convolutional network that relies on a semantic map based on fine-tuning pre-trained models. In particular, KMGCN uses a mask mechanism guided by conceptual knowledge to encounter more aspect terms (including target and non-target aspect terms). Experiments are conducted on 12 SemEval-2014 variant benchmarking datasets, and the results demonstrated the effectiveness of the proposed framework.

A conditional GAN-based approach for enhancing transfer learning performance in few-shot HCR tasks

Supervised learning with the restriction of a few existing training samples is called Few-Shot Learning. FSL is a subarea that puts deep learning performance in a gap, as building robust deep networks requires big training data. Using transfer learning in FSL tasks is an acceptable way to avoid the challenge of building new deep models from scratch. Transfer learning methodology considers borrowing the architecture and parameters of a previously trained model on a large-scale dataset and fine-tuning it for low-data target tasks. But practically, fine-tuning pretrained models in target FSL tasks suffers from overfitting. The few existing samples are not enough to correctly adjust the pretrained model’s parameters to provide the best fit for the target task. In this study, we consider mitigating the overfitting problem when applying transfer learning in few-shot Handwritten Character Recognition (HCR) tasks. A data augmentation approach based on Conditional Generative Adversarial Networks is introduced. CGAN is a generative model that can create artificial instances that appear more real and indistinguishable from the original samples. CGAN helps generate extra samples that hold the possible variations of human handwriting instead of applying traditional image transformations. These transformations are low-level, data-independent operations, and only produce augmented samples with limited diversity. The introduced approach was evaluated in fine-tuning the three pretrained models: AlexNet, VGG-16, and GoogleNet. The results show that the samples generated by CGAN can enhance transfer learning performance in few-shot HCR tasks. This is by achieving model fine-tuning with fewer epochs and by increasing the model’s F1-score and decreasing the Generalization Error (E_{test}).

Towards Multimodal Vision-Language Models Generating Non-generic Text

While text generated by current vision-language models may be accurate and syntactically correct, it is often general. Recent work has used optical character recognition to supplement visual information with text extracted from an image. In many cases, using text in the image improves the specificity and usefulness of generated text. We contend that vision-language models can benefit from additional information extracted from an image. We modify previous multimodal frameworks to accept relevant information from a number of auxiliary classifiers. In particular, we focus on person names as an additional set of tokens and create a novel image-caption dataset to facilitate captioning with person names. The dataset, Politicians and Athletes in Captions (PAC), consists of captioned images of well-known people in context. By fine-tuning pretrained models with this dataset, we demonstrate a model that can naturally integrate facial recognition tokens into generated text by training on limited data.

Deep Learning of Ultrasound Imaging for Evaluating Ambulatory Function of Individuals with Duchenne Muscular Dystrophy.

Duchenne muscular dystrophy (DMD) results in loss of ambulation and premature death. Ultrasound provides real-time, safe, and cost-effective routine examinations. Deep learning allows the automatic generation of useful features for classification. This study utilized deep learning of ultrasound imaging for classifying patients with DMD based on their ambulatory function. A total of 85 individuals (including ambulatory and nonambulatory subjects) underwent ultrasound examinations of the gastrocnemius for deep learning of image data using LeNet, AlexNet, VGG-16, VGG-16TL, VGG-19, and VGG-19TL models (the notation TL indicates fine-tuning pretrained models). Gradient-weighted class activation mapping (Grad-CAM) was used to visualize features recognized by the models. The classification performance was evaluated using the confusion matrix and receiver operating characteristic (ROC) curve analysis. The results show that each deep learning model endows muscle ultrasound imaging with the ability to enable DMD evaluations. The Grad-CAMs indicated that boundary visibility, muscular texture clarity, and posterior shadowing are relevant sonographic features recognized by the models for evaluating ambulatory function. Of the proposed models, VGG-19 provided satisfying classification performance (the area under the ROC curve: 0.98; accuracy: 94.18%) and feature recognition in terms of physical characteristics. Deep learning of muscle ultrasound is a potential strategy for DMD characterization.