Fine-tuned Model Research Articles

MEHunter: Transformer-based mobile element variant detection from long reads.

Mobile genetic elements (MEs) are heritable mutagens that significantly contribute to genetic diseases. The advent of long-read sequencing technologies, capable of resolving large DNA fragments, offers promising prospects for the comprehensive detection of ME variants (MEVs). However, achieving high precision while maintaining recall performance remains challenging mainly brought by the variable length and similar content of MEV signatures, which are often obscured by the noise in long reads. Here, we propose MEHunter, a high-performance MEV detection approach utilizing a fine-tuned transformer model adept at identifying potential MEVs with fragmented features. Benchmark experiments on both simulated and real datasets demonstrate that MEHunter consistently achieves higher accuracy and sensitivity than the state-of-the-art tools. Furthermore, it is capable of detecting novel potentially individual-specific MEVs that have been overlooked in published population projects. MEHunter is available from https://github.com/120L021101/MEHunter. Supplementary data are available at Bioinformatics online.

The InterVision Framework: An Enhanced Fine-Tuning Deep Learning Strategy for Auto-Segmentation in Head and Neck.

Adaptive radiotherapy (ART) workflows are increasingly adopted to achieve dose escalation and tissue sparing under dynamic anatomical conditions. However, recontouring and time constraints hinder the implementation of real-time ART workflows. Various auto-segmentation methods, including deformable image registration, atlas-based segmentation, and deep learning-based segmentation (DLS), have been developed to address these challenges. Despite the potential of DLS methods, clinical implementation remains difficult due to the need for large, high-quality datasets to ensure model generalizability. This study introduces an InterVision framework for segmentation. The InterVision framework can interpolate or create intermediate visuals between existing images to generate specific patient characteristics. The InterVision model is trained in two steps: (1) generating a general model using the dataset, and (2) tuning the general model using the dataset generated from the InterVision framework. The InterVision framework generates intermediate images between existing patient image slides using deformable vectors, effectively capturing unique patient characteristics. By creating a more comprehensive dataset that reflects these individual characteristics, the InterVision model demonstrates the ability to produce more accurate contours compared to general models. Models are evaluated using the volumetric dice similarity coefficient (VDSC) and the Hausdorff distance 95% (HD95%) for 18 structures in 20 test patients. As a result, the Dice score was 0.81 ± 0.05 for the general model, 0.82 ± 0.04 for the general fine-tuning model, and 0.85 ± 0.03 for the InterVision model. The Hausdorff distance was 3.06 ± 1.13 for the general model, 2.81 ± 0.77 for the general fine-tuning model, and 2.52 ± 0.50 for the InterVision model. The InterVision model showed the best performance compared to the general model. The InterVision framework presents a versatile approach adaptable to various tasks where prior information is accessible, such as in ART settings. This capability is particularly valuable for accurately predicting complex organs and targets that pose challenges for traditional deep learning algorithms.

Extractive text summarization on medical insights using fine-tuned transformers

ABSTRACT Text summarization is a fundamental Natural language processing task that plays a crucial role in efficiently condensing large textual documents into concise and clear summaries for human comprehension. The amount of data being generated in the medical domain nowadays requires substantial application of the current deep learning approaches such as transformers. The main goal of this research is to extract relevant summaries from the abstracts of the research articles published related to cancer, blood cancer, tinnitus, and Alzheimer's. As the domain-related data requires special attention, our approach uses a fine-tuned transformer model, to guarantee that the summaries produced are not only brief but also accurate. Moreover, as a part of this research, we have effectively collected the information from PubMed and also prepared the data for analysis. A comparative analysis of the Bidirectional and Auto-Regressive Transformers (BART), Text-to-Text Transfer Transformer (T5), Textrank, and Lexrank models on the dataset is carried out in this study to understand the medical insights effectively. The fine-tuned transformer's performance in comparison with other models brings out a newer dimension for future studies.

Taking it further: Leveraging pseudo-labels for field delineation across label-scarce smallholder regions

Satellite-based field delineation has entered a quasi-operational stage due to recent advances in machine learning for computer vision. Transfer learning allows for the resource-efficient transfer of pre-trained field delineation models across heterogeneous geographies. However, the scarcity of labeled data for complex and dynamic smallholder landscapes remains a major bottleneck. The key innovation of this study is to overcome this challenge by using pre-trained models to generate sparse (i.e., not fully annotated) field delineation pseudo-labels for fine-tuning models across geographies and sensor characteristics. We build on a FracTAL ResUNet trained for crop field delineation in India (median field size of 0.24 ha) based on multi-spectral imagery at 1.5 m spatial resolution. We use this model to generate pseudo-labels for the use in Northern Mozambique (median field size of 0.06 ha) based on sub-meter resolution true-color satellite imagery. We designed multiple pseudo-label selection strategies based on field-level probability scores and compared the quantities, area properties, seasonal distribution, and spatial agreement of the pseudo-labels against human-annotated training labels (n = 1,512). We then used the human-annotated labels and the pseudo-labels for model fine-tuning and compared predictions against human field annotations (n = 2,199). We evaluated performance with regards to object-level spatial agreement and site-level field size estimation. Our results indicate i) a good baseline performance of the pre-trained model in both field delineation (mean intersection over union (mIoU) of 0.634) and field size estimation (mean root mean squared error (mRMSE) of 0.071 ha), and ii) the added value of regional fine-tuning with performance improvements in nearly all experiments (mIoU increases of up to 0.060, mRMSE decreases of up to 0.034 ha). Moreover, we found iii) substantial performance increases when using only pseudo-labels (up to 77 % of the mIoU increases and 68 % of the mRMSE decreases obtained by human-annotated labels), and iv) additional performance increases (mIoU+0.008, mRMSE: −0.003 ha) when complementing human annotations with pseudo-labels. Pseudo-labels are architecture-agnostic, can be efficiently generated at scale, and thus facilitate domain adaptation in label-scarce settings. The workflow presented here is a stepping stone for overcoming the persisting challenges in mapping heterogeneous smallholder agriculture.

FLIGHTED: Inferring Fitness Landscapes from Noisy High-Throughput Experimental Data.

Machine learning (ML) for protein design requires large protein fitness datasets generated by high-throughput experiments for training, fine-tuning, and benchmarking models. However, most models do not account for experimental noise inherent in these datasets, harming model performance and changing model rankings in benchmarking studies. Here we develop FLIGHTED, a Bayesian method of accounting for uncertainty by generating probabilistic fitness landscapes from noisy high-throughput experiments. We demonstrate how FLIGHTED can improve model performance on two categories of experiments: single-step selection assays, such as phage display and SELEX, and a novel high-throughput assay called DHARMA that ties activity to base editing. We then compare the performance of standard machine-learning models on fitness landscapes generated with and without FLIGHTED. Accounting for noise significantly improves model performance, especially of CNN architectures, and changes relative rankings on numerous common benchmarks. Based on our new benchmarking with FLIGHTED, data size, not model scale, currently appears to be limiting the performance of protein fitness models, and the choice of top model architecture matters more than the protein language model embedding. Collectively, our results indicate that FLIGHTED can be applied to any high-throughput assay and any machine learning model, making it straightforward for protein designers to account for experimental noise when modeling protein fitness.

Detecting floating litter in freshwater bodies with semi-supervised deep learning

Researchers and practitioners have extensively utilized supervised Deep Learning methods to quantify floating litter in rivers and canals. These methods require the availability of large amount of labeled data for training. The labeling work is expensive and laborious, resulting in small open datasets available in the field compared to the comprehensive datasets for computer vision, e.g., ImageNet. Fine-tuning models pre-trained on these larger datasets helps improve litter detection performances and reduces data requirements. Yet, the effectiveness of using features learned from generic datasets is limited in large-scale monitoring, where automated detection must adapt across different locations, environmental conditions, and sensor settings. To address this issue, we propose a two-stage semi-supervised learning method to detect floating litter based on the Swapping Assignments between multiple Views of the same image (SwAV). SwAV is a self-supervised learning approach that learns the underlying feature representation from unlabeled data. In the first stage, we used SwAV to pre-train a ResNet50 backbone architecture on about 100k unlabeled images. In the second stage, we added new layers to the pre-trained ResNet50 to create a Faster R-CNN architecture, and fine-tuned it with a limited number of labeled images (≈1.8k images with 2.6k annotated litter items). We developed and validated our semi-supervised floating litter detection methodology for images collected in canals and waterways of Delft (the Netherlands) and Jakarta (Indonesia). We tested for out-of-domain generalization performances in a zero-shot fashion using additional data from Ho Chi Minh City (Vietnam), Amsterdam and Groningen (the Netherlands). We benchmarked our results against the same Faster R-CNN architecture trained via supervised learning alone by fine-tuning ImageNet pre-trained weights. The findings indicate that the semi-supervised learning method matches or surpasses the supervised learning benchmark when tested on new images from the same training locations. We measured better performances when little data (≈200 images with about 300 annotated litter items) is available for fine-tuning and with respect to reducing false positive predictions. More importantly, the proposed approach demonstrates clear superiority for generalization on the unseen locations, with improvements in average precision of up to 12.7%. We attribute this superior performance to the more effective high-level feature extraction from SwAV pre-training from relevant unlabeled images. Our findings highlight a promising direction to leverage semi-supervised learning for developing foundational models, which have revolutionized artificial intelligence applications in most fields. By scaling our proposed approach with more data and compute, we can make significant strides in monitoring to address the global challenge of litter pollution in water bodies.

Generative AI model trained by molecular dynamics for rapid mechanical design of architected graphene

Generative artificial intelligence (AI) is shown to be a useful tool to automatically learn from existing information and generate new information based on their connections, but its usage for quantitative mechanical research is less understood. Here, we focus on the structure-mechanics relationship of architected graphene as graphene with void defects of specific patterns. We use Molecular Dynamics (MD) to simulate uniaxial tension on architected graphene, extract the von Mises stress field in mechanical loading, and use the results to train a fine-tuned generative AI model through a Low-Rank Adaptation method. This model enables the freely designed architected graphene structures and predicts its associated stress field in uniaxial tension loading through simple descriptive language. We demonstrate that the fine-tuned model can be established with a few training images and can quantitatively predict the stress field for graphene with various defect geometries and distributions not included in the training set. We validate the accuracy of the stress field with MD simulations. Moreover, we illustrate that our generative AI model can predict the stress field from a schematic drawing of the architected graphene through image-to-image generation. These features underscore the promising future for employing advanced generative AI models in end-to-end advanced nanomaterial design and characterization, enabling the creation of functional, structural materials without using complex numerical modeling and data processing.

Uniqorn: Unified question answering over RDF knowledge graphs and natural language text

Question answering over RDF data like knowledge graphs has been greatly advanced, with a number of good systems providing crisp answers for natural language questions or telegraphic queries. Some of these systems incorporate textual sources as additional evidence for the answering process, but cannot compute answers that are present in text alone. Conversely, the IR and NLP communities have addressed QA over text, but such systems barely utilize semantic data and knowledge. This paper presents a method for complex questions that can seamlessly operate over a mixture of RDF datasets and text corpora, or individual sources, in a unified framework. Our method, called Uniqorn, builds a context graph on-the-fly, by retrieving question-relevant evidences from the RDF data and/or a text corpus, using fine-tuned BERT models. The resulting graph typically contains all question-relevant evidences but also a lot of noise. Uniqorn copes with this input by a graph algorithm for Group Steiner Trees, that identifies the best answer candidates in the context graph. Experimental results on several benchmarks of complex questions with multiple entities and relations, show that Uniqorn significantly outperforms state-of-the-art methods for heterogeneous QA – in a full training mode, as well as in zero-shot settings. The graph-based methodology provides user-interpretable evidence for the complete answering process.

A Research of Fine-Tuning CNN-LSTM Model for Gold Price Prediction

Gold price and foreign exchange price prediction are usually regarded as serialized data with strong time correlation. It is precisely because of this feature that the performance of traditional convolutional neural networks is often not as good as that of recurrent neural networks. This paper fine-tunes the CNN-LSTM model in two ways. The first is to simplify the input data, convert the time series into binary data, and change the model task from predicting prices to predicting price fluctuations. The second is to add a dropout layer and use the dropout layer to improve the model's overfitting problem. The results demonstrate that the improved model outperforms the first model, with prediction accuracy rising from 54% to 57%. Furthermore, the refined model is not limited to processing the gold price series; it can also be applied to other time series, and when paired with other models, it could yield even better outcomes.

EnzChemRED, a rich enzyme chemistry relation extraction dataset

Expert curation is essential to capture knowledge of enzyme functions from the scientific literature in FAIR open knowledgebases but cannot keep pace with the rate of new discoveries and new publications. In this work we present EnzChemRED, for Enzyme Chemistry Relation Extraction Dataset, a new training and benchmarking dataset to support the development of Natural Language Processing (NLP) methods such as (large) language models that can assist enzyme curation. EnzChemRED consists of 1,210 expert curated PubMed abstracts where enzymes and the chemical reactions they catalyze are annotated using identifiers from the protein knowledgebase UniProtKB and the chemical ontology ChEBI. We show that fine-tuning language models with EnzChemRED significantly boosts their ability to identify proteins and chemicals in text (86.30% F1 score) and to extract the chemical conversions (86.66% F1 score) and the enzymes that catalyze those conversions (83.79% F1 score). We apply our methods to abstracts at PubMed scale to create a draft map of enzyme functions in literature to guide curation efforts in UniProtKB and the reaction knowledgebase Rhea.

Transformer-Based Topic Modeling for Urdu Translations of the Holy Quran

Topic modeling enables the discovery of concealed themes and patterns in extensive text collections. It facilitates a thorough examination of the messages present in religious texts. Topic modeling for Quranic verses is a trending study area, with various translations already explored including Bahasa, English, and Arabic. Yet, there is a need for further research, particularly in Urdu translations of the Quran. In this study, we propose applying the BERTopic framework to Urdu translations of The Holy Quran. By leveraging the BERTopic approach, which incorporates a fine-tuned BERT model, we aim to capture the contextual nuances and linguistic complexities unique to the Quran. In this study, we utilized existing Urdu translations of the Quran from eight different translators sourced from Tanzil, a renowned resource for Quranic text and translations. We assessed the performance of our proposed BERTopic model compared to traditional techniques like LDA and NMF, using coherence and diversity metrics. The results indicate that our BERT-based approach outperforms these conventional methods, achieving an average coherence improvement of 0.03 and a diversity score of 0.83. These findings highlight the effectiveness of BERTopic in extracting meaningful topics from Urdu translations of The Holy Quran and contribute to the computational analysis of religious texts, supporting scholarly endeavours in comparative studies of Quranic translations in Urdu.

Closing the gap between open source and commercial large language models for medical evidence summarization

Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. Using proprietary LLMs introduces multiple risk factors, including a lack of transparency and vendor dependency. While open-source LLMs allow better transparency and customization, their performance falls short compared to the proprietary ones. In this study, we investigated to what extent fine-tuning open-source LLMs can further improve their performance. Utilizing a benchmark dataset, MedReview, consisting of 8161 pairs of systematic reviews and summaries, we fine-tuned three broadly-used, open-sourced LLMs, namely PRIMERA, LongT5, and Llama-2. Overall, the performance of open-source models was all improved after fine-tuning. The performance of fine-tuned LongT5 is close to GPT-3.5 with zero-shot settings. Furthermore, smaller fine-tuned models sometimes even demonstrated superior performance compared to larger zero-shot models. The above trends of improvement were manifested in both a human evaluation and a larger-scale GPT4-simulated evaluation.

CACER: Clinical concept Annotations for Cancer Events and Relations.

Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48000 medical problems and drug events and 10000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction models such as Bidirectional Encoder Representations from Transformers (BERT), Fine-tuned Language Net Text-To-Text Transfer Transformer (Flan-T5), Large Language Model Meta AI (Llama3), and Generative Pre-trained Transformers-4 (GPT-4) using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, large language models offer no performance advantage over the smaller BERT models. We introduce CACER, a novel corpus with fine-grained annotations for medical problems, drugs, and their relationships in clinical narratives of oncology notes. State-of-the-art transformer models achieved performance comparable to IAA for several extraction tasks.

Enhanced diagnostic accuracy for multiple lung diseases using a fine-tuned MobileNetV2 model with advanced pre-processing techniques

Early diagnosis of lung conditions is crucial for effective treatment and improving patient health. However, traditional diagnostic methods using chest X-ray images have some notable drawbacks, such as overlapping anatomical structures obscuring areas of interest, the presence of noise potentially masking abnormalities, and low contrast diminishing differentiation. In this research undertaking, we explored an enhanced MobileNetV2 approach to augmenting the accuracy of diagnosing multiple concomitant lung pathologies. We employed an inclusive methodology, incorporating several progressive steps. We leveraged contrast-limited adaptive histogram equalization to heighten the clarity of the dataset’s images. Bilateral filtering was applied to refine contrast and sharpness, along with utilizing a dense convolutional neural network. Additional techniques were utilized as well, such as image standardization, Gaussian blur, and histogram equalization, to further increase contrast and reduce noise. This was performed through rotations, scaling, horizontal flipping, brightness adjustment, elastic transformations, and random cropping and padding to generate more realistic exemplars. We defined and compiled a purpose-specific MobileNetV2 architecture for this research endeavor. The model was evaluated based on several metrics, including accuracy, precision, recall, F1-score, and specificity, after every epoch, to refine the training process. We used Bayesian optimization for more efficient fine-tuning of the model. Conditional random fields and ensemble averaging methods were employed for postprocessing. These enhancements led to superior results. After image enhancement, the model achieved an accuracy of 99.20 %, with a precision of 98.90 %, a recall of 99.10 %, and an F1-score of 99.00 %. The specificity of the predictions reached 99.40 %. These improvements assist medical workers in making more informed decisions. However, it is important to note that the current algorithm does not influence the effectiveness of treatment or recovery from the condition.

Harmonizing Fine-tuned Llama 2 for Content Generation with Stable Diffusion for Image Synthesis in Article Creation

The research explores the integration of generative AI in multimedia content production using a fine-tuned Llama 2 model for text generation and the Stable Diffusion algorithm for image synthesis. The research analyses the fine-tuned Llama 2-7b-chat model's adaptability to specific content generation contexts, enhanced by a unique dataset and QLoRa, a Quantized Low-Rank Adaptation for parameter-efficient fine-tuning, achieving significant reductions in training loss and nuanced quality in the generated content. Notably, the model's evaluation yielded an impressive perplexity score of 1.49, indicating advanced predictive performance. Additionally, stable diffusion's ability to transform textual descriptions into intricate images, highlighting its potential in AI-mediated content creation is demonstrated. The experiments and qualitative analyses reveal improvements in efficiency and creativity, emphasizing the collaborative potential of these models to revolutionize multidisciplinary content generation. The research underscores the transformative impact of fine-tuned generative models on content creation and offers insights into the broader implications for future AI research, while acknowledging the critical need for ethical considerations in the deployment of such technologies.

Classification of AO/OTA 31A/B femur fractures in X-ray images using YOLOv8 and advanced data augmentation techniques

Femur fractures are a significant worldwide public health concern that affects patients as well as their families because of their high frequency, morbidity, and mortality. When employing computer-aided diagnostic (CAD) technologies, promising results have been shown in the efficiency and accuracy of fracture classification, particularly with the growing use of Deep Learning (DL) approaches. Nevertheless, the complexity is further increased by the need to collect enough input data to train these algorithms and the challenge of interpreting the findings. By improving on the results of the most recent deep learning-based Arbeitsgemeinschaft für Osteosynthesefragen and Orthopaedic Trauma Association (AO/OTA) system classification of femur fractures, this study intends to support physicians in making correct and timely decisions regarding patient care. A state-of-the-art architecture, YOLOv8, was used and refined while paying close attention to the interpretability of the model. Furthermore, data augmentation techniques were involved during preprocessing, increasing the dataset samples through image processing alterations. The fine-tuned YOLOv8 model achieved remarkable results, with 0.9 accuracy, 0.85 precision, 0.85 recall, and 0.85 F1-score, computed by averaging the values among all the individual classes for each metric. This study shows the proposed architecture's effectiveness in enhancing the AO/OTA system's classification of femur fractures, assisting physicians in making prompt and accurate diagnoses.

Impedance-based multivariate analysis for accurate estimation of H2S concentrations using CuCrO2 gas sensor

This study unveils an innovative approach to fabricating H2S gas sensor prototypes for continuous monitoring, leveraging Pd-decorated CuCrO2-based metal oxide semiconductor (MOS) chemiresistors and artificial neural network-assisted impedance-based multivariate analysis. Sensors were exposed to H2S in cross-interfering environments containing humidity, NO2, NH3, CH4, H2, and CO2. Impedance-based parameters (Z, phase difference, Z′ and Z") obtained at various frequencies demonstrated that sensors were reproducible and selective for H2S detection. A neural network-based multilayer perceptron (MLP) regression model was trained with different impedance-based parameters to estimate the H2S concentrations. Continuous operation resulted in larger baseline variation for Z, Z′, and Z" readings; however, the measured phase difference values exhibited less depletion than other parameters. Furthermore, concerns about baseline changes were effectively addressed with a fine-tuned MLP model, which predicted both pure air and H2S atmospheres more correctly under cross-interfering and baseline-depleted conditions by employing phase differences at different frequencies as input. Possible reasons for the accurate prediction can be attributed to the confined behaviour of phase difference and discussed with the help of sensor statistical parameters such as mean variation, standard deviation and principal components.

Extracting lung cancer staging descriptors from pathology reports: A generative language model approach

BackgroundIn oncology, electronic health records contain textual key information for the diagnosis, staging, and treatment planning of patients with cancer. However, text data processing requires a lot of time and effort, which limits the utilization of these data. Recent advances in natural language processing (NLP) technology, including large language models, can be applied to cancer research. Particularly, extracting the information required for the pathological stage from surgical pathology reports can be utilized to update cancer staging according to the latest cancer staging guidelines. ObjectivesThis study has two main objectives. The first objective is to evaluate the performance of extracting information from text-based surgical pathology reports and determining pathological stages based on the extracted information using fine-tuned generative language models (GLMs) for patients with lung cancer. The second objective is to determine the feasibility of utilizing relatively small GLMs for information extraction in a resource-constrained computing environment. MethodsLung cancer surgical pathology reports were collected from the Common Data Model database of Seoul National University Bundang Hospital (SNUBH), a tertiary hospital in Korea. We selected 42 descriptors necessary for tumor-node (TN) classification based on these reports and created a gold standard with validation by two clinical experts. The pathology reports and gold standard were used to generate prompt-response pairs for training and evaluating GLMs which then were used to extract information required for staging from pathology reports. ResultsWe evaluated the information extraction performance of six trained models as well as their performance in TN classification using the extracted information. The Deductive Mistral-7B model, which was pre-trained with the deductive dataset, showed the best performance overall, with an exact match ratio of 92.24% in the information extraction problem and an accuracy of 0.9876 (predicting T and N classification concurrently) in classification. ConclusionThis study demonstrated that training GLMs with deductive datasets can improve information extraction performance, and GLMs with a relatively small number of parameters at approximately seven billion can achieve high performance in this problem. The proposed GLM-based information extraction method is expected to be useful in clinical decision-making support, lung cancer staging and research.

Exploring RoBERTa model for cross-domain suggestion detection in online reviews

Detecting suggestions in online review requires contextual understanding of review text, which is an important real-world application of natural language processing. Given the disparate text domains found in product reviews, a common strategy involves fine-tuning bidirectional encoder representations from transformers (BERT) models using reviews from various domains. However, there hasn't been an empirical examination of how BERT models behave across different domains in tasks related to detecting suggestion sentences from online reviews. In this study, we explore BERT models for suggestion classification that have been fine-tuned using single-domain and cross-domain Amazon review datasets. Our results indicate that while single-domain models achieved slightly better performance within their respective domains compared to cross-domain models, the latter outperformed single-domain models when evaluated on cross-domain data. This was also observed for single-domain data not used for fine-tuning the single-domain model and on average across all tests. Although fine-tuning single-domain models can lead to minor accuracy improvements, employing multi-domain models that perform well across domains can help in cold start problems and reduce annotation costs.

MELEP: A Novel Predictive Measure of Transferability in Multi-label ECG Diagnosis.

In practical electrocardiography (ECG) interpretation, the scarcity of well-annotated data is a common challenge. Transfer learning techniques are valuable in such situations, yet the assessment of transferability has received limited attention. To tackle this issue, we introduce MELEP, which stands for Muti-label Expected Log of Empirical Predictions, a measure designed to estimate the effectiveness of knowledge transfer from a pre-trained model to a downstream multi-label ECG diagnosis task. MELEP is generic, working with new target data with different label sets, and computationally efficient, requiring only a single forward pass through the pre-trained model. To the best of our knowledge, MELEP is the first transferability metric specifically designed for multi-label ECG classification problems. Our experiments show that MELEP can predict the performance of pre-trained convolutional and recurrent deep neural networks, on small and imbalanced ECG data. Specifically, we observed strong correlation coefficients (with absolute values exceeding 0.6 in most cases) between MELEP and the actual average F1 scores of the fine-tuned models. Our work highlights the potential of MELEP to expedite the selection of suitable pre-trained models for ECG diagnosis tasks, saving time and effort that would otherwise be spent on fine-tuning these models.