Multi-stage Transfer Learning Research Articles

Explainable artificial intelligence and multi-stage transfer learning for injection molding quality prediction

Explainable artificial intelligence and multi-stage transfer learning for injection molding quality prediction

Transfer learning for anatomical structure segmentation in otorhinolaryngology microsurgery.

Reducing the annotation burden is an active and meaningful area of artificial intelligence (AI) research. Multiple datasets for the segmentation of two landmarks were constructed based on 41257 labelled images and 6 different microsurgical scenarios. These datasets were trained using the multi-stage transfer learning (TL) methodology. The multi-stage TL enhanced segmentation performance over baseline (mIOU 0.6892 vs. 0.8869). Besides, Convolutional Neural Networks (CNNs) achieved a robust performance (mIOU 0.8917 vs. 0.8603) even when the training dataset size was reduced from 90% (30078 images) to 10% (3342 images). When directly applying the weight from one certain surgical scenario to recognise the same target in images of other scenarios without training, CNNs still obtained an optimal mIOU of 0.6190±0.0789. Model performance can be improved with TL in datasets with reduced size and increased complexity. It is feasible for data-based domain adaptation among different microsurgical fields.

CAM based fine-grained spatial feature supervision for hierarchical yoga pose classification using multi-stage transfer learning

In this paper, we propose a technique for hierarchical yoga pose classification (YPC) in a multi-stage multi-tasking framework. We propose a three-stage transfer learning based end-to-end training methodology. Novelty lies in (a) proposed supervised contrastive combined loss function for stage-1 training, (b) proposed Encoder–Decoder network architecture with attention mechanism for stage-3 training, (c) proposed spatial context aware multi-tasking combined loss function for stage-3 training. Firstly, for stage-1 training, we propose the usage of linear combination of three loss functions: cross-entropy, self-supervised contrastive loss and supervised contrastive loss in a multi-tasking manner. We introduce radial and cosine margin into the formulation of self-supervised and supervised contrastive loss to pull feature embeddings of same class closer together compared to feature embeddings of different classes. Weights learned over stage-1 training are subsequently fine-tuned over cross-entropy multi-tasking loss in stage-2. These stage-2 weights are transfer learned and are further fine-tuned in stage-3 training. For stage-3 training, we propose the usage of spatial context aware multi-tasking combined loss function. This loss function leverages on the fine-grained spatial features obtained from HiResCAM. These are processed in parallel with features obtained from XGrad-CAM (sensitivity and conservation axioms satisfying features) to further supervise the learning of hierarchical yoga pose classifier. We exemplify our methodology on the publicly available Yoga-82 large-scale dataset. We report peak Top-1 YPC accuracy of 95.89% over 6 pose classes (Yoga-6), 93.85% over 20 pose classes (Yoga-20) and 90.0% over 82 pose classes (Yoga-82). Our proposed method achieves 6.1% improvement over Top-1 classification accuracy in Yoga-6 hierarchy, 9.3% improvement in Yoga-20 hierarchy and 10.9% improvement in Yoga-82 hierarchy in comparison with state-of-the-art (SOTA) methodology. We achieve the current best Top-1 classification accuracies in all the three YPC hierarchies.

Targeted Display Advertising using Machine Learning

This paper delves into the intricate challenges of problem formulation and data representation in the context of a large-scale machine learning system for targeted display advertising. Unlike traditional models, this system is not just conceptual but has been operational for years across thousands of advertising campaigns. Since obtaining ideal training data is cost-prohibitive, data is sourced from related domains and tasks and then adapted for the target task. The paper outlines the architecture of this multi-stage transfer learning system, emphasizing the problem formulation aspects. Extensive experiments demonstrate the value of each transfer stage. Real-world results with diverse advertising clients from various industries showcase the system's performance. The paper concludes with valuable insights gained from over half a decade of work on this complex, widely deployed machine learning system.

Accurate Segmentation and Tracking of Chorda Tympani in Endoscopic Middle Ear Surgery with Artificial Intelligence.

Objective: We introduce a novel endoscopic middle ear surgery dataset specifically designed for evaluating deep learning (DL)-based semantic segmentation of chorda tympani. Methods: We curated a dataset comprising 8240 images from 25 patients, divided into a training set (20%, 1648 images), validation set (5%, 412 images), and test set (75%, 6180 images). We employed data enhancement techniques to expand the picture size of the training and validation sets by 5 times (training set: 8240 images, verification set: 2060 images). Subsequently, we employed a multistage transfer learning training method to establish, train, and validate various convolutional neural networks. Results: On the validation set of 2060 labeled images, our proposed network achieved good results, with the U-net exhibiting the highest effectiveness (mIOU = 0.8737, mPA = 0.9263). Furthermore, when applied to the test dataset of 6180 raw images and contrasted with the prediction of otologists, the overall performance of the U-net was excellent (accuracy = 0.911, precision = 0.9823, sensitivity = 0.8777, specificity = 0.9714). Conclusions: Our findings demonstrate that DL can be successfully employed for automatic segmentation of chorda tympani in endoscopic middle ear surgery, yielding high-performance results. This study validates the potential feasibility of future intelligent navigation technologies to assist in endoscopic middle ear surgery.

MSTL-Kace: Prediction of Prokaryotic Lysine Acetylation Sites Based on Multistage Transfer Learning Strategy.

As one of the most important post-translational modifications (PTM), lysine acetylation (Kace) plays an important role in various biological activities. Traditional experimental methods for identifying Kace sites are inefficient and expensive. Instead, several machine learning methods have been developed for Kace site prediction, and hand-crafted features have been used to encode the protein sequences. However, there are still two challenges: the complex biological information may be under-represented by these manmade features and the small sample issue of some species needs to be addressed. We propose a novel model, MSTL-Kace, which was developed based on transfer learning strategy with pretrained bidirectional encoder representations from transformers (BERT) model. In this model, the high-level embeddings were extracted from species-specific BERT models, and a two-stage fine-tuning strategy was used to deal with small sample issue. Specifically, a domain-specific BERT model was pretrained using all of the sequences in our data sets, which was then fine-tuned, or two-stage fine-tuned based on the training data set of each species to obtain the species-specific BERT models. Afterward, the embeddings of residues were extracted from the fine-tuned model and fed to the different downstream learning algorithms. After comparison, the best model for the six prokaryotic species was built by using a random forest. The results for the independent test sets show that our model outperforms the state-of-the-art methods on all six species. The source codes and data for MSTL-Kace are available at https://github.com/leo97king/MSTL-Kace.

A multi-stage transfer learning strategy for diagnosing a class of rare laryngeal movement disorders

Background:It remains hard to directly apply deep learning-based methods to assist diagnosing essential tremor of voice (ETV) and abductor and adductor spasmodic dysphonia (ABSD and ADSD). One of the main challenges is that, as a class of rare laryngeal movement disorders (LMDs), there are limited available databases to be investigated. Another worthy explored research question is which above sub-disorder benefits most from diagnosis based on sustained phonations. The question is from the fact that sustained phonations can help detect pathological voice from healthy voice. Method:A transfer learning strategy is developed for LMD diagnosis with limited data, which consists of three fundamental parts. (1) An extra vocally healthy database from the International Dialects of English Archive (IDEA) is employed to pre-train a convolutional autoencoder. (2) The transferred proportion of the pre-trained encoder is explored. And its impact on LMD diagnosis is also evaluated, yielding a two-stage transfer model. (3) A third stage is designed following the initial two stages to embed information of pathological sustained phonation into the model. This stage verifies the different effects of applying sustained phonation on diagnosing the three sub-disorders, and helps boost the final diagnostic performance. Results:The analysis in this study is based on clinician-labeled LMD data obtained from the Vanderbilt University Medical Center (VUMC). We find that diagnosing ETV shows sensitivity to sustained phonation within the current database. Meanwhile, the results show that the proposed multi-stage transfer learning strategy can produce (1) accuracy of 65.3% on classifying normal and other three sub-disorders all at once, (2) accuracy of 85.3% in differentiating normal, ABSD, and ETV, and (3) accuracy of 77.7% for normal, ADSD and ETV. These findings demonstrate the effectiveness of the proposed approach.

Domain and Histopathology Adaptations–Based Classification for Malignancy Grading System

Accurate proliferation rate quantification can be used to devise an appropriate treatment for breast cancer. Pathologists use breast tissue biopsy glass slides stained with hematoxylin and eosin to obtain grading information. However, this manual evaluation may lead to high costs and be ineffective because diagnosis depends on the facility and the pathologists' insights and experiences. Convolutional neural network acts as a computer-based observer to improve clinicians' capacity in grading breast cancer. Therefore, this study proposes a novel scheme for automatic breast cancer malignancy grading from invasive ductal carcinoma. The proposed classifiers implement multistage transfer learning incorporating domain and histopathologic transformations. Domain adaptation using pretrained models, such as InceptionResNetV2, InceptionV3, NASNet-Large, ResNet50, ResNet101, VGG19, and Xception, was applied to classify the ×40 magnification BreaKHis data set into eight classes. Subsequently, InceptionV3 and Xception, which contain the domain and histopathology pretrained weights, were determined to be the best for this study and used to categorize the Databiox database into grades 1, 2, or 3. To provide a comprehensive report, this study offered a patchless automated grading system for magnification-dependent and magnification-independent classifications. With an overall accuracy (means ±SD) of 90.17%±3.08% to 97.67%±1.09% and an F1 score of 0.9013 to 0.9760 for magnification-dependent classification, the classifiers in this work achieved outstanding performance. The proposed approach could be used for breast cancer grading systems in clinical settings.

Curriculum Reinforcement Learning From Avoiding Collisions to Navigating Among Movable Obstacles in Diverse Environments

Curriculum learning has proven highly effective to speed up training convergence with improved performance in a variety of tasks. Researchers have been studying how a curriculum can be constituted to train reinforcement learning (RL) agents in various application domains. However, discovering curriculum sequencing requires the ranking of sub-tasks or samples in order of difficulty, which is not yet sufficiently studied for robot navigation problems. It is still an open question what navigation strategies can be learned and transferred during multi-stage transfer learning from easy to hard. Furthermore, despite of some attempts of learning real robot manipulation tasks using curriculum, most of existing works are limited to toy or simulated settings rather than realistic scenarios. To address those issues, we first investigated how the model convergence in diverse environments relates to the navigation strategies and difficulty metrics. We found that only some of the environments can be trained from scratch, such as in a relatively open tunnel-like environment that only required wall following. We then carried out a two-stage transfer learning for more difficult environments. We found such approach effective for goal navigation, but failed for more complex tasks where movable obstacles may be on the navigation path. To facilitate more complex policies in the navigation among movable obstacles (NAMO) task, another curriculum with distance and pace functions appropriate to the difficulty of the environment was developed. The proposed scheme was proved effective and the strategies learned were discussed via comprehensive evaluations conducted in simulated and real environments. Supplementary materials can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://arg-nctu.github.io/projects/curl-navi.html</uri>

From benchmark to bedside: transfer learning from social media to patient-provider text messages for suicide risk prediction.

Compared to natural language processing research investigating suicide risk prediction with social media (SM) data, research utilizing data from clinical settings are scarce. However, the utility of models trained on SM data in text from clinical settings remains unclear. In addition, commonly used performance metrics do not directly translate to operational value in a real-world deployment. The objectives of this study were to evaluate the utility of SM-derived training data for suicide risk prediction in a clinical setting and to develop a metric of the clinical utility of automated triage of patient messages for suicide risk. Using clinical data, we developed a Bidirectional Encoder Representations from Transformers-based suicide risk detection model to identify messages indicating potential suicide risk. We used both annotated and unlabeled suicide-related SM posts for multi-stage transfer learning, leveraging customized contemporary learning rate schedules. We also developed a novel metric estimating predictive models' potential to reduce follow-up delays with patients in distress and used it to assess model utility. Multi-stage transfer learning from SM data outperformed baseline approaches by traditional classification performance metrics, improving performance from 0.734 to a best F1 score of 0.797. Using this approach for automated triage could reduce response times by 15 minutes per urgent message. Despite differences in data characteristics and distribution, publicly available SM data benefit clinical suicide risk prediction when used in conjunction with contemporary transfer learning techniques. Estimates of time saved due to automated triage indicate the potential for the practical impact of such models when deployed as part of established suicide prevention interventions. This work demonstrates a pathway for leveraging publicly available SM data toward improving risk assessment, paving the way for better clinical care and improved clinical outcomes.

Protein Secondary Structure Prediction Using Cascaded Feature Learning Model

The protein secondary structure prediction (PSSP) is pivotal for predicting tertiary structure, which is proliferating in demand for drug design and development. Further, it can be used to learn different protein functions. Although there are many computational methods for protein structure prediction, none of them have succeeded 100% in solving the protein structure prediction problem. This paper proposes a deep learning model, namely Cascaded Feature Learning Model (CFLM) that uses a multi-stage transfer learning approach based on the Residual Dense Network (RDN) for predicting protein secondary structure. The model is trained with different protein datasets at various levels of transfer learning and accepts selected protein features such as solvent accessibility, physicochemical properties, PSSM (position-specific scoring matrix), and PSFM (position-specific frequency matrix) as input. The Q3 and Q8 accuracy obtained on CASP 13 and 14 benchmark datasets with the proposed approach are 91.23% and 91.45%, and 76.83% and 78.04%, respectively. The performance of CFLM is also compared with some of the recent PSSP approaches and it is observed that the proposed approach improves the prediction accuracy by 2.8% in terms of Q3 and 1.81% in terms of Q8 in the CASP 14 dataset.

Multi-stage transfer learning with BERTology-based language models for question answering system in vietnamese

Multi-stage transfer learning with BERTology-based language models for question answering system in vietnamese

Cam Based Fine-Grained Spatial Feature Supervision for Hierarchical Yoga Pose Classification Using Multi-Stage Transfer Learning

Cam Based Fine-Grained Spatial Feature Supervision for Hierarchical Yoga Pose Classification Using Multi-Stage Transfer Learning

LSTM Multi-Stage Transfer Learning for Blood Pressure Estimation Using Photoplethysmography

Considerable research has been devoted to developing machine-learning models for continuous Blood Pressure (BP) estimation. A challenging problem that arises in this domain is the selection of optimal features with interpretable models for medical professionals. The aim of this study was to investigate evidence-based physiologically motivating features based on a solid physiological background of BP determinants. A powerful and compact set of features encompassing six physiologically oriented features was extracted in addition to another set of features consisting of six commonly used features for comparison purposes. In this study, we proposed a BP predictive model using Long Short-Term Memory (LSTM) networks with multi-stage transfer learning approach. The proposed model topology consists of three cascaded stages. First, a BP classification stage. Second, a Mean Arterial Pressure (MAP) regression stage to further approximate a quantity proportional to Vascular Resistance (VR) using the extracted Cardiac Output (CO) from the PPG signal. Third, the main BP estimation stage. The final stage (final BP prediction) is able to exploit embedded correlations between BP and the proposed features along with derived outputs carrying hemodynamic characteristics through the sub-sequence stages. We also constructed traditional single-stage Artificial Neural Network (ANN) and LSTM-based models to appraise the performance gain of our proposed model. The models were tested and evaluated on 40 subjects from the MIMIC II database. The LSTM-based multi-stage model attained a MAE ± SD of 2.03 ± 3.12 for SBP and 1.18 ± 1.70 mmHg for DBP. The proposed set of features resulted in drastic error reduction, of up to 86.21%, compared to models trained on the commonly used features. The superior performance of the proposed multi-stage model provides confirmatory evidence that the selected transferable features among the stages coupled with the high-performing multi-stage topology enhance blood pressure estimation accuracy using PPG signals. This indicates the compelling nature and sufficiency of the proposed efficient features set.

Deep neural network incorporating domain and resolution transformations model for histopathological image classification

To accurately diagnose breast cancer, pathologists take biopsy and perform microscopic examination. However, this procedure is inconvenient, time-consuming, and requires high cost. To overcome these challenges, a deep neural network is used to offer a computer-based observation to categorize breast cancer as benign or malignant. In this study, the binary classification of breast cancer from histopathological images is proposed using multistage transfer learning incorporating domain and resolution transformations. We trained our models with a publicly available BreaKHis dataset. We modified and fine-tuned the well-known architectures, VGG19, ResNet (50 and 101), InceptionV3, Xception, Inception-ResNetV2, and NASNet-Large to classify 40 × magnification histopathology images as benign or malignant, then transferred the pre-trained weights for 100 × , 200 × , and 400 × magnification classifications. Translation of the proposed CNN based multistage transfer learning to vision transformer was also performed. Our approach achieved a remarkable performance by demonstrating a test accuracy of 93.97%100% and an F1-score of 0.911.00. The results from our study depict the use of multistage transfer learning improves the breast histopathological images classification performance. Therefore, the proposed approach might be applied in a clinical setting to offer improved breast cancer diagnosis.

BUViTNet: Breast Ultrasound Detection via Vision Transformers.

Convolutional neural networks (CNNs) have enhanced ultrasound image-based early breast cancer detection. Vision transformers (ViTs) have recently surpassed CNNs as the most effective method for natural image analysis. ViTs have proven their capability of incorporating more global information than CNNs at lower layers, and their skip connections are more powerful than those of CNNs, which endows ViTs with superior performance. However, the effectiveness of ViTs in breast ultrasound imaging has not yet been investigated. Here, we present BUViTNet breast ultrasound detection via ViTs, where ViT-based multistage transfer learning is performed using ImageNet and cancer cell image datasets prior to transfer learning for classifying breast ultrasound images. We utilized two publicly available ultrasound breast image datasets, Mendeley and breast ultrasound images (BUSI), to train and evaluate our algorithm. The proposed method achieved the highest area under the receiver operating characteristics curve (AUC) of 1 ± 0, Matthew's correlation coefficient (MCC) of 1 ± 0, and kappa score of 1 ± 0 on the Mendeley dataset. Furthermore, BUViTNet achieved the highest AUC of 0.968 ± 0.02, MCC of 0.961 ± 0.01, and kappa score of 0.959 ± 0.02 on the BUSI dataset. BUViTNet outperformed ViT trained from scratch, ViT-based conventional transfer learning, and CNN-based transfer learning in classifying breast ultrasound images (p &lt; 0.01 in all cases). Our findings indicate that improved transformers are effective in analyzing breast images and can provide an improved diagnosis if used in clinical settings. Future work will consider the use of a wide range of datasets and parameters for optimized performance.

Multi-stage Transfer Learning for Fake News Detection Using AWD-LSTM Network

In the recent decades, the automatic veracity verification of rumors is essential, since online social media platforms allow users to post news item or express opinion towards a circulating piece of information without much restriction. The intention of fake news is to make the readers believe in inaccurate information, where the detection of fake news by using content is a difficult task. So, the auxiliary information: user profile, social engagement of the users, and other user’s comments are useful in the detection of fake news. In this manuscript, a novel multi-stage transfer learning approach is introduced for an effective fake news detection, where it utilizes user’s comments as auxiliary information to detect whether the given tweet is true or false. The stances of the response tweets contain opinions on news/rumors are often used for verifying the veracity of the circulating information. In order to devastate the effects of the specific rumors at the earliest, the multi-stage transfer learning approach automatically predict veracity of rumors jointly with the stances of their response tweets. The proposed multi-stage transfer learning is an inductive transfer learning variation that is used to forecast the stance of responses, then to identify fake news. The proposed model’s effectiveness is evaluated on the two-benchmark datasets: semEval-2017 task 8 and PHEME. The proposed model outperformed the existing approaches by obtaining a classification accuracy of 64.30% and 65.30%, an F-measure of 65.95% and 63.90% on semEval-2017 task 8, and PHEME on event-wise datasets.

In-Field Automatic Identification of Pomegranates Using a Farmer Robot

Ground vehicles equipped with vision-based perception systems can provide a rich source of information for precision agriculture tasks in orchards, including fruit detection and counting, phenotyping, plant growth and health monitoring. This paper presents a semi-supervised deep learning framework for automatic pomegranate detection using a farmer robot equipped with a consumer-grade camera. In contrast to standard deep-learning methods that require time-consuming and labor-intensive image labeling, the proposed system relies on a novel multi-stage transfer learning approach, whereby a pre-trained network is fine-tuned for the target task using images of fruits in controlled conditions, and then it is progressively extended to more complex scenarios towards accurate and efficient segmentation of field images. Results of experimental tests, performed in a commercial pomegranate orchard in southern Italy, are presented using the DeepLabv3+ (Resnet18) architecture, and they are compared with those that were obtained based on conventional manual image annotation. The proposed framework allows for accurate segmentation results, achieving an F1-score of 86.42% and IoU of 97.94%, while relieving the burden of manual labeling.

Detection of Diabetic Retinopathy Using Deep Learning

Abstract: Diabetic retinopathy is one of the most dangerous complications of diabetes, leading to permanent blindness if left untreated. One of the major challenges is early detection, which is very important for the success of treatment. Unfortunately, accurate identification of the stage of diabetic retinopathy is notoriously tricky and requires expert human interpretation of fundus images. Simplifying the detection step is essential and can help millions of people. Convolutional Neural Networks (CNNs) have been successfully used in many neighboring subjects and for the diagnosis of diabetic retinopathy itself. However, the high cost of large annotated datasets as well as inconsistencies between different clinicians hinders the implementation of these methods. In this paper, we propose an automatic method based on deep learning to detect the stage of diabetic retinopathy using a single human fundus image. In addition, we propose a multi-stage transfer learning approach that uses similar datasets with different labels. The presented method can be used as a screening method for the early detection of diabetic retinopathy with a sensitivity and specificity of 0.99 and is ranked 54 out of 2943 competing methods (quadratic weighted kappa score 0.925466) on the APTOS 2019 Blindness Detection Dataset (13,000 images).

Multistage Transfer Learning for Stage Detection of Diabetic Retinopathy

Diabetic retinopathy is one of the most threatening complications of diabetes that leads to permanent blindness if left untreated. Severity of the diabetic retinopathy disease is based on presence of microaneurysms, exudates, neovascularisation and haemorrhages. Convolutional neural networks have been successfully applied in many adjacent subjects, and for diagnosis of diabetic retinopathy itself. In this paper, an automatic deep-learning-based method for stage detection of diabetic retinopathy by single photography of the human fundus is proposed. Additionally, the multistage approach to transfer learning, which makes use of similar datasets with different labelling, is experimented. The proposed architecture gives high accuracy in classification through spatial analysis. Amongst other supervised algorithms involved, proposed solution is to find a better and optimized way to classifying the fundus image with little pre-processing techniques. The proposed architecture deployed with dropout layer techniques yields 78 percent accuracy.