Tri-training for Dependency Parsing Domain Adaptation

In recent years, the research of dependency parsing focuses on improving the accuracy of in-domain data and has made remarkable progress. However, the real world is different from a single scenario dataset, filled with countless scenarios that are not covered by the dataset, namely, out-of-domain. As a result, parsers that perform well on the in-domain data often suffer significant performance degradation on the out-of-domain data. Therefore, in order to adapt the existing in-domain parsers with substantial performance to the new domain scenario, cross-domain transfer learning techniques are essential to solve the domain problem in parsing. In this paper, we examine two scenarios for cross-domain transfer learning: semi-supervised and unsupervised cross-domain transfer learning. Specifically, we adopt a pretrained language model BERT for training on the source domain (in-domain) data at subword level and introduce two tri-training variant methods for the two scenarios so as to achieve the goal of cross-domain transfer learning. The system based on this paper participated in NLPCC-2019-shared-task on cross-domain dependency parsing and won the first place on the “subtask3-un-open” and “subtask4-semi-open” subtasks, indicating the effectiveness of the approaches adopted.

This study aimed to find suitable source domain data in cross-domain transfer learning to extract robust image features. Then, a model was built to preoperatively distinguish lung granulomatous nodules (LGNs) from lung adenocarcinoma (LAC) in solitary pulmonary solid nodules (SPSNs). Data from 841 patients with SPSNs from five centres were collected retrospectively. First, adaptive cross-domain transfer learning was used to construct transfer learning signatures (TLS) under different source domain data and conduct a comparative analysis. The Wasserstein distance was used to assess the similarity between the source domain and target domain data in cross-domain transfer learning. Second, a cross-domain transfer learning radiomics model (TLRM) combining the best performing TLS, clinical factors and subjective CT findings was constructed. Finally, the performance of the model was validated through multicentre validation cohorts. Relative to other source domain data, TLS based on lung whole slide images as source domain data (TLS-LW) had the best performance in all validation cohorts (AUC range: 0.8228-0.8984). Meanwhile, the Wasserstein distance of TLS-LW was 1.7108, which was minimal. Finally, TLS-LW, age, spiculated sign and lobulated shape were used to build the TLRM. In all validation cohorts, The AUC ranges were 0.9074-0.9442. Compared with other models, decision curve analysis and integrated discrimination improvement showed that TLRM had better performance. The TLRM could assist physicians in preoperatively differentiating LGN from LAC in SPSNs. Furthermore, compared with other images, cross-domain transfer learning can extract robust image features when using lung whole slide images as source domain data and has a better effect.

Deep neural networks (DNNs) have achieved remarkable success in various domains; however, their performance often relies heavily on large-scale, high-quality labeled datasets, which are scarce in low-resource environments. Cross-domain transfer learning has emerged as a promising technique for adapting pre-trained models from data-rich source domains to low-resource target domains to address this limitation. This study explores innovative strategies to enhance the performance and applicability of DNNs through cross-domain transfer learning, focusing on challenges such as domain disparity, data scarcity, and computational constraints. We evaluate several transfer learning approaches, including feature-based transfer, parameter fine-tuning, and adversarial domain adaptation, across diverse healthcare, agriculture, and natural language processing applications. Experimental results demonstrate significant improvements in model accuracy and generalization in low-resource environments, with accuracy gains of up to 20% compared to models trained from scratch. Additionally, we analyze the impact of transfer learning on reducing training time and computational requirements, making it a viable solution for resource-constrained settings. Despite its potential, the study highlights critical challenges, including negative transfer, model interpretability, and ethical considerations in domain transfer. Addressing these issues, we propose a framework for selecting optimal source domains and enhancing model robustness through hybrid techniques and unsupervised learning. This research emphasizes the transformative potential of cross-domain transfer learning in bridging the gap between data-rich and low-resource environments, paving the way for more equitable and efficient applications of deep learning technologies worldwide.

Improving the performance of drifted/shifted electronic nose systems by cross-domain transfer using common transfer samples

Automatic biology image classification is essential for biodiversity conservation and ecological study. Recently, due to the record-shattering performance, deep convolutional neural networks (DCNNs) have been used more often in biology image classification. However, training DCNNs requires a large amount of labeled data, which may be difficult to collect for some organisms. This study was carried out to exploit cross-domain transfer learning for DCNNs with limited data. According to the literature, previous studies mainly focus on transferring from ImageNet to a specific domain or transferring between two closely related domains. While this study explores deep transfer learning between species from different domains and analyzes the situation when there is a huge difference between the source domain and the target domain. Inspired by the analysis of previous studies, the effect of biology cross-domain image classification in transfer learning is proposed. In this work, the multiple transfer learning scheme is designed to exploit deep transfer learning on several biology image datasets from different domains. There may be a huge difference between the source domain and the target domain, causing poor performance on transfer learning. To address this problem, multistage transfer learning is proposed by introducing an intermediate domain. The experimental results show the effectiveness of cross-domain transfer learning and the importance of data amount and validate the potential of multistage transfer learning.

A parallel ensemble optimization and transfer learning based intelligent fault diagnosis framework for bearings

Small data sets are an extremely challenging problem in the machine learning (ML) realm, and in specific, in regression scenarios, as the lack of relevant data can lead to ML models that have large bias. However, there are many applications for which a purely data‐driven procedure would be advantageous, but a large amount of data are not available. This article proposes a novel regression‐based transfer learning (TL) model to address this challenge, where TL is defined as knowledge transfer from a large, relevant data set (source domain data) to a small data set (target domain data). The proposed TL model is termed double‐weighted support vector transfer regression (DW‐SVTR), which couples least squares support vector machines for regression (LS‐SVMR) with two weight functions. The first weight function uses kernel mean matching (KMM) to reweight the source domain data such that the mean values of the source and target domain data in a reproduced kernel Hilbert space (RKHS) are close. In this way, the source domain data points relevant to the target domain points have a larger weight than irrelevant source domain points. The second weight is a function of estimated residuals, which aims to further reduce the negative interference of irrelevant source domain points. The proposed approach is assessed and validated via simulated data and by enhanced shear strength prediction of nonductile columns based on limited availability of nonductile column data. Specifically, the results for the latter show that the proposed DW‐SVTR can reduce the root mean square error (RMSE) by 34% and enhance the coefficient of determination (R2) by 229%. These numerical results demonstrate that the DW‐SVTR significantly reduces the effect of small sample bias and improves prediction performance compared to standard ML methods.

A novel classification method for GPR B-scan images based on weak-shot learning

A Word2Vec-ResNet Transfer Learning model for promoter prediction with dimensionality reduction and cross-domain knowledge integration.

Many existing techniques for recognizing sports posture depend heavily on large, labeled datasets, which limits their performance in scenarios with limited data, such as few-shot learning in specific sports. To address this challenge, we introduce a novel framework that integrates transfer learning, LSTM (Long Short-Term Memory) and GAN (Generative Adversarial Network). The transfer learning component extracts common features from large-scale data in the source domain, while the LSTM module captures temporal dependencies crucial for posture recognition. Simultaneously, the GAN component generates synthetic data to supplement the sparse dataset in the target domain, improving the overall model effectiveness. Experimental results on the Human3.6M and Kinetics-400 datasets show that the proposed TGAN–LSTM model outperforms others in key evaluation metrics, achieving an accuracy of 85.2%, mAP of 79.8% and AUC of 87.3% on Human3.6M, and an accuracy of 80.7% with an F1-score of 78.5% on Kinetics-400. In comparison to other baseline models, TGAN–LSTM demonstrates remarkable performance under few-shot learning conditions. This approach not only offers a solution for few-shot sports posture recognition but also contributes valuable insights to research in cross-domain transfer learning and few-shot learning, with wide practical applications.

Transfer learning for multi-objective non-intrusive load monitoring in smart building

The focus of Source-free Unsupervised Domain Adaptation (SFUDA) is to effectively transfer a well-trained model from the source domain to an unlabelled target domain. During the target domain adaptation, the source domain data is no longer accessible. Prevalent methodologies attempt to synchronize the data distributions between the source and target domains, utilizing pseudo-labels to impart categorical information, which has made some progress in improving the model’s performance. However, performance impairments persist due to the introduction of learning bias from the source model and the impact of noisy pseudo-labels generated for the target domain. In this research, we reveal that the central cause for feature misalignment during domain transition is the learning bias, which is generated by the discrepancy of information between source and target domain data. The source domain data may contain distinguishable features that do not appear on the target domain, which causes the pre-trained source model to fail to work during domain adaptation. To overcome the information discrepancy, we propose a Prototypical Feature Compensation (PFC) Network. The network extracts representative feature maps of the source domain. Then use them to minimize the discrepancy information in the target domain feature maps. This mechanism facilitates feature alignment across different domains, allowing the model to generate more accurate categorical data through pseudo-labelling. The experimental results and ablation studies demonstrate exceptional performance on three SFUDA datasets and provide evidence of the proposed PFC method’s ability to adjust the feature distribution of both source and target domain data, ensuring their overlap in the latent space.

Pest infestations have always been a major factor affecting tea production. Real-time detection of tea pests using machine vision is a mainstream method in modern agricultural pest control. Currently, there is a notable absence of machine vision devices capable of real-time monitoring for small-sized tea pests in the market, and the scarcity of open-source datasets available for tea pest detection remains a critical limitation. This manuscript proposes a YOLOv8-FasterTea pest detection algorithm based on cross-domain transfer learning, which was successfully deployed in a novel tea pest monitoring device. The proposed method leverages transfer learning from the natural language character domain to the tea pest detection domain, termed cross-domain transfer learning, which is based on the complex and small characteristics shared by natural language characters and tea pests. With sufficient samples in the language character domain, transfer learning can effectively enhance the tiny and complex feature extraction capabilities of deep networks in the pest domain and mitigate the few-shot learning problem in tea pest detection. The information and texture features of small tea pests are more likely to be lost with the layers of a neural network becoming deep. Therefore, the proposed method, YOLOv8-FasterTea, removes the P5 layer and adds a P2 small target detection layer based on the YOLOv8 model. Additionally, the original C2f module is replaced with lighter convolutional modules to reduce the loss of information about small target pests. Finally, this manuscript successfully applies the algorithm to outdoor pest monitoring equipment. Experimental results demonstrate that, on a small sample yellow board pest dataset, the mAP@.5 value of the model increased by approximately 6%, on average, after transfer learning. The YOLOv8-FasterTea model improved the mAP@.5 value by 3.7%, while the model size was reduced by 46.6%.

Histopathological image classification based on cross-domain deep transferred feature fusion

<p dir="ltr">Recent advancements in Natural Language Processing (NLP), specifically Large Language Models (LLMs), have demonstrated their transformative power in various domains, including software engineering. This thesis focuses on addressing the critical problem of conflict and duplicate requirement identification in requirement engineering, leveraging the potential of structured and non-structured data in software engineering for automation. For this purpose, first, a diverse set of requirements are annotated for entity recognition tasks, enabling the creation of machine learning and transformer-based models which provides 95.8% accuracy in identifying software-specific entities from requirement texts present in DOORS dataset. Second, building upon this foundation, a two-phase algorithm is developed for conflict detection in requirement documents. The algorithm utilizes supervised learning on requirement data to identify potential conflicts and subsequently validates them semantically using entity extraction techniques. Third, to further refine the conflict and duplicate identification process, the problem is formulated as a sentence pair classification task. Transfer learning techniques, such as sequential transfer learning and cross-domain transfer learning, are employed, and a novel architecture called Software-Requirement Bi-directional Encoder Representation Transformer (SR-BERT) is proposed for requirement pair classification. SR-BERT achieves 95.3% F1-score in determining conflicts, duplicate, and neutral requirement pairs. Additionally, misclassifications flagged by cross-domain transfer learning are reevaluated using Actor-Action (AA) extraction, Part of Speech (POS) tagging, and Semantic Role Labeling (SRL) techniques. The requirement pair classification approach demonstrates great potential in conflict detection tasks, and its performance is enhanced by integrating bi-directional encoders with a contrastive learning framework and cross-encoders. Fourth, inspired by recent advancements in prompt-based learning, a Prompt-based Text-to-Text Models for Software Requirement Pairs (PT2-SRP) framework is also developed, leveraging specially crafted prompts for requirement pair datasets and training encoder-decoder models to identify conflicts in requirement pairs. Lastly, the impact of Data Augmentation (DA) on requirement conflict identification is examined. Specif ically, during the curation of datasets for requirement pair classification task, a scarcity of conflict requirement pairs is noted. To overcome this limitation, novel data augmentation techniques are devised, which ensure the generation of meaningful data instances while preserving the class labels in requirement pair datasets.</p>