Cross-domain Tasks Research Articles

Diffraction model-driven neural network trained using hybrid domain loss for real-time and high-quality computer-generated holography.

Learning-based computer-generated holography (CGH) has demonstrated great potential in enabling real-time, high-quality holographic displays. However, most existing learning-based algorithms still struggle to produce high-quality holograms, due to the difficulty of convolutional neural networks (CNNs) in learning cross-domain tasks. Here, we present a diffraction model-driven neural network (Res-Holo) using hybrid domain loss for phase-only hologram (POH) generation. Res-Holo utilizes the weights of the pretrained ResNet34 as the initialization during the encoder stage of the initial phase prediction network to extract more generic features and also to help prevent overfitting. Also, frequency domain loss is added to further constrain the information that the spatial domain loss is insensitive. The peak signal-to-noise ratio (PSNR) of the reconstructed image is improved by 6.05 dB using hybrid domain loss compared to using spatial domain loss alone. Simulation results show that the proposed Res-Holo can generate high-fidelity 2 K resolution POHs with an average PSNR of 32.88 dB at 0.014 seconds/frame on the DIV2K validation set. Both monochrome and full-color optical experiments show that the proposed method can effectively improve the quality of reproduced images and suppress image artifacts.

Tea Leaf Disease Classification using Domain Adaptation Method

Tea trees are extremely vulnerable to diseases and insect pests in the growth process, which seriously affects the yield and quality of tea leaves. This requires the identification and treatment of infected tea leaves in time. With the rapid development of artificial intelligence and computer vision, there is a new trend to identify the images by computer. However, there often exists environmental changes such as illumination intensity, sample angles and background in the leaf images collected in natural scenes, which results in the difference of data distribution. This makes the traditional deep learning method unable to solve the problem of cross-domain classification very well, thus seriously affecting the accuracy of classification. This study mainly focuses on the cross-domain task with unaligned data distribution. A domain-adaptive based method was proposed to realize the cross-domain classification of tea leaf diseases. The experimental results verify the effectiveness of the presented method and provide new thoughts for the cross-domain classification problem in agriculture.

RBNN: Memory-Efficient Reconfigurable Deep Binary Neural Network With IP Protection for Internet of Things

Currently, a high demand for on-device deep neural network (DNN) model deployment is limited by the large model size, computing-intensive floating-point operations (FLOPS), and intellectual property (IP) infringements (i.e., easy access to model duplication for the avoidance of license payments). One appealing solution to addressing the first two concerns is model quantization, which reduces the model size and uses integer operations commonly supported by microcontrollers (MCUs usually do not support FLOPS). To this end, a 1-bit quantized DNN model or deep binary neural network (BNN) significantly improves the memory efficiency, where each parameter in a BNN model has only 1 bit. However, BNN cannot directly provide IP protection (in particular, the functionality of the model is locked unless there is a license payment). In this article, we propose a reconfigurable BNN (RBNN) to further amplify the memory efficiency for resource-constrained Internet of Things (IoT) devices while naturally protecting the model IP. Generally, RBNN can be reconfigured on demand to achieve any one of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M&gt;1$ </tex-math></inline-formula> ) distinct tasks with the same parameter set, thus only a single task determines the memory requirements. In other words, the memory utilization is improved by a factor of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> . Our extensive experiments corroborate that up to seven commonly used tasks ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M=7$ </tex-math></inline-formula> , six of these tasks are image related and the last one is audio) can co-exist (the value of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$M$ </tex-math></inline-formula> can be larger). These tasks with a varying number of classes have no or negligible accuracy drop-off (i.e., within 1%) on three binarized popular DNN architectures, including VGG, ResNet, and ReActNet. The tasks span across different domains, e.g., computer vision and audio domains validated herein, with the prerequisite that the model architecture can serve those cross-domain tasks. To fulfill the IP protection of an RBNN model, the reconfiguration can be controlled by both a user key and a device-unique root key generated by the intrinsic hardware fingerprint (e.g., SRAM memory power-up pattern). By doing so, an RBNN model can only be used per paid user per authorized device, thus benefiting both the user and the model provider. The source code is released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/LearningMaker/RBNN</uri> .

A Dense ResNet Model with RGB Input Mapping for Cross-Domain Mechanical Fault Diagnosis

In actual engineering applications, the mechanical machine is exposed to uncertain conditions such as noise interference and various loads. The commonly used fault diagnosis models suffer degradation in the prediction accuracy in such complex industrial environments where the available label samples are insufficient and the conditions are varied. To combat this challenge, a cross-domain mechanical fault diagnosis method based on the deep-learning networks is proposed. It utilizes small samples, i.e., 10% of the total, and operates on the time-series signal collected from the mechanical equipment. It provides a classification accuracy of more than 97% on the dataset from Case Western Reserve University (CWRU) under variable conditions and 97.56% with the noise interference of 0 dB. The one-dimensional vibration signal is first converted into an image through RGB mapping. Then, the derived RGB image is capable of the time dependent and spatial properties of the time sequence signal and can be directly used as the input of the deep-learning networks. The deep-learning networks model, i.e., the ResNet, is adopted for the fault feature extraction and additional dense connections are added among the residual blocks to supplement the insufficient labeled samples within the networks. Then, an RGB-DResNet is constructed, capable of retaining the robust features for the classification of the mechanical faults in different working conditions. Finally, through retraining the model by use of transfer learning, the derived RGB-TDResNet model gives a fine adaption to the feature distribution with a small amount of target domain information. The performance of the proposed fault diagnosis model was validated on the dataset from CWRU. The results show that it provides a high identification accuracy and strong robustness in variable operating conditions as well as the noise environment. It is a rather promising approach for dealing with the cross-domain tasks of mechanical fault diagnosis.

Reciprocal normalization for domain adaptation

Batch normalization (BN) is widely used in modern deep neural networks, which has been shown to represent the domain-related knowledge, and thus is ineffective for cross-domain tasks like unsupervised domain adaptation (UDA). Existing BN variant methods aggregate source and target domain knowledge in the same channel in normalization module. However, the misalignment between the features of corresponding channels across domains often leads to a sub-optimal transferability. In this paper, we exploit the cross-domain relation and propose a novel normalization method, Reciprocal Normalization (RN). Specifically, RN first presents a Reciprocal Compensation (RC) module to acquire the compensatory for each channel in both domains based on the cross-domain channel-wise correlation. Then RN develops a Reciprocal Aggregation (RA) module to adaptively aggregate the feature with its cross-domain compensatory components. As an alternative to BN, RN is more suitable for UDA problems and can be easily integrated into popular domain adaptation methods. Experiments show that the proposed RN outperforms existing normalization counterparts by a large margin and helps state-of-the-art adaptation approaches achieve better results. The source code is available on https://github.com/Openning07/reciprocal-normalization-for-DA.

Semi-Supervised Adversarial Transfer Networks for Cross-Domain Intelligent Fault Diagnosis of Rolling Bearings

In recent advances, deep learning-based methods have been broadly applied in fault diagnosis, while most existing studies assume that source domain and target domain data follow the same distribution. As differences in operating conditions lead to the deterioration of diagnosis performance, domain adaptation technology has been introduced to bridge the distribution gap. However, most existing approaches generally assume that source domain labels are available under all health conditions during training, which is incompatible with the actual industrial situation. To this end, this paper proposes a semi-supervised adversarial transfer networks for cross-domain intelligent fault diagnosis of rolling bearings. Firstly, the Gramian Angular Field method is introduced to convert time domain vibration signals into images. Secondly, a semi-supervised learning-based label generating module is designed to generate artificial labels for unlabeled images. Finally, the dynamic adversarial transfer network is proposed to extract the domain-invariant features of all signal images and provide reliable diagnosis results. Two case studies were conducted on public rolling bearing datasets to evaluate the diagnostic performance. An experiment under variable operating conditions and an experiment with different numbers of source domain labels were carried out to verify the generalization and robustness of the proposed approach, respectively. Experiment results demonstrate that the proposed method can achieve high diagnosis accuracy when dealing with cross-domain tasks with deficient source domain labels, which may be more feasible in engineering applications than conventional methodologies.

Towards well-generalizing meta-learning via adversarial task augmentation

Meta-learning aims to use the knowledge from previous tasks to facilitate the learning of novel tasks. Many meta-learning models elaborately design various task-shared inductive bias, and learn it from a large number of tasks, so the generalization capability of the learned inductive bias depends on the diversity of the training tasks. A common assumption in meta-learning is that the training tasks and the test tasks come from the same or similar task distributions. However, this is usually not strictly satisfied in practice, so meta-learning models need to cope with various novel in-domain or cross-domain tasks. To this end, we propose to use task augmentation to increase the diversity of training tasks, thereby improving the generalization capability of meta-learning models. Concretely, we consider the worst-case problem around the base task distribution, and derive the adversarial task augmentation method which can generate inductive bias-adaptive ‘challenging’ tasks. Our method can be used as a simple plug-and-play module for various meta-learning models, and improve their generalization capability. We conduct extensive experiments under in-domain and cross-domain few-shot learning and unsupervised few-shot learning settings, and evaluate our method on different types of data (images and text). Experimental results show that our method can effectively improve the generalization capability of various meta-learning models under different settings.

Fourier-inspired neural module for real-time and high-fidelity computer-generated holography.

Learning-based computer-generated holography (CGH) algorithms appear as novel alternatives to generate phase-only holograms. However, most existing learning-based approaches underperform their iterative peers regarding display quality. Here, we recognize that current convolutional neural networks have difficulty learning cross-domain tasks due to the limited receptive field. In order to overcome this limitation, we propose a Fourier-inspired neural module, which can be easily integrated into various CGH frameworks and significantly enhance the quality of reconstructed images. By explicitly leveraging Fourier transforms within the neural network architecture, the mesoscopic information within the phase-only hologram can be more handily extracted. Both simulation and experiment were performed to showcase its capability. By incorporating it into U-Net and HoloNet, the peak signal-to-noise ratio of reconstructed images is measured at 29.16 dB and 33.50 dB during the simulation, which is 4.97 dB and 1.52 dB higher than those by the baseline U-Net and HoloNet, respectively. Similar trends are observed in the experimental results. We also experimentally demonstrated that U-Net and HoloNet with the proposed module can generate a monochromatic 1080p hologram in 0.015 s and 0.020 s, respectively.

Multi-objective dynamic distribution adaptation with instance reweighting for transfer feature learning

In knowledge adaptation, the source and target knowledge are transferred into the same mapping space by simultaneously reducing the difference between the marginal and conditional distributions; however, it is difficult to precisely balance the two distributions at each transformation. To address this problem, a novel multi-objective dynamic distribution adaptation (MODDA) with instance reweighting is proposed to reduce discrepancies between the two distributions. In addition, a customised non-dominated sorting genetic algorithm-II (NSGA2) optimisation method is presented for searching the optimal cumulative weight path, and four genetic operator combinations are compared to determine which one is optimal for MODDA. Moreover, kernel mean matching is proposed for the first time for dynamic compensation based on an individual’s relevance in instance reweighting. The experimental results confirm that MODDA outperforms other state-of-the-art algorithms in terms of the classification accuracy for 16 well-known cross-domain tasks.

Dual Teacher: A Semisupervised Cotraining Framework for Cross-Domain Ship Detection

Cross-domain ship detection tries to identify Synthetic Aperture Radar (SAR) ship by adapting knowledge from labeled optical images, without labor-intensive annotations. In practical applications, a few ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g</i> ., one or three samples) labeled SAR samples are available, which provides an additional supervision for SAR ships. However, the existing cross-domain methods ignore the SAR supervision (a few labeled and unlabeled SAR images), which limits their performances in a practical and under-investigated task: semi-supervised cross-domain ship detection. In this paper, a Dual Teacher framework is proposed to address the mutual interference between the optical supervision and the SAR supervision. First, both optical and SAR supervision are decomposed into two sub-tasks: cross-domain task and semi-supervised task. Then, both cross-domain task and semi-supervised task can be learned interactively in two individual teacher-student models. The teacher-student models generate pseudo-labels on unlabeled SAR images by a teacher network and fine-tune the student network. Finally, the Dual Teacher framework retrains two teacher-student models in co-training strategies. Both cross-domain dataset and semi-supervised dataset are exploited to jointly improve the pseudo-label quality. The effectiveness of the Dual Teacher framework has been fully experimentally demonstrated. The code is available at https://github.com/XiangtaoZheng/DualTeacher.

Active Broad-Transfer Learning Algorithm for Class-Imbalanced Fault Diagnosis

Knowledge transfer with class-imbalanced data is a challenge in predictive maintenance and fault diagnosis. Deep learning algorithms have provided promising results in fault diagnosis. However, their prediction performance is affected by class-imbalanced data in cross-domain tasks. Broad learning algorithms present promising performance in handling class-imbalanced domain-adaptation (CIDA) problems. In the presence of a domain shift, active broad transfer for class-imbalanced learning (ABTCI), an active broad-transfer learning algorithm for CIDA, is proposed. First, the ABTCI algorithm extracts the time–frequency features and feeds them into a recurrent cell to capture spatial–temporal features. Subsequently, it augments the feature space using a sparse autoencoder and an orthogonal mapping projector. By solving the ridge regression problem, the classifier is initialized. Next, the algorithm samples the target data with reliable pseudo-labels and synthesizes new data using random intraclass interpolation among the minor classes containing source and target knowledge. Finally, the classifier is updated using an incremental continuous learning strategy. The performance of the ABTCI algorithm is validated using three datasets, which include 20 class-balanced and 27 class-imbalanced transfer tasks. The performance of the proposed algorithm benchmarked against other deep-learning algorithms is promising.

Unsupervised Domain Adaptation Semantic Segmentation of High-Resolution Remote Sensing Imagery With Invariant Domain-Level Prototype Memory

Semantic segmentation is a key technique involved in automatic interpretation of high-resolution remote sensing (HRS) imagery and has drawn much attention in the remote sensing community. Deep convolutional neural networks (DCNNs) have been successfully applied to the HRS imagery semantic segmentation task due to their hierarchical representation ability. However, the heavy dependency on a large number of training data with dense annotation and the sensitiveness to the variation of data distribution severely restrict the potential application of DCNNs for the semantic segmentation of HRS imagery. This study proposes a novel unsupervised domain adaptation semantic segmentation network (MemoryAdaptNet) for the semantic segmentation of HRS imagery. MemoryAdaptNet constructs an output space adversarial learning scheme to bridge the domain distribution discrepancy between source domain and target domain and to narrow the influence of domain shift. Specifically, we embed an invariant feature memory module to store invariant domain-level context information because the features obtained from adversarial learning only tend to represent the variant feature of current limited inputs. This module is integrated by a category attention-driven invariant domain-level context aggregation module to current pseudo invariant feature for further augmenting the pixel representations. An entropy-based pseudo label filtering strategy is used to update the memory module with high-confident pseudo invariant feature of current target images. Extensive experiments under three cross-domain tasks indicate that our proposed MemoryAdaptNet is remarkably superior to the state-of-the-art methods.

Cross Domain Lifelong Learning Based on Task Similarity.

Humans gradually learn a sequence of cross-domain tasks and seldom experience catastrophic forgetting. In contrast, deep neural networks achieve good performance only in specific tasks within a single domain. To equip the network with lifelong learning capabilities, we propose a Cross-Domain Lifelong Learning (CDLL) framework that fully explores task similarities. Specifically, we employ a Dual Siamese Network (DSN) to learn the essential similarity features of tasks across different domains. To further understand similarity information across domains, we introduce a Domain-Invariant Feature Enhancement Module (DFEM) to better extract domain-invariant features. Moreover, we propose a Spatial Attention Network (SAN) that assigns different weights to various tasks based on the learned similarity features. Ultimately, to maximize the use of model parameters for learning new tasks, we propose a Structural Sparsity Loss (SSL) that can make the SAN as sparse as possible while ensuring accuracy. Experimental results show that our method effectively reduces catastrophic forgetting compared with state-of-the-art methods when continuously learning multiple tasks across different domains. It is worth noting that the proposed method scarcely forgets old knowledge while consistently enhancing the performance of learned tasks, more closely aligning with human learning.

Automated broad transfer learning for cross-domain fault diagnosis

Smart manufacturing system pursues automated modeling algorithms for industrial applications in dynamic environments. The prevalent deep transfer learning (DTL) has achieved promising results in cross-domain fault diagnosis. However, most DTL algorithms are dataset- and domain-specific. They require hyperparameter optimization (HPO) calling for prior knowledge to accomplish a promising prediction performance. This dilemma persists when new data from different domains arrive. An automated broad-transfer learning algorithm (AutoBTL) is proposed to improve predictive modeling for cross-domain tasks. AutoBTL includes three components, a broad classifier, an active estimator, and a hyperparameter optimizer for solving the HPO problem in cross-domain fault diagnosis. At each iteration, AutoBTL initiates a gated broad architecture assigned by the optimizer for target prediction. Then, an active estimator samples the target data with reliable pseudo labels for domain adaptation and performance evaluation. Finally, the optimizer updates a surrogate model and optimizes the hyperparameter space. These steps get repeated until satisfactory model is generated. AutoBTL involves a transductive joint validation strategy, which significantly improves the performance of the existing HPO algorithms in cross-domain tasks. The performance of the AutoBTL algorithm is validated with three benchmark datasets, including 14 cross-domain tasks. The computational results have demonstrated the accuracy and efficiency of the proposed algorithm over the widely used predictive modeling algorithms.

A lightweight and robust model for engineering cross-domain fault diagnosis via feature fusion-based unsupervised adversarial learning

Cross-domain bearing fault diagnosis models have weaknesses such as large size, complex calculation and weak anti-noise ability. Hence, a lightweight and robust model via feature fusion-based unsupervised adversarial learning (LRFFUAL) is proposed, which could be a special benefit for practical engineering applications. A main innovation lies in a customized feature fusion block to achieve a tradeoff between model lightweight and robustness. Accordingly, a channel residual strategy is proposed to apply residual techniques for channels with weak feature information to achieve data augmentation. Concerning cross-domain tasks with huge distribution discrepancy, a new adversarial learning strategy is proposed to improve model convergence rate by inputting marginal features into a discriminator. Experimental results show that the proposed LRFFUAL has advantages of smaller size, less computation, and stronger robustness compared with other existing methods.

A Siamese Vision Transformer for Bearings Fault Diagnosis.

Fault diagnosis methods based on deep learning have progressed greatly in recent years. However, the limited training data and complex work conditions still restrict the application of these intelligent methods. This paper proposes an intelligent bearing fault diagnosis method, i.e., Siamese Vision Transformer, suiting limited training data and complex work conditions. The Siamese Vision Transformer, combining Siamese network and Vision Transformer, is designed to efficiently extract the feature vectors of input samples in high-level space and complete the classification of the fault. In addition, a new loss function combining the Kullback-Liebler divergence both directions is proposed to improve the performance of the proposed model. Furthermore, a new training strategy termed random mask is designed to enhance input data diversity. A comparative test is conducted on the Case Western Reserve University bearing dataset and Paderborn dataset and our method achieves reasonably high accuracy with limited data and satisfactory generation capability for cross-domain tasks.

BAT: Block and token self-attention for speech emotion recognition

Transformers have achieved great success in many artificial intelligence fields, such as computer vision (CV), audio processing and natural language processing (NLP). In speech emotion recognition (SER), transformer-based architectures usually compute attention in a token-by-token (frame-by-frame) manner, but this approach lacks adequate capacity to capture local emotion information and is easily affected by noise. This paper proposes a novel SER architecture, referred to as block and token self-attention (BAT), that splits a mixed spectrogram into blocks and computes self-attention by combining these blocks with tokens, which can alleviate the effect of local noise while capturing authentic sentiment expressions. Furthermore, we present a cross-block attention mechanism to facilitate information interaction among blocks while integrating a frequency compression and channel enhancement (FCCE) module to smooth the attention biases between blocks and tokens. BAT achieves 73.2% weighted accuracy (WA) and 75.2% unweighted accuracy (UA) on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset, surpassing the results of previously developed state-of-the-art approaches with the same dataset partitioning operation. Further experimental results reveal that our proposed method is also well suited for cross-database and cross-domain tasks, achieving 89% WA and 87.4% UA on Emo-DB and producing a top-1 recognition accuracy of 88.32% with only 15.01 Mb of parameters on the CIFAR-10 image dataset under a scenario with no data augmentation or pretraining.

Remote-Sensing Cross-Domain Scene Classification: A Dataset and Benchmark

Domain adaptation for classification has achieved significant progress in natural images but not in remote-sensing images due to huge differences in data-imaging mechanisms between different modalities and inconsistencies in class labels among existing datasets. More importantly, the lack of cross-domain benchmark datasets has become a major obstacle to the development of scene classification in multimodal remote-sensing images. In this paper, we present a cross-domain dataset of multimodal remote-sensing scene classification (MRSSC). The proposed MRSSC dataset contains 26,710 images of 7 typical scene categories with 4 distinct domains that are collected from Tiangong-2, a Chinese manned spacecraft. Based on this dataset, we evaluate several representative domain adaptation algorithms on three cross-domain tasks to build baselines for future research. The results demonstrate that the domain adaptation algorithm can reduce the differences in data distribution between different domains and improve the accuracy of the three tasks to varying degrees. Furthermore, MRSSC also achieved fairly results in three applications: cross-domain data annotation, weakly supervised object detection and data retrieval. This dataset is believed to stimulate innovative research ideas and methods in remote-sensing cross-domain scene classification and remote-sensing intelligent interpretation.

A Hybrid Matching Network for Fault Diagnosis under Different Working Conditions with Limited Data.

Intelligent fault diagnosis methods based on deep learning have achieved much progress in recent years. However, there are two major factors causing serious degradation of the performance of these algorithms in real industrial applications, i.e., limited labeled training data and complex working conditions. To solve these problems, this study proposed a domain generalization-based hybrid matching network utilizing a matching network to diagnose the faults using features encoded by an autoencoder. The main idea was to regularize the feature extractor of the network with an autoencoder in order to reduce the risk of overfitting with limited training samples. In addition, a training strategy using dropout with random changing rates on inputs was implemented to enhance the model's generalization on unseen domains. The proposed method was validated on two different datasets containing artificial and real faults. The results showed that considerable performance was achieved by the proposed method under cross-domain tasks with limited training samples.

Cross‐domain sequence labelling using language modelling and parameter generating

Sequence labelling (SL) tasks are currently widely studied in the field of natural language processing. Most sequence labelling methods are developed on a large amount of labelled training data via supervised learning, which is time-consuming and expensive. As an alternative, domain adaptation is proposed to train a deep-learning model for sequence labelling in a target domain by exploiting existing labelled training data in related source domains. To this end, the authors propose a Bi-LSTM model to extract more-related knowledge from multi-source domains and learn specific context from the target domain. Further, the language modelling training is also applied to cross-domain adaptability facilitating. The proposed model is extensively evaluated with the named entity recognition and part-of-speech tagging tasks. The empirical results demonstrate the effectiveness of the cross-domain adaption. Our model outperforms the state-of-the-art methods used in both cross-domain tasks and crowd annotation tasks.