Fine-tuning Stage Research Articles

Label-guided contrastive learning with weighted pseudo-labeling: A novel mechanical fault diagnosis method with insufficient annotated data

Exploring fault diagnosis methods for mechanical equipment with weak dependency on annotated data is essential for industrial production. Contrastive learning (CL), capable of learning representations without labeling information, has achieved satisfactory performance in mechanical fault diagnosis. However, current CL-based approaches mainly encounter two limitations. First, the pre-training stage uses either unannotated or annotated samples exclusively while the fine-tuning stage solely relies on annotated ones, leading to inefficient sample utilization. Second, the representation learned by contrastive loss alone in the pretext task is sub-optimal for downstream diagnostic tasks. To address these issues, this paper proposed a novel diagnostic framework based on label-guided contrastive learning (LgCL) and weighted pseudo-labeling (WPL) strategy to improve fault diagnosis accuracy. In the pre-training stage, the proposed LgCL integrates two types of contrastive loss together with classification loss, enabling the encoder to learn discriminative representations that directly benefit the downstream diagnostic task. The devised hybrid fine-tuning strategy allows both labeled and unlabeled data to participate in fine-tuning via pseudo-labeling, enhancing model generalization. The pertinently designed WPL strategy mitigates the defect of noisy pseudo labels. Comparison and ablation experiments on two public datasets and one self-designed dataset validate the superiority of the proposed method for fault diagnosis with limited annotated data, with diagnostic accuracies improved by 25.30%, 5.47% and 10.02% over supervised, semi-supervised and contrastive learning methods, respectively.

Unsupervised Adaptation Learning for Real Multiplatform Hyperspectral Image Denoising.

Real hyperspectral images (HSIs) are ineluctably contaminated by diverse types of noise, which severely limits the image usability. Recently, transfer learning has been introduced in hyperspectral denoising networks to improve model generalizability. However, the current frameworks often rely on image priors and struggle to retain the fidelity of background information. In this article, an unsupervised adaptation learning (UAL)-based hyperspectral denoising network (UALHDN) is proposed to address these issues. The core idea is first learning a general image prior for most HSIs, and then adapting it to a real HSI by learning the deep priors and maintaining background consistency, without introducing hand-crafted priors. Following this notion, a spatial-spectral residual denoiser, a global modeling discriminator, and a hyperspectral discrete representation learning scheme are introduced in the UALHDN framework, and are employed across two learning stages. First, the denoiser and the discriminator are pretrained using synthetic noisy-clean ground-based HSI pairs. Subsequently, the denoiser is further fine-tuned on the real multiplatform HSI according to a spatial-spectral consistency constraint and a background consistency loss in an unsupervised manner. A hyperspectral discrete representation learning scheme is also designed in the fine-tuning stage to extract semantic features and estimate noise-free components, exploring the deep priors specific for real HSIs. The applicability and generalizability of the proposed UALHDN framework were verified through the experiments on real HSIs from various platforms and sensors, including unmanned aerial vehicle-borne, airborne, spaceborne, and Martian datasets. The UAL denoising scheme shows a superior denoising ability when compared with the state-of-the-art hyperspectral denoisers.

Implementing a two-stage, shim field-calibrated superconducting shimming method on a 7 T cryogen-free small animal MRI magnet

Ultrahigh field systems (≥ 7 T) can increase the signal-to-noise ratio of magnetic resonance imaging (MRI), improving imaging performance compared to systems with lower fields. However, these enhancements heavily rely on a high B0 magnetic field homogeneity level, which can be achieved through superconducting shimming. This paper presents a novel two-stage superconducting shimming method designed to achieve precise shimming for a 7 T MRI superconducting magnet. In the initial stage, detailed measurements and fittings were conducted to determine the current polarity and the axial or circumferential positions of the shim fields. Subsequently, an optimization strategy was implemented to determine the optimal shim currents with a flexible target field. The second stage involves an iterative process to fine-tune the current of a specific shim coil, identified as having the most significant impact on field homogeneity. The overall fitness of 99.5% underscores the precision in determining the current polarity and position of the shim fields. Significantly, the calibrated shim system substantially improves the peak-to-peak and Root Mean Square Error (RMSE) field homogeneities from 107.42 ppm and 37.00 ppm to 11.12 ppm and 3.26 ppm, respectively, representing improvements of 89.65% and 91.19%. Furthermore, the simulation results of the fine-tuning stage demonstrate additional enhancements in peak-to-peak field homogeneity, to 9.9 ppm by reducing the current of the Z2 shim coil by 51.3 mA. Additionally, the shimmed magnetic field exhibited high time stability, with a maximum variation of only 27 µT observed within 48 h. Thus, the proposed two-stage superconducting shimming framework effectively addresses the challenge of imperfect B0 magnetic fields, enhancing peak-to-peak and RMSE field homogeneity. The stepwise optimized approach also mitigates deviations caused by shim-to-shim coupling, demonstrating its efficacy in achieving precise shimming in ultrahigh-field MRI systems.

A novel cross-receptive field fusion cascade network with adaptive mask update for transfer health state diagnosis of manipulators

Manipulators, particularly planar parallel manipulators, are widely employed in high-end precision equipment to conduct precise positioning and operation tasks due to their advantages of high stiffness, high precision, and high load. Moreover, they are also frequently exposed to changeable working circumstances, which significantly cause inconsistent health state data distribution. Although transfer learning can successfully offset the above distribution discrepancies, it remains unclear how to identify and quantify the source domain knowledge’s contribution to the transfer process. To overcome these challenges, a novel transfer health state diagnosis framework, named cross-receptive field fusion cascade network with adaptive mask update (CFFCN-AMU), is developed and employed for manipulators. Specifically, a unique cross-receptive field fusion cascade module (CFFCM), in which the receptive field self-evaluator and channel attention mechanism are jointly designed, is constructed initially to achieve adaptive extraction and fusion of cascaded features. Subsequently, in the target domain fine-tuning stage, an adaptive mask update (AMU) strategy is implemented to evaluate the contribution of source domain knowledge and selectively guide the parameter updating process. Finally, some mechanistic model-driven cross-working condition transfer scenarios are investigated. Multiple sets of excellent transfer diagnosis results fully illustrate the transferability and superiority of the constructed CFFCN-AMU model.

Arc bubble edge detection method based on deep transfer learning in underwater wet welding.

The stability of arc bubble is a crucial indicator of underwater wet welding process. However, limited research exists on detecting arc bubble edges in such environments, and traditional algorithms often produce blurry and discontinuous results. To address these challenges, we propose a novel arc bubble edge detection method based on deep transfer learning for processing underwater wet welding images. The proposed method integrates two training stages: pre-training and fine-tuning. In the pre-training stage, a large source domain dataset is used to train VGG16 as a feature extractor. In the fine-tuning stage, we introduce the Attention-Scale-Semantics (ASS) model, which consists of a Convolutional Block Attention Module (CBAM), a Scale Fusion Module (SCM) and a Semantic Fusion Module (SEM). The ASS model is further trained on a small target domain dataset specific to underwater wet welding to fine-tune the model parameters. The CBAM can adaptively weight the feature maps, focusing on more crucial features to better capture edge information. The SCM training method maximizes feature utilization and simplifies training by combining multi-scale features. Additionally, the skip structure of SEM effectively mitigates semantic loss in the high-level network, enhancing the accuracy of edge detection. On the BSDS500 dataset and a self-constructed underwater wet welding dataset, the ASS model was evaluated against conventional edge detection models-Richer Convolutional Features (RCF), Fully Convolutional Network (FCN), and UNet-as well as state-of-the-art models LDC and TEED. In terms of Mean Absolute Error (MAE), accuracy, and other evaluation metrics, the ASS model consistently outperforms these models, demonstrating edge detection capabilities that are both effective and stable in detecting arc bubble edges in underwater wet welding images.

Prototype-based contrastive substructure identification for molecular property prediction.

Substructure-based representation learning has emerged as a powerful approach to featurize complex attributed graphs, with promising results in molecular property prediction (MPP). However, existing MPP methods mainly rely on manually defined rules to extract substructures. It remains an open challenge to adaptively identify meaningful substructures from numerous molecular graphs to accommodate MPP tasks. To this end, this paper proposes Prototype-based cOntrastive Substructure IdentificaTion (POSIT), a self-supervised framework to autonomously discover substructural prototypes across graphs so as to guide end-to-end molecular fragmentation. During pre-training, POSIT emphasizes two key aspects of substructure identification: firstly, it imposes a soft connectivity constraint to encourage the generation of topologically meaningful substructures; secondly, it aligns resultant substructures with derived prototypes through a prototype-substructure contrastive clustering objective, ensuring attribute-based similarity within clusters. In the fine-tuning stage, a cross-scale attention mechanism is designed to integrate substructure-level information to enhance molecular representations. The effectiveness of the POSIT framework is demonstrated by experimental results from diverse real-world datasets, covering both classification and regression tasks. Moreover, visualization analysis validates the consistency of chemical priors with identified substructures. The source code is publicly available at https://github.com/VRPharmer/POSIT.

Multimodal self-supervised learning for remote sensing data land cover classification

Deep learning has revolutionized the remote sensing image processing techniques over the past few years. Nevertheless, annotating high-quality samples is difficult and time-consuming, which limits the performance of deep neural networks because of insufficient supervision information. Aiming to solve this contradiction, we investigate the multimodal self-supervised learning (MultiSSL) paradigm for pre-training and classification of remote sensing image. Specifically, the proposed self-supervised feature learning model consists of asymmetric encoder–decoder structure, in which deep unified encoder learns high-level key information characterizing multimodal remote sensing data and task-specific lightweight decoders are developed to reconstruct original data. To further enhance feature extraction capability, the cross-attention layers are utilized to exchange information contained in heterogeneous characteristics, thus learning more complementary information from multimodal remote sensing data. In fine-tuning stage, the pre-trained encoder and cross-attention layer serve as feature extractor, and leaned characteristics are combined with corresponding spectral information for land cover classification through a lightweight classifier. The self-supervised pre-training model can learn high-level key features from unlabeled samples, thereby utilizing the feature extraction capability of deep neural networks while reducing their dependence on annotated samples. Compared with existing classification paradigms, the proposed multimodal self-supervised pre-training and fine-tuning scheme achieves superior performance for remote sensing image land cover classification.

ECC-PolypDet: Enhanced CenterNet With Contrastive Learning for Automatic Polyp Detection.

Accurate polyp detection is critical for early colorectal cancer diagnosis. Although remarkable progress has been achieved in recent years, the complex colon environment and concealed polyps with unclear boundaries still pose severe challenges in this area. Existing methods either involve computationally expensive context aggregation or lack prior modeling of polyps, resulting in poor performance in challenging cases. In this paper, we propose the Enhanced CenterNet with Contrastive Learning (ECC-PolypDet), a two-stage training & end-to-end inference framework that leverages images and bounding box annotations to train a general model and fine-tune it based on the inference score to obtain a final robust model. Specifically, we conduct Box-assisted Contrastive Learning (BCL) during training to minimize the intra-class difference and maximize the inter-class difference between foreground polyps and backgrounds, enabling our model to capture concealed polyps. Moreover, to enhance the recognition of small polyps, we design the Semantic Flow-guided Feature Pyramid Network (SFFPN) to aggregate multi-scale features and the Heatmap Propagation (HP) module to boost the model's attention on polyp targets. In the fine-tuning stage, we introduce the IoU-guided Sample Re-weighting (ISR) mechanism to prioritize hard samples by adaptively adjusting the loss weight for each sample during fine-tuning. Extensive experiments on six large-scale colonoscopy datasets demonstrate the superiority of our model compared with previous state-of-the-art detectors.

EfficientQ: An efficient and accurate post-training neural network quantization method for medical image segmentation

Model quantization is a promising technique that can simultaneously compress and accelerate a deep neural network by limiting its computation bit-width, which plays a crucial role in the fast-growing AI industry. Despite model quantization’s success in producing well-performing low-bit models, the quantization process itself can still be expensive, which may involve a long fine-tuning stage on a large, well-annotated training set. To make the quantization process more efficient in terms of both time and data requirements, this paper proposes a fast and accurate post-training quantization method, namely EfficientQ. We develop this new method with a layer-wise optimization strategy and leverage the powerful alternating direction method of multipliers (ADMM) algorithm to ensure fast convergence. Furthermore, a weight regularization scheme is incorporated to provide more guidance for the optimization of the discrete weights, and a self-adaptive attention mechanism is proposed to combat the class imbalance problem. Extensive comparison and ablation experiments are conducted on two publicly available medical image segmentation datasets, i.e., LiTS and BraTS2020, and the results demonstrate the superiority of the proposed method over various existing post-training quantization methods in terms of both accuracy and optimization speed. Remarkably, with EfficientQ, the quantization of a practical 3D UNet only requires less than 5 min on a single GPU and one data sample. The source code is available at https://github.com/rongzhao-zhang/EfficientQ.

Mixed event separation and identification based on a convolutional neural network trained with the domain transfer method for a φ-OTDR sensing system

In phase-sensitive optical time domain reflectometer (φ-OTDR) based distributed acoustic sensing (DAS), correct identification of event types is challenging in complex environments where multiple events happen simultaneously. In this study, we have proposed a convolutional neural network (CNN) with a separation module and an identification module to simultaneously separate a mixed event into individual single-event components and identify each type of component contained in the mixed event. The domain transfer method is used in the training, fine-tuning, and testing of the proposed CNN, which saves 94% of the workload for massive DAS data collection and signal demodulation. A fine-tuning stage is added to minimize the impact of the dataset shift between the audio data and DAS data, hence enhancing the separation and identification performance. The model has good noise tolerance and achieves nearly 90% identification accuracy even at a relatively low signal-to-noise ratio (SNR). Compared with the conventional method using DAS data for training, domain transfer using a large amount of diverse audio data for training well generalizes the model to the target domain and hence provides more stable performance with only little degradation of identification accuracy.

Drive-by bridge damage detection by vehicle-bridge interaction neural operator

Drive-by bridge damage detection using vibrations of vehicles for damage detection of bridges has been studied since it needs no instrumentation on bridges. However, it is still a challenge to be able to extract features that are sensitive to bridge damage and relevant to the responses of the bridge. To address the challenges of drive-by bridge damage detection, machine learning-based approaches have also been actively investigated in recent research. With supervised learning methods, it is almost impossible to obtain training data on the damage state of a bridge. On the other hand, unsupervised learning approaches have been considered, but they only provide changes in relative features rather than physical properties that are useful for estimating the structural performance. To address the above issues, this study aims to propose and investigate the feasibility of drive-by bridge damage detection using a vehicle-bridge interaction neural operator (VINO). In the proposed method, numerical simulations of vehicle-bridge interactions considering the damage field of the bridge are first carried out to generate physically informed training data. The VINO is then constructed using the physically informed training data. However, VINOs are still insufficient to model real bridges. Therefore, in order to improve the practicality of the physically-informed VINO, data-informed fine-tuning is carried out using available data from real bridges. It is noted that no damage data is required in the fine-tuning stage. The feasibility of the drive-by bridge damage detection using VINO is discussed using an in-house experiment on a model bridge under a moving vehicle.

Probability graph complementation contrastive learning

Graph Neural Network (GNN) has achieved remarkable progress in the field of graph representation learning. The most prominent characteristic, propagating features along the edges, degrades its performance in most heterophilic graphs. Certain researches make attempts to construct KNN graph to improve the graph homophily. However, there is no prior knowledge to choose proper K and they may suffer from the problem of Inconsistent Similarity Distribution (ISD). To accommodate this issue, we propose Probability Graph Complementation Contrastive Learning (PGCCL) which adaptively constructs the complementation graph. We employ Beta Mixture Model (BMM) to distinguish intra-class similarity and inter-class similarity. Based on the posterior probability, we construct Probability Complementation Graphs to form contrastive views. The contrastive learning prompts the model to preserve complementary information for each node from different views. By combining original graph embedding and complementary graph embedding, the final embedding is able to capture rich semantics in the finetuning stage. At last, comprehensive experimental results on 20 datasets including homophilic and heterophilic graphs firmly verify the effectiveness of our algorithm as well as the quality of probability complementation graph compared with other state-of-the-art methods.

MDTGAN: Multi domain generative adversarial transfer learning network for traffic data imputation

Accurate, real-time, and efficient traffic data, crucial for intelligent transportation systems, which is often disrupted in the real world due to the influence of weather, interference, and equipment failures. Generative Adversarial Networks (GANs), have achieved significant results in image restoration, providing insights for traffic data imputation. Recent research has focused on developing more effective and reasonable model by learning transferable common knowledge from different cities or different scenarios, which is also an excellent idea for improving the data imputation performance. In light of, we contrive to design a GANs based transferable traffic data imputation model, namely Multi Domain Generative Adversarial Transfer Learning Network (MDTGAN). Firstly, the model consists of two stages. In pre-training stage, the model utilizes the source domain datasets (Similar cities road network datasets) to update parameters and transferred them. Then, the model parameters are optimized using the target domain dataset (Target city road network dataset) in fine-tuning stage to avoid the issue of insufficient samples caused by data missing. Secondly, to adapt the model to different dataset topologies, MDTGAN models the node-level data by taking node time series as input, enabling the model to autonomously learn the prevalent spatiotemporal correlations inherent in nodes. Additionally, we introduce a domain discriminator module that guides the spatial encoder in learning domain-invariant features of network node spatial information, enhancing the model generalization ability. The experiments conducted on three publicly available datasets, in which one dataset is regarded as the target dataset, while the others serve as source datasets. The experimental results demonstrate that the MDTGAN model consistently outperforms the baseline models. Specifically, the MDTGAN model can transfer valuable knowledge from the source domain datasets to improve data imputation performance on the target domain dataset, providing practical significance for cities lacking historical traffic data.

Focusing On Fine-Tuning

Those who design and deploy generative AI models, such as Large Language Models like GPT-4 or image diffusion models like Stable Diffusion, can shape model behavior in four distinct stages: pretraining, fine-tuning, in-context learning, and input and output filtering. The four stages differ among many dimensions, including cost, access, and persistence of change. Pretraining is always very expensive and in-context learning is nearly costless. Pretraining and fine-tuning change the model in a more persistent manner, while in-context learning and filters make less durable alterations. These are but two of many such distinctions reviewed in this Essay. Legal scholars, policymakers, and judges need to understand the differences between the four stages as they try to shape and direct what these models do. Although legal and policy interventions can (and probably will) occur during all four stages, many will best be directed at the fine-tuning stage. Fine-tuning will often represent the best balance between power, precision, and disruption of the approaches.

CORE-ReID: Comprehensive Optimization and Refinement through Ensemble Fusion in Domain Adaptation for Person Re-Identification

This study introduces a novel framework, “Comprehensive Optimization and Refinement through Ensemble Fusion in Domain Adaptation for Person Re-identification (CORE-ReID)”, to address an Unsupervised Domain Adaptation (UDA) for Person Re-identification (ReID). The framework utilizes CycleGAN to generate diverse data that harmonize differences in image characteristics from different camera sources in the pre-training stage. In the fine-tuning stage, based on a pair of teacher–student networks, the framework integrates multi-view features for multi-level clustering to derive diverse pseudo-labels. A learnable Ensemble Fusion component that focuses on fine-grained local information within global features is introduced to enhance learning comprehensiveness and avoid ambiguity associated with multiple pseudo-labels. Experimental results on three common UDAs in Person ReID demonstrated significant performance gains over state-of-the-art approaches. Additional enhancements, such as Efficient Channel Attention Block and Bidirectional Mean Feature Normalization mitigate deviation effects and the adaptive fusion of global and local features using the ResNet-based model, further strengthening the framework. The proposed framework ensures clarity in fusion features, avoids ambiguity, and achieves high accuracy in terms of Mean Average Precision, Top-1, Top-5, and Top-10, positioning it as an advanced and effective solution for UDA in Person ReID.

Gradient Matching Federated Domain Adaptation for Brain Image Classification.

Federated learning has shown its unique advantages in many different tasks, including brain image analysis. It provides a new way to train deep learning models while protecting the privacy of medical image data from multiple sites. However, previous studies suggest that domain shift across different sites may influence the performance of federated models. As a solution, we propose a gradient matching federated domain adaptation (GM-FedDA) method for brain image classification, aiming to reduce domain discrepancy with the assistance of a public image dataset and train robust local federated models for target sites. It mainly includes two stages: 1) pretraining stage; we propose a one-common-source adversarial domain adaptation (OCS-ADA) strategy, i.e., adopting ADA with gradient matching loss to pretrain encoders for reducing domain shift at each target site (private data) with the assistance of a common source domain (public data) and 2) fine-tuning stage; we develop a gradient matching federated (GM-Fed) fine-tuning method for updating local federated models pretrained with the OCS-ADA strategy, i.e., pushing the optimization direction of a local federated model toward its specific local minimum by minimizing gradient matching loss between sites. Using fully connected networks as local models, we validate our method with the diagnostic classification tasks of schizophrenia and major depressive disorder based on multisite resting-state functional MRI (fMRI), respectively. Results show that the proposed GM-FedDA method outperforms other commonly used methods, suggesting the potential of our method in brain imaging analysis and other fields, which need to utilize multisite data while preserving data privacy.

A multi-resolution self-supervised learning framework for semantic segmentation in histopathology

Modern whole slide imaging technique together with supervised deep learning approaches have been advancing the field of histopathology, enabling accurate analysis of tissues. These approaches use whole slide images (WSIs) at various resolutions, utilising low-resolution WSIs to identify regions of interest in the tissue and high-resolution for detailed analysis of cellular structures. Due to the labour-intensive process of annotating gigapixels WSIs, accurate analysis of WSIs remains challenging for supervised approaches. Self-supervised learning (SSL) has emerged as an approach to build efficient and robust models using unlabelled data. It has been successfully used to pre-train models to learn meaningful image features which are then fine-tuned with downstream tasks for improved performance compared to training models from scratch. Yet, existing SSL methods optimised for WSI are unable to leverage the multi-resolutions and instead, work only in an individual resolution neglecting the hierarchical structure of multi-resolution inputs. This limitation prevents from the effective utilisation of complementary information between different resolutions, hampering discriminative WSI representation learning. In this paper we propose a Multi-resolution SSL Framework for WSI semantic segmentation (MSF-WSI) that effectively learns histopathological features. Our MSF-WSI learns complementary information from multiple WSI resolutions during the pre-training stage; this contrasts with existing works that only learn between the resolutions at the fine-tuning stage. Our pre-training initialises the model with a comprehensive understanding of multi-resolution features which can lead to improved performance in the subsequent tasks. To achieve this, we introduced a novel Context-Target Fusion Module (CTFM) and a masked jigsaw pretext task to facilitate the learning of multi-resolution features. Additionally, we designed Dense SimSiam Learning (DSL) strategy to maximise the similarities of image features from early model layers to enable discriminative learned representations. We evaluated our method using three public datasets on breast and liver cancer segmentation tasks. Our experiment results demonstrated that our MSF-WSI surpassed the accuracy of other state-of-the-art methods in downstream fine-tuning and semi-supervised settings.

An autoencoder-based self-supervised learning for multimodal sentiment analysis

Representation learning is a crucial and challenging task within multimodal sentiment analysis. Effective multimodal sentiment representations contain two key aspects: consistency and difference. However, the state-of-the-art multimodal sentiment analysis approaches failed to capture the difference and consistency of sentiment information across diverse modalities. To address the multimodal sentiment representation problem, we propose an autoencoder-based self-supervised learning framework. In the pre-training stage, an autoencoder is designed for each modality, leveraging unlabeled data to learn richer sentiment representations for each modality through sample reconstruction and modality consistency detection tasks. In the fine-tuning stage, the pre-trained autoencoder is injected into MulT (AE-MT) and enhance the model's ability to extract deep sentiment information by incorporating a contrastive learning auxiliary task. Our experiments on the popular Chinese sentiment analysis benchmark (CH-SIMS v2.0) and English sentiment analysis benchmark (MOSEI) demonstrate significant gains over baseline models.

RoUIE: A Method for Constructing Knowledge Graph of Power Equipment Based on Improved Universal Information Extraction

The current state evaluation of power equipment often focuses solely on changes in electrical quantities while neglecting basic equipment information as well as textual information such as system alerts, operation records, and defect records. Constructing a device-centric knowledge graph by extracting information from multiple sources related to power equipment is a valuable approach to enhance the intelligence level of asset management. Through the collection of pertinent authentic datasets, we have established a dataset for the state evaluation of power equipment, encompassing 35 types of relationships. To better suit the characteristics of concentrated relationship representations and varying lengths in textual descriptions, we propose a generative model called RoUIE, which is a method for constructing a knowledge graph of power equipment based on improved Universal Information Extraction (UIE). This model first utilizes a pre-trained language model based on rotational position encoding as the text encoder in the fine-tuning stage. Subsequently, we innovatively leverage the Distribution Focal Loss (DFL) to replace Binary Cross-Entropy Loss (BCE) as the loss function, further enhancing the model’s extraction performance. The experimental results demonstrate that compared to the UIE model and mainstream joint extraction benchmark models, RoUIE exhibits superior performance on the dataset we constructed. On a general Chinese dataset, the proposed model also outperforms baseline models, showcasing the model’s universal applicability.

MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Classification Tasks

This paper presents a benchmark study of MIDI-domain music classification using the mask language modeling approach of the Bidirectional Encoder Representations from Transformers (BERT). Specifically, with five public-domain datasets of single-track polyphonic piano MIDI files, we pre-train a 12-layer Transformer model using the BERT approach and fine-tune it for four downstream classification tasks. These include two note-level classification tasks---melody extraction and velocity prediction, and two sequence-level classification tasks---artist classification and emotion classification. Compared to strong recurrent neural network (RNN)-based baselines, the BERT approach does lead to higher classification accuracy, with less than 10 epochs of fine-tuning. Ablation studies further show that the pre-training remains effective even if the MIDI data of the downstream tasks are not seen during the pre-training stage, and that freezing the self-attention layers at the fine-tuning stage only slightly degrades the performance. For reproducibility, we share the weights of our pre-trained and fine-tuned models.