Model Training Method Research Articles

Does pension fund ownership reduce market manipulation? Evidence from China

This study demonstrates that Chinese stock pension funds have similar investment preferences to mutual funds (pensions and mutual funds are managed by the same group of fund managers) but consistently underperform (outperform) their mutual fund counterparts in growth (value) stock investing. We call this contradiction China’s pension fund puzzle. To solve this puzzle, we provide evidence that stocks with low pension fund ownership have undertaken high price manipulation risks and should be compensated by a risk manipulation premium. We designed a logit (probit) Model training method using four different datasets to train the Model. The results show that the pension fund shareholding variable significantly reduced the probability of price manipulation (logit: −6.37, t = −4.13; probit:−1.72, t = −4.63). In the asset pricing framework, we constructed a price manipulation factor and evaluate its pricing performance. The Chinese market had a significant price manipulation premium of 1.9% (t = 2.006).

Towards Robust Neural Rankers with Large Language Model: A Contrastive Training Approach

Pre-trained language model-based neural rankers have been widely applied in information retrieval (IR). However, the robustness issue of current IR models has not received sufficient attention, which could significantly impact the user experience in practical applications. In this study, we focus on the defensive ability of IR models against query attacks while guaranteeing their retrieval performance. We discover that improving the robustness of IR models not only requires a focus on model architecture and training methods but is also closely related to the quality of data. Different from previous research, we use large language models (LLMs) to generate query variations with the same intent, which exhibit richer and more realistic expressions while maintaining consistent query intent. Based on LLM-generated query variations, we propose a novel contrastive training framework that substantially enhances the robustness of IR models to query perturbations. Specifically, we combine the contrastive loss in the representation space of query variations with the ranking loss in the retrieval training stage to improve the model’s ability to understand the underlying semantic information of queries. Experimental results on two public datasets, WikiQA and ANTIQUE, demonstrate that the proposed contrastive training approach effectively improves the robustness of models facing query attack scenarios while outperforming baselines in retrieval performance. Compared with the best baseline approach, the improvements in average robustness performance of Reranker IR models are 24.9%, 26.5%, 27.0%, and 75.0% on WikiQA and 8.7%, 1.9%, 6.3%, and 13.6% on ANTIQUE, in terms of the MAP (Mean Average Precision), MRR (Mean Reciprocal Rank), nDCG@10 (Normalized Discounted Cumulative Gain) and P@10 (Precision), respectively.

Layered mixed-precision training: A new training method for large-scale AI models

How to efficiently and quickly train large-scale AI models has become a hot topic in recent deep learning. Mixed-precision training is an effective technique to speed up training and reduce memory usage. At present, the automatic mixed-precision training method mainly uses half-precision (FP16) for the matrix operations of forward and backward propagation of the entire model and accumulates the FP32 weight copies to avoid rounding errors. However, this method is not optimized for each layer individually, leading to poor convergence in large-scale model training because different layers have different data patterns. Therefore, this paper proposes a layered mixed-precision training method, which can flexibly adjust training precisions according to the contribution of each layer to the training effect. Applying the layered mixed-precision method, the ResNet model achieves a 1.9× speedup compared to the baseline and a lower percentage of accuracy loss. In addition, this paper combines the layered mixed-precision method with distributed training strategies. Combining data parallel training, the model achieves a 3.74× speedup by using four Tesla V100 GPUs. The applicability of the layered mixed-precision method in model parallel training has been verified. Combining optimized pipeline parallel training, the model achieves a 3.26× speedup by using three Tesla V100 GPUs.

An ensemble deep-learning framework for landslide susceptibility assessment using multiple blocks: a case study of Wenchuan area, China

The improvement of the landslide susceptibility mapping (LSM) is a long-standing problem, as it provides basics for hazard mitigation. Recently, hybrid ensemble deep learning (DL) techniques have witnessed the potential for this purpose. In this paper, we proposed a novel ensemble DL model, namely GL-ResNet, which employs the conventional ResNet blocks for landslide feature extraction, long-short term memory (LSTM) structures for information storage, and a proposed GoogLeNet block (GBlk) to broaden model perception ability. To validate the model performance, a landslide inventory containing 1147 historical landslide polygons and the data of 12 landslide factors in the Wenchuan area in southwestern China, was presented and separated into training and validating dataset using a 7:3 randomly sampling ratio strategy. Based on AUC and Accuracy, GL-ResNet (0.96 and 0.909) outperformed logistic regression (0.92 and 0.851), support vector machines (0.94 and 0.884), deep belief networks (0.95 and 0.884), gated recurrent unit (0.94 and 0.884) and ResNet (0.95 and 0.894). We also explored the robustness of GL-ResNet for LSM. The results suggested that although GL-ResNet is sensitive to initial training conditions, it showed good robustness to model training method and sample ratios. In detail, GL-ResNet outperformed the conventional models in terms of fitting power and prediction performance by 0.03-0.04 and 0.02 respectively in most cases, with even greater differences in the limited training dataset.

Mobility-Aware Federated Learning Considering Multiple Networks.

Federated learning (FL) is a distributed training method for machine learning models (ML) that maintain data ownership on users. However, this distributed training approach can lead to variations in efficiency due to user behaviors or characteristics. For instance, mobility can hinder training by causing a client dropout when a device loses connection with other devices on the network. To address this issue, we propose a FL coordination algorithm, MoFeL, to ensure efficient training even in scenarios with mobility. Furthermore, MoFeL evaluates multiple networks with different central servers. To evaluate its effectiveness, we conducted simulation experiments using an image classification application that utilizes machine models trained by a convolutional neural network. The simulation results demonstrate that MoFeL outperforms traditional training coordination algorithms in FL, with 156.5% more training cycles, in scenarios with high mobility compared to an algorithm that does not consider mobility aspects.

Multimodal Prompt Learning in Emotion Recognition Using Context and Audio Information

Prompt learning has improved the performance of language models by reducing the gap in language model training methods of pre-training and downstream tasks. However, extending prompt learning in language models pre-trained with unimodal data to multimodal sources is difficult as it requires additional deep-learning layers that cannot be attached. In the natural-language emotion-recognition task, improved emotional classification can be expected when using audio and text to train a model rather than only natural-language text. Audio information, such as voice pitch, tone, and intonation, can give more information that is unavailable in text to predict emotions more effectively. Thus, using both audio and text can enable better emotion prediction in speech emotion-recognition models compared to semantic information alone. In this paper, in contrast to existing studies that use multimodal data with an additional layer, we propose a method for improving the performance of speech emotion recognition using multimodal prompt learning with text-based pre-trained models. The proposed method is using text and audio information in prompt learning by employing a language model pre-trained on natural-language text. In addition, we propose a method to improve the emotion-recognition performance of the current utterance using the emotion and contextual information of the previous utterances for prompt learning in speech emotion-recognition tasks. The performance of the proposed method was evaluated using the English multimodal dataset MELD and the Korean multimodal dataset KEMDy20. Experiments using both the proposed methods obtained an accuracy of 87.49%, F1 score of 44.16, and weighted F1 score of 86.28.

Federated Learning-Based Insulator Fault Detection for Data Privacy Preserving.

Insulators are widely used in distribution network transmission lines and serve as critical components of the distribution network. The detection of insulator faults is essential to ensure the safe and stable operation of the distribution network. Traditional insulator detection methods often rely on manual identification, which is time-consuming, labor-intensive, and inaccurate. The use of vision sensors for object detection is an efficient and accurate detection method that requires minimal human intervention. Currently, there is a considerable amount of research on the application of vision sensors for insulator fault recognition in object detection. However, centralized object detection requires uploading data collected from various substations through vision sensors to a computing center, which may raise data privacy concerns and increase uncertainty and operational risks in the distribution network. Therefore, this paper proposes a privacy-preserving insulator detection method based on federated learning. An insulator fault detection dataset is constructed, and Convolutional Neural Network (CNN) and Multi-Layer Perceptron (MLP) models are trained within the federated learning framework for insulator fault detection. Most of the existing insulator anomaly detection methods use a centralized model training method, which has the advantage of achieving a target detection accuracy of over 90%, but the disadvantage is that the training process is prone to privacy leakage and lacks privacy protection capability. Compared with the existing insulator target detection methods, the proposed method can also achieve an insulator anomaly detection accuracy of more than 90% and provide effective privacy protection. Through experiments, we demonstrate the applicability of the federated learning framework for insulator fault detection and its ability to protect data privacy while ensuring test accuracy.

Noise-aware Local Model Training Mechanism for Federated Learning

As a new paradigm in training intelligent models, federated learning is widely used to train a global model without requiring local data to be uploaded from end devices. However, there are often mislabeled samples (i.e., noisy samples) in the dataset, which will cause the model update to deviate from the correct direction during the training process, thus reducing the convergence accuracy of the global model. Existing works employ noisy label correction techniques to reduce the impact of noisy samples on model updates by correcting labels; however, such methods necessitate the use of prior knowledge and additional communication costs, which cannot be directly applied to federated learning due to data privacy concerns and limited communication resources. Therefore, this paper proposes a noise-aware local model training method that corrects the noisy labels directly at the end device under the constraints of federated learning. By constructing a label correction model, a joint optimization problem is formally defined for optimizing both the label correction model and the client-side local training model (e.g., classification model). As a solution to this optimization problem, we propose a robustness training algorithm using label correction, along with a cross-validation data sampling algorithm that updates both models simultaneously. It is verified through experiments that the mechanism can effectively improve the model convergence accuracy on noisy datasets in federated learning scenarios.

Low-Cost Object Detection Models for Traffic Control Devices through Domain Adaption of Geographical Regions

Automated inspection systems utilizing computer vision technology are effective in managing traffic control devices (TCDs); however, they face challenges due to the limited availability of training datasets and the difficulty in generating new datasets. To address this, our study establishes a benchmark for cost-effective model training methods that achieve the desired accuracy using data from related domains and YOLOv5, a one-stage object detector known for its high accuracy and speed. In this study, three model cases were developed using distinct training approaches: (1) training with COCO-based pre-trained weights, (2) training with pre-trained weights from the source domain, and (3) training with a synthesized dataset mixed with source and target domains. Upon comparing these model cases, this study found that directly applying source domain data to the target domain is unfeasible, and a small amount of target domain data is necessary for optimal performance. A model trained with fine-tuning-based domain adaptation using pre-trained weights from the source domain and minimal target data, proved to be the most resource-efficient approach. These results contribute valuable guidance for practitioners aiming to develop TCD models with limited data, enabling them to build optimal models while conserving resources.

PairTraining: A method for training Convolutional Neural Networks with image pairs

In the field of image classification, the Convolutional Neural Networks (CNNs) are effective. Most of the work focuses on improving and innovating CNN’s network structure. However, using labeled data more effectively for training has also been an essential part of CNN’s research. Combining image disturbance and consistency regularization theory, this paper proposes a model training method (PairTraining) that takes image pairs as input and dynamically modify the training difficulty according to the accuracy of the model in the training set. According to the accuracy of the model in the training set, the training process will be divided into three stages: the qualitative stage, the fine learning stage and the strengthening learning stage. Contrastive learning images are formed using a progressively enhanced image disturbance strategy at different training stages. The input image and contrast learning image are combined into image pairs for model training. The experiments are tested on four public datasets using eleven CNN models. These models have different degrees of improvement in accuracy on the four datasets. PairTraining can adapt to a variety of CNN models for image classification training. This method can better improve the effectiveness of training and improve the degree of generalization of classification models after training. The classification model obtained by PairTraining has better performance in practical application.

A Non-Intrusive Load Decomposition Method Based on Generative Adversarial Networks and Asynchronous Federated Learning

The non-intrusive load decomposition method is an essential approach of appreciating the water requirements of consumers. Traditional techniques of load breakdown have a number of drawbacks, including a poor level of water identification precision. In order to do this, a model for load decomposition based on generative adversarial networks has been developed. By applying a convolutional autoencoder to the total power noise, the generative network constructs the power breakdown sequence of the given home appliance. The discriminant network is then utilized to determine whether or not the created sequence is legitimate. In order to achieve a more natural and authentic sequence, the participants compete against one another. In light of the shortcomings of the centralized model training method, the network model implementation scheme based on asynchronous federated learning is utilized to replace the traditional convolution in order to realize the model’s lightweight nature. This allows the model to be applied to terminal devices such as smart water meters, and it is carried out in a cloud-edge collaborative manner. In addition, the model is carried out in a manner that is collaborative between the cloud and the edge. Training not only eases the strain of communication transmission but also protects the privacy of users and the integrity of their data.

Masked360: Enabling Robust 360-degree Video Streaming with Ultra Low Bandwidth Consumption.

360-degree video streaming has gained tremendous growth over the past years. However, the delivery of 360-degree videos over the Internet still suffers from the scarcity of network bandwidth and adverse network conditions (e.g., packet loss, delay). In this paper, we propose a practical neural-enhanced 360-degree video streaming framework called Masked360, which can significantly reduce bandwidth consumption and achieve robustness against packet loss. In Masked360, instead of transmitting the complete video frame, the video server only transmits a masked low-resolution version of each video frame to reduce bandwidth significantly. When delivering masked video frames, the video server also sends a lightweight neural network model called MaskedEncoder to clients. Upon receiving masked frames, the client can reconstruct the original 360-degree video frames and start playback. To further improve the quality of video streaming, we also propose a set of optimization techniques, such as complexity-based patch selection, quarter masking strategy, redundant patch transmission and enhanced model training methods. In addition to bandwidth savings, Masked360 is also robust to packet loss during the transmission, because packet losses can be concealed by the reconstruction operation performed by the MaskedEncoder. Finally, we implement the whole Masked360 framework and evaluate its performance using real datasets. The experimental results show that Masked360 can achieve 4K 360-degree video streaming with bandwidth as low as 2.4 Mbps. Besides, video quality of Masked360 is also improved significantly, with an improvement of 5.24-16.61% in terms of PSNR and 4.74-16.15% in terms of SSIM compared to other baselines.

Distributed Fire Detection and Localization Model Using Federated Learning

Fire detection and monitoring systems based on machine vision have been gradually developed in recent years. Traditional centralized deep learning model training methods transfer large amounts of video image data to the cloud, making image data privacy and confidentiality difficult. In order to protect the data privacy in the fire detection system with heterogeneous data and to enhance its efficiency, this paper proposes an improved federated learning algorithm incorporating computer vision: FedVIS, which uses a federated dropout and gradient selection algorithm to reduce communication overhead, and uses a transformer to replace a traditional neural network to improve the robustness of federated learning in the context of heterogeneous data. FedVIS can reduce the communication overhead in addition to reducing the catastrophic forgetting of previous devices, improving convergence, and producing superior global models. In this paper’s experimental results, FedVIS outperforms the common federated learning methods FedSGD, FedAVG, FedAWS, and CMFL, and improves the detection effect by reducing communication costs. As the amount of clients increases, the accuracy of other algorithmic models decreases by 2–5%, and the number of communication rounds required increases significantly; meanwhile, our method maintains a superior detection performance while requiring roughly the same number of communication rounds.

An MRI image automatic diagnosis model for lumbar disc herniation using semi-supervised learning

Lumbar disc herniation is a common disease that causes low back pain. Due to the high cost of medical diagnosis, as well as a shortage and uneven distribution of medical resources, a system that can automatically analyze and diagnose lumbar spine Magnetic Resonance Imaging (MRI) is becoming an urgent need. This study uses deep learning methods to establish a classifier to diagnose lumbar disc herniation. An MRI classification dataset of lumbar disc herniation consisting of public MRI images is presented and is used to train the proposed classifier. Because a common difficulty in applying computer vision technology to medical images is labeling training data, we use a semi-supervised model training method, while multilayer transverse axial MRI images are used as the model input. In this method, we first use unlabelled MRI images for random self-supervised pre-training and the pre-trained model as a feature extractor for MRI images. Then, all marked cross-sections of each intervertebral disc are used to calculate the feature vector through the feature extractor. The information of all feature vectors is integrated, while a multilayer perceptron is used for classification training. After training, the model achieved 87.11% accuracy, 87.50% sensitivity, 86.72% specificity and 0.9487 AUC (Area Under the ROC Curve) index on the test set. To analyze the rationality of the diagnostic results more quickly, we output the severity of degenerative changes in each region using a heatmap.

Data-driven comparison of federated learning and model personalization for electric load forecasting

Residential short-term electric load forecasting is essential in modern decentralized power systems. Load forecasting methods mostly rely on neural networks and require access to private and sensitive electric load data for model training. Conventional neural network training aggregates all data on a centralized server to train one global model. However, the aggregation of user data introduces security and data privacy risks. In contrast, this study investigates the modern neural network training methods of federated learning and model personalization as potential solutions to security and data privacy problems. Within an extensive simulation approach, the investigated methods are compared to the conventional centralized method and a pre-trained baseline predictor to compare their respective performances. This study identifies that the underlying data structure of electric load data has a significant influence on the loss of a model. We therefore conclude that a comparison of loss distributions will in fact be considered a comparison of data structures, rather than a comparison of the model performance. As an alternative method of comparison of loss values, this study develops the ”differential comparison”. The method allows for the isolated comparison of model loss differences by only comparing the losses of two models generated by the same data sample to build a distribution of differences. The differential comparison method was then used to identify model personalization as the best performing model training method for load forecasting among all analyzed methods, with a superior performance in 59.1 % of all cases.

A Domain-Based, Adaptive, Multi-Scale, Inter-Subject Sleep Stage Classification Network

Sleep stage classification is of great importance in sleep analysis, which provides information for the diagnosis and monitoring of sleep-related conditions. To accurately analyze sleep structure under comfortable conditions, many studies have applied deep learning to sleep staging based on single-lead electrocardiograms (ECGs). However, there is still great room for improvement in inter-subject classification. In this paper, we propose an end-to-end, multi-scale, subject-adaptive network that improves the performance of the model according to the model architecture, training method, and loss calculation. In our investigation, a multi-scale residual feature encoder extracted various details to support the feature extraction of single-lead ECGs in different situations. After taking the domain shift caused by individual differences and acquisition conditions into consideration, we introduced a domain-aligning layer to confuse the domain. Moreover, to enhance the performance of the model, the multi-class focal loss was used to reduce the negative impact of class imbalance on the learning of the model, and the loss of sequence prediction was added to the classification task to assist the model in judging sleep stages. The model was evaluated on the public test datasets SHHS2, SHHS1, and MESA, and we obtained mean accuracies (Kappa) of 0.849 (0.837), 0.827 (0.790), and 0.868 (0.840) for awake/light sleep/deep sleep/REM stage classification, which confirms that this is an improved solution compared to the baseline. The model also performed outstandingly in cross-dataset testing. Hence, this article makes valuable contributions toward improving the reliability of sleep staging.

Evaluating active learning for blind imbalanced domains

Deep learning, which has been very successful in recent years, requires a large amount of data. Active learning has been widely studied and used for decades to reduce annotation costs and now attracts lots of attention in deep learning. Many real-world deep learning applications use active learning to select the informative data to be annotated. In this paper, we first investigate laboratory settings for active learning. We show significant gaps between the results from different laboratory settings and describe our practical laboratory setting that reasonably reflects the active learning use cases in real-world applications. Then, we introduce a problem setting of blind imbalanced domains. Any data set includes multiple domains, e.g., individuals in handwritten character recognition with different social attributes. Major domains have many samples, and minor domains have few samples in the training set. However, we must accurately infer both major and minor domains in the test phase. We experimentally compare different methods of active learning for blind imbalanced domains in our practical laboratory setting. We show that a simple active learning method using softmax margin and a model training method using distance-based sampling with center loss, both working in the deep feature space, perform well.

RingMo: A Remote Sensing Foundation Model With Masked Image Modeling

Deep learning approaches have contributed to the rapid development of Remote Sensing (RS) image interpretation. The most widely used training paradigm is to utilize ImageNet pre-trained models to process RS data for specified tasks. However, there are issues such as domain gap between natural and RS scenes, and the poor generalization capacity of RS models. It makes sense to develop a foundation model with general RS feature representation. Since a large amount of unlabeled data is available, the self-supervised method has more development significance than the fully supervised method in remote sensing. However, most of the current self-supervised methods use contrastive learning, whose performance is sensitive to data augmentation, additional information, and selection of positive and negative pairs. In this paper, we leverage the benefits of generative self-supervised learning for RS images, and propose a Remote sensing foundation Model framework called RingMo, which consists of two parts. First, a large-scale dataset is constructed by collecting two million RS images from satellite and aerial platforms, covering multiple scenes and objects around the world. Second, we propose an RS foundation model training method designed for dense and small objects in complicated RS scenes. We show that the foundation model trained on our dataset with RingMo method achieves state-of-the-art on eight datasets across four downstream tasks, demonstrating the effectiveness of the proposed framework. Through in-depth exploration, we believe it is time for RS researchers to embrace generative self-supervised learning and leverage its general representation capabilities to speed up the development of RS applications.

A Model Training Method for DDoS Detection Using CTGAN under 5GC Traffic

A Model Training Method for DDoS Detection Using CTGAN under 5GC Traffic

Reinforcement Learning for Stock Prediction and High-Frequency Trading With T+1 Rules

The high-frequency trading framework for the price trend prediction model and trading strategy has been a popular approach for T+0 trading in the stock market. The prediction model is used to predict price trends, and the trading strategy is used to determine the price and volume of the order. Most trading strategies consist of multiple trading logic associated with certain tuning parameters. These parameters significantly affect the profitability of high-frequency trading frameworks. There are two main disadvantages of this framework: 1) the price trend prediction model can not adapt to the current market data distribution, and 2) the trading strategy can not adapt to the current market conditions automatically. Thus, the framework cannot always maintain positive revenue. To address this problem, we propose a novel dynamic parameter optimization algorithm based on reinforcement learning for stock prediction and trading, and to generate an adaptive trading framework. First, we use a rolling model training method for stock price trend prediction. Second, we regard each set of strategy parameters as action and devise an inverse reinforcement learning algorithm for the reward function to accurately estimate the reward of each action. Because of the T+1 trading rules of the Chinese stock market, we consider the constraint of limited short position in the reward function. Finally, a reward-enhanced upper confidence bound (UCB) selection algorithm is proposed to automatically optimize the parameters of the trading logic in real-time trading. The experimental results show that our method achieves competitive performance in the Chinese stock market.