Generalization Performance Of Model Research Articles

The WuC-Adam algorithm based on joint improvement of Warmup and cosine annealing algorithms.

The Adam algorithm is a common choice for optimizing neural network models. However, its application often brings challenges, such as susceptibility to local optima, overfitting and convergence problems caused by unstable learning rate behavior. In this article, we introduce an enhanced Adam optimization algorithm that integrates Warmup and cosine annealing techniques to alleviate these challenges. By integrating preheating technology into traditional Adam algorithms, we systematically improved the learning rate during the initial training phase, effectively avoiding instability issues. In addition, we adopt a dynamic cosine annealing strategy to adaptively adjust the learning rate, improve local optimization problems and enhance the model's generalization ability. To validate the effectiveness of our proposed method, extensive experiments were conducted on various standard datasets and compared with traditional Adam and other optimization methods. Multiple comparative experiments were conducted using multiple optimization algorithms and the improved algorithm proposed in this paper on multiple datasets. On the MNIST, CIFAR10 and CIFAR100 datasets, the improved algorithm proposed in this paper achieved accuracies of 98.87%, 87.67% and 58.88%, respectively, with significant improvements compared to other algorithms. The experimental results clearly indicate that our joint enhancement of the Adam algorithm has resulted in significant improvements in model convergence speed and generalization performance. These promising results emphasize the potential of our enhanced Adam algorithm in a wide range of deep learning tasks.

PaCNN-LSTM: A Localization Scheme Based on Improved Contrastive Learning and Parallel Fusion Neural Network

The deep learning technique plays an important role in Wi-Fi localization systems as it could mine deep features of measurement data. The main challenges are to combat the signal fluctuation resulting in decrease of sample discrimination, and to leverage the broadest information of sample measurements during the training phase, since they are directly related to the location accuracy and robustness. Hence, to address the above issues, this paper proposes an indoor Wi-Fi localization scheme which mainly contains two modules. Firstly, an improved contrastive learning is introduced to handle the sample signal measurements to increase the discrimination. It is from the perspective of learning and encoding, and avoids the drawbacks brought by traditional processing methods. Then, we build a parallel fusion network named as PaCNN-LSTM based on convolutional neural network (CNN) and long short-term memory network (LSTM). Compared with existing networks, PaCNN-LSTM connects neural networks in parallel rather than serial, which improves the generalization performance of model when extracting the spatial and temporal features of signal measurements. In addition, it also considers the large amount of middle layer information that is always ignored. By adding a Flatten layer after the pooling layer, the available information of samples has been broadened. Extensive experimental results show that the localization performance of the proposed scheme is outperformed than others, where the location accuracy is improved by about 22%.

Language-Aware Domain Generalization Network for Cross-Scene Hyperspectral Image Classification

Text information including extensive prior knowledge about land cover classes has been ignored in hyperspectral image classification (HSI) tasks. It is necessary to explore the effectiveness of linguistic mode in assisting HSI classification. In addition, the large-scale pre-training image-text foundation models have demonstrated great performance in a variety of downstream applications, including zero-shot transfer. However, most domain generalization methods have never addressed mining linguistic modal knowledge to improve the generalization performance of model. To compensate for the inadequacies listed above, a Language-aware Domain Generalization Network (LDGnet) is proposed to learn cross-domain invariant representation from cross-domain shared prior knowledge. The proposed method only trains on the source domain (SD) and then transfers the model to the target domain (TD). The dual-stream architecture including image encoder and text encoder is used to extract visual and linguistic features, in which coarse-grained and fine-grained text representations are designed to extract two levels of linguistic features. Furthermore, linguistic features are used as cross-domain shared semantic space, and visual-linguistic alignment is completed by supervised contrastive learning in semantic space. Extensive experiments on three datasets demonstrate the superiority of the proposed method when compared with state-of-the-art techniques.

UNet segmentation network of COVID-19 CT images with multi-scale attention.

In recent years, the global outbreak of COVID-19 has posed an extremely serious life-safety risk to humans, and in order to maximize the diagnostic efficiency of physicians, it is extremely valuable to investigate the methods of lesion segmentation in images of COVID-19. Aiming at the problems of existing deep learning models, such as low segmentation accuracy, poor model generalization performance, large model parameters and difficult deployment, we propose an UNet segmentation network integrating multi-scale attention for COVID-19 CT images. Specifically, the UNet network model is utilized as the base network, and the structure of multi-scale convolutional attention is proposed in the encoder stage to enhance the network's ability to capture multi-scale information. Second, a local channel attention module is proposed to extract spatial information by modeling local relationships to generate channel domain weights, to supplement detailed information about the target region to reduce information redundancy and to enhance important information. Moreover, the network model encoder segment uses the Meta-ACON activation function to avoid the overfitting phenomenon of the model and to improve the model's representational ability. A large number of experimental results on publicly available mixed data sets show that compared with the current mainstream image segmentation algorithms, the pro-posed method can more effectively improve the accuracy and generalization performance of COVID-19 lesions segmentation and provide help for medical diagnosis and analysis.

Distracted driving behaviour recognition based on transfer learning and model fusion

In the recognition of distracted driving behaviour, traditional manual feature extraction is subjective and complex; single deep convolutional network also has problems such as insufficient generalisation performance and stability. To solve the above problems, this paper proposes a distracted driving behaviour recognition method based on transfer learning and model fusion. First, based on the transfer learning method, the deep convolutional neural network models ResNet18 and ResNet34 are used to extract the features of some images, respectively. Furthermore, the pre-trained model is fine-tuned to obtain four deep convolutional neural network models. Finally, the four network models are fused by stacking method, using 5-fold cross-validation method to reduce over-fitting. Experimental results show that the recognition accuracy of distracted driving behaviour after model fusion reaches 95.47%. The fusion model has higher model generalisation performance and recognition accuracy, which can provide certain technical support for the research of distracted driving behaviour recognition.

A point cloud semantic segmentation method for nuclear power reactors based on RandLA-Net Model

The detection of foreign objects in nuclear power plant reactor is a key task in the operation and maintenance of nuclear power plants. Loose and falling foreign objects such as bolts can lead to fuel component damage and unplanned shutdown, posing serious hazards. Therefore, we propose a point cloud semantic segmentation method for foreign objects in nuclear power plant reactor based on the RandLA-Net model. Considering the correlation between point cloud collection error and curvature, the data augmentation method is improved to reduce the risk of model overfitting. By treating the boundary points of different classes as the hard examples, the hard example mining is designed to improve model generalization performance. Adding an improved test time augmentation method during model inference, the more reliable segmentation results are performed by multiple prediction on points. The experimental results indicate that the proposed method can achieve high-accuracy reactor point cloud semantic segmentation with mIoU of 0.992 and mAcc of 0.997.

Towards interpreting vulnerability of object detection models via adversarial distillation

Recent works have shown that deep learning models are highly vulnerable to adversarial examples, limiting the application of deep learning in security-critical systems. This paper aims to interpret the vulnerability of deep learning models to adversarial examples. We propose adversarial distillation to illustrate that adversarial examples are generalizable data features. Deep learning models are vulnerable to adversarial examples because models do not learn this data distribution. More specifically, we obtain adversarial features by introducing a generation and extraction mechanism. The generation mechanism generates adversarial examples, which mislead the source model trained on the original clean samples. The extraction term removes the original features and selects valid and generalizable adversarial features. Valuable adversarial features guide the model to learn the data distribution of adversarial examples and realize the model’s generalization on the adversarial dataset. Extensive experimental evaluations have proved the excellent generalization performance of the adversarial distillation model. Compared with the normally trained model, the mAP has increased by 2.17% on their respective test sets, while the mAP on the opponent’s test set is very low. The experimental results further prove that adversarial examples are also generalizable data features, which obey a different data distribution from the clean data.

Entity Factor: A Balanced Method for Table Filling in Joint Entity and Relation Extraction

The knowledge graph is an effective tool for improving natural language processing, but manually annotating enormous amounts of knowledge is expensive. Academics have conducted research on entity and relation extraction techniques, among which, the end-to-end table-filling approach is a popular direction for achieving joint entity and relation extraction. However, once the table has been populated in a uniform label space, a large number of null labels are generated within the array, causing label-imbalance problems, which could result in a tendency of the model’s encoder to predict null labels; that is, model generalization performance decreases. In this paper, we propose a method to mitigate non-essential null labels in matrices. This method utilizes a score matrix to calculate the count of non-entities and the percentage of non-essential null labels in the matrix, which is then projected by the power of natural constant to generate an entity-factor matrix. This is then incorporated into the scoring matrix. In the back-propagation process, the gradient of non-essential null-labeled cells in the entity factor layer is affected and shrinks, the amplitude of which is related to the size of the entity factor, thereby reducing the feature learning of the model for a large number of non-essential null labels. Experiments with two publicly available benchmark datasets show that the incorporation of entity factors significantly improved model performance, especially in the relation extraction task, by 1.5% in both cases.

Small samples data augmentation and improved MobileNet for surface defects classification of hot-rolled steel strips

Surface defects in hot-rolled steel strips are one of the common product problems for the steel industry, which harm the product appearance, affect the corrosion and wear resistance, and shorten the product service life. The natural defect samples are sparse, category imbalanced, and expensive manual annotations. Therefore, it is crucial to study the data augmentation and classification methods for small sample surface defects. To solve the above problems and improve the accuracy and real-time performance of defect classification, we propose a random offline data augmentation algorithm (Random-CutMix) and an improved MobileNet architecture (SP-MobileNet). The Random-CutMix algorithm expands the dataset by random sampling to balance the number of each defect class. The SP-MobileNet combines the inverse residual module with the channel shuffle mechanism (CSIn-Module) and pyramid split attention (PSA) module, which facilitates cross-group information flow and improves model representation capability and generalization performance with low computational cost. The accuracy, recall, F1 score, parameter, computational complexity, and frame rate of SP-MobileNet with Random-CutMix on the X-SDD dataset were 95.97%, 95.22%, 95.46%, 6.5 M, and 0.54 G, 72 FPS, respectively. The experiment results indicate that our method outperforms the state-of-the-art methods and provides an effective trade-off between accuracy and instantaneity in actual industrial production.

Wearable Inertial Sensor-Based Hand-Guiding Gestures Recognition Method Robust to Significant Changes in the Body-Alignment of Subject

The accuracy of the wearable inertia-measurement-unit (IMU)-sensor-based gesture recognition may be significantly affected by undesired changes in the body-fixed frame and the sensor-fixed frame according to the change in the subject and the sensor attachment. In this study, we proposed a novel wearable IMU-sensor-based hand-guiding gesture recognition method robust to significant changes in the subject’s body alignment based on the floating body-fixed frame method and the bi-directional long short-term memory (bi-LSTM). Through comparative experimental studies with the other two methods, it was confirmed that aligning the sensor-fixed frame with the reference frame of the human body and updating the reference frame according to the change in the subject’s body-heading direction helped improve the generalization performance of the gesture recognition model. As a result, the proposed floating body-fixed frame method showed a 91.7% test accuracy, confirming that it was appropriate for gesture recognition under significant changes in the subject’s body alignment during gestures.

Influence of sample attributes on generalization performance of machine learning models for windage alteration fault diagnosis of the mine ventilation system

Rapid diagnosis and mitigation of windage alteration faults (WAFs) in the mine ventilation system is of great significance to create a good working environment and ensure safe production. The complexity of the ventilation system state evolution in time and space, and the high dimensionality of the feature set characterizing the system state leads to an exceptionally complex fault diagnosis. Therefore, this paper proposes to build a WAFs diagnosis model based on support vector machine (SVM). Meanwhile, the generalization performance of the model is significantly affected by the sample attributes. Therefore, the generalization errors of the model under the combination of four factors, i.e., sample dispersion, sample numbers, input features, and feature numbers, are further analyzed. The results show that the prediction effect of the model leads to a law that with the increase of the degree of sample dispersion, the number of samples, and the number of input features, it improves at the beginning and becomes stable in the end. Furthermore, the best feature for fault diagnosis is found as the combination of wind volume and wind pressure features. Finally, the correctness of the conclusion is verified by comparative experiments. This research provides useful guidance and valuable reference for how to build machine learning (ML) models for mine ventilation system fault diagnosis.

Cervical Cell Image Classification-Based Knowledge Distillation.

Current deep-learning-based cervical cell classification methods suffer from parameter redundancy and poor model generalization performance, which creates challenges for the intelligent classification of cervical cytology smear images. In this paper, we establish a method for such classification that combines transfer learning and knowledge distillation. This new method not only transfers common features between different source domain data, but also realizes model-to-model knowledge transfer using the unnormalized probability output between models as knowledge. A multi-exit classification network is then introduced as the student network, where a global context module is embedded in each exit branch. A self-distillation method is then proposed to fuse contextual information; deep classifiers in the student network guide shallow classifiers to learn, and multiple classifier outputs are fused using an average integration strategy to form a classifier with strong generalization performance. The experimental results show that the developed method achieves good results using the SIPaKMeD dataset. The accuracy, sensitivity, specificity, and F-measure of the five classifications are 98.52%, 98.53%, 98.68%, 98.59%, respectively. The effectiveness of the method is further verified on a natural image dataset.

Anomaly Matters: An Anomaly-Oriented Model for Medical Visual Question Answering.

Medical images contain various abnormal regions, most of which are closely related to the lesions or diseases. The abnormality or lesion is one of the major concerns during clinical practice and therefore becomes the key in answering questions about medical images. However, the recent efforts still focus on constructing a generic Visual Question Answering framework for medical-domain tasks, which is not adequate for practical medical requirements and applications. In this paper, we present two novel medical-specific modules named multiplication anomaly sensitive module and residual anomaly sensitive module to utilize weakly supervised anomaly localization information in medical Visual Question Answering. Firstly, the proposed multiplication anomaly sensitive module designed for anomaly-related questions can mask the feature of the whole image according to the anomaly location map. Secondly, the residual anomaly sensitive module could learn a flexible anomaly feature while preserving the information of the original questioned image, which is more helpful in answering anomaly-unrelated questions. Thirdly, the transformer decoder and multi-task learning strategy are combined to further enhance the question-reasoning ability and the model generalization performance. Finally, qualitative and quantitative experiments on a variety of medical datasets exhibit the superiority of the proposed approaches compared to the state-of-the-art methods.

Generalizable multi-task, multi-domain deep segmentation of sparse pediatric imaging datasets via multi-scale contrastive regularization and multi-joint anatomical priors.

Clinical diagnosis of the pediatric musculoskeletal system relies on the analysis of medical imaging examinations. In the medical image processing pipeline, semantic segmentation using deep learning algorithms enables an automatic generation of patient-specific three-dimensional anatomical models which are crucial for morphological evaluation. However, the scarcity of pediatric imaging resources may result in reduced accuracy and generalization performance of individual deep segmentation models. In this study, we propose to design a novel multi-task, multi-domain learning framework in which a single segmentation network is optimized over the union of multiple datasets arising from distinct parts of the anatomy. Unlike previous approaches, we simultaneously consider multiple intensity domains and segmentation tasks to overcome the inherent scarcity of pediatric data while leveraging shared features between imaging datasets. To further improve generalization capabilities, we employ a transfer learning scheme from natural image classification, along with a multi-scale contrastive regularization aimed at promoting domain-specific clusters in the shared representations, and multi-joint anatomical priors to enforce anatomically consistent predictions. We evaluate our contributions for performing bone segmentation using three scarce and pediatric imaging datasets of the ankle, knee, and shoulder joints. Our results demonstrate that the proposed approach outperforms individual, transfer, and shared segmentation schemes in Dice metric with statistically sufficient margins. The proposed model brings new perspectives towards intelligent use of imaging resources and better management of pediatric musculoskeletal disorders.

Explicit Metric-Based Multiconcept Multi-Instance Learning With Triplet and Superbag.

Multi-instance learning (MIL) has garnered considerable attention in recent years due to its favorable performance in various scenarios. Nonetheless, most previous studies have implicitly expressed the correlation between instances and bags. Moreover, the importance of negative instances has been largely overlooked. Hence, we seek to present an explicit and intuitively understandable method that can compensate for these deficiencies. In this article, we creatively introduce a metric-based multiconcept MIL approach based on two aspects. First, the triplet-based bag embedding method identifies instance categories and builds attention weights for every instance explicitly. Accordingly, bag embedding is accomplished under the limitation of weak supervision. Second, the developed instance correlation metric approach in the superbag considers the multiconcept issue to boost the model generalization performance. We have designed a rich variety of experiments to demonstrate the performance of our algorithm. The artificial data experiment reveals the interpretability of the proposed network. The results of the comparison experiment confirm that our method shows favorable performance in multiple tasks. Finally, we illustrate the motivation of the presented method by the ablation experiments.

Adaptive alarm prediction in optical network based on model generalization in cross-layer AI

Artificial intelligence (AI) applications about optical network management are increasing, especially in fault diagnosis. When the network fault or alarm occurs, the equipment running status and performance parameters are changing. AI algorithms can extract the internal relationship from the data and it is proved that the alarm prediction method based on AI is effective. However, the performance of AI models, especially supervised learning, rely on large data sets. These training sets with only a small number of samples are usually unavailable due to the lack of historical data. How to improve the generalization ability of AI models is still a key problem to be solved in the intelligent maintenance of optical networks. In optical network alarm prediction, performance data of devices have different degrees of relationship to alarms and the generalization performance of models is limited by the data structure. In this paper, we propose a method to optimize the model generalization ability in alarm prediction, i.e., the training of a small sample model is solved through parameter transfer in the training process. We verify the generalization effect of the models by using real data of a commercial optical network. Experimental results show that the models achieve significant performance improvement in the data set with a high correlation between performance and alarm.

Federated learning‐based trajectory prediction model with privacy preserving for intelligent vehicle

The existing trajectory prediction is mainly for specific road sections, which is poor to adapt to complex and changing traffic scenarios. Meanwhile, decentralized trajectory data is hard to be fully utilized in the data silo environment. To solve data silos in the intelligent vehicle industry, introducing federated learning methods in vehicular edge computing has attracted extensive attention. But traditional federated learning still has the potential to suffer from the mining of training data in the case of model privacy leakage. In this paper, a vehicle trajectory prediction method based on federated learning and homomorphic encryption has been presented, which adopts a three-layer architecture with a vehicle cluster, edge computing server, and cloud core network. Compared with traditional centralized deep learning methods, this approach enables joint modeling of multiple parties to improve the model's generalization performance. We used a proxy re-encryption algorithm to implement key distribution and also designed an encrypted federated network algorithm FAHEFL, which uses FAHE1 homomorphic encryption to protect the privacy of the model in parameter transmission. Each local model includes a driving behavior recognition module and trajectory output module. The driving behavior recognition module uses a 1D convolutional neural network to recognize the driving behavior, then input recognition results and historical trajectories to the trajectory output module, which uses LSTM neural network to output predicted trajectories. The experiment results show that the model built with FAHEFL has no more 1% error in the driving behavior recognition module than concentrated learning, while the minimum mean square error of the trajectory prediction module increased by only 2.5%. This article also discusses the performance between FAHEFL and well-known cryptographic federation network algorithms.

Active and Semi-Supervised Graph Neural Networks for Graph Classification

Graph classification aims to predict the class labels of graphs and has a wide range of applications in many real-world domains. However, most of existing graph neural networks for graph classification tasks use 90 <inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> of labeled graphs for training and the remaining 10 <inline-formula><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> for testing, which obviously struggle in solving the problem of the scarcity of labeled graphs in real-world graph classification scenarios. And it is arduous to label a large number of graph examples for training because of the difficulty and resource consumption in the tagging process. Motivated by this, we propose a novel active and semi-supervised graph neural network (ASGNN) framework, which endeavors to complete graph classification tasks with a small number of labeled graph examples and available unlabeled graph examples. In our framework, active learning selects high-uncertain and representative graph examples from the test set and add them to the training set after annotation. Semi-supervised learning is utilized to select the high-confidence unlabeled graph examples containing structural information from the test set, and add them to the training set after pseudo labeling. To improve the generalization performance of the graph classification model, multiple GNNs are trained collaboratively for promoting the expressiveness of each other and increasing the reliability of graph classification results. Overall, the ASGNN framework takes fully use of unlabeled graph examples to reinforce graph classification effectively, and can be applied to any existing supervised graph neural networks for graph classification. Experimental results on benchmark graph datasets demonstrate that the proposed framework yields competitive performance on graph classification tasks with only a small number of labeled graph examples.

Plant6mA: A predictor for predicting N6-methyladenine sites with lightweight structure in plant genomes.

N6-methyladenine (6mA) in DNA, a type of DNA methylation in epigenetic modification, has attracted extensive attention in recent years. In order to improve our understanding of 6mA biological activities and mechanisms in plant genomes, we need to be able to accurately identify 6mA sites. Because traditional wet-lab experiments frequently necessitate a large amount of manpower and time, a plethora of computational methods, particularly machine learning, have emerged to achieve fast and accurate 6mA site prediction. Traditional machine learning methods, on the other hand, rely heavily on manual features and integrated learning to improve performance, resulting in a reliance on prior knowledge and a large model scale. Furthermore, many models are only trained and tested for one species, with no comparison of model generalization performance, resulting in models with limited practical usability. In order to increase the generalization capability of the model, we propose a lightweight structure predictor Plant6mA based on Transformer encoder. Based on results on independent test sets, our proposed Plant6mA has better generalization performance than the most advanced methods in predicting 6mA location in plant genomes. Plant6mA's MultiHead Attention mechanism effectively enhances its expressive ability by capturing potential biological information from multiple scales of the input sequence. Furthermore, we used a dimensionality reduction tool to visualize Plant6mA's training process and visually demonstrate the effectiveness of our model.

MABUSE: A margin optimization based feature subset selection algorithm using boosting principles

Feature subset selection is one of the most common procedures in machine learning tasks. In a broad sense, feature selection methods can be classified into three major groups, embedded, filter and wrapper methods. Although wrappers might attain superior classification performances, they suffer from scalability issues as they are more computationally expensive than the other methods. Filters are typically faster, and sometimes they are the only applicable methods when the datasets are large. In the field of classification, margin optimization has been proven to be an efficient approach for improving the generalization performance of many classification models. Although margins have been used as criteria for feature selection, in most cases, the most advanced methods are wrappers, which suffer from high computational costs and do not outperform the faster algorithms. In this paper, we propose MABUSE, which is a feature selection method that optimizes margins from a filter perspective. We consider a nearest-neighbor margin definition and, borrowing from the strategy of classifier ensemble construction using boosting, we develop a new method that uses a simple heuristic search. Extensive experimental validation demonstrates that our proposed approach outperforms the state-of-the-art algorithms in both classification and reduction, and has a computational cost that is similar to previous algorithms.