Improve Generalization Performance Research Articles

Rollback Ensemble With Multiple Local Minima in Fine-Tuning Deep Learning Networks.

Image retrieval is a challenging problem that requires learning generalized features enough to identify untrained classes, even with very few classwise training samples. In this article, to obtain generalized features further in learning retrieval data sets, we propose a novel fine-tuning method of pretrained deep networks. In the retrieval task, we discovered a phenomenon in which the loss reduction in fine-tuning deep networks is stagnated, even while weights are largely updated. To escape from the stagnated state, we propose a new fine-tuning strategy to roll back some of the weights to the pretrained values. The rollback scheme is observed to drive the learning path to a gentle basin that provides more generalized features than a sharp basin. In addition, we propose a multihead ensemble structure to create synergy among multiple local minima obtained by our rollback scheme. Experimental results show that the proposed learning method significantly improves generalization performance, achieving state-of-the-art performance on the Inshop and SOP data sets.

Skill-learning by observation-training with patients after traumatic brain injury.

Traumatic brain injury (TBI) is a major cause of death and disability in Western society, and often results in functional and neuropsychological abnormalities. Memory impairment is one of the most significant cognitive implications after TBI. In the current study we investigated procedural memory acquisition by observational training in TBI patients. It was previously found that while practicing a new motor skill, patients engage in all three phases of skill learning–fast acquisition, between-session consolidation, and long-term retention, though their pattern of learning is atypical compared to healthy participants. A different set of studies showed that training by observing a motor task, generally prompted effective acquisition and consolidation of procedural knowledge in healthy participants. The aim of our study was to (i) evaluate the potential benefit of action observation in TBI patients. (ii) Examine the possibility of general improvement in performance between the first (24 h post-training) and second (2 weeks post-training) stage of the study. (iii) Investigate the link between patients’ ability to benefit from observational learning (via performance gains–speed and accuracy) and common measures of injury (such as severity of injury, functional and cognitive measures).Materials and methodsPatients hospitalized after moderate to severe TBI, were trained by observation for the finger opposition sequence (FOS) motor task. They were then tested for the observation-trained sequence (A) and a similar control sequence (B), at two different time-points (24 h post-training and 2 weeks later).Results revealed(i) a significant difference in performance between the trained (A) and untrained (B) sequences, in favor of the trained sequence. (ii) An increase in performance for both sequences A and B toward the second (retention) session. (iii) The advantage for sequence A was stable and preserved also in the second session. (iv) Participants with lower moderate Functional Independence Measure (FIM) scores gained more from observational-procedural learning, compared with patients with higher functional abilities.ConclusionOverall, these findings support the notion that TBI patients may achieve procedural memory consolidation and retention through observational learning. Moreover, different functional traits may predict the outcomes of observational training in different patients. These findings may have significant practical implications in the future, regarding skill acquisition methods in TBI patients.

ATM-TCR: TCR-Epitope Binding Affinity Prediction Using a Multi-Head Self-Attention Model.

TCR-epitope pair binding is the key component for T cell regulation. The ability to predict whether a given pair binds is fundamental to understanding the underlying biology of the binding mechanism as well as developing T-cell mediated immunotherapy approaches. The advent of large-scale public databases containing TCR-epitope binding pairs enabled the recent development of computational prediction methods for TCR-epitope binding. However, the number of epitopes reported along with binding TCRs is far too small, resulting in poor out-of-sample performance for unseen epitopes. In order to address this issue, we present our model ATM-TCR which uses a multi-head self-attention mechanism to capture biological contextual information and improve generalization performance. Additionally, we present a novel application of the attention map from our model to improve out-of-sample performance by demonstrating on recent SARS-CoV-2 data.

Implicit Gradient Alignment in Distributed and Federated Learning

A major obstacle to achieving global convergence in distributed and federated learning is the misalignment of gradients across clients or mini-batches due to heterogeneity and stochasticity of the distributed data. In this work, we show that data heterogeneity can in fact be exploited to improve generalization performance through implicit regularization. One way to alleviate the effects of heterogeneity is to encourage the alignment of gradients across different clients throughout training. Our analysis reveals that this goal can be accomplished by utilizing the right optimization method that replicates the implicit regularization effect of SGD, leading to gradient alignment as well as improvements in test accuracies. Since the existence of this regularization in SGD completely relies on the sequential use of different mini-batches during training, it is inherently absent when training with large mini-batches. To obtain the generalization benefits of this regularization while increasing parallelism, we propose a novel GradAlign algorithm that induces the same implicit regularization while allowing the use of arbitrarily large batches in each update. We experimentally validate the benefits of our algorithm in different distributed and federated learning settings.

Improved Feature Pyramid Convolutional Neural Network for Effective Recognition of Music Scores

Music written by composers and performed by multidimensional instruments is an art form that reflects real-life emotions. Historically, people disseminated music primarily through sheet music recording and oral transmission. Among them, recording music in sheet music form was a great musical invention. It became the carrier of music communication and inheritance, as well as a record of humanity's magnificent music culture. The advent of digital technology solves the problem of difficult musical score storage and distribution. However, there are many drawbacks to using data in image format, and extracting music score information in editable form from image data is currently a challenge. An improved convolutional neural network for musical score recognition is proposed in this paper. Because the traditional convolutional neural network SEGNET misclassifies some pixels, this paper employs the feature pyramid structure. Use additional branch paths to fuse shallow image details, shallow texture features that are beneficial to small objects, and high-level features of global information, enrich the multi-scale semantic information of the model, and alleviate the problem of the lack of multiscale semantic information in the model. Poor recognition performance is caused by semantic information. By comparing the recognition effects of other models, the experimental results show that the proposed musical score recognition model has a higher recognition accuracy and a stronger generalization performance. The improved generalization performance allows the musical score recognition method to be applied to more types of musical score recognition scenarios, and such a recognition model has more practical value.

Artificial intelligence in interdisciplinary life science and drug discovery research.

Artificial intelligence in interdisciplinary life science and drug discovery research.

Features of teenagers’ physical performance in different power zones and their consideration in the process of physical education at school

Introduction. The research relevance is due to the need to improve the process of physical education of school teenagers by taking into account the specifics of physical performance in a wide range of available loads and the degree of puberty. The research purpose is to reveal the features of physical performance and motor fitness of schoolchildren aged 13-15 at different puberty stages (PSs), manifested in different relative power zones. Materials and methods. The study involved school teenagers aged 13-15 (n=168), assigned to the main medical group for health reasons. A complex of functional and motor tests was used to assess physical performance and motor fitness. Differences determined by Student’s t-test were considered significant at p<0.05. Results. An analysis of changes in physical performance in the maximum power zone showed that it significantly increases during puberty: when moving from PS I to V, an increase (t=3.09–13.06; ps<0.01–0.001) of all indicators used was observed. A similar situation was observed when working in the submaximal power zone. With an increase in the level of puberty, there was a general improvement in performance (t=2.81-9.14; ps<0.05-0.001). In the high power zone, differences between PSs I and V were revealed only in relation to PWC170 (t=10.29; p<0.001) and six-minute run (t=4.25; p<0.001). In the moderate power zone, the results of the Cooper test (t=7.92; p<0.001) and watt-pulse – WP (t=2.64; p<0.05) improved, the indicator of aerobic capacity decreased – coefficient b of Muller’s equation (t=5.48; p<0.001). In the maximum and submaximal power zones, there was a progressive increase in the integral performance ratings from PS I to V, while in the high and moderate power zones, the maximum level of functionality was observed at PS III. Conclusion. The results obtained give grounds to assume that in physical education lessons at school, when using extra-curricular and out-of-school forms of physical exercise, it is necessary to combine teenagers with the same indicators of biological maturity into one subgroup and, taking this into account, select means and methods for increasing physical performance, developing physical condition and parameters of various metabolic physical activities. The standards for assessing the physical performance and motor fitness of school teenagers of the same passport age should also be developed separately for the initial and final puberty stages.

Generality and specificity in a piecewise linear map model for motor learning

The specificity and generality of practice effects are central issues in motor learning, retention and transfer. The current study implemented the piecewise linear map model to analyze practice effects on five different target time conditions in discrete timing tasks. Five individualized target times were constructed and tested for eighteen adult healthy participants. After the pre-test, all participants practiced the mid-fast condition for 7 sessions before the post-test and the one-week retention-test were performed. In addition to the stochastic and deterministic model parameters, the success rates and ratios of absolute errors to the target windows were analyzed for the task performance. The results showed the general performance improvements as well as reduction of the noise amplitudes in the stochastic component of the map model for the mid-fast, median, mid-slow, and slow conditions but not the fast condition. The decreased absolute value of the trial-to-trial slopes that indicated a more efficient target time adjustment was only observed for the mid-fast condition. The practice effects of generality and specificity in performing sequential timing tasks were revealed in the stochastic and deterministic components of the piecewise linear map model.

Self-Supervised Multi-Label Transformation Prediction for Video Representation Learning

Self-supervised learning is a promising paradigm to address the problem of manual-annotation through effectively leveraging unlabeled videos. By solving self-supervised pretext tasks, powerful video representations can be discovered automatically. However, recent pretext tasks for videos rely on utilizing the temporal properties of videos, ignoring the crucial supervisory signals from the spatial subspace of videos. Therefore, we present a new self-supervised pretext task called Multi-Label Transformation Prediction (MLTP) to sufficiently utilize the spatiotemporal information in videos. In MLTP, all videos are jointly transformed by a set of geometric and color-space transformations, such as rotation, cropping, and color-channel split. We formulate the pretext as a multi-label prediction task. The 3D-CNN is trained to predict a composition of underlying transformations as multiple outputs. Thereby, transformation invariant video features can be learned in a self-supervised manner. Experimental results verify that 3D-CNNs pre-trained using MLTP yield video representations with improved generalization performance for action recognition downstream tasks on UCF101 ([Formula: see text]) and HMDB51 ([Formula: see text]) datasets.

Robust EMRAN-aided coupled controller for autonomous vehicles

This paper presents a coupled, neural network-aided longitudinal cruise and lateral path-tracking controller for an autonomous vehicle with model uncertainties and experiencing unknown external disturbances. Using a feedback error learning mechanism, an inverse vehicle dynamics learning scheme utilizing an adaptive Radial Basis Function (RBF) neural network, referred to as the Extended Minimal Resource Allocating Network (EMRAN) is employed. EMRAN uses an extended Kalman filter for online learning and weight updates, and also incorporates a growing/pruning strategy for maintaining a compact network for easier real-time implementation. The online learning algorithm handles the parametric uncertainties and eliminates the effect of unknown disturbances on the road. Combined with a self-regulating learning scheme for improving generalization performance, the proposed EMRAN-aided control architecture aids a basic PID cruise and Stanley path-tracking controllers in a coupled form. Its performance and robustness to various disturbances and uncertainties are compared with the conventional PID and Stanley controllers, along with a comparison with a fuzzy-based PID controller and an active disturbance rejection control (ADRC) scheme. Simulation results are presented for both slow and high speed scenarios. The root mean square (RMS) and maximum tracking errors clearly indicate the effectiveness of the proposed control scheme in achieving better tracking performance in autonomous vehicles under unknown environments.

Research on Grasping Strategy Based on Residual Network

At present, the research on rectangular grasp strategy is generally based on object detection algorithm, which limits the improvement of model accuracy and generalization performance. This paper studies the semantic segmentation model based on residual network, and uses it to generate grasp strategies. The improved algorithm model not only achieves excellent rectangular grasping strategy prediction, but also has good generalization performance.

AQM based on the queue length: A real-network study.

Active Queue Management (AQM) is recommended by Internet Engineering Task Force to mitigate the bufferbloat phenomenon in the Internet. In this paper, we show the results of comprehensive measurements carried out in our university network, in which a device with an AQM algorithm, designed and programmed for this purpose, was running. The implemented AQM algorithm was based on the dropping function, i.e. arriving packets were dropped randomly, with the probability being a function of the queue length. Several different dropping function forms, proposed in the networking literature, were used, in addition to the classic FIFO queue (no AQM). The experiment lasted over a month, during which the state of the network was measured and recorded several thousand times. This made the results independent of the natural fluctuations of the users’ behavior and the network load. Conclusions on the general performance improvement offered by the implemented AQM, as well as the differences in the performance between particular forms of the dropping function, were reached. Some of these conclusions differ from those drawn previously from simulations. This underlines the need for carrying measurements of new AQMs in real, operating networks, with complex, natural traffic.

Multi-Path U-Net Architecture for Cell and Colony-Forming Unit Image Segmentation.

U-Net is the most cited and widely-used deep learning model for biomedical image segmentation. In this paper, we propose a new enhanced version of a ubiquitous U-Net architecture, which improves upon the original one in terms of generalization capabilities, while addressing several immanent shortcomings, such as constrained resolution and non-resilient receptive fields of the main pathway. Our novel multi-path architecture introduces a notion of an individual receptive field pathway, which is merged with other pathways at the bottom-most layer by concatenation and subsequent application of Layer Normalization and Spatial Dropout, which can improve generalization performance for small datasets. In general, our experiments show that the proposed multi-path architecture outperforms other state-of-the-art approaches that embark on similar ideas of pyramid structures, skip-connections, and encoder–decoder pathways. A significant improvement of the Dice similarity coefficient is attained at our proprietary colony-forming unit dataset, where a score of was achieved for the foreground class.

Footprint Extraction and Sports Training Action Recognition Based on Wireless Network Communication

The combination of scientific and technological achievements and sports has found new opportunities to change people’s sports habits. Sport training takes up an increasing proportion of people’s lives. In order to improve the efficiency of sports training and standardize the training actions of players, this article is based on wireless network communication and uses different types of recognition methods in the field of action recognition to build basic classifications. Iterative mutual training to improve generalization performance can reduce the cost of labeling and realize the complementary advantages of different recognition methods, thereby improving the recognition accuracy of human actions. Finally, the algorithm is used to recognize human movements. This method can effectively overcome the problem of differential degradation of base classifiers in the iterative process of collaborative training and further improve the accuracy of human action recognition. The experimental results prove that the motion recognition of wireless network communication proposed in this paper can effectively improve the accuracy of athletes’ movements, which is more than 20% higher than traditional methods, and, under the guidance of standardized movements, can reduce athletes’ sports injuries.

An Ordered Aggregation-Based Ensemble Selection Method of Lightweight Deep Neural Networks With Random Initialization

Due to the popularity of 5G connectivity and The Internet of Things sensors, deep learning algorithms are being extended to edge devices. Compared with AI(Artificial Intelligence) cloud platforms, the deployment of deep neural networks on edge devices must focus on low power consumption, low latency, stability and reliability. In recent years, the development of lightweight deep neural network architecture has provided a basis for the deployment of deep neural networks on edge devices. However, the shortcomings of deep neural networks, such as overconfidence, vulnerability to adversarial attack, and easy over fitting when samples are insufficient, still limit their applications in many fields. One of the ways to compensate for these defects is to use deep ensemble. An ordered aggregation-based ensemble selection algorithm is proposed, which uses soft-margin as the importance assessment metric to take full advantage of the diversity and complementarity of lightweight deep neural networks obtained from different initialization training, so as to improve the overall performance of multiple edge devices. The experimental results show that this algorithm has a significant improvement in generalization performance compared with random ensemble and ordered aggregation algorithms based on accuracy or diversity, and provides a new complementary idea for the deployment of lightweight deep neural networks on edge devices.

Graph-Based Classification With Multiple Shift Matrices

Due to their effectiveness in capturing similarities between different entities, graphical models are widely used to represent datasets that reside on irregular and complex manifolds. Graph signal processing offers support to handle such complex datasets. In this paper, we propose a novel graph filter design method for semi-supervised data classification. The proposed design uses multiple graph shift matrices, one for each feature, and is shown to provide improved performance when the feature qualities are uneven. We introduce three methods to optimize for the graph filter coefficients and the graph combining coefficients. The first method uses the alternating minimization approach. In the second method, we optimize our objective function by convex relaxation that provides a performance benchmark. The third method adopts a genetic algorithm, which is computationally efficient and better at controlling overfitting. In our simulation experiments, we use both synthetic and real datasets with informative and non-informative features. Monte Carlo simulations demonstrate the effectiveness of multiple graph shift operators in the graph filters. Significant improvements in comparison to conventional graph filters are shown, in terms of average error rate and confidence scores. Furthermore, we perform cross validation to show how our approach can control overfitting and improve generalization performance.

Evaluation and Improvement of Generalization Performance of SAR Ship Recognition Algorithms

As artificial intelligence continues to advance, deep learning has greatly contributed to the advancement of ship recognition using Synthetic Aperture Radar (SAR) images. Deep learning-based SAR ship recognition performance is largely dependent on the sample set used. SAR ship recognition datasets published in recent years, however, are most derived from a single SAR satellite sensor. It needs to be evaluated and analyzed carefully whether the model trained by a single satellite dataset can still achieve the same accuracy with different SAR satellites. The paper focuses on the following research to address these issues. Firstly, using multiple SAR satellite sensors, we create a new SAR ship dataset (named generalization performance evaluation dataset, GPED) containing multi-resolution and multi-polarization data to examine the generalization performance of the deep learning-based SAR ship recognition method. GPED and a marine target detection dataset (MTDD) are then used to evaluate and analyze the generalization performance of current mainstream deep learning methods. According to the the experiment results, the mean average accuracy (mAP) of the ship recognition model trained on GPED is generally higher than that of MTDD, which proves that GPED has a better generalization performance. Furthermore, SAR ship detection datasets have more samples than ship recognition datasets, which inspired us to use transfer learning to transfer knowledge from ship detection to ship recognition. In this paper, a method for ship recognition based on transfer learning that utilizes the knowledge gained from the ship detection task is proposed. The method includes two modules: pretraining module and fine-tuning module. It can apply samples of unlabeled ship types to ship recognition, thus reducing the number of labeled samples that are required for ship recognition. The experimental results on GPED and MTDD show that our method can achieve good recognition performances.

Optimization and comparison of models for core temperature prediction of mother rabbits using infrared thermography

Core temperature is a measurement used to evaluate and diagnose the health status of mother rabbits. The preferred method is to use an indwelling probe to measure the rectal temperature. This procedure can be stressful to rabbits, are time and labor-consuming. The objective of this paper was to develop the fastest, accurate and non-invasive method to measure the rectal temperature of mother rabbits using Infrared thermography (IRT) and machine learning (ML). Two hundred lactating mother rabbits (LR) and two hundred non-lactating mother rabbits (NLR) of the YPLU breed were used in the experiment and IRT data, rectal temperature and ambient temperature were recorded. The results showed that there was a significant correlation between the maximum infrared temperature of eyes (EMIT) and ears (RMIT) and rectal temperature (p < 0.05) and a significant correlation between the EMIT, RMIT and the ambient temperature (p < 0.01) in two kinds of mother rabbits. Least squares support vector machine (LS-SVM), Gaussian process (GP), and partial least squares (PLS) machine learning methods were used to establish models of mother rabbits’ core temperature. Using the over-sampling method to process the training set, three models of LS-SVM, GP, and PLS with improved generalization performance were obtained. The RMSE for the test set by LS-SVM, GP, and PLS methods were 0.13℃, 0.12℃ and 0.07℃ and the R2 were 0.94, 0.96 and 0.98, respectively. Compared with the results obtained from two other regressions, LS-SVM and GP, the PLS method exhibited the best performance. The prediction error of the test set is <0.2℃, which can better predict the core temperature of the mother rabbits.

Adversarial smoothing tri-regression for robust semi-supervised industrial soft sensor

In industrial processes, soft sensor techniques are used to predict the hard-to-measure quality variables under the classic supervised learning paradigm. However, the data challenges, i.e., the widespread noises and inadequate labeled samples, usually make the data-driven sensors weak for application. In this paper, a simple but effective regularization method termed as adversarial smoothing regularization (ASR) is proposed, which measures the local smoothness of prediction around each input sample. By minimizing the divergence between the predictions for noisy and clean inputs, the proposed ASR regulates the learning model to be robust against local perturbation in a semi-supervised manner. Theoretical analysis explains the smoothing capability of ASR, and shows that it is helpful for the improvement of generalization performance. We also design a tri-regression framework to further use the information of unlabeled samples with pseudo labels and present the adversarial smoothing tri-regression (ASTR) model for soft sensor. Based on two industrial processes, comprehensive soft sensor experiments and noise tests are performed to show the robust semi-supervised learning capability of the proposed ASR and ASTR.

Minimum Distribution Support Vector Clustering.

Support vector clustering (SVC) is a boundary-based algorithm, which has several advantages over other clustering methods, including identifying clusters of arbitrary shapes and numbers. Leveraged by the high generalization ability of the large margin distribution machine (LDM) and the optimal margin distribution clustering (ODMC), we propose a new clustering method: minimum distribution for support vector clustering (MDSVC), for improving the robustness of boundary point recognition, which characterizes the optimal hypersphere by the first-order and second-order statistics and tries to minimize the mean and variance simultaneously. In addition, we further prove, theoretically, that our algorithm can obtain better generalization performance. Some instructive insights for adjusting the number of support vector points are gained. For the optimization problem of MDSVC, we propose a double coordinate descent algorithm for small and medium samples. The experimental results on both artificial and real datasets indicate that our MDSVC has a significant improvement in generalization performance compared to SVC.