Overfitting In Training Research Articles

A residual autoencoder-based transformer for fault detection of multivariate processes

The complexity of high-dimensional and noisy process signals reduces the effectiveness of conventional fault detection methods in industrial processes. Based on the hypothesis that data collected from normal and faulty processes has different characteristics, unsupervised deep neural networks, e.g., autoencoders, have been widely applied in process fault detection and achieved good performance. Many variants have been proposed to improve feature learning by combining different network structures. In this paper, a new transformer model, residual autoencoder-based transformer, is proposed for process fault detection. Firstly, autoencoder and transformer are integrated for better unsupervised feature learning of process signals. Secondly, linear embedding and attention mechanisms with bias are proposed to generate effective features from process signals. Finally, residual connections are constructed between the encoder and decoder of RATransformer to address overfitting in training. Four industrial cases are used to test the performance of RATransformer for process fault detection. The results show that the fault detection rate of RATransformer is at least 1 % higher than other comparison methods. Moreover, the testing results show that the model structure improves the fault detection performance of RATransformer. The complex models like RATransformer can be used in the industrial process when sufficient normal process data is available. An end-to-end training method can be further developed to improve the applicability of RATransformer in process fault detection in the future.

Recovering a clean background: A parallel deep network architecture for single-image deraining

Deep convolutional neural networks have been popularly applied in single-image deraining recently. Nevertheless, as the network becomes deeper, it is easy to cause training over-fitting and performance saturation, particularly in the case of insufficient training data. In this paper, we report the design of a new network, namely parallel deraining convolutional neural network (PARDNet), for single-image deraining. Specifically, PARDNet adopts two parallel residual sub-networks based on different receptive fields to extract more comprehensive characteristics of the rain streaks, as well as decrease the depth of the network. The hybrid dilated convolution is employed to enlarge the sub-network’s receptive field to capture more context information. The efficient channel attention module is integrated into the proposed PARDNet to capture rain streaks more effectively and preserve more background details. Furthermore, to facilitate the network training, the residual learning is also fused into PARDNet in a holistic way. Extensive experiments on synthetic and real-world rainy image datasets demonstrate the superiority of PARDNet for single-image deraining.

Improved pitch shifting data augmentation for ship-radiated noise classification

The limited amount of ship-radiated noise data causes machine learning models to be prone to overfitting in training, and data augmentation methods could improve model generalization performance. The frequency stability of harmonic line spectra in ship-radiated noise leads to a lack of sample diversity generated by the pitch shifting method, which is overcome by the proposed improved method. Nine classification algorithms combining three time-frequency features and three classifiers are implemented and evaluated on the DeepShip and the ShipsEar datasets. The average accuracy increased by 1.67% on DeepShip and 2.25% on ShipsEar, using improved pitch shifting and time stretching augmentation methods. The constant-Q transformation-convolutional neural networks (CQT-CNN) performs best among these nine algorithms. Its accuracy improved from 68.33% to 74.08% on the DeepShip, and the F1 score of it improved from 57.92% to 61.45% on the ShipsEar. Data augmentation improves classification performance for each class of ships in different ways, suggesting that augmentation specific to the class and state of the ship would improve classification performance further.

DKFD: Optimizing Common Pediatric Dermatoses Detection with Novel Loss Function and Post-Processing

Using appropriate classification and recognition technology can help physicians make clinical diagnoses and decisions more effectively as a result of the ongoing development of artificial intelligence technology in the medical field. There are currently a number of issues with the detection of common pediatric dermatoses, including the challenge of image collection, the low resolution of some collected images, the intra-class variability and inter-class similarity of disease symptoms, and the mixing of disease symptom detection results. To resolve these problems, we first introduced the Random Online Data Augmentation and Selective Image Super-Resolution Reconstruction (RDA-SSR) method, which successfully avoids overfitting in training, to address the issue of the small dataset and low resolution of collected images, increase the number of images, and improve the image quality. Second, for the issue of an imbalance between difficult and simple samples, which is brought on by the variation within and between classes of disease signs during distinct disease phases. By increasing the loss contribution of hard samples for classification on the basis of the cross-entropy, we propose the DK_Loss loss function for two-stage object detection, allowing the model to concentrate more on the learning of hard samples. Third, in order to reduce redundancy and improve detection precision, we propose the Fliter_nms post-processing method for the intermingling of detection results based on the NMS algorithm. We created the CPD-10 image dataset for common pediatric dermatoses and used the Faster R-CNN network training findings as a benchmark. The experimental results show that the RDA-SSR technique, while needing a similar collection of parameters, can improve mAP by more than 4%. Furthermore, experiments were conducted over the CPD-10 dataset and PASCAL VOC2007 dataset to evaluate the effectiveness of DK_Loss over the two-stage object detection algorithm, and the results of cross-entropy loss-function-based training are used as baselines. The findings demonstrated that, with DK_Loss taken into account, its mAP is 1–2% above the baseline. Furthermore, the experiments confirmed that the Fliter_nms post-processing method can also improve model precision.

MuRCL: Multi-Instance Reinforcement Contrastive Learning for Whole Slide Image Classification.

Multi-instance learning (MIL) is widely adop- ted for automatic whole slide image (WSI) analysis and it usually consists of two stages, i.e., instance feature extraction and feature aggregation. However, due to the "weak supervision" of slide-level labels, the feature aggregation stage would suffer from severe over-fitting in training an effective MIL model. In this case, mining more information from limited slide-level data is pivotal to WSI analysis. Different from previous works on improving instance feature extraction, this paper investigates how to exploit the latent relationship of different instances (patches) to combat overfitting in MIL for more generalizable WSI classification. In particular, we propose a novel Multi-instance Rein- forcement Contrastive Learning framework (MuRCL) to deeply mine the inherent semantic relationships of different patches to advance WSI classification. Specifically, the proposed framework is first trained in a self-supervised manner and then finetuned with WSI slide-level labels. We formulate the first stage as a contrastive learning (CL) process, where positive/negative discriminative feature sets are constructed from the same patch-level feature bags of WSIs. To facilitate the CL training, we design a novel reinforcement learning-based agent to progressively update the selection of discriminative feature sets according to an online reward for slide-level feature aggregation. Then, we further update the model with labeled WSI data to regularize the learned features for the final WSI classification. Experimental results on three public WSI classification datasets (Camelyon16, TCGA-Lung and TCGA-Kidney) demonstrate that the proposed MuRCL outperforms state-of-the-art MIL models. In addition, MuRCL can achieve comparable performance to other state-of-the-art MIL models on TCGA-Esca dataset.

A novel approach for automatic segmentation of prostate and its lesion regions on magnetic resonance imaging.

To develop an accurate and automatic segmentation model based on convolution neural network to segment the prostate and its lesion regions. Of all 180 subjects, 122 healthy individuals and 58 patients with prostate cancer were included. For each subject, all slices of the prostate were comprised in the DWIs. A novel DCNN is proposed to automatically segment the prostate and its lesion regions. This model is inspired by the U-Net model with the encoding-decoding path as the backbone, importing dense block, attention mechanism techniques, and group norm-Atrous Spatial Pyramidal Pooling. Data augmentation was used to avoid overfitting in training. In the experimental phase, the data set was randomly divided into a training (70%), testing set (30%). four-fold cross-validation methods were used to obtain results for each metric. The proposed model achieved in terms of Iou, Dice score, accuracy, sensitivity, 95% Hausdorff Distance, 86.82%,93.90%, 94.11%, 93.8%,7.84 for the prostate, 79.2%, 89.51%, 88.43%,89.31%,8.39 for lesion region in segmentation. Compared to the state-of-the-art models, FCN, U-Net, U-Net++, and ResU-Net, the segmentation model achieved more promising results. The proposed model yielded excellent performance in accurate and automatic segmentation of the prostate and lesion regions, revealing that the novel deep convolutional neural network could be used in clinical disease treatment and diagnosis.

Identification for the cortical 3-Hinges folding pattern based on cortical morphological and structural features.

The Cortical 3-Hinges Folding Pattern (i.e., 3-Hinges) is one of the brain's hallmarks, and it is of great reference for predicting human intelligence, diagnosing eurological diseases and understanding the brain functional structure differences among gender. Given the significant morphological variability among individuals, it is challenging to identify 3-Hinges, but current 3-Hinges researches are mainly based on the computationally expensive Gyral-net method. To address this challenge, this paper aims to develop a deep network model to realize the fast identification of 3-Hinges based on cortical morphological and structural features. The main work includes: (1) The morphological and structural features of the cerebral cortex are extracted to relieve the imbalance between the number of 3-Hinges and each brain image's voxels; (2) The feature vector is constructed with the K nearest neighbor algorithm from the extracted scattered features of the morphological and structural features to alleviate over-fitting in training; (3) The squeeze excitation module combined with the deep U-shaped network structure is used to learn the correlation of the channels among the feature vectors; (4) The functional structure roles that 3-Hinges plays between adolescent males and females are discussed in this work. The experimental results on both adolescent and adult MRI datasets show that the proposed model achieves better performance in terms of time consumption. Moreover, this paper reveals that cortical sulcus information plays a critical role in the procedure of identification, and the cortical thickness, cortical surface area, and volume characteristics can supplement valuable information for 3-Hinges identification to some extent. Furthermore, there are significant structural differences on 3-Hinges among adolescent gender.

Guided Dropout: Improving Deep Networks Without Increased Computation

Deep convolution neural networks are going deeper and deeper. However, the complexity of models is prone to overfitting in training. Dropout, one of the crucial tricks, prevents units from co-adapting too much by randomly dropping neurons during training. It effectively improves the performance of deep networks but ignores the importance of the differences between neurons. To optimize this issue, this paper presents a new dropout method called guided dropout, which selects the neurons to switch off according to the differences between the convolution kernel and preserves the informative neurons. It uses an unsupervised clustering algorithm to cluster similar neurons in each hidden layer, and dropout uses a certain probability within each cluster. Thereby this would preserve the hidden layer neurons with different roles while maintaining the model’s scarcity and generalization, which effectively improves the role of the hidden layer neurons in learning the features. We evaluated our approach compared with two standard dropout networks on three well-established public object detection datasets. Experimental results on multiple datasets show that the method proposed in this paper has been improved on false positives, precision-recall curve and average precision without increasing the amount of computation. It can be seen that the increased performance of guided dropout is thanks to shallow learning in the networks. The concept of guided dropout would be beneficial to the other vision tasks.

A Novel Lightweight Attention-Discarding Transformer for High-Resolution SAR Image Classification

Vision Transformer (ViT) has been introduced in high-resolution synthetic aperture radar (HR SAR) image classification due to its excellent global feature extraction ability. However, small samples of SAR images make it difficult to fit the ViT with excessive trainable parameters, which easily results in over-fitting in training. Meanwhile, poor capability in capturing local features of ViT limits its accuracy in SAR image classification. To solve these problems, this letter proposes a new Lightweight Attention-Discarding Transformer (LAD Transformer) for the classification of HR SAR images. In the proposed model, the backbone of the advanced Swin transformer is used to model global information and extract hierarchical features. Moreover, the vital feature extraction part of the LAD Transformer completely discards the self-attention mechanism and extracts local features of SAR images by introducing lighter group convolution and channel shuffle (GC-CS Block). In addition, to address the estimation shift caused by consecutive batch normalization (BN) layers, a new composite normalization method consisting of Batch normalization and Layer Normalization (BLN) in GC-CS Block is proposed. The experiments show that the proposed network has fewer parameters and higher classification accuracy on two real HR SAR data.

Attention-based advantage actor-critic algorithm with prioritized experience replay for complex 2-D robotic motion planning

Robotic motion planning in dense and dynamic indoor scenarios constantly challenges the researchers because of the motion unpredictability of obstacles. Recent progress in reinforcement learning enables robots to better cope with the dense and unpredictable obstacles by encoding complex features of the robot and obstacles into the encoders like the long-short term memory (LSTM). Then these features are learned by the robot using reinforcement learning algorithms, such as the deep Q network and asynchronous advantage actor critic algorithm. However, existing methods depend heavily on expert experiences to enhance the convergence speed of the networks by initializing them via imitation learning. Moreover, those approaches based on LSTM to encode the obstacle features are not always efficient and robust enough, therefore sometimes causing the network overfitting in training. This paper focuses on the advantage actor critic algorithm and introduces an attention-based actor critic algorithm with experience replay algorithm to improve the performance of existing algorithm from two perspectives. First, LSTM encoder is replaced by a robust encoder attention weight to better interpret the complex features of the robot and obstacles. Second, the robot learns from its past prioritized experiences to initialize the networks of the advantage actor-critic algorithm. This is achieved by applying the prioritized experience replay method, which makes the best of past useful experiences to improve the convergence speed. As results, the network based on our algorithm takes only around 15% and 30% experiences to get rid of the early-stage training without the expert experiences in cases with five and ten obstacles, respectively. Then it converges faster to a better reward with less experiences (near 45% and 65% of experiences in cases with ten and five obstacles respectively) when comparing with the baseline LSTM-based advantage actor critic algorithm. Our source code is freely available at the GitHub (https://github.com/CHUENGMINCHOU/AW-PER-A2C).

ACR-SA: attention-based deep model through two-channel CNN and Bi-RNN for sentiment analysis.

Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) have been successfully applied to Natural Language Processing (NLP), especially in sentiment analysis. NLP can execute numerous functions to achieve significant results through RNN and CNN. Likewise, previous research shows that RNN achieved meaningful results than CNN due to extracting long-term dependencies. Meanwhile, CNN has its advantage; it can extract high-level features using its local fixed-size context at the input level. However, integrating these advantages into one network is challenging because of overfitting in training. Another problem with such models is the consideration of all the features equally. To this end, we propose an attention-based sentiment analysis using CNN and two independent bidirectional RNN networks to address the problems mentioned above and improve sentiment knowledge. Firstly, we apply a preprocessor to enhance the data quality by correcting spelling mistakes and removing noisy content. Secondly, our model utilizes CNN with max-pooling to extract contextual features and reduce feature dimensionality. Thirdly, two independent bidirectional RNN, i.e., Long Short-Term Memory and Gated Recurrent Unit are used to capture long-term dependencies. We also applied the attention mechanism to the RNN layer output to emphasize each word’s attention level. Furthermore, Gaussian Noise and Dropout as regularization are applied to avoid the overfitting problem. Finally, we verify the model’s robustness on four standard datasets. Compared with existing improvements on the most recent neural network models, the experiment results show that our model significantly outperformed the state-of-the-art models.

Forest Fire Smoke Detection Method Based on MoAm-YOLOv4 Algorithm

To improve the performance of the forest fire smoke detection model and achieve a better balance between detection accuracy and speed, an improved YOLOv4 detection model (MoAm-YOLOv4) that combines a lightweight network and attention mechanism was proposed. Based on the YOLOv4 algorithm, the backbone network CSPDarknet53 was replaced with a lightweight network MobilenetV1 to reduce the model’s size. An attention mechanism was added to the three channels before the output to increase its ability to extract forest fire smoke effectively. The algorithm used the K-means clustering algorithm to cluster the smoke dataset, and obtained candidate frames that were close to the smoke images; the dataset was expanded to 2000 images by the random flip expansion method to avoid overfitting in training. The experimental results show that the improved YOLOv4 algorithm has excellent detection effect. Its mAP can reach 93.45%, precision can get 93.28%, and the model size is only 45.58 MB. Compared with YOLOv4 algorithm, MoAm-YOLOv4 improves the accuracy by 1.3% and reduces the model size by 80% while sacrificing only 0.27% mAP, showing reasonable practicability.

Research on early warning of agricultural credit and guarantee risk based on deep learning

Under the impact of agricultural industry differentiation, traditional financial risk model cannot forewarn the guarantee risk of agricultural credit with effectively. This paper proposes an early warning algorithm of agricultural credit and guarantee risk that can effectively overcome the interference of external factors. Using deep learning network, the risk algorithm of agricultural credit and guarantee was built and it could change the deep belief network into supervised learning. To train for an optimal model, two new hidden layers are added to extract image feature vectors, as well as a Softmax classifier. The model is trained and evaluated by the usage of the risk data set of L province from 2017 to 2019, reinforcing the pre-training network and data to deal with the issue of overfitting in training. The results show that the accuracy of the model reaches 92.56%, when the training sample proportion is 90%, with all the 13 factors in the test set taken as input. It shows that the training of the model worked well and that it can effectively predict the risk of agricultural credit and guarantee.

RLSTM: A New Framework of Stock Prediction by Using Random Noise for Overfitting Prevention.

An accurate prediction of stock market index is important for investors to reduce financial risk. Although quite a number of deep learning methods have been developed for the stock prediction, some fundamental problems, such as weak generalization ability and overfitting in training, need to be solved. In this paper, a new deep learning model named Random Long Short-Term Memory (RLSTM) is proposed to get a better predicting result. RLSTM includes prediction module, prevention module, and three full connection layers. Input of the prediction module is a stock or an index which needs to be predicted. That of the prevention module is a random number series. With the index of Shanghai Securities Composite Index (SSEC) and Standard & Poor's 500 (S&P500), simulations show that the proposed RLSTM can mitigate the overfitting and outperform others in accuracy of prediction.

Anchor-free Convolutional Network with Dense Attention Feature Aggregation for Ship Detection in SAR Images

In recent years, with the improvement of synthetic aperture radar (SAR) imaging resolution, it is urgent to develop methods with higher accuracy and faster speed for ship detection in high-resolution SAR images. Among all kinds of methods, deep-learning-based algorithms bring promising performance due to end-to-end detection and automated feature extraction. However, several challenges still exist: (1) standard deep learning detectors based on anchors have certain unsolved problems, such as tuning of anchor-related parameters, scale-variation and high computational costs. (2) SAR data is huge but the labeled data is relatively small, which may lead to overfitting in training. (3) To improve detection speed, deep learning detectors generally detect targets based on low-resolution features, which may cause missed detections for small targets. In order to address the above problems, an anchor-free convolutional network with dense attention feature aggregation is proposed in this paper. Firstly, we use a lightweight feature extractor to extract multiscale ship features. The inverted residual blocks with depth-wise separable convolution reduce the network parameters and improve the detection speed. Secondly, a novel feature aggregation scheme called dense attention feature aggregation (DAFA) is proposed to obtain a high-resolution feature map with multiscale information. By combining the multiscale features through dense connections and iterative fusions, DAFA improves the generalization performance of the network. In addition, an attention block, namely spatial and channel squeeze and excitation (SCSE) block is embedded in the upsampling process of DAFA to enhance the salient features of the target and suppress the background clutters. Third, an anchor-free detector, which is a center-point-based ship predictor (CSP), is adopted in this paper. CSP regresses the ship centers and ship sizes simultaneously on the high-resolution feature map to implement anchor-free and nonmaximum suppression (NMS)-free ship detection. The experiments on the AirSARShip-1.0 dataset demonstrate the effectiveness of our method. The results show that the proposed method outperforms several mainstream detection algorithms in both accuracy and speed.

Fine-Tuning Pre-Trained Convolutional Neural Networks for Women Common Cancer Classification using RNA-Seq Gene Expression

Most of the recent cancer classification methods use gene expression profile as features because it can provide very important information regarding tumor characteristics. Motivated by their success in the computer vision area now deep learning has been successfully applied to medical data because it can read non-linear patterns in a complex feature and can allow the leverage of information from unlabeled data of problems that do not belong to the problem being handled. In this paper, we implement transfer learning, which refers to the use of a model trained on one task to perform classification on another task to classify five cancer types that most commonly affect women. We used VGG16, Xception, DenseNet, and ResNet50 as base models and then added a dense layer to reflect our five-class classification problem. To avoid training over-fitting that can result in a very high training accuracy and a low cross-validation accuracy we used L2-regularization. We retrained (fine-tuned) these models using a five-fold cross-validation approach on RNA-Seq gene expression data after transforming it into 2D-image like data. We used the softmax activation function with the prediction dense layer and adam as optimizer in the model fit for all four architectures. The highest performance is obtained when fine-tuning Xception architecture, which achieved classification accuracy = 98.6%, precision = 98.6%, recall = 97.8%, and F1-score = 98% on five-fold cross-validation training and testing approach.

An ensemble method with sentiment features and clustering support

Long Short Term Memory (LSTM) and Convolutional Neural Network (CNN) are efficiently applied to natural language processing, especially sentiment analysis. CNN employs filters to capture local dependencies while LSTM designs a cell to memorize long-distance information. However, integrating these advantages into one model is challenging because of overfitting in training. To avoid this problem, we propose a freezing technique to learn sentiment-specific vectors from CNN and LSTM. This technique is efficient for integrating the advantages of various deep learning models. We also observe that semantically clustering documents into groups is more beneficial for ensemble methods. According to the experiments, our method achieves competitive results on the five well-known datasets: Pang & Lee movie reviews, Stanford Twitter Sentiment and Stanford Sentiment Treebank for sentence level, IMDB large movie reviews, and SenTube for document level.

Regularized margin-based conditional log-likelihood loss for prototype learning

The classification performance of nearest prototype classifiers largely relies on the prototype learning algorithm. The minimum classification error (MCE) method and the soft nearest prototype classifier (SNPC) method are two important algorithms using misclassification loss. This paper proposes a new prototype learning algorithm based on the conditional log-likelihood loss (CLL), which is based on the discriminative model called log-likelihood of margin (LOGM). A regularization term is added to avoid over-fitting in training as well as to maximize the hypothesis margin. The CLL in the LOGM algorithm is a convex function of margin, and so, shows better convergence than the MCE. In addition, we show the effects of distance metric learning with both prototype-dependent weighting and prototype-independent weighting. Our empirical study on the benchmark datasets demonstrates that the LOGM algorithm yields higher classification accuracies than the MCE, generalized learning vector quantization (GLVQ), soft nearest prototype classifier (SNPC) and the robust soft learning vector quantization (RSLVQ), and moreover, the LOGM with prototype-dependent weighting achieves comparable accuracies to the support vector machine (SVM) classifier.

Genetic training of network using chaos concept: application to QSAR studies of vibration modes of tetrahedral halides.

The chaotic dynamical system is introduced in genetic algorithm to train ANN to formulate the CGANN algorithm. Logistic mapping as one of the most important chaotic dynamic mappings provides each new generation a high chance to hold GA's population diversity. This enhances the ability to overcome overfitting in training an ANN. The proposed CGANN has been used for QSAR studies to predict the tetrahedral modes (nu(1)(A1) and nu(2)(E)) of halides [MX(4)](epsilon). The frequencies predicted by QSAR were compared with those calculated by quantum chemistry methods including PM3, AM1, and MNDO/d. The possibility of improving the predictive ability of QSAR by including quantum chemistry parameters as feature variables has been investigated using tetrahedral tetrahalide examples.