Overfitting Issues Research Articles

Dynamic demand estimation on large scale networks using Principal Component Analysis: The case of non-existent or irrelevant historical estimates

Calibrating DTA models is complex due to the involved indeterminacy, non-linearity, and dimensionality, restricting the application of conventional calibration approaches, especially for larger networks. For this, Principal Component Analysis (PCA) is slowly establishing itself as the new state of the art because it can greatly tackle two well known challenges—i.e. problem dimensionality and non-linearity. PCA application limits the optimization search space in a lower dimension space, defined by orthogonal Principal Components, evaluated upon a set of historical estimates. In this paper, we solve practical implementation problems for PCA-based calibration techniques. Specifically, we formulate a data-assimilation framework to propose multiple OD historical data-set generation methods which allows the use of PC-based algorithms in case the historical data is irrelevant or unavailable, often the case for large-scale DTA models. Furthermore, we propose a simplified problem formulation that leverages properties of the novel data-set generation framework and helps for faster and more efficient calibration. The methodology is implemented using the PC-SPSA algorithm, which combines PCA with the popular Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm, commonly used to calibrate smaller networks. The approach is tested on a large-scale case study of the Munich metropolitan urban network, with encouraging calibration results. The proposed data-assimilation framework can account for spatial, temporal, and day-to-day variations in the demand. Different methods and combinations are tested and compared. The results suggest that all these correlations should be used in order to avoid over-fitting issues. Furthermore, the implementation properties of PCA and PC-SPSA are also explored using different sensitivity analyses to assess the toll and benefits of using PCA i.e., ease in SPSA hyper-parameter, role of historical data-set generation parameters and the algorithm’s performance against different target demand fluctuations. The analysis shows encouraging results for PC-SPSA robustness and helps establishing simplified guidelines for implementing such PCA-methods practically on large-scale DTA models.

Spine Image Segmentation Based on U-Net and Atrous spatial pyramid pooling

In the past ten years, deep learning has achieved remarkable results in the area of natural image segmentation, and has gradually turned to the field of medical image segmentation. The precise segmentation of spine images can be used for early screening of spondylopathy, which is convenient for early detection and treatment of patients. Aiming at the segmentation of the spine image by the U-Net network, which structure will lead to large model calculation, network overfitting, image size, noise information and other issues. This paper introduces a new method based on spatial pyramid pooling module ASpp on U-Net network and applies the proposed network structure to the segmentation of the spine image. The model is first enhanced by the Spp module and Densenet data; secondly, the U-Net encoding and decoding structure is adopted, and the deep separable convolution is used. This method greatly reduces the complexity and computation of the model, and uses a rectangular convolution kernel to increase the computation of the model in a small amount. Based on the volume, the receptive field of the convolution operation is expanded; finally, in order to effectively enhance the segmentation area of the image, the SE-Net attention mechanism module is added to the lateral connection. The method proposed in this paper conducts ablation experiments on a public dataset of spine images and compares it with the U-Net network. The method proposed IOU in this paper can reach 72%, and the F1-score can reach 0.92. The comparison of the experimental results of spine medical image datasets shows that the method proposed in this paper can effectively improve the accuracy and accuracy of spine image segmentation.

Sparsity constrained wavelet neural networks for robust soft sensor design with application to the industrial KIVCET unit

The main objective of this paper is to develop a soft sensor for predictions of the residue element concentrations of feed mixture in a KIVCET furnace, since such measurements are not available online and are crucial in achieving the objective of real-time optimization. To realize this, a wavelet neural network is considered owing to its high accuracy in approximating functions with a few functional components over networks with sigmoid/tanh activation functions. The idea of wavelet decompositions is used to address the challenges associated with random initialization in such networks. A one-norm penalty on the data-fit term is considered to make the approach robust to outliers, and a sparsity constraint on the parameter vector in terms of zero-norm is considered to handle the issues of over-fitting and input redundancy. The efficiency of the proposed method is demonstrated through an industrial KIVCET unit and a simulation study.

Progressive ShallowNet for large scale dynamic and spontaneous facial behaviour analysis in children

COVID-19 has severely disrupted every aspect of society and left negative impact on our life. Resisting the temptation in engaging face-to-face social connection is not as easy as we imagine. Breaking ties within social circle makes us lonely and isolated, that in turns increase the likelihood of depression related disease and even can leads to death by increasing the chance of heart disease. Not only adults, children's are equally impacted where the contribution of emotional competence to social competence has long term implications. Early identification skill for facial behaviour emotions, deficits, and expression may help to prevent the low social functioning. Deficits in young children's ability to differentiate human emotions can leads to social functioning impairment. However, the existing work focus on adult emotions recognition mostly and ignores emotion recognition in children. By considering the working of pyramidal cells in the cerebral cortex, in this paper, we present progressive lightweight shallow learning for the classification by efficiently utilizing the skip-connection for spontaneous facial behaviour recognition in children. Unlike earlier deep neural networks, we limit the alternative path for the gradient at the earlier part of the network by increase gradually with the depth of the network. Progressive ShallowNet is not only able to explore more feature space but also resolve the over-fitting issue for smaller data, due to limiting the residual path locally, making the network vulnerable to perturbations. We have conducted extensive experiments on benchmark facial behaviour analysis in children that showed significant performance gain comparatively.

A Meta-heuristic Algorithm for the Minimal High-Quality Feature Extraction of Online Reviews

Feature extraction and selection are critical in sentiment analysis (SA) to extract and select only the appropriate features by removing those deemed redundant. As such, the successful implementation of this process leads to better classification accuracy. Inevitably, selecting high-quality minimal features can be challenging given the inherent complication in dealing with over-fitting issues. Most of the current studies used a heuristic method to perform the classification process that will result in selecting and examining only a single feature subset, while ignoring the other subsets that might give better results. This study explored the effect of using the meta-heuristic method together with the ensemble classification method in the sentiment classification of online reviews. Adding to that point, the extraction and selection of relevant features used feature ranking, hyper-parameter optimization, crossover, and mutation, while the classification process utilized the ensemble classifier. The proposed method was tested on the polarity movie review dataset v2.0 and product review dataset (books, electronics, kitchen, and music). The test results indicated that the proposed method significantly improved the classification results by 94%, which far exceeded the existing method. Therefore, the proposed feature extraction and selection method can help in improving the performance of SA in online reviews and, at the same time, reduce the number of extracted features.

Feature Importance Score-Based Functional Link Artificial Neural Networks for Breast Cancer Classification.

Growth of malignant tumors in the breast results in breast cancer. It is a cause of death of many women across the world. As a part of treatment, a woman might have to go through painful surgery and chemotherapy that may further lead to severe side effects. However, it is possible to cure it if it is diagnosed in the initial stage. Recently, many researchers have leveraged machine learning (ML) techniques to classify breast cancer. However, these methods are computationally expensive and prone to the overfitting problem. A simple single-layer neural network, i.e., functional link artificial neural network (FLANN), is proposed to overcome this problem. Further, the F-score is used to reduce the issue of overfitting by selecting features having a higher significance level. In this paper, FLANN is proposed to classify breast cancer using Wisconsin Breast Cancer Dataset (WBCD) (with 699 samples) and Wisconsin Diagnostic Breast Cancer (WDBC) (with 569 samples) datasets. Experimental results reveal that the proposed models can diagnose breast cancer with higher performance. The proposed model can be used in the early breast cancer diagnosis with 99.41% accuracy.

Radar Target Detection With Multi-Task Learning in Heterogeneous Environment

From the perspective of feature extraction and classification, deep neural networks are widely used in radar target detection. However, in heterogeneous environment, the traditional deep neural networks are difficult to extract a robust feature, which leads to the degradation of network detection performance. In order to address this problem, a radar target detection method with multi-task learning in heterogeneous environment is proposed. Considering the influence of heterogeneous data distribution, the proposed method designs a contrastive learning module added to a multi-task autoencoder. It can learn a compact and distinguishable feature representation, which enhances the feature separability between the clutter and the target. Simultaneously, a classifier is introduced to realize a binary detection in the feature representation. Comprehensive experiments are carried out to show that the proposed method guarantees a good detection performance in heterogeneous environment and solves the issue of over-fitting to a certain extent. Compared with some classical detectors, the proposed method shows better performance.

Temporal Point Cloud Fusion With Scene Flow for Robust 3D Object Tracking

Non-visual range sensors such as Lidar have shown the potential to detect, locate and track objects in complex dynamic scenes thanks to their higher stability in comparison with vision-based sensors like cameras. However, due to the disorder, sparsity, and irregularity of the point cloud, it is much more challenging to take advantage of the temporal information in the dynamic 3D point cloud sequences, as it has been done in the image sequences for improving detection and tracking. In this paper, we propose a novel scene-flow-based point cloud feature fusion module to tackle this challenge, based on which a 3D object tracking framework is also achieved to exploit the temporal motion information. Moreover, we carefully designed several training schemes that contribute to the success of this new module by eliminating the issues of overfitting and long-tailed distribution of object categories. Extensive experiments on the public KITTI 3D object tracking dataset demonstrate the effectiveness of the proposed method by achieving superior results to the baselines. The source code is available at <uri>https://github.com/Tsinghua-OpenICV/SharingVan-OpenPCDet</uri>.

Generating Hypergraph-Based High-Order Representations of Whole-Slide Histopathological Images for Survival Prediction.

Patient survival prediction based on gigapixel whole-slide histopathological images (WSIs) has become increasingly prevalent in recent years. A key challenge of this task is achieving an informative survival-specific global representation from those WSIs with highly complicated data correlation. This article proposes a multi-hypergraph based learning framework, called "HGSurvNet," to tackle this challenge. HGSurvNet achieves an effective high-order global representation of WSIs via multilateral correlation modeling in multiple spaces and a general hypergraph convolution network. It has the ability to alleviate over-fitting issues caused by the lack of training data by using a new convolution structure called hypergraph max-mask convolution. Extensive validation experiments were conducted on three widely-used carcinoma datasets: Lung Squamous Cell Carcinoma (LUSC), Glioblastoma Multiforme (GBM), and National Lung Screening Trial (NLST). Quantitative analysis demonstrated that the proposed method consistently outperforms state-of-the-art methods, coupled with the Bayesian Concordance Readjust loss. We also demonstrate the individual effectiveness of each module of the proposed framework and its application potential for pathology diagnosis and reporting empowered by its interpretability potential.

A Transfer Ensemble Learning Method for Evaluating Power Transformer Health Conditions with Limited Measurement Data

Health condition evaluation of power transformers is of great importance for safe and reliable operation of power grids. The data-driven methods have been widely studied and applied. However, in practical applications, collecting high-quality data is difficult and expensive. Therefore, it is still challenging to build effective diagnostic models with insufficient labeled training data. This paper proposes a transfer ensemble model to address this data scarcity problem. Firstly, a one-dimensional convolutional neural network is pre-trained using the large source domain dataset. Then, the target model parameters are initialized with those of the source model and fine-tuned using the target dataset. A transfer strategy is proposed to decide what and where the diagnostic knowledge should be transferred from the source domain to the target domain. Finally, an ensemble model is built on the basis of a series of target models with different transfer strategies, which can further alleviate the over-fitting issue and improve the gener-alization ability. The experimental results validate the effectiveness of the proposed method and show the feasibility of practical applications.

Analysis of performance of accuracy by adding new features individually using Relief-F and Budget Tree Random Forest (RFBTRF) method

The education is very important for improving the values of students in the society. Different types of features like school related features, student related features, parent related features and teacher related features are influencing the success rate of students in their education. Identification of best features from the huge set of features for analyzing the success or failure of a student is one important challenge to the research community and academicians. The set of features information is collected for preparing the student dataset also one difficult task in the prediction of student academic performance. We collected a student dataset of different schools that contains 4965 student’s information. The dataset contains information of 45 features of different categories such as school related features, student related features, parent related features and teacher related features. All features are not useful for predicting the academic performance of a student. The Data mining methods are applied in various research domains including education to extract hidden information from datasets. The feature selection algorithms are used to determine the best informative features by eliminating the irrelevant and redundant features. In this work, Relief-F Budget Tree Random Forest feature selection algorithm is used to identify the relevant features in the collected school dataset. Five different machine learning models are used to predict the efficiency of feature selection algorithm. The decision tree model shows best accuracy for student academic performance prediction compared with other models. The experimental results display that the RFBTRF algorithm identifies the best informative features for enhancing the accuracy of student academic performance prediction and also reduces the over-fitting issues. The experiment started with individual features and then continued with combination of different categories of features. It was observed that the accuracy of student academic performance prediction is decreased when some categories of features are added to other categories of features.

Realistic Smile Expression Recognition Approach Using Ensemble Classifier with Enhanced Bagging

A robust smile recognition system could be widely used for many real-world applications. Classification of a facial smile in an unconstrained setting is difficult due to the invertible and wide variety in face images. In this paper, an adaptive model for smile expression classification is suggested that integrates a fast features extraction algorithm and cascade classifiers. Our model takes advantage of the intrinsic association between face detection, smile, and other face features to alleviate the over-fitting issue on the limited training set and increase classification results. The features are extracted taking into account to exclude any unnecessary coefficients in the feature vector; thereby enhancing the discriminatory capacity of the extracted features and reducing the computational process. Still, the main causes of error in learning are due to noise, bias, and variance. Ensemble helps to minimize these factors. Combinations of multiple classifiers decrease variance, especially in the case of unstable classifiers, and may produce a more reliable classification than a single classifier. However, a shortcoming of bagging as the best ensemble classifier is its random selection, where the classification performance relies on the chance to pick an appropriate subset of training items. The suggested model employs a modified form of bagging while creating training sets to deal with this challenge (error-based bootstrapping). The experimental results for smile classification on the JAFFE, CK+, and CK+48 benchmark datasets show the feasibility of our proposed model.

Deep Siamese Semantic Segmentation Network for PCB Welding Defect Detection

Deep learning has been widely used in recent years for printed circuit board (PCB) defect detection because of its excellent performance. However, deep-learning-based approaches often suffer from the over-fitting problem due to the lack of sufficient training data in real applications. Meanwhile, these approaches still have some challenges to detect these defects with small sizes and irregular shapes. To address these problems, this article has developed a novel deep Siamese semantic segmentation network which integrates the similarity measurement of the Siamese network with the encoder&#x2013;decoder semantic segmentation network for PCB welding defect detection. This network includes two encoders sharing weighted values, a decoder, and some correlation modules, in which the decoder integrates deep features from two decoders with their feature difference computed by some correlation modules via skipping connections to recover spatial information on multiple output layers, and thus this proposed network can perform PCB welding small defect semantic segmentation. Moreover, via these correlation modules, this proposed network can pay more attention to semantic difference and further alleviate the over-fitting issue because of insufficient defect samples. Finally, we propose a combined loss function which combines the weighted cross-entropy loss, the Lovasz softmax loss, and the weighted precision&#x2013;recall loss for network training to further improve small defect segmentation and recall improvement. Experimental results demonstrate that the proposed network can be trained on limited training images and achieve high efficiency and outstanding effects for PCB welding small defect segmentation.

BreastNet18: A High Accuracy Fine-Tuned VGG16 Model Evaluated Using Ablation Study for Diagnosing Breast Cancer from Enhanced Mammography Images.

Identification and treatment of breast cancer at an early stage can reduce mortality. Currently, mammography is the most widely used effective imaging technique in breast cancer detection. However, an erroneous mammogram based interpretation may result in false diagnosis rate, as distinguishing cancerous masses from adjacent tissue is often complex and error-prone. Six pre-trained and fine-tuned deep CNN architectures: VGG16, VGG19, MobileNetV2, ResNet50, DenseNet201, and InceptionV3 are evaluated to determine which model yields the best performance. We propose a BreastNet18 model using VGG16 as foundational base, since VGG16 performs with the highest accuracy. An ablation study is performed on BreastNet18, to evaluate its robustness and achieve the highest possible accuracy. Various image processing techniques with suitable parameter values are employed to remove artefacts and increase the image quality. A total dataset of 1442 preprocessed mammograms was augmented using seven augmentation techniques, resulting in a dataset of 11,536 images. To investigate possible overfitting issues, a k-fold cross validation is carried out. The model was then tested on noisy mammograms to evaluate its robustness. Results were compared with previous studies. Proposed BreastNet18 model performed best with a training accuracy of 96.72%, a validating accuracy of 97.91%, and a test accuracy of 98.02%. In contrast to this, VGGNet19 yielded test accuracy of 96.24%, MobileNetV2 77.84%, ResNet50 79.98%, DenseNet201 86.92%, and InceptionV3 76.87%. Our proposed approach based on image processing, transfer learning, fine-tuning, and ablation study has demonstrated a high correct breast cancer classification while dealing with a limited number of complex medical images.

Context-based sentiment analysis on customer reviews using machine learning linear models.

Customer satisfaction and their positive sentiments are some of the various goals for successful companies. However, analyzing customer reviews to predict accurate sentiments have been proven to be challenging and time-consuming due to high volumes of collected data from various sources. Several researchers approach this with algorithms, methods, and models. These include machine learning and deep learning (DL) methods, unigram and skip-gram based algorithms, as well as the Artificial Neural Network (ANN) and bag-of-word (BOW) regression model. Studies and research have revealed incoherence in polarity, model overfitting and performance issues, as well as high cost in data processing. This experiment was conducted to solve these revealing issues, by building a high performance yet cost-effective model for predicting accurate sentiments from large datasets containing customer reviews. This model uses the fastText library from Facebook’s AI research (FAIR) Lab, as well as the traditional Linear Support Vector Machine (LSVM) to classify text and word embedding. Comparisons of this model were also done with the author’s a custom multi-layer Sentiment Analysis (SA) Bi-directional Long Short-Term Memory (SA-BLSTM) model. The proposed fastText model, based on results, obtains a higher accuracy of 90.71% as well as 20% in performance compared to LSVM and SA-BLSTM models.

An Efficient Handwritten Digit Recognition Based on Convolutional Neural Networks with Orthogonal Learning Strategies

The predictions of characters/text/digits from the handwritten images have made the research community spotlight towards recognition. There are enormous applications and ambiguity that made prediction possible with Deep Learning (DL) approaches. Primarily, there are four necessary steps to be carried out with handwriting prediction. First, consideration of a dataset that is more appropriate for DL validation an inefficient manner. Here, Special Database 1 and Special Database 2 are used, which are combined and modified by the National Institute of Standards and Technology (NIST). Next is pre-processing of input handwritten digit recognition data by data normalization, extraction of efficient features which provides better prediction accuracy. The proposed idea uses pixel values as features with the analysis of hyper-parameters to enhance near-human performance. With SVM, non-linear and linear models are built to extract the appropriate features for further processing. The features are separate and placed over the Bag of Features (BoF), which is used by the next processing stage. Finally, a novel Convolutional Neural Network (CNN) is by built modifying the network structure with Orthogonal Learning Particle Swarm Optimization (CNN-OLPSO). This modification is adopted for evolutionarily optimizing the number of hyper-parameters. This proposed optimizer predicts the optimal values from the fitness computation and shows better efficiency when compared to various other conventional approaches. The novelty which relies on CNN adoption is to endeavor a suitable path towards digitalization and preserve the handwritten structure and help automatic feature extraction using CNN by offering better computation accuracy. The optimization approach helps to avoid over-fitting and under-fitting issues. Here, metrics like accuracy, elapsed time, recall, precision, and [Formula: see text]-measure are evaluated. The results of CNN-OLPSO give better accuracy, reduced error rate and better execution time (s) compared to other existing methods. Thus, the proposed model shows better tradeoff in the recognition rate of handwritten digits.

HortNet417v1-A Deep-Learning Architecture for the Automatic Detection of Pot-Cultivated Peach Plant Water Stress.

The biggest challenge in the classification of plant water stress conditions is the similar appearance of different stress conditions. We introduce HortNet417v1 with 417 layers for rapid recognition, classification, and visualization of plant stress conditions, such as no stress, low stress, middle stress, high stress, and very high stress, in real time with higher accuracy and a lower computing condition. We evaluated the classification performance by training more than 50,632 augmented images and found that HortNet417v1 has 90.77% training, 90.52% cross validation, and 93.00% test accuracy without any overfitting issue, while other networks like Xception, ShuffleNet, and MobileNetv2 have an overfitting issue, although they achieved 100% training accuracy. This research will motivate and encourage the further use of deep learning techniques to automatically detect and classify plant stress conditions and provide farmers with the necessary information to manage irrigation practices in a timely manner.

Retracted: Efficient ensemble to combat flash attacks

AbstractFlash event generates enormous traffic and the cloud service providers use sustaining techniques like scaling and content delivery network to up their services. One of the main bottlenecks that the cloud service providers still find difficult to tackle is flash attacks. Illegitimate users send craftily designed packets to land up inside the server for wreaking havoc. As deep learning autoencoder has the potential to detect malicious traffic it has been used in this research study to develop an ensemble. Convolutional neural network is efficacious in overcoming the issue of overfitting; deep autoencoder is proficient in extracting features through dimensionality reduction. In order to obtain both these advantages it was decided to develop an ensemble keeping denoising autoencoder as the core element. The process of addressing a flash attack requires first detecting the presence of bot in malicious traffic, second studying its nature by observing its behavioral manifestations. Detection of botnet was achieved by three ensembles, namely, DAE_CNN, DAE_MLP, and DAE_XGB. But capturing its external manifested behavior is challenging, because the bot signatures are always in a state of flux. The simulated empirical study yielded an appreciable outcome. Its accuracy rate was 99.9% for all the three models and the false positive rates were 0, 0.006, and 0.001, respectively.

Balancing data for generalizable machine learning to predict glass-forming ability of ternary alloys

Machine Learning has thrived on the emergence of data-driven materials science. However, the materials datasets acquired at existing research efforts have significant imbalance issues. This paper investigated the data imbalance for the glass-forming ability of ternary alloy systems, which consists of abundant, low-fidelity high-throughput data, and sparse, high-fidelity traditional experimental data. We demonstrated a new method to handle the data imbalance and trained artificial neural network (ANN) models on the original vs. balanced datasets. The ANN model trained on the balanced dataset solved the overfitting issue suffered by the model trained on the original dataset. More importantly, the generalizability in predicting the new alloy system was improved in the data-balanced model, evidenced by the leave-one-alloy-system-out validation. Our work highlights the importance of handling data imbalance in material datasets to solve the overfitting issues of machine learning models and further enhance generalizability in predicting the characteristics of the new material systems.

Enhanced Deep Hierarchical Long Short-Term Memory and Bidirectional Long Short-Term Memory for Tamil Emotional Speech Recognition using Data Augmentation and Spatial Features

Neural networks have become increasingly popular for language modelling and within these large and deep models, overfitting, and gradient remains an important problem that heavily influences the model performance. As long short-term memory (LSTM) and bidirectional long short-term memory (BILSTM) individually solve long-term dependencies in sequential data, the combination of both LSTM and BILSTM in hierarchical gives added reliability to minimise the gradient, overfitting, and long learning issues. Hence, this paper presents four different architectures such as the Enhanced Deep Hierarchal LSTM &amp; BILSTM (EDHLB), EDHBL, EDHLL &amp; EDHBB has been developed. The experimental evaluation of a deep hierarchical network with spatial and temporal features selects good results for four different models. The average accuracy of EDHLB is 92.12%, EDHBL is 93.13, EDHLL is 94.14% &amp; EDHBB is 93.19% and the accuracy level obtained for the basic models such as the LSTM, which is 74% and BILSTM, which is 77%. By evaluating all the models, EDHBL performs better than other models, with an average efficiency of 94.14% and a good accuracy rate of 95.7%. Moreover, the accuracy for the collected Tamil emotional dataset, such as happiness, fear, anger, sadness, and neutral emotions indicates 100% accuracy in a cross-fold matrix. Emotions such as disgust show around 80% efficiency. Lastly, boredom shows 75% accuracy. Moreover, the training time and evaluation time utilised by EDHBL is less when compared with the other models. Therefore, the experimental analysis shows EDHBL as superior to the other models on the collected Tamil emotional dataset. When compared with the basic models, it has attained 20% more efficiency.