Under-fitting Problem Research Articles

Inception v3 based cervical cell classification combined with artificially extracted features

Traditional cell classification methods generally extract multiple features of the cell manually. Moreover, the simple use of artificial feature extraction methods has low universality. For example, it is unsuitable for cervical cell recognition because of the complexity of the cervical cell texture and the large individual differences between cells. Using the convolutional neural network classification method is a good way to solve this problem. However, although the cell features can be extracted automatically, the cervical cell domain knowledge will be lost, and the corresponding features of different cell types will be missing; hence, the classification effect is not sufficiently accurate. Aiming at addressing the limitations of the two mentioned classification methods, this paper proposes a cell classification algorithm that combines Inception v3 and artificial features, which effectively improves the accuracy of cervical cell recognition. In addition, to address the under-fitting problem and carry out effective deep learning training with a relatively small amount of medical data, this paper inherits the strong learning ability from transfer learning, and achieves accurate and effective cervical cell image classification based on the Herlev dataset. Using this method, an accuracy of more than 98% is achieved, providing an effective framework for computer aided diagnosis of cervical cancer. The proposed algorithm has good universality, low complexity, and high accuracy, rendering it suitable for further extension and application to the classification of other types of cancer cells.

On Training Neural Network Decoders of Rate Compatible Polar Codes via Transfer Learning.

Neural network decoders (NNDs) for rate-compatible polar codes are studied in this paper. We consider a family of rate-compatible polar codes which are constructed from a single polar coding sequence as defined by 5G new radios. We propose a transfer learning technique for training multiple NNDs of the rate-compatible polar codes utilizing their inclusion property. The trained NND for a low rate code is taken as the initial state of NND training for the next smallest rate code. The proposed method provides quicker training as compared to separate learning of the NNDs according to numerical results. We additionally show that an underfitting problem of NND training due to low model complexity can be solved by transfer learning techniques.

Understanding the Disharmony between Weight Normalization Family and Weight Decay

The merits of fast convergence and potentially better performance of the weight normalization family have drawn increasing attention in recent years. These methods use standardization or normalization that changes the weight W to W′, which makes W′ independent to the magnitude of W. Surprisingly, W must be decayed during gradient descent, otherwise we will observe a severe under-fitting problem, which is very counter-intuitive since weight decay is widely known to prevent deep networks from over-fitting. Moreover, if we substitute (e.g., weight normalization) W′ = W∥W∥ in the original loss function ∑i L(ƒ(xi; W′),yi) + ½λ∥W′∥2, it is observed that the regularization term ½λ∥W′∥2 will be canceled as a constant ½ λ in the optimization objective. Therefore, to decay W, we need to explicitly append: ½λ∥W∥2. In this paper, we theoretically prove that ½λ∥W∥2 improves optimization only by modulating the effective learning rate and fairly has no influence on generalization when the weight normalization family is compositely employed. Furthermore, we also expose several serious problems when introducing weight decay term to weight normalization family, including the missing of global minimum, training instability and sensitivity of initialization. To address these problems, we propose an Adaptive Weight Shrink (AWS) scheme, which gradually shrinks the weights during optimization by a dynamic coefficient proportional to the magnitude of the parameter. This simple yet effective method appropriately controls the effective learning rate, which significantly improves the training stability and makes optimization more robust to initialization.

Variational Autoencoder With Optimizing Gaussian Mixture Model Priors

The latent variable prior of the variational autoencoder (VAE) often utilizes a standard Gaussian distribution because of the convenience in calculation, but has an underfitting problem. This paper proposes a variational autoencoder with optimizing Gaussian mixture model priors. This method utilizes a Gaussian mixture model to construct prior distribution, and utilizes the Kullback-Leibler (KL) distance between posterior and prior distribution to implement an iterative optimization of the prior distribution based on the data. The greedy algorithm is used to solve the KL distance for defining the approximate variational lower bound solution of the loss function, and for realizing the VAE with optimizing Gaussian mixture model priors. Compared with the standard VAE method, the proposed method obtains state-of-the-art results on MNIST, Omniglot, and Frey Face datasets, which shows that the VAE with optimizing Gaussian mixture model priors can learn a better model.

Energy Consumption Prediction of Fused Deposition 3D Printer Based on Improved Regularized BP Neural Network

An energy consumption prediction method based on process parameters and neural network was proposed to study the inherent energy consumption characteristics of open source melt deposition 3D printer and improve energy efficiency. An improved regularized network is used for optimization to avoid over-fitting and under-fitting problems. The orthogonal test sample data were trained in MATLAB environment, and the predictive model between process parameters and energy consumption of open source melt deposition 3D printing was established. The energy consumption prediction results of BP networks and regularized networks are compared by analyzing convergence curves and network training charts. The results show that the BP network training has experienced over-fitting, resulting in a prediction energy consumption error of about 10%. The improved regularization method effectively avoids the over-fitting phenomenon and the error of energy consumption prediction is about 1%. It can effectively improve the calculation efficiency of energy consumption prediction, which shows the accuracy of this method in energy consumption prediction.

An Automated Diagnostic System for Heart Disease Prediction Based on ${\chi^{2}}$ Statistical Model and Optimally Configured Deep Neural Network

Different automated decision support systems based on artificial neural network (ANN) have been widely proposed for the detection of heart disease in previous studies. However, most of these techniques focus on the preprocessing of features only. In this paper, we focus on both, i.e., refinement of features and elimination of the problems posed by the predictive model, i.e., the problems of underfitting and overfitting. By avoiding the model from overfitting and underfitting, it can show good performance on both the datasets, i.e., training data and testing data. Inappropriate network configuration and irrelevant features often result in overfitting the training data. To eliminate irrelevant features, we propose to use $\chi ^{2}$ statistical model while the optimally configured deep neural network (DNN) is searched by using exhaustive search strategy. The strength of the proposed hybrid model named $\chi ^{2}$ -DNN is evaluated by comparing its performance with conventional ANN and DNN models, another state of the art machine learning models and previously reported methods for heart disease prediction. The proposed model achieves the prediction accuracy of 93.33%. The obtained results are promising compared to the previously reported methods. The findings of the study suggest that the proposed diagnostic system can be used by physicians to accurately predict heart disease.

A CNN Model for Semantic Person Part Segmentation with Capacity Optimization.

In this paper, a deep learning model with an optimal capacity is proposed to improve the performance of person part segmentation. Previous efforts in optimizing the capacity of a CNN model suffer from a lack of large datasets as well as the over-dependence on a single-modality CNN which is not effective in learning. We make several efforts in addressing these problems. Firstly, other datasets are utilized to train a CNN module for pre-processing image data and a segmentation performance improvement is achieved without a time-consuming annotation process. Secondly, we propose a novel way of integrating two complementary modules to enrich the feature representations for more reliable inferences. Thirdly, the factors to determine the capacity of a CNN model are studied and two novel methods are proposed to adjust (optimize) the capacity of a CNN to match it to the complexity of a task. The over-fitting and under-fitting problems are eased by using our methods. Experimental results show that our model outperforms the state-of-the-art deep learning models with a better generalization ability and a lower computational complexity.

Nonuniformity Correction of Single Infrared Images Based on Deep Filter Neural Network

The fixed-pattern noise (FPN) caused by nonuniform optoelectronic response limits the sensitivity of an infrared imaging system and severely reduces the image quality. Therefore, nonuniform correction of infrared images is very important. In this paper, we propose a deep filter neural network to solve the problems of network underfitting and complex training with convolutional neural network (CNN) applications in nonuniform correction. Our work is mainly based on the idea of deep learning, where the nonuniform image noise features are fully learned from a large number of simulated training images. The network is designed by introducing the filter and the subtraction structure. The background interference of the image is removed by the filter, so the learning model is gathered in the nonuniform noise. The subtraction structure is used to further reduce the input-to-output mapping range, which effectively simplifies the training process. The results from the test on infrared images shows that our algorithm is superior to the state-of-the-art algorithm in visual effects and quantitative measurements, providing a new method for deep learning in nonuniformity correction of single images.

An automatic and intelligent optimal surface modeling method for composite tunnel structures

Automatically modeling and intelligently monitoring composite tunnel structures is of great significance considering the development of composite and diverse tunnel construction materials. Therefore, how to efficiently analyze the deformation of all kinds of tunnel structures with its expanding application is becoming increasingly important. In this paper, we analyze the over- and under-fitting problems under different point cloud profiles to generate an automatic and robust B-spline model which is a suitable method for high accuracy approximation capable of optimizing the composite tunnel model by adjusting the parameters. The innovation of this research is that we combine the maximum likelihood function and the degree of freedom to obtain the optimal parameters model of composite tunnels, which are validated by the various profiles.

Variational Bayesian Learning for Dirichlet Process Mixture of Inverted Dirichlet Distributions in Non-Gaussian Image Feature Modeling.

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations.

Determination of Adequate Hidden Neurons in Combo Neural Network Using New Formulation and Fine Tuning with IMGWOA for Enrich Wind-Speed Forecasting

This article proposes a new formulation to choose the amount of hidden neurons in combo neural networks, and fine tuning is done by improved modified grey wolf optimisation algorithm (IMGWOA) for wind-speed forecasting. The proposed combo neural network-based wind-speed forecasting model designed by aggregating the forecast values from multiple neural network models such as an ELMAN, recursive radial basis function, multilayer perceptron and improved back propagation network to get the most exact wind-speed forecasting with minimal error. Furthermore, combo neural network performance is enriched by means of fine tuning with IMGWOA. The over-fitting or under-fitting problem is caused because of the random choice of hidden neurons in artificial neural networks. This article presents the solution for both-either over-fitting or under-fitting problems using new formulation. Accurate amount of hidden neurons is determined based on 151 different formulations; these formulations are checked by means of the mean square error. The perfect designing of combo neural network using the new formulation is justified based on the convergence theorem. Proposed formulation performance is compared with other approaches for the determination of hidden neurons in neural networks. The result confirms that the suggested formulations are compact with reduced complexity, effective for determination of the hidden neurons in the combo neural network which minimises error to the least, and accuracy is enriched by the optimisation algorithm (IMGWOA), applied for wind-speed forecasting application.

A dual estimate method for aeromagnetic compensation

Scalar aeromagnetic surveys have played a vital role in prospecting. However, before analysis of the surveys’ aeromagnetic data is possible, the aircraft’s magnetic interference should be removed. The extensively adopted linear model for aeromagnetic compensation is computationally efficient but faces an underfitting problem. On the other hand, the neural model proposed by Williams is more powerful at fitting but always suffers from an overfitting problem. This paper starts off with an analysis of these two models and then proposes a dual estimate method to combine them together to improve accuracy. This method is based on an unscented Kalman filter, but a gradient descent method is implemented over the iteration so that the parameters of the linear model are adjustable during flight. The noise caused by the neural model’s overfitting problem is suppressed by introducing an observation noise.

Solving the Under-Fitting Problem for Decision Tree Algorithms by Incremental Swarm Optimization in Rare-Event Healthcare Classification

Solving the Under-Fitting Problem for Decision Tree Algorithms by Incremental Swarm Optimization in Rare-Event Healthcare Classification

Development of robust neighbor embedding based super-resolution scheme

In this paper, we propose a robust neighbor embedding super-resolution (RNESR) scheme to generate a super-resolution (SR) image from a single low-resolution (LR) image. It utilizes histogram matching for selection of best training pair of images. This helps to learn co-occurrence prior to high-resolution (HR) image reconstruction. The global neighborhood size is computed from local neighborhood size, which avoids the over-fitting and under-fitting problem during neighbor embedding. Robust locally linear embedding (RLLE) is used in place of locally linear embedding (LLE) to generate HR image. To validate the scheme, exhaustive simulation has been carried out on standard images. Comparative analysis with respect to different measures like peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and feature similarity index (FSIM) reveals that the RNESR scheme generates high-quality SR image from a LR image as compared to existing schemes.

Locally weighted linear regression for cross-lingual valence-arousal prediction of affective words

Dimensional sentiment analysis that aims to predict a continuous numerical value on multiple dimensions, such as the valence-arousal (VA) space, has attracted more attention in recent years. Compared to the categorical approach that focuses on sentiment classification such as binary classification (positive and negative), the dimensional approach can provide more fine-grained sentiment analysis. Therefore, recent studies have investigated the automatic development of affective lexicons with VA ratings because such resources are fundamental and useful for building dimensional sentiment applications. Due to the limited number of VA lexicons, a cross-lingual approach has emerged that aims to estimate the VA ratings of affective words of one language from those of another language based on linear regression or other regression methods. However, one of the major limitations of linear regression is the under-fitting problem which can cause a poor fit between the algorithm and the training data. To tackle this problem, this study proposes a locally weighted method to improve linear regression for predicting the valence-arousal values of affective words. This method performs a regression around the point of interest using only training data that are “local” to that point, and thus can reduce the impact of noise from unrelated training data. Experimental results show that the proposed method achieved a lower error rate and a higher correlation coefficient for predicting the VA ratings of Chinese affective words from English VA lexicons.

An Interaction Prediction Model of Monitoring Node Based on Observational Learning

Data acquisition anomalies often occur in remote monitoring. In this paper, a software solution is presented to discover the abnormal monitoring node and predict the monitored data, rather than using hardware maintenance. Firstly, by analyzing the distribution characteristics of the monitoring data from each node, the highly correlated nodes of the abnormal node are selected. Then, an integrated BP neural network is applied to build an observational learning model, which can give interactive predictions for the abnormal node. To solve the under-fitting problem caused by small samples and improve the generalization performance of the model, we propose a new observational learning algorithm, in which the weights are calculated using the mean squared error (MSE) of learners on test set. Experiments conducted on the airport noise data set show that the proposed model has satisfying predictive ability, and the improved observational learning algorithm is more stable and effective than the traditional observational learning algorithm.

Variational learning for Dirichlet process mixtures of Dirichlet distributions and applications

In this paper, we propose a Bayesian nonparametric approach for modeling and selection based on a mixture of Dirichlet processes with Dirichlet distributions, which can also be seen as an infinite Dirichlet mixture model. The proposed model uses a stick-breaking representation and is learned by a variational inference method. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the obstacle of estimating the correct number of clusters is sidestepped by assuming an infinite number of clusters. Compared to other approximation techniques, such as Markov chain Monte Carlo (MCMC), which require high computational cost and whose convergence is difficult to diagnose, the whole inference process in the proposed variational learning framework is analytically tractable with closed-form solutions. Additionally, the proposed infinite Dirichlet mixture model with variational learning requires only a modest amount of computational power which makes it suitable to large applications. The effectiveness of our model is experimentally investigated through both synthetic data sets and challenging real-life multimedia applications namely image spam filtering and human action videos categorization.

An Improved Neighbor Embedding Method to Super-resolution Reconstruction of a Single Image

In this paper, an improved neighbor embedding super-resolution method is proposed by offering two extensions. Firstly, an improved neighbor embedding algorithm is presented to overcome over- or under-fitting problem existing in the neighbor embedding algorithm of [7]. In our algorithm, the local geometry of each image patch is characterized by the reconstruction weights with which the patch is reconstructed from all the neighbors in the neighborhood which depends on an empirical threshold. Thus, each image patch is reconstructed from all the neighbors in the neighborhood rather than k nearest neighbors. Secondly, a global reconstruction constraint is incorporated into super-resolution reconstruction process to make the degraded version of target high-resolution image consistent with measured low-resolution image. Experiments show that our method yields better results.

Uncertainty assessment and ensemble flood forecasting using bootstrap based artificial neural networks (BANNs)

A reliable hydrologic prediction is essential for planning, designing and management activities of water resources. This study quantifies the parametric uncertainty involved in flood forecasting using artificial neural network (ANN) models. Hourly water level forecasting models are developed and uncertainty assessment is carried out for hourly water level forecasting. Hourly water level data of five upstream gauging stations of a large river basin are considered for hourly water level forecasting. The uncertainty associated with hourly flood forecast is investigated using the bootstrap based artificial neural networks (BANNs). Ensemble prediction is made by averaging the output of member bootstrapped neural networks. Results obtained indicate that BANN-hydrologic forecasting models with confidence bounds can improve their reliability for flood forecasts. It is illustrated that the confidence intervals based on BANNs are capable of quantifying the parametric uncertainty for short as well as for long lead time forecasts. Study shows the ensemble prediction is more consistent and reproducible. The study also analyzes the effect of length of training datasets and performance of split sample validation in BANNs modeling. The results illustrate that short length of training datasets with appropriate representation can perform similar to models with long length training datasets. The results also illustrate that the bootstrap technique is capable of solving the problems of over-fitting and underfitting during training of BANN models as the results without cross validation show similar performance compared to results obtained using cross validation technique.

Assessing the relative importance of surface ozone influential variables in regional-scale analysis

The multilayer perceptron (MLP) model is frequently used to assess the relative importance (RI) of surface ozone influential variables when they are used to investigate ozone variation mechanisms. Previous studies, however, suffer from two limitations: 1) indentifying or searching the optimal MLP topology to avoid a biased RI assessment inevitably incurs a heavy computational burden, and 2) the model is suitable only for local-scale analysis of ozone variation mechanisms. To tackle both problems, we selected three typical air-quality monitoring sites in Hong Kong as our study targets, as the ozone variations at these sites are inhomogeneously affected by regional and local factors. An MLP model trained by automatic relevance determination (referred to as MLP-ARD), a Bayesian MLP approach, was employed to assess the RI for both regional and local ozone influential variables. The results indicated the following remarks: 1) The MLP-ARD model, due to its high degree of resistance to both the over-fitting and the under-fitting problems, is exempt from identifying or searching the optimal MLP topology when used to obtain an unbiased RI assessment and thus avoid the heavy computational burden. Furthermore, the RI assessment results obtained with the MLP-ARD method are comparable to those of the best assessment method in the literature. Based on these findings, decision-makers can scientifically promulgate a site-specific air pollution control strategy site; 2) Regional-scale analysis of ozone variation is indispensable, as taking the key regional ozone influential variables into account significantly improves the prediction accuracy of the MLP-ARD model, especially on peak ozone days.