True Prediction Error Research Articles

Spatial error prediction method for precision machine tools considering the influence of repeatability error

A spatial error prediction method for precision machine tools considering the influence of repeatability errors is proposed in this paper. Firstly, the reciprocal of the repeatability error is used to evaluate the measurement data confidence to distinguish the difference of useful information contained in the measurement data of different positions. Secondly, the proposed confidence index is used to weight the measured values of PDGEs at different positions, so that the data containing more useful information can play a full role in the error prediction, and the predicted error is substituted into the error model to predict the spatial error. Then, the computer simulation based on statistical analysis is designed, the PDGEs and spatial error prediction accuracy of the weighted algorithm are verified by simulation. Finally, a multi-instrument fusion spatial error measurement scheme is proposed, and the accuracy and reliability of the model are verified on a horizontal machining center prototype.

Integration of physics-based data in deep learning model training for predicting the effect of sulfur content in the directed energy deposition process

The training of a machine learning model solely on experimental data, encompassing both pre- and post-process information, can reveal the general relationship of the directed energy deposition process. However, models trained in this manner encounter limitations in capturing critical in-process information occurring during deposition. This paper details the training of a deep learning model through the integration of in-process physics-based simulation information and a pre-process experiment dataset. The sulfur content of stainless steel 316L was selected as critical in-process information affecting the final track geometry and was captured using computational fluid dynamics simulation of a single-track deposition process, which cannot be captured accurately through experimentation. The physics-based simulation dataset was generated by obtaining the contour of deposition and dilution of the solidified track cross-section. The experiment was conducted using central composite design, and data augmentation was achieved through curve fitting using a response surface methodology regression model. Statistical analysis assessing the quality of simulation and experiment data was conducted. Among six baseline models, a deep learning model with a specified training sequence of experiment and simulation data, denoted as DL-AugExp-Sim-Exp, exhibited the best-performing R2 and root mean square error prediction accuracy for cross-section track shape. Notably, deep learning models trained with both experiment and simulation information demonstrated a lower R2 value compared to models trained solely with experiment data, revealing a tradeoff between R2 value and additional prediction capability. In summary, in this study, the integration of a physics-based simulation dataset demonstrated the additional prediction capability concerning the effect of sulfur content on track geometry.

Deep Regression Network with Prediction Confidence in Time Series Application for Asset Health Estimation

Many works have been focused in developing detection, monitoring and prediction routines for asset health estimation system. Classic machine learning based models benefit from quality of physics-informed features available from domain knowledge. This, however, can be labor intensive and is limited by quality of features developed through available knowledge. Deep learning based approach, if successful, can alleviate this laborious step. On the other hand, users often need to decide whether or not to trust an algorithmic prediction while the true error in the prediction is unknown. In this work, we propose a deep learning based regression network that output both prediction value and confidence score for asset health estimation in short intermittent transients time series application. In the experimental study, we show that our model has low prediction error given short intermittent transients multivariate time series as input. Furthermore, our model also provides a confidence score for each prediction that is highly negatively correlated with true prediction error. Experiments show that by setting an acceptance threshold on confidence score, our model can reach an averaged improvement of 20% on the prediction quality with 90% coverage.

Prediction and model evaluation for space–time data

ABSTRACT Evaluation metrics for prediction error, model selection and model averaging on space–time data are understudied and poorly understood. The absence of independent replication makes prediction ambiguous as a concept and renders evaluation procedures developed for independent data inappropriate for most space–time prediction problems. Motivated by air pollution data collected during California wildfires in 2008, this manuscript attempts a formalization of the true prediction error associated with spatial interpolation. We investigate a variety of cross-validation (CV) procedures employing both simulations and case studies to provide insight into the nature of the estimand targeted by alternative data partition strategies. Consistent with recent best practice, we find that location-based cross-validation is appropriate for estimating spatial interpolation error as in our analysis of the California wildfire data. Interestingly, commonly held notions of bias-variance trade-off of CV fold size do not trivially apply to dependent data, and we recommend leave-one-location-out (LOLO) CV as the preferred prediction error metric for spatial interpolation.

Combining seeded region growing and k-nearest neighbours for the segmentation of routinely acquired spatio-temporal image data

PurposeThe acquisition conditions of medical imaging are often precisely defined, leading to a high homogeneity among different data sets. Nonetheless, outliers or artefacts still appear and need to be reliably detected to ensure a reliable diagnosis. Thus, the algorithms need to handle small sample sizes especially, when working with domain specific imaging modalities.MethodsIn this work, we suggest a pipeline for the detection and segmentation of light pollution in near-infrared fluorescence optical imaging (NIR-FOI), based on a small sample size. NIR-FOI produces spatio-temporal data with two spatial and one temporal dimension. To calculate a two-dimensional light pollution map for the entire image stack, we combine region growing and k-nearest neighbours (kNN), which classifies pixels into fore- and background by its entire temporal component. Thus, decision-making on reduced data is omitted.ResultsWe achieved a F_1 score of 0.99 for classifying a data set as light polluted or pollution-free. Additionally, we reached a total F_1 score of 0.90 for detecting regions of interest within the polluted data sets. Finally, an average Dice’s coefficient measuring the segmentation performance over all polluted data sets of 0.80 was accomplished.ConclusionsA Dice’s coefficient of 0.80 for the area segmentation does not seem perfect. However, there are two main factors, besides true prediction errors, lowering the score: Segmentation mistakes on small areas lead to a rapid decrease in the score and labelling errors due to complex data. However, in combination with the light-polluted data set and pollution area detection, these results can be considered successful and play a key role in our general goal: Exploiting NIR-FOI for the early detection of arthritis within hand joints.

Thermal error compensation method of truss robot beam structure based on mechanism and data drive

As the main supporting component of the truss robot, the thermal deformation of the beam often has a great influence on the overall thermal error of the truss robot due to its large span. In order to improve the thermal error prediction accuracy of long-span truss robot, a thermal error prediction method based on multiple linear regression and long short-term memory network is proposed based on mechanism and data drive. Firstly, the multiple linear regression model is used to predict the thermal error, and the prediction error data processing. Secondly, the long short-term memory network is established. In order to improve the performance of the long short-term memory network more effectively, an improved particle swarm optimization algorithm is proposed to optimize the hyper-parameters of the long short-term memory network. Finally, the improved particle swarm optimization–long short-term memory network is used to correct the prediction error of the multiple linear regression model. The experimental results show that the combined thermal error prediction model based on multiple linear regression and improved particle swarm optimization–long short-term memory algorithm has higher prediction accuracy than multiple linear regression model and long short-term memory network. The method has stable prediction accuracy and can provide a basis for thermal error compensation.

A detailed kinetic model for aromatics formation from small hydrocarbon and gasoline surrogate fuel combustion

This work develops a detailed chemical kinetic model for Polycyclic Aromatic Hydrocarbon (PAH) chemistry that builds on a chemical model for gasoline surrogates by Cai et al. [Proc. Combust. Inst. 37 (2019) 639–647]. Informed by ab-initio and experimental studies, the model development emphasizes the prediction of soot precursors starting from C3H4 isomers up to the size of acepyrene. The proposed model was validated against experimental measurements from 79 publications, including ignition delay times, laminar burning velocities, and speciation data for several fuels in various conditions. A validation against measured peak mole fractions from 31 counterflow flames demonstrates an average prediction error of aromatic species up to the size of acenaphthalene below a factor of three. Additionally, the effects of experimental uncertainty are quantitatively discussed by computing confidence intervals for the true model prediction error. The formation of two-ring aromatic species is often still considered poorly understood. Accordingly, special attention is given to the formation of indene and naphthalene. Reaction flux analyses based on rigorously defined species selection criteria reveal that indene formation pathways are initiated by reactions of phenyl and benzyl and described by the reaction kinetics from theoretical calculations. In contrast, naphthalene formation is dominated by more uncertain pathways relying on estimated rates that are initiated by reactions of benzene and fulvenallenyl. Five-member rings on the periphery of PAH species and soot particles play a crucial role in recent experimental and theoretical studies. Therefore, this work investigates acenaphthene and other C12H8 isomers. The most abundant C12H8 isomers are acenaphthene, 1-ethynylnaphthalene, and 2-ethynylnaphthalene, and their respective concentrations are predicted to be similar in many flames. Furthermore, reaction flux analyses show that their formation is dominated by HACA routes starting from naphthalene.

A medium-scale study of using machine learning fusion to improve TLE prediction precision without external information

This paper presents a machine learning (ML) approach that can improve orbit prediction precision and reduce uncertainties with a properly designed fusion strategy. The ML models are trained based on historical TLE sets and then ML-modifications are generated for the future states and uncertainty predictions. The ML-modification is first interpreted as a pseudo-measurement of the true orbit prediction error and then fused with the conventional orbital state and uncertainty propagation result. The regularized particle filter (PF) with progressive correction and systematic sampling is adopted for uncertainty propagation while using the Simplified General Perturbations #4 (SGP4) model. Comprehensive experiments are conducted and analyzed on over 100 resident space objects (RSOs) in different orbit types. The results demonstrate that the fusion strategy enables the ML approach to work with the advanced PF prediction method. The prediction precision can be significantly improved for the majority of the cases. The prediction accuracy is observed to be improved in some cases but without a clear pattern. The results and analysis also suggest that future studies of a monitoring system facilitating the fusion process would be helpful in practical applications.

Improvement of human error prediction accuracy in single-trial analysis of electroencephalogram.

The prevention of human error is an important task that has already been researched. Previous studies have shown that EEG signals can predict the occurrence of human errors. However, high accuracy has not yet been achieved in a single-trial analysis. This study is aimed to improve the accuracy of single-trial analysis, and propose a method for anomaly detection with auto encoder(AE). In the experiment, we conducted "Press the button(Go)" or "Do nothing(No-Go)" according to the visual stimulus and analyzed the EEG signal from -1000 ms to 0 ms when the stimulus was displayed. We prepared two types of inputs, time series data and frequency spectrum, and an AE was trained to reconstruct the inputs. We then calculated the difference between the reconstructed data and input data and predicted human error by its largeness. In the prediction using Support Vector Machine (SVM) based on the frequency spectrum, some over-fitting occurred and the average accuracy was 43 %. In the prediction using anomaly detection with frequency spectrum was 53 % and could not be classified. The time series data was 63 % which improved the accuracy. A previous study has shown frequency-dependent features such as -band activity and rhythm, as precursors of human error. However, in single-trial analysis, we obtained a higher accuracy by time series data than when by using the frequency spectrum. However, there was no noticeable difference between SVM and anomaly detection methods other than over-fitting. Therefore, in this case, the improvement in accuracy by the anomaly detection method could not be confirmed. However, the result suggests that it is more effective to use the frequency spectrum than the time series data in the single-trial analysis in the future.

A Guide for More Accurate and Precise Estimations in Simulative Unidimensional IRT Models

There is a great deal of research about item response theory (IRT) conducted by simulations. Item and ability parameters are estimated with varying numbers of replications under different test conditions. However, it is not clear what the appropriate number of replications should be. The aim of the current study is to develop guidelines for the adequate number of replications in conducting Monte Carlo simulation studies involving unidimensional IRT models. For this aim, 192 simulation conditions which included four sample sizes, two test lengths, eight replication numbers, and unidimensional IRT models were generated. Accuracy and precision of item and ability parameter estimations and model fit values were evaluated by considering the number of replications. In this context, for the item and ability parameters; mean error, root mean square error, standard error of estimates, and for model fit; M_2, 〖RMSEA〗_2, and Type I error rates were considered. The number of replications did not seem to influence the model fit, it was decisive in Type I error inflation and error prediction accuracy for all IRT models. It was concluded that to get more accurate results, the number of replications should be at least 625 in terms of accuracy of the Type I error rate estimation for all IRT models. Also, 156 replications and above can be recommended. Item parameter biases were examined, and the largest bias values were obtained from the 3PL model. It can be concluded that the increase in the number of parameters estimated by the model resulted in more biased estimates.

Tourist Demand Prediction Model Based on Improved Fruit Fly Algorithm

To accurately predict the development and change trend of the future, tourism market can effectively improve the planning and purpose of tourism development. In order to improve the accuracy of tourist demand prediction, this paper studies the tourist demand prediction model based on improved fruit fly algorithm. Aiming at the optimization defects of the traditional fruit fly optimization algorithm (FOA), the model introduces two concepts of sensitivity and pheromone, improves the optimization strategy and position replacement of fruit fly, improves the diversity of fruit fly population, modifies the global optimization characteristics of the algorithm, and improves the local search ability and search efficiency of the algorithm. By combining the improved AFOA with echo state network (ESN), a two-stage combined prediction model (AAFOA-ESN) is constructed. The experimental results show that the minimum prediction error accuracy of the model is only 0.55%, which has more robust prediction effect, faster convergence speed, and higher prediction accuracy.

Comparative assessment of PEE methods and new performance measurement for RDH

The paper mainly analyzes four typical reversible data hiding (RDH) algorithms that use PEE technology in combination: Conventional Prediction-error-expansion(C-PEE), Adaptive Prediction-error-expansion(A-PEE), Pairwise Prediction-error-expansion (P-PEE), Improve Pairwise Prediction-error-expansion(IP-PEE). And the experimental comparison results are also given. In addition, in the research process, we found that the peak signal-to-noise ratio (PSNR) and embedding capacity (EC) of many algorithms are very close, due to lack of benchmark tools for technical evaluation in the RDH field, thus this paper proposes a new performance measurement method based on the PSNR and EC measurement, we call it the prediction error accuracy rate, Determined by the ratio of the capacity that meets the embedding conditions to the total capacity of the image, combined with the measurement of this method, can better determine whether the performance of the algorithm is optimal. Finally, in this area, challenge and future directions are also given.

Full characterization of the peak-locking error by means of orthogonal functions and application to the flow behind a helicopter fuselage model

This work offers a 1D fully analytic complete modeling of the peak-locking error. Besides error assessment, this model allows for measurement correction. The paradigm of peak locking is revisited. The expressions of the functional dependence of the error are obtained for the two most relevant components of the measurement statistics: (1) the local velocity average, and (2) the local velocity rms. The prediction of the peak-locking error is related to complex interactions between the particle images size and shape and the measurement algorithms (peak fitting, image interpolation, etc.). The proposed model incorporates the different interactions by using a generalized approach that only requires the calibration of a few coefficients. This calibration is done by means of an inexpensive multiple Δt strategy, consisting in measuring the same flow field with different Δt subsets. Differently to previous empirical models from the authors, a robust theoretical foundation has been established. Besides providing rationale to the procedure, the results are significantly more accurate. A calibration using only three coefficients may correct up to 90% of the peak-locking error, while the previous empirical models would correct just in the order of 50%. This leaves a 10% of residual peak-locking error instead of 50%, reducing the final incidence by a factor of 5. The methodology is tested, in the context of turbulent flows, on real images corresponding to the measurement of the flow around the fuselage of a helicopter model. The test campaign was performed in the Italian Aerospace Research Centre (CIRA) CT-1 low speed wind tunnel. The proposed method, combined with the multiple Δt strategy, results in an error prediction accuracy that is good enough for correcting the measurements. Corrections to the average flow are in the order of 0.1 pixel (~ 3% of the convective velocity but in the order of 100% of the local flow variations). Corrections to the rms are in the same order; being the turbulence small, they cancel very large relative rms errors (~ 100%). The coherence between predicted error and actual error in real cases is further supported by the consistent matching that can be observed on their spatial patterns, associated with flow field statistics. An orthogonal expansion allows for full assessment of PIV peak-locking bias errors. The turbulent flow behind a helicopter fuselage model is used as test case. The expansion is calibrated and validated using a multiple Δt strategy. The resulting assessment is good enough to allow for correcting errors in the order of 0.1 pixel in the measured average as well as in the rms.

Sequential Depth Completion With Confidence Estimation for 3D Model Reconstruction

This letter addresses a depth-completion problem for sequential data to reconstruct 3D models of outdoor scenes. While many deep-learning-based approaches have recently achieved promising results, their results are not directly applicable to 3D modeling because of several reasons. First, most results contain a lot of outliers because of irregularly distributed sparse measurements. Second, they ignore temporal coherence in sequential frames and produce temporally inconsistent depths. Therefore, we propose a new method that predicts temporally consistent depths with corresponding confidences from sequential frames. The suggested method can efficiently remove the outliers based on confidence estimates, which accurately represent the true prediction errors. The method also produces temporally consistent depths by integrating the depth information of consecutive frames. In addition, we present a 3D-modeling system that reconstructs a globally consistent 3D model in real-time using the results from the proposed depth completion method. Extensive experiments on synthetic and real-world datasets show that our method outperforms the other state-of-the-art methods in terms of both depth-completion and 3D-modeling accuracies.

Prediction error after model search

Estimation of the prediction error of a linear estimation rule is difficult if the data analyst also uses data to select a set of variables and constructs the estimation rule using only the selected variables. In this work, we propose an asymptotically unbiased estimator for the prediction error after model search. Under some additional mild assumptions, we show that our estimator converges to the true prediction error in $L^{2}$ at the rate of $O(n^{-1/2})$, with $n$ being the number of data points. Our estimator applies to general selection procedures, not requiring analytical forms for the selection. The number of variables to select from can grow as an exponential factor of $n$, allowing applications in high-dimensional data. It also allows model misspecifications, not requiring linear underlying models. One application of our method is that it provides an estimator for the degrees of freedom for many discontinuous estimation rules like best subset selection or relaxed Lasso. Connection to Stein’s Unbiased Risk Estimator is discussed. We consider in-sample prediction errors in this work, with some extension to out-of-sample errors in low-dimensional, linear models. Examples such as best subset selection and relaxed Lasso are considered in simulations, where our estimator outperforms both $C_{p}$ and cross validation in various settings.

Dynamic temperature gradient and unfalsified control approach for machine tool thermal error compensation

In this work, a novel machine tool thermal error modeling method based on dynamic temperature gradient is proposed, and a thermal error compensation method based on unfalsified control is developed. The dynamic temperature gradient is used to optimize the locations of temperature measuring points on the machine tool. Real-time compensation for the thermal error can be achieved using the developed compensation method by establishing the correlations between dynamic temperature gradient and thermal error in the machine tool. Different from traditional model-based methods, the developed compensation approach does not rely on an accurate model of the thermal error but instead uses online input/output data to adaptively select the best controller at any moment, thereby improving thermal error prediction accuracy and robustness. The effectiveness of the developed thermal error compensation method is demonstrated on a turning center, where the spindle thermal error is compensated during the manufacturing of 120 inner bore parts and 120 shaft parts. After compensation using the proposed approach, thermal errors are reduced from 27 ώm to 9 ώm for the inner bore parts and from 31 ώm to 11 ώm for the shaft parts, respectively.

Thermal error modeling of machine tool spindle based on the improved algorithm optimized BP neural network

Thermal error of the machine tool spindle is one of the main factors affecting the machining accuracy. For the complex operating environment of the machine tool, the difficulty of thermal error prediction modeling, and the low accuracy of the traditional thermal error prediction model, a spindle thermal error prediction model based on the improved particle swarm optimization (IPSO) optimize back propagation (BP) neural network is established in this paper. The temperature measurement points are clustered by SOM neural network, and the correlation analysis method is used to explore the correlation between the thermal sensitive points and the thermal error of the spindle. The S-type function is used to improve the inertia weight coefficient of the IPSO algorithm so as to improve the particle optimization effect. IPSO is used to optimize the parameters of BP neural network, such as the initial weights and thresholds. Compared with the GA-BP prediction model, the modeling efficiency, robustness, and accuracy of IPSO-BP neural network prediction model are all superior GA-BP prediction model. Taking the thermal error of the electric spindle of precision CNC machining center as the research object, the intelligent temperature sensor and the laser displacement sensor are used to obtain the machine tool temperature values and the spindle thermal error values. The prediction accuracy of the GA-BP model for the spindle thermal error was 93.1%, and the prediction accuracy of the IPSO-BP model was 96.5%. The results show that the IPSO-BP model can accurately predict the thermal error of the spindle under different working conditions. The model can obtain higher thermal error prediction accuracy and is more suitable for the thermal error compensation model.

AIC for the group Lasso in generalized linear models

When covariates are assumed to be clustered in groups in regression problems, the group Lasso is useful, because it tends to drive all the weights in one group to zero together. The group Lasso includes a tuning parameter which controls a penalty level, and an unbiased estimator of the true prediction error is derived as a $$C_p$$ -type criterion to select an appropriate value of the tuning parameter. However, in general, the $$C_p$$ -type criterion cannot be derived in generalized linear regressions such as a logistic regression. Hence, this paper obtains the AIC for the group Lasso based on its original definition under the framework of generalized linear models. In terms of computational cost, our criterion is clearly better than the cross validation, but it is shown through simulation studies that the performance of the our criterion is almost the same as or better than that of the cross validation.

On the overestimation of random forest's out-of-bag error.

The ensemble method random forests has become a popular classification tool in bioinformatics and related fields. The out-of-bag error is an error estimation technique often used to evaluate the accuracy of a random forest and to select appropriate values for tuning parameters, such as the number of candidate predictors that are randomly drawn for a split, referred to as mtry. However, for binary classification problems with metric predictors it has been shown that the out-of-bag error can overestimate the true prediction error depending on the choices of random forests parameters. Based on simulated and real data this paper aims to identify settings for which this overestimation is likely. It is, moreover, questionable whether the out-of-bag error can be used in classification tasks for selecting tuning parameters like mtry, because the overestimation is seen to depend on the parameter mtry. The simulation-based and real-data based studies with metric predictor variables performed in this paper show that the overestimation is largest in balanced settings and in settings with few observations, a large number of predictor variables, small correlations between predictors and weak effects. There was hardly any impact of the overestimation on tuning parameter selection. However, although the prediction performance of random forests was not substantially affected when using the out-of-bag error for tuning parameter selection in the present studies, one cannot be sure that this applies to all future data. For settings with metric predictor variables it is therefore strongly recommended to use stratified subsampling with sampling fractions that are proportional to the class sizes for both tuning parameter selection and error estimation in random forests. This yielded less biased estimates of the true prediction error. In unbalanced settings, in which there is a strong interest in predicting observations from the smaller classes well, sampling the same number of observations from each class is a promising alternative.

Excess Optimism: How Biased is the Apparent Error of an Estimator Tuned by SURE?

ABSTRACT Nearly all estimators in statistical prediction come with an associated tuning parameter, in one way or another. Common practice, given data, is to choose the tuning parameter value that minimizes a constructed estimate of the prediction error of the estimator; we focus on Stein’s unbiased risk estimator, or SURE, which forms an unbiased estimate of the prediction error by augmenting the observed training error with an estimate of the degrees of freedom of the estimator. Parameter tuning via SURE minimization has been advocated by many authors, in a wide variety of problem settings, and in general, it is natural to ask: what is the prediction error of the SURE-tuned estimator? An obvious strategy would be simply use the apparent error estimate as reported by SURE, that is, the value of the SURE criterion at its minimum, to estimate the prediction error of the SURE-tuned estimator. But this is no longer unbiased; in fact, we would expect that the minimum of the SURE criterion is systematically biased downwards for the true prediction error. In this work, we define the excess optimism of the SURE-tuned estimator to be the amount of this downward bias in the SURE minimum. We argue that the following two properties motivate the study of excess optimism: (i) an unbiased estimate of excess optimism, added to the SURE criterion at its minimum, gives an unbiased estimate of the prediction error of the SURE-tuned estimator; (ii) excess optimism serves as an upper bound on the excess risk, that is, the difference between the risk of the SURE-tuned estimator and the oracle risk (where the oracle uses the best fixed tuning parameter choice). We study excess optimism in two common settings: shrinkage estimators and subset regression estimators. Our main results include a James–Stein-like property of the SURE-tuned shrinkage estimator, which is shown to dominate the MLE; and both upper and lower bounds on excess optimism for SURE-tuned subset regression. In the latter setting, when the collection of subsets is nested, our bounds are particularly tight, and reveal that in the case of no signal, the excess optimism is always in between 0 and 10 degrees of freedom, regardless of how many models are being selected from. Supplementary materials for this article are available online.