Prediction Error Estimation Research Articles

Convergent neural signatures of speech prediction error are a biological marker for spoken word recognition

We use MEG and fMRI to determine how predictions are combined with speech input in superior temporal cortex. We compare neural responses to words in which first syllables strongly or weakly predict second syllables (e.g., “bingo”, “snigger” versus “tango”, “meagre”). We further compare neural responses to the same second syllables when predictions mismatch with input during pseudoword perception (e.g., “snigo” and “meago”). Neural representations of second syllables are suppressed by strong predictions when predictions match sensory input but show the opposite effect when predictions mismatch. Computational simulations show that this interaction is consistent with prediction error but not alternative (sharpened signal) computations. Neural signatures of prediction error are observed 200 ms after second syllable onset and in early auditory regions (bilateral Heschl’s gyrus and STG). These findings demonstrate prediction error computations during the identification of familiar spoken words and perception of unfamiliar pseudowords.

Aberrant neural computation of social controllability in nicotine-dependent humans

Social controllability, or the ability to exert control during social interactions, is crucial for optimal decision-making. Inability to do so might contribute to maladaptive behaviors such as smoking, which often takes place in social settings. Here, we examined social controllability in nicotine-dependent humans as they performed an fMRI task where they could influence the offers made by simulated partners. Computational modeling revealed that smokers under-estimated the influence of their actions and self-reported a reduced sense of control, compared to non-smokers. These findings were replicated in a large independent sample of participants recruited online. Neurally, smokers showed reduced tracking of forward projected choice values in the ventromedial prefrontal cortex, and impaired computation of social prediction errors in the midbrain. These results demonstrate that smokers were less accurate in estimating their personal influence when the social environment calls for control, providing a neurocomputational account for the social cognitive deficits in this population. Pre-registrations: OSF Registries|How interoceptive state interacts with value-based decision-making in addiction (fMRI study). OSF Registries|COVID-19: social cognition, mental health, and social distancing (online study).

Encryption–decryption-based state estimation for nonlinear complex networks subject to coupled perturbation

This paper discusses the encryption-decryption-based state estimation (EDBSE) issue for coupled perturbation complex networks (CPCNs) in the framework of the Kalman-type filtering scheme. A uniform distributed random variable is employed to characterize the coupled perturbation among different network units. A uniform-quantization-dependent encryption-decryption (UQDED) scheme is considered here to orchestrate the transmitted data. A novel EDBSE approach is developed such that the upper bounds of prediction error (PE) covariance (PEC) and estimation error (EE) covariance (EEC) can be derived by resolving Riccati-like difference equations and the estimation parameter (EP) is determined by minimizing the trace of the upper bound of EEC. Furthermore, a uniformly bounded condition is elaborated to evaluate the algorithm performance of EDBSE. Finally, an illustrative example is conducted to verify the validity of the introduced EDBSE method.

Study on applicable coverage extension of theory-based generalization errors bounds to the variants of RVFL network and ELM

This study investigates the coverage extendibility of generalized prediction error estimation for the random vector functional link (RVFL) network and generalized extreme learning machine (GELM). GELM is an ELM applying the limiting behavior of Moore–Penrose generalized inverse (M–P GI) to the output matrix of the last hidden layer of ELM. The RVFL network, the ELM, and their variants are currently employed in various areas due to their fast learning and are expected to be adopted in more applications. Additionally, obtaining prediction error bounds quickly for these networks could greatly increase their utilities. Therefore, exploring the error bounds of the variants is crucial. Recently, the upper and lower bounds of generalized prediction error and their associated tail probabilities for the RVFL network and GELM were analyzed. This study aims to extend the coverage of the existing error bounds to variants of RVFL and ELM. To this end, we investigate each solution of the objective functions for the existing RVFL and ELM variants, compare it with the solution obtained by using the limiting behavior of M–P GI, and then discuss the applicability of the probability inequalities for the prediction error to the variant. Specifically, we conceptualize a restricted solution vector and prove that the restricted solution of a variant of RVFL is identical to that of GELM, with an appropriately selected regularization parameter. The mechanism used in the proof can be applied to other variants, which have different objective functions but have similar forms of solution. Experiments are performed with the CAIDA (Center for Applied Internet Data Analysis) dataset. Numerical results verify the analysis, showing that the generalization prediction error bounds and their associated tail probabilities for ELM (RVFL) variants in applicability coverage are almost identical to those of the GELM (RVFL). In addition, performance-based network similarity is introduced and computed with the obtained results. Based on this analysis, we can conclude that the probability inequalities for prediction error derived can be directly applied to some variants of the RVFL network and ELM, such as robust regularized RVFL and ridge regression ELM. However, it cannot apply directly to the networks that use information obtained during the learning process in its learning, such as RVFL+ and its variants. The significance of this study is that it enables us to figure out the relationship between the size-restricted final weight vector of RVFL and the final weight vectors of ELM, the effect of the relationship on generalized prediction error estimation, and the performance-based similarity between networks. This study provides a theoretical basis for the generalized prediction error's upper and lower bounds and their associated tail probabilities for ELM (RVFL) variants.

The Prognostic Value of the Systemic Immune-Inflammation Index (SII) and Red Cell Distribution Width (RDW) in Patients with Cervical Cancer Treated Using Radiotherapy.

There is growing interest in the prognostic value of routinely performed pre-treatment blood test indices, such as the RDW or SII, with the latter combining the neutrophil-to-lymphocyte ratio (NLR) and platelet-to-lymphocyte ratio (PLR). These indices were shown to be prognostic for survival in some malignancies. The purpose of this study was to evaluate the association between pre-treatment RDW and SII, and OS in patients treated with radiotherapy for primary localised cervical cancer. This retrospective analysis included patients treated with definitive CRT between 2011 and 2017 for histopathologically confirmed FIGO 2018 stage IB2-IVA cervical cancer. Statistical analysis was performed using the Kaplan-Meier method, two-sided log-rank tests, and Cox proportional hazards models, with the AIC serving as a prediction error estimator. The study group included 249 patients with a median age of 57.2 years and a median follow-up of 75.8 months. The majority were diagnosed with squamous cell carcinoma (237; 95.2%) and had FIGO stage III (211; 84.7%). Approximately half of the patients (116; 46.4%) had regional lymph node metastases. Patients with a low RDW (≤13.4%) and low SII (≤986.01) had a significantly longer OS (p = 0.001 and p = 0.002). The RDW remained as an independent prognostic factor in the multivariable model (high vs. low; HR = 2.04; 95% CI: 1.32-3.16; p = 0.001). Including RDW in the model decreased the Akaike Information Criterion from 1028.25 to 1018.15. The RDW is a cheap and widely available index that is simultaneously an independent prognostic factor for survival and could be used to improve pre-treatment prognosis assessments in patients with cervical cancer undergoing CRT. Available data encourage assessing the RDW as a prognostic factor in prospective trials to aid the identification of candidates for treatment escalation.

Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks

Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.

Learning Cognitive Flexibility: Neural Substrates of Adapting Switch-Readiness to Time-varying Demands.

An individual's readiness to switch tasks (cognitive flexibility) varies over time, in part, as the result of reinforcement learning based on the statistical structure of the world around them. Consequently, the behavioral cost associated with task-switching is smaller in contexts where switching is frequent than where it is rare, but the underlying brain mechanisms of this adaptation in cognitive flexibility are not well understood. Here, we manipulated the likelihood of switches across blocks of trials in a classic cued task-switching paradigm while participants underwent fMRI. As anticipated, behavioral switch costs decreased as the probability of switching increased, and neural switch costs were observed in lateral and medial frontoparietal cortex. To study moment-by-moment adjustments in cognitive flexibility at the neural level, we first fitted the behavioral RT data with reinforcement learning algorithms and then used the resulting trial-wise prediction error estimate as a regressor in a model-based fMRI analysis. The results revealed that lateral frontal and parietal cortex activity scaled positively with unsigned switch prediction error and that there were no brain regions encoding signed (i.e., switch- or repeat-specific) prediction error. Taken together, this study documents that adjustments in cognitive flexibility to time-varying switch demands are mediated by frontoparietal cortex tracking the likelihood of forthcoming task switches.

A wind speed interval prediction method for reducing noise uncertainty

Due to the noise uncertainty, the conventional point prediction model is difficult to describe the actual characteristics of wind speed and lacks a description of the wind speed fluctuation range. In this paper, the kernel density estimation according to its error value is given, and then its fluctuation range is found to combine the prediction results of the test set to get its prediction range. Firstly, the singular spectrum analysis (SSA) is introduced to conduct the noise reduction, and variational modal decomposition (VMD) is performed to handle the sequences, then an improved slime mold algorithm (SMA) is proposed to optimize the VMD, and the stochastic configuration networks (SCNs) is applied to perform the prediction. Finally, the interval prediction results are calculated by fusing the point prediction error and kernel density estimation. The experimental results demonstrate that the proposed method can effectively reduce the noise interference in the wind speed prediction.

Robustness investigation of cross-validation based quality measures for model assessment

In this paper the accuracy and robustness of quality measures for the assessment of machine learning models are investigated. The prediction quality of a machine learning model is evaluated model-independent based on a cross-validation approach, where the approximation error is estimated for unknown data. The presented measures quantify the amount of explained variation in the model prediction. The reliability of these measures is assessed by means of several numerical examples, where an additional data set for the verification of the estimated prediction error is available. Furthermore, the confidence bounds of the presented quality measures are estimated and local quality measures are derived from the prediction residuals obtained by the cross-validation approach.

AI-Assisted Fatigue and Stamina Control for Performance Sports on IMU-Generated Multivariate Times Series Datasets.

Optimal sports performance requires a balance between intensive training and adequate rest. IMUs provide objective, quantifiable data to analyze performance dynamics, despite the challenges in quantifying athlete training loads. The ability of AI to analyze complex datasets brings innovation to the monitoring and optimization of athlete training cycles. Traditional techniques rely on subjective assessments to prevent overtraining, which can lead to injury and underperformance. IMUs provide objective, quantitative data on athletes' physical status during action. AI and machine learning can turn these data into useful insights, enabling data-driven athlete performance management. With IMU-generated multivariate time series data, this paper uses AI to construct a robust model for predicting fatigue and stamina. IMUs linked to 19 athletes recorded triaxial acceleration, angular velocity, and magnetic orientation throughout repeated sessions. Standardized training included steady-pace runs and fatigue-inducing techniques. The raw time series data were used to train a supervised ML model based on frequency and time-domain characteristics. The performances of Random Forest, Gradient Boosting Machines, and LSTM networks were compared. A feedback loop adjusted the model in real time based on prediction error and bias estimation. The AI model demonstrated high predictive accuracy for fatigue, showing significant correlations between predicted fatigue levels and observed declines in performance. Stamina predictions enabled individualized training adjustments that were in sync with athletes' physiological thresholds. Bias correction mechanisms proved effective in minimizing systematic prediction errors. Moreover, real-time adaptations of the model led to enhanced training periodization strategies, reducing the risk of overtraining and improving overall athletic performance. In sports performance analytics, the AI-assisted model using IMU multivariate time series data is effective. Training can be tailored and constantly altered because the model accurately predicts fatigue and stamina. AI models can effectively forecast the beginning of weariness before any physical symptoms appear. This allows for timely interventions to prevent overtraining and potential accidents. The model shows an exceptional ability to customize training programs according to the physiological reactions of each athlete and enhance the overall training effectiveness. In addition, the study demonstrated the model's efficacy in real-time monitoring performance, improving the decision-making abilities of both coaches and athletes. The approach enables ongoing and thorough data analysis, supporting strategic planning for training and competition, resulting in optimized performance outcomes. These findings highlight the revolutionary capability of AI in sports science, offering a future where data-driven methods greatly enhance athlete training and performance management.

Pre-Interventional Risk Assessment in The Elderly (PIRATE): Development of a scoring system to predict 30-day mortality using data of the Peri-Interventional Outcome Study in the Elderly.

Risk assessment before interventions in elderly patients becomes more and more vital due to an increasing number of elderly patients requiring surgery. Existing risk scores are often not tailored to marginalized groups such as patients aged 80 years or older. We aimed to develop an easy-to-use and readily applicable risk assessment tool that implements pre-interventional predictors of 30-day mortality in elderly patients (≥80 years) undergoing interventions under anesthesia. Using Cox regression analysis, we compared different sets of predictors by taking into account their ease of availability and by evaluating predictive accuracy. Coefficient estimates were utilized to set up a scoring system that was internally validated. Model building and evaluation were based on data from the Peri-Interventional Outcome Study in the Elderly (POSE), which was conducted as a European multicenter, observational prospective cohort study. Our risk assessment tool, named PIRATE, contains three predictors assessable at admission (urgency, severity and living conditions). Discriminatory power, as measured by the concordance index, was 0.75. The estimated prediction error, as measured by the Brier score, was 0.036 (covariate-free reference model: 0.043). PIRATE is an easy-to-use risk assessment tool that helps stratifying elderly patients undergoing interventions with anesthesia at increased risk of mortality. PIRATE is readily available and applies to a wide variety of settings. In particular, it covers patients needing elective or emergency surgery and undergoing in-hospital or day-case surgery. Also, it applies to all types of interventions, from minor to major. It may serve as a basis for multidisciplinary and informed shared decision-making.

Transcranial magnetic stimulation mapping of the motor cortex: comparison of five estimation algorithms.

There are currently five different kinds of transcranial magnetic stimulation (TMS) motor mapping algorithms available, from ordinary point-based algorithms to advanced field-based algorithms. However, there have been only a limited number of comparison studies conducted, and they have not yet examined all of the currently available algorithms. This deficiency impedes the judicious selection of algorithms for application in both clinical and basic neuroscience, and hinders the potential promotion of a potential superior algorithm. Considering the influence of algorithm complexity, further investigation is needed to examine the differences between fMRI peaks and TMS cortical hotspots that were identified previously. Twelve healthy participants underwent TMS motor mapping and a finger-tapping task during fMRI. The motor cortex TMS mapping results were estimated by five algorithms, and fMRI activation results were obtained. For each algorithm, the prediction error was defined as the distance between the measured scalp hotspot and optimized coil position, which was determined by the maximum electric field strength in the estimated motor cortex. Additionally, the study identified the minimum number of stimuli required for stable mapping. Finally, the location difference between the TMS mapping cortical hotspot and the fMRI activation peak was analyzed. The projection yielded the lowest prediction error (5.27 ± 4.24 mm) among the point-based algorithms and the association algorithm yielded the lowest (6.66 ± 3.48 mm) among field-based estimation algorithms. The projection algorithm required fewer stimuli, possibly resulting from its suitability for the grid-based mapping data collection method. The TMS cortical hotspots from all algorithms consistently deviated from the fMRI activation peak (20.52 ± 8.46 mm for five algorithms). The association algorithm might be a superior choice for clinical applications and basic neuroscience research, due to its lower prediction error and higher estimation sensitivity in the deep cortical structure, especially for the sulcus. It also has potential applicability in various other TMS domains, including language area mapping and more. Otherwise, our results provide further evidence that TMS motor mapping intrinsically differs from fMRI motor mapping.

Assessing automatic data processing algorithms for RGB-D cameras to predict fruit size and weight in apples

Data acquired using an RGB-D Azure Kinect DK camera were used to assess different automatic algorithms to estimate the size, and predict the weight of non-occluded and occluded apples. The programming of the algorithms included: (i) the extraction of images of regions of interest (ROI) using manual delimitation of bounding boxes or binary masks; (ii) estimating the lengths of the major and minor geometric axes for the purpose of apple sizing; and (iii) predicting the final weight by allometric modelling. In addition to the use of bounding boxes, the algorithms also allowed other post-mask settings (circles, ellipses and rotated rectangles) to be implemented, and different depth options (distance between the RGB-D camera and the fruits detected) for subsequent sizing through the application of the thin lens theory. Both linear and nonlinear allometric models demonstrated the ability to predict apple weight with a high degree of accuracy (R2 greater than 0.942 and RMSE < 16 g). With respect to non-occluded apples, the best weight predictions were achieved using a linear allometric model including both the major and minor axes of the apples as predictors. The mean absolute percentage error (MAPE) ranged from 5.1% to 5.7% with respective RMSE of 11.09 g and 13.02 g, depending to whether circles, ellipses, or bounding boxes were used to adjust fruit shape. The results were therefore promising and open up the possibility of implementing reliable in-field apple measurements in real time. Importantly, final weight prediction error and intermediate size estimation errors (from sizing algorithms) interact but in a way that is not easily quantifiable when weight allometric models with implicit prediction error are used. In addition, allometric models should be reviewed when applied to other apple cultivars, fruit development stages or even for different fruit growth conditions depending on canopy management.

Deploying artificial neural network to predict hybrid biodiesel fuel properties from their fatty acid compositions

Measurement-related problems have spurred fuel properties prediction using machine learning techniques. Improved fuel properties offered by hybrid biodiesel (HB) via mixed oils were predicted from their fatty acid compositions (FACs) using artificial neural network (ANN). FACs and fuel properties of HB sourced from the literature were used to develop ANN models. FAC data were used as the input parameters to predict the fuel properties data (kinematic viscosity (KV), density, calorific value (CV), and flash point (FP)) considered as the output parameters of the models. Using the multilayer perception ANN, the models were trained using Levenberg-Marquardt back propagation learning algorithm coupled with different numbers of neurons and activation functions for the prediction of the fuel properties. The models were observed to accurately predict these fuel properties with high prediction accuracy (R 2 = 1). The evaluated model performance errors were 0.1014 and 0.0504, 0.2905 and 0.4225, 0.1848, and 0.1038, and 0.4726 and 0.7833 for KV, density, CV, and FP using root mean square error and average absolute deviation respectively. Prediction performance and error estimates were slightly better than those for single feedstock biodiesel. Hence, this study shows the ability of ANN to accurately predict the fuel properties of HB from the FAs.

Estimation of prediction error in time series

Summary The accurate estimation of prediction errors in time series is an important problem, which has immediate implications for the accuracy of prediction intervals as well as the quality of a number of widely used time series model selection criteria such as the Akaike information criterion. Except for simple cases, however, it is difficult or even impossible to obtain exact analytical expressions for one-step and multi-step predictions. This may be one of the reasons that, unlike in the independent case (see Efron, 2004), up to now there has been no fully established methodology for time series prediction error estimation. Starting from an approximation to the bias-variance decomposition of the squared prediction error, a method for accurate estimation of prediction errors in both univariate and multivariate stationary time series is developed in this article. In particular, several estimates are derived for a general class of predictors that includes most of the popular linear, nonlinear, parametric and nonparametric time series models used in practice, with causal invertible autoregressive moving average and nonparametric autoregressive processes discussed as lead examples. Simulations demonstrate that the proposed estimators perform quite well in finite samples. The estimates may also be used for model selection when the purpose of modelling is prediction.

Estimation of Chlorophyll–a and Total Suspended Solid Based on Observation and Sentinel-2 Imagery in Coastal Water Teluk Awur, Jepara-Indonesia

The existence of aquaculture in Marine Science Techno Park (MSTP), Jepara requires good-quality water. Remote sensing is the right solution to conduct routine, cost-effective, and wide-ranging monitoring. This study aims to estimate the Total Suspended Solid (TSS) and chlorophyll-a (Chl-a) values based on Sentinel-2 imagery. The reflectance values used are from Sentinel-2A (http://marine.copernicus.eu/acquisitions on 14 September and 30 October 2022). The TSS estimation algorithm used is a single band (red), while for Chl-a, it uses the sum of four visible bands namely blue, green, red, and near-infrared. The predicted TSS values from the Sentinel-2 ranged from 16.65-144.78 mg/L (average 44.59 mg/L) and Chl-a was 1.65-5.57 µg/L (average 3.35 µg/L) and 0.59 - 5.284 µg/L (average 2.49 µg/L) in September. While the TSS and Chl-a in-situ in October 2022, ranged from 48.80 - 78.20 mg/L (mean 55.81 mg/L) and 0.882 - 4.736 mg/L (average 3.10 mg/L). The performance of the algorithm used in this study is not suitable for implementation in this study regarding the low prediction error estimation values as follows: RMSE, bias, and MAPE values for TSS are 21.17 mg/L, -10.76, and 31.52%, respectively; and for Chl-a are 1.04 µg/L, 0.25, and 35.83%, respectively. Thus, a special algorithm needs to be developed for the coastal waters of Teluk Awur.

Robust filtering for spacecraft attitude estimation systems with multiplicative noises, unknown measurement disturbances and correlated noises

Unknown measurement disturbances and correlated noise caused by jitter or vibration phenomena during spacecraft operation and the complex external environment are important topics for spacecraft attitude estimation. This paper studies the problem of spacecraft attitude estimation for nonlinear systems with multiplicative noises, unknown measurement disturbances and correlated noises. In addition, a two-step prediction framework is used to complete the noise decoupling. This paper aims to design a robust filter that minimizes the upper bound of the prediction error covariance and estimation error covariance in the presence of multiplicative noises, correlated noises and unknown measurement disturbances. The prediction gain and filtering gain of the robust filter are designed to minimize the upper bound. Finally, the simulation results verify the effectiveness of the proposed filter.

A Guide to Cross-Validation for Artificial Intelligence in Medical Imaging.

Artificial intelligence (AI) is being increasingly used to automate and improve technologies within the field of medical imaging. A critical step in the development of an AI algorithm is estimating its prediction error through cross-validation (CV). The use of CV can help prevent overoptimism in AI algorithms and can mitigate certain biases associated with hyperparameter tuning and algorithm selection. This article introduces the principles of CV and provides a practical guide on the use of CV for AI algorithm development in medical imaging. Different CV techniques are described, as well as their advantages and disadvantages under different scenarios. Common pitfalls in prediction error estimation and guidance on how to avoid them are also discussed. Keywords: Education, Research Design, Technical Aspects, Statistics, Supervised Learning, Convolutional Neural Network (CNN) Supplemental material is available for this article. © RSNA, 2023.

Unbiased prediction errors for partial least squares regression models: Choosing a representative error estimator for process monitoring

Partial least squares (PLS) regression is widely used to predict chemical analytes from spectroscopic data, thus reducing the need for expensive and time-consuming wet chemical reference analysis in industrial process monitoring. However, predictions via PLS by definition carry sample-specific errors, and estimation of these errors is essential for correct interpretation of results. To increase trust in PLS regression-based predictions, reliable prediction error estimates must be reported. This can be achieved by determining realistic sample-specific prediction errors using an unbiased mean squared prediction error estimate. This work provides a guide for estimating sample-specific prediction errors, showing the importance of choosing an appropriate error estimator prior to deploying PLS models for industrial applications. We reviewed recent and established methods for estimating the sample-specific prediction error and test them through simulation studies. The methods were subsequently applied for estimating prediction errors in two real-life datasets from the food ingredients industry, where near-infrared spectroscopy was used to quantify i) urea in process water and ii) individual protein concentrations in ultrafiltration retentates from a protein fractionation process. Both the simulations and real data examples showed that the mean squared error of calibration is always a downward biased estimator. Although leave-one-out-cross-validation performed surprisingly well in the data analysed in this work, this paper demonstrated that the appropriate choice of error estimator requires the user to make an informed, data-centered decision.

Cross-Validation: What Does It Estimate and How Well Does It Do It?

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow’s C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. Supplementary materials for this article are available online.