Holdout Group Research Articles

Efficient population representation with more genetic markers increases performance of a steelhead (Oncorhynchus mykiss) genetic stock identification baseline.

Genetic stock identification (GSI) is an important fisheries management tool to identify the origin of fish harvested in mixed stock fisheries. Periodic updates of genetic baselines can improve performance via the addition of unsampled or under-sampled populations and the inclusion of more informative markers. We used a combination of baselines to evaluate how population representation, marker number, and marker type affected the performance and accuracy of genetic stock assignments (self-assignment, bias, and holdout group tests) for steelhead (Oncorhynchus mykiss) in the Snake River basin. First, we compared the performance of an existing genetic baseline with a newly developed one which had a reduced number of individuals from more populations using the same set of markers. Self-assignment rates were significantly higher (p < 0.001; +5.4%) for the older, larger baseline, bias did not differ significantly between the two, but there was a significant improvement in performance for the new baseline in holdout results (p < 0.001; mean increase of 25.0%). Second, we compared the performance of the new baseline with increased numbers of genetic markers (~2x increase of single-nucleotide polymorphisms; SNPs) for the same set of baseline individuals. In this comparison, results produced significantly higher rates of self-assignment (p < 0.001; +9.7%) but neither bias nor leave-one-out were significantly affected. Third, we compared 334 SNPs versus opportunistically discovered microhaplotypes from the same amplicons for the new baseline, and showed the latter produced significantly higher rates of self-assignment (p < 0.01; +2.6%), similar bias, but slightly lower holdout performance (-0.1%). Combined, we show the performance of genetic baselines can be improved via representative and efficient sampling, that increased marker number consistently improved performance over the original baseline, and that opportunistic discovery of microhaplotypes can lead to small improvements in GSI performance.

Prevalence Patterns and Onset Prediction of High Myopia for Children and Adolescents in Southern China via Real-World Screening Data: Retrospective School-Based Study.

Patients with high myopia have an increased lifetime risk of complications. The prevalence patterns of high myopia in children and adolescents in southern China are unclear. Early identification of high-risk individuals is critical for reducing the occurrence and development of high myopia and avoiding the resulting complications. This study aimed to determine the prevalence of high myopia in children and adolescents in southern China via real-world screening data and to predict its onset by studying the risk factors for high myopia based on machine learning. This retrospective school-based study was conducted in 13 cities with different gross domestic products in southern China. Through data acquisition and filtering, we analyzed the prevalence of high myopia and its association with age, school stage, gross domestic product, and risk factors. A random forest algorithm was used to predict high myopia among schoolchildren and then assessed in an independent hold-out group. There were 1,285,609 participants (mean age 11.80, SD 3.07, range 6-20 years), of whom 658,516 (51.2%) were male. The overall prevalence of high myopia was 4.48% (2019), 4.88% (2020), and 3.17% (2021), with an increasing trend from the age of 11 to 17 years. The rates of high myopia increased from elementary schools to high schools but decreased at all school stages from 2019 to 2021. The coastal and southern cities had a higher proportion of high myopia, with an overall prevalence between 2.60% and 5.83%. Age, uncorrected distance visual acuity, and spherical equivalents were predictive factors for high myopia onset in schoolchildren. The random forest algorithm achieved a high accuracy of 0.948. The area under the receiver operator characteristic curve (AUC) was 0.975. Both indicated sufficient model efficacy. The performance of the model was validated in an external test with high accuracy (0.971) and a high AUC (0.957). High myopia had a high incidence in Guangdong Province. Its onset in children and adolescents was well predicted with the random forest algorithm. Efficient use of real-world data can contribute to the prevention and early diagnosis of high myopia.

Phenotyping of ABCA4 Retinopathy by Machine Learning Analysis of Full-Field Electroretinography.

PurposeBiallelic pathogenic variants in ABCA4 are the commonest cause of monogenic retinal disease. The full-field electroretinogram (ERG) quantifies severity of retinal dysfunction. We explored application of machine learning in ERG interpretation and in genotype–phenotype correlations.MethodsInternational standard ERGs in 597 cases of ABCA4 retinopathy were classified into three functional phenotypes by human experts: macular dysfunction alone (group 1), or with additional generalized cone dysfunction (group 2), or both cone and rod dysfunction (group 3). Algorithms were developed for automatic selection and measurement of ERG components and for classification of ERG phenotype. Elastic-net regression was used to quantify severity of specific ABCA4 variants based on effect on retinal function.ResultsOf the cohort, 57.6%, 7.4%, and 35.0% fell into groups 1, 2, and 3 respectively. Compared with human experts, automated classification showed overall accuracy of 91.8% (SE, 0.169), and 96.7%, 39.3%, and 93.8% for groups 1, 2, and 3. When groups 2 and 3 were combined, the average holdout group accuracy was 93.6% (SE, 0.142). A regression model yielded phenotypic severity scores for the 47 commonest ABCA4 variants.ConclusionsThis study quantifies prevalence of phenotypic groups based on retinal function in a uniquely large single-center cohort of patients with electrophysiologically characterized ABCA4 retinopathy and shows applicability of machine learning. Novel regression-based analyses of ABCA4 variant severity could identify individuals predisposed to severe disease.Translational RelevanceMachine learning can yield meaningful classifications of ERG data, and data-driven scoring of genetic variants can identify patients likely to benefit most from future therapies.

Machine Learning to Determine Risk Factors for Myopia Progression in Primary School Children: The Anyang Childhood Eye Study.

IntroductionTo investigate the risk factors for myopia progression in primary school children and build prediction models by applying machine learning to longitudinal, cycloplegic autorefraction data.MethodsA total of 2740 children from grade 1 to grade 6 were examined annually over a period of 5 years. Myopia progression was determined as change in cycloplegic autorefraction. Questionnaires were administered to gauge environmental factors. Each year, risk factors were evaluated and prediction models were built in a training group and then tested in an independent hold-out group using the random forest algorithm.ResultsSix variables appeared in prediction models on myopia progression for all 5 years, with combined weight of 77% and prediction accuracy over 80%. Uncorrected distance visual acuity (UDVA) had the greatest weight (mean 28%, range 22–39%), followed by spherical equivalent (20%, 7–28%), axial length (13%, 10–14%), flat keratometry reading (K1) (7%, 4–11%), gender (6%, 2–9%), and parental myopia (3%, 1–10%). UDVA and spherical equivalent had peak weight at the second and third study years, respectively. The weight of myopic parents decreased steadily over the 5 years (9.5%, 1.9%, 1.8%, 1%, and 1.3%). Weekly time spent reading, reading distance, reading in bed, and frequency of eating meat were included as variables in different study years.ConclusionsMyopia progression in children was predicted well by machine learning models. UDVA and spherical equivalents were good predictive factors for myopia progression in children through primary school. Parental myopia was found to play a substantial role in the early stage of myopia progression but waned as children grew older.Supplementary InformationThe online version contains supplementary material available at 10.1007/s40123-021-00450-2.

Combining machine learning, space-time cloud restoration and phenology for farm-level wheat yield prediction

Though studies showed the potential of high-resolution optical sensors for crop yield prediction, several factors have limited their wider application. The main factors are obstruction of cloud, identification of phenology, demand for high computing infrastructure and the complexity of statistical methods. In this research, we created a novel approach by combining four methods. First, we implemented the cloud restoration algorithm called gapfill to restore missed Normalized Difference Vegetation Index (NDVI) values derived from Sentinel-2 sensor (S2) due to cloud obstruction. Second, we created square tiles as a solution for high computing infrastructure demand due to the use of high-resolution sensor. Third, we implemented gapfill following critical crop phenology stage. Fourth, observations from restored images combined with original (from cloud-free images) values and applied for winter wheat prediction. We applied seven base machine learning as well as two groups of super learning ensembles. The study successfully applied gapfill on high-resolution image to get good quality estimates for cloudy pixels. Consequently, yield prediction accuracy increased due to the incorporation of restored values in the regression process. Base models such as Generalized Linear Regression (GLM) and Random Forest (RF) showed improved capacity compared to other base and ensemble models. The two models revealed RMSE of 0.001 t/ha and 0.136 t/ha on the holdout group. The two models also revealed consistent and better performance using scatter plot analysis across three datasets. The approach developed is useful to predict wheat yield at field scale, which is a rarely available but vital in many developmental projects, using optical sensors.

Concussion assessment potentially aided by use of an objective multimodal concussion index

Objective Prompt, accurate, objective assessment of concussion is crucial as delays can lead to increased short and long-term consequences. The purpose of this study was to derive an objective multimodal concussion index (CI) using EEG at its core, to identify concussion, and to assess change over time throughout recovery. Methods Male and female concussed ( N = 232) and control ( N = 206) subjects 13–25 years were enrolled at 12 US colleges and high schools. Evaluations occurred within 72 h of injury, 5 days post-injury, at return-to-play (RTP), 45 days after RTP (RTP + 45); and included EEG, neurocognitive performance, and standard concussion assessments. Concussed subjects had a witnessed head impact, were removed from play for ≥ 5 days using site guidelines, and were divided into those with RTP &lt; 14 or ≥14 days. Part 1 describes the derivation and efficacy of the machine learning derived classifier as a marker of concussion. Part 2 describes significance of differences in CI between groups at each time point and within each group across time points. Results Sensitivity = 84.9%, specificity = 76.0%, and AUC = 0.89 were obtained on a test Hold-Out group representing 20% of the total dataset. EEG features reflecting connectivity between brain regions contributed most to the CI. CI was stable over time in controls. Significant differences in CI between controls and concussed subjects were found at time of injury, with no significant differences at RTP and RTP + 45. Within the concussed, differences in rate of recovery were seen. Conclusions The CI was shown to have high accuracy as a marker of likelihood of concussion. Stability of CI in controls supports reliable interpretation of CI change in concussed subjects. Objective identification of the presence of concussion and assessment of readiness to return to normal activity can be aided by use of the CI, a rapidly obtained, point of care assessment tool.

Altered Cortical Brain Structure and Increased Risk for Disease Seen Decades After Perinatal Exposure to Maternal Smoking: A Study of 9000 Adults in the UK Biobank.

Secondhand smoke exposure is a major public health risk that is especially harmful to the developing brain, but it is unclear if early exposure affects brain structure during middle age and older adulthood. Here we analyzed brain MRI data from the UK Biobank in a population-based sample of individuals (ages 44-80) who were exposed (n = 2510) or unexposed (n = 6079) to smoking around birth. We used robust statistical models, including quantile regressions, to test the effect of perinatal smoke exposure (PSE) on cortical surface area (SA), thickness, and subcortical volumes. We hypothesized that PSE would be associated with cortical disruption in primary sensory areas compared to unexposed (PSE-) adults. After adjusting for multiple comparisons, SA was significantly lower in the pericalcarine (PCAL), inferior parietal (IPL), and regions of the temporal and frontal cortex of PSE+ adults; these abnormalities were associated with increased risk for several diseases, including circulatory and endocrine conditions. Sensitivity analyses conducted in a hold-out group of healthy participants (exposed, n = 109, unexposed, n = 315) replicated the effect of PSE on SA in the PCAL and IPL. Collectively our results show a negative, long term effect of PSE on sensory cortices that may increase risk for disease later in life.

Assessment of the accuracy of the Greulich and Pyle hand-wrist atlas for age estimation in a contemporary Australian population

Age estimation of living individuals is topical, and is particularly important globally, owing to increasing migration of undocumented individuals. Radiographic skeletal analysis of hand-wrist skeletal development is often used to infer chronological age based on direct comparison to standards such as the Greulich and Pyle atlas. However, this atlas has been criticised for being applied to foreign populations without due consideration of accuracy. The aims of the current study are to assess the precision and accuracy of the atlas in a contemporary Western Australian population and to develop population specific standards based on the latter system. The study sample comprised 360 individuals (equal sexes) and aged from birth to 25 years; a hold-out group comprising a further 50 individuals was used for model validation. Age estimation was performed through the visual comparison of the study radiographs against the atlas standards; statistical analyses were performed to assess the relationship between estimated skeletal and actual age. Prediction models were formulated; mean SEE values were ±0.005–0.90 (male) and ±0.25–0.421 years (female). Comparisons with prior research demonstrates the importance of contemporary population standards. The models presented here have forensic utility in a Western Australia jurisdiction, albeit the level of accuracy achieved is not suitable for the specific determination of legal majority.

Does Greater Online Assortment Pay? An Empirical Study Using Matched Online and Catalog Shoppers

Most multichannel retailers offer larger assortment in their online channel than in the offline channels. Conventional belief holds that larger assortment leads to more sales. However, offering more products could also cannibalize the sales of the retailer's main product offerings, hence reducing efficiency in assortment management and increasing cost. Recent behavioral research suggests that excessive assortment can lead to customer cognitive overload and deter them from purchasing. In this paper, we investigate the effect of large online assortment on customer spending. We empirically estimate the effect of large online assortment by exploiting the natural experiment of channel migration, which allows us to compare the spending of customers who migrated to the retailer's online channel with that of a holdout group of pure catalog shoppers. We find that online customers outspend those catalog shoppers by 10.7 percent on the retailer's mainline products. We provide evidence that the increased spending by online customers is due to lower product search cost in the online channel. Lastly, our research suggests that the online sales of low-volume niche products can be an important source of revenue for the retailer.

User-Independent Recognition of Sports Activities From a Single Wrist-Worn Accelerometer: A Template-Matching-Based Approach.

To investigate the accuracy of template matching for classifying sports activities using the acceleration signal recorded with a wearable sensor. A population of 29 normal weight and 19 overweight subjects was recruited to perform eight common sports activities, while body movement was measured using a triaxial accelerometer placed at the wrist. User- and axis-independent acceleration signal templates were automatically extracted to represent each activity category and recognize activity types. Five different similarity measures between example signals and templates were compared: Euclidean distance, dynamic time warping (DTW), derivative DTW, correlation and an innovative index, and combining distance and correlation metrics ( Rce). Template-based activity recognition was compared to statistical-learning classifiers, such as Naïve Bayes, decision tree, logistic regression (LR), and artificial neural network (ANN) trained using time- and frequency-domain signal features. Each algorithm was tested on data from a holdout group of 15 normal weight and 19 overweight subjects. The Rce index outperformed other template-matching metrics by achieving recognition rate above 80% for the majority of the activities. Template matching showed robust classification accuracy when tested on unseen data and in case of limited training examples. LR and ANN achieved the highest overall recognition accuracy ∼ 85% but showed to be more vulnerable to misclassification error than template matching on overweight subjects' data. Template matching can be used to classify sports activities using the wrist acceleration signal. Automatically extracted template prototypes from the acceleration signal may be used to enhance accuracy and generalization properties of statistical-learning classifiers.

Inconsistency in albuminuria predictors in type 2 diabetes: a comparison between neural network and conditional logistic regression

Albuminuria is a sensitive marker to predict future cardiovascular events in patients with type 2 diabetes mellitus. However, current studies only use conventional regression models to discover predictors of albuminuria. We have used 2 different statistical models to predict albuminuria in type 2 diabetes mellitus: a multilayer perception neural network and a conditional logistic regression. Neural network models were used to predict the level of albuminuria in patients with type 2 diabetes mellitus, which include a matched case-control study for the population. For each case, we randomly selected 1 control matched by age and body mass index (BMI). The input variables were sex, duration of diabetes, systolic and diastolic blood pressure, glomerular filtration rate, high-density lipoprotein, low-density lipoprotein, triglyceride, high-density lipoprotein/triglyceride ratio, cholesterol, fasting blood sugar, and glycated hemoglobin. Age and BMI were included only in the neural network model. This model included 4 hidden layers and 1 bias. Relative error of predictions was 0.38% in the training group, 0.52% in the testing group, and 1.20% in the holdout group. The most robust predictors of albuminuria were high-density lipoprotein (21%), cholesterol (14.4%), and systolic blood pressure (9.7%). Using the conditional logistic regression model, glomerular filtration rate, time of onset to diabetes, and sex were significant indicators in the onset of albuminuria. Using a neural network model, we show that high-density lipoprotein is the most important factor in predicting albuminuria in type 2 diabetes mellitus. Our neural network model complements the current risk factor models to improve the care of patients with diabetes.

MELD-XI: A rational approach to “sickest first” liver transplantation in cirrhotic patients requiring anticoagulant therapy

Priority for "sickest first" liver transplantation (LT) in the United States is determined by the model for end-stage liver disease (MELD). MELD is a good predictor of short-term mortality in cirrhosis, but it can overestimate risk when international normalized ratio (INR) is artificially elevated by anticoagulation. An alternate prognostic index omitting INR is needed in this situation. We retrospectively analyzed survival data for 554 cirrhotic veterans referred for consideration of LT prior to December 1, 2003 (training group). Using logistic regression we derived a predictive formula for 90-day pretransplant mortality incorporating bilirubin and creatinine but omitting INR. We normalized this formula to the same scale as MELD using linear regression. This yielded MELD-XI (for MELD excluding INR) = 5.11 Ln(bilirubin) + 11.76 Ln(creatinine) + 9.44. Accuracy of MELD-XI was validated in a holdout group of 278 cirrhotic veterans referred after December 1, 2003, and in an independent validation dataset of 7,203 cirrhotic adults listed for LT in the United States between May 1, 2001, and October 31, 2001. MELD-XI and MELD correlated well in training, holdout, and independent validation cohorts (r = 0.930, 0.954, and 0.902, respectively). In the holdout cohort, c-statistics of MELD vs. MELD-XI for mortality were, respectively, 0.939 vs. 0.906 at 30 days;0.860 vs. 0.841 at 60 days; 0.842 vs. 0.829 at 90 days; and 0.795 vs. 0.797 at 180 days. In the independent validation dataset, c-statistics for MELD vs. MELD-XI as predictors of 90-day survival were, respectively, 0.857 vs. 0.843 in noncholestatic liver diseases and 0.905 vs. 0.894 in cholestatic liver diseases. Comparable MELD and MELD-XI scores were associated with comparable prognosis. In conclusion, MELD-XI, despite omission of INR, is nearly as accurate as MELD in predicting short-term survival in cirrhosis. In patients treated with oral anticoagulants, substitution of MELD-XI for MELD may permit more accurate assessment of risk and more rational assignment of "sickest first" priority for LT.

Muscle reflex classification of low-back pain

It has been well documented that low-back pain (LBP) patients have longer muscle response latencies to perturbation than healthy controls. These muscle responses appear to be reflexive and not voluntary in nature, and as a result, might be useful for objectively classifying LBP. The goal of the study was to develop an objective and accurate method for classifying LBP using a sudden load-release protocol. Subjects were divided into two groups: learning group (20 patients and 20 controls), and holdout group (15 patients and 12 controls). Subjects exerted isometric trunk force against a cable in four different directions. Following cable release, the trunk was suddenly displaced eliciting a muscle reflex response. Reflex latencies for muscles switching-on and shutting-off were determined using electromyogram signals from 8 trunk muscles. Independent t tests were performed on the learning group to determine which reflex parameters were to be entered into logistic regression analysis to produce a classification model. The holdout group was used to validate this classification model. The three-parameter model was able to correctly classify 83% of the learning group, and 81% of the holdout group. Using reflex parameters appears to be an accurate and objective method for classifying LBP.

Estimating Success in Predicting a Variable with Nominal Measurement Using Other Variables with Nominal Measurements

In this article we are concerned with the situation where one is estimating the outcome of a variable Y, with nominal measurement, on the basis of the outcomes of several predictor variables, X 1, X 2, ..., X r, each with nominal measurement. We assume that we have a random sample from the population. Here we are interested in estimating p, the probability of successfully predicting a new Y from the population, given the X measurements for this new observation. We begin by proposing an estimator, pa, which is the success rate in predicting Y from the current sample. We show that this estimator is always biased upwards. We then propose a second estimator, pb, which divides the original sample into two groups, a holdout group and a training group, in order to estimate p. We show that procedures such as these are always biased downwards, no matter how we divide the original sample into the two groups. Because one of these estimators tends to overestimate p while the other tends to underestimate p, we propose as a heuristic solution to use the mean of these two estimators, pc, as an estimator for p. We then perform several simulation studies to compare the three estimators with respect to both bias and MSE. These simulations seem to confirm that $ p c is a better estimator than either of the other two.

The prediction of power and efficiency during near-maximal rowing

The relationship between power and gross efficiency during near-maximal rowing, and physiological measures of strength, power, aerobic and anaerobic capacities and United State Rowing Association (USRA) performance tests (independent variables) was investigated among collegiate male rowers. Criterion measures of rowing power and gross efficiency were measured in a moving-water rowing tank, using an oar instrumented with strain gauges to assess force and a potentiometer to assess oar position. Bivariate correlation analysis (n = 28) indicated no relationship between the independent variables and rowing gross efficiency (P > 0.05). Rowing power [mean (SD) 483.4 (34.75) W] was significantly related to inboard leg extension strength (IL strength, r = 0.63), outboard leg extension strength (r = 0.45), combined leg extension strength (r = 0.45), and time to complete the USRA 2000-m simulated rowing race (r = -0.52; P <0.05). Stepwise regression using resampling cross-validation of 15 random samples (21 subjects per sample selected from a total group of 28 intercollegiate oarsmen) indicated that predictors of rowing power were IL strength and blood lactate following a peak oxygen uptake rowing test with significant multiple correlations of R 0.61 to 0.86 (P <0.05). The standard error of estimate (SEM) ranged from 18.1 to 29.9 W, or 5.3 (0.77) percent of the criterion value. Cross-validation with a hold-out group (seven subjects per sample) was performed for each equation and correlations ranged from R = 0.14 to 0.97 (SEM = 8.0 to 38.9 W). In conclusion, data from the present study suggest that to increase rowing power, training should emphasize leg strength and anaerobic training to decrease the level of lactate accumulated during rowing.

Panel appraisal as a criterion in test validation: A demonstration in a law enforcement job

The purpose of this project was to develop and validate an entry level written exam for the job of deputy sheriff. In addition, the study investigated a novel approach to collecting criterion data-panel appraisals. A 265 item pilot test was developed in order to cover the KSAO's identified by a thorough job analysis. A concurrent criterion-related validation strategy was employed in which: (1) criterion data were collected in the form of supervisory job performance ratings (i.e., using the panel appraisal format), performance in training, and performance over time on 373 deputies and corporals, and (2) the pilot test was then administered to these same deputies and corporals. Item analysis procedures on an 80% calibration sample resulted in the selection of 75 items for the final version of the written exam. Validity estimates, calculated on a 20% holdout group, were significant for the job performance and performance in training criteria, r=.26 and r=.40, respectively. Subsequent comparisons of the three performance criteria indicated that the panel appraisal format produced a highly reliable criterion measure.

Discriminant analysis for predicting dystocia in beef cattle. II. Derivation and validation of a prebreeding prediction model.

Discriminant analysis was utilized to derive and validate a model for predicting dystocia using only data available at the beginning of the breeding season. Data were collected from 211 Chianina crossbred cows (2 to 6 yr old) bred to Chianina bulls. A proportionally stratified sampling procedure divided females into an analysis sample (n = 134) on which the model was derived and a hold-out sample (n = 77) on which the prediction model was validated (tested). Variables available during the derivation stage were cow age, cow weight, pelvic height, pelvic width, pelvic area and calf sire. Dystocia was categorized as either unassisted or assisted. Occurrence of dystocia was 17.2 and 18.2% in the analysis and hold-out samples, respectively. All data were standardized to a mean of zero and a variance of one before statistical analysis. The centroid of cows experiencing dystocia differed (P less than .01) from that of cows calving unassisted in the analysis sample. Significant variables were pelvic area and cow age (standardized coefficients = .56 and .51, respectively). This model correctly classified 85.1% of the cows in the analysis sample. This was 13.5% greater than the proportional chance criterion. For model validation, prediction accuracy was 84.4% in the hold-out group, which was 14.2% greater than the proportional chance criterion. However, only 57.1% of the cows that experienced dystocia were correctly classified. Examination of the data revealed that those cows misclassified were 3 yr of age or older.(ABSTRACT TRUNCATED AT 250 WORDS)

A “New” Type of True-False Item

A multiple-alternative item type having the basic characteristics of true-false items but differing from the conventional true-false type item in several respects was described. Evidence was presented supporting the contention that the item type is suited to the task of measurement in the areas of diagnosis and hypothesis formulation. Further, following the construction of a key with the aid of item analysis, the correlation between the test and a performance test criterion measure was .60 for a hold-out group used for purposes of cross-validation.

A "NEW" TYPE OF TRUE-FALSE ITEM

A multiple-alternative item type having the basic characteristics of true-false items but differing from the conventional true-false type item in several respects was described. Evidence was presented supporting the contention that the item type is suited to the task of measurement in the areas of diagnosis and hypothesis formulation. Further, following the construction of a key with the aid of item analysis, the correlation between the test and a performance test criterion measure was .60 for a hold-out group used for purposes of cross-validation.