Complete Cross-validation Research Articles

A comprehensive exploration of complete cross-validation for circular data

Kernel density estimation of circular data has recently received considerable attention for its ability to model and analyse distributions on unit circles and other periodic domains. Our aim is to contribute to the literature on data-driven bandwidth selectors in circular kernel density estimation. We propose a novel circular-specific method that is based on a crossvalidation procedure with a von Mises density used as a kernel function. Using simulated data as well as real-world circular datasets, we evaluate and validate the proposed method and compare it with the existing methods.

Comparison of Gaussian and Epancehnikov Kernels

Kernel regression is a nonparametric analysis with smoothing method. Smoothing has become synonymous with nonparametric methods used to estimate functions. The purpose of smoothing is to remove variability from data that has no effect so that the characteristics of the data will appear clear. Kernel regression has a flexible form and the mathematical calculations are easy to adjust. In kernel regression, an estimator is known which is usually used to estimate the regression function, namely the Nadaraya-Watson estimator. This study aims to show how to estimate data using nonparametric regression Gaussian and Eponocvh kernels with the Nadaraya-Watson estimator and the bandwidth selection methods are "Rule of Thumb" bandwidth, Unbiased Cross Validation, Biased Cross Validation and Complete Cross Validation. The results of this study indicate that the MSE value generated by the Epanechnikov kernel function and the Gaussian kernel uses the optimal bandwidth. Statistically, the MSE value generated by the Epanechnikov kernel is almost close to the value in the Gaussian kernel, so it can be said that the MSE value produced by the two kernel functions is almost the same. Based on the plot of estimation results for the Eponocvh kernel function and the Gaussian kernel using the optimal bandwidth, it is very close, so it can be said that the use of a different kernel function with the optimal bandwidth for each of the kernel functions will produce the same estimated regression curve. The results of this study support the opinion expressed by Hastie and Tibshirani, which states that in kernel regression the selection of the smoothing parameter (bandwidth) is much more important than choosing the kernel function.

PAIREF: paired refinement also for Phenix users.

In macromolecular crystallography, paired refinement is generally accepted to be the optimal approach for the determination of the high-resolution cutoff. The software tool PAIREF provides automation of the protocol and associated analysis. Support for phenix.refine as a refinement engine has recently been implemented in the program. This feature is presented here using previously published data for thermolysin. The results demonstrate the importance of the complete cross-validation procedure to obtain a thorough and unbiased insight into the quality of high-resolution data.

A complete Fourier-synthesis-based backbone-conformation-dependent library for proteins.

While broadening the applicability of (φ/ψ)-dependent target values for the bond angles in the peptide backbone, sequence/conformation categories with too few residues to analyze via previous methods were encountered. Here, a method of describing a conformation-dependent library (CDL) using two-dimensional Fourier coefficients is reported where the number of coefficients for individual categories is determined via complete cross-validation. Sample sizes are increased further by selective blending of categories with similar patterns of conformational dependence. An additional advantage of the Fourier-synthesis-based CDL is that it uses continuous functions and has no artifactual steps near the edges of populated regions of φ/ψ space. A set of libraries for the seven main-chain bond angles, along with the ω and ζ angles, was created based on a set of Fourier analyses of 48 368 residues selected from high-resolution models in the wwPDB. This new library encompasses both trans- and cis-peptide bonds and outperforms currently used discrete CDLs.

Paired refinement under the control of PAIREF.

Crystallographic resolution is a key characteristic of diffraction data and represents one of the first decisions an experimenter has to make in data evaluation. Conservative approaches to the high-resolution cutoff determination are based on a number of criteria applied to the processed X-ray diffraction data only. However, high-resolution data that are weaker than arbitrary cutoffs can still result in the improvement of electron-density maps and refined structure models. Therefore, the impact of reflections from resolution shells higher than those previously used in conservative structure refinement should be analysed by the paired refinement protocol. For this purpose, a tool called PAIREF was developed to provide automation of this protocol. As a new feature, a complete cross-validation procedure has also been implemented. Here, the design, usage and control of the program are described, and its application is demonstrated on six data sets. The results prove that the inclusion of high-resolution data beyond the conventional criteria can lead to more accurate structure models.

Validation of the multi-mission altimeter wave height data for the Baltic Sea region

We present a complete cross-validation of significant wave heights (SWH) extracted from altimetry data from all ten existing satellites with available in situ (buoy and echosounder) wave measurements for the Baltic Sea basin. The main purpose is to select an adequate altimetry data subset for a subsequent evaluation of the wave climate. The satellite measurements with the backscatter coefficients > 13.5 cdb, errors in the SWH normalized standard deviation > 0.5 m and snapshots with centroids closer than 0.2 degrees to the land are not reliable. The ice flag usually denotes the ice concentration of > 50%. The presence of ice affects the SWH data starting from concentrations 10%, but substantial effects are only evident for concentrations > 30%. The altimetry data selected based on these criteria have very good correspondence with in situ data, except for GEOSAT Phase 1 data (1985-1989) that could not be validated. The root-mean-square difference of altimetry and in situ data is in the range of 0.23-0.37, which is significant for the Baltic Sea, compared with an average wave height of ~1 m. The bias for CRYOSAT-2, ERS-2, JASON-1/2 and SARAL data is below 0.06 m. The ENVISAT, ERS-1, GEOSAT and TOPEX satellites revealed larger biases up to 0.23 m. The SWH time series from several satellite pairs (ENVISAT/JASON-1, SARAL/JASON-2, ERS-1/TOPEX) exhibit substantial mutual temporal drift and part of them evidently are not homogeneous in time. A new high-resolution SWH data set from the SARAL satellite reveals a very good correspondence with the in situ data and with the data stream from previous satellites.

Overfitting in prediction models – Is it a problem only in high dimensions?

The growing recognition that human diseases are molecularly heterogeneous has stimulated interest in the development of prognostic and predictive classifiers for patient selection and stratification. In the process of classifier development, it has been repeatedly emphasized that in situations where the number of candidate predictor variables is much larger than the number of observations, the apparent (training set, resubstitution) accuracy of the classifiers can be highly optimistically biased and hence, classification accuracy should be reported based on evaluation of the classifier on a separate test set or using complete cross-validation. Such evaluation methods have however not been the norm in the case of low-dimensional, p<n data that arise, for example, in clinical trials when a classifier is developed on a combination of clinico-pathological variables and a small number of genetic biomarkers selected from an understanding of the biology of the disease. We undertook simulation studies to investigate the existence and extent of the problem of overfitting with low-dimensional data. The results indicate that overfitting can be a serious problem even for low-dimensional data, especially if the relationship of outcome to the set of predictor variables is not strong. We hence encourage the adoption of either a separate test set or complete cross-validation to evaluate classifier accuracy, even when the number of candidate predictor variables is substantially smaller than the number of cases.

Wrapper feature selection for small sample size data driven by complete error estimates

This paper focuses on wrapper-based feature selection for a 1-nearest neighbor classifier. We consider in particular the case of a small sample size with a few hundred instances, which is common in biomedical applications. We propose a technique for calculating the complete bootstrap for a 1-nearest-neighbor classifier (i.e., averaging over all desired test/train partitions of the data). The complete bootstrap and the complete cross-validation error estimate with lower variance are applied as novel selection criteria and are compared with the standard bootstrap and cross-validation in combination with three optimization techniques – sequential forward selection (SFS), binary particle swarm optimization (BPSO) and simplified social impact theory based optimization (SSITO). The experimental comparison based on ten datasets draws the following conclusions: for all three search methods examined here, the complete criteria are a significantly better choice than standard 2-fold cross-validation, 10-fold cross-validation and bootstrap with 50 trials irrespective of the selected output number of iterations. All the complete criterion-based 1NN wrappers with SFS search performed better than the widely-used FILTER and SIMBA methods. We also demonstrate the benefits and properties of our approaches on an important and novel real-world application of automatic detection of the subthalamic nucleus.

Конструктивные оценки полного скользящего контроля для пороговой классификации

Конструктивные оценки полного скользящего контроля для пороговой классификации

NP-completeness of the problem of prototype selection in the nearest neighbor method

The problem of selecting of prototypes is to select a subset in the learning sample for which the set of minimum cardinality would provide the optimum of a given learning quality functional. In this article the problem of classification is considered in two classes, the method of classification by nearest neighbor, and three functional characteristics: the frequency of errors on the entire sample, a cross validation with one separated object, and a complete cross validation with k separated objects. It is shown that the problem of selection of prototypes in all three cases is NP-complete, which justifies the use of well-known heuristic methods for the prototype search.

Prototype sample selection based on minimization of the complete cross validation functional

A method of prototype sample selection from a training set for a classifier of K nearest neighbors (KNN), based on minimization of the complete cross validation functional, is proposed. The optimization leads to reduction of the training set to the minimum sufficient number of prototypes, removal (censoring) of noise samples, and improvement of the generalization ability, simultaneously.

Development and Validation of Predictive Indices for a Continuous Outcome Using Gene Expression Profiles

There have been relatively few publications using linear regression models to predict a continuous response based on microarray expression profiles. Standard linear regression methods are problematic when the number of predictor variables exceeds the number of cases. We have evaluated three linear regression algorithms that can be used for the prediction of a continuous response based on high dimensional gene expression data. The three algorithms are the least angle regression (LAR), the least absolute shrinkage and selection operator (LASSO), and the averaged linear regression method (ALM). All methods are tested using simulations based on a real gene expression dataset and analyses of two sets of real gene expression data and using an unbiased complete cross validation approach. Our results show that the LASSO algorithm often provides a model with somewhat lower prediction error than the LAR method, but both of them perform more efficiently than the ALM predictor. We have developed a plug-in for BRB-ArrayTools that implements the LAR and the LASSO algorithms with complete cross-validation.

Clustering Mass Spectral Peaks Increases Recognition Accuracy and Stability of SVM-based Feature Selection

Mass spectral profiling of serum or plasma is one of the tools widely used to make experimental diagnostic systems for different cancer types. In this approach, a set of discriminatory peaks serves as a multiplex cancer biomarker. Hence, adequate selection of peaks is a crucial stage in the development of diagnostic rule. In the present paper we propose using sequential filter and wrapper feature selection in a complete cross-validation scheme with feature selection performed at each run of crossvalidation separately. Filter feature selection is represented by hierarchical cluster analysis; recursive feature elimination coupled with support vector machine is utilized as a wrapper feature selection method. The method performance is demonstrated on previously obtained dataset with ovarian cancer and non-cancer sera. Application of our approach led to a slight but statistically significant increase in accuracy. Peak clustering favoured more stable results of feature selection and provided a biological meaning to selected m/z values. We recommend clustering of peaks as a filter dimensionality reduction for further use in mass spectral studies.

Classification across gene expression microarray studies.

BackgroundThe increasing number of gene expression microarray studies represents an important resource in biomedical research. As a result, gene expression based diagnosis has entered clinical practice for patient stratification in breast cancer. However, the integration and combined analysis of microarray studies remains still a challenge. We assessed the potential benefit of data integration on the classification accuracy and systematically evaluated the generalization performance of selected methods on four breast cancer studies comprising almost 1000 independent samples. To this end, we introduced an evaluation framework which aims to establish good statistical practice and a graphical way to monitor differences. The classification goal was to correctly predict estrogen receptor status (negative/positive) and histological grade (low/high) of each tumor sample in an independent study which was not used for the training. For the classification we chose support vector machines (SVM), predictive analysis of microarrays (PAM), random forest (RF) and k-top scoring pairs (kTSP). Guided by considerations relevant for classification across studies we developed a generalization of kTSP which we evaluated in addition. Our derived version (DV) aims to improve the robustness of the intrinsic invariance of kTSP with respect to technologies and preprocessing.ResultsFor each individual study the generalization error was benchmarked via complete cross-validation and was found to be similar for all classification methods. The misclassification rates were substantially higher in classification across studies, when each single study was used as an independent test set while all remaining studies were combined for the training of the classifier. However, with increasing number of independent microarray studies used in the training, the overall classification performance improved. DV performed better than the average and showed slightly less variance. In particular, the better predictive results of DV in across platform classification indicate higher robustness of the classifier when trained on single channel data and applied to gene expression ratios.ConclusionsWe present a systematic evaluation of strategies for the integration of independent microarray studies in a classification task. Our findings in across studies classification may guide further research aiming on the construction of more robust and reliable methods for stratification and diagnosis in clinical practice.

An Eight-Gene Blood Expression Profile Predicts the Response to Infliximab in Rheumatoid Arthritis

BackgroundTNF alpha blockade agents like infliximab are actually the treatment of choice for those rheumatoid arthritis (RA) patients who fail standard therapy. However, a considerable percentage of anti-TNF alpha treated patients do not show a significant clinical response. Given that new therapies for treatment of RA have been recently approved, there is a pressing need to find a system that reliably predicts treatment response. We hypothesized that the analysis of whole blood gene expression profiles of RA patients could be used to build a robust predictor to infliximab therapy.Methods and FindingsWe performed microarray gene expression analysis on whole blood RNA samples from RA patients starting infliximab therapy (n = 44). The clinical response to infliximab was determined at week 14 using the EULAR criteria. Blood cell populations were determined using flow cytometry at baseline, week 2 and week 14 of treatment. Using complete cross-validation and repeated random sampling we identified a robust 8-gene predictor model (96.6% Leave One Out prediction accuracy, P = 0.0001). Applying this model to an independent validation set of RA patients, we estimated an 85.7% prediction accuracy (75–100%, 95% CI). In parallel, we also observed a significantly higher number of CD4+CD25+ cells (i.e. regulatory T cells) in the responder group compared to the non responder group at baseline (P = 0.0009).ConclusionsThe present 8-gene model obtained from whole blood expression efficiently predicts response to infliximab in RA patients. The application of the present system in the clinical setting could assist the clinician in the selection of the optimal treatment strategy in RA.

Solution structure and sugar-binding mechanism of mouse latrophilin-1 RBL: a 7TM receptor-attached lectin-like domain.

SummaryLatrophilin-1 (Lat-1), a target receptor for α-Latrotoxin, is a putative G protein-coupled receptor implicated in synaptic function. The extracellular portion of Lat-1 contains a rhamnose binding lectin (RBL)-like domain of unknown structure. RBL domains, first isolated from the eggs of marine species, are also found in the ectodomains of other metazoan transmembrane proteins, including a recently discovered coreceptor of the neuronal axon guidance molecule SLT-1/Slit. Here, we describe a structure of this domain from the mouse Lat-1. RBL adopts a unique α/β fold with long structured loops important for monosaccharide recognition, as shown in the structure of a complex with L-rhamnose. Sequence alignments and mutagenesis show that residues important for carbohydrate binding are often absent in other receptor-attached examples of RBL, including the SLT-1/Slit coreceptor. We postulate that this domain class facilitates direct protein-protein interactions in many transmembrane receptors.

Cross-platform performance of genes predictive of pathologic response to doxorubicin-paclitaxel containing regimens

11061 Background: Estrogen Receptor (ER) positive breast cancers have shown a lower rate of pathologic complete response (pCR) after neoadjuvant chemotherapy (NC) than ER negative tumors. However, the long-term benefits of adjuvant chemotherapy are evident in both subsets according to ER status. This might suggest some benefit for patients without pCR. Methods: Fifty-eight patients with ER positive locally-advanced breast cancer received NC with doxorubicin and paclitaxel (ATx3 ⟶wTx12) (Gianni L JCO 2005). RT-PCR assays were used to quantify expression of 384 genes on paraffin-embedded core biopsies. To reduce the possible confounding effect of the intermediate response phenotype, we developed a predictor comparing best responders (BR=pCR + breast residual ≤3 mm) versus gross residual disease (RD). An internal accuracy estimate was performed using complete leave-one-out cross validation. As external validation we applied the predictor’s genes derived from the Milan cases to an Affymetrix-based dataset of MD Anderson (Hess KR JCO 2006). The main differences of this dataset were the stage (I-III) and the NC regimen (Tx4 or wTx12 ⟶FACx4). A model based on the genes identified from the Milan cases was developed on a training set of MD Anderson cases and then applied in the validation set of the MD Anderson. All the analyses were performed by BRB-ArrayTools. Results: Nine best responder and 19 RD cases were identified. A 32 gene model was developed and complete cross-validation provided performance estimates of Accuracy 96%, Sensitivity 1.0, Specificity 0.95, PPV 0.9, NPV 1.0. Fifty-seven corresponding probes were identified on the Affymetrix platform. Nine pCR (breast) and 37 not- pCR ER positive cases were available in the training set. A 6 probes model (5 genes: TOP2A, MELK, BIRC5, KIFC1, BUB1 all higher in pCR- Breast group) was selected in the training set for its high sensitivity. This model was applied to 4 pCR and 30 not-pCR cases in the validation set and its performance was: Accuracy: 82%, Sens: 1, Spec: 0.8, PPV: 0.4, NPV:1. Conclusions: The identified genes provided good predictive accuracy in slightly dissimilar clinical cases from different institutions assayed on a different platform. An external validation of the identified models is warranted. No significant financial relationships to disclose.

Analysis of Gene Expression Data Using BRB-Array Tools

BRB-ArrayTools is an integrated software system for the comprehensive analysis of DNA microarray experiments. It was developed by professional biostatisticians experienced in the design and analysis of DNA microarray studies and incorporates methods developed by leading statistical laboratories. The software is designed for use by biomedical scientists who wish to have access to state-of-the-art statistical methods for the analysis of gene expression data and to receive training in the statistical analysis of high dimensional data. The software provides the most extensive set of tools available for predictive classifier development and complete cross-validation. It offers extensive links to genomic websites for gene annotation and analysis tools for pathway analysis. An archive of over 100 datasets of published microarray data with associated clinical data is provided and BRB-ArrayTools automatically imports data from the Gene Expression Omnibus public archive at the National Center for Biotechnology Information.

Structural Basis for the Interaction of the Myosin Light Chain Mlc1p with the Myosin V Myo2p IQ Motifs

Calmodulin, regulatory, and essential myosin light chain are evolutionary conserved proteins that, by binding to IQ motifs of target proteins, regulate essential intracellular processes among which are efficiency of secretory vesicles release at synapsis, intracellular signaling, and regulation of cell division. The yeast Saccharomyces cerevisiae calmodulin Cmd1 and the essential myosin light chain Mlc1p share the ability to interact with the class V myosin Myo2p and Myo4 and the class II myosin Myo1p. These myosins are required for vesicle, organelle, and mRNA transport, spindle orientation, and cytokinesis. We have used the budding yeast model system to study how calmodulin and essential myosin light chain selectively regulate class V myosin function. NMR structural analysis of uncomplexed Mlc1p and interaction studies with the first three IQ motifs of Myo2p show that the structural similarities between Mlc1p and the other members of the EF-hand superfamily of calmodulin-like proteins are mainly restricted to the C-lobe of these proteins. The N-lobe of Mlc1p presents a significantly compact and stable structure that is maintained both in the free and complexed states. The Mlc1p N-lobe interacts with the IQ motif in a manner that is regulated both by the IQ motifs sequence as well as by light chain structural features. These characteristic allows a distinctive interaction of Mlc1p with the first IQ motif of Myo2p when compared with calmodulin. This finding gives us a novel view of how calmodulin and essential light chain, through a differential binding to IQ1 of class V myosin motor, regulate this activity during vegetative growth and cytokinesis.

Can Multivariable Risk-Benefit Profiling Be Used to Select Treatment-Favorable Patients for Thrombolysis in Stroke in the 3- to 6-Hour Time Window?

The Stroke-Thrombolytic Predictive Instrument (Stroke-TPI) uses multivariate equations to predict outcomes with and without thrombolysis. We sought to examine whether such a multivariate predictive instrument might be useful in selecting patients with a favorable risk-benefit treatment profile for therapy after 3 hours. We explored outcomes in patients from 5 major randomized clinical trials testing intravenous recombinant tissue plasminogen activator (rt-PA) classified by the Stroke-TPI as "treatment-favorable" or "treatment-unfavorable." We used iterative bootstrap re-sampling to estimate how such a model would perform on independent test data. Among patients treated within the 3- to 6-hour window, 67% of patients were classified by Stroke-TPI predicted outcomes as "treatment-favorable" and 33% were classified as "treatment-unfavorable." Outcomes in the treatment-favorable group demonstrated benefit for thrombolysis (modified Rankin Scale score < or =1: 44.0% with rt-PA versus 34.2 with placebo, P=0.005), whereas harm was demonstrated in the treatment-unfavorable group (modified Rankin Scale score < or =1: 31.3% with rt-PA versus 38.3% with placebo; P=0.004). Bootstrap resampling with complete cross-validation showed that the absolute margin of benefit in the treatment-favorable group diminished on average by 36% between derivation and independent validation sets, but still represented a significant tripling of improvement in benefit compared with conventional inclusion criteria (5.2% [interquartile range, 1.7% to 8.6%] versus 1.8% [interquartile range, -0.5 to 4.1], P<0.0001). Such multivariable risk-benefit profiling may be useful in the selection of acute stroke patients for rt-PA therapy even more than 3 hours after symptom onset. Prospective testing is indicated.