Independent Validation Tests Research Articles

Combination of Multiple Serum Markers Using an Artificial Neural Network to Improve Specificity in Discriminating Malignant from Benign Pelvic Masses

A panel of four selected tumor markers, CA 125 II, CA 72-4, CA 15-3, and lipid-associated sialic acid, was analyzed collectively using an artificial neural network (ANN) approach to differentiate malignant from benign pelvic masses. A dataset of 429 patients, 192 of whom had malignant histology, was retrospectively used in the study. A prototype ANN classifier was developed using a subset of the data which included 73 patients with malignant conditions and 101 patients with benign conditions. The ANN classifier demonstrated a much improved specificity over that of the assay CA 125 II alone (87.5% vs 68.4%) while maintaining a statistically comparable sensitivity (79.0% vs 82.4%) in discriminating malignant from benign pelvic masses in an independent validation test using data from the remaining 255 patients which had been set aside and kept blind to the developers of the ANN system. A similar improvement in specificity was observed among patients under 50 years of age (82.3% vs 62.0%). The ANN system was further tested using additional serum specimens collected from 196 apparently healthy women. The ANN system had a specificity of 100.0% compared to that of 94.8% with the assay CA 125 II alone.

A comparison of the weights-of-evidence method and probabilistic neural networks

The need to integrate large quantities of digital geoscience information to classify locations as mineral deposits or nondeposits has been met by the weights-of-evidence method in many situations. Widespread selection of this method may be more the result of its ease of use and interpretation rather than comparisons with alternative methods. A comparison of the weights-of-evidence method to probabilistic neural networks is performed here with data from Chisel Lake-Andeson Lake, Manitoba, Canada. Each method is designed to estimate the probability of belonging to learned classes where the estimated probabilities are used to classify the unknowns. Using these data, significantly lower classification error rates were observed for the neural network, not only when test and training data were the same (0.02 versus 23%), but also when validation data, not used in any training, were used to test the efficiency of classification (0.7 versus 17%). Despite these data containing too few deposits, these tests of this set of data demonstrate the neural network's ability at making unbiased probability estimates and lower error rates when measured by number of polygons or by the area of land misclassified. For both methods, independent validation tests are required to ensure that estimates are representative of real-world results. Results from the weights-of-evidence method demonstrate a strong bias where most errors are barren areas misclassified as deposits. The weights-of-evidence method is based on Bayes rule, which requires independent variables in order to make unbiased estimates. The chi-square test for independence indicates no significant correlations among the variables in the Chisel Lake–Andeson Lake data. However, the expected number of deposits test clearly demonstrates that these data violate the independence assumption. Other, independent simulations with three variables show that using variables with correlations of 1.0 can double the expected number of deposits as can correlations of −1.0. Studies done in the 1970s on methods that use Bayes rule show that moderate correlations among attributes seriously affect estimates and even small correlations lead to increases in misclassifications. Adverse effects have been observed with small to moderate correlations when only six to eight variables were used. Consistent evidence of upward biased probability estimates from multivariate methods founded on Bayes rule must be of considerable concern to institutions and governmental agencies where unbiased estimates are required. In addition to increasing the misclassification rate, biased probability estimates make classification into deposit and nondeposit classes an arbitrary subjective decision. The probabilistic neural network has no problem dealing with correlated variables—its performance depends strongly on having a thoroughly representative training set. Probabilistic neural networks or logistic regression should receive serious consideration where unbiased estimates are required. The weights-of-evidence method would serve to estimate thresholds between anomalies and background and for exploratory data analysis.

Validation and verification of regression in small data sets

Four different methods of using small data sets in multivariate modelling are compared w.r.t. predictive precision in the long-run. The modelling in this case concerns multivariate calibration: ŷ= f( X). The study consists of a Monte Carlo simulation within a large data base of real data; X=NIR reflectance spectra and y=protein percentage, measured in 922 whole maize plant samples. Small data sets (40–120 objects) were repeatedly selected at random from the data base, each time simulating the situation of having only a small set of samples available for estimating, optimizing and assessing the calibration model. The `true' apparent prediction error was each time controlled in the remaining data base. This was replicated 100 times in order to study the statistical performance of the four different validation methods. In each Monte Carlo replicate, the splitting of the available data set into calibration set and test set was compared to full cross validation. The results demonstrated that removing samples from an already limited set of available samples to an independent VALIDATION TEST SET seriously reduced the predictive performance of the calibrated models, and at the same time gave uncertain, systematically over-optimistic assessment of the models' predictive performance. Full CROSS VALIDATION gave improved predictive performance, and gave only slightly over-optimistic assessment of this predictive performance. Further removal of even more of the available samples for use in an independent VERIFICATION TEST SET gave in-the-long-run correct, although uncertain estimates of the predictive performance of the calibrated models, but this performance level had seriously deteriorated. Alternative verification of the model's predictive performance by the method of CROSS VERIFICATION gave results very similar to those of the cross validation. These results from real data correspond closely to previous findings for artificially simulated data. It appears that full cross validation is superior to both the use of independent validation test set and independent verification test set.

The seasonal risk of pediatric/juvenile acute lymphocytic leukemia in the United States

Acute lymphocytic leukemia (ALL) occurring in pediatric and juvenile patients of nine U.S.A. locations during 1973 through 1980 was investigated by epidemiologic methodology. Time series periodic regression analysis of the monthly cumulative risk for ages 0–19 years revealed evidence of trimodal periodicity at seven of the nine locations tested. Combined analyses revealed a significant effect of latitude in causing asynchrony of the trimodal patterns observed. The average risk maxima occurred in April, August, and December for U.S.A. locations with latitude above 40 ° (Seattle, Nebraska, Iowa, Detroit, and Connecticut), and in February, July, and October for other locations (San Francisco, Utah, New Mexico, and Atlanta). Because periodic regression analysis is a complex method, independent validation tests were performed to supplement the original results. Here, the best Symmetric trimodal sine curve model of each location was utilized to predict the average pattern among the remaining locations of similar latitude. These validation tests supported the existence of similar trimodal patterns for all of the southern locations, and all of the northern locations except Connecticut. The observed peaks in monthly ALL risk coincide with seasonal elevations in the rates of allergenic and infectious diseases, elements of which are capable of promoting lymphocytic proliferation and transformation. Annual disease rates were significantly lower in populations with a high proportion of Blacks or American Indians compared to predominantly Caucasian populations. The possibility that seasonal factors (environmental allergens and/or infectious agents) promote leukomogenesis via indirect mechanisms is briefly discussed.