Cross-validating models of continuous data from simulation and experiment by using linear regression and artificial neural networks

Abstract

We are increasingly surrounded by sensors gathering massive amounts of data, and patterns in continuous variables are often discovered by using artificial neural networks (ANN), while linear regression (LR) is useful for detecting linear relationships. LR also provide preliminary estimates of potentially complex associations, and serve as a benchmark for the performance of ANNs. We show that while cross-validation (CV) is indispensable for insuring the robustness of the discovered patterns, it systematically leads, when combined with LR, to specific artefacts that underestimate the extent of the associations between predictor and target variables. We explain how this previously unnoticed type of artefact arises specifically from the combination of CV with LR and does not affect non-linear methods such as ANN. We also demonstrate through simulations that ANN were able to discover a wide range of complex associations missed by LR. The results were confirmed by the analysis of physiological, behavioural and subjective data collected from N = 31 human subjects performing laparoscopy training experiments.