Best practices in biomarker performance evaluation: Common tool and standards with neuropredict

Abstract

AbstractBackgroundThe need for standards in neuroimaging research has been clearly recognized recently in a number of research areas [1]. However, the recommendations made so far do not sufficiently address the complexities involved in the full PRedictive ANAlysis Workflow (PRANAW). In this presentation, I would like to highlight the need, and present some guidelines, towards standardizing generic predictive analysis, that is increasingly becoming common in many areas of neuroscience research. A key application to the Alzheimer’s community being the performance evaluation of biomarkers.MethodIncomplete/non‐standard reporting of the PRANAW prevents from engaging in any meaningful comparisons. Moreover, given its complexity, it's easy to make choices for individual steps (such as how one performs cross‐validation) in PRANAW, that may not fit well in the full workflow, or produce less accurate results as a whole (with variations in bias or variance). Lack of standards in PRANAW led to the publication of hundreds of papers without clear insight emerging towards whether there is a reliable biomarker for a particular application, and their relative performances.ResultI will present some ideas to facilitate standardized and easy predictive analysis: 1) A tool (accessible to all skill levels) to perform predictive analysis that implements the best practices, as well as standardized reports. A preliminary attempt in this direction is neuropredict [2]. 2) A cloud repository to receive various standardized reports (either using the tool above, or user's own). 3) A web portal (for visualization and comparison) which lets users and reviewers compare new methods to those in the database.ConclusionI believe the publicly available and openly validated tools/infrastructure is necessary now, and prove to be of great value in the long run. Moreover, this would enable a number of Alzheimer’s researchers without native expertise in machine learning or informatics to apply the best practices in biomarker performance evaluation and report more stable and reproducible results.