A machine learning model to determine the accuracy of variant calls in capture-based next generation sequencing

Alicia Y. Zhou,Gilad Mishne,Jeroen van den Akker,Anjali D. Zimmer

doi:10.1186/s12864-018-4659-0

Abstract

BackgroundNext generation sequencing (NGS) has become a common technology for clinical genetic tests. The quality of NGS calls varies widely and is influenced by features like reference sequence characteristics, read depth, and mapping accuracy. With recent advances in NGS technology and software tools, the majority of variants called using NGS alone are in fact accurate and reliable. However, a small subset of difficult-to-call variants that still do require orthogonal confirmation exist. For this reason, many clinical laboratories confirm NGS results using orthogonal technologies such as Sanger sequencing. Here, we report the development of a deterministic machine-learning-based model to differentiate between these two types of variant calls: those that do not require confirmation using an orthogonal technology (high confidence), and those that require additional quality testing (low confidence). This approach allows reliable NGS-based calling in a clinical setting by identifying the few important variant calls that require orthogonal confirmation.ResultsWe developed and tested the model using a set of 7179 variants identified by a targeted NGS panel and re-tested by Sanger sequencing. The model incorporated several signals of sequence characteristics and call quality to determine if a variant was identified at high or low confidence. The model was tuned to eliminate false positives, defined as variants that were called by NGS but not confirmed by Sanger sequencing. The model achieved very high accuracy: 99.4% (95% confidence interval: +/− 0.03%). It categorized 92.2% (6622/7179) of the variants as high confidence, and 100% of these were confirmed to be present by Sanger sequencing. Among the variants that were categorized as low confidence, defined as NGS calls of low quality that are likely to be artifacts, 92.1% (513/557) were found to be not present by Sanger sequencing.ConclusionsThis work shows that NGS data contains sufficient characteristics for a machine-learning-based model to differentiate low from high confidence variants. Additionally, it reveals the importance of incorporating site-specific features as well as variant call features in such a model.

Highlights

Generation sequencing (NGS) has become a common technology for clinical genetic tests
Model performance We developed and trained a machine learning model to differentiate between high confidence Next generation sequencing (NGS) calls that do not require orthogonal confirmation and low confidence NGS calls that do require confirmation
Due to the high cost of false positive predictions, we set a strict threshold to achieve a 100% true positive prediction rate: any NGS call that the model predicts as high confidence is confirmed, but some of the NGS calls that the model indicates as low confidence are confirmed present too

Summary

Introduction

Generation sequencing (NGS) has become a common technology for clinical genetic tests. We report the development of a deterministic machine-learning-based model to differentiate between these two types of variant calls: those that do not require confirmation using an orthogonal technology (high confidence), and those that require additional quality testing (low confidence). This approach allows reliable NGS-based calling in a clinical setting by identifying the few important variant calls that require orthogonal confirmation. Developing a robust and reliable method for differentiating between these high and low confidence variant calls is essential to create an NGS assay with the highest possible accuracy for clinical testing Such a method could pinpoint the variants which require orthogonal confirmation, ensuring that those variants are reported correctly. Other multi-parameter algorithms, such as Variant Quality Score Recalibration, have been developed to assay the quality of NGS variant calls but are designed for whole exome sequencing and whole genome sequencing-sized data sets, and have not been optimized for smaller targeted panels [10]

Methods

Results

Discussion

Conclusion