Evaluation of template‐based modeling in CASP13

Tristan I Croll,Andriy Kryshtafovych,Randy J Read,Massimo D Sammito

doi:10.1002/prot.25800

Abstract

Performance in the template‐based modeling (TBM) category of CASP13 is assessed here, using a variety of metrics. Performance of the predictor groups that participated is ranked using the primary ranking score that was developed by the assessors for CASP12. This reveals that the best results are obtained by groups that include contact predictions or inter‐residue distance predictions derived from deep multiple sequence alignments. In cases where there is a good homolog in the wwPDB (TBM‐easy category), the best results are obtained by modifying a template. However, for cases with poorer homologs (TBM‐hard), very good results can be obtained without using an explicit template, by deep learning algorithms trained on the wwPDB. Alternative metrics are introduced, to allow testing of aspects of structural models that are not addressed by traditional CASP metrics. These include comparisons to the main‐chain and side‐chain torsion angles of the target, and the utility of models for solving crystal structures by the molecular replacement method. The alternative metrics are poorly correlated with the traditional metrics, and it is proposed that modeling has reached a sufficient level of maturity that the best models should be expected to satisfy this wider range of criteria.

Highlights

Even though the worldwide Protein Data Bank[1] continues to expand quickly, growth in this database is outpaced by the growth in genomic information, leading to an escalation in the need for protein structure modeling
In CASP7 we introduced a metric scoring individual model predictions based on their usefulness in molecular replacement (MR).[6]
Comparing this to the SCASP12-accuracy estimates (ASE) score aggregated over all models (Figure 5) revealed that high Template-based modeling (TBM) scores are no guarantee of good local geometry—the otherwise field-leading contributions from the Zhang lab[15] score in the bottom quintile of all groups by the torsion-only metric

Summary

| INTRODUCTION

Even though the worldwide Protein Data Bank[1] (wwPDB) continues to expand quickly, growth in this database is outpaced by the growth in genomic information, leading to an escalation in the need for protein structure modeling. The conventional metrics, and the new torsion metrics, evaluate respectively the correctness of the predicted folds and the adherence of predicted fine-scale features to those observed in the target structures Users of such models will primarily be interested in their utility for particular purposes, such as providing targets for the design of new therapeutics or explaining the impact of mutations found in inherited diseases. In CASP7 we introduced a metric scoring individual model predictions based on their usefulness in MR.[6] As a result of continuing improvement in structure prediction, the use of TBM to improve MR models has been greatly expanding in recent years.[13,14,15,16] TBM is the focus of this work, it should be noted that free modeling of whole proteins or fragments can yield useful models for MR, under favorable circumstances of relatively small proteins and high-resolution data.[17,18,19,20]

| MATERIALS AND METHODS

| RESULTS

| DISCUSSION

| CONCLUSIONS

CONFLICT OF INTEREST