Abstract
The secondary structure prediction (SSP) of proteins has long been an essential structural biology technique with various applications. Despite its vital role in many research and industrial fields, in recent years, as the accuracy of state-of-the-art secondary structure predictors approaches the theoretical upper limit, SSP has been considered no longer challenging or too challenging to make advances. With the belief that the substantial improvement of SSP will move forward many fields depending on it, we conducted this study, which focused on three issues that have not been noticed or thoroughly examined yet but may have affected the reliability of the evaluation of previous SSP algorithms. These issues are all about the sequence homology between or within the developmental and evaluation datasets. We thus designed many different homology layouts of datasets to train and evaluate SSP prediction models. Multiple repeats were performed in each experiment by random sampling. The conclusions obtained with small experimental datasets were verified with large-scale datasets using state-of-the-art SSP algorithms. Very different from the long-established assumption, we discover that the sequence homology between query datasets for training, testing, and independent tests exerts little influence on SSP accuracy. Besides, the sequence homology redundancy between or within most datasets would make the accuracy of an SSP algorithm overestimated, while the redundancy within the reference dataset for extracting predictive features would make the accuracy underestimated. Since the overestimating effects are more significant than the underestimating effect, the accuracy of some SSP methods might have been overestimated. Based on the discoveries, we propose a rigorous procedure for developing SSP algorithms and making reliable evaluations, hoping to bring substantial improvements to future SSP methods and benefit all research and application fields relying on accurate prediction of protein secondary structures.
Highlights
The secondary structure prediction (SSP) of proteins is a technique to predict the backbone conformation of proteins based only on amino acid sequences
In the development and evaluation of machine-learning-based SSP methods, it has long been generally assumed that the homology between query datasets for training and test will affect the reliability of predictions
Perhaps different from most SSP researchers’ expectations, Fig 1 shows that the homology between training and testing query sets has little effect on SSP accuracy
Summary
The secondary structure prediction (SSP) of proteins is a technique to predict the backbone conformation of proteins based only on amino acid sequences. By investigating the influence of the sequence redundancy of developmental datasets on SSP accuracy, this work aims to program a sophisticated SSP dataset preparation procedure and a rigorous evaluation strategy to prevent over/underestimating an SSP method during development. Eight-state predictors use the 8 secondary structure elements (SSE) described by DSSP [24]. In 1999, PSIPRED [27], a neural network machine-learning method, first utilized the position-specific scoring matrices (PSSM) generated by PSI-BLAST [28] to be the main feature set and made Q3 achieve a new high, 76.5%. State-of-the-art algorithms, such as RaptorX, SpineX, Scorpion, Spider2/3, DeepCNF, MUFOLD-SS, NetSurfP-2, and Porter 5 [30,31,32,33,34,35,36,37,38], all used PSSM, and their Q3 and Q8 accuracy approximately fell in 81–85% and 71–75%, respectively
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
