Report on the Barbados Workshop

Abstract

The HUPO Workshop on “Test Samples for Quantitative Proteomics” was held in Barbados January 9–16, 2009 at the McGill Bellairs Research Institute co-chaired by Drs. Jan van Oostrum (Chair, HUPO Industrial Advisory Board) and John Bergeron (McGill University). The Workshop was organized by Ms. Sylvie Ouellette (Administrative Coordinator, HUPO). The Workshop built on the foundation of a HUPO test samples effort in press in Nature Methods (Ref. 1Bell A.W. Deutsch E.W. Au C.E. Kearney R.E. Sechi S. Nilsson T. Bergeron J.J.M. HUPO Test Sample Working Group A HUPO test sample study reveals common problems in mass spectrometry-based proteomics.Nat. Methods. 2009; 10.1038/nmeth.1333Crossref Scopus (272) Google Scholar). This paper demonstrated how 27 international labs selected by a HUPO Test Samples Committee were able to ensure 100% success by all participating labs for the characterization of 20 proteins at a 5 picomole abundance for each protein. For 20 of the 27 labs iterative guidance was necessary for the 100% successful outcome. A further test for stochastic sampling of tryptic peptides involved the characterization of 22 tryptic peptides of mass 1250 ± 5 Da found among the 20 proteins (all of different amino acid sequence and at least one per protein). Here, the 27 labs were less successful with only 4 labs generating sufficient data for all 22 peptides and only one of these 4 labs able to report on the characterization of the 22 peptides. Several bottlenecks were uncovered in this study; effective education and training remains the most important bottleneck for a successful proteomics pipeline.In the January 2009 workshop, several presentations dealt with relative quantification efforts using two-dimensional gels and imaging as well as the advantages and disadvantages of commercially available computer software packages to standardize such data sets (Will Dracup, Nonlinear Dynamics; Tom Slyker, Bio-Rad; Francesc Canals, ProteoRed, and Ola Forsstrom-Olsson, Ludesi). As well, the standardization of the methodology and operating procedure processes involved were presented (Judit Nagy, Imperial College; Jun Wheeler, National Institute for Biological Standards and Control).Challenges of the HUPO equimolar test sample reported and discussed included sample distribution to individual analysis (Alex Bell, McGill University) and data reprocessing and a centralized analysis (Eric Deutsch, Institute for Systems Biology). As well, heat maps which visualized the frequency with which tryptic peptides were observed as deduced from the tandem mass spectra reported for each lab were found to be a striking visual method to represent the data (Rob Kearney, McGill University). The importance of test sample preparation (high protein purity) and centralized analysis of all data were emphasized for quality control and the discovery of bottlenecks related to high throughput proteomics. The importance of highly curated and well annotated data bases was highlighted by Amos Bairoch (Swiss Institute of Bioinformatics), who presented the most recent version of the human Swiss-Prot protein data base. Remarkably, nearly 8,000 of the predicted products of the 20,300 protein coding human genes have not yet been observed at the protein level.For tandem mass spectrometry, insightful experiences were shared by an industrial representative (Martha Stapels, Waters), academic investigators (Bruno Domon, Institute of Molecular Systems Biology and Gilles Lajoie, University of Western Ontario), and colleagues participating in organized efforts such as Clinical Proteomic Technology Assessment for Cancer (CPTAC) (David Tabb, Vanderbilt University) and The Association of Biomolecular Resource Facilities (ABRF) (Mike MacCoss, University of Washington). The pioneering and detailed studies from The Association of Biomolecular Resource Facilities (ABRF) illustrated the challenges of establishing a successful quantitation effort with test samples. The most recent study (ABRF-PRG07: Advanced Quantitative Proteomics Study) showed that of 36 participating labs, none was 100% successful. Henry Duewel (Sigma-Aldrich) recommended that the various or most commonly utilized proteomics platforms (e.g. protein identification, quantitation, and post-translational modification characterization) might each require a unique, specially tailored set of reference materials. Henry Duewel highlighted some of the insights and experiences that Sigma-Aldrich has gained in preparing and commercializing their Equimolar and Dynamic Range Protein Standards (Universal Proteomics Standards).Tommy Nilsson (Gothenburg University) summarized efforts of ABRF over the past 10 years and the HUPO equimolar test samples study to conclude that the proteomics community still has a lot to prove with respect to comprehensive and accurate reporting. There is a need for better databases that are compatible with mass spectrometry-based proteomics; preferably gene-centric in design such that there exists only one top entry for each of the 20,300 human protein products. Isoforms and splice variants should then be assigned to each of these 20,300 proteins. Such gene-centric databases are likely to increase the fidelity of mass spectrometry-based proteomics. To underscore the need, Nilsson highlighted the complete failure of the entire proteomics community to provide even a handful of Food and Drug Administration approved biomarkers for disease. “A lot of work is still needed”, he concluded.From the investigators present at the HUPO workshop, proposals for a quality assurance (self-assessment) experiment and for a relative abundance test sample were put forward. There was a consensus for the use of stable isotope labeled peptides and for recombinant proteins labeled with 13C and 15N isotopes spiked into complex mixtures. Along with light versions of recombinant proteins spiked into previously characterized complex mixtures, this was considered as a reasonable test sample to ensure that the community can achieve 100% success in quantitative proteomics. Whether the lower limit of 0.1 fmol or 1 fmol is achievable by all was debated.Discussion on the need for a standardized trypsinization methodology was led by Christoph Borchers (University of Victoria) who showed that different digestion procedures and conditions have diverse effects for individual proteins. Christoph Borchers also illustrated the efficacy of quantitation of multiple proteins in human plasma by Multi-Reaction Monitoring.Reproducibility was discussed by Will Dracup (NonLinear Dynamics) who defined the problems and proposed solutions for cross-lab reproducibility and quality control. The pitfalls of only focusing on reproducibility without having appropriate metrics for data correctness were presented by Ola Forsstrom-Olsson (Ludesi), who proposed alternative metrics for quality control. Jan van Oostrum (Zeptosens, co-chair) used examples taken from reverse phase protein microarray experiments to highlight that reproducibility criteria in an experimental science like proteomics should be a common standard. Jim Dasch (Protein Forest) presented technology to reduce peptide complexity in order to improve on the stochastic aspect of sampling at the mass spectrometer.The importance of bioinformatics in data analysis and reproducibility in quantitative proteomics was not ignored. Investigators presented who were responsible for data deposition and a centralized analysis of the data from the 27 labs for the HUPO equimolar test samples paper (1Bell A.W. Deutsch E.W. Au C.E. Kearney R.E. Sechi S. Nilsson T. Bergeron J.J.M. HUPO Test Sample Working Group A HUPO test sample study reveals common problems in mass spectrometry-based proteomics.Nat. Methods. 2009; 10.1038/nmeth.1333Crossref Scopus (272) Google Scholar).The importance of Tranche (Phil Andrews, University of Michigan) and PRIDE (Lennart Martens, EBI) as data repositories and the preferred formats for submission were indicated. For reprocessing, raw mass spectrometer output files in the native instrument vendor format were preferred with the exception of data acquired in MSe mode (2Geromanos S.J. Vissers J.P.C. Silva J.C. Dorschel C.A. Li G.-Z. Gorenstein M.V. Bateman R.H. Langridge J.I. The detection, correlation and comparison of peptide precursor and product ions from data independent LC-MS with data dependant LC-MS/MS.Proteomics. 2009; 9: 1683-1695Crossref PubMed Scopus (392) Google Scholar). The importance of making data available through public data repositories for reprocessing and further analysis was highlighted. In addition, Eric Deutsch and Rob Kearney demonstrated how a centralized analysis of the data submitted from labs participating in the proposed quantitation exercise could be analyzed separately from that of each individual lab.Scott Gerber (Dartmouth University) discussed translational or clinical proteomics emphasizing the indexing of data points rather than biomarker discovery. In addition, Bruno Domon (Institute of Molecular Systems Biology) discussed quantitative proteomics and Selected Reaction Monitoring (SRM)-based assays emphasizing the need to minimize sample preparation and gathered a consensus along with Graham Scott (Agilent Technologies) for the relative abundance Test Samples.A highly successful initiative encompassing test sample design and application for biomarker discovery was presented by Hishashi Narimatsu (National Institute of Advanced Industrial Science and Technology, Japan). Here, the glycomics community has established standard operating protocols for the characterization of N-glycosyl modifications on N-linked glycoproteins. Sylvie LaBoissiere (Genome Quebec) presented insight into matching proteomics with genomics for the pathogen C. difficile. The Workshop also had presentations representing large-scale proteomics efforts with biological insight. Catherine Au's (McGill University) presentation was on “Organellar Proteomics to Create the Cell Map” and Michel Desjardins's (Université de Montréal) was on “Phosphoproteomics of the Macrophage Phagosome”. Here the applications of the technology to biological discovery were particularly evident.The proposed HUPO test sample exercise for quantitation is expected to build on other efforts proposed for the community (3Anderson N.L. Anderson N.G. Pearson T.W. Borchers C.H. Paulovich A.G. Patterson S.D. Gillette M. Aebersold R. Carr S.A. Human proteome detection and quantitation project.Mol. Cell. Proteomics. 2009; 5: 883-886Abstract Full Text Full Text PDF Scopus (173) Google Scholar) as well as the pioneering work of ABRF and Clinical Proteomic Technology Assessment for Cancer (CPTAC) in ensuring high sensitivity and reproducibility in quantitative proteomics by all labs. The HUPO Workshop on “Test Samples for Quantitative Proteomics” was held in Barbados January 9–16, 2009 at the McGill Bellairs Research Institute co-chaired by Drs. Jan van Oostrum (Chair, HUPO Industrial Advisory Board) and John Bergeron (McGill University). The Workshop was organized by Ms. Sylvie Ouellette (Administrative Coordinator, HUPO). The Workshop built on the foundation of a HUPO test samples effort in press in Nature Methods (Ref. 1Bell A.W. Deutsch E.W. Au C.E. Kearney R.E. Sechi S. Nilsson T. Bergeron J.J.M. HUPO Test Sample Working Group A HUPO test sample study reveals common problems in mass spectrometry-based proteomics.Nat. Methods. 2009; 10.1038/nmeth.1333Crossref Scopus (272) Google Scholar). This paper demonstrated how 27 international labs selected by a HUPO Test Samples Committee were able to ensure 100% success by all participating labs for the characterization of 20 proteins at a 5 picomole abundance for each protein. For 20 of the 27 labs iterative guidance was necessary for the 100% successful outcome. A further test for stochastic sampling of tryptic peptides involved the characterization of 22 tryptic peptides of mass 1250 ± 5 Da found among the 20 proteins (all of different amino acid sequence and at least one per protein). Here, the 27 labs were less successful with only 4 labs generating sufficient data for all 22 peptides and only one of these 4 labs able to report on the characterization of the 22 peptides. Several bottlenecks were uncovered in this study; effective education and training remains the most important bottleneck for a successful proteomics pipeline. In the January 2009 workshop, several presentations dealt with relative quantification efforts using two-dimensional gels and imaging as well as the advantages and disadvantages of commercially available computer software packages to standardize such data sets (Will Dracup, Nonlinear Dynamics; Tom Slyker, Bio-Rad; Francesc Canals, ProteoRed, and Ola Forsstrom-Olsson, Ludesi). As well, the standardization of the methodology and operating procedure processes involved were presented (Judit Nagy, Imperial College; Jun Wheeler, National Institute for Biological Standards and Control). Challenges of the HUPO equimolar test sample reported and discussed included sample distribution to individual analysis (Alex Bell, McGill University) and data reprocessing and a centralized analysis (Eric Deutsch, Institute for Systems Biology). As well, heat maps which visualized the frequency with which tryptic peptides were observed as deduced from the tandem mass spectra reported for each lab were found to be a striking visual method to represent the data (Rob Kearney, McGill University). The importance of test sample preparation (high protein purity) and centralized analysis of all data were emphasized for quality control and the discovery of bottlenecks related to high throughput proteomics. The importance of highly curated and well annotated data bases was highlighted by Amos Bairoch (Swiss Institute of Bioinformatics), who presented the most recent version of the human Swiss-Prot protein data base. Remarkably, nearly 8,000 of the predicted products of the 20,300 protein coding human genes have not yet been observed at the protein level. For tandem mass spectrometry, insightful experiences were shared by an industrial representative (Martha Stapels, Waters), academic investigators (Bruno Domon, Institute of Molecular Systems Biology and Gilles Lajoie, University of Western Ontario), and colleagues participating in organized efforts such as Clinical Proteomic Technology Assessment for Cancer (CPTAC) (David Tabb, Vanderbilt University) and The Association of Biomolecular Resource Facilities (ABRF) (Mike MacCoss, University of Washington). The pioneering and detailed studies from The Association of Biomolecular Resource Facilities (ABRF) illustrated the challenges of establishing a successful quantitation effort with test samples. The most recent study (ABRF-PRG07: Advanced Quantitative Proteomics Study) showed that of 36 participating labs, none was 100% successful. Henry Duewel (Sigma-Aldrich) recommended that the various or most commonly utilized proteomics platforms (e.g. protein identification, quantitation, and post-translational modification characterization) might each require a unique, specially tailored set of reference materials. Henry Duewel highlighted some of the insights and experiences that Sigma-Aldrich has gained in preparing and commercializing their Equimolar and Dynamic Range Protein Standards (Universal Proteomics Standards). Tommy Nilsson (Gothenburg University) summarized efforts of ABRF over the past 10 years and the HUPO equimolar test samples study to conclude that the proteomics community still has a lot to prove with respect to comprehensive and accurate reporting. There is a need for better databases that are compatible with mass spectrometry-based proteomics; preferably gene-centric in design such that there exists only one top entry for each of the 20,300 human protein products. Isoforms and splice variants should then be assigned to each of these 20,300 proteins. Such gene-centric databases are likely to increase the fidelity of mass spectrometry-based proteomics. To underscore the need, Nilsson highlighted the complete failure of the entire proteomics community to provide even a handful of Food and Drug Administration approved biomarkers for disease. “A lot of work is still needed”, he concluded. From the investigators present at the HUPO workshop, proposals for a quality assurance (self-assessment) experiment and for a relative abundance test sample were put forward. There was a consensus for the use of stable isotope labeled peptides and for recombinant proteins labeled with 13C and 15N isotopes spiked into complex mixtures. Along with light versions of recombinant proteins spiked into previously characterized complex mixtures, this was considered as a reasonable test sample to ensure that the community can achieve 100% success in quantitative proteomics. Whether the lower limit of 0.1 fmol or 1 fmol is achievable by all was debated. Discussion on the need for a standardized trypsinization methodology was led by Christoph Borchers (University of Victoria) who showed that different digestion procedures and conditions have diverse effects for individual proteins. Christoph Borchers also illustrated the efficacy of quantitation of multiple proteins in human plasma by Multi-Reaction Monitoring. Reproducibility was discussed by Will Dracup (NonLinear Dynamics) who defined the problems and proposed solutions for cross-lab reproducibility and quality control. The pitfalls of only focusing on reproducibility without having appropriate metrics for data correctness were presented by Ola Forsstrom-Olsson (Ludesi), who proposed alternative metrics for quality control. Jan van Oostrum (Zeptosens, co-chair) used examples taken from reverse phase protein microarray experiments to highlight that reproducibility criteria in an experimental science like proteomics should be a common standard. Jim Dasch (Protein Forest) presented technology to reduce peptide complexity in order to improve on the stochastic aspect of sampling at the mass spectrometer. The importance of bioinformatics in data analysis and reproducibility in quantitative proteomics was not ignored. Investigators presented who were responsible for data deposition and a centralized analysis of the data from the 27 labs for the HUPO equimolar test samples paper (1Bell A.W. Deutsch E.W. Au C.E. Kearney R.E. Sechi S. Nilsson T. Bergeron J.J.M. HUPO Test Sample Working Group A HUPO test sample study reveals common problems in mass spectrometry-based proteomics.Nat. Methods. 2009; 10.1038/nmeth.1333Crossref Scopus (272) Google Scholar). The importance of Tranche (Phil Andrews, University of Michigan) and PRIDE (Lennart Martens, EBI) as data repositories and the preferred formats for submission were indicated. For reprocessing, raw mass spectrometer output files in the native instrument vendor format were preferred with the exception of data acquired in MSe mode (2Geromanos S.J. Vissers J.P.C. Silva J.C. Dorschel C.A. Li G.-Z. Gorenstein M.V. Bateman R.H. Langridge J.I. The detection, correlation and comparison of peptide precursor and product ions from data independent LC-MS with data dependant LC-MS/MS.Proteomics. 2009; 9: 1683-1695Crossref PubMed Scopus (392) Google Scholar). The importance of making data available through public data repositories for reprocessing and further analysis was highlighted. In addition, Eric Deutsch and Rob Kearney demonstrated how a centralized analysis of the data submitted from labs participating in the proposed quantitation exercise could be analyzed separately from that of each individual lab. Scott Gerber (Dartmouth University) discussed translational or clinical proteomics emphasizing the indexing of data points rather than biomarker discovery. In addition, Bruno Domon (Institute of Molecular Systems Biology) discussed quantitative proteomics and Selected Reaction Monitoring (SRM)-based assays emphasizing the need to minimize sample preparation and gathered a consensus along with Graham Scott (Agilent Technologies) for the relative abundance Test Samples. A highly successful initiative encompassing test sample design and application for biomarker discovery was presented by Hishashi Narimatsu (National Institute of Advanced Industrial Science and Technology, Japan). Here, the glycomics community has established standard operating protocols for the characterization of N-glycosyl modifications on N-linked glycoproteins. Sylvie LaBoissiere (Genome Quebec) presented insight into matching proteomics with genomics for the pathogen C. difficile. The Workshop also had presentations representing large-scale proteomics efforts with biological insight. Catherine Au's (McGill University) presentation was on “Organellar Proteomics to Create the Cell Map” and Michel Desjardins's (Université de Montréal) was on “Phosphoproteomics of the Macrophage Phagosome”. Here the applications of the technology to biological discovery were particularly evident. The proposed HUPO test sample exercise for quantitation is expected to build on other efforts proposed for the community (3Anderson N.L. Anderson N.G. Pearson T.W. Borchers C.H. Paulovich A.G. Patterson S.D. Gillette M. Aebersold R. Carr S.A. Human proteome detection and quantitation project.Mol. Cell. Proteomics. 2009; 5: 883-886Abstract Full Text Full Text PDF Scopus (173) Google Scholar) as well as the pioneering work of ABRF and Clinical Proteomic Technology Assessment for Cancer (CPTAC) in ensuring high sensitivity and reproducibility in quantitative proteomics by all labs.