Precise benchmarking of single-cell RNA-seq methods including mcSCRB-seq

Abstract

Single-cell RNA-sequencing is useful tool to identify cell-type and cell-state in population cells. And, various single-cell RNA-seq methods have been already reported. Recently, mcSCRB-seq has been reported in Nature Communications 9:2937 (2018). The paper claims that mcSCRB-seq is one of most sensitive single-cell RNA-seq method on ground of their ERCC detection efficiency. We took an interest in quantitative performance of mcSCRB-seq. Worryingly, we found some of dubious points in paper. Usage of ERCC spike-in RNA is one of strategies to validate quantitative performance for single-cell RNA-seq methods. It is generally known that copy number of ERCC spike-in RNA at 50% detection capability (also called as ERCC detection efficiency) highly depend on initial fastq reads. The authors also showed and confirmed dependency in Supplementary Figure 13b. The authors compared ERCC detection efficiency among single-cell RNA-seq methods in Figure 2c. The authors mentioned that the average ERCC detection efficiency is most representative measure to compare sensitivities across many protocols. in main text. However, authors did not show information about sequence reads per sample in Figure 2c. Therefore, we investigated “sequence reads per sample” for some methods in Figure 2c. We noticed that authors compared copy number of respective methods, which used greatly differing initial data size (fastq reads)(Figure X1). For example, mcSCRB-seq uses 2 million reads per cell. Yanai's lab C1-CEL-seq2 used about 1.3 million reads per cell. Quartz-Seq2 used about 0.24 million reads per cell. The authors used more fastq reads for their developed method (mcSCRB-seq) than that for other methods. The authors loaded dice in favor of their developed method (mcSCRB-seq). Again, note that ERCC detection efficiency highly depend on initial fastq reads. To solve unfair comparison, we decided to re-analyze these methods in an accurate manner by ourselves. At first, we re-analyzed data sets, which used in Figure 2c of Nature Communications 9:2937 (2018). We observed considerable variation of ERCC detection efficiency among data set of mcSCRB-seq (Figure X2). Reanalyzed score of J1 mcSCRB-seq was almost consistent with score of Figure 2c in Nature Communications 9:2937 (2018). Moreover, average ERCC score of CEL-seq2 and Quartz-Seq2 were better than that of mcSCRB-seq (Figure X2). The result do not support superiority of mcSCRB-seq for ERCC detection efficiency. Nevertheless, it is also known that ERCC detection efficiency is affected by experimental condition, including type of cell line. In Figure 2c of Nature Communications 9:2937 (2018), authors compared data set from different cell lines. To validate quantitative performance in a precise manner, we should analyze data set, which used same cell line (2i-treated J1 ES cell)(Figure X3). We analyzed data according to previous study (Molecular Cells 2017 65(4) P631-643). We found that mcSCRB-seq and Quartz-Seq2 has a superior ability to detect molecule of ERCC spike-in RNA (Figure X3). Moreover, Quartz-Seq2 and CEL-seq2 has a superior ability to detect UMI and gene count (Figure X3). However, mcSCRB-seq showed lower ability to detect UMI and gene count. We do not know why mcSCRB-seq preferred ERCC spike-in RNA rather than internal RNA. It seems that mcSCRB-seq is not skillful at detection of internal gene expression. These results indicate that ERCC detection efficiency poorly correlated with detection efficiency of UMI and gene count. We feel researchers should attach weigth to assessment with internal gene expression, because assessment with ERCC spike-in RNA is vulnerable to technical noise and experimental conditions. Our precise validation unveiled actual quantitative performance for respective single-cell RNA-seq methods. We are happy if results help members of single-cell community to precisely understand respective methods.