Benchmarking automated flow cytometry data analysis software using synthetic datasets

Abstract

Background & Aim Recent years have seen the development of many computational tools for automated Flow Cytometry (FC) data analysis. These software tools enable automatic cell population identification, classification and novel data visualisations. Because different software will use different mathematical approaches to identify and quantify cells, outputs can potentially vary when analysing the same dataset. Often, researchers simply accept the results without understanding the underlying black box algorithms and the potential for variation of output. Comparability studies between automated FC software requires high-quality benchmarking datasets that assess the performance and robustness of the algorithms. However, currently available datasets from FC experiments lack “gold standards” due to the nature of live cell biological materials. Furthermore, the datasets that meet data complexity requirements are costly to produce and only come from a narrow range of cell or disease models. Computer simulated FC datasets offer many advantages over their real-world counterparts. Synthetic standards can potentially be created, with data properties being highly controlled to objectively and systematically compare between software. Methods, Results & Conclusion In this research a range of synthetic FC cluster data have been systematically designed and generated containing controlled separation between clusters, rare cell populations, normal and non-normal probability distributions. These reference datasets have been used to evaluate the initial performance characteristics of a number of popular flow cytometry data analysis software, with the accuracy and repeatability of software being measured as the difference between the output and a reference absolute cell number. The initial results reveal similar trends in performance between software when analysing clusters with specific degrees of separation. Certain software platforms were more suited to analysis of rare cell populations. When using non-normally distributed clusters, the presence of skewed cell populations did not affect clustering performance, but the orientation of cluster skewness did. This demonstrates that benchmarking of FC automated software platforms will be possible with a high level of testing integrity using synthetic cluster dataset design leading to enhanced confidence in the data quality of cell characterisations.