On the proper use of statistical design of experiments

Abstract

In a recent article published by Yao et al. [1Yao C.L. Chu I.M. Hsieh T.B. Hwang S.M. A systematic strategy to optimize ex vivo expansion medium for human hematopoietic stem cells derived from umbilical cord blood mononuclear cells.Exp Hematol. 2004; 32: 720-727Abstract Full Text Full Text PDF PubMed Scopus (76) Google Scholar], the authors have presented a very interesting application of the design of experiments (DOE) approach to the optimization of a serum-free medium for the expansion of cord blood (CB) hematopoietic stem cells (HSC). In the first part of their work, the authors have performed screening as well as optimization experimental designs on four serum replacement additives: bovine serum albumin (BSA), insulin (I), transferin (TF), and 2-mercaptoethanol (2-ME). In the second part, they screened and optimized a cytokine cocktail starting from 10 factors: thrombopoietin (TPO), interleukin (IL)-3, stem cell factor (SCF), Flt-3 ligand (FL), IL-6, granulocyte-macrophage colony-stimulating factor (GM-CSF), granulocyte colony-stimulating factor (G-CSF), stem cell growth factor α (SCGF), IL-11, and hepatocyte growth factor (HGF). In both parts, the optimization objective was to maximize the expansion of both white blood cells (WBC) and CD34+ cells. Our main concerns in regard to this work are as follows: 1) the a priori assumption that second- and higher-order interactions between factors were not significant, without providing any statistical evidence to this effect, 2) a general lack of statistical significance testing, and 3) the inefficient use of some empirical optimization methods. In the first part, a 24 full factorial design (16 experiments) was performed, followed by least-squares multilinear regression, to estimate the coefficients of a model based solely on the main effects (i.e., individual effects) of the four serum replacement components. Based on the data provided in the paper, we could reproduce the regression coefficients for Equation 2 (effect on WBC expansion) but the coefficients we obtained for Equation 3 (effect on CD34+ cells) differ significantly from those published in the article (constant: 6.28 instead of 12.39, BSA coefficient: 5.02 instead of 9.33, I coefficient: 0.76 instead of 1.67, TF coefficient: 0.65 instead of 1.62, and 2-ME coefficient: −0.23 instead of −0.11). For both models, the authors considered all positive coefficients to be significant. However, using standard 95% confidence intervals, we found for the model described by Equation 2 that only 3 out of the 5 coefficients were indeed significant (the constant term and the coefficients for BSA and I), whereas 4 coefficients out of 5 were significant for Equation 3 (2-ME coefficient was nonsignificant). It was possible to evaluate confidence intervals in this case since one can estimate as many coefficients as the number of experiments. Thus, 11 degrees of freedom are available here. A very interesting feature of full factorial designs, such as this 24, is that they allow the experimenter to simultaneously and independently estimate all main (individual) effects as well as all factor interactions. The authors did not make use of this powerful capability of the technique and rather seemed to neglect all potential interactions between factors. After reanalyzing the data and estimating interactions, we found that the BSA/TF two-factor interaction as well as the BSA/I/TF three-factor interaction had a significant negative effect on WBC expansion, whereas BSA/I interaction had significant positive effect on this response. In addition, the following interactions had significant positive effects on CD34+ cell expansion: TF/2-ME, BSA/I, and BSA/TF/2-ME. No negative interaction was found for CD34+ cell expansion. Omitting these interactions can be very misleading and could lead to a suboptimal component mixture (see chapter 15 of Box et al. [2Box G.E.P. Hunter W.G. Hunter J.S. Statistics for experimenters: An introduction to design, data analysis, and model building. John Wiley & Sons Inc., New York1978Google Scholar] for an example). In this new analysis, significance testing was performed using normal probability plots of the estimated effects instead of using confidence intervals since no degrees of freedom are available (16 coefficients are estimated using 16 experiments). In the second part, the authors performed a 210-6 two-level fractional factorial design to screen the effects of 10 cytokines with only 16 experiments. In this design, each main effect of individual cytokines is confounded with at least 2 two-factor interactions and numerous higher-order interactions [2Box G.E.P. Hunter W.G. Hunter J.S. Statistics for experimenters: An introduction to design, data analysis, and model building. John Wiley & Sons Inc., New York1978Google Scholar]. The use of such a highly fractionated experimental design must therefore always be followed by further experiments in order to break the confounding pattern and to guarantee the identification of causal relationships between factors and responses. Instead, Yao et al. [1Yao C.L. Chu I.M. Hsieh T.B. Hwang S.M. A systematic strategy to optimize ex vivo expansion medium for human hematopoietic stem cells derived from umbilical cord blood mononuclear cells.Exp Hematol. 2004; 32: 720-727Abstract Full Text Full Text PDF PubMed Scopus (76) Google Scholar] concluded without any significance testing that 9 of the 10 factors were significant. This conclusion cannot be supported by only 16 experiments unless all interactions are known a priori to be inexistent, which is not the case here as we have previously shown. The authors also used a steepest ascent method (SA) to optimize the concentration of serum substitutes and cytokines. The SA is a powerful gradient-based line search method designed to experimentally seek a global optimum by successively performing DOE to evaluate the local gradient and then moving along that direction until no further improvements are found. This procedure (DOE followed by line search) is repeated until no improvement directions can be found [2Box G.E.P. Hunter W.G. Hunter J.S. Statistics for experimenters: An introduction to design, data analysis, and model building. John Wiley & Sons Inc., New York1978Google Scholar]. The use of this method by the authors is very inefficient since they applied the line search within the initial experimental domain when it could have been found using surface response methods with no or very few additional experiments. In addition, if their linear model is adequate, but has never been tested, then the optimum necessarily lies on one corner of the design. The authors should have saved these efforts and associated costs to explore outside the initial domain to seek for a global optimum. In conclusion, we firmly believe that the use of DOE can be very useful in the field of cell physiology and the development of culture media. In that respect, Yao et al. [1Yao C.L. Chu I.M. Hsieh T.B. Hwang S.M. A systematic strategy to optimize ex vivo expansion medium for human hematopoietic stem cells derived from umbilical cord blood mononuclear cells.Exp Hematol. 2004; 32: 720-727Abstract Full Text Full Text PDF PubMed Scopus (76) Google Scholar] have great merit in applying these contemporary tools to the very important problems of ex vivo stem cell culture. However, we have shown that their work suffered from serious methodological flaws, which questions the validity of their conclusions. In fact, DOE methods are often simplistically presented in commercial statistical package, and this is often the cause of data misinterpretation. A better integration of these concepts in the science and engineering curricula could possibly contribute to improve this situation, which is often encountered in the literature. A systematic strategy to optimize ex vivo expansion medium for human hematopoietic stem cells derived from umbilical cord blood mononuclear cellsExperimental HematologyVol. 32Issue 8PreviewIn this study, a serum-free, stroma-free, and chemically defined medium for hematopoietic stem cell (HSC) expansion was systematically developed and optimized using factorial design and the steepest ascent method. Full-Text PDF Open ArchiveIn reply to Michaud et al.: Systematic strategy approach in medium designExperimental HematologyVol. 33Issue 11PreviewIn response to the letter by Francois-Thomas Michaud, Victor Parent, Alain Garnier, and Carl Duchesne [1], we, as authors of “A Systematic Strategy to Optimize Ex Vivo Expansion Medium for Human Hematopoietic Stem Cells Derived from Umbilical Cord Blood Mononuclear Cells” in Experimental Hematology 2004;32:720–727 [2], gratefully acknowledge their correction of the error in Equation 3 for serum substitute screening. The coefficients of all parameters of Equation 3 should be divided by a factor of 2. Full-Text PDF Open Access