Machine learning with knowledge constraints for process optimization of open-air perovskite solar cell manufacturing

Abstract

•ML accelerates open-air process optimization for perovskite solar cells•With a budget of 100 process conditions, 18.5% device efficiency is achieved•Researchers’ domain knowledge can be incorporated into ML process optimization•Benchmarking results show an advantage of ML over traditional methods Perovskite photovoltaics (PVs) have achieved rapid improvement in the past decade for the power conversion efficiency of small-area lab-scale devices. However, successful commercialization still requires the development of low-cost, scalable, and high-throughput manufacturing techniques. Machine learning (ML) for materials science and engineering has been developed in recent years, and it can readily be used to accelerate perovskite manufacturing scale-up. We demonstrate a Bayesian optimization framework that allows the incorporation of researchers’ domain knowledge into the ML-guided loop. In the case of optimizing perovskite solar cells by the open-air rapid spray plasma processing (RSPP) technique, the proposed framework enables a faster optimization in comparison with other conventional researcher-driven design-of-experiment methods. Although it has been shown for RSPP, the ML framework can be broadly used for accelerated development of manufacturing technologies for perovskite PVs. Developing a scalable manufacturing technique for perovskite solar cells requires process optimization in high-dimensional parameter space. Herein, we present a machine learning (ML)-guided framework of sequential learning for manufacturing the process optimization of perovskite solar cells. We apply our methodology to the rapid spray plasma processing (RSPP) technique for open-air perovskite device fabrication. With a limited experimental budget of screening 100 process conditions, we demonstrated an efficiency improvement to 18.5% as the best result from a device fabricated by RSPP. Our model is enabled by three innovations: flexible knowledge transfer between experimental processes by incorporating data from prior experimental data as a probabilistic constraint, incorporation of both subjective human observations and ML insights when selecting next experiments, and adaptive strategy of locating the region of interest using Bayesian optimization before conducting local exploration for high-efficiency devices. Furthermore, in virtual benchmarking, our framework achieves faster improvements with limited experimental budgets than traditional design-of-experiments methods. Developing a scalable manufacturing technique for perovskite solar cells requires process optimization in high-dimensional parameter space. Herein, we present a machine learning (ML)-guided framework of sequential learning for manufacturing the process optimization of perovskite solar cells. We apply our methodology to the rapid spray plasma processing (RSPP) technique for open-air perovskite device fabrication. With a limited experimental budget of screening 100 process conditions, we demonstrated an efficiency improvement to 18.5% as the best result from a device fabricated by RSPP. Our model is enabled by three innovations: flexible knowledge transfer between experimental processes by incorporating data from prior experimental data as a probabilistic constraint, incorporation of both subjective human observations and ML insights when selecting next experiments, and adaptive strategy of locating the region of interest using Bayesian optimization before conducting local exploration for high-efficiency devices. Furthermore, in virtual benchmarking, our framework achieves faster improvements with limited experimental budgets than traditional design-of-experiments methods. Metal halide perovskites are efficient solar absorbers compatible with low-cost solution processing methods that show promise as an emerging thin-film photovoltaic (PV) technology. Scaling up the fabrication process is currently one of the critical research areas of perovskite technology on the path to commercialization.1Li Z.C. Klein T.R. Kim D.H. Yang M. Berry J.J. van Hest M.F.A.M. Zhu K. Scalable fabrication of perovskite solar cells.Nat. Rev. Mater. 2018; 3: 1-20Google Scholar, 2Perini C.A.R. Doherty T.A.S. Stranks S.D. Correa-Baena J.-P. Hoye R.L.Z. Pressing challenges in halide perovskite photovoltaics—from the atomic to module level.Joule. 2021; 5: 1024-1030Google Scholar, 3Li D. Zhang D. Lim K.-S. Hu Y. Rong Y. Mei A. Park N.-G. Han H. A review on scaling up perovskite solar cells.Adv. Funct. Mater. 2021; 31: 2008621Google Scholar Despite the success of >25% efficient perovskite solar cells in academic labs using spin coating,4Almora O. Baran D. Bazan G.C. Berger C. Cabrera C.I. Catchpole K.R. Erten-Ela S. Guo F. Hauch J. Ho-Baillie A.W.Y. et al.Device performance of emerging photovoltaic materials.Adv. Energy Mater. 2021; 11: 2002774Google Scholar,5Yoo J.J. Seo G. Chua M.R. Park T.G. Lu Y. Rotermund F. Kim Y.-K. Moon C.S. Jeon N.J. Correa-Baena J.-P. et al.Efficient perovskite solar cells via improved carrier management.Nature. 2021; 590: 587-593Google Scholar this processing method is not scalable to a manufacturing line. Recently, Rolston et al.6Rolston N. Scheideler W.J. Flick A.C. Chen J.P. Elmaraghi H. Sleugh A. Zhao O. Woodhouse M. Dauskardt R.H. Rapid open-air fabrication of perovskite solar modules.Joule. 2020; 4: 2675-2692Google Scholar demonstrated a high-throughput rapid spray plasma processing (RSPP) method as a scalable open-air fabrication process because it has the potential of achieving low-cost perovskite PV modules at a manufacturing cost of ∼$0.2 /W. In addition to cost, the main advantage of this spray-deposition-based technique is its ultrahigh throughput and improved mechanical properties of the thin film in comparison with other scalable processing methods, such as blade coating, slot-die coating, and roll-to-roll printing. For new scalable perovskite manufacturing processes (including RSPP), it typically takes months to years to achieve process control and reproducibility on the module scale, and several years to estimate the upper potential of the technology. One of the key challenges is that there are many processing parameters to co-optimize1Li Z.C. Klein T.R. Kim D.H. Yang M. Berry J.J. van Hest M.F.A.M. Zhu K. Scalable fabrication of perovskite solar cells.Nat. Rev. Mater. 2018; 3: 1-20Google Scholar,6Rolston N. Scheideler W.J. Flick A.C. Chen J.P. Elmaraghi H. Sleugh A. Zhao O. Woodhouse M. Dauskardt R.H. Rapid open-air fabrication of perovskite solar modules.Joule. 2020; 4: 2675-2692Google Scholar—e.g., precursor composition, speed, temperature, head/nozzle height, and curing methods. High-throughput experimentation has been introduced to explore the regions in the high-dimensional parameter space.7Ahmadi M. Ziatdinov M. Zhou Y. Lass E.A. Kalinin S.V. Machine learning for high-throughput experimental exploration of metal halide perovskites.Joule. 2021; 5: 2797-2822Google Scholar, 8Zhao Y. Zhang J. Xu Z. Sun S. Langner S. Hartono N.T.P. Heumueller T. Hou Y. Elia J. Li N. et al.Discovery of temperature-induced stability reversal in perovskites using high-throughput robotic learning.Nat. Commun. 2021; 12: 2191Google Scholar, 9Du X. Lüer L. Heumueller T. Wagner J. Berger C. Osterrieder T. Wortmann J. Langner S. Vongsaysy U. Bertrand M. et al.Elucidating the full potential of OPV materials utilizing a high-throughput robot-based platform and machine learning.Joule. 2021; 5: 495-506Google Scholar, 10Saliba M. Polyelemental, multicomponent perovskite semiconductor libraries through combinatorial screening.Adv. Energy Mater. 2019; 9: 1803754Google Scholar, 11Zhao Y. Heumueller T. Zhang J. Luo J. Kasian O. Langner S. Kupfer C. Liu B. Zhong Y. Elia J. et al.A bilayer conducting polymer structure for planar perovskite solar cells with over 1,400 hours operational stability at elevated temperatures.Nat. Energy. 2022; 7: 144-152Google Scholar However, in some cases, this high-dimensional optimization problem can be nearly impossible to solve by brute force or even by the sophisticated design of experiments (DoEs). Sequential machine learning (ML) methods, e.g., Bayesian optimization (BO), have emerged as effective optimization strategies to explore a wide range of chemical reaction synthesis12Burger B. Maffettone P.M. Gusev V.V. Aitchison C.M. Bai Y. Wang X. Li X. Alston B.M. Li B. Clowes R. et al.A mobile robotic chemist.Nature. 2020; 583: 237-241Google Scholar, 13Granda J.M. Donina L. Dragone V. Long D.L. Cronin L. Controlling an organic synthesis robot with machine learning to search for new reactivity.Nature. 2018; 559: 377-381Google Scholar, 14Shields B.J. Stevens J. Li J. Parasram M. Damani F. Alvarado J.I.M. Janey J.M. Adams R.P. Doyle A.G. Bayesian reaction optimization as a tool for chemical synthesis.Nature. 2021; 590: 89-96Google Scholar and material optimization.15Gongora A.E. Xu B. Perry W. Okoye C. Riley P. Reyes K.G. Morgan E.F. Brown K.A. A Bayesian experimental autonomous researcher for mechanical design.Sci. Adv. 2020; 6eaaz1708Google Scholar, 16MacLeod B.P. Parlane F.G.L. Morrissey T.D. Häse F. Roch L.M. Dettelbach K.E. Moreira R. Yunker L.P.E. Rooney M.B. Deeth J.R. et al.Self-driving laboratory for accelerated discovery of thin-film materials.Sci. Adv. 2020; 6eaaz8867Google Scholar, 17Mekki-Berrada F. Ren Z. Huang T. Wong W.K. Zheng F. Xie J. Tian I.P.S. Jayavelu S. Mahfoud Z. Bash D. et al.Two-step machine learning enables optimized nanoparticle synthesis.npj Comput. Mater. 2021; 7: 55Google Scholar, 18Attia P.M. Grover A. Jin N. Severson K.A. Markov T.M. Liao Y.H. Chen M.H. Cheong B. Perkins N. Yang Z. et al.Closed-loop optimization of fast-charging protocols for batteries with machine learning.Nature. 2020; 578: 397-402Google Scholar, 19Lookman T. Balachandran P.V. Xue D. Yuan R. Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design.npj Comput. Mater. 2019; 5: 21Google Scholar, 20Nikolaev P. Hooper D. Webber F. Rao R. Decker K. Krein M. Poleski J. Barto R. Maruyama B. Autonomy in materials research: a case study in carbon nanotube growth.npj Comput. Mater. 2016; 2: 1-6Google Scholar, 21Balachandran P.V. Kowalski B. Sehirlioglu A. Lookman T. Experimental search for high-temperature ferroelectric perovskites guided by two-step machine learning.Nat. Commun. 2018; 9: 1668Google Scholar, 22Ling J. Hutchinson M. Antono E. Paradiso S. Meredig B. High-dimensional materials and process optimization using data-driven experimental design with well-calibrated uncertainty estimates.Integr. Mater. Manuf. Innov. 2017; 6: 207-217Google Scholar, 23Rohr B. Stein H.S. Guevarra D. Wang Y. Haber J.A. Aykol M. Suram S.K. Gregoire J.M. Benchmarking the acceleration of materials discovery by sequential learning.Chem. Sci. 2020; 11: 2696-2706Google Scholar, 24Erps T. Foshey M. Luković M.K. Shou W. Goetzke H.H. Dietsch H. Stoll K. von Vacano B. Matusik W. Accelerated discovery of 3D printing materials using data-driven multiobjective optimization.Sci. Adv. 2021; 7eabf7435Google Scholar, 25Tran A. Tranchida J. Wildey T. Thompson A.P. Multi-fidelity machine-learning with uncertainty quantification and Bayesian optimization for materials design: application to ternary random alloys.J. Chem. Phys. 2020; 153074705Google Scholar BO has been shown to work well for optimization problems with under 20 variables26James V. Miranda L. PySwarms: a research toolkit for particle swarm optimization in Python.J. Open Source Softw. 2018; 3: 433Google Scholar or up to 30 variables with some algorithm modifications.27Wang Z. Gehring C. Kohli P. Jegelka S. Batched large-scale Bayesian optimization in high-dimensional spaces.in: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics. AISTATS, 2018: 745-754Google Scholar,28Harris S.J. Harris D.J. Li C. Failure statistics for commercial lithium ion batteries: a study of 24 pouch cells.J. Power Sources. 2017; 342: 589-597Google Scholar Therefore, we have chosen to study sequential-learning-based optimization strategies in our current study of RSPP perovskite PV devices. Current reports of sequential learning studies with classical BO have two common drawbacks: (1) no direct channel to incorporate information from previous relevant studies and (2) inflexibility to adapt researchers’ qualitative feedbacks into the iterative loop. On one hand, the ML model sometimes requires a significant amount of data to learn what has already become apparent to the researchers.29Ziatdinov M.A. Ghosh A. Kalinin S.V. Physics makes the difference: Bayesian optimization and active learning via augmented Gaussian process.Mach. Learn. Sci. Technol. 2022; 3015022Google Scholar On the other hand, these drawbacks could discourage material science researchers to adopt ML tools because their domain knowledge could not be utilized in the classical BO iterations. Previous data and researchers’ knowledge are useful information sources for experimental planning. An acquisition function in a BO framework is the “decision-maker” to produce an experimental plan in the next round. Therefore, we can incorporate domain knowledge as probabilistic constraints for the acquisition function (introduced by Gelbart et al.30Gelbart M.A. Snoek J. Adams R.P. Bayesian optimization with unknown constraints.Preprint at arXiv. 2014; (1403.5607)Google Scholar). With the aim of rapid optimization, Sun et al. incorporated density functional theory (DFT) calculations of phase stability as a probabilistic constraint for the experimental acceleration of composition optimization to improve perovskite stability, avoiding compositions susceptible to phase segregation.31Sun S. Tiihonen A. Oviedo F. Liu Z. Thapa J. Zhao Y. Hartono N.T.P. Goyal A. Heumueller T. Batali C. et al.A data fusion approach to optimize compositional stability of halide perovskites.Matter. 2021; 4: 1305-1322Google Scholar Simple information such as a visual assessment of film thickness, color, and structural defects can be a powerful tool that provides additional guidance for intelligent and efficient optimization. In other words, if an experienced researcher identifies a low-quality perovskite film, the subsequent device fabrication is no longer necessary. Defining a probabilistic constraint is an effective and flexible way to incorporate this information into a sequential learning framework. In this work, we develop a sequential learning framework with probabilistic constraints for rapid process optimization with power conversion efficiency (PCE) as the target variable. As illustrated in Figure 1, the sequential learning framework iteratively learns the process-efficiency relation and suggests new experiments to achieve the optimal target. For a general framework, we start with experimental planning of process conditions with a model-free sampling method for the initial round. Then, perovskite solar cells are fabricated by the RSPP method, and PCE is measured with a solar simulator under standard testing conditions (STCs). With the experimental data of the process parameters and the device PCEs, we train the regression model to learn the process-efficiency relation, and the regression model is subsequently used to predict the PCE (and its prediction uncertainty) for the unsampled regions. Finally, the prediction results are evaluated by an acquisition function together with the constraint information, and therefore, a new round of experiments is planned. By including two additional knowledge constraints (i.e., visual assessment of film quality and previous experimental data from a related study), the optimization framework maps out the parameter space and tends to avoid the less promising regions based on observational film-quality information and prior device data. Hence, sampling suggestions focus on the most promising regions within the parameter space. To demonstrate the capability of our optimization approach, we consider six key RSPP input variables for the perovskite absorber layer in a device fabrication process. We aimed to exceed the best PCE of solar cells produced with our RSPP. First, we show that efficiency improvements can be obtained with our sequential learning framework, reaching 18.5% efficiency in five experimental iterations. Second, we describe how multiple sources of information were fused into our sequential learning framework as a probabilistic constraint. Third, we analyze the learned relationships between input variables and efficiency, extracting some generalizable insights. Fourth, we benchmark the acceleration factor and enhancement factor of our optimization process against conventional model-free sampling methods for the DoEs sampling methods with optimization simulations, demonstrating an excellent acceleration within the limited experimental budget of fewer than 100 conditions. Figure 2A plots the process optimization guided by the sequential learning framework and probabilistic constraints of visual inspection and previous experimental data. PCEs were measured in batches consisting of 20 conditions at a time to enable iteration and model feedback to suggest the subsequent round of process parameters. The highest PCE device of each process condition (the dark-green dots in Figure 2A) was used in the optimization algorithm. In addition, based on the previous dataset of 45 process conditions (Figure 2C), we defined a top performer with PCE to be a device with PCE above 17% and a good performer to be a device with PCE exceeding 15%. Thus, we had 1 top performer and 6 good performers. Among the 85 of 100 conditions in BO-guided experiments in this work (excluding the low-quality films that did not pass the visual inspection), 45 process conditions achieved >15% PCE (good performers), and 10 process conditions achieved >17% PCE (top performers). The success rate was therefore 47% for the good performers and 12% for the top performers. In contrast, among the Latin hypercube sampling (LHS)-guided experiments of 50 process conditions (Figure 2B), we found only 6 good performers (12% success rate) and 1 top performer (2% success rate). Furthermore, the champion process condition produced a best-in-our-lab RSPP device efficiency of 18.5% in fewer than 100 conditions. The champion device in both LHS and previous experiments had never reached 18% PCE. According to a recent review paper on spray-deposited perovskite solar cells,32Bishop J.E. Smith J.A. Lidzey D.G. Development of spray-coated perovskite solar cells.ACS Appl. Mater. Interfaces. 2020; 12: 48237-48245Google Scholar this PCE is comparable with the highest-efficiency devices fabricated by spray deposition in the open air (18.5%)33Su J. Cai H. Yang J. Ye X. Han R. Ni J. Li J. Zhang J. Perovskite ink with an ultrawide processing window for efficient and scalable perovskite solar cells in ambient air.ACS Appl. Mater. Interfaces. 2020; 12: 3531-3538Google Scholar and in the N2 glove box (19%).34Ding J. Han Q. Ge Q.Q. Xue D.J. Ma J.Y. Zhao B.Y. Chen Y.X. Liu J. Mitzi D.B. Hu J.S. Fully air-bladed high-efficiency perovskite photovoltaics.Joule. 2019; 3: 402-416Google Scholar Visual inspection of the perovskite films was done after depositing the perovskite layer with RSPP. The film quality was rated by evaluating color, uniformity, and pinholes. The ratings were intended to be very conservative so that only the lowest-quality films were “tossed.” This evaluation step is typical when optimizing with conventional researcher-driven device processing. Example low-quality and high-quality films are shown in Figure S1. To confirm the validity of our visual criteria, we fabricated additional devices from those low-quality films. We confirmed low PCE values for these conditions (i.e., all below 13.5% with an average PCE of 7.8%), and the corresponding PCE distribution is shown in Figure S2. This validation step confirms that the device fabrication for low-quality films can be skipped. When training our regression model with sequential learning, the film quality ratings were incorporated as a constraint function. Note that only 6 conditions after the first batch produced low-quality films, which further demonstrates that the model sampled the parameter space more effectively over time. Figure 3 shows the ML method used in this study. Because a six-dimensional parameter space is difficult to visualize, two parameters of processing speed and substrate temperature were chosen as illustrations. A similar plot was obtained for each pair of parameters used in the regression analysis. A regression model was generated for the objective function and two constraint functions. The objective function contains the primary information measured from the experiments by plotting the process-efficiency relationship (Figure 3A). The combination of probabilistic constraints are additional layers included based on visual inspection of film quality (constraint function 1; Figure 3B) and previous experimental results (constraint function 2; Figure 3C). The resulting acquisition value was determined based on the selection criteria used (i.e., PCE or film quality). These are plotted in Figures 3D–3F and provide the framework for the sequential learning approach, namely how the model learned from each batch to suggest new process conditions with the goal of higher performance. The raw acquisition function in Figure 3D was the upper confidence bound (UCB) of the objective function. Probabilistic constraint 1 is calculated based on the probability of a good-quality film, and probabilistic constraint 2 is calculated based on the probability of a device achieving the above-average PCE in a previous experimental dataset. Both probabilistic constraint functions were scaled to reduce the impact of these constraints and prevent the modification of the acquisition function from being too harsh. More specifically, for constraint function 1, the film quality was rated in a binary outcome of either 0 (fail) or 1 (pass). The regression model of film quality was built to interpolate the unsampled regions (see Figure 3E) and sequentially converted to the probability function of passing the film quality assessment. The probabilistic constraint function (i.e., constraint function 1) was then softened to a range of 0.5–1 and multiplied by the raw acquisition function (Figure 3B). See supplemental information section 1.2 for the mathematical definition of probabilistic constraint function. The scaling means that we weigh the device data 2 times more important than the film quality data to avoid any potential bias from our qualitative visual inspection of film quality. The previous data (that can be found in Figure 1B) for constraint function 2 were acquired during a preliminary optimization of the RSPP setup. We also use these data as a probabilistic constraint instead of a model prior because other variables beyond the six considered herein were slightly modified. For example, the spray and plasma nozzles used in previous experiments were different than those used in this work. Although we believe there is essential knowledge to be transferred (that is not affected too strongly by these modifications), the previous data are not equivalent to the current dataset. This required a more subtle way of representing the information as a probabilistic constraint to the BO framework, instead of directly adding the previous data in the model training. Figure 4 visualizes the model learning through iteration as the acquisition function evolved and the process conditions began to converge round by round. Including the initial sampling by LHS, five experimental rounds were conducted for the device optimization. The experiments in rounds 1–3 followed the suggestions from the BO acquisition function. Figure 4 also shows that the probabilistic constraint began to affect the acquisition from round 1. For example, the higher-temperature and higher-speed regions were sampled less after the initial sampling round because of the probabilistic constraint. This observation is consistent with the probabilistic constraint and acquisition function shown in Figures 3E and 3F. Due to the limited budget of 100 experimental conditions (or a total of 5 experimental rounds), we opted for a different acquisition method in the final round. Aiming at a further improvement of PCE, we conducted a local optimization in a smaller window of process conditions around the best condition predicted by the regression model. The best condition was found by the particle swarm optimization method,26James V. Miranda L. PySwarms: a research toolkit for particle swarm optimization in Python.J. Open Source Softw. 2018; 3: 433Google Scholar,35Kennedy J. Eberhart R. Particle swarm optimization.Proceedings of ICNN'95—International Conference on Neural Networks. 4. 1995: 1942-1948Google Scholar which is a common method for the global optimum for any given regression model. The parameter space of process conditions for the final round is shown in Table S2. The reason for this change is that our Gaussian process (GP) model has the tendency to “smooth out” features in the response surface as shown in Figure 5. A similar finding was found in a previous study in materials science,17Mekki-Berrada F. Ren Z. Huang T. Wong W.K. Zheng F. Xie J. Tian I.P.S. Jayavelu S. Mahfoud Z. Bash D. et al.Two-step machine learning enables optimized nanoparticle synthesis.npj Comput. Mater. 2021; 7: 55Google Scholar where the GP fit was also very smooth to avoid potential overfitting to a small number of data points. To increase the probability of finding the optimum value, we use GP as a ringfencing technique to identify a region of the highest probability (i.e., a window of process conditions), rather than a specific point or condition of highest probability. Within this region of highest probability, we combined a few techniques to achieve a balance of exploitation and exploration. Therefore, the final 20 conditions consisted of a best model-predicted condition, nearest neighbors of the best condition for exploitation, and LHS conditions in this region for more exploration. We visualized the trained regression model of process-efficiency relations by projecting the six-dimensional parameter space into 2D pair-wise contour plots. For each contour plot, we sampled two process variables of interest in a 20 × 20 grid. For every point in the contour plot (i.e., a combination of the two variables), we sampled the remaining four variables 200 times, and we predicted the PCEs using the regression model for the 200 process conditions. Because we were interested in maximizing efficiency, we took the maximum PCE of the 200 process conditions. Therefore, these contour plots are the manifolds of the maximum PCEs in a 2D reduced space. For six process variables, there are a total number of 15 possible contour plots. Figure 5 shows some examples of the dimension-reduced manifolds. For example, a data point in the contour plot indicates the combination of temperature = 140°C and plasma duty cycle = 20% can achieve a PCE of >16.5% when the other four process variables are fully optimized. Note that the predicted best efficiency by the trained regressor shown in the contour plots is lower than the experimentally measured PCE (i.e., the ground truth) due to the training error of the regression model. This difference is common in BO, but it is still effective because an acquisition function balancing exploitation and exploration compensates for the model error. The training errors after each experimental iteration are found in Figure S4. The manifold maps also inform correlations between different process variables and their impact on device efficiency. We also projected the suggested experimental conditions onto the contour plots, which helps interpret the decision-making of the algorithm and prevent mistakes at every round. Some correlations in Figure 5 led to new insights of the RSPP method, including a negative correlation between both plasma gas flow/duty cycle and temperature (Figures 5A and 5B). Since a certain curing dose is required to convert the precursor solution to a crystalline perovskite, this result indicated that the higher curing properties of increased plasma gas and duty cycle offset the lower temperature to balance the energy delivered to the perovskite during processing. Additionally, some correlative trends (consistent with previous experimentation) were also observed. For example, higher spray flow correlates with faster speed (see Figure 5C), since a constant precursor dose should be delivered to the substrate to form the desired film thickness. We conducted a simulated “virtual” optimization with the fully trained “teacher” regression model for a benchmarking comparison between the sequential learning and conventional DoE methods. The “teacher” model is a gradient boosting regression model with decision trees, and it was trained with all the experimental data acquired for this work (i.e., the data from the BO framework and the LHS shown in Figure 2B). The training outcome is shown in Figure 6A. For the “virtual” optimization, a good “teacher” model only needs to ensure the predictions follow a monotonic correlation with the ground truth (i.e., the actual efficiency measured from experimentation). Therefore, following the study by Häse et al.,36Häse F. Aldeghi M. Hickman R.J. Roch L.M. Christensen M. Liles E. Hein J.E. Aspuru-Guzik A. Olympus: a benchmarking framework for noisy optimization and experiment planning.Mach. Learn. Sci. Technol. 2021; 2035021Google Scholar the spearman coefficient was used as an evaluation metric of the model training to assess the strength of the monotonic correlation between predicted and ground-truth efficiencies. This “teacher” regression model approximates the ground-truth manifold with a spearman coefficient of 0.93, and this mod