Evaluating Autoselection Methods Used for Choosing Solutions from Pareto-Optimal Set: Does Nondominance Persist from Calibration to Validation Phase?

Abstract

The calibration of hydrological models using multiobjective algorithms generates several competitive solutions, usually referred to as a Pareto-optimal set. Pareto-optimal solutions are nondominated (i.e., accurate in different ways) and give users several decision scenarios and alternative trade-offs between model evaluation objectives. For decision-making purposes, a single solution is often chosen to represent the properties of the problem under consideration. From a practical standpoint, users are interested to know if selected solutions will continue to be nondominated when evaluated for future periods. In this paper demonstrates an evaluation framework to compare four autoselection methods commonly applied to select a solution from the Pareto-optimal set. The Pareto-optimal sets were generated by using the nondominated sorting genetic algorithm-II (NSGA-II) to calibrate the soil and water assessment tool (SWAT) for simulations of streamflow in the Fairchild Creek watershed in southern Ontario, Canada. The analysis was conducted for 15 calibration outputs in different periods, and each output was evaluated for another 15 different validation periods, resulting in a total of 225 evaluations for each autoselection method. Only a subset of nondominated solutions during the calibration phase remain equally accurate when evaluated at a future time. The results showed that a selection criteria based on a compromise between a representative pathway in parameter space and a dominant variability in objective space is important to finding solutions that remain nondominated for several validation periods. That is, the most suitable solutions are those that have commonalities in parameter space and whose responses at the watershed outlet are similar.