Comments on “The Shortcomings of Nonlinear Principal Component Analysis in Identifying Circulation Regimes”

Abstract

Christiansen (2005) presents a critique of the use of nonlinear principal component analysis (NLPCA) for the detection of atmospheric circulation regimes. He first notes that for the NLPCA model architecture considered in Monahan et al. (2001, 2003) only a limited class of approximation shapes are possible, and then that the parameterization of the associated time series (and thus its probability distribution) is arbitrary up to a homeomorphism. It is further argued through numerical experiments that NLPCA of Gaussian data produces spurious multimodality, and that the analysis of 20-hPa geopotential height described in Monahan et al. (2003) is not robust. This comment addresses all three of these concerns, starting with the third. The central point of Christiansen (2005) is that the results of Monahan et al. (2000, 2001, 2003) and Teng et al. (2004) are not robust because NLPCA will diagnose spurious regimes in data that are Gaussian (or nearly so). For this argument to hold, the analysis carried out must be identical to the analyses used in these earlier studies. In fact, the NLPCA algorithm described in Christiansen (2005) differs fundamentally from that used in the Monahan et al. and Teng et al. papers in such a way that the spurious results described are to be expected. As is discussed in detail in Monahan (2000), a primary concern with any statistical technique is avoiding overfitting: that is, a good statistical model should be robust to the introduction of new data not used in the estimation of its parameters. As no analytic solution exists to the variational problem central to NLPCA, iterative numerical minimization techniques must be employed: a good statistical model should also be robust to the choice of initial parameter values. These two issues are discussed extensively in the context of cluster analysis by Michelangeli et al. (1995), where they are referred to respectively as the problems of reproducibility and classifiability. In the Monahan et al. and Teng et al. studies, reproducibility and classifiability were assessed using an ensemble approach. For any given dataset, NLPCA was carried out N times (where N was typically at least 50); each ensemble member used different initial parameter values and a different subset of the full dataset (generally 70%–80%, subsampled in such a way to take into account autocorrelation in the time series). The remaining (“validation”) data were not used in parameter estimation, but retained to assess the skill of the resulting statistical model on “new” data. Parameter values for each ensemble member were estimated using conjugate-gradient methods with early stopping: the iterative minimization was carried out for a fixed number of iterations, or until the mean-squared error over the validation set started increasing (indicative of overfitting). Only those ensemble members for which the mean-squared error over the validation data was smaller than that over the estimation data were judged to pass the reproducibility test and therefore to Corresponding author address: Adam H. Monahan, School of Earth and Ocean Sciences, University of Victoria, P.O. Box 3055, STN CSC, Victoria, BC V8W 3P6, Canada. E-mail: monahana@uvic.ca 15 JANUARY 2007 N O T E S A N D C O R R E S P O N D E N C E 375