The effect of end-point adjustments on smoothing splines used for tree-ring standardization

Abstract

The cubic smoothing spline has been a popular method for detrending tree-ring data since the 1980s. The common implementation of this procedure (e.g., ARSTAN, dplR) uses a unique method for determining the smoothing parameter that is widely known as the %n criterion. However, this smoothing parameter selection method carries the assumption that end point effects are ignorable. In this paper, we complete the mathematical derivation and show how the original method differs from the complete version, both in the interpretation of the smoothing parameter and in the spline fit. Frequency response curves (FRC) demonstrate how the smoothing parameter is affected by the original assumption. For example, the FRC results indicate that a tree core of 250-year length has a 14% difference in the cut-off frequency when looking at the 67%n criterion. The FRC analysis shows that the existing approach produces a more flexible fit than anticipated, i.e., it is removing more variance than previously thought. For example, a 67%n spline under the existing approach corresponds to a 53%n spline fit. By using both simulated tree-core sequences and a dataset from a Midwest forest, we discuss which conditions result in greater differences between the spline fits and which conditions will have small differences. Tree-core sequences that have more curvature, such as a large-amplitude growth release, will lead to greater differences. Finally, we provide approximations to the end-point effect procedure. For example, using an 83%n criterion under the original approach produces a spline fit approximating the 67%n fit under the complete approach. These approximations could be easily implemented within existing programs like ARSTAN.