Detrending Methods Research Articles

Influence of different de-trending methods on stem water relations of <i>Picea meyeri </i>derived from Dendrometer measurements

Influence of different de-trending methods on stem water relations of <i>Picea meyeri </i>derived from Dendrometer measurements

Spurious OLS Estimators of Detrending Method by Adding a Linear Trend in Difference-Stationary Processes—A Mathematical Proof and Its Verification by Simulation

Adding a linear trend in regressions is a frequent detrending method in economic literatures. The traditional literatures pointed out that if the variable considered is a difference-stationary process, then it will artificially create pseudo-periodicity in the residuals. In this paper, we further show that the real problem might be more serious. As the Ordinary Least Squares (OLS) estimators themselves are of such a detrending method is spurious. The first part provides a mathematical proof with Chebyshev’s inequality and Sims–Stock–Watson’s algorithm to show that the OLS estimator of trend converges toward zero in probability, and the other OLS estimator diverges when the sample size tends to infinity. The second part designs Monte Carlo simulations with a sample size of 1,000,000 as an approximation of infinity. The seed values used are the true random numbers generated by a hardware random number generator in order to avoid the pseudo-randomness of random numbers given by software. This paper repeats the experiment 100 times, and gets consistent results with mathematical proof. The last part provides a brief discussion of detrending strategies.

Does signal-free detrending increase chronology coherence in large tree-ring networks?

Over the past decade, dendrochronologists have increasingly adopted the signal-free detrending (SFD) method to remove age-size trends in tree-ring measurement series, amplify the common stand-wide signal in composite chronologies, and recover medium- to low-frequency patterns that may be inadvertently removed by other detrending approaches. However, since its introduction in 2008, no systematic evaluation of the effects of SFD on tree-ring chronologies has been performed. Here we conduct the first review of SFD in dendrochronology and assess its effects when applied to large tree-ring networks. We analyzed the PAGES North America 2 K database of nearly 300 temperature-sensitive chronologies and the Missouri River database of over 350 chronologies curated for the purpose of reconstructing Missouri River streamflow. Both databases contain multiple versions of each chronology generated by different detrending methods, including those produced with (and without) the signal-free procedure applied. We evaluated (i) whether SFD increases chronology coherence at the site level by boosting the between-tree agreement, (ii) whether SFD increases coherence on a regional basis by making neighboring chronologies more similar to each other, and (iii) whether signal-free chronologies retained more medium- to low-frequency variability than their traditional counterparts. We find that, while SFD increased the strength of common signals in many instances, the effect was not universal and some sites even show a decrease in signal coherence. At regional scales, SFD increases chronology coherence in temperature-sensitive records but had no detectable effect on moisture-sensitive records. Our results demonstrate the importance of evaluating the effects of SFD prior to deploying this method for chronology development and paleoclimate reconstruction.

Detection of the Occultation of 55 Cancri e with TESS

Abstract 55 Cancri e is an ultra-short period transiting Super-Earth observed by TESS in Sector 21. Using this photometry, we measure the occultation depth in the TESS bandpass, leveraging the precise transit light curve and comparing multiple detrending methods. We measure the occultation depth to be (15.0 ± 4.8) ppm—a staggeringly small change in brightness, yet one detected by TESS in just a single sector of data. This implies a brightness temperature of 2800 − 160 + 130 K, which is around 1.5σ greater than expected given the mean depth measured with Spitzer. This is not a formally significant difference, and may be accounted for by the known variability, or by an albedo of ∼0.5. In any case, future TESS observations of this system will provide an exciting opportunity to further study this diminutive world’s atmosphere.

Towards standardized processing of eddy covariance flux measurements of carbonyl sulfide

Abstract. Carbonyl sulfide (COS) flux measurements with the eddy covariance (EC) technique are becoming popular for estimating gross primary productivity. To compare COS flux measurements across sites, we need standardized protocols for data processing. In this study, we analyze how various data processing steps affect the calculated COS flux and how they differ from carbon dioxide (CO2) flux processing steps, and we provide a method for gap-filling COS fluxes. Different methods for determining the time lag between COS mixing ratio and the vertical wind velocity (w) resulted in a maximum of 15.9 % difference in the median COS flux over the whole measurement period. Due to limited COS measurement precision, small COS fluxes (below approximately 3 pmol m−2 s−1) could not be detected when the time lag was determined from maximizing the covariance between COS and w. The difference between two high-frequency spectral corrections was 2.7 % in COS flux calculations, whereas omitting the high-frequency spectral correction resulted in a 14.2 % lower median flux, and different detrending methods caused a spread of 6.2 %. Relative total uncertainty was more than 5 times higher for low COS fluxes (lower than ±3 pmol m−2 s−1) than for low CO2 fluxes (lower than ±1.5 µmol m−2 s−1), indicating a low signal-to-noise ratio of COS fluxes. Due to similarities in ecosystem COS and CO2 exchange, we recommend applying storage change flux correction and friction velocity filtering as usual in EC flux processing, but due to the low signal-to-noise ratio of COS fluxes, we recommend using CO2 data for time lag and high-frequency corrections of COS fluxes due to the higher signal-to-noise ratio of CO2 measurements.

Structural Features of the Surface Region of the CdS/por-Si/p-Si Heterostructure with a Porous Silicon Film Formed by Metal-Assisted Chemical Etching

The surface region of the CdS/por-Si/p-Si heterostructure in which the porous layer was formed by metal-assisted etching of the single-crystal p-Si substrate is studied by scanning electron microscopy and Raman spectroscopy methods. Experimental results of the CdS film morphology study are processed using two-dimensional detrended fluctuation analysis and average mutual information methods. It is shown that the porous layer has a significant effect on structural features of the CdS film.

Disentangling ionospheric refraction and diffraction effects in GNSS raw phase through fast iterative filtering technique

We contribute to the debate on the identification of phase scintillation induced by the ionosphere on the global navigation satellite system (GNSS) by introducing a phase detrending method able to provide realistic values of the phase scintillation index at high latitude. It is based on the fast iterative filtering signal decomposition technique, which is a recently developed fast implementation of the well-established adaptive local iterative filtering algorithm. FIF has been conceived to decompose nonstationary signals efficiently and provide a discrete set of oscillating functions, each of them having its frequency. It overcomes most of the problems that arise when using traditional time–frequency analysis techniques and relies on a consolidated mathematical basis since its a priori convergence and stability have been proved. By relying on the capability of FIF to efficiently identify the frequencies embedded in the GNSS raw phase, we define a method based on the FIF-derived spectral features to identify the proper cutoff frequency for phase detrending. To test such a method, we analyze the data acquired from GPS and Galileo signals over Antarctica during the September 2017 storm by the ionospheric scintillation monitor receiver (ISMR) located in Concordia Station (75.10° S, 123.33° E). Different cases of diffraction and refraction effects are provided, showing the capability of the method in deriving a more accurate determination of the sigma_{phi } index. We found values of cutoff frequency in the range of 0.73–0.83 Hz, providing further evidence of the inadequacy of the choice of 0.1 Hz, which is often used when dealing with ionospheric scintillation monitoring at high latitudes.

A Distance-based Method for Spatial Prediction in the Presence of Trend

A new method based on distances for modeling continuous random data in Gaussian random fields is presented. In non-stationary cases in which a trend or drift is present, dealing with information in regionalized mixed variables (including categorical, discrete and continuous variables) is common in geosciences and environmental sciences. The proposed distance-based method is used in a geostatistical model to estimate the trend and the covariance structure, which are key features in interpolation and monitoring problems. This strategy takes full advantage of the information at hand due to the relationship between observations, by using a spectral decomposition of a selected distance and the corresponding principal coordinates. Unconditional simulations are performed to validate the efficiency of the proposed method under a variety of scenarios, and the results show a statistical gain when compared with a more traditional detrending method. Finally, our method is illustrated with two applications: earth’s average daily temperatures in Croatia, and calcium concentration measured at a depth of 0–20 cm in Brazil. Supplementary materials accompanying this paper appear online.

Quality Control and Pre-Analysis Treatment of the Environmental Datasets Collected by an Internet Operated Deep-Sea Crawler during Its Entire 7-Year Long Deployment (2009-2016).

Deep-sea environmental datasets are ever-increasing in size and diversity, as technological advances lead monitoring studies towards long-term, high-frequency data acquisition protocols. This study presents examples of pre-analysis data treatment steps applied to the environmental time series collected by the Internet Operated Deep-sea Crawler “Wally” during a 7-year deployment (2009–2016) in the Barkley Canyon methane hydrates site, off Vancouver Island (BC, Canada). Pressure, temperature, electrical conductivity, flow, turbidity, and chlorophyll data were subjected to different standardizing, normalizing, and de-trending methods on a case-by-case basis, depending on the nature of the treated variable and the range and scale of the values provided by each of the different sensors. The final pressure, temperature, and electrical conductivity (transformed to practical salinity) datasets are ready for use. On the other hand, in the cases of flow, turbidity, and chlorophyll, further in-depth processing, in tandem with data describing the movement and position of the crawler, will be needed in order to filter out all possible effects of the latter. Our work evidences challenges and solutions in multiparametric data acquisition and quality control and ensures that a big step is taken so that the available environmental data meet high quality standards and facilitate the production of reliable scientific results.

Asymmetric Multifractal Analysis of Rebar Futures and Spot Market in China

In this study, a wavelet denoising method is first used to eliminate the influence of noise. Then, an overlapping smooth window technique is introduced into the asymmetric multifractal detrended cross-correlation analysis method, which was combined with the multiscale multifractal analysis method, resulting in the proposed asymmetric multiscale multifractal detrended cross-correlation analysis method. This method not only remedies the pseudo-fluctuation defect of the traditional method, but also explores the asymmetric multifractal cross-correlation between China’s rebar futures and spot markets at different scales. The results show the existence of an asymmetric multifractal cross-correlation between rebar futures and spot markets with upward and downward trends at different scales. This cross-correlation is highly complex at the small-scale, and more pronounced when the futures market is in an uptrend.

Projections on climate internal variability and climatological mean at fine scales over South Korea

Climate internal variability (CIV) plays an important role in understanding climate and is one of the principal uncertainties in climate projections. This study aims to estimate CIV and climatological mean (CM) in predictions using different emission scenarios for South Korea. A stochastic weather generator is employed to generate 100 ensembles of 30-year hourly time series for 40 meteorological stations. CIV is then estimated from the detrended method and compared with the noise computed by the two approaches. The extremely high value of the coefficient of determination between CIV values and noise indicates that the methodologies are seamless. The key results of this study include: (1) national average CM and CIV will increase in the future, and that increase will be greater in Representative Concentration Pathway 8.5 and end periods; (2) the nature of future changes in CM and CIV differ according to the indices of interest. Characteristics of three precipitation-quantity indices (total precipitation, totPr; daily maximum precipitation, maxDa; and hourly maximum precipitation, maxHr) and the precipitation-occurrence index (the number of days without precipitation, nonPr) are largely distinct; (3) examining the relationship between factors of changes of CIV and CM reveal a high correlation between them for maxDa and maxHr, but not for other indices; (4) The tail information of distribution for the FOC ratio implies that future changes in total and extreme precipitation are likely to be decoupled for some months or at some locations. The degree of decoupling is more noticeable on the hourly than the daily scale; and (5) the spatial deviation of CIV is also larger during the summer when CIV values are spatially large; this is valid only for totPr and maxDa. Methodologies and results for finer scales help assess the impact of climate change and develop appropriate adaptation and response strategies.

Wave Signatures in Total Electron Content Variations: Filtering Problems

Over recent years, global navigation satellite systems (GNSSs) have been increasingly used to study near-Earth space. The basis for such studies is the total electron content (TEC) data. Standard procedures for detecting TEC wave signatures include variation selection and detrending. Our experience showed that the inaccurate procedure causes artifacts in datasets which might affect data interpretation, particularly in automated processing. We analyzed the features of various detrending and variation selection methods. We split the problem of the GNSS data filtering into two subproblems: detrending and variation selection. We examined centered moving average, centered moving median, 6th-order polynomial, Hodrick–Prescott filter, L1 filter, cubic smoothing spline, double-applied moving average for the GNSS-TEC detrending problem, and centered moving average, centered moving median, Butterworth filter, type I Chebyshev filter for the GNSS-TEC variation selection problem in this paper. We carried out the analysis based on both model and experimental data. Modeling was based on simple analytical models as well as the International Reference Ionosphere. Analysis of TEC variations of 2–10 min, 10–20 min, and 20–60 min under insufficient detrending conditions showed that the higher errors appear for the longer periods (20–60 min). For the detrending problem, the smoothing cubic spline provided the best results among the algorithms discussed in this paper. The spline detrending featured the minimal value of the mean bias error (MBE) and the root-mean-square error (RMSE), as well as high correlation coefficient. The 6th-order polynomial also produced good results. Spline detrending does not introduce a RMSE more than 0.25 TECU and MBE > 0.35 TECU for IRI trends, while, for the 6th-order polynomial, these errors can exceed 1 TECU in some cases. However, in 95% of observations the RMSE and MBE do not exceed 0.05 TECU. For the variation selection, the centered moving average filter showed the best performance among the algorithms discussed in this paper.

Probabilistic properties of detrended fluctuation analysis for Gaussian processes.

Detrended fluctuation analysis (DFA) is one of the most widely used tools for the detection of long-range dependence in time series. Although DFA has found many interesting applications and has been shown to be one of the best performing detrending methods, its probabilistic foundations are still unclear. In this paper, we study probabilistic properties of DFA for Gaussian processes. Our main attention is paid to the distribution of the squared error sum of the detrended process. We use a probabilistic approach to derive general formulas for the expected value and the variance of the squared fluctuation function of DFA for Gaussian processes. We also get analytical results for the expected value of the squared fluctuation function for particular examples of Gaussian processes, such as Gaussian white noise, fractional Gaussian noise, ordinary Brownian motion, and fractional Brownian motion. Our analytical formulas are supported by numerical simulations. The results obtained can serve as a starting point for analyzing the statistical properties of DFA-based estimators for the fluctuation function and long-memory parameter.

Wind effects on soil thermal properties measured by the dual‐probe heat pulse method

AbstractWhen a dual‐probe heat pulse (DPHP) sensor is installed in soil near the soil–atmosphere interface, the basic assumptions of the infinite line heat source (ILS) model and its improvement, the infinite line heat source model with an adiabatic boundary condition (ILS‐ABC) might not be satisfied because of wind. This study aims to explore the effects of wind on DPHP measurements and to compare the performance of the ILS and ILS‐ABC models at different values of wind velocity (υ) and burial depth (d). Our study shows that the results of laboratory experiments, COMSOL simulations, and field experiments are consistent with each other. For dry sand with d ≤ 4 mm, the effects of wind is non‐negligible at υ ≥ 3.5 m s−1 and the DPHP method does not provide accurate estimations whether the ILS model or the ILS‐ABC model is used. Field experiments are prone to large background temperature fluctuations, which can cause a commonly used linear detrending method to perform poorly. In general, at υ < 2.7 m s−1 in field and d = 5 mm, the ILS‐ABC model provides more accurate estimations with relative errors of <15% in thermal conductivity and relative errors of <9% in heat capacity.

Attribution Analysis of Long-Term Trends of Aridity Index in the Huai River Basin, Eastern China

This paper aims to combinedly investigate the spatiotemporal trends of precipitation (Pre), reference evapotranspiration (ET0), and aridity index (AI) by employing nonparametric methods based on daily datasets from 137 meteorological stations during 1961–2014 in the Huai River Basin (HRB). The dominant factors influencing ET0 and AI trends were also explored using the detrended and differential equation methods. Results show that (1) Pre, ET0, and AI were much larger in summer than in other seasons, and AI had a nonsignificant increasing trend in annual time scale, while Pre and ET0 exhibited decreasing trends, but AI showed a downward trend in spring and autumn (becoming drier) and an upward trend during summer and winter due to increased Pre (becoming wetter); (2) lower AI values were identified in north and higher in south, and lower ET0 was identified in south and higher in north in annual time scale, growing season and spring, while ET0 decreased from west to east in summer and winter, the spatial distribution of Pre was similar to that of AI; (3) for ET0 trends, in general, wind speed at two-meter height (u2) was the dominant factor in spring, autumn, winter, and annual time scale, while in other seasons, solar radiation (Rs) played a dominant role; (4) for AI trends, AI was mostly contributed by Pre in spring, autumn, and winter, the Rs contributed the most to AI trend in growing season and summer, then in annual time scale, u2 was the dominant factor; (5) overall, the contribution of Pre to AI trends was much larger than that of ET0 in spring, autumn, and winter, while AI was mostly contributed by ET0 in annual time scale, growing season and summer. The outcomes of the study may improve our scientific understanding of recent climate change effects on dry–wet variations in the HRB; moreover, this information may be utilized in other climatic regions for comparison analyses.

Detrending Exoplanetary Transit Light Curves with Long Short-term Memory Networks

Abstract The precise derivation of transit depths from transit light curves is a key component for measuring exoplanet transit spectra, and henceforth for the study of exoplanet atmospheres. However, it is still deeply affected by various kinds of systematic errors and noise. In this paper we propose a new detrending method by reconstructing the stellar flux baseline during transit time. We train a probabilistic long short-term memory (LSTM) network to predict the next data point of the light curve during the out-of-transit, and use this model to reconstruct a transit-free light curve—i.e., including only the systematics—during the in-transit. By making no assumption about the instrument, and using only the transit ephemeris, this provides a general way to correct the systematics and perform a subsequent transit fit. The name of the proposed model is TLCD-LSTM, standing for transit light-curve detrending-LSTM. Here we present the first results on data from six transit observations of HD 189733b with the IRAC camera on board the Spitzer Space Telescope, and discuss some of its possible further applications.

The cyclically-adjusted primary balance: A novel approach for the euro area

This paper presents novel estimates for the cyclically-adjusted primary balance for 18 countries of the Euro area over years 1999–2017. We improve the methodology adopted by the European Commission by using quarterly rather than annual frequency data and providing accurate identification of the budgetary items whose response can be considered automatic to the economic cycle. This disaggregated outcome combined with high frequency data marks a significant improvement with respect to previous studies. The empirical analysis is implemented on two sub-periods to examine the impact of governments’ discretionary fiscal policy before and after the Great Recession. The most striking policy implication is that even though the budgetary policy of most European countries can be qualified in principle as anticyclical, this outcome has been weakened by the impact of discretionary policies of many governments especially after the crisis. The results are robust to the use of different de-trending methods.

Glacial Aerodynamic Roughness Estimates: Uncertainty, Sensitivity, and Precision in Field Measurements

AbstractCalculation of the sensible and latent heat (turbulent) fluxes is required in order to close the surface energy budget of glaciers and model glacial melt. The aerodynamic roughness length, z0, is a key parameter in the bulk approach to calculating sensible heat flux; yet, z0 is commonly considered simply as a tuning parameter or generalized between surfaces and over time. Spatially and temporally distributed observations of z0 over ice are rare. Both direct (from wind towers and sonic anemometers) and indirect (from microtopographic surveys) measurements of z0 are subject to sensitivities and uncertainties that are often unstated or overlooked. In this study, we present a quantitative evaluation of aerodynamic profile‐based and microtopographic methods and their effect on z0 using data collected from Storglaciären and Sydöstra Kaskasatjäkkaglaciären, Tarfala Valley, Arctic Sweden. Aggressive data filters discard most of the wind tower data but still produce realistic z0 values of 1.9 mm and 2 mm. Despite uncertainty introduced by scale and resolution dependence, microtopographic methods produced estimates of z0 comparable to wind tower values and those found on similar surfaces. We conclude that (1) in the absence of direct turbulent flux measurements from sonic anemometers, the profile and microtopographic methods provide realistic z0 values, (2) both 2D and 3D microtopographic methods are dependent on scale, resolution, and the chosen detrending method, and (3) careful calibration of these parameters could enable glacier‐wide investigations of z0 from remotely sensed data, including those increasingly available from satellite platforms.

Ranking of tree-ring based hydroclimate reconstructions of the past millennium

To place recent hydroclimate changes, including drought occurrences, in a long-term historical context, tree-ring records serve as an important natural archive. Here, we evaluate 46 millennium-long tree-ring based hydroclimate reconstructions for their Data Homogeneity, Sample Replication, Growth Coherence, Chronology Development, and Climate Signal based on criteria published by Esper et al. (2016) to assess tree-ring based temperature reconstructions. The compilation of 46 individually calibrated site reconstructions includes 37 different tree species and stem from North America (n = 29), Asia (n = 10); Europe (n = 5), northern Africa (n = 1) and southern South America (n = 1). For each criterion, the individual reconstructions were ranked in four groups, and results showed that no reconstruction scores highest or lowest for all analyzed parameters. We find no geographical differences in the overall ranking, but reconstructions from arid and semi-arid environments tend to score highest. A strong and stable hydroclimate signal is found to be of greater importance than a long calibration period. The most challenging trade-off identified is between high continuous sample replications, as well as a well-mixed age class distribution over time, and a good internal growth coherence. Unlike temperature reconstructions, a high proportion of the hydroclimate reconstructions are produced using individual series detrending methods removing centennial-scale variability. By providing a quantitative and objective evaluation of all available tree-ring based hydroclimate reconstructions we hope to boost future improvements in the development of such records and provide practical guidance to secondary users of these reconstructions.

Fractal Analysis of Power Grid Faults and Cross Correlation for the Faults and Meteorological Factors

Power-grid faults pose a great threat to the economy and to social stability. In this paper, the detrended fluctuation analysis method is used to investigate the scaling properties of power grid faults and the correlation between faults and meteorological factors that are closely related to power grid faults. Multifractal detrended fluctuation analysis showed that the fault time series were multifractal. Further investigations revealed the origins of multifractality in power-grid fault time series, and the results showed that the temporal correlations in the data represent the distribution of the returns, which are a significant source of multifractal scaling. Then, cross correlations between the fault and the four meteorological factors were investigated using the detrended cross-correlation analysis method. The results showed that maximum wind speed has a considerable impact on the number of transmission system line faults per day, whereas daily precipitation, daily mean air temperature, and maximum wind speed have a considerable impact on the number of distribution system line faults per day.