Detrending Algorithm Research Articles

Eliminating Noise at the Box-fitting Spectrum

AbstractNon gaussian sources of erros need to be taken into consideration when searching for planetary transits. Such phenomena are mostly caused by the impact of high energetic particles on the detector (Pinheiro da Silva et al. 2008). The detection efficiency of transits, therefor, depend significantly on the data quality and the algorithms utilized to deal with these errors sources. In this work we show that a modified detrend algorithm CDA (CoRoT Detrend Algorithm; Mislis et al. 2010) using a robust statistics and an empirical fit, instead of a polynomial one, can eliminate more efficiently gaps in the data and other long-term trends from the light-curve. Using this algorithm enables us to obtain a reconstructed light-curve with better signal-to-noise ratio that allows to improve the detection of exoplanet transits, although long term signals are destroyed. The results show that these modifications lead to an improved BLS (Box-fitting Least Squares; Kovács, Zucker & Mazeh 2002) algorithm spectrum. At the end we have compared our planetary search results with CoRoT (Convection, Rotation and planetary Transits) satellite chromatic light-curves available in the literature.

Fetal Electrocardiogram Extraction and Performance Analysis

In biomedical engineering, to extract the fetal electrocardiogram (FECG) exactly is a significant and challenging research topic, and so it has been a hot field in biomedical research. Now, a variety of different methods have been proposed to address this problem. From the perspective of blind signal processing, FECG extraction can be modeled as Blind source separation (BSS). In this paper, we present a novel approach, which apply the technique of independent component analysis (ICA) and the theory of wavelet transform, to obtain FECG from the real-life sampled recordings. And in this system diagram, we firstly adopt wavelet de-trending and wavelet de-noising as preprocessing stages to eliminate various kinds of noise, then for these ECG signals processed, in the view of BSS, FastICA algorithm as an ICA method was used to estimate the fetal electrocardiogram signals. Moreover, two different de-trending algorithms are presented to remove the baseline noise. The last but not the least, Performance analysis was provided on the results of the experiment.

A tutorial introduction to adaptive fractal analysis

The authors present a tutorial description of adaptive fractal analysis (AFA). AFA utilizes an adaptive detrending algorithm to extract globally smooth trend signals from the data and then analyzes the scaling of the residuals to the fit as a function of the time scale at which the fit is computed. The authors present applications to synthetic mathematical signals to verify the accuracy of AFA and demonstrate the basic steps of the analysis. The authors then present results from applying AFA to time series from a cognitive psychology experiment on repeated estimation of durations of time to illustrate some of the complexities of real-world data. AFA shows promise in dealing with many types of signals, but like any fractal analysis method there are special challenges and considerations to take into account, such as determining the presence of linear scaling regions.

Improved Time-Series Photometry and Calibration Method for Non-Crowded Fields: MMT Megacam and HAT-South Experiences

AbstractWe present a new photometric reduction method for precise time-series photometry of non-crowded fields that does not need to involve relatively complicated and CPU intensive techniques such as point-spread-function (PSF) fitting or difference image analysis. This method, which combines multi-aperture index photometry and a spatio-temporal de-trending algorithm, gives much superior performance in data recovery and light-curve precision. In practice, the brutal filtering that is often applied to remove outlying data points can result in the loss of vital data, with seriously negative impacts on short-term variations such as flares. Our method utilizes nearly 100% of available data and reduces the rms scatter to several times smaller than that for archived light curves for brighter stars. We outline the details of our new method, and apply it to cases of sample data from the MMT survey of the M37 field, and the HAT-South survey.

Comparison of detrending methods for fluctuation analysis in hydrology

Trends within a hydrologic time series can significantly influence the scaling results of fluctuation analysis, such as rescaled range (RS) analysis and (multifractal) detrended fluctuation analysis (MF-DFA). Therefore, removal of trends is important in the study of scaling properties of the time series. In this study, three detrending methods, including adaptive detrending algorithm (ADA), Fourier-based method, and average removing technique, were evaluated by analyzing numerically generated series and observed streamflow series with obvious relative regular periodic trend. Results indicated that: (1) the Fourier-based detrending method and ADA were similar in detrending practices, and given proper parameters, these two methods can produce similarly satisfactory results; (2) detrended series by Fourier-based detrending method and ADA lose the fluctuation information at larger time scales, and the location of crossover points is heavily impacted by the chosen parameters of these two methods; and (3) the average removing method has an advantage over the other two methods, i.e., the fluctuation information at larger time scales is kept well-an indication of relatively reliable performance in detrending. In addition, the average removing method performed reasonably well in detrending a time series with regular periods or trends. In this sense, the average removing method should be preferred in the study of scaling properties of the hydrometeorolgical series with relative regular periodic trend using MF-DFA.

An algorithm for correcting CoRoT raw light curves

We introduce the CoRoT detrend algorithm (CDA) for detrending CoRoT stellar light curves. The algorithm CDA has the capability to remove random jumps and systematic trends encountered in typical CoRoT data in a fully automatic fashion. Since enormous jumps in flux can destroy the information content of a light curve, such an algorithm is essential. From a study of 1030 light curves in the CoRoT IRa01 field, we developed three simple assumptions which upon CDA is based. We describe the algorithm analytically and provide some examples of how it works. We demonstrate the functionality of the algorithm in the cases of CoRoT0102702789, CoRoT0102874481, CoRoT0102741994, and CoRoT0102729260. Using CDA in the specific case of CoRoT0102729260, we detect a candidate exoplanet around the host star of spectral type G5, which remains undetected in the raw light curve, and estimate the planetary parameters to be Rp=6.27Re and P=1.6986 days.

An iterative filter to reconstruct planetary transit signals in the presence of stellar variability

The detrending algorithms which are widely used to reduce the impact of stellar variability on space-based transit surveys are ill-suited for estimating the parameters of confirmed planets, as they unavoidably alter the transit signal. We present a post-detection detrending algorithm, which filters out signal on other timescales than the period of the transit while preserving the transit signal. We compare the performance of this new filter to a well-established pre-detection detrending algorithm, by applying both to a set of 20 simulated light curves containing planetary transits, stellar variability, and instrumental noise as expected for the CoRoT space mission, and performing analytic fits to the transits. Compared to the pre-detection benchmark, the new post-detection filter systematically yields significantly reduced errors (median reduction in relative error over our sample of about 40%) on the planet-to-star radius ratio, system scale and impact parameter. This is particularly important for active stars, where errors induced by variability can otherwise dominate the final error budget on the planet parameters. Aside from improving planet parameter estimates, the new filter preserves all signal at the orbital period of the planet, and thus could also be used to search for light reflected by the planet.

Detrending time series for astronomical variability surveys

We present a detrending algorithm for the removal of trends in time series. Trends in time series could be caused by various systematic and random noise sources such as cloud passages, changes of airmass, telescope vibration, CCD noise or defects of photometry. Those trends undermine the intrinsic signals of stars and should be removed. We determine the trends from subsets of stars that are highly correlated among themselves. These subsets are selected based on a hierarchical tree clustering algorithm. A bottom-up merging algorithm based on the departure from normal distribution in the correlation is developed to identify subsets, which we call clusters. After identification of clusters, we determine a trend per cluster by weighted sum of normalized light curves. We then use quadratic programming to detrend all individual light curves based on these determined trends. Experimental results with synthetic light curves containing artificial trends and events are presented. Results from other detrending methods are also compared. The developed algorithm can be applied to time series for trend removal in both narrow and wide field astronomy.

Multifractal analysis of sunspot time series: the effects of the 11-year cycle and Fouriertruncation

Multifractal theory provides an elegant statistical characterization of many complex dynamicalvariations in Nature and engineering. It is conceivable that it may enrich characterizationof the sun’s magnetic activity and its dynamical modeling. Recently, on applying Fouriertruncation to remove the 11-year cycle and carrying out multifractal detrended fluctuationanalysis of the filtered sunspot time series, Movahed et al reported that sunspot data arecharacterized by multifractal scaling laws with the exponent for the second-order moment,h(2), being 1.12. Moreover, they think the filtered sunspot data are like a fractional Brownian motionprocess with anti-persistent long-range correlations characterized by the Hurst parameterH = h(2)−1 = 0.12. By designing an adaptive detrending algorithm and critically assessing the effectiveness ofFourier truncation in eliminating the 11-year cycle, we show that the values of the fractalscaling exponents obtained by Movahed et al are artifacts of the filtering algorithm thatthey used. Instead, sunspot data with the 11-year cycle properly filtered are characterizedby a different type of multifractal with persistent long-range correlations characterized byH≈0.74.

Prediction of Sprouting Capacity Using Near-infrared Spectroscopy in Potato Tubers

The potential of near infra-red spectroscopy (NIRS) for predicting the sprouting capacity of potato (Solanum tuberosum L.) tubers was investigated. The experiment was conducted for 2 years using ‘Superior’ and ‘Atlantic’ potatoes. A total of 380 potato tubers (200, Superior; 180, Atlantic) weighing 100–200 g were scanned by NIRS, and the sample sets were divided into four groups; three groups were classified based on the cultivars or years as follows: ‘Superior’, 2004; ‘Superior’, 2005; ‘Atlantic’, 2005; the fourth group contained the total number of samples. The reference value for the sprouting capacities was measured by determining the sprout weight after 30 days of incubation at 20°C. The first derivative transformation equation for raw optical data was standardized by applying standard normal variate (SNV) and detrending (DT) algorithms, and calibration equations between them were developed by modified partial least square (MPLS) regression. The coefficient of determination (r2) ranged from 0.868 to 0.965 with a standard error of calibration (SEC) from 0.31 to 0.40 for the calibration sets; r2 ranged from 0.724 to 0.904 with a standard error of prediction (SEP) from 0.532 to 0.602 for the four validation sets. These results indicate that the calibration model obtained from this experiment is applicable to all the different cultivars for different years, and the NIRS method is a powerful tool for the predictive assessment of the sprouting capacity of potato tubers.

Robust RLS Methods for Online Estimation of Power System Electromechanical Modes

This paper proposes a robust recursive least square (RRLS) algorithm for online identification of power system modes based on measurement data. The measurement data can be either ambient or ringdown. Also, the mode estimation is provided in real-time. The validity of the proposed RRLS algorithm is demonstrated with both simulation data from a 17-machine model and field measurement data from a wide area measurement system (WAMS). Comparison with the conventional recursive least square (RLS) and least mean square (LMS) algorithms shows that the proposed RRLS algorithm can identify the modes from the combined ringdown and ambient signals with outliers and missing data in real-time without noticeable performance degradation. An adaptive detrend algorithm is also proposed to remove the signal trend based on the RRLS algorithm. It is shown that the algorithm can keep up with the measurement data flow and work online to provide real-time mode estimation.

Screening Inorganic Arsenic in Rice by Visible and Near-Infrared Spectroscopy

The potential of near-infrared spectroscopy (NIRS) for screening the inorganic arsenic (i-As) content in commercial rice was assessed. Forty samples of rice were freeze-dried and scanned by NIRS. The i-As contents of the samples were obtained by acid digestion-solvent extraction followed by hydride generation atomic absorption spectrometry, and were regressed against different spectral transformations by modified partial least square (MPLS) regression. The second derivative transformation equation of the raw optical data, previously standardized by applying standard normal variate (SNV) and De-trending (DT) algorithms, resulted in a coefficient of determination in the cross-validation (1-VR) of 0.65, indicative of equations useful for correct separation of the samples in low, medium and high groups. The standard deviation (SD) to standard error of cross-validation (SECV) ratio, expressed in the second derivative equation, was similar to those obtained for other trace metal calibrations reported in NIRS reflectance. Spectral information relating to starch, lipids and fiber in the rice grain, and also pigments in the caryopsis, were the main components used by MPLS for modeling the selected prediction equation. This pioneering use of NIRS to predict the i-As content in rice represents an important reduction in labor input and cost of analysis.

Visible and near-infrared spectroscopy as a technique for screening the inorganic arsenic content in the red crayfish (procambarus clarkii Girard).

The potential of near-infrared spectroscopy (NIRS) for screening the inorganic arsenic (i-As) content in the red crayfish (Procambarus clarkii Girard 1852) was assessed. Sixty-two samples belonging to this species were freeze-dried and scanned by NIRS. The i-As contents of the samples were obtained by acid digestion-solvent extraction followed by hydride generation atomic absorption spectrometry and were regressed against different spectral transformations by modified partial least-squares (MPLS) regression. Second derivative transformation equations of the raw optical data, previously standardized by applying standard normal variate and de-trending algorithms, resulted in a coefficient of determination in the cross-validation (1-VR) of 0.84, indicative of equations of good quantitative information. The standard error of cross-validation to standard deviation ratio, shown by the second derivative equation, was similar to those obtained for other trace metal calibrations reported in NIRS reflectance. Spectral information related to chromophores and lipids of the red crayfish tissues, and also the plant matter contained in their stomachs, were the main organic components used by MPLS for modeling the selected prediction equation. This pioneering use of NIRS to predict the i-As content in red crayfish represents an important savings in time and cost of analysis.

Multivariate estimation of the systematic error of scanning probe microscopes

In the sense of metrology, sample stage actuators (piezos) of scanning probe microscopes are far from being perfect. Therefore raw surface profile data may contain trending components owing to the non-linearity of the actuators. Consequently, raw data should be subjected to a detrending algorithm. According to current standards, this procedure is limited to the removal of piston, slope and curvature. It is a well known effect for experimenters that undesired artifacts may arise, which negatively influence the precision of scanning probe measurements. However, by using the covariance matrix of the surface a method for the multivariate estimation of the systematic error can be derived. It leads to the concept of the principal surface for the removal of instrument errors.

Singular-spectrum analysis: A toolkit for short, noisy chaotic signals

Singular-spectrum analysis (SSA) is developed further, based on experience with applications to geophysical time series. It is shown that SSA provides a crude but robust approximation of strange attractors by tori, in the presence of noise. The method works well for short, noisy time series. The lagged-covariance matrix of the processes studied is the basis of SSA. We select subsets of eigenelements and associated principal components (PCs) in order to provide (i) a noise-reduction algorithm, (ii) a detrending algorithm, and (iii) an algorithm for the identification of oscillatory components. Reconstructed components (RCs) are developed to provide optimal reconstruction of a dynamic process at precise epochs, rather than averaged over the window length of the analysis. SSA is combined with advanced spectral-analysis methods - the maximum entropy method (MEM) and the multi-taper method (MTM) - to refine the interpretation of oscillatory behavior. A combined SSA-MEM method is also used for the prediction of selected subsets of RCs. The entire toolkit is validated against a set of four prescribed time series generated by known processes, quasi-periodic or chaotic. It is also applied to a time series of global surface air temperatures, 130 years long, which has attracted considerable attention in the context of the global warming issue and provides a severe test for noise reduction and prediction.