Uncontaminated Data Research Articles

A robust M-estimator for Gaussian ARMA time series based on the Whittle approximation

In the frequency domain, the Whittle approach using the classical periodogram is traditionally used to estimate the parameters of a stationary time series. The M-periodogram has recently become an alternative tool for analyzing the dependence of time series. It is particularly useful for dealing with outliers and heavy-tailed noise. This paper proposes an alternative to the standard Whittle approach, the M-Whittle estimator, built from the M-periodogram. We show that the proposed method is a consistent estimator of the true parameters of an ARMA process. A finite sample investigation is carried out to assess the performance of the estimator in the scenarios of contaminated and uncontaminated time series. As expected, for the uncontaminated data, the M-Whittle estimator performs similarly to the classical Whittle approach. However, the superiority of the first method is clear in terms of root mean squared error when the series has additive outliers. Two applications are considered to illustrate the methodologies in real data contexts. Regardless of whether or not the data are contaminated by additive outliers, the results presented here strongly motivate using the M-Whittle estimator in practical problems.

A screen-time-based mixture model for identifying and monitoring careless and insufficient effort responding in ecological momentary assessment data.

Ecological momentary assessment (EMA) involves repeated real-time sampling of respondents' current behaviors and experiences. The intensive repeated assessment imposes an increased burden on respondents, rendering EMAs vulnerable to respondent noncompliance and/or careless and insufficient effort responding (C/IER). We developed a mixture modeling approach that equips researchers with a tool for (a) gauging the degree of C/IER contamination of their EMA data and (b) studying the trajectory of C/IER across the study. For separating attentive from C/IER behavior, the approach leverages collateral information from screen times, which are routinely recorded in electronically administered EMAs, and translates theoretical considerations on respondents' behavior into component models for attentive and careless screen times as well as for the functional form of C/IER trajectories. We show how a sensible choice of component models (a) allows disentangling short screen times due to C/IER from familiarity effects due to repeated exposure to the same measures, (b) aids in gaining a fine-grained understanding of C/IER trajectories by distinguishing within-day from between-day effects, and (c) allows investigating interindividual differences in attentiveness. The approach shows good parameter recovery when attentive and C/IER screen time distributions exhibit sufficient separation and yields valid conclusions even in scenarios of uncontaminated data. The approach is illustrated on EMA data from the German Socio-Economic Panel innovation sample. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Robust second-order stationary spatial blind source separation using generalized sign matrices

Consider a spatial blind source separation model in which the observed multivariate spatial data are assumed to be a linear mixture of latent stationary spatially uncorrelated random fields. The objective is to recover an unknown mixing procedure as well as the latent random fields. Recently, spatial blind source separation methods that are based on the simultaneous diagonalization of two or more scatter matrices were proposed. In cases involving uncontaminated data, such methods can solve the blind source separation problem, however, in the presence of outlying observations, these methods perform poorly. We propose a robust blind source separation method that employs robust global and local covariance matrices based on generalized spatial signs in simultaneous diagonalization. Simulation studies are employed to illustrate the robustness and efficiency of the proposed methods in various scenarios.

Consistency factor for the MCD estimator at the Student-t distribution

It is well known that trimmed estimators of multivariate scatter, such as the Minimum Covariance Determinant (MCD) estimator, are inconsistent unless an appropriate factor is applied to them in order to take the effect of trimming into account. This factor is widely recommended and applied when uncontaminated data are assumed to come from a multivariate normal model. We address the problem of computing a consistency factor for the MCD estimator in a heavy-tail scenario, when uncontaminated data come from a multivariate Student-t distribution. We derive a remarkably simple computational formula for the appropriate factor and show that it reduces to an even simpler analytic expression in the bivariate case. Exploiting our formula, we then develop a robust Monte Carlo procedure for estimating the usually unknown number of degrees of freedom of the assumed and possibly contaminated multivariate Student-t model, which is a necessary ingredient for obtaining the required consistency factor. Finally, we provide substantial simulation evidence about the proposed procedure and apply it to data from image processing and financial markets.

Trajectory-based RFI subtraction and calibration for radio interferometry

ABSTRACT Radio interferometry calibration and radio frequency interference (RFI) removal are usually done separately. Here we show that jointly modelling the antenna gains and RFI has significant benefits when the RFI follows precise trajectories, such as for satellites. One surprising benefit is improved calibration solutions, by leveraging the RFI signal itself. We present trajectory-based RFI subtraction and calibration (tabascal), a new algorithm that jointly models the RFI and calibration parameters in visibilities. We test tabascal on simulated MeerKAT calibration observations contaminated by satellite-based RFI. We obtain gain estimates that are both unbiased and up to an order of magnitude better constrained compared to uncontaminated data. When combined with an ad hoc RFI subtraction scheme, tabascal solutions can be further applied to an adjacent target observation: 5 min of calibration data results in an image with about a third the noise achieved when using flagging alone. The recovered flux distribution of RFI subtracted data was on par with uncontaminated data. In contrast, RFI flagging alone resulted in a higher detection threshold and consistent underestimation of source fluxes. For a mean RFI amplitude of 17 Jy, using RFI subtraction leads to less than 1 per cent loss of data compared to 75 per cent data loss from an ideal 3σ flagging algorithm, a very significant increase in data available for science analysis. Although we have examined the case of satellite RFI, tabascal should work for any RFI moving on parametrizable trajectories, relative to the phase centre, such as planes and/or objects fixed to the ground.

Enhanced Robust Univariate Classification Methods for Solving Outliers and Overfitting Problems

The robustness of some classical univariate classifiers is hampered if the data are contaminated. Overfitting is another hiccup when the data sets are uncontaminated with a considerable sample size. The performance of the classification models can be easily biased by the outliers’ problems, of which the constructed model tends to be overfitted. Previous studies often used the Bayes Classifier (BC) and the Predictive Classifier (PC) to address two groups of univariate classification problems. Unfortunately for substantial large sample sizes and uncontaminated data, the BC method overfits when the Optimal Probability of Exact Classification (OPEC) is used as an evaluation benchmark. Meanwhile, for small sample sizes, the BC and PC methods are extremely susceptible to outliers. To overcome these two problems, we proposed two methods: the Smart Univariate Classifier (SUC) and the hybrid classifier. The latter is a combination of the SUC and the BC methods, known as the Smart Univariate Bayes Classifier (SUBC). The performance of the new classification methods was evaluated and compared with the conventional BC and PC methods using the OPEC as a benchmark value. To validate the performance of these classification methods, the Probability of Exact Classification (PEC) was compared with the OPEC value. The results showed that the proposed methods outperformed the conventional BC and PC methods based on the real data sets applied. Numerical results also revealed that the SUC method could solve the overfitting problem. The results further indicated that the two proposed methods were robust against outliers. Therefore, these new methods are helpful when practitioners are confronted with overfitting and data contamination problems.

Signal preservation of exomoon transits during light curve folding

In the search for moons around extrasolar planets (exomoons), astronomers are confronted with a stunning observation. Although 3400 of the 4500 exoplanets were discovered with the transit method and although there are well over 25 times as many moons than planets known in the Solar System (two of which are larger than Mercury), no exomoon has been discovered to date. In the search for exoplanet transits, stellar light curves are usually phase-folded over a range of trial epochs and periods. This approach, however, is not applicable in a straightforward manner to exomoons. Planet-moon transits either have to be modeled in great detail (including their orbital dynamics, mutual eclipses, etc.), which is computationally expensive, or key simplifications have to be assumed in the modeling. One such simplification is to search for moon transits outside of the planetary transits. The question we address in this report is how much in-transit data of an exomoon remains uncontaminated by the near-simultaneous transits of its host planet. We develop an analytical framework based on the probability density of the sky-projected apparent position of an exomoon relative to its planet and test our results with a numerical planet-moon transit simulator. For exomoons with planet-moon orbital separations similar to the Galilean moons, we find that only a small fraction of their in-transit data is uncontaminated by planetary transits: 14% for Io, 20% for Europa, 42% for Ganymede, and 73% for Callisto. The signal-to-noise ratio (S/N) of an out-of-planetary-transit folding technique is reduced compared to a full photodynamical model to about 38% (Io), 45% (Europa), 65% (Ganymede), and 85% (Callisto), respectively. For the Earth’s Moon, we find an uncontaminated data fraction of typically just 18% and a resulting S/N reduction to 42%. These values are astonishingly small and suggest that the gain in speed for any exomoon transit search algorithm that ignores the planetary in-transit data comes at the heavy price of losing a substantial fraction of what is supposedly a tiny signal in the first place. We conclude that photodynamical modeling of the entire light curve has substantial, and possibly essential, advantages over folding techniques of exomoon transits outside the planetary transits, in particular for small exomoons comparable to those of the Solar System.

Automatic classification and reduction of wind noise in spectral data.

Outdoor acoustic data often include non-acoustic pressures caused by atmospheric turbulence, particularly below a few hundred Hz in frequency, even when using microphone windscreens. This paper describes a method for automatic wind-noise classification and reduction in spectral data without requiring measured wind speeds. The method finds individual frequency bands matching the characteristic decreasing spectral slope of wind noise. Uncontaminated data from several short-timescale spectra can be used to obtain a decontaminated long-timescale spectrum. This method is validated with field-test data and can be applied to large datasets to efficiently find and reduce the negative impact of wind noise contamination.

Robust Estimation of Ability and Mental Speed Employing the Hierarchical Model for Responses and Response Times

AbstractVan der Linden's hierarchical model for responses and response times can be used in order to infer the ability and mental speed of test takers from their responses and response times in an educational test. A standard approach for this is maximum likelihood estimation. In real‐world applications, the data of some test takers might be partly irregular, resulting from rapid guessing or item preknowledge. The maximum likelihood estimator is not robust against contamination with irregular data. In this article, we propose a robust estimator of ability and mental speed. The estimator consists of two steps. In the first step, the mental speed is estimated with the estimator of Gervini and Yohai that ignores outlying response times. In the second step, the ability is estimated with an M‐estimator that down weights unusual responses given at unusual response times. This is achieved by combining the hard‐rejection weights of Gervini and Yohai with the M‐estimator suggested by Croux and Haesbroeck for the logistic regression model. The proposed estimator is consistent, almost as efficient as the maximum likelihood estimator in uncontaminated data and robust in contaminated data. The performance of the estimator is analyzed in a simulation study and an empirical example.

On a class of mixed EWMA‐CUSUM median control charts for process monitoring

AbstractMonitoring of any manufacturing, production, or industrial process can be controlled and improved by removing these special cause of variations using control charts. Shewhart‐type control charts are effective to control a large amount of special variations, whereas, cumulative sum (CUSUM) and exponentially weighted moving average (EWMA) charts detect small and moderate variations efficiently in the process parameters. Monitoring of location parameter can be done with mean control charts under the assumption that the parameters are known or correctly estimated from in‐control samples and data are free from outliers (but in practice data occasionally have outliers). In this study, we have proposed generalized mixed EWMA‐CUSUM median control charts structures for known and unknown parameters based on auxiliary variables for detecting shifts in process location parameter. The proposed charts are compared with the corresponding charts for the mean, based on contaminated and uncontaminated data. Different performance measures are used to evaluate the performance of proposed control charts and revealed through results that the median‐based charts are more sensitive to detect a shift in process location parameter in the presence of outliers. An illustrative example using real data is also shown for practical consideration.

Trimmed Statistical Estimation via Variance Reduction

In this paper, we show how to transform any optimization problem that arises from fitting a machine learning model into one that (1) detects and removes contaminated data from the training set while (2) simultaneously fitting the trimmed model on the uncontaminated data that remains. To solve the resulting nonconvex optimization problem, we introduce a fast stochastic proximal-gradient algorithm that incorporates prior knowledge through nonsmooth regularization. For data sets of size n, our approach requires O(n2/3/ℇ) gradient evaluations to reach ℇ-accuracy, and when a certain error bound holds, the complexity improves to O(κn2/3 log(1/ℇ)), where κ is a “condition number.” These rates are n1/3 times better than those achieved by typical, nonstochastic methods.

Dating medieval texts by classification with flexible time intervals

AbstractDetermining the date of writing is a practical problem often encountered in the study of ancient and medieval texts. The problem is compounded by differences in genre, register and style, but in particular by our frequent dependence on much later copies of lost originals. This article examines how a method of classification with flexible time intervals has been developed for a corpus of medieval Irish annals (c.700–c.1600). It is shown that the method can deal successfully with an unparsed, complex corpus containing contaminated data. A method of extending the model from good quality, uncontaminated data to more complex texts is also demonstrated by identifying temporal characteristics and typical entries in the high-quality source.

Robust two-sample test of high-dimensional mean vectors under dependence

A basic problem in modern multivariate analysis is testing the equality of two mean vectors in settings where the dimension p increases with the sample size n. This paper proposes a robust two-sample test for high-dimensional data against sparse and strong alternatives, in which the mean vectors of the populations differ in only a few dimensions, but the magnitude of the differences is large. The test is based on trimmed means and robust precision matrix estimators. The asymptotic joint distribution of the trimmed means is established, and the proposed test statistic is shown to have a Gumbel distribution in the limit. Simulation studies suggest that the numerical performance of the proposed test is comparable to that of non-robust tests for uncontaminated data. For cell-wise contaminated data, it outperforms non-robust tests. An illustration involves biomarker identification in an Alzheimer’s disease dataset.

High-dimensional robust precision matrix estimation: Cellwise corruption under $\epsilon $-contamination

We analyze the statistical consistency of robust estimators for precision matrices in high dimensions. We focus on a contamination mechanism acting cellwise on the data matrix. The estimators we analyze are formed by plugging appropriately chosen robust covariance matrix estimators into the graphical Lasso and CLIME. Such estimators were recently proposed in the robust statistics literature, but only analyzed mathematically from the point of view of the breakdown point. This paper provides complementary high-dimensional error bounds for the precision matrix estimators that reveal the interplay between the dimensionality of the problem and the degree of contamination permitted in the observed distribution. We also show that although the graphical Lasso and CLIME estimators perform equally well from the point of view of statistical consistency, the breakdown property of the graphical Lasso is superior to that of CLIME. We discuss implications of our work for problems involving graphical model estimation when the uncontaminated data follow a multivariate normal distribution, and the goal is to estimate the support of the population-level precision matrix. Our error bounds do not make any assumptions about the the contaminating distribution and allow for a nonvanishing fraction of cellwise contamination.

Resolution of habitat-associated ecogenomic signatures in bacteriophage genomes and application to microbial source tracking

Just as the expansion in genome sequencing has revealed and permitted the exploitation of phylogenetic signals embedded in bacterial genomes, the application of metagenomics has begun to provide similar insights at the ecosystem level for microbial communities. However, little is known regarding this aspect of bacteriophage associated with microbial ecosystems, and if phage encode discernible habitat-associated signals diagnostic of underlying microbiomes. Here we demonstrate that individual phage can encode clear habitat-related 'ecogenomic signatures', based on relative representation of phage-encoded gene homologues in metagenomic data sets. Furthermore, we show the ecogenomic signature encoded by the gut-associated ɸB124-14 can be used to segregate metagenomes according to environmental origin, and distinguish 'contaminated' environmental metagenomes (subject to simulated in silico human faecal pollution) from uncontaminated data sets. This indicates phage-encoded ecological signals likely possess sufficient discriminatory power for use in biotechnological applications, such as development of microbial source tracking tools for monitoring water quality.

Detecting central fixation by means of artificial neural networks in a pediatric vision screener using retinal birefringence scanning

BackgroundReliable detection of central fixation and eye alignment is essential in the diagnosis of amblyopia (“lazy eye”), which can lead to blindness. Our lab has developed and reported earlier a pediatric vision screener that performs scanning of the retina around the fovea and analyzes changes in the polarization state of light as the scan progresses. Depending on the direction of gaze and the instrument design, the screener produces several signal frequencies that can be utilized in the detection of central fixation. The objective of this study was to compare artificial neural networks with classical statistical methods, with respect to their ability to detect central fixation reliably.MethodsA classical feedforward, pattern recognition, two-layer neural network architecture was used, consisting of one hidden layer and one output layer. The network has four inputs, representing normalized spectral powers at four signal frequencies generated during retinal birefringence scanning. The hidden layer contains four neurons. The output suggests presence or absence of central fixation. Backpropagation was used to train the network, using the gradient descent algorithm and the cross-entropy error as the performance function. The network was trained, validated and tested on a set of controlled calibration data obtained from 600 measurements from ten eyes in a previous study, and was additionally tested on a clinical set of 78 eyes, independently diagnosed by an ophthalmologist.ResultsIn the first part of this study, a neural network was designed around the calibration set. With a proper architecture and training, the network provided performance that was comparable to classical statistical methods, allowing perfect separation between the central and paracentral fixation data, with both the sensitivity and the specificity of the instrument being 100%. In the second part of the study, the neural network was applied to the clinical data. It allowed reliable separation between normal subjects and affected subjects, its accuracy again matching that of the statistical methods.ConclusionWith a proper choice of a neural network architecture and a good, uncontaminated training data set, the artificial neural network can be an efficient classification tool for detecting central fixation based on retinal birefringence scanning.

A new robust variable weighting coefficients diffusion LMS algorithm

We introduce a new robust algorithm that is insensitive to impulsive noise (IN) for distributed estimation problem over adaptive networks. Motivated by the fact that each node can access to multiple spatial data, we propose to discard IN-contaminated data. Under the assumption that IN is successfully detected, we propose a cost function that considers only the uncontaminated data. The derived algorithm is the ATC diffusion LMS algorithm that has variable weighting coefficients depending on IN detection, which leads both to insensitivity to IN and to good estimation performance. A method to detect IN is also presented. Simulation results show that the proposed algorithm has good estimation performance in an environment that is subject to IN, and outperforms the conventional robust algorithms.

Optimal Data Scaling for Principal Component Pursuit: A Lyapunov Approach to Convergence

In principle component pursuit (PCP), the essential idea is to replace the original non-convex optimization problem of the matrix rank and the count of non-zero entries by a convex optimization problem of the nuclear and I <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norms. In the PCP literature, it is rigorously proved that the validity of this idea depends on the coherence of the uncontaminated data. Specifically, the lower the coherence is, the equivalence of the convex optimization problem to the original non-convex one will hold by a larger probability. Although the coherence index is fixed for a given data set, it is possible to adjust this index by introducing different scalings to the data. The target of this work is thus to find the optimal scaling of the data such that the coherence index is minimized. Based on the analysis of the PCP problem structure, a non-convex optimization problem with implicit dependence on the scaling parameters is firstly formulated. To solve this problem, a coordinate descent algorithm is proposed. Under mild conditions on the structure of the data matrix, the convergence of the algorithm to a global optimal point is rigorously proved by treating the algorithm as a discrete-time dynamic system and utilizing a Lyapunov-type approach. Monte Carlo simulation experiments are performed to verify the effectiveness of the developed results.

Correcting for Sample Contamination in Genotype Calling of DNA Sequence Data

DNA sample contamination is a frequent problem in DNA sequencing studies and can result in genotyping errors and reduced power for association testing. We recently described methods to identify within-species DNA sample contamination based on sequencing read data, showed that our methods can reliably detect and estimate contamination levels as low as 1%, and suggested strategies to identify and remove contaminated samples from sequencing studies. Here we propose methods to model contamination during genotype calling as an alternative to removal of contaminated samples from further analyses. We compare our contamination-adjusted calls to calls that ignore contamination and to calls based on uncontaminated data. We demonstrate that, for moderate contamination levels (5%-20%), contamination-adjusted calls eliminate 48%-77% of the genotyping errors. For lower levels of contamination, our contamination correction methods produce genotypes nearly as accurate as those based on uncontaminated data. Our contamination correction methods are useful generally, but are particularly helpful for sample contamination levels from 2% to 20%.

Detection of Television Frequency Interference with Satellite Microwave Imager Observations over Oceans

AbstractThe geostationary satellite television (TV) signals that are broadcasted over various continents can be reflected back to space when they reach ocean surfaces. If the reflected signals are intercepted by the antenna of the microwave imager on board polar-orbiting satellites, they are mixed with the thermal emission from the earth and result in direct contamination of the satellite microwave imager measurements. This contamination is referred to as television frequency interference (TFI) and can result in erroneous retrievals of oceanic environmental parameters (e.g., sea surface temperature and sea surface wind speed) from microwave imager measurements. In this study, a principal component analysis (PCA)-based method is applied for detecting the TFI signals over oceans from the Advanced Microwave Scanning Radiometer for the Earth Observing System (AMSR-E) Aqua satellite. It is found that the third principal component of the data matrix of the AMSR-E spectral difference indices from each AMSR-E swath captures the TFI contamination. The TFI-contaminated data on the AMSR-E descending node at both 10.65- and 18.7-GHz frequencies can be separated from uncontaminated data over oceanic areas near the coasts of Europe and the United States based on the intensity of the data projection onto the third principal component (PC). Compared to the earlier methods, the proposed PCA-based algorithm works well on the observations without a priori information and is thus applicable for broader user applications.