Imperfect Sampling Research Articles

Bayesian mixture modelling with ranked set samples.

We consider the Bayesian estimation of the parameters of a finite mixture model from independent order statistics arising from imperfect ranked set sampling designs. As a cost-effective method, ranked set sampling enables us to incorporate easily attainable characteristics, as ranking information, into data collection and Bayesian estimation. To handle the special structure of the ranked set samples, we develop a Bayesian estimation approach exploiting the Expectation-Maximization (EM) algorithm in estimating the ranking parameters and Metropolis within Gibbs Sampling to estimate the parameters of the underlying mixture model. Our findings show that the proposed RSS-based Bayesian estimation method outperforms the commonly used Bayesian counterpart using simple random sampling. The developed method is finally applied to estimate the bone disorder status of women aged 50 and older.

Impact of an imperfect sampling device on the security of a continuous-variable source-independent quantum random-number generator with a phase-randomized local oscillator

Impact of an imperfect sampling device on the security of a continuous-variable source-independent quantum random-number generator with a phase-randomized local oscillator

A Novel Classification of Uncertain Stream Data using Ant Colony Optimization Based on Radial Basis Function

There are many potential sources of data uncertainty, such as imperfect measurement or sampling, intrusive environmental monitoring, unreliable sensor networks, and inaccurate medical diagnoses. To avoid unintended results, data mining from new applications like sensors and location-based services needs to be done with care. When attempting to classify data with a high degree of uncertainty, many researchers have turned to heuristic approaches and machine learning (ML) methods. We propose an entirely new ML method in this paper by fusing the Radial Basis Function (RBF) network based on ant colony optimization (ACO). After introducing a large amount of uncertainty into a dataset, we normalize the data and finish training on clean data. The ant colony optimization algorithm is then used to train a recurrent neural network. Finally, we evaluate our proposed method against some of the most popular ML methods, including a k-nearest neighbor, support vector machine, random forest, decision tree, logistic regression, and extreme gradient boosting (Xgboost). Error metrics show that our model significantly outperforms the gold standard and other popular ML methods. Using industry-standard performance metrics, the results of our experiments show that our proposed method does a better job of classifying uncertain data than other methods

Effects of noise correlation and imperfect data sampling on indicators of critical slowing down

Effects of noise correlation and imperfect data sampling on indicators of critical slowing down

A guide to pre‐processing high‐throughput animal tracking data

Modern, high‐throughput animal tracking increasingly yields ‘big data’ at very fine temporal scales. At these scales, location error can exceed the animal's step size, leading to mis‐estimation of behaviours inferred from movement. ‘Cleaning’ the data to reduce location errors is one of the main ways to deal with position uncertainty. Although data cleaning is widely recommended, inclusive, uniform guidance on this crucial step, and on how to organise the cleaning of massive datasets, is relatively scarce.A pipeline for cleaning massive high‐throughput datasets must balance ease of use and computationally efficiency, in which location errors are rejected while preserving valid animal movements. Another useful feature of a pre‐processing pipeline is efficiently segmenting and clustering location data for statistical methods while also being scalable to large datasets and robust to imperfect sampling. Manual methods being prohibitively time‐consuming, and to boost reproducibility, pre‐processing pipelines must be automated.We provide guidance on building pipelines for pre‐processing high‐throughput animal tracking data to prepare it for subsequent analyses. We apply our proposed pipeline to simulated movement data with location errors, and also show how large volumes of cleaned data can be transformed into biologically meaningful ‘residence patches’, for exploratory inference on animal space use. We use tracking data from the Wadden Sea ATLAS system (WATLAS) to show how pre‐processing improves its quality, and to verify the usefulness of the residence patch method. Finally, with tracks from Egyptian fruit bats Rousettus aegyptiacus, we demonstrate the pre‐processing pipeline and residence patch method in a fully worked out example.To help with fast implementation of standardised methods, we developed the R package atlastools, which we also introduce here. Our pre‐processing pipeline and atlastools can be used with any high‐throughput animal movement data in which the high data‐volume combined with knowledge of the tracked individuals' movement capacity can be used to reduce location errors. atlastools is easy to use for beginners while providing a template for further development. The common use of simple yet robust pre‐processing steps promotes standardised methods in the field of movement ecology and leads to better inferences from data.

Refining sampling protocols for cavefishes and cave crayfishes to account for environmental variation

Subterranean habitats represent focal habitats in many conservation strategies; however, these environments are some of the most difficult to sample. New sampling methods, such as environmental DNA (eDNA), show promise to improve stygobiont detection, but sources of sampling bias are poorly understood. Therefore, we determined the factors affecting detection probability using traditional visual surveys and eDNA surveys for both cavefishes and cave crayfishes and demonstrated how detection affects survey efforts for these taxa. We sampled 40 sites (179 visual and 183 eDNA surveys) across the Ozark Highlands ecoregion. We estimated the detection probability of cave crayfishes and cavefishes using both survey methods under varying environmental conditions. The effectiveness of eDNA or visual surveys varied by environmental conditions (i.e., water volume, prevailing substrate, and water velocity) and the target taxa. When sampling in areas with average water velocity, no flow, and coarse substrate, eDNA surveys had a higher detection probability (0.49) than visual surveys (0.35) for cavefishes and visual surveys (0.67) had a higher detection probability than eDNA surveys (0.40) for cave crayfishes. Under the same sampling conditions, 5 visual surveys compared to 10 eDNA surveys would be needed to confidently detect cave crayfishes and 9 visual surveys compared to 4 eDNA surveys for cavefishes. Environmental DNA is a complementary tool to traditional visual surveys; however, the limitations we identified indicate eDNA currently cannot replace visual surveys in subterranean environments. Although sampling designs that account for imperfect sampling are particularly useful, they may not be practical; thus, increasing sampling efforts to offset known detection bias would benefit conservation strategies.

Medium‐term environmental changes influence age‐specific survival estimates in a salmonid population

Abstract Human‐induced environmental change is a major stressor on freshwater habitats that has resulted in the population declines of many freshwater species. Ontogenetic shifts in habitat use and associated (st)age‐specific requirements mean that impacts of environmental stressors can influence (st)ages in a population differently, and yet relatively few studies of freshwater fish populations account for their detail. We aimed to identify environmental and biotic factors affecting survival estimated for six age‐classes of a European grayling population in the River Wylye, UK over a 17‐year period. We used a Bayesian age‐structured state space model to estimate survival of grayling cohorts between subsequent life stages (eggs to age 5 adults) for 16 annual transitions (2003–2004 to 2018–2019), whilst accounting for imperfect sampling of the population. We quantified the effects of seasonal water flow and temperature, in‐stream habitat and prey resource, and potential competitors and predators on survival between subsequent life stages. We used Bayesian variable selection to gauge their relative importance on survival. Grayling abundances declined during the study period (>75% in all age‐classes), predominately driven by a loss of mature adults. Changes to seasonal flows negatively influenced their survival: increased days of summer low flow related to decreased survival of subadults and mature adults, and lower winter flows related to reduced recruitment of juveniles from eggs. Higher summer macrophyte cover negatively influenced juvenile and subadult survival and increasing days of high temperature in summer appeared detrimental to juvenile survival. Abundance of brown trout (a potential competitor and predator) did not negatively influence grayling survival. Our results reveal the implications of environmental change on a salmonid population, where recent low summer flows and high temperatures, and below average winter flows, have negatively influenced grayling survival. These conditions appear to be becoming more frequent and persistent in our study river, which is towards the species’ southern range limit, which could render the population vulnerable to climate change. Our study demonstrates how careful analysis of long‐term population monitoring and environmental datasets can identify factors affecting (st)age‐specific fish population dynamics, and when combined with local expertise, results in realistic mitigation proposals to promote wildlife population persistence.

DQN based single-pixel imaging.

For an orthogonal transform based single-pixel imaging (OT-SPI), to accelerate its speed while degrading as little as possible of its imaging quality, the normal way is to artificially plan the sampling path for optimizing the sampling strategy based on the characteristic of the orthogonal transform. Here, we propose an optimized sampling method using a Deep Q-learning Network (DQN), which considers the sampling process as decision-making, and the improvement of the reconstructed image as feedback, to obtain a relatively optimal sampling strategy for an OT-SPI. We verify the effectiveness of the method through simulations and experiments. Thanks to the DQN, the proposed single-pixel imaging technique is capable of obtaining an optimal sampling strategy directly, and therefore it requires no artificial planning of the sampling path there, which eliminates the influence of the imperfect sampling path planning on the imaging performance.

Detecting community structure in wild populations: a simulation study based on male elephant, Loxodonta africana, data

Many social networks exhibit community structure, where individuals form discrete subgroups. The composition of such groupings is important for numerous research directions, but their characterization is challenged by data sampling issues. In wild populations, where individuals range over large distances and observation can be limited, social data required to resolve community structure are difficult to collect. Recent studies used simulated data sets to determine the robustness of individual level network metrics under suboptimal sampling conditions, but the sensitivity of community detection algorithms to imperfect sampling has not been assessed. Here, we used simulated data sets to determine how sampling effort and skew influence the ability of three community detection algorithms (fastgreedy, walktrap and louvain) to recover the ‘true’ community structure of networks under two sampling regimes (field-based observational sampling and sampling through biologgers, e.g. proximity detectors). We also examined the robustness of a measure of uncertainty in estimated community structure ( r com ). We based our simulated societies on contact patterns in wild male African elephants, a model system reflecting common sampling challenges of large wild populations. Our results indicate that the accuracy of the algorithms improved with increasing sampling effort and decreasing sampling skew. Under the field sampling regime, when sampling effort is constrained, mid-levels of sampling skew may provide a reasonable compromise between maximizing the mean numbers of observation per individual and minimizing sampling skew. Even with skewed data, r com can provide a reliable measure of uncertainty in the estimated community assignments, but it should be interpreted cautiously with highly skewed data. The network structures explored represent common sampling challenges for wild populations, but unexplored sampling regimes may drive somewhat different dynamics. Our simulations indicate that adequate sampling even when skewed can be informative and maximization of the number of observations among all individuals should be a general objective. • Useful information about community structure can be obtained with skewed sampling. • Accurate community assignment requires many observations of each individual. • Moderately skewed sampling can improve results while reducing sampling requirements. • r com is a robust measure of uncertainty even when sampling is skewed. • Results from biologgers are robust with as few as 30% of the population tagged.

Using Arctic ice mass balance buoys for evaluation of modelled ice energy fluxes

Abstract. A new method of sea ice model evaluation is demonstrated. Data from the network of Arctic ice mass balance buoys (IMBs) are used to estimate distributions of vertical energy fluxes over sea ice in two densely sampled regions – the North Pole and Beaufort Sea. The resulting dataset captures seasonal variability in sea ice energy fluxes well, and it captures spatial variability to a lesser extent. The dataset is used to evaluate a coupled climate model, HadGEM2-ES (Hadley Centre Global Environment Model, version 2, Earth System), in the two regions. The evaluation shows HadGEM2-ES to simulate too much top melting in summer and too much basal conduction in winter. These results are consistent with a previous study of sea ice state and surface radiation in this model, increasing confidence in the IMB-based evaluation. In addition, the IMB-based evaluation suggests an additional important cause for excessive winter ice growth in HadGEM2-ES, a lack of sea ice heat capacity, which was not detectable in the earlier study. Uncertainty in the IMB fluxes caused by imperfect knowledge of ice salinity, snow density and other physical constants is quantified (as is inaccuracy due to imperfect sampling of ice thickness) and in most cases is found to be small relative to the model biases discussed. Hence the IMB-based evaluation is shown to be a valuable tool with which to analyse sea ice models and, by extension, better understand the large spread in coupled model simulations of the present-day ice state. Reducing this spread is a key task both in understanding the current rapid decline in Arctic sea ice and in constraining projections of future Arctic sea ice change.

Adaptation of the iterative Marchenko scheme for imperfectly sampled data

SUMMARY The Marchenko method retrieves the responses to virtual sources in the Earth’s subsurface from reflection data at the surface, accounting for all orders of multiple reflections. The method is based on two integral representations for focusing- and Green’s functions. In discretized form, these integrals are represented by finite summations over the acquisition geometry. Consequently, the method requires ideal geometries of regularly sampled and colocated sources and receivers. Recently new representations were derived, which handle imperfectly sampled data. These new representations use point spread functions (PSFs) that reconstruct results as if they were acquired using a perfect geometry. Here, the iterative Marchenko scheme is adapted, using these new representations, to account for imperfect sampling. This new methodology is tested on a 2-D numerical data example. The results show clear improvement of the proposed scheme over the standard iterative scheme. By removing the requirement for perfect geometries, the Marchenko method can be more widely applied to field data.

NGO online disclosures index in the presence of auxiliary information.

This study highlights the need for analysis of online disclosure practices followed by non-governmental organizations; furthermore, it justifies the crucial role of potential correlates of online disclosure practices followed by non-governmental organizations. We propose a novel index for analyzing the extent of online disclosure of non-governmental organizations (NGO). Using the information stored in an auxiliary variable, we propose a new estimator for gauging the average value of the proposed index. Our approach relies on the use of two factors: imperfect ranked-set sampling procedure to link the auxiliary variable with the study variable, and an NGO disclosure index under simple random sampling that uses information only about the study variable. Relative efficiency of the proposed index is compared with the conventional estimator for the population average under the imperfect ranked-set sampling scheme. Mathematical conditions required for retaining the efficiency of the proposed index, in comparison to the imperfect ranked set sampling estimator, are derived. Numerical scrutiny of the relative efficiency, in response to the input variables, indicates; if the variance of the NGO disclosure index is less than the variance of the estimator under imperfect ranked set sampling, then the proposed index is universally efficient compared to the estimator under imperfect ranked set sampling. If the condition on variances is unmet, even then the proposed estimator remains efficient if majority of the NGO share online data on the auxiliary variable. This work can facilitate nonprofit regulation in the countries where most of the non-governmental organizations maintain their websites.

How sure are you? A web-based application to confront imperfect detection of respiratory pathogens in bighorn sheep.

The relationships between host-pathogen population dynamics in wildlife are poorly understood. An impediment to progress in understanding these relationships is imperfect detection of diagnostic tests used to detect pathogens. If ignored, imperfect detection precludes accurate assessment of pathogen presence and prevalence, foundational parameters for deciphering host-pathogen dynamics and disease etiology. Respiratory disease in bighorn sheep (Ovis canadensis) is a significant impediment to their conservation and restoration, and effective management requires a better understanding of the structure of the pathogen communities. Our primary objective was to develop an easy-to-use and accessible web-based Shiny application that estimates the probability (with associated uncertainty) that a respiratory pathogen is present in a herd and its prevalence given imperfect detection. Our application combines the best-available information on the probabilities of detection for various respiratory pathogen diagnostic protocols with a hierarchical Bayesian model of pathogen prevalence. We demonstrated this application using four examples of diagnostic tests from three herds of bighorn sheep in Montana. For instance, one population with no detections of Mycoplasma ovipneumoniae (PCR assay) still had an 6% probability of the pathogen being present in the herd. Similarly, the apparent prevalence (0.32) of M. ovipneumoniae in another herd was a substantial underestimate of estimated true prevalence (0.46: 95% CI = [0.25, 0.71]). The negative bias of naïve prevalence increased as the probability of detection of testing protocols worsened such that the apparent prevalence of Mannheimia haemolytica (culture assay) in a herd (0.24) was less than one third that of estimated true prevalence (0.78: 95% CI = [0.43, 0.99]). We found a small difference in the estimates of the probability that Mannheimia spp. (culture assay) was present in one herd between the binomial sampling approach (0.24) and the hypergeometric approach (0.22). Ignoring the implications of imperfect detection and sampling variation for assessing pathogen communities in bighorn sheep can result in spurious inference on pathogen presence and prevalence, and potentially poorly informed management decisions. Our Shiny application makes the rigorous assessment of pathogen presence, prevalence and uncertainty straightforward, and we suggest it should be incorporated into a new paradigm of disease monitoring.

Discrete representations for Marchenko imaging of imperfectly sampled data

Marchenko imaging is based on integral representations for focusing functions and Green’s functions. In practice, the integrals are replaced by finite summations. This works well for regularly sampled data, but the quality of the results degrades in a case of imperfect sampling. We have developed discrete representations that account for imperfect sampling of the sources (or, via reciprocity, of the receivers). These representations contain point-spread functions that explain the blurring of the focusing functions and Green’s functions due to imperfect sampling. Deblurring the focusing functions and Green’s functions involves multidimensional deconvolution for the point-spread functions. The discrete representations form the basis for modifying Marchenko imaging to account for imperfectly sampled data, which is important for field data applications.

Classically simulating near-term partially-distinguishable and lossy boson sampling

Boson sampling is the problem of sampling from the same distribution as indistinguishable single photons at the output of a linear optical interferometer. It is an example of a non-universal quantum computation which is believed to be feasible in the near term and cannot be simulated on a classical machine. Like all purported demonstrations of ‘quantum computational supremacy’, this motivates optimising classical simulation schemes for a realistic model of the problem, in this case boson sampling when the implementations experience lost or distinguishable photons. Although current simulation schemes for sufficiently imperfect boson sampling are classically efficient, in principle the polynomial runtime can be infeasibly large. In this work, we develop a scheme for classical simulation of boson sampling under uniform distinguishability and loss, based on the idea of sampling from distributions where at most k photons are indistinguishable. We show that asymptotically this scheme can provide a polynomial improvement in the runtime compared to classically simulating idealised boson sampling. More significantly, we show that in the regime considered experimentally relevant, our approach gives an substantial improvement in runtime over other classical simulation approaches.

Characterization of Historical Methane Occurrence Frequencies from U.S. Underground Natural Gas Storage Facilities with Implications for Risk Management, Operations, and Regulatory Policy.

Defining a baseline for the frequency of occurrences at underground natural gas storage facilities is critical to maintaining safe operation and to the development of appropriate risk management plans and regulatory approaches. Currently used frequency-estimation methods are reviewed and broadened in this article to include critical factors of cause, severity, and uncertainty that contribute to risk. A Bayesian probabilistic analysis characterizes the aleatoric historical occurrence frequencies given imperfect sampling. Frequencies for the three main storage facility types in the United States (depleted oil-and-gas field storage, aquifer storage, solution-mined salt cavern storage) are generally on the order of 3 to 9 × 10-2 occurrences, of all causes (surface, well integrity, subsurface integrity) and severities (nuisance, serious, catastrophic), per facility-year. Loss of well integrity is associated with many, but not all, occurrences either within the subsurface or from there up to the surface. The probability of one serious or catastrophic leakage occurrence to the ground surface within the next 10 years, assuming constant number of facilities, is approximately 0.1-0.3% for any facility type. Storage operators and industry regulators can use occurrence frequencies, their associated probabilities and uncertainties, and forecasts of severity magnitudes to better prioritize resources, establish a baseline against which progress toward achieving a reduction target could be measured, and develop more effective mitigation/monitoring/reduction programs in a risk management plan.

Detection error influences both temporal seroprevalence predictions and risk factors associations in wildlife disease models.

Understanding the prevalence of pathogens in invasive species is essential to guide efforts to prevent transmission to agricultural animals, wildlife, and humans. Pathogen prevalence can be difficult to estimate for wild species due to imperfect sampling and testing (pathogens may not be detected in infected individuals and erroneously detected in individuals that are not infected). The invasive wild pig (Sus scrofa, also referred to as wild boar and feral swine) is one of the most widespread hosts of domestic animal and human pathogens in North America.We developed hierarchical Bayesian models that account for imperfect detection to estimate the seroprevalence of five pathogens (porcine reproductive and respiratory syndrome virus, pseudorabies virus, Influenza A virus in swine, Hepatitis E virus, and Brucella spp.) in wild pigs in the United States using a dataset of over 50,000 samples across nine years. To assess the effect of incorporating detection error in models, we also evaluated models that ignored detection error. Both sets of models included effects of demographic parameters on seroprevalence. We compared our predictions of seroprevalence to 40 published studies, only one of which accounted for imperfect detection.We found a range of seroprevalence among the pathogens with a high seroprevalence of pseudorabies virus, indicating significant risk to livestock and wildlife. Demographics had mostly weak effects, indicating that other variables may have greater effects in predicting seroprevalence.Models that ignored detection error led to different predictions of seroprevalence as well as different inferences on the effects of demographic parameters.Our results highlight the importance of incorporating detection error in models of seroprevalence and demonstrate that ignoring such error may lead to erroneous conclusions about the risk associated with pathogen transmission. When using opportunistic sampling data to model seroprevalence and evaluate risk factors, detection error should be included.

Foraging as an evidence accumulation process.

The patch-leaving problem is a canonical foraging task, in which a forager must decide to leave a current resource in search for another. Theoretical work has derived optimal strategies for when to leave a patch, and experiments have tested for conditions where animals do or do not follow an optimal strategy. Nevertheless, models of patch-leaving decisions do not consider the imperfect and noisy sampling process through which an animal gathers information, and how this process is constrained by neurobiological mechanisms. In this theoretical study, we formulate an evidence accumulation model of patch-leaving decisions where the animal averages over noisy measurements to estimate the state of the current patch and the overall environment. We solve the model for conditions where foraging decisions are optimal and equivalent to the marginal value theorem, and perform simulations to analyze deviations from optimal when these conditions are not met. By adjusting the drift rate and decision threshold, the model can represent different “strategies”, for example an incremental, decremental, or counting strategy. These strategies yield identical decisions in the limiting case but differ in how patch residence times adapt when the foraging environment is uncertain. To describe sub-optimal decisions, we introduce an energy-dependent marginal utility function that predicts longer than optimal patch residence times when food is plentiful. Our model provides a quantitative connection between ecological models of foraging behavior and evidence accumulation models of decision making. Moreover, it provides a theoretical framework for potential experiments which seek to identify neural circuits underlying patch-leaving decisions.

Insect population trends and the IUCN Red List process

Reliable assessment of extinction risk is a key factor in the preparation of Red Lists and in prioritizing biodiversity conservation. Temporal population trends can provide important evidence for such assessments, but imperfect sampling (observation errors) and short-term stochastic variation in population levels caused by environmental variability (process errors) can reduce the reliability of trends and lead to incorrect quantification of extinction risk. The assessment of insect taxa is likely to be particularly prone to these problems, due to the highly dynamic nature of many insect populations, driven by short life-cycles and sensitivity to environmental factors such as the weather. Using long-term United Kingdom monitoring data for 54 butterfly and 431 macro-moth species, we demonstrate the impact of insect population variability on the assessment of extinction risk using the International Union for Conservation of Nature (IUCN) Red List Criterion A (reduction in population size over the last 10 years). For both taxa, varying the start year of the 10-year population trend had a substantial effect on whether particular species met Red List thresholds and on the overall number of species assessed as threatened. We conclude that for these insect taxa strict application of the 10-year rule produces Red List classifications that are unacceptably biased by the start year. Use of long-term trends with adjustment based on species performance over the last decade may offer a pragmatic solution to this problem. We call for further IUCN guidance for practitioners undertaking Red List assessments of taxa with populations that have high temporal variability.

Analytical decomposition of Zernike and hexagonal modes over a hexagonal segmented optical aperture

Zernike polynomials are widely used to describe common optical aberrations of a wavefront as they are well suited to the circular geometry of various optical apertures. Non-conventional optical systems, such as future large optical telescopes with highly segmented primary mirrors or advanced wavefront control devices using segmented mirror membrane facesheets, exhibit a hexagonal geometry, making the hexagonal orthogonal polynomials a valued basis. A cost-benefit trade-off study for deriving practical upper limits in, e.g., polishing, phasing, alignment, and stability of hexagons imposes analytical calculation to avoid time-consuming end-to-end simulations, for the sake of exactness. It is important to include global modes over the pupil for Zernike decomposition over a hexagonal segmented optical aperture into the error budget. However, numerically calculated Zernike decomposition is not optimal due to the discontinuities at the segment boundaries that result in imperfect hexagon sampling. In this paper, we present a novel approach for a rigorous Zernike and hexagonal mode decomposition adapted to hexagonal segmented pupils by means of analytical calculations. By contrast to numerical approaches that are dependent on the sampling of the segment, the decomposition expressed analytically only relies on the number and positions of segments comprising the pupil. Our analytical method allows extremely quick results minimizing computational and memory costs. Further, the proposed formulae can be applied independently from the geometrical architecture of segmented optical apertures. Consequently, the method is universal and versatile per se. For instance, this work finds applications in optical metrology and active correction of phase aberrations. In modern astronomy with extremely large telescopes, it can contribute to sophisticated analytical specification of the contrast in the focal plane in presence of aberrations.