Horvitz-Thompson Estimator Research Articles

The Bahadur representations of quantile estimators in general unequal probability sampling

In this study, we establish the Bahadur representations of quantile estimators based on the Hájek and the Horvitz–Thompson estimators for the population distribution function in unequal probability sampling designs under general conditions. We also present the asymptotic normality of the above quantile estimators. The simulation studies using real data confirm the validity of the asymptotic normality.

Risk minimization using robust experimental or sampling designs and mixture of designs

For both experimental and sampling designs, the efficiency or balance of designs has been extensively studied. There are many ways to incorporate auxiliary information into designs. However, when we use balanced designs to decrease the variance due to an auxiliary variable, the variance may increase due to an effect which we define as lack of robustness. This robustness can be written as the largest eigenvalue of the variance operator of a sampling or experimental design. If this eigenvalue is large, then it might induce a large variance in the Horvitz–Thompson estimator of the total. We calculate or estimate the largest eigenvalue of the most common designs. We determine lower, upper bounds and approximations of this eigenvalue for different designs. Then, we compare these results with simulations that show the trade-off between efficiency and robustness. Those results can be used to determine the proper choice of designs for experiments such as clinical trials or surveys. We also propose a new and simple method for mixing two sampling designs, which allows to use a tuning parameter between two sampling designs. This method is then compared to the Gram–Schmidt walk design, which also governs the trade-off between robustness and efficiency. A set of simulation studies shows that our method of mixture gives similar results to the Gram–Schmidt walk design while having an interpretable variance matrix.

A semi-parametric maximum-likelihood analysis of measurement error in population size estimation

Abstract This work addresses the challenge of measurement errors in capture–recapture (CR) studies with covariates. These errors can introduce bias and undermine inference quality. To address this issue, we introduce a nonparametric measurement error model tailored to the ‘repeated counts’ setting, employing EM-type algorithms for parameter estimation. We use the Horvitz–Thompson estimator for population size estimates. Rigorous simulations, covering varying degrees of measurement error reliability, confirm our approach’s effectiveness. Applied to benchmark datasets, it consistently provides more accurate point estimates and robust uncertainty quantification, enhancing the reliability of CR analyses.

Estimator Comparison for the Prediction of Election Results

Cluster randomized experiments and estimator comparisons are well-documented topics. In this paper, using the datasets of the popular vote in the presidential elections of the United States of America (2012, 2016, 2020), we evaluate the properties (SE, MSE) of three cluster sampling estimators: Ratio estimator, Horvitz–Thompson estimator and the linear regression estimator. While both the Ratio and Horvitz–Thompson estimators are widely used in cluster analysis, we propose a linear regression estimator defined for unequal cluster sizes, which, in many scenarios, performs better than the other two. The main objective of this paper is twofold. Firstly, to indicate which estimator is most suited for predicting the outcome of the popular vote in the United States of America. We do so by applying the single-stage cluster sampling technique to our data. In the first partition, we use the 50 states plus the District of Columbia as primary sampling units, whereas in the second one, we use 3112 counties instead. Secondly, based on the results of the aforementioned procedure, we estimate the number of clusters in a sample for a set standard error while also considering the diminishing returns from increasing the number of clusters in the sample. The linear regression estimator is best in the majority of the examined cases. This type of comparison can also be used for the estimation of any other country’s elections if prior voting results are available.

One‐Inflation and Zero‐Truncation Count Data Modelling Revisited With a View on Horvitz–Thompson Estimation of Population Size

SummaryEstimating the size of a hard‐to‐count population is a challenging matter. We consider uni‐list approaches in which the count of identifications per unit is the basis of analysis. Unseen units have a zero count and do not occur in the sample leading to a zero‐truncated setting. Because of various mechanisms, one‐inflation is often an occurring phenomena that can lead to seriously biased estimates of population size. The current work reviews some recent advances on one‐inflation and zero‐truncation modelling, and furthermore focuses here on the impact it has on population size estimation. The zero‐truncated one‐inflated and the one‐inflated zero‐truncated model is compared (also with the model ignoring one‐inflation) in terms of Horvitz–Thompson estimation of population size. The simulation work shows clearly the biasing effect of ignoring one‐inflation. Both models, the zero‐truncated one‐inflated and the one‐inflated zero‐truncated one, are suitable to model ongoing one‐inflation. It is also important to choose an appropriate base‐line distributional model. Finally, all models derived in the paper are illustrated on a number of case studies.

Inference in Coarsened Time Series via Generalized Method of Moments

We study statistical inference procedures in coarsened time series through the generalized method of moments. A new model for the coarsened time series via multiple potential outcomes is proposed. It can be naturally extended for inferring multi‐variate coarsened time series. We show that this framework generates a general class of estimators. It neatly generalizes the classical Horvitz–Thompson estimator for handling coarsened time series data. Asymptotic properties, including consistency and limiting distribution, of the proposed estimators are investigated. Estimators of the optimal weight matrix and the long‐run covariance matrix are also derived. In particular, confidence intervals of the mean function of the potential outcome as a function of coarsening index can be constructed. A real‐data application on air quality in the USA is investigated.

Digital loans and buy now pay later from LendTech versus bank loans in the era of ‘black swans’: Complementarity in the area of consumer financing

Research background: People who take non-banking loans are primarily perceived as excluded from accessing bank services. The growth of e-commerce and the increasing digitalisation of customer interactions with banks was particularly accelerated by the COVID-19 pandemic (the first ‘black swan’). These processes have also influenced the rapid growth of the LendTech (LT) sector within FinTech with its digital loans and buy-now-pay-later (BNPL) services. The war in Ukraine (the second ‘black swan’) has led to an energy crisis, increased inflation, interest rates and credit costs, and reduced credit accessibility. In this context, the following research questions are addressed: Are the LT and banking sectors complementary or substitutive in the area of consumer financing? Does complementarity apply to all customer segments and loan amounts? How does the extent of complementarity or substitutability of the LT sector depend on, and to what extent do changes in the regulatory and macroeconomic environment affect, the interaction between the banking and LT sectors? Purpose of the article: The aim of the article is to identify trends in the LT sector within FinTech in the context of assessing the scale and determinants of current and future complementarity of the non-bank sector and the banking sector in the area of consumer credit in time of black swans. An additional purpose of the article is to estimate revenues from the basic operating activities of companies from the on-line channel. Methods: The research process was multi-stage and the research procedure was structured. Due to the lack of a uniform source of data on LT products and services, the study used many data sources — data from the Credit Information Bureau, a primary nationwide survey on LT users a primary survey of people representing LT's managerial staff. The selection of LTs was carried out according to the concept of the monetary sampling unit. The Horvitz-Thompson estimator with Sen‐Yates‐Grundy variance form was used to estimate net operating income for LT from the online channel in 2021. Findings & value added: The LT companies surveyed state that black swans (the COVID-19 pandemic and the war in Ukraine) and the current macroeconomic situation have not significantly affected demand for non-banking loans. The reduction in the opportunities for the LT sector as a result of anti-usury regulations will only lead to a shift in consumer demand to the pawnshop sector and the shadow economy, which will be detrimental to consumers. Complementarity between LT and the banking sector is revealed especially in crisis situations, thus limiting the effects of market shock (limited supply of loans offered by banks). In periods of stabilisation, a rather clear division of preferences is noticeable – in the case of seniors and loans for higher amounts, banks dominate, while in the case of lower amounts and in younger age groups, non-bank institutions are more popular. The mechanism by which shrinking banking services are replaced by LT in short-term crises confirms the importance of LT in balancing the Consumer Finance market in the face of unstable periods. From a medium- and long-term perspective, it should be noted that inflation and rising interest rates will increase the scale of credit exclusion in commercial banks, shifting part of the demand to the non-bank sector. These phenomena have an international dimension. Similar observations were made already in 2012 by the CFPB in the USA and the British FTA, when analysing the consumer finance market immediately after the subprime crisis, emphasising the effects of limiting access to bank consumer loans and the resulting growth of the LT market (Gębski, 2013).

Ratio Type Estimator for Balanced Sampling Plan excluding Adjacent Units

The study variables are usually correlated with the auxiliary variables and therefore their correlation could easily be computed. In this paper, the correlation coefficient is used for the estimation procedure, and therefore, we proposed a Horvitz-Thompson ratio-type estimator using correlation coefficient for Balanced Sampling plan excluding Adjacent units (BSA plan). It has been illustrated theoretically and empirically that the proposed Horvitz-Thompson ratio-type estimator is more precise than the Horvitz-Thompson estimator based on BSA plan. The proposed estimator provides an opportunity to utilise the auxiliary information for the estimation of population mean for BSA plan and useful for many real-life experiments. International Journal of Statistical Sciences, Vol.24(1), March, 2024, pp 103-114

Calibration Estimation of Population Proportion in Probability Proportional to Size Sampling in the Presence of Non-Response

Objectives: In this article, we addressed the problem of estimation of the finite population proportion under the Probability Proportional to Size (PPS) sampling technique, when the complete information is unavailable due to the presence of non-response. We developed calibrated estimators of the population proportion under PPS sampling in the presence of nonresponse based on the availability of auxiliary information. Methods: The expressions for the mean squared errors of the suggested estimators were developed to the first order of approximation. The developed estimators of the population proportion are compared with the design-based Horvitz-Thompson estimator and Horvitz-Thompson type calibration estimator which were obtained on the complete response units along with the design-based Hansen and Hurwitz type estimator in the presence of non-response. A Simulation study has also been conducted to support the performance of the developed estimators of population proportion with the help of two real datasets, by computing Percentage Absolute Relative Bias (%ARB) and Percentage Relative Root Mean Squared Error (%RRMSE) using R software. Findings: The simulation study supported the performance of the developed estimators of the finite population proportion based on %ARB and %RRMSE. The proposed calibration estimators of population proportion are more efficient than the other considered estimators in the presence of non-response. Novelty: The proposed new calibrated estimators have practical implications in the estimation of finite population proportions. Keywords: Auxiliary information, Calibration Approach, Non­response, Population proportion, PPS sampling

Detection Probability and Bias in Machine-Learning-Based Unoccupied Aerial System Non-Breeding Waterfowl Surveys

Unoccupied aerial systems (UASs) may provide cheaper, safer, and more accurate and precise alternatives to traditional waterfowl survey techniques while also reducing disturbance to waterfowl. We evaluated availability and perception bias based on machine-learning-based non-breeding waterfowl count estimates derived from aerial imagery collected using a DJI Mavic Pro 2 on Missouri Department of Conservation intensively managed wetland Conservation Areas. UASs imagery was collected using a proprietary software for automated flight path planning in a back-and-forth transect flight pattern at ground sampling distances (GSDs) of 0.38–2.29 cm/pixel (15–90 m in altitude). The waterfowl in the images were labeled by trained labelers and simultaneously analyzed using a modified YOLONAS image object detection algorithm developed to detect waterfowl in aerial images. We used three generalized linear mixed models with Bernoulli distributions to model availability and perception (correct detection and false-positive) detection probabilities. The variation in waterfowl availability was best explained by the interaction of vegetation cover type, sky condition, and GSD, with more complex and taller vegetation cover types reducing availability at lower GSDs. The probability of the algorithm correctly detecting available birds showed no pattern in terms of vegetation cover type, GSD, or sky condition; however, the probability of the algorithm generating incorrect false-positive detections was best explained by vegetation cover types with features similar in size and shape to the birds. We used a modified Horvitz–Thompson estimator to account for availability and perception biases (including false positives), resulting in a corrected count error of 5.59 percent. Our results indicate that vegetation cover type, sky condition, and GSD influence the availability and detection of waterfowl in UAS surveys; however, using well-trained algorithms may produce accurate counts per image under a variety of conditions.

Correlated Cluster-Based Randomized Experiments: Robust Variance Minimization

Experimentation is prevalent in online marketplaces and social networks to assess the effectiveness of new market intervention. To mitigate the interference among users in an experiment, a common practice is to use a cluster-based experiment, where the designer partitions the market into loosely connected clusters and assigns all users in the same cluster to the same variant (treatment or control). Given the experiment, we assume an unbiased Horvitz–Thompson estimator is used to estimate the total market effect of the treatment. We consider the optimization problem of choosing (correlated) randomized assignments of clusters to treatment and control to minimize the worst-case variance of the estimator under a constraint that the marginal assignment probability is [Formula: see text] for all clusters. This problem can be formulated as a linear program where both the number of decision variables and constraints are exponential in the number of clusters—and hence is generally computationally intractable. We develop a family of practical experiments that we refer to as independent block randomization (IBR) experiments. Such an experiment partitions clusters into blocks so that each block contains clusters of similar size. It then treats a fraction q of the clusters in each block (chosen uniformly at random) and does so independently across blocks. The optimal cluster partition can be obtained in a tractable way using dynamic programming. We show that these policies are asymptotically optimal when the number of clusters grows large and no cluster size dominates the rest. In the special case where cluster sizes take values in a finite set and the number of clusters of each size is a fixed proportion of the total number of clusters, the loss is only a constant that is independent of the number of clusters. Beyond the asymptotic regime, we show that the IBR experiment has a good approximation for any problem instance when q is not very tiny. We also examine the performance of the IBR experiments on data-driven numerical examples, including examples based on Airbnb and Facebook data. This paper was accepted by Itai Ashlagi, revenue management and market analytics. Funding: O. Candogan acknowledges NSF [Award 2216912] for “Institute for Data, Econometrics, Algorithms and Learning.” Supplemental Material: The online appendix is available at https://doi.org/10.1287/mnsc.2021.02741 .

The Small Area Estimation of Economic Security: A Proposal

The objective of this work is to propose a small area estimation strategy for an economic security indicator. In the last decade the interest for the measurement of economic security or insecurity has grown constantly, especially since the financial crisis of 2008 and the pandemic period. In this work, economic security is measures through a longitudinal indicator that compares levels of equivalized household income over time. To solve a small area estimation problem, due to possible sample sizes too low in some areas, a small area estimation strategy is suggested to obtain reliable estimates of the indicator of interest. We consider small area models specified at area level. Besides the basic Fay-Herriot area-level model, we propose to consider some longitudinal extensions, including time-specific random effects following an AR(1) process or an MA(1) process. A simulation study based on EU-SILC data shows that all the small area models considered provide a significant efficiency gain with respect to the Horvitz-Thompson estimator, especially the small area model with MA(1) specification for random effects.

An improved method for line transect sampling in Antarctic minke whale surveys

The series of abundance estimates for Antarctic minke whales obtained using standard line transect methods from IWC/SOWER surveys imply drastic (and probably unrealistic) changes in true abundance. One possible factor is that the detection probability on the trackline, g(0), may have decreased with the introduction of inexperienced observers in the most recent surveys. Additionally, mean observed school size may have decreased in the third circumpolar survey in comparison with the second survey. This paper introduces an extended and generalised hazard probability model without the assumption that g(0)=1 to estimate true school size distribution in the population. The proposed method uses a survey design that combines the use of both passing mode with independent observers and closing mode in which the vessel turns off the trackline and closes with the sighting for confirmation of school size and species. The abundance estimate is based on the Horvitz-Thompson estimator in an unequal detectability sampling scheme. The method is applied to the IDCR/SOWER dataset of Antarctic minke whales for illustrative purposes.

Adaptive normalization for IPW estimation

Abstract Inverse probability weighting (IPW) is a general tool in survey sampling and causal inference, used in both Horvitz–Thompson estimators, which normalize by the sample size, and Hájek/self-normalized estimators, which normalize by the sum of the inverse probability weights. In this work, we study a family of IPW estimators, first proposed by Trotter and Tukey in the context of Monte Carlo problems, that are normalized by an affine combination of the sample size and a sum of inverse weights. We show how selecting an estimator from this family in a data-dependent way to minimize asymptotic variance leads to an iterative procedure that converges to an estimator with connections to regression control methods. We refer to such estimators as adaptively normalized estimators. For mean estimation in survey sampling, the adaptively normalized estimator has asymptotic variance that is never worse than the Horvitz–Thompson and Hájek estimators. Going further, we show that adaptive normalization can be used to propose improvements of the augmented IPW (AIPW) estimator, average treatment effect (ATE) estimators, and policy learning objectives. Appealingly, these proposals preserve both the asymptotic efficiency of AIPW and the regret bounds for policy learning with IPW objectives, and deliver consistent finite sample improvements in simulations for all three of mean estimation, ATE estimation, and policy learning.

A decade of humpback whale abundance estimates at Bermuda, an oceanic migratory stopover site

We constructed annual abundance of a migratory baleen whale at an oceanic stopover site to elucidate temporal changes in Bermuda, an area with increasing anthropogenic activity. The annual abundance of North Atlantic humpback whales visiting Bermuda between 2011 and 2020 was estimated using photo-identification capture-recapture data for 1,204 whales, collected between December 2009 and May 2020. Owing to a sparse data set, we combined a Cormack-Jolly-Seber (CJS) model, fit through maximum likelihood estimation, with a Horvitz-Thompson estimator to calculate abundance and used stratified bootstrap resampling to derive 95% confidence intervals (CI). We accounted for temporal heterogeneity in detection and sighting ratesviaa catch-effort model and, guided by goodness-of-fit testing, considered models that accounted for transience. A model incorporating modified sighting effort and time-varying transience was selected using (corrected) Akaike’s Information Criterion (AICc). The survival probability of non-transient animals was 0.97 (95% CI 0.91-0.98), which is comparable with other studies. The rate of transience increased gradually from 2011 to 2018, before a large drop in 2019. Abundance varied from 786 individuals (95% CI 593-964) in 2016 to 1,434 (95% CI 924-1,908) in 2020, with a non-significant linear increase across the period and interannual fluctuations. These abundance estimates confirm the importance of Bermuda for migrating North Atlantic humpback whales and should encourage a review of cetacean conservation measures in Bermudian waters, including area-based management tools. Moreover, in line with the time series presented here, regional abundance estimates should be updated across the North Atlantic to facilitate population monitoring over the entire migratory range.

Apparent survival probability and abundance of juvenile green turtles in the foraging ground at Kuroshima Island, Ryukyu Archipelago

An understanding of the survival probability and abundance of endangered species is critical for their effective conservation and management. In this study, long-term capture-mark-recapture data were used to estimate the apparent survival probability and abundance of green turtles Chelonia mydas at Kuroshima Island in the Yaeyama Islands, Japan. A total of 453 turtles were captured from 2009 to 2020 using entangling nets. All captured turtles were juveniles and subadults; no adult turtles were captured during this study. The estimated apparent survival rate of this aggregation was 0.82 (95% CI = 0.76-0.86) using the Cormack-Jolly-Seber model, which accounts for both permanent emigration and death. Annual abundance was estimated using the Horvitz-Thompson estimator. Estimated abundance increased from 234 (95% CI = 182-286) in 2009 to 418 (95% CI = 325-510) in 2020. This trend in abundance for green turtles in the foraging ground may result from the increasing nesting population in this region. Our study provides information for understanding the life cycle of green turtles in the Yaeyama Islands and for the conservation and management of marine ecosystems in this region.

Rate-optimal cluster-randomized designs for spatial interference

We consider a potential outcomes model in which interference may be present between any two units but the extent of interference diminishes with spatial distance. The causal estimand is the global average treatment effect, which compares outcomes under the counterfactuals that all or no units are treated. We study a class of designs in which space is partitioned into clusters that are randomized into treatment and control. For each design, we estimate the treatment effect using a Horvitz–Thompson estimator that compares the average outcomes of units with all or no neighbors treated, where the neighborhood radius is of the same order as the cluster size dictated by the design. We derive the estimator’s rate of convergence as a function of the design and degree of interference and use this to obtain estimator-design pairs that achieve near-optimal rates of convergence under relatively minimal assumptions on interference. We prove that the estimators are asymptotically normal and provide a variance estimator. For practical implementation of the designs, we suggest partitioning space using clustering algorithms.

Weighting estimation under bipartite incidence graph sampling

Bipartite incidence graph sampling provides a unified representation of many sampling situations for the purpose of estimation, including the existing unconventional sampling methods, such as indirect, network or adaptive cluster sampling, which are not originally described as graph problems. We develop a large class of design-based linear estimators, defined for the sample edges and subjected to a general condition of design unbiasedness. The class contains as special cases the classic Horvitz-Thompson estimator, as well as the other unbiased estimators in the literature of unconventional sampling, which can be traced back to Birnbaum et al. (1965). Our generalisation allows one to devise other unbiased estimators in future, thereby providing a potential of efficiency gains. Illustrations are given for adaptive cluster sampling, line-intercept sampling and simulated graphs.

Variation in smoking attributable all-cause mortality across municipalities in Belgium, 2018: application of a Bayesian approach for small area estimations

BackgroundSmoking is one of the leading causes of preventable mortality and morbidity worldwide, with the European Region having the highest prevalence of tobacco smoking among adults compared to other WHO regions. The Belgian Health Interview Survey (BHIS) provides a reliable source of national and regional estimates of smoking prevalence; however, currently there are no estimates at a smaller geographical resolution such as the municipality scale in Belgium. This hinders the estimation of the spatial distribution of smoking attributable mortality at small geographical scale (i.e., number of deaths that can be attributed to tobacco). The objective of this study was to obtain estimates of smoking prevalence in each Belgian municipality using BHIS and calculate smoking attributable mortality at municipality level.MethodsData of participants aged 15 + on smoking behavior, age, gender, educational level and municipality of residence were obtained from the BHIS 2018. A Bayesian hierarchical Besag-York-Mollie (BYM) model was used to model the logit transformation of the design-based Horvitz-Thompson direct prevalence estimates. Municipality-level variables obtained from Statbel, the Belgian statistical office, were used as auxiliary variables in the model. Model parameters were estimated using Integrated Nested Laplace Approximation (INLA). Deviance Information Criterion (DIC) and Conditional Predictive Ordinate (CPO) were computed to assess model fit.Population attributable fractions (PAF) were computed using the estimated prevalence of smoking in each of the 589 Belgian municipalities and relative risks obtained from published meta-analyses. Smoking attributable mortality was calculated by multiplying PAF with age-gender standardized and stratified number of deaths in each municipality.ResultsBHIS 2018 data included 7,829 respondents from 154 municipalities. Smoothed estimates for current smoking ranged between 11% [Credible Interval 3;23] and 27% [21;34] per municipality, and for former smoking between 4% [0;14] and 34% [21;47]. Estimates of smoking attributable mortality constituted between 10% [7;15] and 47% [34;59] of total number of deaths per municipality.ConclusionsWithin-country variation in smoking and smoking attributable mortality was observed. Computed estimates should inform local public health prevention campaigns as well as contribute to explaining the regional differences in mortality.

On Variance Estimation for the Population Size Estimator under One-Inflated Positive Poisson Distribution

Let NO1PPbe the Horvitz-Thompson estimator for the population size with one-inflated positive Poisson distribution as the underlying distribution. We estimate the variance of this estimator using conditional expectation technique and provide some descriptions on the variance and its associated confidence interval based on simulation study and real data applications.