Multilevel Regression And Poststratification Research Articles

Using leave-one-out cross validation (LOO) in a multilevel regression and poststratification (MRP) workflow: A cautionary tale.

In recent decades, multilevel regression and poststratification (MRP) has surged in popularity for population inference. However, the validity of the estimates can depend on details of the model, and there is currently little research on validation. We explore how leave-one-out cross validation (LOO) can be used to compare Bayesian models for MRP. We investigate two approximate calculations of LOO: Pareto smoothed importance sampling (PSIS-LOO) and a survey-weighted alternative (WTD-PSIS-LOO). Using two simulation designs, we examine how accurately these two criteria recover the correct ordering of model goodness at predicting population and small-area estimands. Focusing first on variable selection, we find that neither PSIS-LOO nor WTD-PSIS-LOO correctly recovers the models' order for an MRP population estimand, although both criteria correctly identify the best and worst model. When considering small-area estimation, the best model differs for different small areas, highlighting the complexity of MRP validation. When considering different priors, the models' order seems slightly better at smaller-area levels. These findings suggest that, while not terrible, PSIS-LOO-based ranking techniques may not be suitable to evaluate MRP as a method. We suggest this is due to the aggregation stage of MRP, where individual-level prediction errors average out. We validate these results by applying to the real world National Health and Nutrition Examination Survey (NHANES) data in the United States. Altogether, these results show that PSIS-LOO-based model validation tools need to be used with caution and might not convey the full story when validating MRP as a method.

Embedded multilevel regression and poststratification: Model-based inference with incomplete auxiliary information.

Health disparity research often evaluates health outcomes across demographic subgroups. Multilevel regression and poststratification (MRP) is a popular approach for small subgroup estimation as it can stabilize estimates by fitting multilevel models and adjust for selection bias by poststratifying on auxiliary variables, which are population characteristics predictive of the analytic outcome. However, the granularity and quality of the estimates produced by MRP are limited by the availability of the auxiliary variables' joint distribution; data analysts often only have access to the marginal distributions. To overcome this limitation, we embed the estimation of population cell counts needed for poststratification into the MRP workflow: embedded MRP (EMRP). Under EMRP, we generate synthetic populations of the auxiliary variables before implementing MRP. All sources of estimation uncertainty are propagated with a fully Bayesian framework. Through simulation studies, we compare different methods of generating the synthetic populations and demonstrate EMRP's improvements over alternatives on the bias-variance tradeoff to yield valid subpopulation inferences of interest. We apply EMRP to the Longitudinal Survey of Wellbeing and estimate food insecurity prevalence among vulnerable groups in New York City. We find that all EMRP estimators can correct for the bias in classical MRP while maintaining lower standard errors and narrower confidence intervals than directly imputing with the weighted finite population Bayesian bootstrap (WFPBB) and design-based estimates. Performances from the EMRP estimators do not differ substantially from each other, though we would generally recommend using the WFPBB-MRP for its consistently high coverage rates.

Abstract 4257: Using Multilevel Regression and Poststratification (MRP) to inform estimates of biomarker prevalence for target populations of interest using linked clinico-genomic databases

Abstract Background Contemporary evidence on the size of biomarker-defined cancer populations is crucial for clinical and economic healthcare decision makers. Estimates can be derived from real-world data (RWD), but may be affected by biases stemming from differences in observed populations vs. general populations of patients under the purview of decision makers. These differences may be driven by patient or clinical characteristics, as well as differences in assays used to assess biomarker status. This study demonstrates the applicability of an emerging method in survey science (Multilevel Regression and Poststratification (MRP)), which leverages common covariates across source data and larger, more representative set of observations to inform estimates of biomarker prevalence for target populations of interest. Methods In disease-agnostic cohorts of patients with solid-tumor cancers from two, large clinico-genomic cancer databases (AACR GENIE 12.0, and nationwide (US-based) de-identified Flatiron Health - Foundation Medicine Clinico-Genomic Database(CGDB)), we used covariate-adjusted regression (age, sex, race, stage at diagnosis (for CGDB patients), and cancer type) to predict the probability of a patient having an NTRK fusion and/or a BRAF V600E mutation detected in their sequenced cancer tissue. AACR Project GENIE (n = 118,649) is a cancer registry of real-world data assembled through data sharing between 19 international cancer centers. The de-identified data in the CGDB (n = 91,719) originated from approximately 280 US cancer clinics (~800 sites of care).We then implemented post-stratification adjustments (Gelman, 2006; Downes 2018) using the Surveillance, Epidemiology, and End Results (SEER) 17 2022 cancer registry to estimate prevalence of the alterations in the large, population-based registry. Results Prevalence estimates were generally consistent between sources. Adjusted prevalence (%) estimates for NTRK Fusion (AACR: 0.25% (raw), adjusted to 0.21% (0.01%-0.48%) (MRP-adjusted mean % (95% quantile intervals)); CGDB: 0.29, adjusted to 0.26 (0.02-0.63), and for BRAF V600E (AACR: 2.19, adjusted to 1.72 (0.77-2.66); CGDB: 2.44, adjusted to 2.68 (1.50-3.89)) reflected differential patient characteristics (predominantly tumor type) in these sources compared to SEER. MRP also enabled creation, from posterior distributions, of tumor-level estimates, along with credible interval bounds. Conclusion While both clinico-genomic databases offered generally consistent estimates of biomarker prevalence, representativeness methods that are used in the survey literature may be suited for addressing biases in RWD sources to improve biomarker prevalence estimates of target populations of patients with cancer. Citation Format: Jeremy W. Snider, Brennan Beal, Lilia Bouzit, Cheryl Cho-Phan, Leah Comment, Garrett Frampton, Tamara Snow. Using Multilevel Regression and Poststratification (MRP) to inform estimates of biomarker prevalence for target populations of interest using linked clinico-genomic databases. [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 4257.

Re-Evaluating Machine Learning for MRP Given the Comparable Performance of (Deep) Hierarchical Models

Multilevel regression and post-stratification (MRP) is a popular use of hierarchical models in political science. Multiple papers have suggested that relying on machine learning methods can provide substantially better performance than traditional approaches that use hierarchical models. However, these comparisons are often unfair to traditional techniques as they omit possibly important interactions or nonlinear effects. I show that complex (“deep”) hierarchical models that include interactions can nearly match or outperform state-of-the-art machine learning methods. Combining multiple models into an ensemble can improve performance, although deep hierarchical models are themselves given considerable weight in these ensembles. The main limitation to using deep hierarchical models is speed. This paper derives new techniques to further accelerate estimation using variational approximations. I provide software that uses weakly informative priors and can estimate nonlinear effects using splines. This allows flexible and complex hierarchical models to be fit as quickly as many comparable machine learning techniques.

Geographic and demographic variation in worry about extreme heat and COVID-19 risk in summer 2020

Extreme heat is a major health hazard that is exacerbated by ongoing human-caused climate change. However, how populations perceive the risks of heat in the context of other hazards like COVID-19, and how perceptions vary geographically, are not well understood. Here we present spatially explicit estimates of worry among the U.S. public about the risks of heat and COVID-19 during the summer of 2020, using nationally representative survey data and a multilevel regression and poststratification (MRP) model. Worry about extreme heat and COVID-19 varies both across states and across demographic groups, in ways that reflect disparities in the impact of each risk. Black or African American and Hispanic or Latino populations, who face greater health impacts from both COVID-19 and extreme heat due to institutional and societal inequalities, also tend to be much more worried about both risks than white, non-Hispanic populations. Worry about heat and COVID-19 were correlated at the individual and population level, and patterns tended to be related to underlying external factors associated with the risk environment. In the face of a changing climate there is an urgent need to address disparities in heat risk and develop responses that ensure the most at-risk populations are protected.

Model-based community health surveillance via multilevel small area estimation using state behavioral risk factor surveillance system (BRFSS): a case study in Connecticut

PurposeThe main objective of state behavioral risk factor surveillance system (BRFSS) is to produce reliable state-level estimates of various population health outcomes. A multilevel Regression and Post-stratification (MRP) methodology for small area estimation has been applied to the 500 Cities Project to provide population estimates at both city-level and census tract-level using national BRFSS data. To date, MRP has not been applied to any state BRFSS to produce health data at local geographic areas. In addition, the use of single year BRFSS might produce temporary inconsistency in small area estimates (SAEs). The predicted standard errors (SEs) and confidence intervals (CIs) of SAEs using Monte Carlo simulation could be substantially underestimated or overestimated. MethodsBy extending the current MRP approach and applying a parametric bootstrapping approach to Connecticut BRFSS (CT BRFSS), we were able to produce SAEs as well as SEs and CIs of SAEs for Connecticut counties and towns. We also applied this model to 5-year CT BRFSS (2011–2015) with an aim to improve the temporary consistency of SAEs. ResultsBoth single-year and 5-year estimates with SEs and CIs were generated for six selected population health indicators at town, county and state levels. Model-based SAEs were internally evaluated by comparing to single-year and 5-year direct BRFSS survey (2011–2015). SAEs were also externally validated when external data were available. ConclusionsModel-based SAEs are valid and could be used to characterize local geographic variations using single state BRFSS data.

Race, Locality, and Representative Bureaucracy: Does Community Bias Matter?

Abstract Despite burgeoning research on representative bureaucracy theory, there is limited examination of how environmental contexts shape the manner in which the demographic makeup of a bureaucracy is linked to distributional bureaucratic outcomes. Scholars in the field of social psychology, however, have suggested that community-level variation in the pervasiveness of biases against particular social groups helps to explain inequitable outcomes in such diverse settings as education, policing, and health care. Incorporating social psychology research into representative bureaucracy theory, this article examines how community racial biases shape the association between the demographic makeup of an organization’s personnel and its bureaucratic outcomes. Using county-level implicit and explicit bias measures that are estimated by multilevel regression and poststratification (MRP) based on a dataset containing more than 1.2 million respondents, we find that more inequitable educational outcomes occur for black students in counties where white residents hold stronger anti-black biases. Our findings also suggest that while black teachers are associated with more favorable outcomes for black students in the zero-sum context of assignment to gifted classes, the association of outcomes with passive representation is more limited in counties with strong racial biases. By accounting for the racial biases exhibited in the communities where both clients and bureaucrats are socially and culturally embedded, this article extends our understanding of how contextual factors shape the nature of bureaucratic representation.

Fast and Accurate Estimation of Non-Nested Binomial Hierarchical Models Using Variational Inference

Non-linear hierarchical models are commonly used in many disciplines. However, inference in the presence of non-nested effects and on large datasets is challenging and computationally burdensome. This paper provides two contributions to scalable and accurate inference. First, I derive a new mean-field variational algorithm for estimating binomial logistic hierarchical models with an arbitrary number of non-nested random effects. Second, I propose “marginally augmented variational Bayes” (MAVB) that further improves the initial approximation through a step of Bayesian post-processing. I prove that MAVB provides a guaranteed improvement in the approximation quality at low computational cost and induces dependencies that were assumed away by the initial factorization assumptions. I apply these techniques to a study of voter behavior using a high-dimensional application of the popular approach of multilevel regression and post-stratification (MRP). Existing estimation took hours whereas the algorithms proposed run in minutes. The posterior means are well-recovered even under strong factorization assumptions. Applying MAVB further improves the approximation by partially correcting the under-estimated variance. The proposed methodology is implemented in an open source software package.

Know your population and know your model: Using model-based regression and poststratification to generalize findings beyond the observed sample.

Psychology research often focuses on interactions, and this has deep implications for inference from nonrepresentative samples. For the goal of estimating average treatment effects, we propose to fit a model allowing treatment to interact with background variables and then average over the distribution of these variables in the population. This can be seen as an extension of multilevel regression and poststratification (MRP), a method used in political science and other areas of survey research, where researchers wish to generalize from a sparse and possibly nonrepresentative sample to the general population. In this article, we discuss areas where this method can be used in the psychological sciences. We use our method to estimate the norming distribution for the Big Five Personality Scale using open source data. We argue that large open data sources like this and other collaborative data sources can potentially be combined with MRP to help resolve current challenges of generalizability and replication in psychology. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Is It Us? Is It Them? Or Is It This Place? Predicting Civility in State Legislatures

AbstractWhile many scholars and analysts have observed a decline in civility in recent years, there have been few examinations of how political, economic, and institutional structures may partially explain inter-state differences in these trends. We suggest three potential explanations: (1) institutional structures, such as legislative professionalism and gubernatorial power, have created different contexts in which legislators build and maintain inter-personal relationships; (2) partisan competition has led to less bipartisan cooperation and contributed to strained relationships between members of different parties; and, (3) economic inequity and change has contributed to economic anxiety among citizens, contributing to conflict in legislative bodies as elected officials attempt to navigate emerging policy challenges. To test these explanations, we develop an innovative measure of civility using a national survey of lobbyists and a partial Multilevel Regression and Poststratification (MRP) design. Findings suggest that there is some validity to all three explanations, and signifying that civility is at least partially a result of structural issues.

Using Multilevel Regression and Poststratification to Estimate Physical Activity Levels from Health Surveys.

Background: Large-scale health surveys often consider sociodemographic characteristics and several health indicators influencing physical activity that often vary across subpopulations. Data in a survey for some small subpopulations are often not representative of the larger population. Objective: We developed a multilevel regression and poststratification (MRP) model to estimate leisure-time physical activity across Brazilian state capitals and evaluated whether the MRP outperforms single-level regression estimates based on the Brazilian cross-sectional national survey VIGITEL (2018). Methods: We used various approaches to compare the MRP and single-level model (complete-pooling) estimates, including cross-validation with various subsample proportions tested. Results: MRP consistently had predictions closer to the estimation target than single-level regression estimations. The mean absolute errors were smaller for the MRP estimates than single-level regression estimates with smaller sample sizes. MRP presented substantially smaller uncertainty estimates compared to single-level regression estimates. Overall, the MRP was superior to single-level regression estimates, particularly with smaller sample sizes, yielding smaller errors and more accurate estimates. Conclusion: The MRP is a promising strategy to predict subpopulations’ physical activity indicators from large surveys. The observations present in this study highlight the need for further research, which could, potentially, incorporate more information in the models to better interpret interactions and types of activities across target populations.

Economic Preferences and the Local Coronavirus Pandemic

Early in the pandemic, individuals in numerous countries experienced quite different rates of COVID-19 infections and deaths dependent on where they lived. This within-country variation offers an opportunity to study how the intensity of a catastrophic shock to systems affects individuals’ economic preferences – a topic without consensus in the literature. In April 2020, we conducted an online survey with approximately 1500 subjects in China, 800 in Chile, and 800 in Italy. Our sampling strategy deliberately sampled subjects with exposure to different levels of local COVID-19 infections. We find that respondents condition their behavior and economic preferences on this intensity – levels of COVID-19 preventive behavior are correlated with the intensity of community infections; exposure to intense infection rates correlates, positively, with risk aversion and patience; and, negatively, with other-regarding preferences. Using machine-learning to estimate individual-level effects, we find notable effect heterogeneity with respect to education levels. Finally, using multilevel regression and poststratification (MRP) we demonstrate province-level estimates of economic preferences for 107 Italian provinces.

Multilevel regression and poststratification for estimating population quantities from large health studies: a simulation study based on US population structure

IntroductionRecruiting a representative sample of participants is becoming increasingly difficult in large-scale health surveys. Multilevel regression and poststratification (MRP) has been shown to be effective in estimating population descriptive quantities...

The big, the bad, and the ugly: Geographic estimation with flawed psychological data.

The geographic distribution of psychological constructs has long been an area of focus for psychological researchers. Recently, however, there has been increased interest in investigations of the so-called subnational distribution of psychological variables, which focus on localized groupings of individuals within spatial units, such as counties or states. By estimating the subnational distribution of a given outcome (e.g., estimating its state- or county-level means), researchers have been able to address questions about the spatial variation of a variety of psychological constructs and investigate the regional association between psychological phenomena and real-world outcomes, such as health outcomes, prosocial behavior, and racial inequity. Unfortunately, however, there are many challenges to estimating a construct's subnational distribution, such as those raised by response biases and subnational sparsity. To help psychological researchers address these issues, we provide a comprehensive discussion of subnational estimation and introduce multilevel regression and poststratification (MrP), a method that is widely considered to be the gold standard for subnational estimation with random samples. As psychologists often do not have access to large, national random samples, we also report 3 studies evaluating MrP's performance under simulated and real-world conditions of sample biases. Ultimately, we find that MrP is likely to outperform the subnational estimation methods that psychological researchers currently use. Based on this, we suggest that psychologists interested in understanding how psychological phenomena vary below the nation level use MrP to conduct these investigations. To help facilitate this, we have made all code and data used for the reported studies publicly available. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Improving multilevel regression and poststratification with structured priors.

A central theme in the field of survey statistics is estimating population-level quantities through data coming from potentially non-representative samples of the population. Multilevel regression and poststratification (MRP), a model-based approach, is gaining traction against the traditional weighted approach for survey estimates. MRP estimates are susceptible to bias if there is an underlying structure that the methodology does not capture. This work aims to provide a new framework for specifying structured prior distributions that lead to bias reduction in MRP estimates. We use simulation studies to explore the benefit of these prior distributions and demonstrate their efficacy on non-representative US survey data. We show that structured prior distributions offer absolute bias reduction and variance reduction for posterior MRP estimates in a large variety of data regimes.

A framework for improving electoral forecasting based on time-aware polling

The prediction of opinion distribution in real-world scenarios represents a major scientific challenge for current social networks analysis. In terms of electoral forecasting, we find several prediction solutions that try to combine statistics with economic indices, and machine learning, like multilevel regression and post-stratification (MRP). Nevertheless, recent studies pinpoint toward the importance of temporal characteristics in the diffusion of opinion. As such, we take inspiration from micro-scale temporal epidemic models and develop an original time-aware (TA) forecasting methodology which is able to improve the prediction of opinion distribution in an electoral context. After a parametric analysis, we validate our TA method with pre-election survey data from three presidential elections (2012–2019) and the UK Brexit (2016). Benchmarking our TA method against two classic statistical approaches, like survey averaging (SA), and cumulative vote counting (CC), and the best pollster predictions, we find that our method is substantially closer to the real election outcomes. On average, we measure prediction errors of 9.8% (SA), 9.6% (CC), 5.1% (MRP), and only 3.0% for TA; these differences translate into increases of prediction accuracy of $$\approx 71\%$$ for the TA method (40% better than best pollsters). Moreover, TA does not require socio-economical contextual information, while the more complex MRP method depends on them for prediction. This work builds upon existing studies on the microscopic temporal dynamics of social networks and offers new insight on how macroscopic prediction can be improved using time-awareness.

Multilevel Regression and Poststratification Versus Survey Sample Weighting for Estimating Population Quantities in Large Population Health Studies.

Multilevel regression and poststratification (MRP) is a model-based approach for estimating a population parameter of interest, generally from large-scale surveys. It has been shown to be effective in highly selected samples, which is particularly relevant to investigators of large-scale population health and epidemiologic surveys facing increasing difficulties in recruiting representative samples of participants. We aimed to further examine the accuracy and precision of MRP in a context where census data provided reasonable proxies for true population quantities of interest. We considered 2 outcomes from the baseline wave of the Ten to Men study (Australia, 2013-2014) and obtained relevant population data from the 2011 Australian Census. MRP was found to achieve generally superior performance relative to conventional survey weighting methods for the population as a whole and for population subsets of varying sizes. MRP resulted in less variability among estimates across population subsets relative to sample weighting, and there was some evidence of small gains in precision when using MRP, particularly for smaller population subsets. These findings offer further support for MRP as a promising analytical approach for addressing participation bias in the estimation of population descriptive quantities from large-scale health surveys and cohort studies.

Splitting the Tactical Vote? Coordination Problems with Polling Model‐Driven Tactical Voting Websites

AbstractThe 2019 general election gave prominence to tactical voting advice websites, particularly those projecting constituency‐level results using national polling data and sophisticated multi‐level regression and poststratification (MRP) polling models. We see tactical voting as an example of a coordination game and argue that polling‐model driven advice websites are disruptive to the existing focal points for strategic voting. Although such websites may well help identify the strongest candidate based on constituency demographics, the article argues they are unlikely to coalesce support behind that single candidate, a prerequisite for successful tactical voting. In practice, there are multiple competing models and websites as well as other key sources of information to voters. Furthermore, the heuristic of relying on tactical voting websites is vulnerable to strategic action by partisans. In short, it is argued that the emergence of these websites risks muddying the waters further for electors seeking to vote tactically, potentially splitting the tactical vote rather than unifying it.

Stacked Regression and Poststratification

I develop a procedure for estimating local-area public opinion called stacked regression and poststratification (SRP), a generalization of classical multilevel regression and poststratification (MRP). This procedure employs a diverse ensemble of predictive models—including multilevel regression, LASSO, k-nearest neighbors, random forest, and gradient boosting—to improve the cross-validated fit of the first-stage predictions. In a Monte Carlo simulation, SRP significantly outperforms MRP when there are deep interactions in the data generating process, without requiring the researcher to specify a complex parametric model in advance. In an empirical application, I show that SRP produces superior local public opinion estimates on a broad range of issue areas, particularly when trained on large datasets.

Model-based pre-election polling for national and sub-national outcomes in the US and UK

We describe a strategy for applying multilevel regression and post-stratification (MRP) methods to pre-election polling. Using a combination of contemporaneous polling, census data, past election polling, past election results, and other sources of information, we are able to construct probabilistic, internally consistent estimates of national votes and the sub-national electoral districts that determine seats or electoral votes in many electoral systems. We report on the performance of the general framework in three applications that were conducted and released publicly in advance of the 2016 UK Referendum on EU Membership, the 2016 US Presidential Election, and the 2017 UK General Election.