Multilevel Regression And Poststratification Research Articles

BARP: Improving Mister P Using Bayesian Additive Regression Trees

Multilevel regression and post-stratification (MRP) is the current gold standard for extrapolating opinion data from nationally representative surveys to smaller geographic units. However, innovations in nonparametric regularization methods can further improve the researcher’s ability to extrapolate opinion data to a geographic unit of interest. I test an ensemble of regularization algorithms and find that there is room for substantial improvement on the multilevel model via more flexible methods of regularization. I propose a modified version of MRP that replaces the multilevel model with a nonparametric approach called Bayesian additive regression trees (BART or, when combined with post-stratification, BARP). I compare both methods across a number of data contexts, demonstrating the benefits of applying more powerful regularization methods to extrapolate opinion data to target geographical units. I provide an R package that implements the BARP method.

Multilevel regression and poststratification as a modeling approach for estimating population quantities in large population health studies: A simulation study.

There are now a growing number of applications of multilevel regression and poststratification (MRP) in population health and epidemiological studies. MRP uses multilevel regression to model individual survey responses as a function of demographic and geographic covariates. Estimated mean outcome values for each demographic-geographic respondent subtype are then weighted by the proportions of each subtype in the population to produce an overall population-level estimate. We recently reported an extensive case study of a large nationwide survey and found that MRP performed favorably compared to conventional survey sampling weights for the estimation of population descriptive quantities in a highly selected sample. In this study, we aimed to evaluate, by way of a simulation experiment, both the accuracy and precision of MRP versus survey sampling weights in the context of large population health studies. While much of the research into MRP has been focused on U.S. political and social science, we considered an alternative population structure of smaller size and with notably fewer geographic subsets. We explored the impact on MRP performance of sample size, model misspecification, interactions, and the addition of a geographic-level covariate. MRP was found to achieve generally superior performance in both accuracy and precision at both the national and state levels. Results were generally robust to model misspecification, and MRP performance was further improved by the inclusion of a geographic-level covariate. These findings offer further evidence that MRP provides a promising analytic approach for addressing participation bias in the estimation of population descriptive quantities from large-scale health surveys and cohort studies.

Differential Dynamic Responsiveness in the U.S. Congress: Evidence from NAFTA

Do incumbent politicians adapt their policy positions in response to changes in public opinion? Existing studies of dynamic responsiveness cannot account for changes in the legislative agenda in Congress over time. We exploit an original dataset on the positions of members of Congress on the North American Free Trade Agreement (NAFTA) at various points leading up to the November 1993 roll-call vote and generate original estimates of constituency public opinion using multi-level regression and post-stratification (MRP). We track whether legislator positioning responds to changes in constituency opinion. We find no evidence of responsiveness to shifting public opinion on NAFTA. Our findings suggest a deficit of dynamic responsiveness in the United States Congress.

Does Responsiveness Stop at the Water's Edge? Examining the Foreign Policy Electoral Connection

Are legislators responsive to public opinion on foreign policy issues? Despite the prevalence of preference-based approaches to international relations, no existing study has systematically assessed the relationship between constituency preferences and foreign policy roll-call votes in the U.S. House. Using multi-level regression and post-stratification (MRP), we create original estimates of constituency preferences, broken down by party, on various trade, security, and immigration bills and measure the responsiveness of members of the U.S. House of Representatives. We find that members are responsive to their median constituent on all issue areas, but there is less responsiveness on trade bills than on security and immigration. We also find that members of Congress are more responsive to co-partisan constituents than to the median on security bills--the reverse holds for trade bills. Additional analyses suggest that cases of greater responsiveness to co-partisans do not result from voters taking partisan cues from their representatives.

Modeling Geographic Variation in Household Disaster Preparedness across U.S. States and Metropolitan Areas

Emerging statistical techniques, combined with the increasing accessibility of primary social survey data, can provide policy-relevant tools for understanding how perceptions and behaviors vary geographically. Planning for natural disasters requires local data, yet data on topics such as household preparedness behaviors are often unavailable at the appropriate spatial resolution. This article presents new nationwide estimates of one element of household preparedness—having supplies in the home to use in case of a disaster—across all states and metropolitan areas in the United States. Estimates are based on a 2015 national survey combined with multilevel regression and poststratification (MRP), a statistical technique to develop subnational estimates from national data sets. The model uses sociodemographic and geographic predictors informed by prior research. Estimates were externally validated against independent surveys, including data from the 2013 American Housing Survey. Comparing the estimates against historical disaster losses demonstrates broad variation in preparedness even among places with historically high rates of death and injury from natural disasters and allows identification of high-risk places with high disaster losses and low preparedness according to this survey item. Leveraging large survey data sets in combination with MRP can be an effective tool for researchers and decision makers to understand geographic variation in perceptions and behaviors at subnational scales.

Evaluating Policy Representation with Dynamic MRP Estimates: Direct Democracy and Same-Sex Relationship Policies in the United States

AbstractDoes direct democracy strengthen popular control of public policy in the United States? A major challenge in evaluating policy representation is the measurement of state-level public opinion and public policy. Although recent studies of policy responsiveness and congruence have provided improved measures of public opinion using multilevel regression and poststratification (MRP) techniques, these analyses are limited by their static nature and cross-sectional design. Issue attitudes, unlike more general political orientations, often vary considerably over time. Unless the dynamics of issue-specific public opinion are appropriately incorporated into the analyses, tests of policy responsiveness and congruence may be misleading. Thus, we assess the degree of policy representation in direct democracy states regarding same-sex relationship recognition policies using dynamic models of policy adoption and congruence that employ dynamic MRP estimates of attitudes toward same-sex marriage. We find that direct democracy institutions increase both policy responsiveness and congruence with issue-specific public opinion.

The spatial distribution of Republican and Democratic climate opinions at state and local scales

Even as US partisan polarization shapes climate and energy attitudes, substantial heterogeneity in climate opinions still exists among both Republicans and Democrats. To date, our understanding of this partisan heterogeneity has been limited to analysis of national- or, less commonly, state-level opinion poll subsamples. However, the dynamics of political representation and issue commitments play out over more finely resolved state and local scales. Here we use previously validated multilevel regression and post-stratification (MRP) models (Howe et al., Nat Clim Chang 5(6):596–603 2015; Mildenberger et al., PLoS One 11(8):e0159774 2016) combined with a novel approach to measuring the distribution of party members to model, for the first time, the spatial distribution of partisan climate and energy opinions. We find substantial geographic variation in Republican climate opinions across states and congressional districts. While Democratic party members consistently think human-caused global warming is happening and support climate policy reforms, the intensity of their climate beliefs also varies spatially at state and local scales. These results have policy-relevant implications for the trajectory of US climate policy reforms.

Extending the Use and Prediction Precision of Subnational Public Opinion Estimation

AbstractThe comparative study of subnational units is on the rise. Multilevel regression and poststratification (MrP) has become the standard method for estimating subnational public opinion. Unfortunately, MrP comes with stringent data demands. As a consequence, scholars cannot apply MrP in countries without detailed census data, and when such data are available, the modeling is restricted to a few variables. This article introduces multilevel regression with synthetic poststratification (MrsP), which relaxes the data requirement of MrP to marginal distributions, substantially increases the prediction precision of the method, and extends its use to countries without census data. The findings of Monte Carlo, U.S., and Swiss analyses show that, using the same predictors, MrsP usually performs in standard applications as well as the currently used standard approach, and it is superior when additional predictors are modeled. The better performance and the more straightforward implementation promise that MrsP will further stimulate subnational research.

Underestimation of Inequality in Japan

This paper asks who within a polity underestimates inequality and what consequences does this underestimation have for politics. Previous studies show that people misperceive overall inequality or where their own income is in the income distribution. In Japan, there are more people who underestimate inequality than people who overestimate it. Why do Japanese underestimate inequality rather than overestimate it? I argue that people living in a relatively equal area are more likely to underestimate inequality than those living in a relatively unequal area. Furthermore, I show that underestimation changes electoral outcomes, demonstrating this by analyzing Japanese data. To reveal demographic and geographical differences in underestimation, I implement multilevel regression and poststratification (MRP) with individual level data collected by Japanese Electoral Study III and aggregate data at the prefectural level. The statistical analysis reveals three things. First, certain demographic and political factors systematically affect the probability of underestimation at the individual level. Second, the degree of underestimation varies across prefectures in Japan. Third, underestimation of inequality affects electoral outcomes.

Tracking state trends in environmental public opinion

Trends in state-level public opinion on the environment within the U.S. are examined, using data from the General Social Survey (GSS) from 1976 to 2008. Multilevel Regression and Post-Stratification (MRP) approach estimates public support for environmental spending at the U.S. state-level over three decades. This allows for an analysis of inter-state homogeneity of environmental public opinion, over the latter half of the twenty century. The findings indicate state-level trends mirror those at the national-level, but state-level public opinion is becoming more analogous over time.

Geographic Variation in Disaster Preparedness Across U.S. States and Metropolitan Areas

Effective planning for natural disasters requires accurate local data relevant to disaster mitigation efforts. However, data on topics such as individual risk perceptions or household preparedness behaviors are often unavailable at the spatial or temporal resolution necessary for local mitigation efforts. To address these data needs, this paper presents new nationwide estimates of one element of household preparedness — setting aside supplies in the home to use in case of a disaster — across states and metropolitan areas in the United States. Using national online survey data collected in 2015, a multilevel regression and poststratification (MRP) model was fit using demographic, social, and geographic predictors. The resulting model estimates are presented for all U.S. states and all 917 metropolitan and micropolitan areas. Estimates were externally validated against representative surveys at the state and metropolitan level, including data from the 2013 American Housing Survey. Validation indicated that model estimates have low absolute errors and are strongly correlated with external data. Comparing the estimates against historical disaster losses demonstrates broad variation in preparedness even among places with historically high rates of death and injury from natural disasters. These findings suggest that cost-effective online data collection techniques combined with MRP methods can be an effective tool for researchers and decision makers interested in understanding geographic variation in social survey data at subnational scales.

Estimating anti-immigrant sentiment for the American states using multi-level modeling and post-stratification, 2004–2008

An estimate is provided of an innovative state-level measure of anti-immigrant sentiment for use in future policy and behavioral studies. State governments became increasingly active in adopting immigrant policies in the 2000s. Previous research highlights the role of public opinion, especially attitudes toward immigrants, in explaining policy priorities and outcomes. Unfortunately, most extant studies utilize political ideology or immigrant populations as rough proxies for public opinion. In this article, we estimate a reliable and valid measure of anti-immigrant sentiment at the state-level using survey aggregation with multilevel regression and post-stratification (MRP) for the period 2004 to 2008. We compare our estimates of anti-immigrant sentiment to alternative measures of immigrant presence and political ideology in predicting multiple variations of state immigrant policies. Ultimately, we find theoretical and statistical advantages of using anti-immigrant sentiment over previous measures in predicting immigrant policies.

The Distribution of Climate Change Public Opinion in Canada

Historic Canadian elections in 2015, both federally and provincially, created an opportunity for action on climate policy. While climate scientists have developed high resolution data sets on the distribution of Canadian climate risks, we still lack comparable data on the local distribution of public climate change opinions. This paper provides the first effort to estimate local climate and energy opinion variability outside the United States. Using a multi-level regression and post-stratification (MRP) approach, we estimate opinion in federal electoral districts and provinces. We demonstrate that a majority of the Canadian public consistently believes that climate change is happening. Belief in climate change's causes varies geographically, with more attributing it to human activity in urban as opposed to rural areas. Most prominently, we find majority support for carbon cap and trade policy in every province and district. By contrast, support for carbon taxation is more heterogeneous. Compared to the distribution of US climate opinions, Canadians believe climate change is happening at higher levels. This new opinion data set will support climate policy analysis and climate policy decision making at national, provincial and local levels.

Assessing the Validity of Enns and Koch's Measure of State Policy Mood

Enns and Koch (hereafter E&K) use multilevel regression and poststratification (MRP) along with survey aggregation to measure state policy mood. As E&K rely on direct information about public opinion, it would be preferable to Berry et al.‘s widely used indirect measure relying on data about the issue positions and vote shares of members of Congress, if E&K's measure were valid. Assessing the validity of E&K's measure takes on special importance because the measure proves to be nearly uncorrelated with Berry et al.‘s measure, implying that at least one is invalid. Because the “true” policy mood of states is unknown, it is impossible to definitively assess the validity of E&K's measure. Instead, we raise some concerns about E&K's measurement methodology and present evidence pertaining to the indicator's face validity, convergent validity, and construct validity. Our analyses leave us doubtful that the E&K measure is valid because its characterization of state moods departs significantly from conventional wisdom and current scholarship.

The states of public opinion on the environment

ABSTRACTState-level public opinion on the environment within the US is examined, using data from the General Social Survey (GSS) from 2000 to 2010. A Multilevel Regression and Post-Stratification (MRP) approach is taken to estimate three different survey items on environmental issues at the US state level. This allows for a comparable understanding of state-level public opinion on environmental issues, which is then connected to economic, environmental, and political conditions, and representation in policy making. The findings indicate the MRP technique is a sophisticated way to make accurate estimations of state-level public opinion, and those estimates provide further insight into the shaping of public opinion and public policy.

Public Opinion in the U.S. States: 1956 to 2010

AbstractIn this article, we create, validate, and analyze new dynamic measures of state partisanship, state policy mood, and state political ideology. The measures of partisanship and policy mood begin in 1956 and the measure of ideology begins in 1976. Our approach uses the advantages of two leading techniques for measuring state public opinion—multilevel regression and poststratification (MRP) and survey aggregation. The resulting estimates are based on nearly 500 different surveys with a total of more than 740,000 respondents. After validating our measures, we show that during the last half century, policy preferences in the states have shifted in important and sometimes surprising ways. For example, we find that differences in political attitudes across time can be as important as differences across states.

Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Subgroups

Using multilevel regression and poststratification (MRP), we estimate voter turnout and vote choice within deeply interacted subgroups: subsets of the population that are defined by multiple demographic and geographic characteristics. This article lays out the models and statistical procedures we use, along with the steps required to fit the model for the 2004 and 2008 presidential elections. Though MRP is an increasingly popular method, we improve upon it in numerous ways: deeper levels of covariate interaction, allowing for nonlinearity and nonmonotonicity, accounting for unequal inclusion probabilities that are conveyed in survey weights, postestimation adjustments to turnout and voting levels, and informative multidimensional graphical displays as a form of model checking. We use a series of examples to demonstrate the flexibility of our method, including an illustration of turnout and vote choice as subgroups become increasingly detailed, and an analysis of both vote choice changes and turnout changes from 2004 to 2008.

How Does Multilevel Regression and Poststratification Perform with Conventional National Surveys?

Multilevel regression and poststratification (MRP) is a method to estimate public opinion across geographic units from individual-level survey data. If it works with samples the size of typical national surveys, then MRP offers the possibility of analyzing many political phenomena previously believed to be outside the bounds of systematic empirical inquiry. Initial investigations of its performance with conventional national samples produce generally optimistic assessments. This article examines a larger number of cases and a greater range of opinions than in previous studies and finds substantial variation in MRP performance. Through empirical and Monte Carlo analyses, we develop an explanation for this variation. The findings suggest that the conditions necessary for MRP to perform well will not always be met. Thus, we draw a less optimistic conclusion than previous studies do regarding the use of MRP with samples of the size found in typical national surveys.

Testing Policy Responsiveness with Dynamic MRP Estimates: Direct Democracy and Gay Rights in the U.S.

One major challenge in testing whether direct democracy institutions increase governmental responsiveness is the measurement of state-level public opinion and public policy. However, recent developments in using multilevel regression and post-stratification (MRP) techniques to estimate state-level public opinion have allowed for more direct tests (Lax and Phillips 2009a, 2009b). Yet these studies find no evidence that direct democracy increases policy congruence with public preferences, even on typically salient issues like gay rights.In this paper we attempt to address the problem of dynamic public opinion by developing dynamic MRP estimates of attitudes toward homosexuality. We then test whether direct democracy institutions increase policy responsiveness to these attitudes using event history models of the adoption of several gay rights policies. The analyses focus on gay rights policies because this allows us to also examine the negative side of increased policy responsiveness – tyranny of the majority.

How Should We Measure District-Level Public Opinion on Individual Issues?

Due to insufficient sample sizes in national surveys, strikingly little is known about public opinion at the level of Congressional and state legislative districts in the United States. As a result, there has been virtually no study of whether legislators accurately represent the will of their constituents on individual issues. This article solves this problem by developing a multilevel regression and poststratification (MRP) model that combines survey and census data to estimate public opinion at the district level. We show that MRP estimates are excellent predictors of public opinion and referenda results for both congressional and state senate districts. Moreover, they have less error, higher correlations, and lower variance than either disaggregated survey estimates or presidential vote shares. The MRP approach provides American and Comparative Politics scholars with a valuable new tool to measure issue-specific public opinion at low levels of geographic aggregation.