Iterative Proportional Fitting Algorithm Research Articles

Adjustment of initial estimates of voter transition probabilities to guarantee consistency and completeness

The estimation of voter shifts (stayers and switchers) between elections is an active area of research that, for decades, has attracted the interest of many scholars. The voter transitions are typically summarised in a row-standardised proportion (probability) matrix. This matrix is usually unknown, despite it being of interest to many agents, including party teams, the media and political scientists. When surveys are used to approximate this matrix, it is not uncommon for the estimated matrix to be inconsistent and even incomplete. The iterative proportional fitting algorithm solves inconsistency but cannot fix incompleteness. Hierarchical Bayesian models that combine aggregate and survey estimates can solve both problems, but are extremely complex and data-demanding. This paper details all the scenarios concerning the available information that can be reasonably considered and, within the linear programming framework, develops specific models to reach consistency and completeness. The models are, moreover, quite flexible as they allow analysts to have missing values and to introduce through weights their relative confidences in the different a priori transition proportions. The usefulness of the proposed models is illustrated with real data. Interested readers can easily use these new models with their data as they have been programmed in the function lp_apriori of the R-package lphom.

Incidence, Etiology, and Healthcare Utilization for Acute Gastroenteritis in the Community, United States.

Knowledge of the epidemiology of sporadic acute gastroenteritis (AGE) in the United States is limited. During September 2016-September 2017, we surveyed Kaiser Permanente Northwest members in Oregon and Washington, USA, to collect data on the 30-day prevalence of dually defined AGE and diarrhea disease and related health-seeking behavior; from a subset of participants, we obtained a stool specimen. Using the iterative proportional fitting algorithm with raked weights, we generated AGE prevalence and annualized rate estimates. We detected norovirus, rotavirus, astrovirus, and sapovirus from submitted stool specimens through real-time quantitative reverse transcription PCR (qRT-PCR). We estimated a 30-day prevalence of 10.4% for AGE and 7.6% for diarrhea only; annual rates were 1.27 cases/person/year for AGE and 0.92 cases/person/year for diarrhea only. Of those with AGE, 19% sought medical care. Almost one quarter (22.4%) of stool specimens from those reporting AGE tested positive for ≥1 viral pathogen, compared with 8.2% from those without AGE.

High Engagement of Patients Monitored by a Digital Health Ecosystem Indicates Significant Improvements of Key r-hGH Treatment Metrics.

The early adoption of digital health solutions in the treatment of growth disorders has enabled the collection and analysis of more than 10 years of real-world data using the easypod™ connect platform. Using this rich dataset, we were able to study the impact of engagement on three key treatment-related outcomes: adherence, persistence of use, and growth. In total, data for 17,906 patients were available. The three features, regularity of injection (≤2h vs >2h), change of comfort setting (yes/no), and opting-in to receive injection reminders (yes/no), were used as a proxy for engagement. Patients were assigned to the low-engagement group (n=1,752) when all of their features had the low-engagement flag (>2h, no, no) and to the high-engagement group (n=1,081) when all of their features had the high-engagement flag (≤2h, yes, yes). The low-engagement group was down-sampled to 1,081 patients (subsample of n=37 for growth) using the iterative proportional fitting algorithm. Statistical tests were used to study the impact of engagement to the outcomes. The results show that all three outcomes were significantly improved by a factor varying from 1.8 up to 2.2 when the engagement level was high. These results should encourage the promotion of engagement and associated behaviors by both patients and healthcare professionals.

The health costs of ethnic distance: evidence from sub-Saharan Africa

This paper shows that children of mothers who are ethnically more distant from their neighbours have worse health outcomes. I combine individual-level micro data from DHS surveys for 14 sub-Saharan African countries with a novel high-resolution dataset on the spatial distribution of ethnic groups at the $$1\,\hbox {km} \times 1\,\hbox {km}$$ level. I measure ethnic distance using linguistic distance and construct the spatial distribution of ethnic groups using an iterative proportional fitting algorithm. Using a time-varying ethnicity fixed effects framework to curb unobserved heterogeneity across ethnic groups, I show that children whose mothers are linguistically more distant from their neighbours face higher mortality rates and are shorter in stature. The pernicious effects of linguistic distance are more pronounced in areas where malaria is endemic. I argue that higher linguistic distance impedes the transmission of information. Consistent with this interpretation, mothers who are linguistically more distant from their neighbours are less likely to receive health-related information. Linguistic distances driven by splits that occurred thousands of years ago are more relevant than more recent splits.

Evaluations of small area composite estimators based on the iterative proportional fitting algorithm

This article deals with the use of sample size dependent composite estimators in spatial microsimulation approaches for small area estimation. This approach has been applied to regression-based small area estimation approaches but never to our knowledge to spatial microsimulation approaches. In this paper, we extend the iterative proportional fitting (IPF) spatial microsimulation technique to small area composite estimators. Using a simulation study, we show both the impact of sample size and the gains from composite estimation to the mean squared error of IPF-based composite estimators. The target variable used is a binary variable reporting good health or bad health.

Population Synthesis with Quasirandom Integer Sampling

Established methods for synthesising a population from geographically aggregated data are robust and well understood. However, most rely on the potentially detrimental process of integerisation if a whole individual population is required, e.g. for use in agent-based modelling (ABM). This paper describes and investigates the use of quasirandom sequences to sample populations from known marginal constraints whilst preserving those marginal distributions. We call this technique Quasirandom Integer Without-replacement Sampling (QIWS) and show that the statistical properties of quasirandomly sampled populations to be superior to those of pseudorandomly sampled ones in that they tend to yield entropies much closer to populations generated using the entropy-maximising iterative proportional fitting (IPF) algorithm. The implementation is extremely efficient, easily outperforming common IPF implementations. It is freely available as an open source R package called humanleague. Finally, we suggest how the current limitations of the implementation can be overcome, providing a direction for future work.

Statistical matching and uncertainty analysis in combining household income and expenditure data

Among the goals of statistical matching, a very important one is the estimation of the joint distribution of variables not jointly observed in a sample survey but separately available from independent sample surveys. The absence of joint information on the variables of interest leads to uncertainty about the data generating model since the available sample information is unable to discriminate among a set of plausible joint distributions. In the present paper a short review of the concept of uncertainty in statistical matching under logical constraints, as well as how to measure uncertainty for continuous variables is presented. The notion of matching error is related to an appropriate measure of uncertainty and a criterion of selecting matching variables by choosing the variables minimizing such an uncertainty measure is introduced. Finally, a method to choose a plausible joint distribution for the variables of interest via iterative proportional fitting algorithm is described. The proposed methodology is then applied to household income and expenditure data when extra sample information regarding the average propensity to consume is available. This leads to a reconstructed complete dataset where each record includes measures on income and expenditure.

SimAthens: A spatial microsimulation approach to the estimation and analysis of small area income distributions and poverty rates in the city of Athens, Greece

Published during a severe economic crisis, this study presents the first spatial microsimulation model for the analysis of income inequalities and poverty in Greece. First, we present a brief overview of the method and discuss its potential for the analysis of multidimensional poverty and income inequality in Greece. We then present the SimAthens model, based on a combination of small-area demographic and socioeconomic information available from the Greek census of population with data from the European Union Statistics on Income and Living Conditions (EU-SILC). The model is based on an iterative proportional fitting (IPF) algorithm, and is used to reweight EU-SILC records to fit in small-area descriptions for Athens based on 2001 and 2011 censuses. This is achieved by using demographic and socioeconomic characteristics as constraint variables. Finally, synthesis of the labor market and occupations are chosen as the main variables for externally validating our results, in order to verify the integrity of the model. Results of this external validation process are found to be extremely satisfactory, indicating a high goodness of fit between simulated and real values. Finally, the study presents a number of model outputs, illustrating changes in social and economic geography, during a severe economic crisis, offering a great opportunity for discussing further potential of this model in policy analysis.

Choice Set Imputation in Atomistic Spatial Choice Models

Constructing the universal choice set in spatial choice models developed at the level of elemental alternatives (atomistic models) is challenging because disaggregate data on the attributes of nonchosen alternatives are often not available. Even when the disaggregate data on nonchosen alternatives are available, matching two data sources will inevitably be error prone given that they might be collected at different times and they might have different coding for categorical variables. An important practical question in the estimation of such atomistic models, therefore, is how to construct the universal choice set in the absence of disaggregate data on the attributes of the nonchosen alternatives. This paper presents a novel approach for spatial imputation of attributes of nonchosen alternatives for estimation and application of atomistic spatial choice models in the absence of disaggregate data. The proposed approach uses the iterative proportional fitting algorithm to impute the attributes of nonchosen alternatives from aggregated data on elemental alternatives. The proposed method is validated with a Monte Carlo experiment and applied to real data in the London residential location choice context.

The iterative proportional fitting algorithm for adjusted agreement in a non‐inferiority diagnostic clinical trial

In this paper, we describe a method of comparing agreement between two diagnostic contingency tables after adjustment to more clinically relevant marginal distributions using the iterative proportional fitting algorithm. When the categories of a contingency table represent mild, moderate, and severe outcomes, the majority of patients often are in the mild category. Because it is often of more interest to evaluate agreement when patients are uniformly distributed among categories, we present the primary results of two clinical trials with adjustment to this structure. We also describe the relationship between the sponsor's pre-specified agreement measure for the observed contingency table and kappa for the adjusted table; and by either criterion, we then show that the agreement of the new diagnostic tool with the standard diagnostic tool is comparably non-inferior to the agreement of the standard diagnostic tool with itself.

Independence analysis of nominal data with the use of log-linear models in R

Log-linear models are used to analyze the relationship between two or more categorical (e.g. nominal or ordinal) variables. The term log-linear derives from the fact that one can, through logarithmic transformations, restate the problem of analyzing multi-way frequency tables in terms that are very similar to ANOVA. Specifically, one may think of the multi-way frequency table to reflect various main effects and interaction effects that add together in a linear fashion to bring about the observed table of frequencies. There are several types of models between dependence and independence: homogenous association, partial association, conditional association and null model. Expected cell frequencies are obtained with the use of iterative proportional fitting algorithm (IPF) [Deming, Stephen 1940]. The next step is to derive model coefficients for single variables as well as for interaction parameter and the most useful tool for interpreting model parameter is odds and odds ratio. Log-linear models are available in R software with the use of loglm function in MASS library and glm function in stats library. In this paper log-linear analysis will be presented with the use of available packages on empirical datasets in economic area.

Robust Gaussian Graphical Modeling Via l1 Penalization

Gaussian graphical models have been widely used as an effective method for studying the conditional independency structure among genes and for constructing genetic networks. However, gene expression data typically have heavier tails or more outlying observations than the standard Gaussian distribution. Such outliers in gene expression data can lead to wrong inference on the dependency structure among the genes. We propose a l(1) penalized estimation procedure for the sparse Gaussian graphical models that is robustified against possible outliers. The likelihood function is weighted according to how the observation is deviated, where the deviation of the observation is measured based on its own likelihood. An efficient computational algorithm based on the coordinate gradient descent method is developed to obtain the minimizer of the negative penalized robustified-likelihood, where nonzero elements of the concentration matrix represents the graphical links among the genes. After the graphical structure is obtained, we re-estimate the positive definite concentration matrix using an iterative proportional fitting algorithm. Through simulations, we demonstrate that the proposed robust method performs much better than the graphical Lasso for the Gaussian graphical models in terms of both graph structure selection and estimation when outliers are present. We apply the robust estimation procedure to an analysis of yeast gene expression data and show that the resulting graph has better biological interpretation than that obtained from the graphical Lasso.

Comparisons of Three Approaches for Discrete Conditional Models

This article describes three methods for computing a discrete joint density from full conditional densities. They are the Gibbs sampler, a hybrid method, and an interaction-based method. The hybrid method uses the iterative proportional fitting algorithm, and it is derived from the mixed parameterization of a contingency table. The interaction-based approach is derived from the canonical parameters, while the Gibbs sampler can be regarded as based on the mean parameters. In short, different approaches are motivated by different parameterizations. The setting of a bivariate conditionally specified distribution is used as the premise for comparing the numerical accuracy of the three methods. Detailed comparisons of marginal distributions, odds ratios and expected values are reported. We give theoretical justifications as to why the hybrid method produces better approximation than the Gibbs sampler. Generalizations to more than two variables are discussed. In practice, Gibbs sampler has certain advantages: it is conceptually easy to understand and there are many software tools available. Nevertheless, the hybrid method and the interaction-based method are accurate and simple alternatives when the Gibbs sampler results in a slowly mixing chain and requires substantial simulation efforts.

Dependence-Preserving Approach to Synthesizing Household Characteristics

One effective approach to study day-to-day traveler behavior is through the activity-based traffic demand model, in which all travelers are treated as individual agents and interact under a computation-intensive framework. Nevertheless, because of high survey costs, low response rate, and privacy concerns, detailed household and personal characteristics are usually unavailable. Various population synthesizers were therefore proposed to reconstruct a methodologically rigorous estimate of household characteristics from different surveys. For instance, the iterative proportional fitting (IPF) algorithm is used to synthesize the full population from the public use microdata sample (PUMS) and Census Summary File 3 (SF3) in the popular activity-based traffic demand model, TRANSIMS. However, some fundamental limitations of IPF (e.g., zero cells in the contingency table as a result of small sample size) have drawn sufficient attention and resulted in the development of enhanced IPF algorithms and other strategies. This paper proposes a copula-based method to synthesize household characteristics that preserves marginal distributions and dependence structure between variables. The proposed method is tested for the state of Iowa, and the results are compared with the IPF approach of TRANSIMS. The synthesized households resulted in the same local SF3 statistics at each block group. But having similar inter-variable correlations as described in the PUMS suggest the applicability of the copula-based approach. Because marginal distributions and dependence structure can be faithfully preserved, the proposed method could be a suitable alternative to synthesize realistic agent characteristics for further activity-based traffic demand modeling.

Generating synthetic baseline populations from register data

The paper presents a population synthesiser based on the method of Iterative Proportional Fitting (IPF) algorithm developed for the new Danish national transport model system. The synthesiser is designed for large population matrices and allows target variables to be represented in several target constraints. As a result, constraints for the IPF are cross-linked, which makes it difficult to ensure consistency of targets in a forecast perspective. The paper proposes a new solution strategy to ensure internal consistency of the population targets in order to guarantee proper convergence of the IPF algorithm. The solution strategy consists in establishing a harmonisation process for the population targets, which combined with a linear programming approach, is applied to generate a consistent target representation. The model approach is implemented and tested on Danish administrative register data. A test on historical census data shows that a 2006 population could be predicted by a 1994 population with an overall percentage deviation of 5–6% given that targets were known. It is also indicated that the deviation is approximately a linear function of the length of the forecast period.

The mode oriented stochastic search (MOSS) algorithm for log-linear models with conjugate priors

We describe a novel stochastic search algorithm for rapidly identifying regions of high posterior probability in the space of decomposable, graphical and hierarchical log-linear models. Our approach is based on the Diaconis–Ylvisaker conjugate prior for log-linear parameters. We discuss the computation of Bayes factors through Laplace approximations and the Bayesian iterative proportional fitting algorithm for sampling model parameters. We use our model determination approach in a sparse eight-way contingency table.

Computer‐aided diagnosis with potential application to rapid detection of disease outbreaks

Our objectives are to quickly interpret symptoms of emergency patients to identify likely syndromes and to improve population-wide disease outbreak detection. We constructed a database of 248 syndromes, each syndrome having an estimated probability of producing any of 85 symptoms, with some two-way, three-way, and five-way probabilities reflecting correlations among symptoms. Using these multi-way probabilities in conjunction with an iterative proportional fitting algorithm allows estimation of full conditional probabilities. Combining these conditional probabilities with misdiagnosis error rates and incidence rates via Bayes theorem, the probability of each syndrome is estimated. We tested a prototype of computer-aided differential diagnosis (CADDY) on simulated data and on more than 100 real cases, including West Nile Virus, Q fever, SARS, anthrax, plague, tularaemia and toxic shock cases. We conclude that: (1) it is important to determine whether the unrecorded positive status of a symptom means that the status is negative or that the status is unknown; (2) inclusion of misdiagnosis error rates produces more realistic results; (3) the naive Bayes classifier, which assumes all symptoms behave independently, is slightly outperformed by CADDY, which includes available multi-symptom information on correlations; as more information regarding symptom correlations becomes available, the advantage of CADDY over the naive Bayes classifier should increase; (4) overlooking low-probability, high-consequence events is less likely if the standard output summary is augmented with a list of rare syndromes that are consistent with observed symptoms, and (5) accumulating patient-level probabilities across a larger population can aid in biosurveillance for disease outbreaks.

Use of Iterative Proportional Fitting Algorithm for Combining Traffic Count Data with Missing Dimensions

This paper proposes a modified version of the iterative proportional fitting (IPF) algorithm that can be used to combine contingency tables with missing dimensions. In many situations data from more than one survey could be used to answer a certain question of interest. Although using only one of the data sets is the simplest approach, it is uneconomical because information contained in the other data sets is not used. When combining data from different sources, one often faces the problem that the data have not been collected in exactly the same way, so certain information is available only in some data sets. Dominici proposed a hierarchical Bayesian model for combining contingency tables with missing dimensions. Although this is an elegant theory, it has a number of shortcomings, the most important one being that it has a high time and memory complexity. In other words, this method is too slow to be useful for traffic count data in which the number of cells easily exceeds several thousands. The IPF algorithm proposed here consists of iterations whose time complexity is linear only in the number of cells, so it can easily be applied to huge contingency tables such as those obtained from traffic surveys.

Multivariate normal integrals and contingency tables with ordered categories

Ak-dimensional multivariate normal distribution is made discrete by partitioning thek-dimensional Euclidean space with rectangular grids. The collection of probability integrals over the partitioned cubes is ak-dimensional contingency table with ordered categories. It is shown that loglinear model with main effects plus two-way interactions provides an accurate approximation for thek-dimensional table. The complete multivariate normal integral table is computed via the iterative proportional fitting algorithm from bivariate normal integral tables. This approach imposes no restriction on the correlation matrix. Comparisons with other numerical integration algorithms are reported. The approximation suggests association models for discretized multivariate normal distributions and contingency tables with ordered categories.

Collapsing multiway contingency tables: Simpson’s paradox and homogenization

Simpson’s paradox results from the collapsing of multiway contingency tables in the presence of nonadditivity. An extension of the iterative proportional fitting algorithm (Deming & Stephan, 1940) is described that may be used to homogenize such tables to produce the required additivity over the unwanted (collapsed-over) effects.