Iterative Proportional Fitting Procedure Research Articles

Regression‐based network‐flow and inner‐matrix reconstruction

AbstractNetwork or matrix reconstruction is a general problem that occurs if the row‐ and column sums of a matrix are given, and the matrix entries need to be predicted conditional on the aggregated information. In this paper, we show that the predictions obtained from the iterative proportional fitting procedure (IPFP) or equivalently maximum entropy (ME) can be obtained by restricted maximum likelihood estimation relying on augmented Lagrangian optimization. Based on this equivalence, we extend the framework of network reconstruction, conditional on row and column sums, toward regression, which allows the inclusion of exogenous covariates and bootstrap‐based uncertainty quantification. More specifically, the mean of the regression model leads to the observed row and column margins. To exemplify the approach, we provide a simulation study and investigate interbank lending data, provided by the Bank for International Settlement. This dataset provides full knowledge of the real network and is, therefore, suitable to evaluate the predictions of our approach. It is shown that the inclusion of exogenous information leads to superior predictions in terms of and errors.

Quantitative uniform stability of the iterative proportional fitting procedure

Quantitative uniform stability of the iterative proportional fitting procedure

Analysis of electrification and its greenhouse gas reduction potential in the industrial sector of Korea using mixed methods

Carbon neutrality has heightened the necessity for deep decarbonization in the industrial sector, emphasizing the significance of ambitious mitigation measures, including system-level energy and material efficiency, carbon capture and utilization, electrification, and green hydrogen utilization. Electrification, particularly in the hard-to-abate industrial sector, emerges as a critical tool assuming the power sector's decarbonization. This study aims to evaluate Korea's industrial sector electrification potential and its greenhouse gas (GHG) reduction potential by 2050. To achieve this, we employ a mixed-method approach, integrating quantitative methods, such as the Iterative Proportional Fitting Procedure (IPFP) for data consistency, energy forecast, and elasticity of substitution models, along with a qualitative method involving energy experts through the Delphi technique. The results indicate an estimated 41% electrification potential for Korea's industrial sector by 2050, with a potential GHG reduction of approximately 26.3% compared to the Business-as-Usual emissions.

Comparison of methods used for filling partially unobserved contingency tables

In this article, we investigate contingency tables where the entries containing small counts are unknown for data privacy reasons. We propose and test two competitive methods for estimating the unknown entries: our modification of the Iterative Proportional Fitting Procedure (IPFP), and one of the Monte Carlo Markov Chain methods called Shake-and-Bake. We use simulation experiments to test these methods in terms of time complexity and the accuracy of searching the space of feasible solutions. To simplify the estimation procedure, we propose to pre-process partially unknown contingency tables with simple heuristics and dimensionality-reduction techniques to find and fill all trivial entries. Our results demonstrate that if the number of missing cells is not very large, the pre-processing is often enough to find fillings for the unknown values in contingency tables. In the cases where simple heuristics are insufficient, the Shake-and-Bake technique outperforms the modified IPFP in terms of time complexity and the accuracy of searching the space of feasible solutions.

Downscaled energy demand projection at the local level using the Iterative Proportional Fitting procedure

In this paper we forecast energy demand at the local level using a new two-dimensional downscaling methodology that respects regional and subnational variation and, as well as consistency at the aggregate level. This methodology combines the one-dimensional statistical downscaling and the Iterative Proportional Fitting (IPF) procedure. Previous projection studies based on downscaling methods have focused on allocating one-dimensional quantity indicators such as GDP, population, and emissions, into geographically or administratively lower levels. In case of energy demand projections, however, downscaling involves not only regional allocation of energy demand quantities but also energy mixes of the demands in the regions. Our novel two-dimensional downscaling methodology allows us to downscale national energy projections in both demand quantities and their energy mix to regional levels by combining the statistical downscaling and the IPF procedure. We illustrate the methodology by deriving energy demand projections for provinces and metropolitan cities in Korea from national scenarios. In addition, we provide two more simple applications of the IPF to show utilization potentials of the IPF in energy demand forecasting and statistics.

An efficient hierarchical model for multi-source information fusion

In urban and transportation research, important information is often scattered over a wide variety of independent datasets which vary in terms of described variables and sampling rates. As activity-travel behavior of people depends particularly on socio-demographics and transport/urban-related variables, there is an increasing need for advanced methods to merge information provided by multiple urban/transport household surveys. In this paper, we propose a hierarchical algorithm based on a Hidden Markov Model (HMM) and an Iterative Proportional Fitting (IPF) procedure to obtain quasi-perfect marginal distributions and accurate multi-variate joint distributions. The model allows for the combination of an unlimited number of datasets. The model is validated on the basis of a synthetic dataset with 1,000,000 observations and 8 categorical variables. The results reveal that the hierarchical model is particularly robust as the deviation between the simulated and observed multivariate joint distributions is extremely small and constant, regardless of the sampling rates and the composition of the datasets in terms of variables included in those datasets. Besides, the presented methodological framework allows for an intelligent merging of multiple data sources. Furthermore, heterogeneity is smoothly incorporated into micro-samples with small sampling rates subjected to potential sampling bias. These aspects are handled simultaneously to build a generalized probabilistic structure from which new observations can be inferred. A major impact in term of expert systems is that the outputs of the hierarchical model (HM) model serve as a basis for a qualitative and quantitative analyses of integrated datasets.

Mipfp: An R Package for Multidimensional Array Fitting and Simulating Multivariate Bernoulli Distributions

This paper explains the mipfp package for R with the core functionality of updating an d-dimensional array with respect to given target marginal distributions, which in turn can be multi-dimensional. The implemented methods include the iterative proportional fitting procedure (IPFP), the maximum likelihood method, the minimum chi-square and least squares procedures. The package also provides an application of the IPFP to simulate data from a multivariate Bernoulli distribution. The functionalities of the package are illustrated through two practical examples: the update of a 3-dimensional contingency table to match the targets for a synthetic population and the estimation and simulation of the joint distribution of the binary attribute impaired pulmonary function as used by Qaqish, Zink, and Preisser (2012).

Estimating Cross-Classified Population Counts of Multidimensional Tables: An Application to Regional Australia to Obtain Pseudo-Census Counts

Abstract Estimating population counts for multidimensional tables based on a representative sample subject to known marginal population counts is not only important in survey sampling but is also an integral part of standard methods for simulating area-specific synthetic populations. In this article several estimation methods are reviewed, with particular focus on the iterative proportional fitting procedure and the maximum likelihood method. The performance of these methods is investigated in a simulation study for multidimensional tables, as previous studies are limited to 2 by 2 tables. The data are generated under random sampling but also under misspecification models, for which sample and target populations differ systematically. The empirical results show that simple adjustments can lead to more efficient estimators, but generally, at the expense of increased bias. The adjustments also generally improve coverage of the confidence intervals. The methods discussed in this article along with standard error estimators, are made freely available in the R package mipfp. As an illustration, the methods are applied to the 2011 Australian census data available for the Illawarra Region in order to obtain estimates for the desired three-way table for age by sex by family type with known marginal tables for age by sex and for family type.

Continuity of f-projections and applications to the iterative proportional fitting procedure

ABSTRACTThis paper proves continuity of f-projections and the continuous dependence of the limit matrix of the iterative proportional fitting procedure (IPF procedure) on the given matrix as well as the given marginals under certain regularity constraints. For finite spaces, the concept of f-projections of finite measures on a compact and convex set is introduced and continuity of f-projections is proven. This result is applied to the IPF procedure. Given a nonnegative matrix as well as row and column marginals the IPF procedure generates a sequence of matrices, called the IPF sequence, by alternately fitting rows and columns to match their respective marginals. The procedure is equivalent to cyclic f-projections. If the IPF sequence converges, the application of the continuity of f-projections yields the continuous dependence of the limit matrix on the given matrix. By generalized convex programming and under some constraints, it is shown that the limit matrix of the IPF sequence continuously depends not only on the given matrix but also on the marginals.

BOUNDING WRONG‐WAY RISK IN CVA CALCULATION

AbstractA credit valuation adjustment (CVA) is an adjustment applied to the value of a derivative contract or a portfolio of derivatives to account for counterparty credit risk. Measuring CVA requires combining models of market and credit risk to estimate a counterparty's risk of default together with the market value of exposure to the counterparty at default. Wrong‐way risk refers to the possibility that a counterparty's likelihood of default increases with the market value of the exposure. We develop a method for bounding wrong‐way risk, holding fixed marginal models for market and credit risk and varying the dependence between them. Given simulated paths of the two models, a linear program computes the worst‐case CVA. We analyze properties of the solution and prove convergence of the estimated bound as the number of paths increases. The worst case can be overly pessimistic, so we extend the procedure by constraining the deviation of the joint model from a baseline reference model. Measuring the deviation through relative entropy leads to a tractable convex optimization problem that can be solved through the iterative proportional fitting procedure. Here, too, we prove convergence of the resulting estimate of the penalized worst‐case CVA and the joint distribution that attains it. We consider extensions with additional constraints and illustrate the method with examples.

Copula-Based Approach to Synthetic Population Generation.

Generating synthetic baseline populations is a fundamental step of agent-based modeling and simulation, which is growing fast in a wide range of socio-economic areas including transportation planning research. Traditionally, in many commercial and non-commercial microsimulation systems, the iterative proportional fitting (IPF) procedure has been used for creating the joint distribution of individuals when combining a reference joint distribution with target marginal distributions. Although IPF is simple, computationally efficient, and rigorously founded, it is unclear whether IPF well preserves the dependence structure of the reference joint table sufficiently when fitting it to target margins. In this paper, a novel method is proposed based on the copula concept in order to provide an alternative approach to the problem that IPF resolves. The dependency characteristic measures were computed and the results from the proposed method and IPF were compared. In most test cases, the proposed method outperformed IPF in preserving the dependence structure of the reference joint distribution.

Alternate Scaling algorithm for biproportional divisor methods

In parliamentary elections biproportional divisor methods translate votes into seats so that for each district fixed seat contingents are met and that every party receives as many seats as the overall vote counts reflect. A set of district-divisors and party-divisors ensures that proportionality is respected both within the districts and within the parties. The divisors can be calculated by means of the Alternate Scaling algorithm (AS-algorithm). It is the discrete variant of the iterative proportional fitting procedure (IPF-procedure). The AS-algorithm iteratively generates scaled vote matrices that after rounding alternately fulfill the district-contingents and the party-seats. Thus it defines two sequences: the AS-scaling-sequence and the AS-seat-sequence. The central question in this paper is: under which conditions does the AS-algorithm generate a biproportional apportionment? The conjecture of Balinski and Pukelsheim (2006) is partially proven. We show that if the set of biproportional apportionments does not contain more than three elements then the AS-algorithm is able to determine it. In the rare event that the set of biproportional apportionments cannot be determined by the AS-algorithm, the complementary AS-Tie&Transfer-combination puts things right. Its analysis leads to a constructive proof of necessary and sufficient conditions for the existence of biproportional apportionments. If these conditions are violated the sequences generated by the AS-algorithms may have more than two accumulation points. On the contrary, the IPF-procedure has at most two accumulation points.

Insights into European interbank network contagion

Purpose – The purpose of this paper is to analyse the importance of interbank connections and shocks on banks’ capital ratios to financial stability by looking at a network comprising a large number of European and UK banks. Design/methodology/approach – The authors model interbank contagion using insights from the Susceptible Infected Recovered model. The authors construct scale-free networks with preferential attachment and growth, applying simulated interbank data to capture the size and scale of connections in the network. The authors proceed to shock these networks per country and perform Monte Carlo simulations to calculate mean total losses and duration of infection. Finally, the authors examine the effects of contagion in terms of Core Tier 1 Capital Ratios for the affected banking systems. Findings – The authors find that shocks in smaller banking systems may cause smaller overall losses but tend to persist longer, leading to important policy implications for crisis containment. Originality/value – The authors infer the interbank domestic and cross-border exposures of banks employing an iterative proportional fitting procedure, called the RAS algorithm. The authors use an extend sample of 169 European banks, that also captures effects on the UK as well as the Eurozone interbank markets. Finally, the authors provide evidence of the contagion effect on each bank by allowing heterogeneity. The authors compare the bank’s relative financial strength with the contagion effect which is modelled by the number and the volume of bilateral connections.

Bounding Wrong-Way Risk in CVA Calculation

A credit valuation adjustment (CVA) is an adjustment applied to the value of a derivative contract or a portfolio of derivatives to account for counterparty credit risk. Measuring CVA requires combining models of market and credit risk to estimate a counterparty's risk of default together with the market value of exposure to the counterparty at default. Wrong-way risk refers to the possibility that a counterparty's likelihood of default increases with the market value of the exposure. We develop a method for bounding wrong-way risk, holding fixed marginal models for market and credit risk and varying the dependence between them. Given simulated paths of the two models, a linear program computes the worst-case CVA. We analyze properties of the solution and prove convergence of the estimated bound as the number of paths increases. The worst case can be overly pessimistic, so we extend the procedure by constraining the deviation of the joint model from a baseline reference model. Measuring the deviation through relative entropy leads to a tractable convex optimization problem that can be solved through the iterative proportional fitting procedure. Here, too, we prove convergence of the resulting estimate of the penalized worst-case CVA and the joint distribution that attains it. We consider extensions with additional constraints and illustrate the method with examples.

Iterative Scaling in Curved Exponential Families

AbstractThe paper describes a generalized iterative proportional fitting procedure that can be used for maximum likelihood estimation in a special class of the general log‐linear model. The models in this class, called relational, apply to multivariate discrete sample spaces that do not necessarily have a Cartesian product structure and may not contain an overall effect. When applied to the cell probabilities, the models without the overall effect are curved exponential families and the values of the sufficient statistics are reproduced by the MLE only up to a constant of proportionality. The paper shows that Iterative Proportional Fitting, Generalized Iterative Scaling, and Improved Iterative Scaling fail to work for such models. The algorithm proposed here is based on iterated Bregman projections. As a by‐product, estimates of the multiplicative parameters are also obtained. An implementation of the algorithm is available as an R‐package.

On the Estimation of Parameters and Best Model Fits of Log Linear Model for Contingency Table

In this paper, we proposed the generalized method and algorithms developed for estimation of parameters and best model fits of log linear model for n-dimensional contingency table. For purpose of this work, the method was used to provide parameter estimates of log-linear model for three-dimensional contingency table. In estimating parameter estimates and best model fit, computer programs in R were developed for the implementation of the algorithms. The iterative proportional fitting procedure was used to find the parameter estimates and goodness of fits of the log linear model. Akaike information criteria (AIC) and Bayesian information criteria (BIC) were used to check the adequacy of the model of the best fit. Secondary data were used for illustration and the result obtained showed that the best model fit for three-dimensional contingency table had a generating class: [CA, AB]. This showed that the best model fit had sufficient evidence to fit the data without loss of information. This model also revealed that breed was independent of chick loss given age. The best model in harmony with the hierarchy principle is Logmijk＝μ＋μC(i)＋ μA(j)＋ μB(k)＋ μCA(ij)＋ μAB(jk).

Termination of the iterative proportional fitting procedure

The iterative proportional fitting procedure (IPF procedure) alternately fits a given nonnegative matrix to given positive row marginals and given positive column marginals. This paper proves that if the IPF procedure terminates, then this has to be within the first two steps.

Synthetic Population Generation at Disaggregated Spatial Scales for Land Use and Transportation Microsimulation

The execution of agent-based microsimulation requires an initial set of agents with detailed socioeconomic and demographic attributes to support subsequent behavioral and market models. Data limitations and privacy reasons often restrict the scope and detail with which a synthetic population can be generated by the traditional population synthesis approach. For the accommodation of the growing requirement of microsimulation on spatial resolution and variety, considering new data sources that overcome the data limitations and support population synthesis at more disaggregated levels is necessary. This paper presents a two-stage population synthesis approach not only to improve the accuracy of population generation with imperfect microdata and marginal data, but also to use additional data sets when the spatial details of the synthetic population are interpolated. A general iterative proportional fitting (IPF) method is used in the first stage to estimate the joint distribution of household and individual characteristics under multiple levels of constraints. Additional building information is collected from multiple sources and used to estimate spatial patterns of housing and household characteristics that are then preserved through a second IPF procedure. Preliminary tests of the proposed two-stage IPF-based approach with Singapore data show that the method yields better fitted population realizations at more fine-grained levels than do traditional one-step population synthesis methods.

Biproportional scaling of matrices and the iterative proportional fitting procedure

A short proof is given of the necessary and sufficient conditions for the convergence of the Iterative Proportional Fitting procedure. The input consists of a nonnegative matrix and of positive target marginals for row sums and for column sums. The output is a sequence of scaled matrices to approximate the biproportional fit, that is, the scaling of the input matrix by means of row and column divisors in order to fit row and column sums to target marginals. Generally it is shown that certain structural properties of a biproportional scaling do not depend on the particular sequence used to approximate it. Specifically, the sequence that emerges from the Iterative Proportional Fitting procedure is analyzed by means of the L 1-error that measures how current row and column sums compare to their target marginals. As a new result a formula for the limiting L 1-error is obtained. The formula is in terms of partial sums of the target marginals, and easily yields the other well-known convergence characterizations.

Accumulation points of the iterative proportional fitting procedure

The asymptotic behavior of the iterative proportional fitting procedure (IPF procedure) is analyzed comprehensively. Given a nonnegative matrix as well as row and column marginals the IPF procedure generates a sequence of matrices, called the IPF sequence, by alternately fitting rows and columns to match their respective marginals. We prove that the IPF sequence has at most two accumulation points. They originate as the limits of the even-step subsequence, and of the odd-step subsequence. The well-known IPF convergence criteria are then retrieved easily. Our proof is based on Csiszár’s and Tusnády’s (Stat Decis Suppl Issue 1:205–237, 1984) results on the interplay of the I-divergence geometry and alternating minimization procedures.