Marginal Sums Research Articles

Estimating the number of zero-one multi-way tables via sequential importance sampling

In 2005, Chen et al. introduced a sequential importance sampling (SIS) procedure to analyze zero-one two-way tables with given fixed marginal sums (row and column sums) via the conditional Poisson (CP) distribution. They showed that compared with Monte Carlo Markov chain (MCMC)-based approaches, their importance sampling method is more efficient in terms of running time and also provides an easy and accurate estimate of the total number of contingency tables with fixed marginal sums. In this paper, we extend their result to zero-one multi-way ( $$d$$ -way, $$d \ge 2$$ ) contingency tables under the no $$d$$ -way interaction model, i.e., with fixed $$d-1$$ marginal sums. Also, we show by simulations that the SIS procedure with CP distribution to estimate the number of zero-one three-way tables under the no three-way interaction model given marginal sums works very well even with some rejections. We also applied our method to Samson’s monks data set.

Interactions in the Analysis of Variance

The standard model for the analysis of variance is over-parameterized. The resulting identifiability problem is typically solved by placing linear constraints on the parameters. In the case of the interactions, these require that the marginal sums be zero. Although seemingly neutral, these conditions have unintended consequences: the interactions are of necessity connected whether or not this is justified, the minimum number of nonzero interactions is four, and, in particular, it is not possible to have a single interaction in one cell. There is no reason why nature should conform to these constraints. The approach taken in this article is one of sparsity: the linear factor effects are chosen so as to minimize the number of nonzero interactions subject to consistency with the data. The resulting interactions are attached to individual cells making their interpretation easier irrespective of whether they are isolated or form clusters. In general, the calculation of a sparse solution is a difficult combinatorial problem but the special nature of the analysis of variance simplifies matters considerably. In many cases, the sparse L 0 solution coincides with the L 1 solution obtained by minimizing the sum of the absolute residuals and that can be calculated quickly. The identity of the two solutions can be checked either algorithmically or by applying known sufficient conditions for equality.

Scale effect and bimodality in the frequency distribution of species occupancy

Community patterns in species-by-site matrices provide valuable clues for inferring ecological processes at work. One such pattern is the occupancy frequency distribution (OFD) depicting the frequency distribution of row sums (i.e., occupancy) with a quarter OFDs of bimodal forms. Another pattern that also reflects the structure of row sums is the ranked species occupancy curve (RSOC), and has been shown to imply a 50% of bimodality in OFDs. The use of RSOCs has been advocated in literature over the OFD based on two conclusions from a 6-model inference using only 24 matrices: (i) RSOCs have two general forms, with half representing bimodal OFDs; (ii) there are no effects of spatial and study scales on RSOCs of different forms. Using a much more representative dataset of 289 matrices, I cast doubt on these two conclusions. A missing but dominant RSOC model (the truncated power law) is added. The number of species and the nestedness of the community differ significantly among matrices of different RSOC forms; however, the number of sites and the taxa in the studies do not differ among RSOC or OFD forms. The quarter OFDs of bimodal forms is reassured, with the least frequent occupancy consistent with Raunkiaer’s law of frequency. Importantly, a RSOC is mathematically transferrable to an OFD, with the derivative of the occupancy ranking curve being equal to the negative reciprocal of the occupancy frequency. Based on the type of the community (null versus interactive) and site environment (homogenous versus heterogeneous), four scenarios are needed to identify pre-inferring assemblage mechanisms. The results highlight the need for shifting research from the emphasis of marginal sums to the analysis of matrix structure for an in-depth understanding of the community assemblage patterns and mechanisms.

An efficient algorithm for Lindner’s test (configural frequency analysis)

Lindner’s (Psychologische Beitrage 26:393–415, 1984) test is a generalisation of Fisher’s exact test for 2 × 2 contingency tables to 2k contingency tables (configural frequency analysis). It has been deemed to be impractical for large numbers of parameters and large marginal sums. This paper transforms the test’s formula and presents an algorithm suitable for large sets of data without a loss of accuracy. The algorithm that is used is useful for detecting and testing syndromes in psychology, sociology, medicine, biology and other sciences.

Asymptotic enumeration of integer matrices with large equal row and column sums

Let s, t, m, n be positive integers such that sm = tn. Let M(m, s; n, t) be the number of m×n matrices over {0, 1, 2, …} with each row summing to s and each column summing to t. Equivalently, M(m, s; n, t) counts 2-way contingency tables of order m×n such that the row marginal sums are all s and the column marginal sums are all t. A third equivalent description is that M(m, s; n, t) is the number of semiregular labelled bipartite multigraphs with m vertices of degree s and n vertices of degree t. When m = n and s = t such matrices are also referred to as n×n magic or semimagic squares with line sums equal to t. We prove a precise asymptotic formula for M(m, s; n, t) which is valid over a range of (m, s; n, t) in which m, n→∞ while remaining approximately equal and the average entry is not too small. This range includes the case where m/n, n/m, s/n and t/m are bounded from below.

Efficient importance sampling for binary contingency tables

Importance sampling has been reported to produce algorithms with excellent empirical performance in counting problems. However, the theoretical support for its e¢ ciency in these applications has been very limited. In this paper, we propose a methodology that can be used to design e¢ cient importance sampling algorithms for counting and test their e¢ ciency rigorously. We apply our techniques after transforming the problem into a rare-event simulation problem –thereby connecting complexity analysis of counting problems with e¢ ciency in the context of rare-event simulation. As an illustration of our approach, we consider the problem of counting the number of binary tables with …xed column and row sums, cj’s and ri’s respectively, and total marginal sums d = P j cj. Assuming that maxj cj = o d 1=2 � , P c 2 = O(d) and the rj’s are bounded we show that a suitable importance sampling algorithm (proposed by Chen, Diaconis, Holmes and Liu (2005)) requires O(d 2 2 � 1 ) operations to produce an estimate that has -relative error with probability 1 �. In addition, if maxj cj = o d 1=4 �0 � for some �0 > 0, the same coverage can be guaranteed with O d 2 2 log � 1 �� operations.

Complementary Logic Gates and Ring Oscillators on Plastic Substrates by Use of Printed Ribbons of Single-Crystalline Silicon

CMOS inverters and three-stage ring oscillators were formed on flexible plastic substrates by transfer printing of p-type and n-type single crystalline ribbons of silicon. The gain and the sum of high and low noise margins of the inverters were as high as ~150 and 4.5 V at supply voltages of 5 V, respectively. The frequencies of the ring oscillators reached 2.6 MHz at supply voltages of 10 V. These results, as obtained with devices that have relatively large critical dimensions (i.e., channel lengths in the several micrometer range), taken together with good mechanical bendability, suggest promise for the use of this type of technology for flexible electronic systems.

Conditional Inference on Tables With Structural Zeros

We develop a set of sequential importance sampling (SIS) strategies for sampling nearly uniformly from two-way zero-one or contingency tables with fixed marginal sums and a given set of structural zeros. The SIS procedure samples tables column by column or cell by cell by using appropriate proposal distributions, and enables us to approximate closely the null distributions of a number of test statistics involved in such tables. When structural zeros are on the diagonal or follow certain patterns, more efficient SIS algorithms are developed which guarantee that every generated table is valid. Examples show that our methods can be applied to make conditional inference on zero-one and contingency tables, and are more efficient than other existing Monte Carlo algorithms.

The Rasch Sampler

The Rasch sampler is an efficient algorithm to sample binary matrices with given marginal sums. It is a Markov chain Monte Carlo (MCMC) algorithm. The program can handle matrices of up to 1024 rows and 64 columns. A special option allows to sample square matrices with given marginals and fixed main diagonal, a problem prominent in social network analysis. In all cases the stationary distribution is uniform. The user has control on the serial dependency.

Sequential Monte Carlo Methods for Statistical Analysis of Tables

We describe a sequential importance sampling (SIS) procedure for analyzing two-way zero–one or contingency tables with fixed marginal sums. An essential feature of the new method is that it samples the columns of the table progressively according to certain special distributions. Our method produces Monte Carlo samples that are remarkably close to the uniform distribution, enabling one to approximate closely the null distributions of various test statistics about these tables. Our method compares favorably with other existing Monte Carlo-based algorithms, and sometimes is a few orders of magnitude more efficient. In particular, compared with Markov chain Monte Carlo (MCMC)-based approaches, our importance sampling method not only is more efficient in terms of absolute running time and frees one from pondering over the mixing issue, but also provides an easy and accurate estimate of the total number of tables with fixed marginal sums, which is far more difficult for an MCMC method to achieve.

Fertilizer market development: a comparative analysis of Ethiopia, Kenya, and Zambia

This article synthesizes case studies from Kenya, Zambia, and Ethiopia to assess how differences in the implementation of fertilizer marketing policies have affected the costs and risks borne by marketing actors, the investment response by private traders, and fertilizer consumption.Financial cost accounting techniques indicate that domestic marketing costs account for 50% or more of farm-gate prices. The sum of importer, wholesaler and retailer profit margins generally account for less than 10%. There are opportunities to reduce domestic marketing costs through the following: reducing port fees, coordinating the timing of fertilizer clearance from the port with up-country transport, reducing transport costs through port, rail, and road improvements, reducing high fuel taxes, and reducing the uncertainty associated with government input distribution programs that impose additional marketing costs on traders. Estimated reductions in the farm-gate price of fertilizer from implementing the full range of options identified in each country range from 11 to 18%. Price reductions of this magnitude, if passed along to farmers, would increase farmers’ effective demand for fertilizer. Investments in selected publically provided goods, often considered outside the scope of fertilizer marketing policy per se, strongly affect the costs of fertilizer, farmers’ willingness to pay for it, and hence the performance of markets.

The Maximum Probability 2 × c Contingency Tables and the Maximum Probability Points of the Multivariate Hypergeometric Distribution

The problem of obtaining the maximum probability 2 × c contingency table with fixed marginal sums, R = (R 1, R 2) and C = (C 1, … , C c ), and row and column independence is equivalent to the problem of obtaining the maximum probability points (mode) of the multivariate hypergeometric distribution MH(R 1; C 1, … , C c ). The most simple and general method for these problems is Joe's (Joe, H. (1988). Extreme probabilities for contingency tables under row and column independence with application to Fisher's exact test. Commun. Statist. Theory Meth. 17(11):3677–3685.) In this article we study a family of MH's in which a connection relationship is defined between its elements. Based on this family and on a characterization of the mode described in Requena and Martín (Requena, F., Martín, N. (2000). Characterization of maximum probability points in the multivariate hypergeometric distribution. Statist. Probab. Lett. 50:39–47.), we develop a new method for the above problems, which is completely general, non recursive, very simple in practice and more efficient than the Joe's method. Also, under weak conditions (which almost always hold), the proposed method provides a simple explicit solution to these problems. In addition, the well-known expression for the mode of a hypergeometric distribution is just a particular case of the method in this article.

Characterization of maximum probability points in the Multivariate Hypergeometric distribution

This paper presents the characterization of the mode of the Multivariate Hypergeometric distribution MH( R 1; C 1,…, C c ) or, equivalently, of the maximum probability 2× c contingency table with fixed marginal sums and row and column independence. Some results concerning the uniqueness of this mode are given. In addition, a relation between the maximum probability points, for different fixed values of one of the variables, is studied, enabling us to obtain a recurrence relation between its maximum probabilities.

Ecological Inference and Entropy-Maximizing: An Alternative Estimation Procedure for Split-Ticket Voting

Publication of King's A Solution to the Ecological Inference Problem has rekindled interest in the estimation of unknown cell values in two- and three-dimensional matrices from knowledge of the marginal sums. This paper outlines an entropy-maximizing (EM) procedure which employs more constraints than King's EI method and produces mathematical rather than statistical procedures: the estimates are maximum-likelihood values. The mathematics are outlined, and the procedure's use illustrated with a study of ticket-splitting at New Zealand's first (1996) general election using the mixed-member proportional representation system, for which official figures provide a check against the EM estimate of the number voting a straight party ticket in each constituency.

Chain ladder, marginal sum and maximum likelihood estimation

The chain ladder method is one of the most famous methods used in reserving. It exploits all data from the run-off triangle and provides simple estimates of the expected ultimate aggregate claims. Simplicity of an estimator is important for its application in practice, but its performance usually depends upon the stochastic mechanism generating the data. In the present paper we consider two stochastic models which reflect certain elementary ideas on the occurrence and the delay in reporting of claims. We show that in these models the chain ladder estimators of the expected ultimate aggregate claims result from classical statistical estimation principles.

Computing maximum likelihood estimates of loglinear models from marginal sums with special attention to loglinear item response theory

In this paper algorithms are described for obtaining the maximum likelihood estimates of the parameters in loglinear models. Modified versions of the iterative proportional fitting and Newton-Raphson algorithms are described that work on the minimal sufficient statistics rather than on the usual counts in the full contingency table. This is desirable if the contingency table becomes too large to store. Special attention is given to loglinear IRT models that are used for the analysis of educational and psychological test data. To calculate the necessary expected sufficient statistics and other marginal sums of the table, a method is described that avoids summing large numbers of elementary cell frequencies by writing them out in terms of multiplicative model parameters and applying the distributive law of multiplication over summation. These algorithms are used in the computer program LOGIMO. The modified algorithms are illustrated with simulated data.

Exact Simultaneous Comparisons with Control in an r × c Contingency Table

AbstractLet categorical data coming from a control group and (r ‐ 1) treated groups be given in an r × c contingency table. A simultaneous test procedure of the (r ‐ 1) hypotheses that the probabilities of all c categories do not differ between the i‐th treated group and the control is derived. For small tables and small cell frequencies it is exactly performed by generation of all tables having the given marginal sums. If 2 categories or 2 groups only are given the asymptotic distribution of the test statistic is known; otherwise its distribution may be simulated if the computational expenditure of performing an exact test is too large. By means of a Monte Carlo study it is shown that this method meets its level more reliably and that it has a better power than others.

A Network Algorithm for Performing Fisher's Exact Test inr×cContingency Tables

Abstract An exact test of significance of the hypothesis that the row and column effects are independent in an r × c contingency table can be executed in principle by generalizing Fisher's exact treatment of the 2 × 2 contingency table. Each table in a conditional reference set of r × c tables with fixed marginal sums is assigned a generalized hypergeometric probability. The significance level is then computed by summing the probabilities of all tables that are no larger (on the probability scale) than the observed table. However, the computational effort required to generate all r × c contingency tables with fixed marginal sums severely limits the use of Fisher's exact test. A novel technique that considerably extends the bounds of computational feasibility of the exact test is proposed here. The problem is transformed into one of identifying all paths through a directed acyclic network that equal or exceed a fixed length. Some interesting new optimization theorems are developed in the process. The numer...

On the existence and uniqueness of maximum-likelihood estimates in the Rasch model

Necessary and sufficient conditions for the existence and uniqueness of a solution of the so-called “unconditional” (UML) and the “conditional” (CML) maximum-likelihood estimation equations in the dichotomous Rasch model are given. The basic critical condition is essentially the same for UML and CML estimation. For complete data matricesA, it is formulated both as a structural property ofA and in terms of the sufficient marginal sums. In case of incomplete data, the condition is equivalent to complete connectedness of a certain directed graph. It is shown how to apply the results in practical uses of the Rasch model.

Hierarchical Internal Migration Regions of France

Internal migration tables can be employed to ascertain functional migration regions?collections of geographic units with relatively large intraregional flows and relatively small interregional movements. A two-stage algorithm that has been utilized to study tables for various nations is applied here to a 21 × 21 French matrix. In the first step, the square table is adjusted to possess uniform marginal (row and column) sums. The doubly standardized table obtained can be regarded as an asymmetric similarity matrix. It serves, in the second stage, as input to a hierarchical clustering procedure, based on the concep of the strong components of a directed graph. Region Parisienne and Centre are shown to have the broadest migration bases. Well-defined southern and northwestern regions are found.