Small Number Of Bins Research Articles

Constructing outlier-free histograms with variable bin-width based on distance minimization

We propose a new method of constructing a variable bin width histogram that can accommodate the unbalanced distribution of the samples yet retaining, as a whole, the good aspect of both equal width (EW) and equal-area (EA) histograms that are being used popularly for data visualization and analysis. We formulate this as an optimal change point detection problem in which the bin boundaries are determined by minimizing the sum of the absolute error or the squared error in each bin. The former is based on Distance Minimization (DM) and new, and the latter is based on Variance Minimization (VM) and is considered the state-of-the-art. The constructed histograms can effectively be used to detect and visualize hidden outliers/anomalies by applying the interquartile range method in each bin. The final histograms are obtained by adjusting bin boundaries and heights accordingly after removing the detected outliers/anomalies. We further propose a method to annotate the constructed bins if the data for annotation is given for each sample as a set of nominal variables, using z-score with respect to their distribution within each bin. We applied our method to both real vinyl greenhouse datasets and two different sets of three synthetic datasets, and confirmed that both DM and VM methods work as intended, both can represent the sample distribution with a smaller number of bins than those by EW and EA methods, The use of interquartile range method can detect anomalies as well as outliers, and the terms selected for annotation are interpretable and reasonable. EW and EA methods have contrasting properties. DM and VM methods lie in between, but the former is closer to EA method and the latter to EW method. DM method runs substantially faster than VM method and performs slightly better than VM method in outlier detection and annotation tasks.

Best Fit Bin Packing with Random Order Revisited

Best Fit is a well known online algorithm for the bin packing problem, where a collection of one-dimensional items has to be packed into a minimum number of unit-sized bins. In a seminal work, Kenyon [SODA 1996] introduced the (asymptotic) random order ratio as an alternative performance measure for online algorithms. Here, an adversary specifies the items, but the order of arrival is drawn uniformly at random. Kenyon’s result establishes lower and upper bounds of 1.08 and 1.5, respectively, for the random order ratio of Best Fit. Although this type of analysis model became increasingly popular in the field of online algorithms, no progress has been made for the Best Fit algorithm after the result of Kenyon. We study the random order ratio of Best Fit and tighten the long-standing gap by establishing an improved lower bound of 1.10. For the case where all items are larger than 1/3, we show that the random order ratio converges quickly to 1.25. It is the existence of such large items that crucially determines the performance of Best Fit in the general case. Moreover, this case is closely related to the classical maximum-cardinality matching problem in the fully online model. As a side product, we show that Best Fit satisfies a monotonicity property on such instances, unlike in the general case. In addition, we initiate the study of the absolute random order ratio for this problem. In contrast to asymptotic ratios, absolute ratios must hold even for instances that can be packed into a small number of bins. We show that the absolute random order ratio of Best Fit is at least 1.3. For the case where all items are larger than 1/3, we derive upper and lower bounds of 21/16 and 1.2, respectively.

Fewer mocks and less noise: Reducing the dimensionality of cosmological observables with subspace projections

Creating accurate and low-noise covariance matrices represents a formidable challenge in modern-day cosmology. We present a formalism to compress arbitrary observables into a small number of bins by projection into a model-specific subspace that minimizes the prior-averaged log-likelihood error. The lower dimensionality leads to a dramatic reduction in covariance matrix noise, significantly reducing the number of mocks that need to be computed. Given a theory model, a set of priors, and a simple model of the covariance, our method works by using singular value decompositions to construct a basis for the observable that is close to Euclidean; by restricting to the first few basis vectors, we can capture almost all the constraining power in a lower-dimensional subspace. Unlike conventional approaches, the method can be tailored for specific analyses and captures non-linearities that are not present in the Fisher matrix, ensuring that the full likelihood can be reproduced. The procedure is validated with full-shape analyses of power spectra from BOSS DR12 mock catalogs, showing that the 96-bin power spectra can be replaced by 12 subspace coefficients without biasing the output cosmology; this allows for accurate parameter inference using only $\sim 100$ mocks. Such decompositions facilitate accurate testing of power spectrum covariances; for the largest BOSS data chunk, we find that: (a) analytic covariances provide accurate models (with or without trispectrum terms); and (b) using the sample covariance from the MultiDark-Patchy mocks incurs a $\sim 0.5\sigma$ shift in $\Omega_m$, unless the subspace projection is applied. The method is easily extended to higher order statistics; the $\sim 2000$-bin bispectrum can be compressed into only $\sim 10$ coefficients, allowing for accurate analyses using few mocks and without having to increase the bin sizes.

Improving the estimation of educational attainment: New methods for assessing average years of schooling from binned data.

BackgroundThe accurate measurement of educational attainment is of great importance for population research. Past studies measuring average years of schooling rely on strong assumptions to incorporate binned data. These assumptions, which we refer to as the standard duration method, have not been previously evaluated for bias or accuracy.MethodsWe assembled a database of 1,680 survey and census datasets, representing both binned and single-year education data. We developed two models that split bins of education into single year values. We evaluate our models, and compare them to the standard duration method, using out-of-sample predictive validity.ResultsOur results indicate that typical methods used to split bins of educational attainment introduce substantial error and bias into estimates of average years of schooling, as compared to new approaches. Globally, the standard duration method underestimates average years of schooling, with a median error of -0.47 years. This effect is especially pronounced in datasets with a smaller number of bins or higher true average attainment, leading to irregular error patterns between geographies and time periods. Both models we developed resulted in unbiased predictions of average years of schooling, with smaller average error than previous methods. We find that one approach using a metric of distance in space and time to identify training data, had the best performance, with a root mean squared error of mean attainment of 0.26 years, compared to 0.92 years for the standard duration algorithm.ConclusionsEducation is a key social indicator and its accurate estimation should be a population research priority. The use of a space-time distance bin-splitting model drastically improved the estimation of average years of schooling from binned education data. We provide a detailed description of how to use the method and recommend that future studies estimating educational attainment across time or geographies use a similar approach.

Fast texture‐preserving bilateral filtering

A new method for fast bilateral filtering with texture‐preserving properties is presented. In general, the texture area is composed of several tiny regions that have almost similar intensity, and therefore bilateral filtering combines similar gray scales of the texture area to form a smooth region. Adaptive boundary filtering solves this problem by using image segmentation to define a new weighting boundary at each pixel filtering. With the new type of boundary, the output image is smoothed while preserving both edges and textures details. The drawback of this algorithm is that it is time consuming. In this paper, a histogram‐based filtering is proposed to improve the speed of this algorithm. However, as texture details are blurred when a small number of bins are used, it is unable to increase speed by using a small number of bins. Deciles‐based algorithm is therefore applied to overcome this limitation, as it can increase the speed while preserving the texture details. The new algorithm, named fast adaptive boundary filtering, increases the speed by more than 40% when compared with adaptive boundary filtering at a boundary threshold T = 40. The output image of new algorithm shows that it is similar to the output of adaptive boundary filtering. Moreover, the speed of new algorithm is compared with the normal 256‐bin histogram‐based bilateral filtering. From the experimental results, it is seen that the new algorithm has higher speed when it is processed with a small to moderate radius window. In addition, a better output image can be obtained from the new algorithm as it preserves both edges and textures. © 2015 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Coarse-grain model for internal energy excitation and dissociation of molecular nitrogen

A rovibrational collisional coarse-grain model has been developed to reduce a detailed mechanism for the internal energy excitation and dissociation processes behind a strong shockwave in a nitrogen flow. The rovibrational energy levels of the electronic ground state of the nitrogen molecule were lumped into a smaller number of bins. The reaction rate coefficients of an ab initio database developed at NASA Ames Research Center were averaged for each bin based on a uniform distribution of the energy levels within the bin. The results were obtained by coupling the Master equation for the reduced mechanism with a one-dimensional flow solver for conditions expected for reentry into Earth’s atmosphere at 10km/s. The coarse-grain collisional model developed allow us to describe accurately the internal energy relaxation and dissociation processes based on a smaller number of equations, as opposed to existing reduced models assuming thermal equilibrium between the rotational and translational energy modes.

The choice of the number of bins for the [formula omitted] statistic

Methods to monitor spatial patterns of disease in populations are of interest in public health practice. The M statistic uses interpoint distances between cases to detect abnormalities in the spatial patterns of diseases. This statistic compares the observed distribution of interpoint distances with that which is expected when no unusual spatial patterns exist. We show the relationship of M to Pearson’s Chi Square statistic, X n 2 . Both statistics require the discretization of continuous data into bins and then are formed by creating a quadratic form, scaled by an appropriate variance covariance matrix. We seek to choose the number and type of these bins for the M statistic so as to maximize the power to detect spatial anomalies. By showing the relationship between M to X n 2 , we argue for the extension of the theory that has been developed for the selection of the number and type of bins for X n 2 to M . We further show that spatial data provides a unique insight into the problem through examples with simulated data and spatial data from a health care provider. In the spatial setting, these indicate that the optimal number of bins depends on the size of the cluster. For large clusters, a smaller number of bins appears to be preferable, however for small clusters having many bins increases the power. Further, results indicate that the number of bins does not appear to vary with m , the number of spatial locations. We discuss the implications of this result for further work.

Fast approximate PCPs for multidimensional bin-packing problems

We consider approximate PCPs for multidimensional bin-packing problems. In particular, we show how a verifier can be quickly convinced that a set of multidimensional blocks can be packed into a small number of bins. The running time of the verifier is bounded by O(log d n ) where n is the number of blocks and d is the dimension.

The analysis and applications of adaptive-binning color histograms

Histograms are commonly used in content-based image retrieval systems to represent the distributions of colors in images. It is a common understanding that histograms that adapt to images can represent their color distributions more efficiently than do histograms with fixed binnings. However, existing systems almost exclusively adopt fixed-binning histograms because, among existing well-known dissimilarity measures, only the computationally expensive Earth Mover’s Distance (EMD) can compare histograms with different binnings. This paper addresses the issue by defining a new dissimilarity measure that is more reliable than the Euclidean distance and yet computationally less expensive than EMD. Moreover, a mathematically sound definition of mean histogram can be defined for histogram clustering applications. Extensive test results show that adaptive histograms produce the best overall performance, in terms of good accuracy, small number of bins, no empty bin, and efficient computation, compared to existing methods for histogram retrieval, classification, and clustering tasks.

Exploring texture ensembles by efficient Markov chain Monte Carlo-Toward a "trichromacy" theory of texture

Presents a mathematical definition of texture, the Julesz ensemble /spl Omega/(h), which is the set of all images (defined on Z/sup 2/) that share identical statistics h. Then texture modeling is posed as an inverse problem: Given a set of images sampled from an unknown /spl Omega/(h/sub */), we search for the statistics h/sub */ which define the ensemble. A /spl Omega/(h) has an associated probability distribution q(I; h), which is uniform over the images in /spl Omega/(h) and has zero probability outside. The authors previously (1999) showed q(I; h) to be the limit distribution of the FRAME (filter, random field, and minimax entropy) model, as the image lattice /spl Lambda//spl rarr/Z/sup 2/. This conclusion establishes the intrinsic link between the scientific definition of texture on Z/sup 2/ and the mathematical models of texture on finite lattices. It brings two advantages: the practice of texture image synthesis by matching statistics is put on a mathematical foundation; and we need not learn the expensive FRAME model in feature pursuit, model selection and texture synthesis. An efficient Markov chain Monte Carte algorithm is proposed for sampling Julesz ensembles. It generates random texture images by moving along the directions of filter coefficients and, thus, extends the traditional single site Gibbs sampler. We compare four popular statistical measures in the literature, in terms of their descriptive abilities. Our experiments suggest that a small number of bins in marginal histograms are sufficient for capturing a variety of texture patterns.

Jointly optimal receivers for the optical pulse-position modulation channel

Jointly optimal receivers that make decisions in the absence of symbol synchronization are derived and analyzed for a pulse-position modulation, optical direct-detection channel. Both receivers that observe the complete sample-path (count record data) and receivers that observe histogram data are studied, and approximations to them are derived. It is seen that jointly optimal receivers are superior to conventional receivers that have separately designed synchronization and decision subsystems. However, their performance advantage is significant only at very low signal levels. Perhaps more significant is that an approximation to the jointly optimal receivers that is much easier to implement than an optimal, separately synchronized receiver achieves nearly the same performance as the latter. Simulation results indicate that the much less complicated receivers that observe histogram data perform as well as receivers that observe the complete sample path at a rather small number of bins per slot. >

Graphical presentation of multidimensional flow histogram using hexagonal segmentation.

Improvements in the method of presenting a multidimensional histogram for flow cytometric data are described. Two-dimensional data were systematically quantized into hexagonal subareas, which facilitated an accurate presentation of the distribution with a smaller number of bins and sample cells.