Data Shuffling Research Articles

Interactive Mesostructures with Volumetric Collisions

This paper presents a technique for interactively colliding with and deforming mesostructures at a per-texel level. It is compatible with a broad range of existing mesostructure rendering techniques including both safe and unsafe ray-height field intersection algorithms. This technique is able to replace traditional 3D geometrical deformations (vertex-based) with 2D image space operations (pixel-based) that are parallelized on a GPU without CPU-GPU data shuffling and integrates well with existing physics engines. Additionally, surface and material properties may be specified at a per-texel level enabling a mesostructure to possess varying attributes intrinsic to its surface and collision behavior. Furthermore, this approach may replace traditional decals with image-based operations that naturally accumulate deformations without inserting any new geometry. This technique provides a simple and efficient way to make almost every surface in a virtual world responsive to user actions and events. It requires no preprocessing time and storage requirements of one additional texture or less. The algorithm uses existing inverse displacement map algorithms as well as existing physics engines and can be easily incorporated into new or existing game pipelines.

OPTIMAL FOLDING OF DATA FLOW GRAPHS BASED ON FINITE PROJECTIVE GEOMETRY USING VECTOR SPACE PARTITIONING

A number of computations exist, especially in area of error-control coding and matrix computations, whose underlying data flow graphs are based on finite projective-geometry (PG) based balanced bipartite graphs. Many of these applications of finite projective geometry are actively being researched upon, especially in coding theory. Almost all these applications need large bipartite graphs, whose nodes represent parallel computations. To reduce its implementation cost, reducing amount of system/hardware resources during design is an important engineering objective. In this context, we present a scheme to reduce resource utilization while designing systems modeled using PG-based graphs. In such systems, the number of processing units is equal to the number of vertices, each performing an atomic computation. We present a novel way of partitioning the vertex set assigned to various atomic computations, into blocks. Each block of partition is then assigned to a processing unit. A processing unit performs the computations corresponding to the vertices in the block assigned to it in a sequential fashion, thus creating the effect of folding the overall computation. The symmetric properties of projective space lattices enable us to develop a conflict-free communication schedule. We employed the technique of coset decomposition of a finite field for partitioning. The folding scheme achieves the best possible throughput, in lack of any overhead of shuffling data across memories while scheduling another computation on the same processing unit. We first provide a scheme for a finite projective space of dimension five, and the corresponding schedules. This specific scheme is then generalized for arbitrary finite projective spaces. Both the folding schemes have been verified by both simulation as well as hardware prototyping. For example, a semi-parallel decoder architecture for a new class of expander codes was designed and implemented using this scheme, with potential deployment in DVD-R/CD-ROM drives.

Chaotic Image Encryption with Random Shuffling of Data

Security of valuable multimedia contents such as images in personal photograph albums, electronic publishing, frames of multicast video conference can be achieved by image encryption. Secure transmission of these contents is required to be rapid, efficient and practical. Hence, image encryption process must be chosen not only to satisfy the security goals but also to fulfill these requirements. Due to the inadequacy and inefficiency of conventional text based information encryption methods, researchers have proposed several encryption schemes. Many of them are based on chaotic algorithms. Recently, the studies are concentrated on some weaknesses of chaotic algorithms and most of the presented solutions came up with complex structured chaotic maps. In this paper, we present a self-diagonal shuffler mechanism embedded to one dimensional chaotic encryption system to overcome its leak points while keeping simplicity and efficiency properties.

OPTIMAL FOLDING OF DATA FLOW GRAPHS BASED ON FINITE PROJECTIVE GEOMETRY USING VECTOR SPACE PARTITIONING

A number of computations exist, especially in area of error-control coding and matrix computations, whose underlying data flow graphs are based on finite projective-geometry (PG)-based balanced bipartite graphs. Many of these applications of finite projective geometry are actively being researched upon, especially in coding theory. Almost all these applications need large bipartite graphs, whose nodes represent parallel computations. To reduce its implementation cost, reducing amount of system/hardware resources during design is an important engineering objective. In this context, we present a scheme to reduce resource utilization while designing systems modeled using PG-based graphs. In such systems, the number of processing units is equal to the number of vertices, each performing an atomic computation. We present a novel way of partitioning the vertex set assigned to various atomic computations, into blocks. Each block of partition is then assigned to a processing unit. A processing unit performs the computations corresponding to the vertices in the block assigned to it in a sequential fashion, thus creating the effect of folding the overall computation. The symmetric properties of projective space lattices enable us to develop a conflict-free communication schedule. We employed the technique of coset decomposition of a finite field for partitioning. The folding scheme achieves the best possible throughput, in lack of any overhead of shuffling data across memories while scheduling another computation on the same processing unit. We first provide a scheme for a finite projective space of dimension five, and the corresponding schedules. This specific scheme is then generalized for arbitrary finite projective spaces. Both the folding schemes have been verified by both simulation as well as hardware prototyping. For example, a semi-parallel decoder architecture for a new class of expander codes was designed and implemented using this scheme, with potential deployment in DVD-R/CD-ROM drives.

Pruned Bit-Reversal Permutations: Mathematical Characterization, Fast Algorithms and Architectures

A mathematical characterization of serially pruned permutations (SPPs) employed in variable-length permuters and their associated fast pruning algorithms and architectures are proposed. Permuters are used in many signal processing systems for shuffling data and in communication systems as an adjunct to coding for error correction. Typically, only a small set of discrete permuter lengths are supported. Serial pruning is a simple technique to alter the length of a permutation to support a wider range of lengths, but results in a serial processing bottleneck. In this paper, parallelizing SPPs is formulated in terms of recursively computing sums involving integer floor functions using integer operations, in a fashion analogous to evaluating Dedekind sums. A mathematical treatment for bit-reversal permutations (BRPs) is presented, and closed-form expressions for BRP statistics including descents/ascents, major index, excedances/descedances, inversions, and serial correlations are derived. It is shown that BRP sequences have weak correlation properties. Moreover, a new statistic called permutation inliers that characterizes the pruning gap of pruned interleavers is proposed. Using this statistic, a recursive algorithm that computes the minimum inliers count of a pruned BR interleaver (PBRI) in logarithmic time is presented. This algorithm enables parallelizing a serial PBRI algorithm by any desired parallelism factor by computing the pruning gap in lookahead rather than a serial fashion, resulting in significant reduction in interleaving latency and memory overhead. Extensions to 2-D block and stream interleavers, as well as applications to pruned fast Fourier transforms and LTE turbo interleavers, are also presented. Moreover, hardware-efficient architectures for the proposed algorithms are developed. Simulation results of interleavers employed in modern communication standards demonstrate three to four orders of magnitude improvement in interleaving time compared to existing approaches.

Multifractal analysis of ground–level ozone concentrations at urban, suburban and rural background monitoring sites in Southwestern Iberian Peninsula

This paper seeks to enhance understanding of the data distribution of ground–level ozone time series by analysing their multifractal spectra. Emphasis is placed on the suitability of the box–counting and moments methods for characterizing scaling properties of ozone concentration, enabling us to describe similarities and differences inferred from the multifractal spectra by analysing various types of monitoring stations (urban, suburban and rural monitoring sites) under identical atmospheric conditions. It is herein demonstrated that that multifractal features are to a considerable extent similar for each type of monitoring station under warm atmospheric conditions and high solar radiation, owing to the fact that these weather characteristics homogenize the scaling behaviour of ozone. On the contrary, location and chemical precursors play a more prominent role under low temperatures and solar radiation, highlighting differences among multifractal features of ozone concentrations in the monitoring sites. At rural stations, the absence of anthropogenic emissions promotes less variability in ozone data and homogenization of multifractal behaviour throughout the year. Furthermore, a data shuffling procedure was performed in order to analyse changes occurring in multifractal spectra as time series are subjected to varying degrees of data position disturbance. Results indicate that multifractal analysis is a useful tool for describing the temporal scaling behaviour of ozone time series at different monitoring sites which refines results that have been traditionally provided by statistical analyses on ozone pollution.

A design methodology for optimally folded, pipelined architectures in VLSI applications using projective space lattices

Semi-parallel, or folded, VLSI architectures are used whenever hardware resources need to be saved. Most recent applications that are based on Projective Geometry (PG) based balanced bipartite graphs also fall in this category. Many of these applications are actively being researched upon, especially in the area of coding theory and matrix computations. Almost all these applications need bipartite graphs of the order of tens of thousands in practice, whose nodes represent parallel processing. To reduce its implementation cost, reducing amount of hardware resources is an important engineering objective. In this paper, we provide a high-level, top-down design methodology to design optimal semi-parallel architectures for applications, whose Data Flow Graph (DFG) is based on PG bipartite graph. Unlike many other folding schemes, the topology of connections between physical elements nodesdoes not change at runtime in this methodology. Hence the folding scheme achieves the best possible throughput, in lack of any overhead of shuffling data across memories while scheduling another computation on the same processing unit. Another advantage is the ease of implementation. To lessen the throughput loss due to folding, we also incorporate a multi-tier pipelining strategy in the design methodology. A C++-based synthesis tool has been developed and tested for automatic generation of RTL models, and is publicly available. A specific high-performance design of a low-density parity check (LDPC) decoder based on this methodology was worked out in past, and has been patent pending.

Two Noise Addition Methods For Privacy-Preserving Data Mining

In the last decade, more and more researches have focused on privacy-preserving data min ing(PPDM). The previous work can be div ided into two categories: data modification and data encryption. Data encryption is not used as widely as data modificat ion because of its high cost on computing and communications. Data perturbation, including additive noise, mu lt iplicative noise, matrix mu ltip lication, data swapping, data shuffling, k-anonymization, Blocking, is an important technology in data modification method. PPDM has two targets: privacy and accuracy, and they are often at odds with each other. Th is paper begins with a proposal of two new noise addition methods for perturbing the original data, followed by a discussion of how they meet the two targets. Experiments show that the methods given in this paper have higher accuracy than existing ones under the same condition of privacy strength.

Deriving to an Optimum Policy for Designing the Operating Parameters of Mahshahr Gas Turbine Power Plant Using a Self Learning Pareto Strategy

In the last decades, analyzing and optimizing the power plants based on thermodynamic laws and intelligent control techniques absorb an incremental interest of researchers. This is because deriving the efficient operating parameters for designing and optimizing the performance of power plants will lead to an acceptable investment and avoiding from discarding the energy. However, there are a few areas of application of mathematical optimization method. Optimizing the governing equations and designing parameters of power plants simultaneously leads to a multi-objective problem in industry. Some of these objectives are nonlinear, non-convex and multi-modal with different type of real life engineering constraints. In this paper a new method called Synchronous Parallel Shuffling Self Organized Pareto Strategy Algorithm (SPSSOPSA) is presented which synthesized evolutionary computing, swarm intelligence techniques and Time Adaptive Self Organizing Map(TASOM) simultaneously incorporating with a data shuffling behavior. Thereafter it will be applied to verifying the optimum decision making for parameter designing of Mahshahr power plant that produced about 117MW electricity, sited in Iran, as a multi-objective and multi-modal problem. The results show the deep relation of the unit cost on the change of the operating parameters. Key words : Economic optimizing; Exergetic optimizing; Work output maximization; Evolutionary algorithm; Self organized map; Power plant

The Effectiveness of Data Shuffling for Privacy-Preserving Data Mining Applications

AbstractPreserving the confidentiality of sensitive data, while permitting knowledge discovery, is an important goal in privacy-preserving data mining. This paper investigates the effectiveness of data shuffling for classification tree and regression analysis. We compare the effectiveness of data shuffling to the tree based data perturbation method which was developed specifically for the purpose of data mining. Results suggest that data shuffling provides the higher levels of data security and more effectively preserves data mining knowledge than tree based data perturbation method.

IMapReduce: A Distributed Computing Framework for Iterative Computation

Iterative computation is pervasive in many applications such as data mining, web ranking, graph analysis, online social network analysis, and so on. These iterative applications typically involve massive data sets containing millions or billions of data records. This poses demand of distributed computing frameworks for processing massive data sets on a cluster of machines. MapReduce is an example of such a framework. However, MapReduce lacks built-in support for iterative process that requires to parse data sets iteratively. Besides specifying MapReduce jobs, users have to write a driver program that submits a series of jobs and performs convergence testing at the client. This paper presents iMapReduce, a distributed framework that supports iterative processing. iMapReduce allows users to specify the iterative computation with the separated map and reduce functions, and provides the support of automatic iterative processing within a single job. More importantly, iMapReduce significantly improves the performance of iterative implementations by (1) reducing the overhead of creating new MapReduce jobs repeatedly, (2) eliminating the shuffling of static data, and (3) allowing asynchronous execution of map tasks. We implement an iMapReduce prototype based on Apache Hadoop, and show that iMapReduce can achieve up to 5 times speedup over Hadoop for implementing iterative algorithms.

Asymmetric properties of long-term and total heart rate variability.

We report on two new physiological phenomena: the long-term and total heart rate asymmetry, which describe a significantly larger contribution of heart rate accelerations to long-term and total heart rate variability. In addition to the existing pair of indices, {text {SD1}}_{rm d}, {text {SD1}}_{rm a}, which are based on partitioning short-term variance, we introduce two other pairs of descriptors based on partitioning long-term ({text {SD2}}_{rm d}, {text {SD2}}_{rm a}) and total ( {text {SDNN}}_{rm d}, {text {SDNN}}_{rm a}) heart rate variability. The new asymmetric descriptors are used to analyze RR intervals time series derived from the 30-min ECG recordings of 241 healthy subjects resting in supine position. It is shown that both new types of asymmetry are present in 76% of the subjects. The new phenomena reported here are real physiological findings rather than artifacts of the method since they vanish after data shuffling.

Practical Oblivious Outsourced Storage

In this article we introduce a technique, guaranteeing access pattern privacy against a computationally bounded adversary, in outsourced data storage, with communication and computation overheads orders of magnitude better than existing approaches. In the presence of a small amount of temporary storage (enough to store O (√ n log n ) items and IDs, where n is the number of items in the database), we can achieve access pattern privacy with computational complexity of less than O (log 2 n ) per query (as compared to, for instance, O (log 4 n ) for existing approaches). We achieve these novel results by applying new insights based on probabilistic analyses of data shuffling algorithms to Oblivious RAM, allowing us to significantly improve its asymptotic complexity. This results in a protocol crossing the boundary between theory and practice and becoming generally applicable for access pattern privacy. We show that on off-the-shelf hardware, large data sets can be queried obliviously orders of magnitude faster than in existing work.

Multifractal analysis of China's agricultural commodity futures markets

Abstract We investigate the multifractal characteristics of the volatility time series from China's agricultural commodity futures markets, using Multifractal Detrended Fluctuation Analysis and multifractal spectrum analysis. We find that prominent multifractal features exit in China's major agricultural commodity futures markets, including the Hard Winter Wheat (HW) futures, the Strong Gluten Wheat (SG) futures, Soy Bean (SB) futures and corn futures. Furthermore, the multifractality strength and multifractal spectrum width of HW futures are both bigger than that of SG, SB and corn futures, implying that the market risk for HW futures might be the strongest among all the four futures contracts. Finally, comparing empirical results of shuffling and surrogate data, we also find that nonlinear temporal correlations instead of non-Gaussian distribution constitute the major contributions in the formation of multifractal features in these four agricultural commodity futures markets.

Maintaining tail dependence in data shuffling using [formula omitted] copula

Data shuffling is a recently proposed technique for masking numerical data where the confidential values are shuffled between records while maintaining all monotonic relationships between the variables in the data set. Data shuffling is based on the multivariate normal copula which assumes that there is no tail dependence in the data set. In many practical situations, however, tail dependence plays a crucial role in decision making. Hence, it is desirable that the data masking procedure be capable of preserving tail dependence when present. In this study, we provide a new data shuffling approach based on t copulas that is capable of maintaining tail dependence in the masked data in a large number of applications.

Intraplate seismicity in Canada: a graph theoretic approach to data analysis and interpretation

Abstract. Intraplate seismicity occurs in central and northern Canada, but the underlying origin and dynamics remain poorly understood. Here, we apply a graph theoretic approach to characterize the statistical structure of spatiotemporal clustering exhibited by intraplate seismicity, a direct consequence of the underlying nonlinear dynamics. Using a recently proposed definition of "recurrences" based on record breaking processes (Davidsen et al., 2006, 2008), we have constructed directed graphs using catalogue data for three selected regions (Region 1: 45°−48° N/74°−80° W; Region 2: 51°−55° N/77°−83° W; and Region 3: 56°−70° N/65°−95° W), with attributes drawn from the location, origin time and the magnitude of the events. Based on comparisons with a null model derived from Poisson distribution or Monte Carlo shuffling of the catalogue data, our results provide strong evidence in support of spatiotemporal correlations of seismicity in all three regions considered. Similar evidence for spatiotemporal clustering has been documented using seismicity catalogues for southern California, suggesting possible similarities in underlying earthquake dynamics of both regions despite huge differences in the variability of seismic activity.

Multifractal features of spot rates in the Liquid Petroleum Gas shipping market

We investigate for the first time the spot rate dynamics of Very Large Gas Carriers (VLGCs) by means of multifractal detrended fluctuation analysis (MF-DFA) and rescaled range (R/S) analysis. Both non-parametric methods allow for a rigorous statistical analysis of the freight process by detecting correlation, scaling and fluctuation behavior regardless of nonlinearity issues. By applying different data-frequencies and a temporal framework, the Hurst exponents indicate that freight rates exhibit trend-reinforcement and persistence subject to limited time-dependency and controlled volatility. The found long-range dependence corroborates that a predictive freight model can be built undermining the efficient market hypothesis. Memory effects seem to each time build up until they are interrupted by seasonal transitions, stochastic events or cycles which all spark a sudden loss in correlations or increase in nonlinearities. The surrogate and shuffling data procedures demonstrate that, dependent on the data-frequency used, memory effects and fat-tail distributions should be contained differently in freight rate models.

MacroSS

SIMD (Single Instruction, Multiple Data) engines are an essential part of the processors in various computing markets, from servers to the embedded domain. Although SIMD-enabled architectures have the capability of boosting the performance of many application domains by exploiting data-level parallelism, it is very challenging for compilers and also programmers to identify and transform parts of a program that will benefit from a particular SIMD engine. The focus of this paper is on the problem of SIMDization for the growing application domain of streaming. Streaming applications are an ideal solution for targeting multi-core architectures, such as shared/distributed memory systems, tiled architectures, and single-core systems. Since these architectures, in most cases, provide SIMD acceleration units as well, it is highly beneficial to generate SIMD code from streaming programs. Specifically, we introduce MacroSS, which is capable of performing macro-SIMDization on high-level streaming graphs. Macro-SIMDization uses high-level information such as execution rates of actors and communication patterns between them to transform the graph structure, vectorize actors of a streaming program, and generate intermediate code. We also propose low-overhead architectural modifications that accelerate shuffling of data elements between the scalar and vectorized parts of a streaming program. Our experiments show that MacroSS is capable of generating code that, on average, outperforms scalar code compiled with the current state-of-art auto-vectorizing compilers by 54%. Using the low-overhead data shuffling hardware, performance is improved by an additional 8% with less than 1% area overhead.

MacroSS

SIMD (Single Instruction, Multiple Data) engines are an essential part of the processors in various computing markets, from servers to the embedded domain. Although SIMD-enabled architectures have the capability of boosting the performance of many application domains by exploiting data-level parallelism, it is very challenging for compilers and also programmers to identify and transform parts of a program that will benefit from a particular SIMD engine. The focus of this paper is on the problem of SIMDization for the growing application domain of streaming. Streaming applications are an ideal solution for targeting multi-core architectures, such as shared/distributed memory systems, tiled architectures, and single-core systems. Since these architectures, in most cases, provide SIMD acceleration units as well, it is highly beneficial to generate SIMD code from streaming programs. Specifically, we introduce MacroSS, which is capable of performing macro-SIMDization on high-level streaming graphs. Macro-SIMDization uses high-level information such as execution rates of actors and communication patterns between them to transform the graph structure, vectorize actors of a streaming program, and generate intermediate code. We also propose low-overhead architectural modifications that accelerate shuffling of data elements between the scalar and vectorized parts of a streaming program. Our experiments show that MacroSS is capable of generating code that, on average, outperforms scalar code compiled with the current state-of-art auto-vectorizing compilers by 54%. Using the low-overhead data shuffling hardware, performance is improved by an additional 8% with less than 1% area overhead.

Data shuffling and statistical analysis on microarray data for gene selection: a comparative study on filtering methods

Computational analysis have been broadly used to discover disease-relevant genes from microarray expression data. In this paper, we extend a traditional statistical metric to a second level to measure gene-disease relations, testing such relation whether can be replicated by randomly shuffling the gene expression data. The traditional metric can be considered as a first-level metric; the relevance of each gene is then verified through the second-level significance testing based on the first-level metric calculated on the original data and shuffled data. We show that this method can also produce high classification performance, compared with other filter-based methods.