Large Processor Research Articles

Special Issue: Advances in High‐Performance Computing and Communications

Special Issue: Advances in High‐Performance Computing and Communications

Workload performance characterization of DARPA HPCS benchmarks

AbstractIt is critical to understand the workload characteristics and resource usage patterns of available applications to guide the design and development of hardware and software stacks of future machines. In this article, we analyze the workload performance characteristics of three large‐scale DARPA HPCS benchmarks: Hybrid Coordinate Ocean Model, Parallel Ocean Program, and Lattice Boltzemann Magneto‐Hydrodynamics Code while executing on IBM Power5+ processor machines. Our analysis is focused on the CPU/memory performance using Cycles Per Instruction (CPI) model and multiprocess communication performance using MPI traces. For each benchmark, we provide a high‐level performance analysis followed by the hotspot analysis for selected input parameters. Then we present a detailed workload performance characterization using CPI model with data from a unique set of performance counters available on the Power5+ processor system. From communication performance analysis, we describe the sources of load imbalances in the applications and identify the potential impediments to the scalability of the applications under large processor counts. We identify several sources of performance problems that are potential bottlenecks and discuss methods to ameliorate them. We also present a comparative analysis of these benchmarks to summarize the similarities and differences in their performance characteristics. Copyright © 2009 John Wiley & Sons, Ltd.

BlueGene/L applications: Parallelism On a Massive Scale

BlueGene/L (BG/L), developed through a partnership between IBM and Lawrence Livermore National Laboratory (LLNL), is currently the world's largest system both in terms of scale, with 131,072 processors, and absolute performance, with a peak rate of 367 Tflop/s. BG/L has led the last four Top500 lists with a Linpack rate of 280.6 Tflop/s for the full machine installed at LLNL and is expected to remain the fastest computer in the next few editions. However, the real value of a machine such as BG/L derives from the scientific breakthroughs that real applications can produce by successfully using its unprecedented scale and computational power. In this paper, we describe our experiences with eight large scale applications on BG/ L from several application domains, ranging from molecular dynamics to dislocation dynamics and turbulence simulations to searches in semantic graphs. We also discuss the challenges we faced when scaling these codes and present several successful optimization techniques. All applications show excellent scaling behavior, even at very large processor counts, with one code even achieving a sustained performance of more than 100 Tflop/s, clearly demonstrating the real success of the BG/L design.

Parallel simulation of particle suspensions with the lattice Boltzmann method

A description of the steps taken to produce a massively parallel code for particle suspension problems using the lattice Boltzmann method is presented. A number of benchmarks based on a binary fluid lattice Boltzmann model are used to assess the performance of the code in terms of the computational overhead required for the particle problem compared with the fluid-only problem, and for the scaling of the code to large processor numbers. On the Blue Gene/L architecture, the additional computational cost of particle suspensions of up to 40% solid volume fraction (here over a million particles) is negligible compared with the fluid-only code.

The uniform estimation of the M/G/1 processor sharing response time distribution

AbstractThis contribution avoids frequently used transform and inversion problems by a decomposition of the transition state model in the original domain. First, the relevant differential‐recurrence backward equation system valid for the M/M/1 processor sharing (PS) is converted to a sum of a service component and an arrival–departure component. The first one yields a pure differential equation system, which can be solved exactly and proves to be uniformly valid for M/G/1 PS systems. The second component vanishes for small and large processor utilisations but contributes to shorter response times otherwise. Thus, the PS service component defines an estimate of the true response time distribution. Second, the higher order moments of the estimates form simple upper bounds of their true much more complicated counterparts where the first moments agree. Partly simulative case studies subject to M, D, H2, heavy‐tailed service time distributions, and PS with permanent customers confirm the new estimation. Third, reference values of the arrival–departure component provide an approximate solution of the differential‐recurrence equation system. This significantly improves the accuracy of the estimates for moderate utilisations. Finally it is shown that the remaining inaccuracies of the estimates are far below expectable forecast errors for high economic loads and otherwise the approximation significantly supports PS system engineering for moderate loads too. Copyright © 2007 John Wiley & Sons, Ltd.

A fine grained parallel smooth particle mesh Ewald algorithm for biophysical simulation studies: Application to the 6-D torus QCDOC supercomputer

In order to model complex heterogeneous biophysical macrostructures with non-trivial charge distributions such as globular proteins in water, it is important to evaluate the long range forces present in these systems accurately and efficiently. The Smooth Particle Mesh Ewald summation technique (SPME) is commonly used to determine the long range part of electrostatic energy in large scale molecular simulations. While the SPME technique does not give rise to a performance bottleneck on a single processor, current implementations of SPME on massively parallel, supercomputers become problematic at large processor numbers, limiting the time and length scales that can be reached. Here, a synergistic investigation involving method improvement, parallel programming and novel architectures is employed to address this difficulty. A relatively simple modification of the SPME technique is described which gives rise to both improved accuracy and efficiency on both massively parallel and scalar computing platforms. Our fine grained parallel implementation of the modified SPME method for the novel QCDOC supercomputer with its 6D-torus architecture is then given. Numerical tests of algorithm performance on up to 1024 processors of the QCDOC machine at BNL are presented for two systems of interest, a β-hairpin solvated in explicit water, a system which consists of 1142 water molecules and a 20 residue protein for a total of 3579 atoms, and the HIV-1 protease solvated in explicit water, a system which consists of 9331 water molecules and a 198 residue protein for a total of 29508 atoms.

Comprehensive multivariate extrapolation modeling of multiprocessor cache miss rates

Cache miss rates are an important subset of system model inputs. Cache miss rate models are used for broad design space exploration in which many cache configurations cannot be simulated directly due to limitations of trace collection setups or available resources. Often it is not practical to simulate large caches. Large processor counts and consequent potentially high degree of cache sharing are frequently not reproducible on small existing systems. In this article, we present an approach to building multivariate regression models for predicting cache miss rates beyond the range of collectible data. The extrapolation model attempts to accurately estimate the high-level trend of the existing data, which can be extended in a natural way. We extend previous work by its applicability to multiple miss rate components and its ability to model a wide range of cache parameters, including size, line size, associativity and sharing. The stability of extrapolation is recognized to be a crucial requirement. The proposed extrapolation model is shown to be stable to small data perturbations that may be introduced during data collection.We show the effectiveness of the technique by applying it to two commercial workloads. The wide design space contains configurations that are much larger than those for which miss rate data were available. The fitted data match the simulation data very well. The various curves show how a miss rate model is useful for not only estimating the performance of specific configurations, but also for providing insight into miss rate trends.

Computational modeling of multicellular constructs with the material point method

Computational modeling of the mechanics of cells and multicellular constructs with standard numerical discretization techniques such as the finite element (FE) method is complicated by the complex geometry, material properties and boundary conditions that are associated with such systems. The objectives of this research were to apply the material point method (MPM), a meshless method, to the modeling of vascularized constructs by adapting the algorithm to accurately handle quasi-static, large deformation mechanics, and to apply the modified MPM algorithm to large-scale simulations using a discretization that was obtained directly from volumetric confocal image data. The standard implicit time integration algorithm for MPM was modified to allow the background computational grid to remain fixed with respect to the spatial distribution of material points during the analysis. This algorithm was used to simulate the 3D mechanics of a vascularized scaffold under tension, consisting of growing microvascular fragments embedded in a collagen gel, by discretizing the construct with over 13.6 million material points. Baseline 3D simulations demonstrated that the modified MPM algorithm was both more accurate and more robust than the standard MPM algorithm. Scaling studies demonstrated the ability of the parallel code to scale to 200 processors. Optimal discretization was established for the simulations of the mechanics of vascularized scaffolds by examining stress distributions and reaction forces. Sensitivity studies demonstrated that the reaction force during simulated extension was highly sensitive to the modulus of the microvessels, despite the fact that they comprised only 10.4% of the volume of the total sample. In contrast, the reaction force was relatively insensitive to the effective Poisson's ratio of the entire sample. These results suggest that the MPM simulations could form the basis for estimating the modulus of the embedded microvessels through a parameter estimation scheme. Because of the generality and robustness of the modified MPM algorithm, the relative ease of generating spatial discretizations from volumetric image data, and the ability of the parallel computational implementation to scale to large processor counts, it is anticipated that this modeling approach may be extended to many other applications, including the analysis of other multicellular constructs and investigations of cell mechanics.

Private and public milk standards in Argentina and Brazil

This article focuses on the development of public and private standards for milk as an input to the dairy processing sector in Argentina and Brazil. From 1950 to 1990, the dominant trend was the development of public standards for basic safety and hygiene of milk. These induced the incipient modernization of dairy farming, post harvest, and processing technologies, even though implementation of the regulations was only partial. The dairy sectors were liberalized and privatized circa 1990. This spurred the rapid rise of private standards set by large processors over the 1990s and into the 2000s. These standards are much more stringent than public standards, and induced rapid concentration at the processing and farm levels due to stiff investment requirements. In Brazil, private standards were set mainly to drive down costs in the supply chain in order to reduce the consumer prices in the mass commodities market. In Argentina, private standards were used to develop quality differentiation in the product market. In both countries, in the 2000s a debate has emerged on new roles and approaches for public standards.

Large‐scale density‐driven flow simulations using parallel unstructured Grid adaptation and local multigrid methods

AbstractAdvanced parallel applications based on the message‐passing paradigm are difficult to design and implement, especially when solution adaptive techniques are used and three‐dimensional problems on complex geometries are faced, which yield the use of unstructured Grids. We present the building blocks for a parallel‐adaptive scheme for the solution of time‐dependent and nonlinear partial differential equations. To minimize computational requirements, h‐adaptivity is introduced via parallel, local Grid adaptation. Novel techniques to avoid hanging nodes are introduced, these assure conforming meshes of hybrid element type in three space dimensions. As a core of the adaptive scheme, local multigrid methods are used to solve the arising linear systems rapidly in parallel. Dynamic Grid changes from h‐adaptivity lead to load imbalance during run time, therefore dynamic load balancing and migration is performed to exploit the aggregated performance of large processor sets efficiently. Real‐world calculations arising from density‐driven flow problems in porous media are performed using the presented parallel‐adaptive solution strategy. The computations are analyzed with regard to speedup. Timings of Grid adaptation, dynamic load balancing/migration and numerical solution scheme show that large‐scale runs on 512 processors gain an overall parallel, numerical speedup of up to 278. A further reduction of the element count by h‐adaptivity by a factor of up to 195 shows the enormous capabilities of the presented parallel‐adaptive multigrid based solution scheme. Copyright © 2005 John Wiley & Sons, Ltd.

Scalable system software: a component-based approach

The growth in computing resources at scientific computing centers has created new challenges for system software. These multi-teraflop systems often exceed the capabilities of the system software and require new approaches to accommodate these large processor counts. The costs associated with development and maintenance of this software are also significant impediments, which are compounded by a lack of interoperability because of site-specific enhancements. The Scalable System Software project seeks to address these issues through a component based approach to system software development. An overview of this design and the benefits of such an approach will be discussed in this paper.

A Ranked Order Filter Implementation for Parallel Analog Processing

Order statistic filtering, the generalization of which is ranked order filtering, is needed for many image-processing functions including median filtering and mathematical morphology. Combining order statistic functionality with the parallel operation and local connectivity of array processing approaches such as the cellular nonlinear network model, has the potential for very high performance in image processing. This paper examines the implementation of programmable ranked order extraction with a very compact hardware realization of an analog current-mode ranked order filter. The considerable savings in the required circuit area, compared to other circuits, make it possible to use the structure as a building block in a massively parallel signal processing array. The operation of the circuit is analyzed in detail with the help of simulations and measurement results obtained from a test chip manufactured in a 0.18-/spl mu/m standard digital CMOS technology are also presented. The simulations and measurement results verify the correct operation of the circuit and show that it is very suitable for inclusion in every cell of a large parallel processor array. This makes many grayscale processing functions available with truly parallel operation and therefore very high performance.

A Fully Parallel CMOS Analog Median Filter

A fully integrated CMOS implementation of a continuous-time analog median filter is presented. The median filter uses two compact analog circuits as building blocks to implement the variable delay and median detection. Median detectors are based on current saturating transconductance comparators, while the time delay is implemented using first-order all-pass filters. Both circuits allow modular expansion for the implementation of large median filter array processors. Based on these blocks, a new fast technique for parallel image processing is presented. It is shown that an image of 91/spl times/80 pixels can be processed in less than 8 /spl mu/s using an array of median filter cells. Experimental results of a test chip prototype in 2-/spl mu/m CMOS MOSIS technology are presented.

A load balancing framework for adaptive and asynchronous applications

We describe the design of a flexible load balancing framework and runtime software system for supporting the development of adaptive applications on distributed-memory parallel computers. The runtime system supports a global namespace, transparent object migration, automatic message forwarding and routing, and automatic load balancing. These features can be used at the discretion of the application developer in order to simplify program development and to eliminate complex bookkeeping associated with mobile data objects. An evaluation of this system in the context of a three-dimensional tetrahedral advancing front parallel mesh generator shows that overall runtime improvements of 15 percent compared to common stop-and-repartition load balancing methods, 30 percent compared to explicit intrusive load balancing methods, and 42 percent compared to no load balancing are possible on large processor configurations. At the same time, the overheads attributable to the runtime system are a fraction of 1 percent of the total runtime. The parallel advancing front method is a coarse-grained and highly adaptive application and therefore exercises all of the features of the runtime system.

A‐scalability and an integrated computational technology and framework for non‐linear structural dynamics. Part 2: Implementation aspects and parallel performance results

AbstractAn integrated framework and computational technology is described that addresses the issues to foster absolute scalability (A‐scalability) of the entire transient duration of the simulations of implicit non‐linear structural dynamics of large scale practical applications on a large number of parallel processors. Whereas the theoretical developments and parallel formulations were presented in Part 1, the implementation, validation and parallel performance assessments and results are presented here in Part 2 of the paper. Relatively simple numerical examples involving large deformation and elastic and elastoplastic non‐linear dynamic behaviour are first presented via the proposed framework for demonstrating the comparative accuracy of methods in comparison to available experimental results and/or results available in the literature. For practical geometrically complex meshes, the A‐scalability of non‐linear implicit dynamic computations is then illustrated by employing scalable optimal dissipative zero‐order displacement and velocity overshoot behaviour time operators which are a subset of the generalized framework in conjunction with numerically scalable spatial domain decomposition methods and scalable graph partitioning techniques. The constant run times of the entire simulation of ‘fixed‐memory‐use‐per‐processor’ scaling of complex finite element mesh geometries is demonstrated for large scale problems and large processor counts on at least 1024 processors. Copyright © 2003 John Wiley & Sons, Ltd.

A‐scalability and an integrated computational technology and framework for non‐linear structural dynamics. Part 1: Theoretical developments and parallel formulations

AbstractFor large‐scale problems and large processor counts, the accuracy and efficiency with reduced solution times and attaining optimal parallel scalability of the entire transient duration of the simulation for general non‐linear structural dynamics problems poses many computational challenges. For transient analysis, explicit time operators readily inherit algorithmic scalability and consequently enable parallel scalability. However, the key issues concerning parallel simulations via implicit time operators within the framework and encompassing the class of linear multistep methods include the totality of the following considerations to foster the proposed notion of A‐scalability: (a) selection of robust scalable optimal time discretized operators that foster stabilized non‐linear dynamic implicit computations both in terms of convergence and the number of non‐linear iterations for completion of large‐scale analysis of the highly non‐linear dynamic responses, (b) selecting an appropriate scalable spatial domain decomposition method for solving the resulting linearized system of equations during the implicit phase of the non‐linear computations, (c) scalable implementation models and solver technology for the interface and coarse problems for attaining parallel scalability of the computations, and (d) scalable parallel graph partitioning techniques. These latter issues related to parallel implicit formulations are of interest and focus in this paper. The former involving parallel explicit formulations are also a natural subset of the present framework and have been addressed previously in Reference 1 (Advances in Engineering Software 2000; 31: 639–647). In the present context, of the key issues, although a particular aspect or a solver as related to the spatial domain decomposition may be designed to be numerically scalable, the totality of the aforementioned issues simultaneously play an important and integral role to attain A‐scalability of the parallel formulations for the entire transient duration of the simulation and is desirable for transient problems. As such, the theoretical developments of the parallel formulations are first detailed in Part 1 of this paper, and the subsequent practical applications and performance results of general non‐linear structural dynamics problems are described in Part 2 of this paper to foster the proposed notion of A‐scalability. Copyright © 2003 John Wiley & Sons, Ltd.

Probabilistic analysis of interconnect coupling noise

Noise simulators and noise avoidance tools are playing an increasingly critical role in the design of deep submicron circuits. However, noise estimates produced by these simulators are often very pessimistic. For large, high-performance industrial ICs, which can contain hundreds of thousands of nets, the worst case estimates of the noise results in thousands of reported violations, without any information about the likelihood of the possible noise violation. In this paper, we present a probabilistic approach to prioritize the violating nets based on the likelihood of occurrence of the reported noise. We derive an upper bound on the probability that the total noise injected on a given victim net by a specific set of aggressors exceeds a threshold. This is equivalent to a lower bound on the expected number of clock cycles required to realize the noise violation for the first time, i.e., mean time-to-failure. If the probability of a failure in a victim is sufficiently small, it is possible that even during the operation of the part for a number of years, the probability of failure on the net is negligible and the net can be assigned a lower priority for the application of noise avoidance strategies. We demonstrate the utility of this approach through experiments carried out on an large industrial processor design using a state-of-the-art industrial noise analysis tool.

Efficient and Optimal Parallel Algorithms for Cholesky Decomposition

In this dissertation we consider design issues such as data, layout, data, dependency, communication scheduling, and parallel models in order to develop parallel algorithms for Cholesky decomposition. Our intention is to design efficient and portable parallel algorithms for large scale dense linear systems to be run in the distributed memory environment. To consider the issue of portability, we utilize the well known LogP model for designing parallel algorithms and for performing theoretical run time analysis. We developed a contention free communication scheduling for the parallel algorithms in order to reduce the communication cost. To study the impact of data layout on performance, we designed the parallel algorithms based on column, row and block data distribution utilizing the fan-in and fan-out methods. Specifically we introduced panel size for 1D layouts and granularity size for 2D layouts for the task partitioning. To determine the efficiency of the parallel algorithms, we derived lower bound results and theoretical run time performance. The analysis on the parallel algorithms shows that parallel algorithms for 1D data layouts are optimal. However for 2D data layouts, the parallel algorithms are efficient, if not necessarily asymptotically optimal. We then implemented the parallel algorithms using MPI. The experimental results show that fan-in outperforms fan-out for 2D layouts, especially for larger processor size. The performance for the 1D fan-out row-based parallel algorithm was no worse than that of the column-based parallel algorithms. This seems to refute the common notion that row-based parallel algorithms are not, as efficient as column-based parallel algorithms. The experimental results indicated that performance was impacted as panel size and/or granularity size varied. This provides insights that the constant factors for lower bound and run time performance call not be ignored.

Mobilization of state program resources for a steel mill accident.

Radioactive materials in scrap metal are a serious problem for dealers and processors as well as for state radiation protection programs. Most large dealers and processors have monitoring devices that detect radiation sources at their scale house or feed to the furnaces. For example, in Florida since July 1995, a fiscal year average of 41 such alarms has been reported to the Department of Health's Bureau of Radiation Control. The majority of these alarms involve small amounts of radioactive material, usually naturally occurring radioactive material such as 226Ra, contained in luminous dials or scale associated with mining activities in the state. Occasionally, industrial devices that contain radioactive source material are found. The Bureau classifies these alarms as incidents, and inspectors are dispatched to help identify the source and determine the disposition of the material. On 13 July 2001, a steel mill in north Florida reported that a 137Cs source had been melted in their furnace. This paper describes the actions undertaken by the Bureau in assessing the extent of contamination and subsequent cleanup activities undertaken by the mill.

Quality and contractual choice: a transaction cost approach to the polish hog market

This paper analyses the marketing arrangements between Polish hog producers and buyers at a time when high‐quality markets are emerging. Interrelationships between four key factors are investigated: governance structures, prices, grading and investment in quality production. A sample of 200 Polish hog producers is analysed. A multinomial logit analysis indicates that producers' choice between large processors as opposed to traders and local slaughterhouses can be predicted by type of contract, grading and quality. A cluster analysis reveals four distinct groups of farmers according to investment in specific assets, ability to safeguard assets, degree of coordination with buyers, use of grading and written contracts, and extent of bargaining power