Opteron Processors Research Articles

Simulating subsurface flow and transport on ultrascale computers using PFLOTRAN

We describe PFLOTRAN, a recently developed code for modeling multi-phase, multi-component subsurface flow and reactive transport using massively parallel computers. PFLOTRAN is built on top of PETSc, the Portable, Extensible Toolkit for Scientific Computation. Leveraging PETSc has allowed us to develop—with a relatively modest investment in development effort—a code that exhibits excellent performance on the largest-scale supercomputers. Very significant enhancements to the code are planned during our SciDAC-2 project. Here we describe the current state of the code, present an example of its use on Jaguar, the Cray XT3/4 system at Oak Ridge National Laboratory consisting of 11706 dual-core Opteron processor nodes, and briefly outline our future plans for the code.

Just-in-time connectivity for very large neuronal networks

Memory storage remains a limitation when running very large neuronal networks (VLNN). With number of neurons n, connectivity storage grows as n2. With connectivity densities of 0.1–10%, 1e6 neurons will require 1e9-1e11 synapses. A single connection requires at least an associated weight and delay as well as additional pointers or offsets to store the connectivity matrix. Conservatively, this will require 10 bytes (e.g., 2 floats and 2 chars) which will then bring the total synaptic memory load to 10 GB-1 TB. The former value may be barely executable on a single large machine. We have exploited an algorithmic space-time trade-off to build large event-driven artificial-cell simulations in the NEURON simulator by utilizing just-in-time connections (JITCONs) that are generated at the time of presynaptic cell spiking. JITCON utilizes a presynaptic-cell-specific random-number-generator seed based on presynaptic-cell serial number that permits it to generate a list of postsynaptic cell targets on the fly, and seeds based on a multiple of presynaptic-cell and postsynaptic-cell serial numbers for generating weights. We have utilized the JITCON algorithm to readily run simulations of >2e6 neurons. These simulations include a moderate level of cellular detail with AMPA, NMDA, GABAA and GABAB synapses, as well as multiple intrinsic properties such as bursting, depolarization blockade and an afterhyperpolarizing channel. Note that these are event-driven simulations and therefore do not utilize continuously integrated compartmental neurons. Since these simulations are event-driven, there is no overhead unless there is activity: simulation time varies widely depending on the level of network activity. An active network of 1.2e5 cells with >8.9e6 synapses, generating >1.1e7 spikes in 1 s simulation time, took 32.3 minutes to run on a 2.4 GHz AMD Opteron processor. Large, active simulations still develop space problems due to the need for a variable-size queue to accommodate varying delivery delays. This limitation is minimized by restrictions on the range and variability of permitted delays. We have begun to explore algorithms that permit a nuanced approach to the space-time trade-off. We permit individual presynaptic cells to store their list of postsynaptic targets in a compressed format. This additional storage can be turned on or off on a per-cell basis. We will explore making this storage dynamic so that a cell can maintain its connectivity list during a period of high activity and then return the memory when its activity is reduced. An additional direction for future development will be the incorporation of an entire encapsulated artificial-cell network as an independent piece of a compiled code (a mod file in NEURON). Such a network module could then be plugged in to other network modules or to a more detailed network that used compartmental models or compartmental/artificial cell hybrids, running in the main NEURON simulator. Running such simulations on parallel supercomputers will permit execution of very-VLNNs of order 100 million neurons.

TU‐EE‐A1‐01: Validation of a Prototype Deterministic Solver for Photon Beam Dose Calculations On Acquired CT Data in the Presence of Narrow Beams and Heterogeneities

Purpose: To evaluate a deterministic method for solving the neutral and charged particle transport equations in cases where electron disequilibrium is significant. Methods and Materials: A prototype deterministic solver, Acuros®, has been developed and validated for photon beam radiotherapy. Acuros is based on the Attila® radiation transport code, which solves the differential form of the linear Boltzmann transport equation for neutral particles, and the linear Boltzmann‐Fokker‐Plank transport equation for charged particles. The angular and energy dependent photon and electron flux is solved at every spatial unknown in the computational domain, and quantities such as dose‐to‐medium (DM) are obtained by multiplying the energy dependent particle flux by the energy dependent energy deposition cross section for the associated image pixel material. Comparisons were made with the Monte Carlo code EGSnrc (DOSXYZnrc) for a head‐and‐neck case having eight 1.5×1.5 cm2 photon beams with a 6 MV photon spectrum. The Acuros calculation used 123,000 elements. Since four spatial unknowns were solved in each element, this equated to 492,000 spatial degrees of freedom.The DOSXYZnrc calculation used 2.5×2.5×2.5 mm3 voxels. Results: Results were compared at 95,183 image pixels where dose > 5% of Dmax. 99.02% of pixels satisfied the 3%/3mm criteria (94,251 out of 95,183 pixels). Computational times for the Acuros and DOSXYZnrc calculations were approximately 19.6 CPU minutes (2.2 GHz Opteron processor)and 2,890 CPU minutes (0.4% avg. uncertainty in voxels > Dmax/2), respectively. Through a rewritten, radiotherapy specific solver, Acuros computational times were further reduced to 7.45 CPU minutes without affecting accuracy. When only doses in regions greater than 20% of Dmax are of interest, computational times are further reduced to 3.08 CPU minutes. Conclusions: A clinically viable combination of dose calculation speed and accuracy has been achieved using a radiotherapy specific deterministic solver.Research funded in part by NIH grant 1R43 CA105806‐01A1.

The AMD Opteron Northbridge Architecture

To increase performance while operating within a fixed power budget, the AMD opteron processor integrates multiple times86-64 cores with a router and memory controller. AMD's experience with building a wide variety of system topologies using Opteron's hypertransport-based processor interface has provided useful lessons that expose the challenges to be addressed when designing future system interconnect, memory hierarchy, and I/O to scale with both the number of cores and sockets in future times86-64 CMP architectures.

An Integrated Pipeline of Open Source Software Adapted for Multi-CPU Architectures: Use in the Large-Scale Identification of Single Nucleotide Polymorphisms

The large amounts of EST sequence data available from a single species of an organism as well as for several species within a genus provide an easy source of identification of intra- and interspecies single nucleotide polymorphisms (SNPs). In the case of model organisms, the data available are numerous, given the degree of redundancy in the deposited EST data. There are several available bioinformatics tools that can be used to mine this data; however, using them requires a certain level of expertise: the tools have to be used sequentially with accompanying format conversion and steps like clustering and assembly of sequences become time-intensive jobs even for moderately sized datasets. We report here a pipeline of open source software extended to run on multiple CPU architectures that can be used to mine large EST datasets for SNPs and identify restriction sites for assaying the SNPs so that cost-effective CAPS assays can be developed for SNP genotyping in genetics and breeding applications. At the International Crops Research Institute for the Semi-Arid Tropics (ICRISAT), the pipeline has been implemented to run on a Paracel high-performance system consisting of four dual AMD Opteron processors running Linux with MPICH. The pipeline can be accessed through user-friendly web interfaces at http://hpc.icrisat.cgiar.org/PBSWeb and is available on request for academic use. We have validated the developed pipeline by mining chickpea ESTs for interspecies SNPs, development of CAPS assays for SNP genotyping, and confirmation of restriction digestion pattern at the sequence level.

A Parallel Implementation of the Katsevich Algorithm for 3-D CT Image Reconstruction

Yu and Wang [1, 2] implemented the first theoretically exact spiral cone-beam reconstruction algorithm developed by Katsevich [3, 4]. This algorithm requires a high computational cost when the data amount becomes large. Here we study a parallel computing scheme for the Katsevich algorithm to facilitate the image reconstruction. Based on the proposed parallel algorithm, several numerical tests are conducted on a high performance computing (HPC) cluster with thirty two 64-bit AMD-based Opteron processors. The standard phantom data [5] is used to establish the performance benchmarks. The results show that our parallel algorithm significantly reduces the reconstruction time, achieving high speedup and efficiency.

TH‐E‐224C‐02: Performance Assessment of a Deterministic Method Incorporating Coupled Photon‐Electron Transport for Photon Beam Dose Calculations On Acquired CT Data

Purpose: To evaluate the performance of a neutral and charged particle deterministic solution method for external photon beam dose calculations using acquired CT data. Methods and Materials: The Attila® radiation transport code, which solves the differential form of the linear Boltzmann transport equation for neutral particles and the Boltzmann‐Fokker‐Planck transport equation for charged particles, has been adapted for calculating dose distributions from acquired CT images. Comparisons were made with the Monte Carlo code EGSnrc (DOSXYZnrc) for a sample prostate treatment consisting of 8 10×10 cm2 open beams with a realistic 6 MV photon spectrum. The Attila computational mesh consisted of 125,000 arbitrary tetrahedral elements, of approximately uniform size, which encompassed an imaged torso region. CT numbers were mapped to the Attila tetrahedral elements using one of four materials: air, tissue, adipose tissue, and bone, each having four discrete densities. Attila dose was determined at the image resolution by extracting the energy dependent particle flux at each 1×1×2.5 mm3 pixel from Attila's calculated linear discontinuous finite element spatial representation, and multiplying by the energy dependent dose deposition response for that pixel material and density.The DOSXYZnrc calculation used 2.5×2.5×2.5 mm3 voxels, and was run until an average statistical uncertainty of 0.4% was achieved for voxels in the target region. Results: Computational times Attila and DOSXYZnrc were 36 CPU minutes (2.4 GHz Opteron processor) and approximately 8,500 CPU minutes, respectively. Employing a deterministic electron cut‐off below 300 keV reduced the Attila computational time to 22 CPU minutes. Agreement between both codes was excellent in both high and low dose regions, with differences less than 2%/2mm for greater than 95% of points on a 2‐D plane through the entire torso. Conclusions: A general purpose deterministic solver was successfully applied for dose calculations using CT image data.Research funded by NIH grant 1R43 CA105806‐01A1.

GLARE: A New Approach for Filtering Large Reagent Lists in Combinatorial Library Design Using Product Properties

We present a novel computer algorithm, called GLARE (Global Library Assessment of REagents), that addresses the issue of optimal reagent selection in combinatorial library design. This program reduces or eliminates the time a medicinal chemist spends examining reagents which a priori cannot be part of a "good" library. Our approach takes the large reagent sets returned by standard chemical database queries and produces often considerably reduced reagent sets that are well-behaved with respect to a specific template. The pruning enforces "goodness" constraints such as the Lipinski rule of five on the product properties such that any reagent selection from the resulting sets produces only "good" products. The algorithm we implemented has three important features: (i) As opposed to genetic algorithms or other stochastic algorithms, GLARE uses a deterministic greedy procedure that smoothly filters out nonviable reagents. (ii) The pruning method can be biased to produce reagent sets with a balanced size, conserving proportionally more reagents in smaller sets. (iii) For very large combinatorial libraries, a partitioning scheme allows libraries as large as 10(12) to be evaluated in 0.25 s on an IBM AMD Opteron processor. This algorithm is validated on a diverse set of 12 libraries. The results that we obtained show an excellent compliance to the product property requirements and very fast timings.

Optimization of lattice QCD codes for the AMD Opteron processor

We report our experience of the optimization of the lattice QCD codes for the new Opteron cluster at DESY Hamburg, including benchmarks. Details of the optimization using SSE/SSE2 instructions and the effective use of prefetch instructions are discussed.

Numerical performance and throughput benchmark for electronic structure calculations in PC-Linux systems with new architectures, updated compilers, and libraries.

A number of recently released numerical libraries including Automatically Tuned Linear Algebra Subroutines (ATLAS) library, Intel Math Kernel Library (MKL), GOTO numerical library, and AMD Core Math Library (ACML) for AMD Opteron processors, are linked against the executables of the Gaussian 98 electronic structure calculation package, which is compiled by updated versions of Fortran compilers such as Intel Fortran compiler (ifc/efc) 7.1 and PGI Fortran compiler (pgf77/pgf90) 5.0. The ifc 7.1 delivers about 3% of improvement on 32-bit machines compared to the former version 6.0. Performance improved from pgf77 3.3 to 5.0 is also around 3% when utilizing the original unmodified optimization options of the compiler enclosed in the software. Nevertheless, if extensive compiler tuning options are used, the speed can be further accelerated to about 25%. The performances of these fully optimized numerical libraries are similar. The double-precision floating-point (FP) instruction sets (SSE2) are also functional on AMD Opteron processors operated in 32-bit compilation, and Intel Fortran compiler has performed better optimization. Hardware-level tuning is able to improve memory bandwidth by adjusting the DRAM timing, and the efficiency in the CL2 mode is further accelerated by 2.6% compared to that of the CL2.5 mode. The FP throughput is measured by simultaneous execution of two identical copies of each of the test jobs. Resultant performance impact suggests that IA64 and AMD64 architectures are able to fulfill significantly higher throughput than the IA32, which is consistent with the SpecFPrate2000 benchmarks.

New chips stop buffer overflow attacks

Chip makers are designing a new generation of microprocessors to stop buffer overflow assaults, exploits that hackers often use to attack and extract data from PCs or servers. AMD's Athlon-64 chips for notebook and desktop computers and its Opteron processors for servers include features that provide buffer-overflow protection. Intel offers buffer-overflow protection in its Itanium chips for servers. A buffer overflow occurs when a program or process tries to store more data in a buffer than it was designed to hold. Operating systems that support the buffer-overflow prevention approach mark certain data in memory with a bit that identifies them as executable or nonexecutable. The AMD chips let users turn off the new security feature for legacy programs not written to work with the technology so that the applications can continue to function.