Single-precision Operations Research Articles

Seismic modeling and inversion using half-precision floating-point numbers

New processors are increasingly supporting half-precision floating-point numbers, often with a significant throughput gain over single-precision operations. Seismic modeling, imaging, and inversion could benefit from such an acceleration, but it is not obvious how the accuracy of the solution can be preserved with a very narrow 16-bit representation. By scaling the finite-difference expression of the isotropic elastic wave equation, we have found that a stable solution can be obtained despite the very narrow dynamic range of the half-precision format.We develop an implementation with the CUDA platform, which, on most recent graphics processing units (GPU), is nearly twice as fast and uses half the memory of the equivalent single-precision version. The error on seismograms caused by the reduced precision is shown to correspond to a negligible fraction of the total seismic energy and is mostly incoherent with seismic phases. Finally, we find that this noise does not adversely impact full-waveform inversion nor reverse time migration, which both benefit from the higher throughput of half-precision computation.

A novel parallel finite element procedure for nonlinear dynamic problems using GPU and mixed-precision algorithm

Purpose The purpose of this paper is to improve the computational speed of solving nonlinear dynamics by using parallel methods and mixed-precision algorithm on graphic processing units (GPUs). The computational efficiency of traditional central processing units (CPUs)-based computer aided engineering software has been difficult to satisfy the needs of scientific research and practical engineering, especially for nonlinear dynamic problems. Besides, when calculations are performed on GPUs, double-precision operations are slower than single-precision operations. So this paper implemented mixed precision for nonlinear dynamic problem simulation using Belytschko-Tsay (BT) shell element on GPU. Design/methodology/approach To minimize data transfer between heterogeneous architectures, the parallel computation of the fully explicit finite element (FE) calculation is realized using a vectorized thread-level parallelism algorithm. An asynchronous data transmission strategy and a novel dependency relationship link-based method, for efficiently solving parallel explicit shell element equations, are used to improve the GPU utilization ratio. Finally, this paper implements mixed precision for nonlinear dynamic problems simulation using the BT shell element on a GPU and compare it to the CPU-based serially executed program and a GPU-based double-precision parallel computing program. Findings For a car body model containing approximately 5.3 million degrees of freedom, the computational speed is improved 25 times over CPU sequential computation, and approximately 10% over double-precision parallel computing method. The accuracy error of the mixed-precision computation is small and can satisfy the requirements of practical engineering problems. Originality/value This paper realized a novel FE parallel computing procedure for nonlinear dynamic problems using mixed-precision algorithm on CPU-GPU platform. Compared with the CPU serial program, the program implemented in this article obtains a 25 times acceleration ratio when calculating the model of 883,168 elements, which greatly improves the calculation speed for solving nonlinear dynamic problems.

Fast Deep Neural Network Training on Distributed Systems and Cloud TPUs

Since its creation, the ImageNet-1k benchmark set has played a significant role as a benchmark for ascertaining the accuracy of different deep neural net (DNN) models on the image classification problem. Moreover, in recent years it has also served as the principal benchmark for assessing different approaches to DNN training. Finishing a 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires $10^{18}$1018 single precision operations in total. On the other hand, the world's current fastest supercomputer can finish $3 \times 10^{17}$3×1017 single precision operations per second (according to the Nov 2018 Top 500 results). If we can make full use of the computing capability of the fastest supercomputer, we should be able to finish the training in several seconds. Over the last two years, researchers have focused on closing this significant performance gap through scaling DNN training to larger numbers of processors. Most successful approaches to scaling ImageNet training have used the synchronous mini-batch stochastic gradient descent (SGD). However, to scale synchronous SGD one must also increase the batch size used in each iteration. Thus, for many researchers, the focus on scaling DNN training has translated into a focus on developing training algorithms that enable increasing the batch size in data-parallel synchronous SGD without losing accuracy over a fixed number of epochs. In this paper, we investigate supercomputers’ capability of speeding up DNN training. Our approach is to use a large batch size, powered by the Layer-wise Adaptive Rate Scaling (LARS) algorithm, for efficient usage of massive computing resources. Our approach is generic, as we empirically evaluate the effectiveness on five neural networks: AlexNet, AlexNet-BN, GNMT, ResNet-50, and ResNet-50-v2 trained with large datasets while preserving the state-of-the-art test accuracy. Compared to the baseline of a previous study from Goyal et al. [1] , our approach shows higher test accuracy on batch sizes that are larger than 16K. When we use the same baseline, our results are better than Goyal et al. for all the batch sizes (Fig. 20 ). Using 2,048 Intel Xeon Platinum 8160 processors, we reduce the 100-epoch AlexNet training time from hours to 11 minutes. With 2,048 Intel Xeon Phi 7250 Processors, we reduce the 90-epoch ResNet-50 training time from hours to 20 minutes. Our implementation is open source and has been released in the Intel distribution of Caffe, Facebook's PyTorch, and Google's TensorFlow. The difference between this paper and the conference-version of our work [2] includes: (1) we implement our approach on Google's cloud Tensor Processing Unit (TPU) platform, which verifies our previous success on CPUs and GPUs. (2) we scale the batch size of ResNet-50-v2 to 32K and achieve 76.3 percent accuracy, which is better than the 75.3 percent accuracy achieved in our conference paper. (3) we apply our approach to Google's Neural Machine Translation (GNMT) application, which helps us to achieves 4× speedup on the cloud TPUs.

Efficient Multiple-Precision Floating-Point Fused Multiply-Add with Mixed-Precision Support

In this paper, an efficient multiple-precision floating-point fused multiply-add (FMA) unit is proposed. The proposed FMA supports not only single-precision, double-precision, and quadruple-precision operations, as some previous works do, but also half-precision operations. The proposed FMA architecture can execute one quadruple-precision operation, or two parallel double-precision operations, or four parallel single-precision operations, or eight parallel half-precision operations every clock cycle. In addition to the support of normal FMA operations, the proposed FMA also supports mixed-precision FMA operations and mixed-precision dot-product operations. Specifically, the products of two lower precision multiplications can be accumulated to a higher precision addend. By setting the operands of one multiplication to zeros, the proposed FMA can also perform mixed-precision FMA operations. Support for mixed-precision FMA and mixed-precision dot-product is newly added but it only consumes 6.5 percent more area compared to a normal multiple-precision FMA unit. Compared to the state-of-the-art multiple-precision FMA design, the proposed FMA supports more floating-point operations such as half-precision FMA operations and mixed-precision operations with only 10.6 percent larger area.

Compressible lattice Boltzmann simulations on high‐performance and low‐cost GeForce GPU

SummaryThe feasibility of compressible lattice Boltzmann simulations on high‐performance and low‐cost NVIDIA GeForce GPU is studied to enhance computational efficiency of compressible lattice Boltzmann by using much cheaper computing hardware. The NVIDIA GeForce GTX 1080 GPU, which has similar single precision operation computing performance with NVIDIA scientific computational Tesla K80 GPU and much lower price (seven to ten times) than the latter, is used in this work. The problem of GeForce GPUs is with low double precision operation performance and lack of error‐correcting code memory, which will influence the accuracy of compressible lattice Boltzmann simulations. To evaluate its feasibility, the double‐distribution‐function lattice Boltzmann method is adopted for compressible fluid flows. Three typical compressible problems are tested, ie, the Riemann problem, the regular shock reflection problem, and the supersonic boundary layer problem. The results by GeForce GPU with single precision operation have numerical fluctuations in the simulations with shock waves. Moreover, GeForce GPUs have similar numerical accuracy with CPUs in the regular shock reflection problem and supersonic boundary layer problem.

New Features of Parallel Implementation of N-Body Problems on GPU

This paper focuses on the parallel implementation of a direct $N$-body method~(particle-particle algorithm) and the application of multiple GPUs for galactic dynamics simulations. Application of a hybrid OpenMP-CUDA technology is considered for models with a number of particles $N \sim 10^5 \div 10^7$. By means of $N$-body simulations of gravitationally unstable stellar galactic we have investigated the algorithms parallelization efficiency for various Nvidia Tesla graphics processors~(K20, K40, K80). Particular attention was paid to the parallel performance of simulations and accuracy of the numerical solution by comparing single and double floating-point precisions~(SP and DP). We showed that the double-precision simulations are slower by a factor of~$1.7$ than the single-precision runs performed on Nvidia Tesla K-Series processors. We also claim that application of the single-precision operations leads to incorrect result in the evolution of the non-axisymmetric gravitating $N$-body systems. In particular, it leads to significant quantitative and even qualitative distortions in the galactic disk evolution. For instance, after $10^4$ integration time steps for the single-precision numbers the total energy, momentum, and angular momentum of a system with $N = 2^{20}$ conserve with accuracy of $10^{-3}$, $10^{-2}$ and $10^{-3}$ respectively, in comparison to the double-precision simulations these values are $10^{-5}$, $10^{-15}$ and $10^{-13}$, respectively. Our estimations evidence in favour of usage of the second-order accuracy schemes with double-precision numbers since it is more efficient than in the fourth-order schemes with single-precision numbers.

High performance and energy efficient single‐precision and double‐precision merged floating‐point adder on FPGA

A high performance and energy efficient single-precision and double-precision merged floating-point adder based on the two-path FP addition algorithm designed and implemented on field programmable gate array (FPGA) is presented. With a fully pipelined architecture, the proposed adder can accomplish one double-precision addition or two parallel single-precision additions in six clock cycles. The proposed architecture is designed based on the double-precision adder and each major component is segmented to support dual single-precision operations. In addition, all the components of the proposed adder are optimised for mapping on FPGA. The proposed architecture is implemented on both Altera Stratix-III and Xilinx Virtex-5 devices and it has a faster clock frequency when compared with the double-precision intellectual property (IP) core adder provided by the FPGA vendors. Since the dual single-precision operations support, the proposed adder has higher throughput compared with the single-precision IP core adder. In addition, the proposed adder has better energy efficiency compared with both single-precision and double-precision IP core adder. The implementation results of the proposed adder on the latest Altera Arria-10 and Xilinx Virtex-7 devices are provided. A direct implementation of the proposed architecture on STM-90 nm technology ASIC platform is also performed.

An area efficient multi-mode quadruple precision floating point adder

Most of the scientific and engineering applications require accurate computations. Double precision floating point computations are not enough for many applications like climate modelling, computational physics, etc. Efficient design of quadruple precision floating point adder is needed for these applications. The proposed multi-mode quadruple precision floating point adder architecture supports four single precision operations in parallel, as well as two double precision operations in parallel and also supports one quadruple precision operation. Compared to existing Quadruple precision floating point adders and Dual mode Quadruple precision floating point adder, the proposed architecture can perform more computations with less area because of resource sharing among different precision operands. The proposed Multi-mode quadruple precision adder supports both normal and subnormal operations and also the exceptional case handling such as infinity, Not a Number (NaN) and zero cases. The proposed adder has been designed and implemented in both ASIC and FPGA. During ASIC implementation with 90nm technology using the synopsis tool, the proposed Multi-mode quadruple precision floating point adder has a 38.57% smaller area compared to the existing quadruple precision floating point adder. Similarly, the proposed design reduces the area by 29.28% and 35.68% when implemented on Virtex 4 and Virtex 5 FPGAs respectively.

Guest Editorial: Computing Frontiers

Computing Frontiers is a forum where researchers meet to exchange ideas in all areas of Computer Science. While its roots are in computer systems research, Computing Frontiers has diversified and it is now closer to an interdisciplinary forum. Its main focus is on giving exposure to innovative and visionarywork in the broad field of Computer Science and Engineering. The 2011 edition of the conference covered research in awide range of topics including processor andmemory architecture, quantum computing, parallel applications, dynamic binary translation, and fault tolerant design. From all the accepted work, we selected the top four articles for an extended publication in this edition. In “Emerging Architectures Enable to Boost Massively Parallel Data Mining using Adaptive SparseGrids,”AlexanderHeinecke andDirk Pfluger present a parallel implementation for the sparse grid algorithms using the vectorization and streaming capabilities of both CPUs and GPUs. The authors target state-of-the-art processors from leading vendors. On a hybrid CPU/GPU system the authors achieve an impressive 188x speedup for single precision operations. In “Addressing GPU On-chip Shared Memory Bank Conflicts Using Elastic Pipeline,” Chunyang Gou and Georgi N. Gaydadjiev propose optimizations to the GPU pipeline as to avoid the on-chip shared-memory bank conflicts. With this technique the authors are able to improve the performance by up to 21%.

Configurable Multimode Embedded Floating-Point Units for FPGAs

Performance of field-programmable gate arrays (FPGAs) used for floating-point applications is poor due to the complexity of floating-point arithmetic. Implementing floating-point units (FPUs) on FPGAs consume a large amount of resources. This makes FPGAs less attractive for use in floating-point intensive applications. Therefore, there is a need for embedded FPUs in FPGAs. However, if unutilized, embedded FPUs waste space on the FPGA die. To overcome this issue, we propose a flexible multimode embedded FPU for FPGAs that can be configured to perform a wide range of operations. The floating-point adder and multiplier in our embedded FPU can each be configured to perform one double-precision operation or two single-precision operations in parallel. To increase flexibility further, access to the large integer multiplier, adder and shifters in the FPU is provided. Benchmark circuits were implemented on both a standard Xilinx Virtex-II FPGA and on our FPGA with embedded FPU blocks. The results using our embedded FPUs showed a mean area improvement of 5.5 times and a mean delay improvement of 5.8 times for the double-precision benchmarks, and a mean area improvement of 3.8 times and a mean delay improvement of 4.2 times for the single-precision benchmarks. The embedded FPUs were also shown to provide significant area and delay benefits for fixed-point and integer circuits.

High-speed and low-power reconfigurable architectures of 2-digit two-dimensional logarithmic number system-based recursive multipliers

In new DSP applications, reconfigurable architectures have emerged to provide a flexible, high-performance, high-speed and low-power implementation platform for wireless embedded devices. Since some DSP algorithms rely heavily on multiplication, there are still demands for more efficient multiplication structures. In this study, two reconfigurable recursive multipliers are presented. The authors' architectures combine some of the flexibility of software with the high performance of hardware through implementing different levels of recursive multiplication schemes on a two-dimensional logarithmic number system (2DLNS) processing structure. The data are split into a number of smaller sections, where each section is converted to a 2-digit 2DLNS (2 bases) representation. The dynamic range reduction and logarithmic characteristics of computing with two orthogonal base exponents in this number system allows multiplication to be implemented with simple parallel small adders. The authors' architectures are able to perform single and double precision multiplications, as well as fault tolerant and dual throughput single precision operations. The implementations demonstrate the efficiency of 2DLNS in multiplication intensive DSP applications and show outstanding results in terms of operation delay and dynamic power consumption.

The PMS project: Poor man's supercomputer

We briefly describe the Poor Man's Supercomputer (PMS) project carried out at Eötvös University, Budapest. The goal was to construct a cost effective, scalable, fast parallel computer to perform numerical calculations of physical problems that can be implemented on a lattice with nearest neighbour interactions. To this end we developed the PMS architecture using PC components and designed a special, low cost communication hardware and the driver software for Linux OS. Our first implementation of PMS includes 32 nodes (PMS1). The performance of PMS1 was tested by Lattice Gauge Theory simulations. Using pure SU(3) gauge theory or the bosonic part of the minimal supersymmetric extention of the standard model (MSSM) on PMS1 we obtained 3$ / Mflops and 0.60$ / Mflops price-to-sustained performance ratio for double and single precision operations, respectively. The design of the special hardware and the communication driver are freely available upon request for non-profit organizations.

Multiple precision square root using the Dwandwa square-root algorithm

Abstract This paper presents the use of an algorithm called Dwandwa square-root algorithm to compute large multiple-precision square roots on computers using existing single-precision operations. A few modifications to the basic algorithm are proposed to improve its efficiency. The implementation of this algorithm on two different processor architectures is discussed. An analysis of the performance of this algorithm in comparison with other known techniques is then presented to highlight the advantages of this technique.

Parallel efficiency for solving linear systems on dawn1000

Here we give a parallelLU decomposition method using block technique. The triangular systems are solved by using modified method of [1]. The numerical results are tested on Dawn 1000 multiprocessor system. From these results, we see that block technique is very important in matrix operations. In our method of solving linear system, single processor can get 44MFOLPS, and 32 processors can get 1117MFLOPS in single precision operation, i.e., the speed up of 32 processors is over 25.

Modified straight division: A computer implementation of multiple-precision division

The ‘Straight division’ algorithm is an in-place division technique that has been known in India as a mental computation technique. This technique is found to be suitable for implementing multiple-precision division on computers using existing single-precision operations. This paper presents some modifications carried out on this basic technique to improve the efficiency of the algorithm. It also discusses an implementation of multiple-precision division using this modified straight division technique on two different processor architectures. This is followed by an analysis of these implementations in comparison with other existing division techniques. It is found that the modified straight division is superior in performance to other known methods.

Fast rounding in multiprecision floating-slash arithmetic

A computational algorithm is described which quickly rounds large fractions into a fixed-length multiprecision floating-slash representation, using single-precision operations where possible. An easily calculated criterion for terminating the rounding process is given.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Fast Multiple-Precision Evaluation of Elementary Functions

Let ƒ( x ) be one of the usual elementary functions (exp, log, artan, sin, cosh, etc.), and let M ( n ) be the number of single-precision operations required to multiply n -bit integers. It is shown that ƒ( x ) can be evaluated, with relative error Ο (2 - n ), in Ο ( M ( n )log ( n )) operations as n → ∞, for any floating-point number x (with an n -bit fraction) in a suitable finite interval. From the Schönhage-Strassen bound on M ( n ), it follows that an n -bit approximation to ƒ( x ) may be evaluated in Ο ( n log 2 ( n ) log log( n )) operations. Special cases include the evaluation of constants such as π, e , and e π . The algorithms depend on the theory of elliptic integrals, using the arithmetic-geometric mean iteration and ascending Landen transformations.