Reduce Memory Requirements Research Articles

Optimal survey design for big data

The volume of data routinely acquired in many industrial geoscientific settings has increased to a point that it poses serious technical challenges to store and process. Optimal experimental design is a tool that can address some of these challenges. First, before data acquisition, optimal design can forecast optimum acquisition geometries and can thus be used to minimize the acquired data volume and reduce the cost of the data life cycle. Second, after data acquisition, optimal design can optimally select the best data to process, thus minimizing the data volume throughput in processing workflows. The catch is that optimal survey design is itself subject to the computational difficulties of big data. We have developed a parallelizable dimensionality reduction method, based on sparse rank-revealing QR-factorization to reduce computing and storage costs for optimal survey design. We implemented the distributable procedure for a guided Bayesian survey design and developed it as part of a workflow to optimize a marine seismic survey of more than 100 million source-receiver pairs in the Gulf of Mexico. The method improved computation times by more than an order of magnitude and reduced memory requirements by nearly two orders of magnitude. Although the procedure involves approximations in exchange for subsequent computational efficiency, the marine survey design results were identical to those achieved without compression. The distributable reduction method may be generally useful for matrix reduction problems in which the matrix data are distributed over multiple machines.

An FPGA Architecture for Extracting Real-Time Zernike Coefficients from Measured Phase Gradients

Abstract Zernike modes are commonly used in adaptive optics systems to represent optical wavefronts. However, real-time calculation of Zernike modes is time consuming due to two factors: the large factorial components in the radial polynomials used to define them and the large inverse matrix calculation needed for the linear fit. This paper presents an efficient parallel method for calculating Zernike coefficients from phase gradients produced by a Shack-Hartman sensor and its real-time implementation using an FPGA by pre-calculation and storage of subsections of the large inverse matrix. The architecture exploits symmetries within the Zernike modes to achieve a significant reduction in memory requirements and a speed-up of 2.9 when compared to published results utilising a 2D-FFT method for a grid size of 8×8. Analysis of processor element internal word length requirements show that 24-bit precision in precalculated values of the Zernike mode partial derivatives ensures less than 0.5% error per Zernike coefficient and an overall error of <1%. The design has been synthesized on a Xilinx Spartan-6 XC6SLX45 FPGA. The resource utilisation on this device is <3% of slice registers, <15% of slice LUTs, and approximately 48% of available DSP blocks independent of the Shack-Hartmann grid size. Block RAM usage is <16% for Shack-Hartmann grid sizes up to 32×32.

Ultrasound tomography for simultaneous reconstruction of acoustic density, attenuation, and compressibility profiles

A fast and efficient forward scattering solver is developed for use in ultrasound tomography. The solver is formulated so as to enable the calculation of scattering from large and relatively high-contrast objects with inhomogeneous physical properties that vary simultaneously in acoustic attenuation, compressibility, and density. It is based on the method of moments in conjunction with a novel implementation of the conjugate gradient algorithm which requires the use of the adjoints of the scattering operators. The solver takes advantage of the symmetric block Toeplitz matrix with symmetric Toeplitz blocks property of the Green's function matrix to increase efficiency and only stores the first row of this matrix to reduce memory requirements. This row is then used for the matrix-vector multiplication using the fast Fourier transform technique, thus, resulting in the computational complexity of O(n log n). The marching-on-source technique is also used to provide a good initial guess which allows the conjugate gradient technique to converge faster than initializing with an arbitrary guess. This feature is important in tomographic inversion algorithms which require that the object to be imaged be interrogated via several incident fields. Forward scattering and inversion examples, based on the Conjugate Gradient Least Squares regularized Born Iterative Method, are shown, in two-dimensions, for objects varying in all three physical properties.

Two-photon absorption cross sections within equation-of-motion coupled-cluster formalism using resolution-of-the-identity and Cholesky decomposition representations: Theory, implementation, and benchmarks.

The equation-of-motion coupled-cluster (EOM-CC) methods provide a robust description of electronically excited states and their properties. Here, we present a formalism for two-photon absorption (2PA) cross sections for the equation-of-motion for excitation energies CC with single and double substitutions (EOM-CC for electronically excited states with single and double substitutions) wave functions. Rather than the response theory formulation, we employ the expectation-value approach which is commonly used within EOM-CC, configuration interaction, and algebraic diagrammatic construction frameworks. In addition to canonical implementation, we also exploit resolution-of-the-identity (RI) and Cholesky decomposition (CD) for the electron-repulsion integrals to reduce memory requirements and to increase parallel efficiency. The new methods are benchmarked against the CCSD and CC3 response theories for several small molecules. We found that the expectation-value 2PA cross sections are within 5% from the quadratic response CCSD values. The RI and CD approximations lead to small errors relative to the canonical implementation (less than 4%) while affording computational savings. RI/CD successfully address the well-known issue of large basis set requirements for 2PA cross sections calculations. The capabilities of the new code are illustrated by calculations of the 2PA cross sections for model chromophores of the photoactive yellow and green fluorescent proteins.

Symbol-Decision Successive Cancellation List Decoder for Polar Codes

Polar codes are of great interests because they provably achieve the capacity of both discrete and continuous memoryless channels while having an explicit construction. Most existing decoding algorithms of polar codes are based on bit-wise hard or soft decisions. In this paper, we propose symbol-decision successive cancellation (SC) and successive cancellation list (SCL) decoders for polar codes, which use symbol-wise hard or soft decisions for higher throughput or better error performance. First, we propose to use a recursive channel combination to calculate symbol-wise channel transition probabilities, which lead to symbol decisions. Our proposed recursive channel combination also has a lower complexity than simply combining bit-wise channel transition probabilities. The similarity between our proposed method and Arikan's channel transformations also helps to share hardware resources between calculating bit- and symbol-wise channel transition probabilities. Second, a two-stage list pruning network is proposed to provide a trade-off between the error performance and the complexity of the symbol-decision SCL decoder. Third, since memory is a significant part of SCL decoders, we propose a pre-computation memory-saving technique to reduce memory requirement of an SCL decoder. Finally, to evaluate the throughput advantage of our symbol-decision decoders, we design an architecture based on a semi-parallel successive cancellation list decoder. In this architecture, different symbol sizes, sorting implementations, and message scheduling schemes are considered. Our synthesis results show that in terms of area efficiency, our symbol-decision SCL decoders outperform both bit- and symbol-decision SCL decoders.

Modified Fruit Fly Optimization Algorithm for Analysis of Large Antenna Array

This research paper deals with the optimization of a large antenna array for maximum directivity using a modified fruit fly optimization algorithm (MFOA) with random search of two groups of swarm and adaptive fruit fly swarm population size. The MFOA is utilized to determine three nonlinear mathematical test functions, analysis of the optimal number of elements and optimal element spacing of the large antenna array, and analysis of nonuniform amplitude of antenna array. The numerical results demonstrate that the MFOA is effective in solving all test function and electromagnetic problems. The advantages of the proposed algorithm are ease of implementation, large search range, less processing time, and reduced memory requirement.

Implementation of Adaptive Digital Beamforming using Cordic

Sonar imaging is one of the simplest technique for detection of under water drowned bodies. There is a need for design of conventional beamforming which are robust and simple. Adaptive beamformer is used to improve the quality of the sonar image. As a result we get an image containing more useful and correct information. The CORDIC computing technique a highly efficient method to compute elementary functions like sine, cosine, translate, rotate values using CORDIC algorithm. The system simulation was carried out using ModelSim and Xilinx ISE Design Suite 9.2i.. Matlab code is used to implement sin and cos using cordic angles and amplitude response of beamformed data by optimized method in order to enlarge the validity region of beamforming. Synthesis results of cordic shows the reduced memory requirement and less power consumption.

Complex Function Approximation Using Two-Dimensional Interpolation

This paper presents a new scheme for evaluating complex reciprocal and exponential functions in hardware. The proposed method utilizes a two-dimensional convolution algorithm to interpolate bivariate functions from tabulated function values in the complex domain. To reduce the memory requirements for lookup tables, the interpolation is decomposed into independent row and column computations, such that the same coefficient table can be shared. Three different interpolation kernels from degree-1 (linear) to degree-2 (quadratic Lagrange) and degree-3 (cubic Lagrange) are explored to find the optimal design parameters and the most acceptable trade-offs between performance and hardware resources. Moreover, a generic hardware architecture is designed to provide scalable implementation capabilities for computation precision and interpolation degree. To verify the proposed architecture, eight complex reciprocal and eight complex exponential design instances are implemented. The ASIC- and FPGA-based experimental results show that the proposed scheme can efficiently approximate the complex reciprocal and exponential functions with up to 16-bit precision, as well as achieve a considerable reduction of memory requirements compared with traditional bipartite and multipartite schemes. The proposed method is also applicable to other complex functions.

NeatFreq: reference-free data reduction and coverage normalization for De Novo sequence assembly.

BackgroundDeep shotgun sequencing on next generation sequencing (NGS) platforms has contributed significant amounts of data to enrich our understanding of genomes, transcriptomes, amplified single-cell genomes, and metagenomes. However, deep coverage variations in short-read data sets and high sequencing error rates of modern sequencers present new computational challenges in data interpretation, including mapping and de novo assembly. New lab techniques such as multiple displacement amplification (MDA) of single cells and sequence independent single primer amplification (SISPA) allow for sequencing of organisms that cannot be cultured, but generate highly variable coverage due to amplification biases.ResultsHere we introduce NeatFreq, a software tool that reduces a data set to more uniform coverage by clustering and selecting from reads binned by their median kmer frequency (RMKF) and uniqueness. Previous algorithms normalize read coverage based on RMKF, but do not include methods for the preferred selection of (1) extremely low coverage regions produced by extremely variable sequencing of random-primed products and (2) 2-sided paired-end sequences. The algorithm increases the incorporation of the most unique, lowest coverage, segments of a genome using an error-corrected data set. NeatFreq was applied to bacterial, viral plaque, and single-cell sequencing data. The algorithm showed an increase in the rate at which the most unique reads in a genome were included in the assembled consensus while also reducing the count of duplicative and erroneous contigs (strings of high confidence overlaps) in the deliverable consensus. The results obtained from conventional Overlap-Layout-Consensus (OLC) were compared to simulated multi-de Bruijn graph assembly alternatives trained for variable coverage input using sequence before and after normalization of coverage. Coverage reduction was shown to increase processing speed and reduce memory requirements when using conventional bacterial assembly algorithms.ConclusionsThe normalization of deep coverage spikes, which would otherwise inhibit consensus resolution, enables High Throughput Sequencing (HTS) assembly projects to consistently run to completion with existing assembly software. The NeatFreq software package is free, open source and available at https://github.com/bioh4x/NeatFreq.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-014-0357-3) contains supplementary material, which is available to authorized users.

Large-Scale Liquid Simulation on Adaptive Hexahedral Grids.

Regular grids are attractive for numerical fluid simulations because they give rise to efficient computational kernels. However, for simulating high resolution effects in complicated domains they are only of limited suitability due to memory constraints. In this paper we present a method for liquid simulation on an adaptive octree grid using a hexahedral finite element discretization, which reduces memory requirements by coarsening the elements in the interior of the liquid body. To impose free surface boundary conditions with second order accuracy, we incorporate a particular class of Nitsche methods enforcing the Dirichlet boundary conditions for the pressure in a variational sense. We then show how to construct a multigrid hierarchy from the adaptive octree grid, so that a time efficient geometric multigrid solver can be used. To improve solver convergence, we propose a special treatment of liquid boundaries via composite finite elements at coarser scales. We demonstrate the effectiveness of our method for liquid simulations that would require hundreds of millions of simulation elements in a non-adaptive regime.

Efficient over-determined implementation of the immersed boundary conditions method

An efficient implementation of the spectrally-accurate immersed boundary conditions method based on the over-determined formulation is presented. The linear system resulting from the discretization is solved using a special solver that takes advantage of the structure of the coefficient matrix. Part of the system corresponding to the field equation is solved exactly while the part corresponding to the boundary relations is solved in the least squares sense. This process permits use of a larger number of constraints to enforce the boundary conditions along the complex flow boundaries. A significant expansion of the range of geometries that can be handled accurately using the new implementation has been documented. Furthermore, a significant acceleration of computations as well as a significant reduction of memory requirements have been demonstrated.

Developing a scalable hybrid MPI/OpenMP unstructured finite element model

The trend of all modern computer architectures, and the path to exascale, is towards increasing numbers of lower power cores, with a decreasing memory to core ratio. This imposes a strong evolutionary pressure on algorithms and software to efficiently utilise all levels of parallelism available on a given platform while minimising data movement. Unstructured finite elements codes have long been effectively parallelised using domain decomposition methods, implemented using libraries such as the Message Passing Interface (MPI). However, there are many optimisation opportunities when threading is used for intra-node parallelisation for the latest multi-core/many-core platforms. The benefits include increased algorithmic freedom, reduced memory requirements, cache sharing, reduced number of partitions, less MPI communication and I/O overhead.In this paper, we report progress in implementing a hybrid OpenMP–MPI version of the unstructured finite element code Fluidity. For matrix assembly kernels, the OpenMP parallel algorithm uses graph colouring to identify independent sets of elements that can be assembled concurrently with no race conditions. In this phase there are no MPI overheads as each MPI process only assembles its own local part of the global matrix. We use an OpenMP threaded fork of PETSc to solve the resulting sparse linear systems of equations. We experiment with a range of preconditioners, including HYPRE which provides the algebraic multigrid preconditioner BoomerAMG where the smoother is also threaded. Since unstructured finite element codes are well known to be memory latency bound, particular attention is paid to ccNUMA architectures where data locality is particularly important to achieve good intra-node scaling characteristics. We also demonstrate that utilising non-blocking algorithms and libraries are critical to mixed-mode application so that it can achieve better parallel performance than the pure MPI version.

BIG-OH: BInarization of gradient orientation histograms

Extracting local keypoints and keypoint descriptions from images is a primary step for many computer vision and image retrieval applications. In the literature, many researchers have proposed methods for representing local texture around keypoints with varying levels of robustness to photometric and geometric transformations. Gradient-based descriptors such as the Scale Invariant Feature Transform (SIFT) are among the most consistent and robust descriptors. The SIFT descriptor, a 128-element vector consisting of multiple gradient histograms computed from local image patches around a keypoint, is widely considered as the gold standard keypoint descriptor. However, SIFT descriptors require at least 128bytes of storage per descriptor. Since images are typically described by thousands of keypoints, it may require more space to store the SIFT descriptors for an image than the original image itself. This may be prohibitive in extremely large-scale applications and applications on memory-constrained devices such as tablets and smartphones. In this paper, with the goal of reducing the memory requirements of keypoint descriptors such as SIFT, without affecting their performance, we propose BIG-OH, a simple yet extremely effective method for binary quantization of any descriptor based on gradient orientation histograms. BIG-OH's memory requirements are very small—when it uses SIFT's default parameters for the construction of the gradient orientation histograms, it only requires 16bytes per descriptor. BIG-OH quantizes gradient orientation histograms by computing a bit vector representing the relative magnitudes of local gradients associated with neighboring orientation bins. In a series of experiments on keypoint matching with different types of keypoint detectors under various photometric and geometric transformations, we find that the quantized descriptor has performance comparable to or better than other descriptors, including BRISK, CARD, BRIEF, D-BRIEF, SQ, and PCA-SIFT. Our experiments also show that BIG-OH is extremely effective for image retrieval, with modestly better performance than SIFT. BIG-OH's drastic reduction in memory requirements, obtained while preserving or improving the image matching and image retrieval performance of SIFT, makes it an excellent descriptor for large image databases and applications running on memory-constrained devices.

High Throughput Radix-4 SISO Decoding Architecture with Reduced Memory Requirement

High Throughput Radix-4 SISO Decoding Architecture with Reduced Memory Requirement

English

Digital image processing(DIP) is the use of computer algorithms to perform image processing on digital images. The basic operation performed by a simple digital camera is, to convert the light energy to electrical energy, then the energy is converted to digital format and a compression algorithm is used to reduce memory requirement for storing the image. This compression algorithm is frequently called for capturing and storing the images. This leads us to develop an efficient compression algorithm which will give the same result as that of the existing algorithms with low power consumption. Image compression is useful as it helps in reduction of the usage of expensive resources, such as memory, or the transmission bandwidth required. But on the downside, compression techniques result in distortion and also additional computational resources are required for compression- decompression of the medical image data. In this, we proposed DHT based image compression technique. The computational complexity is further reduced by using pipelined CORDIC architecture for computation of sin and cos terms in DHT. The CORDIC based DHT is partially simulated and synthesized by using Xilinx ISE design suite and the same was implemented on targeted FPGA. The hardware requirements and gate delay values are observed and noted down.

Adaptive grouping for the higher‐order multilevel fast multipole method

ABSTRACTAn alternative parameter‐free adaptive approach for the grouping of the basis function patterns in the multilevel fast multipole method is presented, yielding significant memory savings compared to the traditional Octree grouping for most discretizations, particularly when using higher‐order basis functions. Results from both a uniformly and nonuniformly meshed scatterer are presented, showing how the technique is worthwhile even for regular meshes, and demonstrating that there is no loss of accuracy in spite of the large reduction in memory requirements and the relatively low computational cost. © 2014 Wiley Periodicals, Inc. Microwave Opt Technol Lett 56:2451–2456, 2014

Fast Covariance Estimation for High-dimensional Functional Data.

We propose two fast covariance smoothing methods and associated software that scale up linearly with the number of observations per function. Most available methods and software cannot smooth covariance matrices of dimension J > 500; a recently introduced sandwich smoother is an exception but is not adapted to smooth covariance matrices of large dimensions, such as J = 10, 000. We introduce two new methods that circumvent those problems: 1) a fast implementation of the sandwich smoother for covariance smoothing; and 2) a two-step procedure that first obtains the singular value decomposition of the data matrix and then smoothes the eigenvectors. These new approaches are at least an order of magnitude faster in high dimensions and drastically reduce computer memory requirements. The new approaches provide instantaneous (a few seconds) smoothing for matrices of dimension J = 10,000 and very fast (< 10 minutes) smoothing for J = 100, 000. R functions, simulations, and data analysis provide ready to use, reproducible, and scalable tools for practical data analysis of noisy high-dimensional functional data.

A node linkage approach for sequential pattern mining.

Sequential Pattern Mining is a widely addressed problem in data mining, with applications such as analyzing Web usage, examining purchase behavior, and text mining, among others. Nevertheless, with the dramatic increase in data volume, the current approaches prove inefficient when dealing with large input datasets, a large number of different symbols and low minimum supports. In this paper, we propose a new sequential pattern mining algorithm, which follows a pattern-growth scheme to discover sequential patterns. Unlike most pattern growth algorithms, our approach does not build a data structure to represent the input dataset, but instead accesses the required sequences through pseudo-projection databases, achieving better runtime and reducing memory requirements. Our algorithm traverses the search space in a depth-first fashion and only preserves in memory a pattern node linkage and the pseudo-projections required for the branch being explored at the time. Experimental results show that our new approach, the Node Linkage Depth-First Traversal algorithm (NLDFT), has better performance and scalability in comparison with state of the art algorithms.

Diffeomorphic metric mapping and probabilistic atlas generation of hybrid diffusion imaging based on BFOR signal basis.

We first propose a large deformation diffeomorphic metric mapping algorithm to align multiple b-value diffusion weighted imaging (mDWI) data, specifically acquired via hybrid diffusion imaging (HYDI). We denote this algorithm as LDDMM-HYDI. We then propose a Bayesian probabilistic model for estimating the white matter atlas from HYDIs. We adopt the work given in Hosseinbor et al. (2013) and represent the q-space diffusion signal with the Bessel Fourier orientation reconstruction (BFOR) signal basis. The BFOR framework provides the representation of mDWI in the q-space and the analytic form of the emsemble average propagator (EAP) reconstruction, as well as reduces memory requirement. In addition, since the BFOR signal basis is orthonormal, the L(2) norm that quantifies the differences in the q-space signals of any two mDWI datasets can be easily computed as the sum of the squared differences in the BFOR expansion coefficients. In this work, we show that the reorientation of the q-space signal due to spatial transformation can be easily defined on the BFOR signal basis. We incorporate the BFOR signal basis into the LDDMM framework and derive the gradient descent algorithm for LDDMM-HYDI with explicit orientation optimization. Additionally, we extend the previous Bayesian atlas estimation framework for scalar-valued images to HYDIs and derive the expectation-maximization algorithm for solving the HYDI atlas estimation problem. Using real HYDI datasets, we show that the Bayesian model generates the white matter atlas with anatomical details. Moreover, we show that it is important to consider the variation of mDWI reorientation due to a small change in diffeomorphic transformation in the LDDMM-HYDI optimization and to incorporate the full information of HYDI for aligning mDWI. Finally, we show that the LDDMM-HYDI outperforms the LDDMM algorithm with diffusion tensors generated from each shell of HYDI.

Evaluation of unsteady computational fluid dynamics models applied to the analysis of a transonic high-pressure turbine stage

This paper presents an efficient ‘Phase-Lagged’ method developed for turbomachinery applications. The method is based on the generalized-shape-correction model. Moving average techniques as well as double-passage domain formulation were adopted in order to reduce memory requirements and improve the model robustness. The model was used to evaluate the aerodynamic performance of the high-pressure transonic turbine stage CT3, experimentally studied at the von Kármán Institute for Fluid Dynamics in the framework of the EU funded TATEF2 project. The results are discussed and compared with both the available experimental data and the results obtained by means of both steady and unsteady scaled full-annulus approaches. Computational requirements of the generalized-shape-correction model are evaluated and discussed showing that nowadays unsteady results can be obtained at an affordable computational cost.