Array Data Structure Research Articles

APPLICATION OF NON-RELATIONAL DATABASES IN THE SOCIAL NETWORKS

The purpose of this research is to present knowledge in the field of work of databases that contain semi-structured data, also known as NoSQL, in applications using large array data structures. The characteristics of the main categories of the non-relational databases and some of their typical representatives (MongoDB, Redis) have been reviewed from the point of view of using them for saving and analysis of data in the social networks. The article presents a study, conducted on the already existing studies in regards to the implementation of the non-relational databases in social networks. Some of the perspectives of the future research are outlined as well.

Automatic conversion of OSM data into LULC maps: comparing FOSS4G based approaches towards an enhanced performance

OSM2LULC is a software package developed to automatically convert OpenStreetMap (OSM) data into Land Use Land Cover (LULC) maps using Free and Open Source Software for Geospatial (FOSS4G) tools. It needs to be highly efficient given the increasing detail of OSM data and the need to apply it to large extent regions. In this article, a comparison between the implementation of OSM2LULC in different available GIS platforms is made using both vector and raster data structures, which resulted in different versions. A description of the differences of each version is made and, to assess their performance, they were applied to four different study areas with different characteristics, in terms of available OSM data and area size. The performance of each version was evaluated taking into account: the overall processing time required to obtain LULC maps; and differences in the results obtained when different data structures (vector and raster) were used. Results showed that the adoption of a strategy that favors interoperability between FOSS4G and the combined use of both vector and raster data promotes a performance increase. After analysing the topological relationships of OSM data, the conversion to raster data format and the execution of procedural parts with such data indicated significant performance gains, without any positional distortions that significantly compromise the applicability of the final result in further case scenarios.

Pre-conditioning strategies to accelerate the convergence of iterative methods in multiphase flow simulations

A computational bottleneck during the solution to multiphase formulations of the incompressible Navier–Stokes equations is often during the implicit solution of the pressure-correction equation that results from operator-splitting methods. Since density is a coefficient in the pressure-correction equation, large variations or discontinuities among the phase densities greatly increase the condition number of the pressure-correction matrix and retard the convergence of iterative methods employed in its solution. To alleviate this shortcoming, the open-source multiphase code MFiX is interfaced with the linear solver library PETSc. Through an appropriate mapping of matrix and vector data structures between the two software packages, an access to a suite of robust, scalable, solver options in PETSc is obtained. Verification of the implementation is demonstrated through predictions that are identical to those obtained from MFiX’s native solvers for a class of single-phase and multiphase flow problems. For a low Reynolds number, flow over a cylinder case, applying Right Side Block Jacobi Preconditioning to the BiCGSTAB iterative solver in PETSc was faster than MFiX’s native solver. This speed-up increased with higher mesh resolution and for higher-order spatial discretizations. In a fluidized bed simulation, this solver–preconditioner combination resulted in a 25% decrease in solve time compared to MFiX’s native solver.

Enspara: Modeling molecular ensembles with scalable data structures and parallel computing

Markov state models (MSMs) are quantitative models of protein dynamics that are useful for uncovering the structural fluctuations that proteins undergo, as well as the mechanisms of these conformational changes. Given the enormity of conformational space, there has been ongoing interest in identifying a small number of states that capture the essential features of a protein. Generally, this is achieved by making assumptions about the properties of relevant features-for example, that the most important features are those that change slowly. An alternative strategy is to keep as many degrees of freedom as possible and subsequently learn from the model which of the features are most important. In these larger models, however, traditional approaches quickly become computationally intractable. In this paper, we present enspara, a library for working with MSMs that provides several novel algorithms and specialized data structures that dramatically improve the scalability of traditional MSM methods. This includes ragged arrays for minimizing memory requirements, message passing interface-parallelized implementations of compute-intensive operations, and a flexible framework for model construction and analysis.

A Simplified Description of Child Tables for Sequence Similarity Search.

Finding related nucleotide or protein sequences is a fundamental, diverse, and incompletely-solved problem in bioinformatics. It is often tackled by seed-and-extend methods, which first find "seed" matches of diverse types, such as spaced seeds, subset seeds, or minimizers. Seeds are usually found using an index of the reference sequence(s), which stores seed positions in a suffix array or related data structure. A child table is a fundamental way to achieve fast lookup in an index, but previous descriptions have been overly complex. This paper aims to provide a more accessible description of child tables, and demonstrate their generality: they apply equally to all the above-mentioned seed types and more. We also show that child tables can be used without LCP (longest common prefix) tables, reducing the memory requirement.

Terra Populus' Architecture for Integrated Big Geospatial Services.

Big geospatial data is an emerging sub-area of geographic information science, big data, and cyberinfrastructure. Big geospatial data poses two unique challenges to these and other cognate disciplines. First, raster and vector data structures and analyses have developed on largely separate paths for the last twenty years and this creates an impediment to researchers utilizing big data platforms that do not promote the integration for these classes. Second, big spatial data repositories have yet to be integrated with big data computation platforms in ways that allow researchers to spatio-temporally analyze big geospatial datasets. IPUMS-Terra, a National Science Foundation cyberInfrastructure project, begins to address these challenges. IPUMS-Terra is a spatial data infrastructure project that provides a unified framework for accessing, analyzing, and transforming big heterogeneous spatio-temporal data, and is part of the IPUMS (Integrated Public Use Microdata Series) data infrastructure. It supports big geospatial data analysis and provides integrated big geospatial services to its users. As IPUMS-Terra's data volume grows, we seek to integrate geospatial platforms that will scale geospatial analyses and address current bottlenecks within our system. However, our work shows that there are still unresolved challenges for big geospatial analysis. The most pertinent is that there is a lack of a unified framework for conducting scalable integrated vector and raster data analysis. We conducted a comparative analysis between PostgreSQL with PostGIS and SciDB and concluded that SciDB is the superior platform for scalable raster zonal analyses.

Homology-Aware Phylogenomics at Gigabase Scales.

Obstacles to inferring species trees from whole genome data sets range from algorithmic and data management challenges to the wholesale discordance in evolutionary history found in different parts of a genome. Recent work that builds trees directly from genomes by parsing them into sets of small n}{}k-mer strings holds promise to streamline and simplify these efforts, but existing approaches do not account well for gene tree discordance. We describe a “seed and extend” protocol that finds nearly exact matching sets of orthologous n}{}k-mers and extends them to construct data sets that can properly account for genomic heterogeneity. Exploiting an efficient suffix array data structure, sets of whole genomes can be parsed and converted into phylogenetic data matrices rapidly, with contiguous blocks of n}{}k-mers from the same chromosome, gene, or scaffold concatenated as needed. Phylogenetic trees constructed from highly curated rice genome data and a diverse set of six other eukaryotic whole genome, transcriptome, and organellar genome data sets recovered trees nearly identical to published phylogenomic analyses, in a small fraction of the time, and requiring many fewer parameter choices. Our method’s ability to retain local homology information was demonstrated by using it to characterize gene tree discordance across the rice genome, and by its robustness to the high rate of interchromosomal gene transfer found in several rice species.

VizGen

This paper introduces a novel domain-specific compiler, which translates visual computing programs written in dynamic languages to highly efficient code. We define "dynamic" languages as those such as Python and MATLAB, which feature dynamic typing and flexible array operations. Such language features can be useful for rapid prototyping, however, the dynamic computation model introduces significant overheads in program execution time. We introduce a compiler framework for accelerating visual computing programs, such as graphics and vision programs, written in generalpurpose dynamic languages. Our compiler allows substantial performance gains (frequently orders of magnitude) over general compilers for dynamic languages by specializing the compiler for visual computation. Specifically, our compiler takes advantage of three key properties of visual computing programs, which permit optimizations: (1) many array data structures have small, constant, or bounded size, (2) many operations on visual data are supported in hardware or are embarrassingly parallel, and (3) humans are not sensitive to small numerical errors in visual outputs due to changing floating-point precisions. Our compiler integrates program transformations that have been described previously, and improves existing transformations to handle visual programs that perform complicated array computations. In particular, we show that dependent type analysis can be used to infer sizes and guide optimizations for many small-sized array operations that arise in visual programs. Programmers who are not experts on visual computation can use our compiler to produce more efficient Python programs than if they write manually parallelized C, with fewer lines of application logic.

Analisis Kinerja Struktur Data Kd-Tree Pada Metode K-Nearest Neighbors

K-Nearest Neighbors is a commonly used classification technique that can be categorized into instance-based classification method. The performance of KNN is mostly determined by the size of the training data. This research compared and analyzed KD-Tree and Array data structures on KNN implementation. Dataset used in this research has large multidimensional features. From the experiment conducted we can conclude that KD-Tree data structure has better and relatively stable performance compared to Array data structure.

Limiting Self-Propagating Malware Based On Connection Failure Behavior Through Hyper-Compact Estimators

Self-propagating malware (e.g., an Internet worm) exploits security loopholes in software to infect servers and then use them to scan the Internet for more vulnerable servers. While the mechanisms of worm infection and their propagation models are well understood, defense against worms remains an open problem. One branch of defense research investigates the behavioral difference between worm-infected hosts and normal hosts to set them apart. One particular observation is that a worm-infected host, which scans the Internet with randomly selected addresses, has a much higher connection-failure rate than a normal host. Rate-limit algorithms have been proposed to control the spread of worms by traffic shaping based on connection failure rate. However, these rate-limit algorithms can work properly only if it is possible to measure failure rates of individual hosts efficiently and accurately. This paper points out a serious problem in the prior method. To address this problem, we first propose a solution based on a highly efficient double-bitmap data structure, which places only a small memory footprint on the routers, while providing good measurement of connection failure rates whose accuracy can be tuned by system parameters. Furthermore, we propose another solution based on shared register array data structure, achieving better memory efficiency and much larger estimation range than our double-bitmap solution.

Tree dependence analysis

We develop a new framework for analyzing recursive methods that perform traversals over trees, called tree dependence analysis. This analysis translates dependence analysis techniques for regular programs to the irregular space, identifying the structure of dependences within a recursive method that traverses trees. We develop a dependence test that exploits the dependence structure of such programs, and can prove that several locality- and parallelism- enhancing transformations are legal. In addition, we extend our analysis with a novel path-dependent, conditional analysis to refine the dependence test and prove the legality of transformations for a wider range of algorithms. We then use these analyses to show that several common algorithms that manipulate trees recursively are amenable to several locality- and parallelism-enhancing transformations. This work shows that classical dependence analysis techniques, which have largely been confined to nested loops over array data structures, can be extended and translated to work for complex, recursive programs that operate over pointer-based data structures.

A predictable hardware to exploit temporal reuse in real-time and embedded systems

In this paper we propose a new hardware data cache (FAFB, fully-associative FIFO tagged buffers) to complement the data cache in processors. It provides predictability when exploiting temporal reuse in array data structures, i.e. it allows an accurate WCET analysis, which is required in real-time systems. With our hardware proposal, compiler transformations that exploit such reuse (essentially tiling) can be safely applied. Moreover, our proposal has other features of particular interest to embedded systems, where a set of well-tuned applications run in a hardware platform which may be constrained in size, complexity and energy consumption. In order to test the most uncommon features of the FAFBs (predictability and effectiveness with a small size), we perform a worst-case analysis on several kernel algorithms for embedded and real-time computing, showing the interaction between tiling and our hardware architecture. Our results show that the number of data cache misses is reduced between 1.3 and 19 times on such algorithms.

Concordance: A Measure of Similarity Between Matrices of Time Series with Applications in Dendroclimatology

A fundamental assumption in dendroclimatology is that the common signal produced by multiple trees of the same species, growing under similar environmental conditions within the same climate region, relates to changes in the climate within that region in the same way. However, there are concerns that the climate response of young kauri trees may differ to older kauri. As a result, the inclusion of radii from young kauri may weaken the climate signal of the composite chronology. To address this concern, a subset of the data containing tree rings formed when the trees were young was compared to those formed when the trees were old. These subsets contained time series of correlated tree rings aligned by year with start and end years differing for each series. Existing techniques for comparing subsets of time series lack reliability for ragged arrays of dependent non-stationary time series. The concordance method was developed to overcome this. Concordance is a non-parametric method based on bootstrapping that is used to test the hypothesis that two subsets of time series are similar in terms of mean, variance or both. Simulations show that concordance is effective for detecting difference in both the level and scale of two submatrices containing non-stationary and dependent time series. When applied to tree-ring data, the concordance method was able to detect evidence against the subset of young tree rings having the same mean, variance or both than older, more established trees.

SFA-SPA: a suffix array based short peptide assembler for metagenomic data.

The determination of protein sequences from a metagenomic dataset enables the study of metabolism and functional roles of the organisms that are present in the sampled microbial community. We had previously introduced algorithm and software for the accurate reconstruction of protein sequences from short peptides identified on nucleotide reads in a metagenomic dataset. Here, we present significant computational improvements to the short peptide assembly algorithm that make it practical to reconstruct proteins from large metagenomic datasets containing several hundred million reads, while maintaining accuracy. The improved computational efficiency is achieved using a suffix array data structure that allows for fast querying during the assembly process, and a significant redesign of assembly steps that enables multi-threaded execution. The program is available under the GPLv3 license from sourceforge.net/projects/spa-assembler.

Fast construction of space‐optimized recursive automaton

SummaryFinite‐state automata are important components in information retrieval and natural language processing software. A recursive automaton is the most compact representation of the acyclic deterministic finite‐state automata. It is based on merging not only the equivalent states but also the identical substructures in an automaton. The LZ trie variant is the state‐of‐the‐art in automata compression regarding space, but the time needed for its construction was, until now, quadratic, which has made it impractical for large inputs. In this paper, we present the first algorithm for LZ trie construction that runs in effectively linear time thereby making it an attractive choice for finite‐state automata implementation. We achieve this goal by adding a new functionality to the enhanced suffix array data structure. We present two variants of the construction procedure – an optimal method regarding the final size and a method that sacrifices some compression for low intermediate memory usage. We have made the implementation of our algorithms available in an open source software package LzLex.Copyright © 2014 John Wiley & Sons, Ltd.

EXHAUSTIVE COMPUTATION OF EXACT DUPLICATIONS VIA SUPER AND NON-NESTED LOCAL MAXIMAL REPEATS

We propose and implement a method to obtain all duplicated sequences (repeats) from a chromosome or whole genome. Unlike existing approaches our method makes it possible to simultaneously identify and classify repeats into super, local, and non-nested local maximal repeats. Computation verification demonstrates that maximal repeats for a genome of several gigabases can be identified in a reasonable time, enabling us to identified these maximal repeats for any sequenced genome. The algorithm used for the identification relies on enhanced suffix array data structure to achieve practical space and time efficiency, to identify and classify the maximal repeats, and to perform further post-processing on the identified duplicated sequences. The simplicity and effectiveness of the implementation makes the method readily extendible to more sophisticated computations. Maxmers can be exhaustively accounted for in few minutes for genome sequences of dozen megabases in length and in less than a day or two for genome sequences of few gigabases in length. One application of duplicated sequence identification is to the study of duplicated sequence length distributions, which our found to exhibit for large lengths a persistent power-law behavior. Variation of estimated exponents of this power law are studied among different species and successive assembly release versions of the same species. This makes the characterization of the power-law regime of sequenced genomes via maximal repeats identification and classification, an important task for the derivation of models that would help us to elucidate sequence duplication and genome evolution.

A New Top-Down Context-Free Parsing for Syntactic Pattern Recognition

The numerous different mathematical methods used to solve pattern recognition snags may be assembled into two universal approaches:the decision-theoretic approach and the syntactic (structural) approach. In this paper,at first syntactic pattern recognition method and formal grammars are described and then has been investigated one of the techniques in syntactic pattern recognition called top –down tabular parser known as Earley’s algorithm Earley's tabular parser is one of the methods of context -free grammar parsing for syntactic pattern recognition. Earley's algorithm uses array data structure for implementing, which is the main problemand for this reasontakes a lots of time,searching in array and grammar parsing,and wasting lots of memory. In order to solve these problems and most important,the cubic time complexity,in this article,a new algorithm has been introduced,which reduces wasting the memory to zero, with using linked list data structure.Also, withthe changes in the implementation and performance of the algorithm,cubic time complexity has transformed into O (n*R) order.

Dynamic Generalized Suffix Arrays

Document retrieval is the basic task of search engines, and seize amount of attention by the pattern matching community. In this paper, we focused on the dynamic version of this problem, in which the text insertion and deletion is allowable. By using the generalized suffix array and other data structure, we proposed a new index structure. Our scheme achieved better time complexity than the existing ones, and a bit more space overhead is needed as return.

Practical Efficient String Mining

In recent years, several algorithms for mining frequent and emerging substring patterns from databases of string data (such as proteins and natural language texts) have been discovered, all of which traverse an enhanced suffix array data structure. All of these algorithms lie at either extreme of the efficiency spectrum; they are either fast and use enormous amounts of space, or they are compact and orders of magnitude slower. In this paper, we present an algorithm that achieves the best of both these extremes, having runtime comparable to the fastest published algorithms while using less space than the most space efficient ones. This excellent practical performance is underpinned by theoretical guarantees. Our main mechanism for keeping memory usage low is to build the enhanced suffix array incrementally, in blocks. Once built, a block is traversed to output patterns with required support before its space is reclaimed to be used for the next block.

RAPSearch2: a fast and memory-efficient protein similarity search tool for next-generation sequencing data

Summary: With the wide application of next-generation sequencing (NGS) techniques, fast tools for protein similarity search that scale well to large query datasets and large databases are highly desirable. In a previous work, we developed RAPSearch, an algorithm that achieved a ~20–90-fold speedup relative to BLAST while still achieving similar levels of sensitivity for short protein fragments derived from NGS data. RAPSearch, however, requires a substantial memory footprint to identify alignment seeds, due to its use of a suffix array data structure. Here we present RAPSearch2, a new memory-efficient implementation of the RAPSearch algorithm that uses a collision-free hash table to index a similarity search database. The utilization of an optimized data structure further speeds up the similarity search—another 2–3 times. We also implemented multi-threading in RAPSearch2, and the multi-thread modes achieve significant acceleration (e.g. 3.5X for 4-thread mode). RAPSearch2 requires up to 2G memory when running in single thread mode, or up to 3.5G memory when running in 4-thread mode.Availability and implementation: Implemented in C++, the source code is freely available for download at the RAPSearch2 website: http://omics.informatics.indiana.edu/mg/RAPSearch2/.Contact: yye@indiana.eduSupplementary information: Available at the RAPSearch2 website.