Special Data Structure Research Articles

Information Retrieval: Textual Indexing Using an Oriented Object Database

The growth in the volume of text data such as books and articles in libraries for centuries has imposed to establish effective mechanisms to locate them. Early techniques such as abstraction, indexing and the use of classification categories have marked the birth of a new field of research called "Information Retrieval". Information Retrieval (IR) can be defined as the task of defining models and systems whose purpose is to facilitate access to a set of documents in electronic form (corpus) to allow a user to find the relevant ones for him, that is to say, the contents which matches with the information needs of the user. Most of the models of information retrieval use a specific data structure to index a corpus which is called "inverted file" or "reverse index". This inverted file collects information on all terms over the corpus documents specifying the identifiers of documents that contain the term in question, the frequency of each term in the documents of the corpus, the positions of the occurrences of the word. In this paper we use an oriented object database (db4o) instead of the inverted file, that is to say, instead to search a term in the inverted file, we will search it in the db4o database. The purpose of this work is to make a comparative study to see if the oriented object databases may be competing for the inverse index in terms of access speed and resource consumption using a large volume of data.

JiTTree: A Just-in-Time Compiled Sparse GPU Volume Data Structure.

Sparse volume data structures enable the efficient representation of large but sparse volumes in GPU memory for computation and visualization. However, the choice of a specific data structure for a given data set depends on several factors, such as the memory budget, the sparsity of the data, and data access patterns. In general, there is no single optimal sparse data structure, but a set of several candidates with individual strengths and drawbacks. One solution to this problem are hybrid data structures which locally adapt themselves to the sparsity. However, they typically suffer from increased traversal overhead which limits their utility in many applications. This paper presents JiTTree, a novel sparse hybrid volume data structure that uses just-in-time compilation to overcome these problems. By combining multiple sparse data structures and reducing traversal overhead we leverage their individual advantages. We demonstrate that hybrid data structures adapt well to a large range of data sets. They are especially superior to other sparse data structures for data sets that locally vary in sparsity. Possible optimization criteria are memory, performance and a combination thereof. Through just-in-time (JIT) compilation, JiTTree reduces the traversal overhead of the resulting optimal data structure. As a result, our hybrid volume data structure enables efficient computations on the GPU, while being superior in terms of memory usage when compared to non-hybrid data structures.

Massively parallel adaptive mesh refinement and coarsening for dynamic fracture simulations

We use the graphical processing unit (GPU) to perform dynamic fracture simulation using adaptively refined and coarsened finite elements and the inter-element cohesive zone model. Due to the limited memory available on the GPU, we created a specialized data structure for efficient representation of the evolving mesh given. To achieve maximum efficiency, we perform finite element calculation on a nodal basis (i.e., by launching one thread per node and collecting contributions from neighboring elements) rather than by launching threads per element, which requires expensive graph coloring schemes to avoid concurrency issues. These developments made possible the parallel adaptive mesh refinement and coarsening schemes to systematically change the topology of the mesh. We investigate aspects of the parallel implementation through microbranching examples, which has been explored experimentally and numerically in the literature. First, we use a reduced-scale version of the experimental specimen to demonstrate the impact of variation in floating point operations on the final fracture pattern. Interestingly, the parallel approach adds some randomness into the finite element simulation on the structured mesh in a similar way as would be expected from a random mesh. Next, we take advantage of the speedup of the implementation over a similar serial implementation to simulate a specimen whose size matches that of the actual experiment. At this scale, we are able to make more direct comparisons to the original experiment and find excellent agreement with those results.

An Execution Framework for Grid-clustering Methods

Cluster analysis methods have proven extremely valuable for explorative data analysis and are also fundamental for data mining methods. Goal of cluster analysis is to find correlations of the value space and to separate the data values into a priori unknown set of subgroups based on a similarity metrics. In case of high dimensional data, i.e. data with a large number of describing attributes, clustering can result into a very time consuming task, which often limits the number of observations to be clustered in practice. To overcome this problem, Grid clustering methods have been developed, which do not calculate similarity values between the data value each, but organize the value space surrounding the data values, e.g. by specific data structure indices. In this paper we present a framework which allows to evaluate different data structures for the generation of a multi-dimensional grid structure grouping the data values into blocks. The values are then clustered by a topological neighbor search algorithm on the basis of the block structure. As first data structure to be evaluated we present the BANG file structure and show its feasibility as clustering index. The developed framework is planned to be contributed as package to the WEKA software.

Joint components based pedestrian detection in crowded scenes using extended feature descriptors

This paper presents a framework for human detection based on the joint component model using extended feature descriptors. This framework provides two contributions for handling the partially occluded problem of pedestrian detection in crowded environments. First, it presents feature descriptors based on extension of two well-known feature descriptors for accommodation in partially occluded pedestrian detection. The feature descriptors, which use multiple scale blocks-based histograms of oriented gradients (MHOG) and parallelogram based Haar-like feature (PHF), are proposed for improving the accuracy of the detection system. As a result of using MHOG, an extensive feature space allows for the obtaining of highly discriminated features, which supports robust detection. On the other hand, the PHF is adaptive for the shape of human limbs in detection of pedestrians. This contribution also presents special data structures to store image intensities and image gradients for using an integral image method, which is helpful for fast computation for both feature descriptors. Second, the joint component model for human detection is proposed for the training and detecting of pedestrians in crowded scenes. The proposed detection method based on fusion of the boosting technique and the support vector machine (SVM) is presented. The SVM is known as one of the most efficient learning models for classification. The advantage of boosting is a strong classifier based on a combination set of weak classifiers. However, the performance of boosting depends on the kernel of element classifier. The method based on the extended feature descriptors and the joint component model is helpful for constructing an efficient classification. The experimental results demonstrate that this method for pedestrian detection outperforms current methods in crowded situations under a variety of outdoor environments.

Maximal Covering Location Problems on networks with regional demand

Covering problems are well studied in the Operations Research literature under the assumption that both the set of users and the set of potential facilities are finite. In this paper, we address the following variant, which leads to a Mixed Integer Nonlinear Program (MINLP): locations of p facilities are sought along the edges of a network so that the expected demand covered is maximized, where demand is continuously distributed along the edges. This MINLP has a combinatorial part (which edges of the network are chosen to contain facilities) and a continuous global optimization part (once the edges are chosen, which are the optimal locations within such edges).A branch-and-bound algorithm is proposed, which exploits the structure of the problem: specialized data structures are introduced to successfully cope with the combinatorial part, inserted in a geometric branch-and-bound algorithm.Computational results are presented, showing the appropriateness of our procedure to solve covering problems for small (but non-trivial) values of p.

An evolutionary local search for the capacitated vehicle routing problem minimizing fuel consumption under three-dimensional loading constraints

This study introduces a new practical variant of the combined routing and loading problem called the capacitated vehicle routing problem minimizing fuel consumption under three-dimensional loading constraints (3L-FCVRP). It presents a meta-heuristic algorithm for solving the problem. The aim is to design routes for a fleet of homogeneous vehicles that will serve all customers, whose demands are formed by a set of three-dimensional, rectangular, weighted items. Unlike the well-studied capacitated vehicle routing problem with 3D loading constraints (3L-CVRP), the objective of the 3L-FCVRP is to minimize total fuel consumption rather than travel distance. The fuel consumption rate is assumed to be proportionate to the total weight of the vehicle. A route is feasible only if a feasible loading plan to load the demanded items into the vehicle exists and the loading plan must satisfy a set of practical constraints.To solve this problem, the evolutionary local search (ELS) framework incorporating the recombination method is used to explore the solution space, and a new heuristic based on open space is used to examine the feasibility of the solutions. In addition, two special data structures, Trie and Fibonacci heap, are adopted to speed up the procedure. To verify the effectiveness of our approach, we first test the ELS on the 3L-CVRP, which can be seen as a special case of the 3L-FCVRP. The results demonstrate that on average ELS outperforms all of the existing approaches and improves the best-known solutions for most instances. Then, we generate data for 3L-FCVRP and report the detailed results of the ELS for future comparisons.

Computational Cost Reduction of Nondominated Sorting Using the M-Front

Many multiobjective evolutionary algorithms rely on the nondominated sorting procedure to determine the relative quality of individuals with respect to the population. In this paper, we propose a new method to decrease the cost of this procedure. Our approach is to determine the nondominated individuals at the start of the evolutionary algorithm run and to update this knowledge as the population changes. In order to do this efficiently, we propose a special data structure called the M-front, to hold the nondominated part of the population. The M-front uses the geometric and algebraic properties of the Pareto dominance relation to convert orthogonal range queries into interval queries using a mechanism based on the nearest neighbor search. These interval queries are answered using dynamically sorted linked lists. Experimental results show that our method can perform significantly faster than the state-of-the-art Jensen–Fortin’s algorithm, especially in many-objective scenarios. A significant advantage of our approach is that, if we change a single individual in the population we still know which individuals are dominated and which are not.

Variable latency L1 data cache architecture design in multi-core processor under process variation

In this paper, we propose a new variable latency L1 data cache architecture for multi-core processors. Our proposed architecture extends the traditional variable latency cache to be geared toward the multi-core processors. We added a specialized data structure for recording the latency of the L1 data cache. Depending on the added latency to the L1 data cache, the value stored to the data structure is determined. It also tracks the remaining cycles of the L1 data cache which notifies data arrival to the reservation station in the core. As in the variable latency cache of the single-core architecture, our proposed architecture flexibly extends the cache access cycles considering process variation. The proposed cache architecture can reduce yield losses incurred by L1 cache access time failures to nearly 0%. Moreover, we quantitatively evaluate performance, power, energy consumption, power-delay product, and energy-delay product when increasing the number of cache access cycles.

Boosting the FM-Index on the GPU: Effective Techniques to Mitigate Random Memory Access.

The recent advent of high-throughput sequencing machines producing big amounts of short reads has boosted the interest in efficient string searching techniques. As of today, many mainstream sequence alignment software tools rely on a special data structure, called the FM-index, which allows for fast exact searches in large genomic references. However, such searches translate into a pseudo-random memory access pattern, thus making memory access the limiting factor of all computation-efficient implementations, both on CPUs and GPUs. Here, we show that several strategies can be put in place to remove the memory bottleneck on the GPU: more compact indexes can be implemented by having more threads work cooperatively on larger memory blocks, and a k-step FM-index can be used to further reduce the number of memory accesses. The combination of those and other optimisations yields an implementation that is able to process about two Gbases of queries per second on our test platform, being about 8 × faster than a comparable multi-core CPU version, and about 3 × to 5 × faster than the FM-index implementation on the GPU provided by the recently announced Nvidia NVBIO bioinformatics library.

Functional pearl: a SQL to C compiler in 500 lines of code

We present the design and implementation of a SQL query processor that outperforms existing database systems and is written in just about 500 lines of Scala code -- a convincing case study that high-level functional programming can handily beat C for systems-level programming where the last drop of performance matters. The key enabler is a shift in perspective towards generative programming. The core of the query engine is an interpreter for relational algebra operations, written in Scala. Using the open-source LMS Framework (Lightweight Modular Staging), we turn this interpreter into a query compiler with very low effort. To do so, we capitalize on an old and widely known result from partial evaluation known as Futamura projections, which state that a program that can specialize an interpreter to any given input program is equivalent to a compiler. In this pearl, we discuss LMS programming patterns such as mixed-stage data structures (e.g. data records with static schema and dynamic field components) and techniques to generate low-level C code, including specialized data structures and data loading primitives.

Associated types and constraint propagation for generic programming in Scala

Generic programming is a programming paradigm for creation of highly resuable software components through decoupling algorithms from specific data structures which are being processed. The rise of research on ways of handling generic programming in various programming languages took place last years. We analyze and develop a number of generic programming features, in particular associated types and constraint propagation, for the Scala programming language designed by Martin Odersky in Ecole Polytechnique Federale de Lausanne.

Efficient image-aware version control systems using GPU

Version control is considered to be a vital component for supporting professional software development. While it has been widely used for textual artifacts, such as source code or documentation, little attention has been given to binary artifacts. This omission can place huge restrictions on projects in the game and media industries as they contain large amounts of binary data, such as images, videos, three-dimensional models, and animations, along with their source code. For these kinds of artifacts, existing strategies such as storing the file as a whole for each revision or saving conventional binary deltas consume significant storage space with duplicate data and, even worse, do not provide any understandable information on which modifications were made. As a response to this problem, this paper introduces a change-set model infrastructure to support version control of image artifacts using a specialized data structure. Additionally, our approach can deal with the maintenance of duplicate nearly identical images through a merge operation. Because of the amount of data that has to be processed, we designed our solution based on a parallel architecture, which permits a massively parallel approach to version control. The paper also compares our approach with some popular open-source version control systems, showing their repository growth in relation to ours as well as the time required to process image artifacts. Finally, we demonstrate that our architecture requires less storage space and runs much faster than current methods. Copyright © 2015 John Wiley & Sons, Ltd.

A High Efficient and Real Time Data Aggregation Scheme for WSNs

Data aggregation is an important method of improving transmitting efficiency of WSNs, but existing researches have some disadvantages as follows: several periods delay will be generated when filtering messages; the reduplicated messages filtering ratio is low; complex calculations need to be executed; and extra hardware should be added to obtain high performance. To resolve these problems, this paper proposes a real time and high efficient data aggregation scheme (dynamical message list based data aggregation, DMLDA) based on clustering routing algorithm. DMLDA includes three procedures: activating nodes, clustering nodes, and filtering messages. In filtering procedure, a special data structure named dynamic list will be established in every filtering node; it is designed to store messages transmitted by filtering node, and, comparing current message with list's items, the message's redundancy can be judged without any delay. To improve the filtering efficiency, the message list can be adjusted dynamically. The three procedures of data aggregation are all introduced, and the filtering method is designed in detail in this paper. At last, a series of experiments are simulated to prove our scheme's performance, and the advantages are analyzed in theory.

Mechanical model of geometric cell and topological algorithm for cell dynamics from single-cell to formation of monolayered tissues with pattern.

Geometric and mechanical properties of individual cells and interactions among neighboring cells are the basis of formation of tissue patterns. Understanding the complex interplay of cells is essential for gaining insight into embryogenesis, tissue development, and other emerging behavior. Here we describe a cell model and an efficient geometric algorithm for studying the dynamic process of tissue formation in 2D (e.g. epithelial tissues). Our approach improves upon previous methods by incorporating properties of individual cells as well as detailed description of the dynamic growth process, with all topological changes accounted for. Cell size, shape, and division plane orientation are modeled realistically. In addition, cell birth, cell growth, cell shrinkage, cell death, cell division, cell collision, and cell rearrangements are now fully accounted for. Different models of cell-cell interactions, such as lateral inhibition during the process of growth, can be studied in detail. Cellular pattern formation for monolayered tissues from arbitrary initial conditions, including that of a single cell, can also be studied in detail. Computational efficiency is achieved through the employment of a special data structure that ensures access to neighboring cells in constant time, without additional space requirement. We have successfully generated tissues consisting of more than 20,000 cells starting from 2 cells within 1 hour. We show that our model can be used to study embryogenesis, tissue fusion, and cell apoptosis. We give detailed study of the classical developmental process of bristle formation on the epidermis of D. melanogaster and the fundamental problem of homeostatic size control in epithelial tissues. Simulation results reveal significant roles of solubility of secreted factors in both the bristle formation and the homeostatic control of tissue size. Our method can be used to study broad problems in monolayered tissue formation. Our software is publicly available.

RViz: a toolkit for real domain data visualization

In computational science and computer graphics, there is a strong requirement to represent and visualize information in the real domain, and many visualization data structures and algorithms have been proposed to achieve this aim. Unfortunately, the dataflow model that is often selected to address this issue in visualization systems is not flexible enough to visualize newly invented data structures and algorithms because this scheme can accept only specific data structures. To address this problem, we propose a new visualization tool, RViz, which is independent of the input information data structures. Since there is no requirement for additional efforts to manage the flow networks and the interface to abstracted information is simple in RViz, any scientific information visualization algorithms are easier to implement than the dataflow model. In this paper, we provide case studies in which we have successfully implemented new data structures and related algorithms using RViz, including geometry synthesis, distance field representation, and implicit surface reconstruction. Through these cases, we show how RViz helps users visualize and understand any hidden insights in input information.

Efficiently computing the drainage network on massive terrains using external memory flooding process

We present EMFlow, a very efficient algorithm and its implementation, to compute the drainage network (i.e. the flow direction and flow accumulation) on huge terrains stored in external memory. Its utility lies in processing the large volume of high resolution terrestrial data newly available, which internal memory algorithms cannot handle efficiently. The flow direction is computed using an adaptation of our previous method RWFlood that uses a flooding process to quickly remove internal depressions or basins. Flooding, proceeding inward from the outside of the terrain, works oppositely to the common method of computing downhill flow from the peaks. To reduce the number of I/O operations, EMFlow adopts a new strategy to subdivide the terrain into islands that are processed separately. The terrain cells are grouped into blocks that are stored in a special data structure managed as a cache memory. EMFlow's execution time was compared against the two most recent and most efficient published methods: TerraFlow and r.watershed.seg. It was, on average, 25 and 110 times faster than TerraFlow and r.watershed.seg respectively. Also, EMFlow could process larger datasets. Processing a 50000 × 50000 terrain on a machine with 2GB of internal memory took about 4500 seconds, compared to 87000 seconds for TerraFlow while r.watershed.seg failed on terrains larger than 15000 ×15000. On very small, say1000 ×1000 terrains, EMFlow takes under a second, compared to 6 and 20 seconds in r.watershed.seg and TerraFlow respectively. So EMFlow could be a component of a future interactive system where a user could modify terrain and immediately see the new hydrography.

Approximation of function evaluation over sequence arguments via specialized data structures

This paper proposes strategies for maintaining a database of computational results of functions f on sequence arguments x→, where x→ is sorted in non-decreasing order and f(x→) has greatest dependence on the first few terms of x→. This scenario applies also to symmetric functions f, where the partial derivatives approach zero as the corresponding component value increases. The goal is to pre-compute exact values f(u→) on a tight enough net of sequence arguments, so that given any other sequence x→, a neighboring sequence u→ in the net giving a close approximation can be efficiently found. Our scheme avoids pre-computing the more-numerous partial-derivative values. It employs a new data structure that combines ideas of a trie and an array implementation of a heap, representing grid values compactly in the array, yet still allowing access by a single index lookup rather than pointer jumping. We demonstrate good size/approximation performance in a natural application.

On the detection of custom memory allocators in C binaries

Many reverse engineering techniques for data structures rely on the knowledge of memory allocation routines. Typically, they interpose on the system's malloc and free functions, and track each chunk of memory thus allocated as a data structure. However, many performance-critical applications implement their own custom memory allocators. Examples include webservers, database management systems, and compilers like gcc and clang. As a result, current binary analysis techniques for tracking data structures fail on such binaries. We present MemBrush, a new tool to detect memory allocation and deallocation functions in stripped binaries with high accuracy. We evaluated the technique on a large number of real world applications that use custom memory allocators. We demonstrate that MemBrush can detect allocators/deallocators with a high accuracy which is 52 out of 59 for allocators, and 29 out of 31 for deallocators in SPECINT 2006. As we show, we can furnish existing reverse engineering tools with detailed information about the memory management API, and as a result perform an analysis of the actual application specific data structures designed by the programmer. Our system uses dynamic analysis and detects memory allocation and deallocation routines by searching for functions that comply with a set of generic characteristics of allocators and deallocators.

Transforming XML documents to OWL ontologies: A survey

The aims of XML data conversion to ontologies are the indexing, integration and enrichment of existing ontologies with knowledge acquired from these sources. The contribution of this paper consists in providing a classification of the approaches used for the conversion of XML documents into OWL ontologies. This classification underlines the usage profile of each conversion method, providing a clear description of the advantages and drawbacks belonging to each method. Hence, this paper focuses on two main processes, which are ontology enrichment and ontology population using XML data. Ontology enrichment is related to the schema of the ontology (TBox), and ontology population is related to an individual (Abox). In addition, the ontologies described in these methods are based on formal languages of the Semantic Web such as OWL (Ontology Web Language) or RDF (Resource Description Framework). These languages are formal because the semantics are formally defined and take advantage of the Description Logics. In contrast, XML data sources are without formal semantics. The XML language is used to store, export and share data between processes able to process the specific data structure. However, even if the semantics is not explicitly expressed, data structure contains the universe of discourse by using a qualified vocabulary regarding a consensual agreement. In order to formalize this semantics, the OWL language provides rich logical constraints. Therefore, these logical constraints are evolved in the transformation of XML documents into OWL documents, allowing the enrichment and the population of the target ontology. To design such a transformation, the current research field establishes connections between OWL constructs (classes, predicates, simple or complex data types, etc.) and XML constructs (elements, attributes, element lists, etc.). Two different approaches for the transformation process are exposed. The instance approaches are based on XML documents without any schema associated. The validation approaches are based on the XML schema and document validated by the associated schema. The second approaches benefit from the schema definition to provide automated transformations with logic constraints. Both approaches are discussed in the text.