Storage Schema Research Articles

A SPARQL query processing system using map-phase-multi join for big data in clouds

Big data refers to large datasets which are huge, complex and hard to be stored and analysed by traditional data processing tools. Linked data is one of the approaches to deal with big data which are stored and processed in TripleStore. For huge dataset, TripleStore requires more scalable techniques. 'MapReduce' programming model is the most representative of cloud technology. There are several approaches using MapReduce to serve SPARQL query but still exhibit unacceptable performance in complex queries. In this paper, we propose a map-phase-multi-join algorithm for processing SPARQL queries. Using multi-join, job initialisation time is reduced by avoiding iterative of MapReduce jobs. Furthermore, map-phase join can save bandwidth by preventing join-less data to be transferred among computing nodes. We also design a storage schema and a join-order rule which enhance the performance of our system. The evaluation results show that our system outperforms traditional join approaches in most queries.

NoSQL databases for big data

NoSQL solutions have been created to respond to many issues encountered when dealing with some specific applications, e.g., storage of very large datasets. In fact, traditional RDMS ensure data integrity and transaction consistency. But, this is at the cost of a rigid storage schema and a complex management. Certainly, data integrity and consistency are required in many cases like in financial applications but they are not always needed. The goal of this paper is to establish a precise picture about NoSQL's evolution and mechanisms as well as the advantages and disadvantages of the main NoSQL data models and frameworks. For this purpose, first, a deep comparison between SQL and NoSQL databases is presented. Many criteria are examined such as: scalability, performance, consistency, security, analytical capabilities and fault-tolerance mechanisms. Second, the four major types of NoSQL databases are defined and compared: key-value stores, document databases, column-oriented databases and graph databases. Third, we compare for each NoSQL data model the main available technical solutions.

A systematic view on data descriptors for the visual analysis of tabular data

Visualization has become an important ingredient of data analysis, supporting users in exploring data and confirming hypotheses. At the beginning of a visual data analysis process, data characteristics are often assessed in an initial data profiling step. These include, for example, statistical properties of the data and information on the data’s well-formedness, which can be used during the subsequent analysis to adequately parametrize views and to highlight or exclude data items. We term this information data descriptors, which can span such diverse aspects as the data’s provenance, its storage schema, or its uncertainties. Gathered descriptors encapsulate basic knowledge about the data and can thus be used as objective starting points for the visual analysis process. In this article, we bring together these different aspects in a systematic form that describes the data itself (e.g. its content and context) and its relation to the larger data gathering and visual analysis process (e.g. its provenance and its utility). Once established in general, we further detail the concept of data descriptors specifically for tabular data as the most common form of structured data today. Finally, we utilize these data descriptors for tabular data to capture domain-specific data characteristics in the field of climate impact research. This procedure from the general concept via the concrete data type to the specific application domain effectively provides a blueprint for instantiating data descriptors for other data types and domains in the future.

A MongoDB-Based Management of Planar Spatial Data with a Flattened R-Tree

This paper addresses how to manage planar spatial data using MongoDB, a popular NoSQL database characterized as a document-oriented, rich query language and high availability. The core idea is to flatten a hierarchical R-tree structure into a tabular MongoDB collection, during which R-tree nodes are represented as collection documents and R-tree pointers are expressed as document identifiers. By following this strategy, a storage schema to support R-tree-based create, read, update, and delete (CRUD) operations is designed and a module to manage planar spatial data by consuming and maintaining flattened R-tree structure is developed. The R-tree module is then seamlessly integrated into MongoDB, so that users could manipulate planar spatial data with existing command interfaces oriented to geodetic spatial data. The experimental evaluation, using real-world datasets with diverse coverage, types, and sizes, shows that planar spatial data can be effectively managed by MongoDB with our flattened R-tree and, therefore, the application extent of MongoDB will be greatly enlarged. Our work resulted in a MongoDB branch with R-tree support, which has been released on GitHub for open access.

Reliable and Secure Distributed Cloud Data Storage Using Reed-Solomon Codes

Despite the popularity and many advantages of using cloud data storage, there are still major concerns about the data stored in the cloud, such as security, reliability and confidentiality. In this paper, we propose a reliable and secure distributed cloud data storage schema using Reed-Solomon codes. Different from existing approaches to achieving data reliability with redundancy at the server side, our proposed mechanism relies on multiple cloud service providers (CSP), and protects users’ cloud data from the client side. In our approach, we view multiple cloud-based storage services as virtual independent disks for storing redundant data encoded with erasure codes. Since each CSP has no access to a user’s complete data, the data stored in the cloud would not be easily compromised. Furthermore, the failure or disconnection of a CSP will not result in the loss of a user’s data as the missing data pieces can be readily recovered. To demonstrate the feasibility of our approach, we developed a prototype distributed cloud data storage application using three major CSPs. The experimental results show that, besides the reliability and security related benefits of our approach, the application outperforms each individual CSP for uploading and downloading files.

Storage schema and ontology-independent SPARQL to HiveQL translation

Growing size of Semantic Web data demands scalable semantic stores. Hadoop-based distributed and parallel processing frameworks such as HBase and Hive are becoming increasingly popular for storing ...

An Optimized Splitting Attribute Algorithm for Inconsistent Conflict in Context Lattice

With the emergence of Cloud computing and Internet of Things, Context-aware applications face new challenges. One of them is big data from huge context application and sources. The main stream of applications have used not only realtime versions but also history versions of context data. This paper concerned about optimization techniques of storage and reasoning in the CMS (context management system). For our storage of context data from different sources, FCA Lattice has been employed as a kind of storage schema to support modeling and fusion of these different context data. Further, context conditions about data are essential to logical reasoning. Under different context conditions, context data can be promoted to be knowledge, which makes context reasoning readily. In the dynamic environment, to get reasonable results, reasoning services require their input to keep consistent in the changeable conditions. The changeable conditions can be represented as context attributes, intervals and relations etc. To make consistent knowledge available in the conditions, our pervious works have analyzed incremental cache and check of consistent intervals, and proposed a context lattice-based distributed optimized update algorithm. In this paper, based on the algorithm, our problem is to optimize the split function. The split is needed when current lattice has no condition making knowledge consistent. The main aim of this paper is to improve time performance of splitting attributes or intervals or fuzzy relations that could be detailed. We propose a new parallel split algorithm. This algorithm computes the priorities of candidates. To reduce time cost, it decreases the split scope by choosing the split candidate with the highest priority value. To decrease the full lattice update time in the split process, it generates the sub lattices split by the candidates concurrently and merges them after. On the theory, we analyze the feasibility of the algorithm. On the test, as a new part of the whole update algorithm, it is compared with the nai ve one, and it shows the better time performance. What’s more, it makes multi-threads execute on the same lattice to avoid producing more memory cost caused by copying the lattice for an independent thread.

Study and Implementation of Index Mechanism for GML Data

GML is the encoding specification of geospatial data, more and more geospatial data are represented in GML documents. It’s efficient to use database technologies to manage and maintain the geospatial data described by GML. The index in a database can help to locate the tuples for a query. In order to query GML data efficiently, appropriate index structure should be designed. In this paper, a method to build path index for the GML document is proposed based on its storage schema. The path index records the sequences and levels of elements in the GML document. Tuples stored in tables can be quickly located for a query with the path index, and the structure information recorded in the path index can help to reconstruct the GML document.

Adaptive Distributed Data Storage for Context-Aware Applications

Context-aware computing is a paradigm that relies on the active use of information coming from a variety of sources, ranging from smartphones to sensors. The paradigm usually leads to storing large volumes of data that need to be processed to derive higher-level context information. The paper presents a cloud-based storage layer for managing sensitive context data. To handle the storage and aggregation of context data for context-aware applications, Clouds are perfect candidates. But a Cloud platform for context-aware computing needs to cope with several requirements: high concurrent access (all data needs to be available to potentially a large number of users), mobility support (such platform should actively use the caches on mobile devices whenever possible, but also cope with storage size limitations), real-time access guarantees – local caches should be located closer to the end-user whenever possible, and persistency (for traceability, a history of the context data should remain available). BlobSeer, a framework for Cloud data storage, is a perfect candidate for storing context data for large-scale applications. It offers capabilities such as persistency, concurrency and support for flexible storage schema requirement. On top of BlobSeer, Context Aware Framework is designed as an extension that offers context-aware data management to higherlevel applications, and enables scalable high-throughput under high-concurrency. On a logical level, the most important capabilities offered by Context Aware Framework are transparency, support for mobility, real-time guarantees and support for access based on meta-information. On the physical layer, the most important capability is persistent Cloud storage.

Research on Vector Spatial Data Storage Schema Based on Hadoop Platform

Cloud computing technology is changing the mode of the spatial information industry which is applied and provides new ideas for it. Since Hadoop platform provides easy expansion, high performance, high fault tolerance and other advantages, we propose a novel vector spatial data storage schema based on it to solve the problems on how to use cloud computing technology to directly manage spatial data and present data topological relations. Firstly, vector spatial data storage schema is designed based on column-oriented storage structures and key/value mapping to express spatial topological relations. Secondly, we design middleware and merge with vector spatial data storage schema in order to directly store spatial data and present geospatial data access refinement schemes based on GeoTools toolkit. Thirdly, we verify the middleware and the data storage schema through Hadoop cluster experiments. Comprehensive experiments demonstrate that our proposal is efficient and applicable to directly storing large-scale vector spatial data and timely express spatial topological relations.

Effective Hierarchical Information Management in Mobile Environment

Problem statement: As performance of mobile devices is developed highly, several kinds of data is stored in mobile devices. For effective data management and information retrieval, some researches to apply ontology concept to mobile devices are progressed. However, in conventional researches, they apply conventional ontology storage structure used in PC environment to mobile platform. Conclusion/Recommendations: Therefore, performance of search about data is low and not effective. Therefore, we suggested new ontology storage schema with ontology path for effective hierarchical information in mobile environment.

OWL 문서의 변경 탐지 및 관리 기법

정보 자원에 대한 정확한 검색을 위해서는 점진적으로 변화하는 온톨로지의 변경 내용을 효율적으로 관리할 필요가 있다. 최근에는 OWL을 이용해 온톨로지를 기술하는 경우가 많기 때문에 OWL 문서로 작성된 온톨로지의 변경 내용을 관리할 수 있는 기법이 필요하다. 이러한 요구에 따라 본 논문에서는 OWL로 기술된 온톨로지의 변화를 탐지할 수 있도록 변경 가능한 요소를 분류하고 각 요소의 특성에 맞게 변경 내용을 관리할 수 있는 저장 스키마를 제안한다. 그리고 온톨로지 버전별로 클래스나 프로퍼티에 대한 정보를 제공하는 뷰를 이용하여 질의 처리 성능을 개선할 수 있는 가능성을 제시한다. 본 논문에서 제안한 저장 스키마는 각 온톨로지 버전과 관련된 메타데이터의 변경 내용도 함께 저장한다. 또한 온톨로지가 변경되면 자동적으로 추가 또는 삭제되어야 하는 메타데이터의 내용도 추론을 통해 관리할 수 있다. 따라서 본 논문에서 제안한 저장 스키마를 이용하면 온톨로지의 변경 이력에 대한 검색이 가능하고 사용자가 선택한 온톨로지 버전에 맞는 정확하고 유효한 메타데이터를 제공할 수 있다. For accurate search on information resources, it is needed to manage gradual changes in ontology efficiently. Recently, because ontology is often written using OWL, techniques that can manage changes in OWL documents are required. To meet these needs, in this paper, we classify changeable elements to detect changes in OWL ontology and propose a storage schema that can manage the changes according to the characteristics of each element. And we suggest the possibility of improving performance of query processing using views that provide information about classes or properties in each ontology version. The proposed storage schema stores changes in metadata associated with each ontology version. In addition, it can manage metadata that must be added or deleted through reasoning when ontology changes. So, the proposed storage schema can support queries about history of changes in ontology and provide accurate and valid metadata that is suitable for user-selected ontology version.

Schema Evolution via Multi-Version Metadata in SaaS

The Basic-Table combined with Extension-Table (BT&ET) layout has become the popular data storage architecture for SaaS currently. For the sake of improving the processing efficiency we improve the BT&ET layout, and migrate some tenants’ frequently accessed extension fields into the basic table based on the tenants’ constantly need on data access. With the development of cloud computing, Multi-Tenant data need to be stored in multiple data nodes. When tenant's data storage schema evolution happens, all data nodes hava to do data migration simultaneously and tenants may perceive the influence. In order to minimize the costs and negative load of schema evolution we propose the Multi-Version metadata technology. Tenant's data on each data node may contain different version metadata, and schema envolution can run asynchronously. We carry out experiments and the results figure out that the workload decreases significantly.

Efficient and self-adaptive storage schema in flash-based database system

It has been a new approach that flash memory is used as storage media to improve database system performance.In the current flash database system,to solve the problems of low search performance,improper log region allocation,and high index update cost in storage management,a forecasting algorithm based on the latest version of Bloom Filter was proposed,record locator and log summary structure were introduced,and a self-adaptive mechanism based on flash search and update cost estimate model were given.The experimental results prove that it can make a proper log region allocation,efficiently improve search performance,and reduce non-clustered index update cost.

Pareto-Based Dominant Graph: An Efficient Indexing Structure to Answer Top-K Queries

Given a record set D and a query score function F, a top-k query returns k records from D, whose values of function F on their attributes are the highest. In this paper, we investigate the intrinsic connection between top-k queries and dominant relationships between records, and based on which, we propose an efficient layer-based indexing structure, Pareto-Based Dominant Graph (DG), to improve the query efficiency. Specifically, DG is built offline to express the dominant relationship between records and top-k query is implemented as a graph traversal problem, i.e., Traveler algorithm. We prove theoretically that the size of search space (that is the number of retrieved records from the record set to answer top-k query) in our algorithm is directly related to the cardinality of skyline points in the record set (see Theorem 3). Considering I/O cost, we propose cluster-based storage schema to reduce I/O cost in Traveler algorithm. We also propose the cost estimation methods in this paper. Based on cost analysis, we propose an optimization technique, pseudorecord, to further improve the search efficiency. In order to handle the top-k query in the high-dimension record set, we also propose N-Way Traveler algorithm. In order to handle DG maintenance efficiently, we propose “Insertion” and “Deletion” algorithms for DG. Finally, extensive experiments demonstrate that our proposed methods have significant improvement over its counterparts, including both classical and state art of top-k algorithms.

A Prediction-based Energy-conserving Approximate Storage and Query Processing Schema in Object-Tracking Sensor Networks

Energy efficiency is one of the most critical issues in the design of wireless sensor networks. In object-tracking sensor networks, the data storage and query processing should be energy-conserving by decreasing the message complexity. In this paper, a Prediction-based Energy-conserving Approximate StoragE schema (P-EASE) is proposed, which can reduce the query error of EASE by changing its approximate area and adopting predicting model without increasing the cost. In addition, focusing on reducing the unnecessary querying messages, P-EASE enables an optimal query algorithm to taking into consideration to query the proper storage node, i.e., the nearer storage node of the centric storage node and local storage node. The theoretical analysis illuminates the correctness and efficiency of the P-EASE. Simulation experiments are conducted under semi-random walk and random waypoint mobility. Compared to EASE, P-EASE performs better at the query error, message complexity, total energy consumption and hotspot energy consumption. Results have shown that P-EASE is more energy-conserving and has higher location precision than EASE.

Pivoted Table Index for Querying Product-Property-Value Information

The query for triple information on product–attribute (property)–value is one of the most frequent queries in e-commerce. In storing the triple (product–attribute–value) information, a vertical schema is effective for avoiding sparse data and schema evolution, while a conventional horizontal schema often shows better query performance, since the properties are queried as groups clustered by each product. Therefore, we propose two storage schemas: a vertical schema as a primary table structure for the triple information in RDBMS and a pivoted table index created from the basic vertical table as an additional index structure for accelerating query processing. The pivoted table index is beneficial to improving the performance of the frequent pattern query on the group properties associated with each product class.

GML 문서 저장을 위한 저장 스키마 및 하부 저장 관리자의 설계 및 구현

GML은 OGC(OpenGIS Consortium)에서 공간지리정보의 저장 및 전송을 위한 인코딩 표준으로 제안한 마크업 언어이다. 일반적인 공간 네트워크 데이터베이스에서 GML 지원을 위한 연구는 GML 문서의 파싱, GML 문서의 저장, GML 문서의 질의어로 분류된다. 이러한 3가지 주제 가운데 GML 문서 저장에 관한 연구는 효율적인 GML 문서 검색을 위해 필수적인 연구이다. 그러나 기존 XML 문서의 저장 스키마를 다루는 연구는 다수인 데 반해, GML 문서의 저장 스키마에 관한 연구는 거의 전무한 형편이다. 또한 기존 XML 문서 저장 스키마는 공간지리정보 저장에 적합하지 않다. 따라서 본 논문에서는 기존의 XML저장방식의 단점인 많은 중복데이터 저장, 엘리먼트를 얻기 위해 여러 테이블을 탐색해야 하는 단점을 보완하는 GML 문서의 효율적 저장을 위한 3가지의 저장 스키마를 제안한다. 아울러 제안하는 저장 스키마에 적합한 GML 지원 하부 저장 관리자를 선계 및 구현한다. GML is a markup language presented as exchange standard for geographic information by the OGC(Open GIS Consortium). In spatial network databases, researches for supporting GML(Geographic Markup Language) can be divided into the parsing, the storing and the retrieval of GML documents. Among them, the study on the storage of GML documents is essential for their efficient retrieval. However there is little research on the storing of GML documents whereas there have been a lot of researches on the storing of n documents. Because the storage schema designed for XML documents are not appropriate for geographic information, we, in this paper, propose three storage schema for efficiently storing GML documents including geographic information in order to solve the problem that the XML storage schema store duplicate data and need to search many tables for obtaining elements. In addition, we design and implement a low-level storage manager which can store GML documents using the proposed GML storage schema.

Storing and retrieving multimedia web notes

Digital annotation of multi-media documents adds personal information to them. Sharing of annotations among different users allows discussions and cooperative work. We discuss architectural solutions and storage schemas to the problem of annotating multimedia documents with multimedia objects. Annotations can refer to whole documents or single portions, but also to groups of objects in the same document. We present a new digital annotation system with a client-server architecture, where the client is a plug-in for a standard web browser and servers are annotation repositories. Annotations can be retrieved and filtered, based on their metadata descriptors, and possibly on their content.

Answering queries using views: A survey

The problem of answering queries using views is to find efficient methods of answering a query using a set of previously defined materialized views over the database, rather than accessing the database relations. The problem has recently received significant attention because of its relevance to a wide variety of data management problems. In query optimization, finding a rewriting of a query using a set of materialized views can yield a more efficient query execution plan. To support the separation of the logical and physical views of data, a storage schema can be described using views over the logical schema. As a result, finding a query execution plan that accesses the storage amounts to solving the problem of answering queries using views. Finally, the problem arises in data integration systems, where data sources can be described as precomputed views over a mediated schema. This article surveys the state of the art on the problem of answering queries using views, and synthesizes the disparate works into a coherent framework. We describe the different applications of the problem, the algorithms proposed to solve it and the relevant theoretical results.