Hybrid Database System Research Articles

Verifiable Access Technology of Hybrid Database in Distributed System Under Big Data

Targeting at the problem that the existing hybrid database solutions can not simultaneously realize the client’s verification and access function to the encrypted data in the public cloud database and the cross database query function in the heterogeneous database environment, this paper designs an access and integration middleware to realize the client’s verification and access function to the heterogeneous hybrid database in the public cloud and the cross database query function. Access and integration middleware is composed of verification access component and heterogeneous database data integration method. The verification and access component reconstructs the protocol interaction process based on MySQL protocol, and encrypts and decrypts the encrypted data in the public cloud database through the existing hybrid database scheme, so as to realize the client’s verification and access to the encrypted data in the cloud database. The heterogeneous database data integration method unifies the data from different types of databases, and then integrates them into the same database, and uses this database for query operation to realize cross database query. This paper first analyzes the hybrid database system model, then designs from the aspects of MySQL protocol analysis, verification access, data integration, and finally analyzes the access and integration middleware from the aspects of functionality and security.

Development of high-precision wind, wave and current forecast system for offshore wind energy industry in Taiwan: a two-stage method of numerical simulation and AI correction

ABSTRACT At the Changhwa Fuhai wind farm, the high-resolution numerical models for wind, wave and current simulations are crucial to provide daily forecasts as a decision-support system for Taiwan’s offshore wind energy industry. In this study, the two-stage method, consisting of numerical simulation and deep-learning correction based on the observed data at Fuhai wind farm site, is implemented to further improve the meteorological and marine forecast accuracy. The recurrent neural network-long short-term memory (RNN-LSTM) model is adopted in the deep learning analytics. Currently, this system is capable of daily providing the next 4-day wind field forecast and the next 7-day wave and current field forecast. Through the testing, it is found that the error correction of wind simulations can reduce root mean square error (RMSE) by up to 36.7%. Furthermore, through adopting the SQL and HBase hybrid database system, a wide range of historical data as well as daily forecast data can be quickly queried and displayed on the web interface to provide important decision support to the relevant wind power companies in scheduling offshore construction plans, forecasting wind energy production, and arranging wind turbines’ shutdown and maintenance sequence.

Replication at the speed of change

The IBM Db2 Analytics Accelerator (IDAA) is a state-of-the art hybrid database system that seamlessly extends the strong transactional capabilities of Db2 for z/OS with the very fast column-store processing in Db2 Warehouse. The Accelerator maintains a copy of the data from Db2 for z/OS in its backend database. Data can be synchronized at a single point in time with a granularity of a table, one or more of its partitions, or incrementally as rows changed using replication technology. IBM Change Data Capture (CDC) has been employed as replication technology since IDAA version 3. Version 7.5.0 introduces a superior replication technology as a replacement for IDAA's use of CDC - Integrated Synchronization. In this paper, we present how Integrated Synchronization improves the performance by orders of magnitudes, paving the way for near real-time Hybrid Transactional and Analytical (HTAP) processing.

An integration approach of hybrid databases based on SQL in cloud computing environment

SummaryAs the applications with big data in cloud computing environment grow, many existing systems expect to expand their service to support the dramatic increase of data, and modern software development for services computing and cloud computing software systems is no longer based on a single database but on existing multidatabases and this convergence needs new software architecture design. This paper proposes an integration approach to support hybrid database architecture, including MySQL, MongoDB, and Redis, to make it possible of allowing users to query data simultaneously from both relational SQL systems and NoSQL systems in a single SQL query. Two mechanisms are provided for constructing Redis's indexes and semantic transforming between SQL and MongoDB API to add the SQL feature for these NoSQL databases. With the proposed approach, hybrid database systems can be performed in a flexible manner, ie, access can be either relational database or NoSQL, depending on the size of data. The approach can effectively reduce development complexity and improve development efficiency of the software systems with multidatabases. This is the result of further research on the related topic, which fills the gap ignored by relevant scholars in this field to make a little contribution to the further development of NoSQL technology.

A GEOSPATIAL ANALYSIS FRAMEWORK FOR FINE SCALE URBAN INFRASTRUCTURE NETWORKS

Abstract. Understanding the spatial connectivity of urban infrastructure networks that connect assets to buildings is important for the fine-scale spatial analysis and modelling of the resource flows within cities. However, rarely are spatially explicit representations of infrastructure networks available for such analysis. Further, an appropriate database system is the core of development of an infrastructure assets information and management platform, capable of handling the wide range of data for infrastructure system modelling and analysis. In this paper, we develop a geospatial simulation and analysis framework, which is capable of generating fine-scale urban infrastructure networks and storing the network instances in a hybrid database system for further modelling and analysis needs. We demonstrate the use of this platform by simulating the entire-city electricity distribution network for the city of Newcastle upon Tyne. Validation of the resulting network is performed using the network layout diagram from the local power company. The heuristically derived network was found to have a 91 % spatial accuracy.

Measuring transaction performance based on storage approaches of Native XML database

Many organizations today store their critical business information permanently in XML format. XML data can be managed using: XML-Enabled Database (XED) systems which convert and store XML files in traditional database systems; Native XML Database (NXD) systems which store XML data natively using three main storage technologies – text-based, model-based, and schema-based techniques; and Hybrid Database systems which are comprised of both XML-Enabled and Native XML database systems. NXDs are faster than other database technologies because there is no need to convert the format of the data prior to storage. No performance evaluation has been carried out to compare all three storage strategies, hence, this paper reports on the first attempt to evaluate all three storage strategies by using open source products to measure the response time taken for each of the database basic tasks such as database creation, dataset insertion, and data manipulation. The results of the evaluation show that the schema-based storage strategy: performs 3.5 times faster than the other two storage techniques in data insertion; shows very good performance in query processing on small and large datasets; performs 10.33 times faster than text-based, and 7.5 times faster than model-based storage techniques in query processing of large datasets.

Semi-static operator graphs for accelerated query execution on FPGAs

This paper introduces the concept of Semi-static Operator Graphs (SOG) to provide a runtime reconfigurable accelerator for query execution based on a Field Programmable Gate Array (FPGA). Instead of generating an FPGA configuration for a given arbitrary query during system runtime, we deploy a general query structure on the FPGA consisting of multiple small reconfigurable partitions (RP). During deployment of the hybrid database system, for each RP various query operators are prepared as reconfigurable modules (RM). At system runtime, the proposed approach dynamically chooses and reconfigures RMs into the RPs regarding a given query. As a result the reconfiguration overhead during system runtime is significantly reduced and enables the utilization of our hybrid architecture in real-world scenarios.

HYBRID DATABASE SYSTEM FOR BIG DATA STORAGE AND MANAGEMENT

Relational database systems have been the standard storage system over the last forty years. Recently, advancements in technologies have led to an exponential increase in data volume, velocity and variety beyond what relational databases can handle. Developers are turning to NoSQL which is a non- relational database for data storage and management. Some core features of database system such as ACID have been compromised in NOSQL databases. This work proposed a hybrid database system for the storage and management of extremely voluminous data of diverse components known as big data, such that the two models are integrated in one system to eliminate the limitations of the individual systems. The system is implemented in MongoDB which is a NoSQL database and SQL. The results obtained, revealed that having these two databases in one system can enhance storage and management of big data bridging the gap between relational and NoSQL storage approach.

CrowdDB

Databases often give incorrect answers when data are missing or semantic understanding of the data is required. Processing such queries requires human input for providing the missing information, for performing computationally difficult functions, and for matching, ranking, or aggregating results based on fuzzy criteria. In this demo we present CrowdDB, a hybrid database system that automatically uses crowdsourcing to integrate human input for processing queries that a normal database system cannot answer. CrowdDB uses SQL both as a language to ask complex queries and as a way to model data stored electronically and provided by human input. Furthermore, queries are automatically compiled and optimized. Special operators provide user interfaces in order to integrate and cleanse human input. Currently CrowdDB supports two crowdsourcing platforms: Amazon Mechanical Turk and our own mobile phone platform. During the demo, the mobile platform will allow the VLDB crowd to participate as workers and help answer otherwise impossible queries.

A demonstration of HYRISE

We propose to demonstrate HYRISE, a main memory hybrid database system, which automatically partitions tables into vertical partitions consisting of variable numbers of columns based on access patterns to each table. Using an accurate model of cache misses, HYRISE is able to predict the performance of different partitionings, and to automatically select the best partitions using an automated database partitioning algorithm. Our demonstration will show the results of the physical partitioning based on different query workloads, allowing demo attendees to visualize, fine-tune, and modify the partitioning using a GUI. It will then show how the various physical designs affect the query plans and the performance of the database as a whole. Attendees can thus experiment with various physical models, and can grasp the potential of hybrid partitionings, which achieve a 20% to 400% performance improvement over pure all-column or all-row designs on our realistic hybrid workload derived from customer applications.

HYRISE

In this paper, we describe a main memory hybrid database system called HYRISE, which automatically partitions tables into vertical partitions of varying widths depending on how the columns of the table are accessed. For columns accessed as a part of analytical queries (e.g., via sequential scans), narrow partitions perform better, because, when scanning a single column, cache locality is improved if the values of that column are stored contiguously. In contrast, for columns accessed as a part of OLTP-style queries, wider partitions perform better, because such transactions frequently insert, delete, update, or access many of the fields of a row, and co-locating those fields leads to better cache locality. Using a highly accurate model of cache misses, HYRISE is able to predict the performance of different partitionings, and to automatically select the best partitioning using an automated database design algorithm. We show that, on a realistic workload derived from customer applications, HYRISE can achieve a 20% to 400% performance improvement over pure all-column or all-row designs, and that it is both more scalable and produces better designs than previous vertical partitioning approaches for main memory systems.

Cost-based optimization in DB2 XML

DB2 XML is a hybrid database system that combines the relational capabilities of DB2 Universal DatabaseTM (UDB) with comprehensive native XML support. DB2 XML augments DB2® UDB with a native XML store, XML indexes, and query processing capabilities for both XQuery and SQL/XML that are integrated with those of SQL. This paper presents the extensions made to the DB2 UDB compiler, and especially its cost-based query optimizer, to support XQuery and SQL/XML queries, using much of the same infrastructure developed for relational data queried by SQL. It describes the challenses to the relational infrastructure that supporting XQuery and SQL/XML poses and provides the rationale for the extensions that were made to the three main parts of the optimizer: the plan operators, the cardinality and cost model, and statistics collection.

Development of a dynamic data interchange scheme to support product design in agile manufacturing

A proficient information system features the capability to response to customer changes in a timely manner such that it facilitates enterprise to achieve the concept of agile manufacturing. The efficient sharing of data and knowledge within an organization is the prerequisite to enhance the competitiveness of companies in this information age. In today's industrial environment, there are a great variety of computer systems and applications, which make the task of data integration among heterogeneous database systems even more difficult and costly, thus hindering the progress of tasks automation. In general, there is a lack of research literature related to the generic approach to link dissimilar databases together for seamless data communication. Therefore, a comprehensive mechanism is required to handle various data format in complex product design processes in order to achieve efficient knowledge retrieval and update information manipulation. This paper introduces a dynamic data interchange scheme which attempts to exchange the data automatically as well as enables the filtering of valuable data between traditional relational database model and case-based reasoning knowledge repository. The significance of this paper is concerned with the intelligent data exchange within a hybrid database system, embracing the empirical data to help agile manufacturing enterprises make critical decisions.