Memory Database Research Articles

클라우드 데이터 서비스를 위한 대용량 데이터 처리 분산 파일 아키텍처 설계

최근 클라우드 컴퓨팅 시장에 진출했거나 진출을 선언한 글로벌 IT 기업들을 이미 보유하고 있는 하드웨어, 소프트웨어 기반 기술들을 활용하거나 상호 협력을 통해 다양한 클라우드 서비스들을 제공함으로써 불특정 다수를 대상으로 급격하게 성장하고 있는 클라우드 컴퓨팅 시장에서 자신들의 영역을 지속적으로 확장해 나가고 있다. 분산 파일 시스템은 데이터의 저장과 관리뿐만 아니라 상위 계층 서비스가 요구하는 충분한 성능과 안정성을 보장해주기 위한 클라우드 컴퓨팅의 핵심 기술 중의 하나이다. 본 논문 에서는 클라우드 컴퓨팅을 위해 분산 파일 시스템이 갖추어야 할 사항들과 클라우드 컴퓨팅에서 활용 가능한 오픈 소스 기반의 하둡 분산 파일 시스템, 메모리 데이터베이스 기술, 고가용성 데이터베이스 시스템을 소개하고 현재 클라우드 컴퓨팅 시장에서 활용되고 있는 분산 파일 시스템의 동향을 통한 다양한 분산처리 기술을 참고하여 대용량 분산 데이터 처리 아키텍처를 구현하였다. In these day, some of IT venders already were going to cloud computing market, as well they are going to expand their territory for the cloud computing market through that based on their hardware and software technology, making collaboration between hardware and software vender. Distributed file system is very mainly technology for the cloud computing that must be protect performance and safety for high levels service requests as well data store. This paper introduced distributed file system for cloud computing and how to use this theory such as memory database, Hadoop file system, high availability database system. now In the market, this paper define a very large distributed processing architect as a reference by kind of distributed file systems through using technology in cloud computing market.

A novel automatic virtual metrology system architecture for TFT-LCD industry based on main memory database

The semiconductor and thin-film-transistor-liquid-crystal-display (TFT-LCD) industries widely value Automatic Virtual Metrology System (AVMS). AVMS needs to handle a large volume of VM-related data, which may cause poor internal database performance. In general, AVMS adopts efficient but expensive commercial database management systems (DBMSs) to yield good AVMS performance. This usually makes the AVMS construction cost very high. Therefore, the industries require a novel AVMS architecture with lower cost and greater efficiency in database. This paper proposes a novel AVMS architecture based on Main Memory Database (MMDB) technology. Specifically, the MMDB is used to improve the performance bottlenecks of the current Disk Resident Database (DRDB). Also, we design automatic data-backup and automatic data-query sources integration mechanisms to effectively relieve rapidly increased data volume in the original AVMS architecture. In addition, the novel AVMS architecture adopts a free commercial MMDB to significantly reduce total system cost. Integrated testing results show that the proposed AVMS architecture and developed technologies can enable the AVMS to have better data-storage efficiency, superior data-query performance, and lower database cost. The proposed AVMS architecture and research results in this paper can be a useful reference for TFT-LCD manufacturing companies in constructing their own AVM systems. The proposed AVMS architecture can also be applied in the semiconductor and solar-cell industries.

Research on Real-Time Database of Configuration Software

This article using three kinds of storage structure combined to store field data, that is memory database, relational database and document management system. This paper also gives the basic realization scheme of real-time database, and using Windows dynamic link library technology and the global shared memory technology to establish real-time database system.

Machines, translations and memories: language transfer in the web browser

Classic translation memory (TM) revolutionised technical translation in the 1990s. ‘Enhanced’ TM will revolutionise most types of translation this decade. The massive memory databases collected over 20 years can now be accessed from TM tools for exact and fuzzy matches. Sub-segmental information (terminology and phraseology) can also be gained from them via manual or automated concordancing. Stripped of metadata and formatting information, they have made possible the swift development of statistical machine translation (MT), which can be integrated with TM to provide matches where the memories themselves cannot. At the same time, cloud computing facilitates the adoption of web-based MT (and of web-based TM and translation management systems). Postediting, previously limited to a narrow section of the industry, becomes mainstream. The Read/Write Web demands (and allows for) a new model of translation, with dynamic, user-generated content not well suited to the conventional translation-editing-proofreading model. The web browser seems to drive those who want to make technical translation a profession into either semi-skilled workers enduring quasi sweatshop conditions, or else into the highly-skilled language engineer who helps make the decisions on which tool or process those translators will use for which task.

Bridging two worlds with RICE

The growing need to use large amounts of data as the basis for sophisticated business analysis conflicts with the current capabilities of statistical software systems as well as the functions provided by most modern databases. We developed two novel approaches towards a solution for this basic conflict, based on the widely-used statistical software package R and the SAP In-Memory Computing Engine (IMCE). We thereby propose an alternative data exchange mechanism with R. Instead of using standard SQL interfaces like JDBC or ODBC we introduced SQL-SHM, a shared memory-based data exchange to incorporate R's vertical data structure. Furthermore, we extended this approach to R-Op introducing R scripts equivalent to native database operations like join or aggregation within the execution plans. With the calculation engine, IMCE provides a framework to model logical execution plans and thereby offers a convenient way to use the full functionality of R via SQL interface. Moreover, this enables us to run R scripts in parallel without the necessity of extending the R interpreter itself.

IPL-P

A great deal of research has been done on solid-state storage media such as flash memory and non-volatile memory in the past few years. While NAND-type flash memory is now considered a top alternative to magnetic disk drives, different types of non-volatile memory have also begun to appear in the market recently. Although some advocates of storage class memory (SCM) predicted that flash memory would give way to SCM in the very near future, we believe that they will co-exist, complementing each other, for a while until the hurdles in its manufacturing process are lifted and storage class memory becomes commercially competitive in both capacity and price. This demo presents an improved design of In-Page Logging (IPL) by augmenting it with Phase Change RAM (PCRAM) in its log area. IPL is a buffer and storage management strategy that has been proposed for flash memory database systems. Due to the byte-addressability of PCRAM and its faster speed for small reads and writes, the IPL scheme with PCRAM can improve the performance of flash memory database systems even further by storing frequent log records in PCRAM instead of flash memory. We report a few advantages of this new design that will make IPL more suitable for flash memory database systems.

A Novel Main Memory Database T-tree Index Optimization Algorithm

Now main memory database obtains more and more widespread application in critical application, along with the continuous increasing of the data throughput capacity, the store-in data are getting more and more, the complexity is getting higher and higher, how to enhance the overall performance of the main memory database, this paper proposed a novel T tree index optimization algorithm, has designed new data structures and algorithms to increase the efficiency of the main memory database(MMDB), has optimized T tree index of the memory database, reduces the balanced revolving times in the T tree, raises the T tree's index efficiency. Results indicate that T-tail Tree provides very good performance in MMDB.

Efficiently compiling efficient query plans for modern hardware

As main memory grows, query performance is more and more determined by the raw CPU costs of query processing itself. The classical iterator style query processing technique is very simple and exible, but shows poor performance on modern CPUs due to lack of locality and frequent instruction mispredictions. Several techniques like batch oriented processing or vectorized tuple processing have been proposed in the past to improve this situation, but even these techniques are frequently out-performed by hand-written execution plans. In this work we present a novel compilation strategy that translates a query into compact and efficient machine code using the LLVM compiler framework. By aiming at good code and data locality and predictable branch layout the resulting code frequently rivals the performance of hand-written C++ code. We integrated these techniques into the HyPer main memory database system and show that this results in excellent query performance while requiring only modest compilation time.

Improving performance by creating a native join-index for OLAP

The performance of online analytical processing (OLAP) is critical for meeting the increasing requirements of massive volume analytical applications. Typical techniques, such as in-memory processing, column-storage, and join indexes focus on high performance storage media, efficient storage models, and reduced query processing. While they effectively perform OLAP applications, there is a vital limitation: mainmemory database based OLAP (MMOLAP) cannot provide high performance for a large size data set. In this paper, we propose a novel memory dimension table model, in which the primary keys of the dimension table can be directly mapped to dimensional tuple addresses. To achieve higher performance of dimensional tuple access, we optimize our storage model for dimension tables based on OLAP query workload features. We present directly dimensional tuple accessing (DDTA) based join (DDTAJOIN), a technique to optimize query processing on the memory dimension table by direct dimensional tuple access. We also contribute by proposing an optimization of the predicate tree to shorten predicate operation length by pruning useless predicate processing. Our experimental results show that the DDTA-JOIN algorithm is superior to both simulated row-store main memory query processing and the open-source column-store main memory database MonetDB, thanks to the reduced join cost and simple yet efficient query processing.

Making T-Trees Cache Conscious on Commodity Microprocessors *

Recent research shows that the database performance can be significantly improved by the effective cache utilization of the conventional microprocessors. Researchers have modified existing index structures into ones optimized for CPU cache performance in main memory database environments. A Cache Sensitive B+-Tree is designed to minimize the impact of cache misses for B+-Trees and it has been known to be more effective than other types of main memory index structure including T-Trees. In this paper, we introduce a Cache Sensitive T-Tree (CST-Tree) and show how T-Trees can also be redesigned to be cache sensitive. We present an experimental performance study which shows that our Cache Sensitive T-Trees can outperform the original T-Trees and Cache Sensitive B+-Trees on commodity microprocessors.

Using Actors to Build a Parallel DBMS

In this paper, we present the design and the architecture of a parallel main memory database management system. We focus on concurrency control scheme and recovery. Our prototype is based on the concept of "database actors", an object-oriented data model well suited for parallelmanipulations. The storage sub system is built upon distributed Ram-files using SDDS (Scalable Distributed Data Structures) techniques. A nested transaction model is proposed and used to handle concurrency access and recovery. We have also proposed novel approach, based on wait-die, to implement a distributed deadlock prevention technique for our model of nested transactions.

Hypothetical Analysis on Basis of View

With the development of computer’s hardware, main memory becomes bigger and bigger and can hold more data, so some operation before need disk I/O, now can be completed in main memory. In OLAP what-if or hypothetical analysis may need modify data or alter structure of cube, it should recalculate cube and is a long process. This paper presents how to do What-if analysis in main memory database environments and the experience shows that main memory OLAP is obvious superior to disk OLAP.

Optimization of T-Tree Index of Main Memory Database in Critical Application

One approach to achieving high performance in the DBMS in the critical application is to store the database in main memory rather than on disk. One can then design new data structures and algorithms oriented towards increasing the efficiency of the main memory database -MMDB. In this paper we present some results on index structures from an ongoing study of MMDB. We propose a new index structure, the T-tail Tree. We give the main algorithm of the T-tail Tree and the performance of these algorithms. Our results indicate that T-tail Tree provides good overall performance in main memory.

Database compression on graphics processors

Query co-processing on graphics processors (GPUs) has become an effective means to improve the performance of main memory databases. However, this co-processing requires the data transfer between the main memory and the GPU memory via a low-bandwidth PCI-E bus. The overhead of such data transfer becomes an important factor, even a bottleneck, for query co-processing performance on the GPU. In this paper, we propose to use compression to alleviate this performance problem. Specifically, we implement nine lightweight compression schemes on the GPU and further study the combinations of these schemes for a better compression ratio. We design a compression planner to find the optimal combination. Our experiments demonstrate that the GPU-based compression and decompression achieved a processing speed up to 45 and 56 GB/s respectively. Using partial decompression, we were able to significantly improve GPU-based query co-processing performance. As a side product, we have integrated our GPU-based compression into MonetDB, an open source column-oriented DBMS, and demonstrated the feasibility of offloading compression and decompression to the GPU.

SIMD-scan

The availability of huge system memory, even on standard servers, generated a lot of interest in main memory database engines. In data warehouse systems, highly compressed column-oriented data structures are quite prominent. In order to scale with the data volume and the system load, many of these systems are highly distributed with a shared-nothing approach. The fundamental principle of all systems is a full table scan over one or multiple compressed columns. Recent research proposed different techniques to speedup table scans like intelligent compression or using an additional hardware such as graphic cards or FPGAs. In this paper, we show that utilizing the embedded Vector Processing Units (VPUs) found in standard superscalar processors can speed up the performance of mainmemory full table scan by factors. This is achieved without changing the hardware architecture and thereby without additional power consumption. Moreover, as on-chip VPUs directly access the system's RAM, no additional costly copy operations are needed for using the new SIMD-scan approach in standard main memory database engines. Therefore, we propose this scan approach to be used as the standard scan operator for compressed column-oriented main memory storage. We then discuss how well our solution scales with the number of processor cores; consequently, to what degree it can be applied in multi-threaded environments. To verify the feasibility of our approach, we implemented the proposed techniques on a modern Intel multi-core processor using Intel® Streaming SIMD Extensions (Intel® SSE). In addition, we integrated the new SIMD-scan approach into SAP® Netweaver® Business Warehouse Accelerator. We conclude with describing the performance benefits of using our approach for processing and scanning compressed data using VPUs in column-oriented main memory database systems.

Prefetching J+-Tree: A Cache-Optimized Main Memory Database Index Structure

As the speed gap between main memory and modern processors continues to widen, the cache behavior becomes more important for main memory database systems (MMDBs). Indexing technique is a key component of MMDBs. Unfortunately, the predominant indexes — B+-trees and T-trees — have been shown to utilize cache poorly, which triggers the development of many cache-conscious indexes, such as CSB+-trees and pB+-trees. Most of these cache-conscious indexes are variants of conventional B+-trees, and have better cache performance than B+-trees. In this paper, we develop a novel J+-tree index, inspired by the Judy structure which is an associative array data structure, and propose a more cache-optimized index — Prefetching J+-tree (pJ+-tree), which applies prefetching to J+-tree to accelerate range scan operations. The J+-tree stores all the keys in its leaf nodes and keeps the reference values of leaf nodes in a Judy structure, which makes J+-tree not only hold the advantages of Judy (such as fast single value search) but also outperform it in other aspects. For example, J+-trees can achieve better performance on range queries than Judy. The pJ+-tree index exploits prefetching techniques to further improve the cache behavior of J+-trees and yields a speedup of 2.0 on range scans. Compared with B+-trees, CSB+-trees, pB+-trees and T-trees, our extensive experimental study shows that pJ+-trees can provide better performance on both time (search, scan, update) and space aspects.

All-IP 이동 통신망에서 HSS 시스템의 효과적인 백업과 실시간 이중화 기법

디지털 이동통신 서비스의 가입자 정보를 실시간으로 관리하는 HSS(Home Subscriber Server) 시스템은 사용자의 콜처리를 위한 기본 정보뿐만 아니라 사용자 인증과 운영 등에 관한 추가적인 정보를 제어하기 위해 가입자 데이터베이스를 주기억장치에 상주시켜 운영하는 주기억 장치 데이터베이스(main memory database system)를 필요로 한다. 그러나, 신뢰성과 안정성이 중요시되는 HSS-DBS의 경우, 초기 로딩시간이 길고 메모리 손실시 그 영향이 훨씬 크기 때문에 보다 안전한 데이터의 저장 방식과 백업 기법이 필요하다. 본 논문에서는 HSS 시스템의 안전성과 성능을 고려한 효율적인 백업 이중화 기법을 제시하였다. 제안한 쉐도잉 백업 기법을 통해 위치 등록으로 인한 실시간 백업의 과부하를 최소화하고 일정 시간 간격을 두고 지연 회복시킴으로써 오류 발생 시 신속하게 복구할 수 있으며 원격 서버에 실시간으로 백업하는 백업 이중화 기법을 통해 안정적인 시스템 운영을 보장한다. An HSS(Home Subscriber Server) system requires a main-memory database on main-memory unit for the real-tine management of the subscriber information in the mobile communication service, in that the system controls not only basic data for handling calls of users, but also additional service data related to user authentication and operational data. Nonetheless, HSS-DBS system, requiring the reliability and stability, need more secure data store method and a back-up technique because the system have a long startup time and the big problem on the failures of main-memory. This paper proposes an efficient back-up replication technique, on the basis of enhancing the stability and performance of HSS system. The proposed shadowing back-up technique adopting the delayed recovery process, can help minimize the real-time back-up overloads by location registration, while the proposed backup replication method enables more stable system operations with replicating the data to remote server in real time.

Main Memory Database TPC-H Workload Characterization on Modern Processor

Ailamaki等人1999年研究了数据库管理系统(database management system,简称DBMS)在处理器上的时间开销分解.此后,相关研究集中在分析DBMS在处理器上的瓶颈.但这些研究工作均是在磁盘数据库DRDBs(disk resident databases)上开展的,而且都是分析DBMS上的TPC-C类负载.然而,随着硬件技术的进步,现代计算机的多级缓存结构(memory hierarchy)在逐渐地"上移".例如,容量越来越大的芯片内缓存(on-chip caches)和芯片外缓存(off-chip caches),容量越来越大的RAM,Flash Memory等等.为此,处理器负载分析的研究工作也应随之"上移".研究内存数据MMDBs(main memory resident databases)在计算密集型负载下的处理器行为特性.由于磁盘数据库的主要性能瓶颈是磁盘I/O,因而可以用索引、压缩等技术进行优化;然而,内存数据库的性能瓶颈却在于处理器和内存之间的数据交换.针对这一问题,首先分析了磁盘数据库和内存数据库在TPC-H负载下处理器性能瓶颈的差异,并给出了一些优化建议,提出了通过预取的优化方法.其次,通过实验比较了不同存储体系结构(行存储与列存储)对处理器利用率的差异,并探索了下一代内存数据库体系结构方面的解决方案.此外,还研究了索引结构对处理器多级缓存的影响,并给出了索引的优化建议.最后,提出一个微测试集用于评估内存数据库在DSS(decision support system)负载下处理器的性能及行为特性.研究结果会对运行于下一代处理器上的内存数据库体系结构设计和性能优化提供一定的实验依据.;In 1999, the research of database systems' execution time breakdown on modern computer platforms has been analyzed by Ailamaki, et al. The primary motivation of these studies is to improve the performance of Disk Resident Databases (DRDBs), which form the main stream of database systems until now. The typical benchmark used in those studies is TPC-C. However, continuing hardware advancements have moved-up on the memory hierarchy, such as the larger and larger on-chip and off-chip caches, the steadily increasing RAM space, and the commercial availability of huge flash memory (solid-state disk) on top of regular disk, etc. To reflect such a trend, the target of workload characterization research along the memory hierarchy is also studied. This paper focuses on Main Memory Databases (MMDBs), and the TPC-H benchmark. Unlike the performance of DRDB which is I/O bound and may be optimized by high-level mechanisms such as indexing, the performance of MMDB is basically CPU and memory bound. In this study, the paper first compares the execution time breakdown of DRDB and MMDB, and the paper proposes an optimize strategy to optimize the memory resident aggregate. Then, the paper explores the difference between column-oriented and row-oriented storage models in CPU and cache utilization. Furthermore, the paper measures performance of MMDBs on different generational CPUs. In addition, the paper analyzes the index influence and gives a strategy for main memory database index optimization. Finally, the paper analyzes each query in the full TPC-H benchmark in detail, and obtains systematic results, which help design micro-benchmarks for further analysis of CPU cache stall. Results of this study are expected to benefit the performance optimization of MMDBs, and the architecture design memory-oriented databases of the next generation.

Translating and Revising for Localisation: What do We Know? What do We Need to Know?

Translating and revising for localisation require specific skills that are distinct from those required for translating and revising for publishing or other purposes. Their distinctiveness is due to several factors, particularly the need to work within a translation memory (TM) editor environment, rather than the usual word-processor. When using TM, translators and revisers are constrained by strict requirements regarding terminology, and leveraging and reuse from memory databases. Recent studies on translating with TM have shown that segmentation imposes a narrow, bottom-up view of the text, which restricts translators in their preferred approach. Other recent studies of revision in general have indicated, moreover, that there is a significant risk that revision may not always improve final text quality. This risk is especially likely to apply to the revision of TM-mediated translation, since revisers are placed in the top-down role of pulling together (sometimes not even sequentially ordered) the segments into a coherent whole. This article argues that the practice of translation and revision for localisation is being shaped by the needs of a new and unprecedented type of user: the TM apparatus itself. Research and discussion will enable participants to better understand their roles and responsibilities within this new scenario.

CC-GiST: 임의의 캐시 인식 검색 트리를 효율적으로 구현하기 위한 일반화된 프레임워크

최근 메인 메모리 가격이 하락하고 용량이 크게 증가함에 따라 메인 메모리 데이터베이스에 기반한 응용이 급격히 증가하고 있다. 캐시 미스 (cache miss)는 CPU에서 액세스하고자 하는 데이터가 캐시에 존재하지 않아 메모리로부터 읽어 들이는 과정이며, 메인 메모리 데이터베이스의 성능 감소의 중요한 원인이다. 메인 메모리 데이터베이스에서의 캐시 미스를 줄이고 캐시를 최대한 활용하기 위하여 여러 가지 캐시 인식 트리들(cache conscious trees)이 제안되었다. 이러한 캐시 인식 트리들은 각각 특성이 다르므로 하나의 응용에서 둘 이상의 캐시 인식 트리들이 동시에 관리될 수 있다. 또한, 만약 기존의 캐시 인식 트리가 응용에서의 요구를 만족시키지 못하면 새로운 캐시 인식 트리를 구현하여야 한다. 본 논문에서는 캐시 인식하는 일반화된 검색 트리(Cache-Conscious Generalized Search Tree, CC-GiST)를 제안한다. CC-GiST는 디스크 기반의 일반화된 검색 트리 (Generalized Search Tree, GiST) [HNP95]를 캐시 인식하도록 확장한 것이며, 포인터 압축(pointer compression)과 키 압축(key compression) 기법을 비롯하여 임의의 캐시 인식 트리의 공통적인 기능 및 알고리즘들을 동시에 제공한다. CC-GiST를 기반으로 특정 캐시 인식트리를 구현하려면 그 트리에 해당된 기능만을 구현하면 된다. 본 논문에서는 CC-GiST를 기반으로 기존의 대표적인 캐시 인식 트리인 CSB+-트리, pkB-트리, CR-트리를 구현하는 방법을 기술한다. CC-GiST를 이용함에 따라 메인 메모리 데이터베이스 응용에서 여러 개의 캐시 인식 트리를 관리하는 번거로움에서 벗어날 수 있고, 응용의 요구에 따른 새로운 캐시 인식 트리를 최소한의 노력으로 효율적으로 구현할 수 있다. According to recent rapid price drop and capacity growth of main memory, the number of applications on main memory databases is dramatically increasing. Cache miss, which means a phenomenon that the data required by CPU is not resident in cache and is accessed from main memory, is one of the major causes of performance degradation of main memory databases. Several cache-conscious trees have been proposed for reducing cache miss and making the most use of cache in main memory databases. Since each cache-conscious tree has its own unique features, more than one cache-conscious tree can be used in a single application depending on the application's requirement. Moreover, if there is no existing cache-conscious tree that satisfies the application's requirement, we should implement a new cache-conscious tree only for the application's sake. In this paper, we propose the cache-conscious generalized search tree (CC-GiST). The CC-GiST is an extension of the disk-based generalized search tree (GiST) [HNP95] to be tache-conscious, and provides the entire common features and algorithms in the existing cache-conscious trees including pointer compression and key compression techniques. For implementing a cache-conscious tree based on the CC-GiST proposed in this paper, one should implement only a few functions specific to the cache-conscious tree. We show how to implement the most representative cache-conscious trees such as the CSB+-tree, the pkB-tree, and the CR-tree based on the CC-GiST. The CC-GiST eliminates the troublesomeness caused by managing mire than one cache-conscious tree in an application, and provides a framework for efficiently implementing arbitrary cache-conscious trees with new features.