Main-memory Processing Research Articles

Farewell to Disks

Sophisticated processing of huge, complex datasets requires us to rethink the relationship between disk-based storage and main-memory processing. Some features of modern systems—namely, 64-bit architectures, memory-mapped sparse files, virtual memory, and copy on write support—let us process our data with readable and efficient RAM-based algorithms, using slow disks and file systems only for their large capacity and to secure the data's persistence. The author demonstrates this approach through a short C++ program that locates the shortest path on very large graphs, like that of Wikipedia. Although RAM-based processing opens up again many problems that database systems already solve, the author believes that this is the right move because it provides us with a unified programming and performance model for all our data operations, irrespective of where the data resides.

DBToaster

We present DBToaster, a novel query compilation framework for producing high performance compiled query executors that incrementally and continuously answer standing aggregate queries using in-memory views. DBToaster targets applications that require efficient main-memory processing of standing queries ( views ) fed by high-volume data streams, recursively compiling view maintenance (VM) queries into simple C++ functions for evaluating database updates ( deltas ). While today's VM algorithms consider the impact of single deltas on view queries to produce maintenance queries, we recursively consider deltas of maintenance queries and compile to thoroughly transform queries into code. Recursive compilation successively elides certain scans and joins, and eliminates significant query plan interpreter overheads. In this demonstration, we walk through our compilation algorithm, and show the significant performance advantages of our compiled executors over other query processors. We are able to demonstrate 1--3 orders of magnitude improvements in processing times for a financial application and a data warehouse loading application, both implemented across a wide range of database systems, including PostgreSQL, HSQLDB, a commercial DBMS 'A', the Stanford STREAM engine, and a commercial stream processor 'B'.

Data link driver program design for the IBM token ring network PC adapter

A driver program has been designed and developed to operate on the data link layer of the IBM token ring Personal Computer adapter. The program is written in the PC assembly language. It supports the Logical Link Control type 1 procedure, and serves as a basic building block for the construction of higher level communications software. The design provides a mechanism by which to construct multiple Service Access Points and multiple Command Control Blocks for a SAP so that it can support the transmission of multiple frames without having to wait for the command completion of the earlier request. As a result, main memory processing in the PC can continue in parallel while the adapte is preparing a frame for transmission, or awaiting its return. This overlapping in processing greatly enhances the efficiency of transmission, and reduces the link level processing overhead to a minimum. This is confirmed by performance tests undertaken on the PC adapter.