Homogenizing Data and Metadata Retrieval in Scientific Applications

Abstract

Support for large arrays has been increasingly gaining attention by the database community. Array databases are a quickly expanding category of database management systems that treat large, multidimensional array data as first-class database citizens, allowing convenient and efficient storage and retrieval. Large array data on its own, however, is rarely meaningful or machine understandable. Therefore, arrays are almost always linked to additional, small metadata, typically managed in relational databases. This leads to an information integration gap, an impedance mismatch that we address with the work presented in this paper. We aim to remedy this with ASQL (Array SQL), an augmentation of the standard relational database language SQL with multidimensional array processing capabilities. Besides core ASQL concepts, this paper presents a general mediator approach to implementing it by combining a relational with an array DBMS. This improves on the most common situation of systems specializing in either relational or array processing, but not both. In particular we implement ASQLDB on top of the array DBMS rasdaman and relational DBMS HSQLDB, and demonstrate its potential for practical applicability through real world use cases. Initial benchmark comparisons against another array DBMS with a similar goal but rather different approach reveal advantageous results for ASQL.