MDMF: A comprehensive framework for managing large-scale heterogeneous data in eSoC collaborative environment

Abstract

Computational Chemistry as a data-intensive application involves the geographically dispersed extraction of complex data information from very large collections of measured or computed data. And many chemists from different domains have to work together to explore, query, analyze, visualize and process large-scale heterogeneous data sets. Therefore, in order to address these challenges, we present and design a comprehensive framework Massive Data Management Framework (MDMF), which comprises three critical modules. It integrates the data management of CGSP and GOS, and even implements the interoperation to handle large scale data in distributed environment. And it also provides an easy-to-use graphical Chemical Data Visual Management Tool, which affords not only common database functions but also the functions of displaying and editing many types of chemical elements. Furthermore, it even offers a user-friendly Data Management Client Tool which is a uniform data viewer to access and manage the underlying data management in grid environment. Finally, we demonstrate several applications in eSoC system and the results indicate that the framework is an effective data management way to research on computational chemistry.