Large-scale Data Management Research Articles

The development of an ICT framework for business intelligence at the Brazilian National Cancer Institute: a case study of organizational learning and innovation

This paper presents the development of an ICT framework for the management, processing and analysis of large-scale Brazilian cancer data. This was a qualitative and descriptive study conducted at Brazilian National Cancer Institute (INCA). The benefits derived from the framework implementation, so far, are: real-time knowledge access; patient’s focus and satisfaction; enhanced knowledge transfer and diffusion of best health care practices; reduction in treatment costs; relief of the INCA’s labor shortage; and finally speediness of cancer diagnosis and enhanced quality of both diagnosis and data used in epidemiological studies. The study also identifies the critical aspects that must be observed by the Brazilian health organizations in early stages of development of BI systems and thus contributes to the faster adoption of such systems.

Advancing data management and analysis in different scientific disciplines

Over the past several years, rapid growth of data has affected many fields of science. This has often resulted in the need for overhauling or exchanging the tools and approaches in the disciplines’ data life cycles. However, this allows the application of new data analysis methods and facilitates improved data sharing.The project Large-Scale Data Management and Analysis (LSDMA) of the German Helmholtz Association has been addressing both specific and generic requirements in its data life cycle successfully since 2012. Its data scientists work together with researchers from the fields such as climatology, energy and neuroscience to improve the community-specific data life cycles, in several cases even all stages of the data life cycle, i.e. from data acquisition to data archival. LSDMA scientists also study methods and tools that are of importance to many communities, e.g. data repositories and authentication and authorization infrastructure.

Unlocking the circular economy through new business models based on large-scale data: An integrative framework and research agenda

This work makes the case for the integration of the circular economy (CE) and large-scale data (LD), also known as big data. The paper is one of the first to integrate conceptual and practical trends regarding: (a) the ReSOLVE based models of the circular economy; (b) key stakeholders roles in pursuing a more sustainable society; and (c) the volume, velocity, variety, and veracity (4V's) of large-scale data (LD) management. This study's contributions include: (1) introducing a new integrative framework to enhance the understanding of the CE-LD nexus; (2) a relational matrix which illustrates the complexity of large-scale data and stakeholders management; and (3) a research agenda, with clear research propositions and future research direction. The proposed CE-LD integrative framework provides socio-technical insights for academics, practitioners, managers, and policy decision-makers.

Split key management framework for Open Stack Swift object storage cloud

Object storage systems are emerging as new generation service oriented store technologies of large scale data management and delivery in clouds. Recently, several object storage technologies are emerging, one such technology is OpenStack Swift. Swift is based on container services methodology for data organization and retrieval, managed via open standards like web services REST protocols. Swift has several components for data management and retrieval, but, lacks in providing security features (like the key management and encryption) of the data while the data is at rest and in transit. In this paper, we propose a framework for OpenStack Swift key management based on multi-factor split keys, and propose a methodology for key organisation using the document database Mongo DB. First, we describe the object based storage mechanisms in swift, its limitations, followed by key management framework and its components. In this paper we compare the performance of the proposed symmetric-split-key encryption/decryption methodology with the symmetric key, and the performance of the key retrieval when organized in Relational database MySQL and Document database Mongo DB.

Moving metadata from ad hoc files to database tables for robust, highly available, and scalable HDFS

As a representative large-scale data management technology, Apache Hadoop is an open-source framework for processing a variety of data such as SNS, medical, weather, and IoT data. Hadoop largely consists of HDFS, MapReduce, and YARN. Among them, we focus on improving the HDFS metadata management scheme responsible for storing and managing big data. We note that the current HDFS incurs many problems in system utilization due to its file-based metadata management. To solve these problems, we propose a novel metadata management scheme based on RDBMS for improving the functional aspects of HDFS. Through analysis of the latest HDFS, we first present five problems caused by its metadata management and derive three requirements of robustness, availability, and scalability for resolving these problems. We then design an overall architecture of the advanced HDFS, A-HDFS, which satisfies these requirements. In particular, we define functional modules according to HDFS operations and also present the detailed design strategy for adding or modifying the individual components in the corresponding modules. Finally, through implementation of the proposed A-HDFS, we validate its correctness by experimental evaluation and also show that A-HDFS satisfies all the requirements. The proposed A-HDFS significantly enhances the HDFS metadata management scheme and, as a result, ensures that the entire system improves its stability, availability, and scalability. Thus, we can exploit the improved distributed file system based on A-HDFS for various fields and, in addition, we can expect more applications to be actively developed.

A scalable Map Reduce tasks scheduling: a threading-based approach

The Map Reduce paradigm is now considered a standard platform that is used for large-scale data processing and management. A major operation that the Map Reduce platform relies on greatly is tasks scheduling. Although many schedulers have been presented, task scheduling is still one of the major problems that face Map Reduce frameworks. Schedulers need to maintain data locality to achieve an acceptable performance by avoiding several data transmissions. Hence, in this paper, we propose a new scheduling algorithm named 'MTL' that utilises multi-threading principles. The MTL scheduler assigns a dedicated thread for each data block. Indeed, the multi-threading approach shows great results that make our MTL scheduler a scalable one that performs well. At the same time, it maintains the locality property. During the evaluation of the MTL scheduler performance, two main factors were taken into consideration; the simulation time and the energy consumption. The MTL scheduler is then compared with other existing schedulers such as FIFO, matchmaking, and delay schedulers. The MTL scheduler showed favourable results and proved its advantages over other existing schedulers.

A scalable Map Reduce tasks scheduling: a threading-based approach

The Map Reduce paradigm is now considered a standard platform that is used for large-scale data processing and management. A major operation that the Map Reduce platform relies on greatly is tasks scheduling. Although many schedulers have been presented, task scheduling is still one of the major problems that face Map Reduce frameworks. Schedulers need to maintain data locality to achieve an acceptable performance by avoiding several data transmissions. Hence, in this paper, we propose a new scheduling algorithm named 'MTL' that utilises multi-threading principles. The MTL scheduler assigns a dedicated thread for each data block. Indeed, the multi-threading approach shows great results that make our MTL scheduler a scalable one that performs well. At the same time, it maintains the locality property. During the evaluation of the MTL scheduler performance, two main factors were taken into consideration; the simulation time and the energy consumption. The MTL scheduler is then compared with other existing schedulers such as FIFO, matchmaking, and delay schedulers. The MTL scheduler showed favourable results and proved its advantages over other existing schedulers.

Clinical Databases and Registries in Congenital and Pediatric Cardiac Surgery, Cardiology, Critical Care, and Anesthesiology Worldwide.

The growth in large-scale data management capabilities and the successful care of patients with congenital heart defects have coincidentally paralleled each other for the last three decades, and participation in multicenter congenital heart disease databases and registries is now a fundamental component of cardiac care. This manuscript attempts for the first time to consolidate in one location all of the relevant databases worldwide, including target populations, specialties, Web sites, and participation information. Since at least 1,992 cardiac surgeons and cardiologists began leveraging this burgeoning technology to create multi-institutional data collections addressing a variety of specialties within this field. Pediatric heart diseases are particularly well suited to this methodology because each individual care location has access to only a relatively limited number of diagnoses and procedures in any given calendar year. Combining multiple institutions data therefore allows for a far more accurate contemporaneous assessment of treatment modalities and adverse outcomes. Additionally, the data can be used to develop outcome benchmarks by which individual institutions can measure their progress against the field as a whole and focus quality improvement efforts in a more directed fashion, and there is increasing utilization combining clinical research efforts within existing data structures. Efforts are ongoing to support better collaboration and integration across data sets, to improve efficiency, further the utility of the data collection infrastructure and information collected, and to enhance return on investment for participating institutions.

Guest Editorial: Large-scale Multimedia Data Management: Techniques and Applications

Guest Editorial: Large-scale Multimedia Data Management: Techniques and Applications

Tuning Heterogeneous Computing Platforms for Large-Scale Hydrology Data Management

HydroTerre is a research prototype platform developed at Penn State for the hydrology community. It provides access to aggregated scientific data sets that are useful for hydrological modeling and research. HydroTerre’s frontend is a web service, and a user query can request creation of a data bundle whose size can vary from a few megabytes to 100’s of gigabytes. In this article, we present software tuning and optimization strategies for various hardware configurations of the HydroTerre platform. Our goal is to minimize access time to a wide range of data bundle creation queries from users. We use automated schemes to estimate the computational work required for various queries, and identify the best-performing hardware/software configuration. We hope this study is instructive for researchers developing similar data management cyberinfrastructure in other science and engineering fields.

Efficient indexing and retrieval of large-scale geo-tagged video databases

We are witnessing a significant growth in the number of smartphone users and advances in phone hardware and sensor technology. In conjunction with the popularity of video applications such as YouTube, an unprecedented number of user-generated videos (UGVs) are being generated and consumed by the public, which leads to a Big Data challenge in social media. In a very large video repository, it is difficult to index and search videos in their unstructured form. However, due to recent development, videos can be geo-tagged (e.g., locations from GPS receiver and viewing directions from digital compass) at the acquisition time, which can provide potential for efficient management of video data. Ideally, each video frame can be tagged by the spatial extent of its coverage area, termed Field-Of-View (FOV). This effectively converts a challenging video management problem into a spatial database problem. This paper attacks the challenges of large-scale video data management using spatial indexing and querying of FOVs, especially maximally harnessing the geographical properties of FOVs. Since FOVs are shaped similar to slices of pie and contain both location and orientation information, conventional spatial indexes, such as R-tree, cannot index them efficiently. The distribution of UGVs' locations is non-uniform (e.g., more FOVs in popular locations). Consequently, even multilevel grid-based indexes, which can handle both location and orientation, have limitations in managing the skewed distribution. Additionally, since UGVs are usually captured in a casual way with diverse setups and movements, no a priori assumption can be made to condense them in an index structure. To overcome the challenges, we propose a class of new R-tree-based index structures that effectively harness FOVs' camera locations, orientations and view-distances, in tandem, for both filtering and optimization. We also present novel search strategies and algorithms for efficient range and directional queries on our indexes. Our experiments using both real-world and large synthetic video datasets (over 30 years' worth of videos) demonstrate the scalability and efficiency of our proposed indexes and search algorithms.

Rucio, the next-generation Data Management system in ATLAS

Rucio is the next-generation of Distributed Data Management (DDM) system benefiting from recent advances in cloud and ”Big Data” computing to address HEP experiments scaling requirements. Rucio is an evolution of the ATLAS DDM system Don Quixote 2 (DQ2), which has demonstrated very large scale data management capabilities with more than 160 petabytes spread worldwide across 130 sites, and accesses from 1,000 active users. However, DQ2 is reaching its limits in terms of scalability, requiring a large number of support staff to operate and being hard to extend with new technologies. Rucio addresses these issues by relying on new technologies to ensure system scalability, cover new user requirements and employ new automation framework to reduce operational overheads. This paper shows the key concepts of Rucio, details the Rucio design, and the technology it employs, the tests that were conducted to validate it and finally describes the migration steps that were conducted to move from DQ2 to Rucio.

TACIT: An open-source text analysis, crawling, and interpretation tool.

As human activity and interaction increasingly take place online, the digital residues of these activities provide a valuable window into a range of psychological and social processes. A great deal of progress has been made toward utilizing these opportunities; however, the complexity of managing and analyzing the quantities of data currently available has limited both the types of analysis used and the number of researchers able to make use of these data. Although fields such as computer science have developed a range of techniques and methods for handling these difficulties, making use of those tools has often required specialized knowledge and programming experience. The Text Analysis, Crawling, and Interpretation Tool (TACIT) is designed to bridge this gap by providing an intuitive tool and interface for making use of state-of-the-art methods in text analysis and large-scale data management. Furthermore, TACIT is implemented as an open, extensible, plugin-driven architecture, which will allow other researchers to extend and expand these capabilities as new methods become available.

Evaluating the Corporate Social Responsibility and Organizational Performance based on Large-Scale Data Access and Management

Information technology has entered a big-data era. Large-scale data is becoming the basic factors of production and it would affect the corporate social responsibility disclosure deeply, the use of big data has become the key factor to improve the competitiveness of enterprises. Through this empirical analysis, we can draw the following conclusions: corporate social responsibility is mainly composed of economic responsibility, legal responsibility, ethical responsibility and spontaneous responsibility. And CSR will have positive effect on organizational performance. All of these indicate that prompting enterprises staff to fulfill corporate social responsibility is helpful to improve the organizational performance.

Cloud Support Data Management Infrastructure for Upcoming Smart Cities

This paper presents a novel large scale data management model for future smart cities. The system is exploiting the emerging cloud computing services availability over the globe by processing the collected data at the edge. To handle the large scale requirements in smart city, we introduce a Mobile Edge Computing (MEC) framework in order to increase the reliability of the deployed applications. MEC is a promising framework to reduce the cloud core utilization as well as to provide applications that require low latency to mobile end users. As near as possible to the location of the end user, MEC relates to the mobile network applications and data stream acceleration through caching and/or compressing of relevant data at the edge of the mobile network. Although MEC is not yet deployed in real-life systems, recent studies discussed some technical details and related concepts. This paper studies the definition of MEC and similar concepts with respect to typical application scenarios, like in smart cities, and provides scope and limitations that may be encountered when implementing and deploying MEC. Then, a MEC framework is used to explore the data management infrastructure for upcoming smart cities.

Progress in Multi-Disciplinary Data Life Cycle Management

Modern science is most often driven by data. Improvements in state-of-the-art technologies and methods in many scientific disciplines lead not only to increasing data rates, but also to the need to improve or even completely overhaul their data life cycle management.Communities usually face two kinds of challenges: generic ones like federated authorization and authentication infrastructures and data preservation, and ones that are specific to their community and their respective data life cycle. In practice, the specific requirements often hinder the use of generic tools and methods.The German Helmholtz Association project ’’Large-Scale Data Management and Analysis” (LSDMA) addresses both challenges: its five Data Life Cycle Labs (DLCLs) closely collaborate with communities in joint research and development to optimize the communities data life cycle management, while its Data Services Integration Team (DSIT) provides generic data tools and services.We present most recent developments and results from the DLCLs covering communities ranging from heavy ion physics and photon science to high-throughput microscopy, and from DSIT.

Unlocking data: federated identity with LSDMA and dCache

X.509, the dominant identity system from grid computing, has proved unpopular for many user communities. More popular alternatives generally assume the user is interacting via their web-browser. Such alternatives allow a user to authenticate with many services with the same credentials (user-name and password). They also allow users from different organisations form collaborations quickly and simply.Scientists generally require that their custom analysis software has direct access to the data. Such direct access is not currently supported by alternatives to X.509, as they require the use of a web-browser.Various approaches to solve this issue are being investigated as part of the Large Scale Data Management and Analysis (LSDMA) project, a German funded national R&D project. These involve dynamic credential translation (creating an X.509 credential) to allow backwards compatibility in addition to direct SAML- and OpenID Connect-based authentication.We present a summary of the current state of art and the current status of the federated identity work funded by the LSDMA project along with the future road map.

Energy-Efficient Location and Activity-Aware On-Demand Mobile Distributed Sensing Platform for Sensing as a Service in IoT Clouds

The Internet of Things (IoT) envisions billions of sensors deployed around us and connected to the Internet, where the mobile crowd sensing technologies are widely used to collect data in different contexts of the IoT paradigm. Due to the popularity of Big Data technologies, processing and storing large volumes of data has become easier than ever. However, large scale data management tasks still require significant amounts of resources that can be expensive regardless of whether they are purchased or rented (e.g. pay-as-you-go infrastructure). Further, not everyone is interested in such large scale data collection and analysis. More importantly, not everyone has the financial and computational resources to deal with such large volumes of data. Therefore, a timely need exists for a cloud-integrated mobile crowd sensing platform that is capable of capturing sensors data, on-demand, based on conditions enforced by the data consumers. In this paper, we propose a context-aware, specifically, location and activity-aware mobile sensing platform called C-MOSDEN ( Context-aware Mobile Sensor Data ENgine) for the IoT domain. We evaluated the proposed platform using three real-world scenarios that highlight the importance of 'selective sensing'. The computational effectiveness and efficiency of the proposed platform are investigated and is used to highlight the advantages of context-aware selective sensing.

Abstract 4870: GenePool: A cloud-based technology for rapidly data mining large-scale, patient-derived cancer genomic cohorts including The Cancer Genome Atlas

Abstract GenePool is a cloud-based system for the secure storage, management, analysis, visualization and interpretation of large-scale human genomics data. The GenePool platform enables rapid interrogation of massive amounts of data generated by current laboratory methods and was designed to meet the needs of scientists engaged in cancer biomarker discovery and characterization. The software solution provides users with an intuitive interface, an easy way to store and select genomics datasets according to sample-associated metadata for analysis, and an automated system for performing routine, well-characterized genomics workflows. Results are presented with multiple options for annotation, sorting and data export, with a visualization tool that facilitates browsing of genomic data for biomarker identification. We will demonstrate how GenePool removes the data download, management, and computing burdens faced by researchers interested in working with large amounts of patient-derived cancer genomics exemplified by The Cancer Genome Atlas. To date, TCGA has completed molecular characterization of more than 9,000 patient samples from a target of 17,000, and will have released an estimated 2.5 petabytes of data at project completion. We will present case studies using the >25 cancer cohorts worth of RNA-Seq, miRNA-Seq, Exome Sequencing, Protein Expression, Copy Number, and DNA Methylation data (provided to the community through the TCGA Data Portal) and made readily available for sample selection and analysis within GenePool. As validation of the GenePool platform, we will present results obtained within minutes that validate knowledge reported in the literature over the course of decades. We will then present novel findings for less understood cancer indications. Note: This abstract was not presented at the meeting. Citation Format: Sandeep Sanga, Antoaneta Vladimirova, Richard D. Goold, Tod M. Klingler. GenePool: A cloud-based technology for rapidly data mining large-scale, patient-derived cancer genomic cohorts including The Cancer Genome Atlas. [abstract]. In: Proceedings of the 106th Annual Meeting of the American Association for Cancer Research; 2015 Apr 18-22; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Res 2015;75(15 Suppl):Abstract nr 4870. doi:10.1158/1538-7445.AM2015-4870

Towards scalable real-time analytics

We present an overview of our work on the SAP HANA Scale-out Extension, a novel distributed database architecture designed to support large scale analytics over real-time data. This platform permits high performance OLAP with massive scale-out capabilities, while concurrently allowing OLTP workloads. This dual capability enables analytics over real-time changing data and allows fine grained user-specified service level agreements (SLAs) on data freshness. We advocate the decoupling of core database components such as query processing, concurrency control, and persistence, a design choice made possible by advances in high-throughput low-latency networks and storage devices. We provide full ACID guarantees and build on a logical timestamp mechanism to provide MVCC-based snapshot isolation, while not requiring synchronous updates of replicas. Instead, we use asynchronous update propagation guaranteeing consistency with timestamp validation. We provide a view into the design and development of a large scale data management platform for real-time analytics, driven by the needs of modern enterprise customers.