Large-scale Data Management Systems Research Articles

Qanaat

Today's large-scale data management systems need to address distributed applications' confidentiality and scalability requirements among a set of collaborative enterprises. This paper presents Qanaat , a scalable multi-enterprise permissioned blockchain system that guarantees the confidentiality of enterprises in collaboration workflows. Qanaat presents data collections that enable any subset of enterprises involved in a collaboration workflow to keep their collaboration private from other enterprises. A transaction ordering scheme is also presented to enforce only the necessary and sufficient constraints on transaction order to guarantee data consistency. Furthermore, Qanaat supports data consistency across collaboration workflows where an enterprise can participate in different collaboration workflows with different sets of enterprises. Finally, Qanaat presents a suite of consensus protocols to support intra-shard and cross-shard transactions within or across enterprises.

Towards a New Data Replication Management in Cloud Systems

Applications produce huge volumes of data that are distributed on remote and heterogeneous sites. This generates problems related to access and sharing data. As a result, managing data in large-scale environments is a real challenge. In this context, large-scale data management systems often use data replication, a well-known technique that treats generated problems by storing multiple copies of data, called replicas, across multiple nodes. Most of the replication strategies in these environments are difficult to adapt to cloud environments. They aim to achieve the best performance of the system without meeting the important objectives of the cloud provider. This article proposes a new dynamic replication strategy. The proposed algorithm significantly improves provider gain without neglecting customer satisfaction.

Survey of Large-Scale Data Management Systems for Big Data Applications

Today, data is flowing into various organizations at an unprecedented scale. The ability to scale out for processing an enhanced workload has become an important factor for the proliferation and popularization of database systems. Big data applications demand and consequently lead to the developments of diverse large-scale data management systems in different organizations, ranging from traditional database vendors to new emerging Internet-based enterprises. In this survey, we investigate, characterize, and analyze the large-scale data management systems in depth and develop comprehensive taxonomies for various critical aspects covering the data model, the system architecture, and the consistency model. We map the prevailing highly scalable data management systems to the proposed taxonomies, not only to classify the common techniques but also to provide a basis for analyzing current system scalability limitations. To overcome these limitations, we predicate and highlight the possible principles that future efforts need to be undertaken for the next generation large-scale data management systems.

Data replication strategies with performance objective in data grid systems: a survey

Replicating for performance constitutes an important issue in large-scale data management systems. In this context, a significant number of replication strategies have been proposed for data grid systems. Some works classified these strategies into static vs. dynamic or centralised vs. decentralised or client vs. server initiated strategies. Very few works deal with a replication strategy classification based on the role of these strategies when building a replica management system. In this paper, we propose a new replication strategy classification based on objective functions of these strategies. Also, each replication strategy is designed according to the data grid topology for which it was proposed. We point out the impact of the topology on replication performance although most of these strategies have been proposed for a hierarchical grid topology. We also study the impact of some factors on performance of these strategies, e.g. access pattern, bandwidth consumption and storage capacity.

Design and Implementation of Large-Scale Data Management System Based on C/S Structure

With the development of computer and network technology, the storage and analysis for complex and large-scale of data can be implemented. According to the requirement, this paper designs a large-scale data management system based on C/S structure which realizes concentration, integration, security management, in-depth analysis and report generation of the field data. Application of the system can improve the scientific management level of enterprises, enhance employee productivity, ensure security, integrity and accuracy of the data, and provide important data for the performance review and decisions of enterprises development, thus improving economic efficiency of enterprises.

RDF in the clouds: a survey

The Resource Description Framework (RDF) pioneered by the W3C is increasingly being adopted to model data in a variety of scenarios, in particular data to be published or exchanged on the Web. Managing large volumes of RDF data is challenging, due to the sheer size, the heterogeneity, and the further complexity brought by RDF reasoning. To tackle the size challenge, distributed storage architectures are required. Cloud computing is an emerging paradigm massively adopted in many applications for the scalability, fault-tolerance, and elasticity feature it provides, enabling the easy deployment of distributed and parallel architectures. In this article, we survey RDF data management architectures and systems designed for a cloud environment, and more generally, those large-scale RDF data management systems that can be easily deployed therein. We first give the necessary background, then describe the existing systems and proposals in this area, and classify them according to dimensions related to their capabilities and implementation techniques. The survey ends with a discussion of open problems and perspectives.

Popularity framework for monitoring user workload

This paper describes a monitoring framework for large scale data management systems with frequent data access. This framework allows large data management systems to generate meaningful information from collected tracing data and to be queried on demand for specific user usage patterns in respect to source and destination locations, period intervals, and other searchable parameters. The feasibility of such a system at the petabyte scale is demonstrated by describing the implementation and operational experience of a real world management information system for the ATLAS experiment employing the proposed framework. Our observations suggest that the proposed user monitoring framework is capable of scaling to meet the needs of very large data management systems.

Entangled queries

Many data-driven social and Web applications involve collaboration and coordination. The vision of Declarative Data-Driven Coordination (D3C), proposed in Kot et al. [2010], is to support coordination in the spirit of data management: to make it data-centric and to specify it using convenient declarative languages. This article introduces entangled queries , a language that extends SQL by constraints that allow for the coordinated choice of result tuples across queries originating from different users or applications. It is nontrivial to define a declarative coordination formalism without arriving at the general (NP-complete) Constraint Satisfaction Problem from AI. In this article, we propose an efficiently enforceable syntactic safety condition that we argue is at the sweet spot where interesting declarative power meets applicability in large-scale data management systems and applications. The key computational problem of D3C is to match entangled queries to achieve coordination. We present an efficient matching algorithm which statically analyzes query workloads and merges coordinating entangled queries into compound SQL queries. These can be sent to a standard database system and return only coordinated results. We present the overall architecture of an implemented system that contains our evaluation algorithm. We also describe a proof-of-concept Facebook application we have built on top of this system to allow friends to coordinate flight plans. Finally, we evaluate the performance of the matching algorithm experimentally on realistic coordination workloads.

Medical informatics: the future of organized healthcare or Orwellian medicine?

Public healthcare information technologies or ‘medical informatics’ may very well be-come the preeminent medical development of the 21st century, akin to the breakthrough role antimicrobial therapies played starting in the late 19th century and the impact that genomics were starting to have in the 20th century. If current trends prevail, large-scale data management systems, fueled by the combination of rapid developments in medical information systems and rapidly decreasing costs for data storage and retrieval, will be connected almost directly to all but the most remote or destitute of medical practices anywhere in the world. Currently, United States government agencies collect a wide variety of hospital discharge data on most inpatients. With such administrative datasets, it is then up to the holder of the data to determine how our practices are rated and what support we will receive for our care delivery in the future. Never mind that some items such as ‘hyponatremia’ (one of the currently collected screening values) will likely have little or next to nothing to do with the quality of care that a spine surgery patient may have received. If a patient turns up ‘hyponatremic’ on a federal discharge database it results in a demerit for the practitioner and the hospital without chance for appeal or explanation. Are current practices, such as recording a sodium level, the harbinger of quality of care for remote specialties such as spine surgery in the future? Probably not, but it isn’t all that farfetched to imagine comprehensive data sets being gathered to screen our practice patterns. The sophistication of data gathering will undoubtedly grow, allowing for iden-tifi cation of blemishes and real complications alike in an unprecedented fashion. From there it is but a small step to envision future healthcare systems that are driven by data analysis of apparent ‘evidence’ to decree treatment decisions based on abstract principles without the power of human touch. This is the tipping point where the currently popular, but largely irrelevant ‘guidelines’ (usually turned out by professional societies) will be rendered completely irrelevant by implementation of healthcare ‘policies’ decreed by governmental organizations and powerful third-party payor corporations. This vision is very much in keeping with the dystopian prophecies of author George Orwell, perhaps most well known for his bestsellers “Animal Farm” written in 1945 and “Nineteen Eighty-Four” written in 1949, who predicted healthcare being administered through a remote healthcare bureaucracy that took it upon itself to decide what was best for its citizens. The current, undeniably fascinating, leap in information technology could indeed result in the realization of the Orwellian nightmare: healthcare decreed by ‘careless’ anonymous bureaucratic calculation with, human disease and its treatment reduced to a formula and issued in the form of a mandated decision.Aside from this nightmarish scenario—are there some potential benefi ts for patient care inherent to a progressive implementation of healthcare informatics? Sure! There are fi ve major points that immediately come to mind:• Rare diseases will become much more transparent with modes of detection and treatment as long as they are well-identifi ed a priori in large-scale data gathering.

Cognitive networking of large scale wireless systems

We propose the concept of cognitive networking for large-scale wireless systems, which opportunistically utilises network resources including both spectrum bandwidth and radio availability. Both types of resources cannot be predetermined in large-scale wireless systems, due to various reasons such as interferences and dynamic traffic load. The proposed technology not only establishes dynamic wireless networks, but also provides for reliable network quality of services (QoS). The supporting network architecture, embedded wireless interconnect (EWI), is proposed to implement the cognitive networking concept and supply an effective application-programming interface for large-scale data management systems. Two example applications are presented, including wireless mesh networks for broadband wireless internet access and wireless sensor networks for target tracking. Major advantages of the technology are further discussed. We suggest that the performance of the proposed system would improve with larger network scale and the implementation complexity could be independent of the network scale.

A research agenda for query processing in large-scale peer data management systems

Peer Data Management Systems (P dms) are a novel, useful, but challenging paradigm for distributed data management and query processing. Conventional integrated information systems have a hierarchical structure with an integration component that manages a global schema and distributes queries against this schema to the underlying data sources. P dms are a natural extension to this architecture by allowing each participating system (peer) to act both as a data source and as an integrator. Peers are interconnected by schema mappings, which guide the rewriting of queries between the heterogeneous schemas, and thus form a P2P (peer-to-peer)-like network. Despite several years of research, the development of efficient P dms still holds many challenges. In this article we first survey the state of the art on peer data management: We classify P dms by characteristics concerning their system model, their semantics, their query planning schemes, and their maintenance. Then we systematically examine open research directions in each of those areas. In particular, we observe that research results from both the domain of P2P systems and of conventional distributed data management can have an impact on the development of P dms.