Large-scale Scientific Data Research Articles

An exploration of SciDB in the context of emerging technologies for data stores in particle physics and cosmology

Traditional relational databases have not always been well matched to the needs of data-intensive sciences, but efforts are underway within the database community to attempt to address many of the requirements of large-scale scientific data management. One such effort is the open-source project SciDB. Since its earliest incarnations, SciDB has been designed for scalability in parallel and distributed environments, with a particular emphasis upon native support for array constructs and operations. Such scalability is of course a requirement of any strategy for large-scale scientific data handling, and array constructs are certainly useful in many contexts, but these features alone do not suffice to qualify a database product as an appropriate technology for hosting particle physics or cosmology data. In what constitutes its 1.0 release in June 2011, SciDB has extended its feature set to address additional requirements of scientific data, with support for user-defined types and functions, for data versioning, and more. This paper describes an evaluation of the capabilities of SciDB for two very different kinds of physics data: event-level metadata records from proton collisions at the Large Hadron Collider (LHC), and the output of cosmological simulations run on very-large-scale supercomputers. This evaluation exercises the spectrum of SciDB capabilities in a suite of tests that aim to be representative and realistic, including, for example, definition of four-vector data types and natural operations thereon, and computational queries that match the natural use cases for these data.

A semantic framework for automatic generation of computational workflows using distributed data and component catalogues

Computational workflows are a powerful paradigm to represent and manage complex applications, particularly in large-scale distributed scientific data analysis. Workflows represent application components that result in individual computations as well as their interdependences in terms of dataflow. Workflow systems use these representations to manage various aspects of workflow creation and execution for users, such as the automatic assignment of execution resources. This article describes an approach to automating a new aspect of the process: the selection of application components and data sources. We present a novel approach that enables users to specify varying degrees of detail and amount of constraints in a workflow request, including the specification of constraints on input, intermediate or output data in the workflow, abstract workflow component classes rather than specific component implementations, and generic reusable workflow templates that express a pre-defined combination of components. The algorithm elaborates the user request into a set of fully ground workflows with specific choices of data sources and codes to be used so that they can be submitted for mapping and execution. The algorithm searches through the space of possible candidate workflows by creating increasingly more specialized versions of the original template and eliminating candidates that violate constraints cumulated in the candidate workflow as components and data sources are selected. A novel feature of our approach is that it assumes a distributed architecture where data and component catalogues are separate from the workflow system. The algorithm explicitly poses queries to external catalogues, and therefore any reasoning regarding data or component properties is not assumed to occur within the workflow system. We describe our implementation of this approach in the Wings workflow system. This implementation uses the W3C Web Ontology Language and associated reasoners to implement the workflow system as well as the data and component catalogues. This research demonstrates the use of artificial intelligence techniques to support the kinds of automation envisioned by the scientific community for large-scale distributed scientific data analysis.

Feature-Based Statistical Analysis of Combustion Simulation Data

We present a new framework for feature-based statistical analysis of large-scale scientific data and demonstrate its effectiveness by analyzing features from Direct Numerical Simulations (DNS) of turbulent combustion. Turbulent flows are ubiquitous and account for transport and mixing processes in combustion, astrophysics, fusion, and climate modeling among other disciplines. They are also characterized by coherent structure or organized motion, i.e. nonlocal entities whose geometrical features can directly impact molecular mixing and reactive processes. While traditional multi-point statistics provide correlative information, they lack nonlocal structural information, and hence, fail to provide mechanistic causality information between organized fluid motion and mixing and reactive processes. Hence, it is of great interest to capture and track flow features and their statistics together with their correlation with relevant scalar quantities, e.g. temperature or species concentrations. In our approach we encode the set of all possible flow features by pre-computing merge trees augmented with attributes, such as statistical moments of various scalar fields, e.g. temperature, as well as length-scales computed via spectral analysis. The computation is performed in an efficient streaming manner in a pre-processing step and results in a collection of meta-data that is orders of magnitude smaller than the original simulation data. This meta-data is sufficient to support a fully flexible and interactive analysis of the features, allowing for arbitrary thresholds, providing per-feature statistics, and creating various global diagnostics such as Cumulative Density Functions (CDFs), histograms, or time-series. We combine the analysis with a rendering of the features in a linked-view browser that enables scientists to interactively explore, visualize, and analyze the equivalent of one terabyte of simulation data. We highlight the utility of this new framework for combustion science; however, it is applicable to many other science domains.

From social data mining to forecasting socio-economic crises.

.The purpose of this White Paper of the EU Support Action “Visioneer”(see www.visioneer.ethz.ch) is to address the following goals: Develop strategies to quickly increase the objective knowledge about social and economic systems.Describe requirements for efficient large-scale scientific data mining of anonymized social and economic data.Formulate strategies how to collect stylized facts extracted from large data set.Sketch ways how to successfully build up centers for computational social science.Propose plans how to create centers for risk analysis and crisis forecasting.Elaborate ethical standards regarding the storage, processing, evaluation, and publication of social and economic data.

Computation in large-scale scientific and internet data applications is a focus of MMDS 2010

A report is provided for the ACM SIGKDD community about the 2010 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2010), its origin in MMDS 2006 and MMDS 2008, and future directions for this interdisciplinary research area.

Semantic enabled metadata management in PetaShare

We designed a semantic enabled metadata framework using ontology for multi-disciplinary and multi-institutional large-scale scientific data sets in a Data Grid setting. Two main issues are addressed: data integration for semantically and physically heterogeneous distributed knowledge stores, and semantic reasoning for data verification and inference in such a setting. This framework enables data interoperability between otherwise semantically incompatible data sources, cross-domain query capabilities and multi-source knowledge extraction. In this paper, we present the basic system architecture for this framework, as well as an initial implementation. We also analyse a real-life scenario and show integration of our framework into the PetaShare Data Grid where multi-disciplinary data archives are geographically distributed across six research institutions in Louisiana.

Common Data Format Archiving of Large-Scale Intelligent Transportation Systems Data for Efficient Storage, Retrieval, and Portability

Intelligent transportation system (ITS) sensor networks, such as road weather information and traffic sensor networks, typically generate enormous amounts of data. As a result, archiving, retrieval, and exchange of ITS sensor data for planning and performance analysis are becoming increasingly difficult. An efficient ITS archiving system that is compact and exchangeable and allows efficient and fast retrieval of large amounts of data is essential. A proposal is made for a system that can meet the present and future archiving needs of large-scale ITS data. This system is referred to as common data format (CDF) and was developed by the National Space Science Data Center for archiving, exchange, and management of large-scale scientific array data. CDF is an open system that is free and portable and includes self-describing data abstraction. Archiving traffic data by using CDF is demonstrated, and its archival and retrieval performance is presented for the Minnesota Department of Transportation–s 30-s traffic data collected from about 4,000 loop detectors around Twin Cities freeways. For comparison of the archiving performance, the same data were archived by using a commercially available relational database, which was evaluated for its archival and retrieval performance. This result is presented, along with reasons that CDF is a good fit for large-scale ITS data archiving, retrieval, and exchange of data.

Grids, the TeraGrid and beyond

The correlation, combination, and statistical analysis of large data volumes derived from multiple sources depend on joining a new generation of high-resolution scientific instruments, high-performance computing systems, and large-scale scientific data archives via high-speed networks and a software infrastructure that enables resource and data sharing by collaborating groups of distributed researchers. Scheduled for completion in 2003, the National Science Foundation's TeraGrid, a massive research computing infrastructure, will combine five large computing and data management facilities and support many additional academic institutions and research laboratories in just such endeavors. When operational, the TeraGrid will help researchers solve problems in fields such as genomics, biology, and high-energy physics.

A scalable virtual environment for large scale scientific data analysis

The growing complexity and sheer amount of data resulting from supercomputer simulations increases the need for new and intuitive analysis techniques such as VR. In this paper we present a scalable VR system integrated in a distributed and collaborative visualization and simulation environment which can be used for scientific data analysis and computational steering in several application domains. After discussing the requirements for such a system we describe the basic architecture of the VR environment. We then give some examples of currently investigated application domains in the areas of water turbine optimization and space vehicle re-entry and comet impact simulation involving metacomputing on a trans-atlantic scale.

Adding Intelligence to Scientific Data Management

First Page