Architectural challenges and solutions for petascale postprocessing

Abstract

With petascale applications comes petascale data; gleaning knowledge and insight from this large-scale data is widely accepted to be a limiting factor in many fields of scientific endeavor. Large-scale data presents two incredible challenges: scale and complexity. The scale challenge refers to the problem of being able to process tremendous numbers of bytes of data within some time constraint. The complexity challenge is more subtle; petascale data is inherently complex and reducing this data to comprehensible forms is difficult.For the scale challenge, it is often assumed that continued usage of techniques viable at the terascale will provide a recipe for success at the petascale. We argue the opposite: following this path will result in cost prohibitive solutions. Instead, we must pursue ‘smarter’ techniques, such as in situ and multi-resolution processing, which are well established by a decade's worth of research.The forms of postprocessing are very diverse, but they can be summarized in broad use cases: data exploration, presentation graphics, quantitative analysis, comparative analysis, and visual debugging. Together, these use cases are responsive to the complexity challenge. Moreover, these use cases are responsive to the demands of the user community. Making pretty pictures is just one component of the simulation community's postprocessing needs and all of these needs must be addressed for petascale data.A major focus of this paper is to construct a road map for realizing this broad set of use cases at the petascale, and, further, to do this in the most economical way possible. None of the smart processing techniques we consider are a panacea; none are capable of supporting every use case. However, a software architecture that underpins these processing techniques and that can dynamically select between them will provide an economical way to meet these challenging requirements, and in this paper we discuss such an architecture.