Abstract
We designed and implemented a parallel visualisation system for the analysis of large scale time-dependent particle type data. The particular challenge we address is how to analyse a high performance computation style dataset when a visual representation of the full set is not possible or useful, and one is only interested in finding and inspecting smaller subsets that fulfil certain complex criteria. We used Paraview as the user interface, which is a familiar tool for many HPC users, runs in parallel, and can be conveniently extended. We distributed the data in a supercomputing environment using the Hadoop file system. On top of it, we run Hive or Impala, and implemented a connection between Paraview and them that allows us to launch programmable SQL queries in the database directly from within Paraview. The queries return a Paraview-native VTK object that fits directly into the Paraview pipeline. We find good scalability and response times. In the typical supercomputer environment like the one we used for implementation the queue and management system make it difficult to keep local data in between sessions, which imposes a bottleneck in the data loading stage. This makes our system most useful when permanently installed on a dedicated cluster.
Highlights
High performance computer simulations can routinely reach the peta-scale, producing up to tens or even hundreds of GB of data per single time step
Mitchell, Ahrens and Wang [2] proposed a hybrid approach in which data was distributed across an HPC cluster using the Hadoop Distributed File System (HDFS), and used Kit-Wares ParaView as the user interface
The geometrical locality of data allowed to distribute the information such that each ParaView server had local access to the data needed for the distributed rendering
Summary
High performance computer simulations can routinely reach the peta-scale, producing up to tens or even hundreds of GB of data per single time step. The performance increased linearly with the number of nodes used, following the effective bandwidth available to ParaView This approach would have difficulty handling data that has time dependent geometry, or that requires parallel distribution along more than one variable. By using HDFS we are able to handle huge numbers of particles and time steps, and the coupling between ParaView and Hive allows us to interact, process, and visualise them in almostreal time. By not using HDFS directly but through Hive, our system allows us to distribute particles using criteria like physical properties or the number of particles. Another advantage of this setup is that the set of nodes running HDFS/Hive need not be the same used for Paraview. The set of filters we developed and tested were geared towards this biomechanical simulation, our system should be useful for all types of simulation that track particles moving in fluids or vector fields (e.g. supernova explosions or agent based modelling)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
