TH-E-BRC-04: Monte-Carlo Simulation in a Cloud Computing Environment with MapReduce

Abstract

Purpose: Monte Carlo simulation is considered the most reliable method for modeling photon and electron transport in heterogeneous media. However, its widespread use is prohibited by the high computational cost. The purpose of this work is to report our implementation of a simple MapReduce method for performing fault‐tolerant Monte‐Carlo computations in a massively‐ parallel cloud computing environment. Designed for solving Internet‐scale problems, MapReduce hides the complexity of parallelization, data distribution, fault‐tolerance, and load balancing, thereby vastly simplifying parallel code development. Methods: We ported MC321, a commonly‐used Monte‐Carlo package for optical photontransport to Hadoop, an open‐ source MapReduce framework. Photon histories were computed using parallel Map tasks, and photon absorption scored using Reduce tasks. The distributed implementation was executed on a commercial compute cloud, using the propagation of optical photons in a turbid phantom as a benchmark. Results: This novel implementation simulated 100 billion photon histories on 240 virtual nodes in 1 hour 14 minutes, a 378X speed‐up compared to the single‐threaded execution of the Monte‐Carlo program. The performance of the MapReduce implementation scaled nearly linearly with the number of nodes. The overall computational throughput was 85,178 photon histories per node per second, with a latency of 100 seconds. The distributed simulation produced the same output as the original implementation, and was resilient to hardware failure: the correctness of the simulation was unaffected by the failure of 50% of all worker nodes. Conclusions: MapReduce is a new tool to rapidly deploy reliable and scalable applications on hundreds of nodes in the cloud. Applied to medical physics, these technologies could substantially accelerate the translation of novel algorithms into practical use.