The Parallel Image Processing Environment (PIPE): automated parallelization of satellite data analyses

Abstract

AbstractA Beowulf‐type cluster can: (1) mitigate many issues associated with the analysis of large, complex remotely sensed data sets; (2) shorten the response time of operational agencies to crisis‐management situations; and (3) expedite the reanalysis of large archives of satellite data. Whereas most Beowulf‐type designs support modeling applications, the Parallel Image Processing Environment (PIPE) addresses the unique requirements of remote sensing applications. PIPE has four hierarchical layers: hardware, operating system, middleware and applications. Rocks, a middleware sublayer, manages the cluster. DIAL‐developed interprocess communication and control daemons form the second middleware sublayer. They encapsulate user‐defined applications and thereby support automated, user‐transparent parallelization of satellite data analyses, implemented in the applications layer using generalized constructs. The daemons also monitor resource (computational and I/O) utilization on a node/thread basis, a feature not supported by other generally available monitoring utilities. The application support libraries are fully extensible, facilitate the reuse of modular and commonly used software functions in new applications and thereby reduce both the cost and time to implement new applications. Two applications (signal analysis, image classification) show PIPE's versatility and performance characteristics. PIPE is intrinsically scalable, reliable and can be incrementally implemented. A comparison with other embarrassingly parallel systems is also provided. Copyright © 2006 John Wiley & Sons, Ltd.