Data Mining Meets Grid Computing: Time to Dance?

Abstract

A grand challenge problem (Wah, 1993) refers to a computing problem that cannot be solved in a reasonable amount of time with conventional computers. While grand challenge problems canbefoundinmanydomains,scienceapplicationsaretypicallyattheforefrontoftheselargescalecomputingproblems.Fundamentalscientificproblemscurrentlybeingexploredgenerate increasingly complex data, require more realistic simulations of the processes under study and demand greater and more intricate visualizations of the results. These problems often require numerous complex calculations and collaboration among people with multiple disciplines and geographic locations. Examples of scientific grand challenge problems include multi-scale environmentalmodellingandecosystemsimulations,biomedicalimagingandbiomechanics,nuclear power and weapons simulations, fluid dynamics and fundamental computational science (use of computation to attain scientific knowledge) (Butler, 1999; Gomes and Selman, 2005). Many grand challenge problems involve the analysis of very large volumes of data. Data mining (also known as knowledge discovery in databases) (Frawley, Piatetsky-Shapiro and Matheus, 1992) is a well stablished field of computer science concerned with the automated search of large volumes of data for patterns that can be considered knowledge about the data. Dataminingisoftendescribedasderivingknowledgefromtheinputdata.Applyingdatamining to grand challenge problems brings its own computational challenges. One way to address these computational challenges is grid computing (Kesselman and Foster, 1998). ‘Grid’ refers topersistentcomputingenvironmentsthatenablesoftwareapplicationstointegrateprocessors, storage, networks, instruments, applications and other resources that are managed by diverse organizations in widespread locations. This chapter describes how both paradigms ‐ data mining and grid computing ‐ can benefit from each other: data mining techniques can be efficiently deployed in a grid environment and operational grids can be mined for patterns that may help to optimize the effectiveness and efficiency of the grid computing infrastructure. The chapter will also briefly outline the chapters of this volume.