Self-optimising data farming for Web applications

Abstract

Many Web applications walk the thin line between the need for dynamic data and the need to meet user performance expectations. In environments where funds are not available to constantly upgrade hardware inline with user demand, alternative approaches need to be considered. We introduce a 'data farming' model whereby dynamic data, which is 'grown' in operational applications, is 'harvested' and 'packaged' for various consumer markets. Like any well managed agricultural operation, crops are harvested according to historical and perceived demand as inferred by a self-optimising process. This approach aims to make enhanced use of available resources through better utilisation of system downtime - thereby improving application performance and increasing the availability of key business data.