Data warehouse processing and query by mail

Abstract

Much effort in I/S is currently going into creating "Data Warehouses." These are stores of data periodically extracted from older legacy applications, converted to common standards and made accessible for user analysis. The warehouse acts as a WORM (Write Once, Read Many times) storage. Where the extract and transfer is performed nightly, they provide access to what is termed "Near Operational" data and can be used to replace much of the existing reporting. In other cases they are used to store mostly historical data for analysis of trends, market impact, financial status and do on. While often implemented with a variety of different clean up tools, languages, data base products and query tools, this article describes an implementation done almost entirely with APL. It includes a query capability termed "Query by Mail" which enables anyone with access to E-Mail to send queries to the Warehouse and receive responses or extracts of data by return mail. The "query" includes customized analysis of field content to allow identification of fields and records containing invalid data. Built upon a proprietary inverted file system, it provides rapid response to user queries and little load on the server system.