Messy data – clean software – brilliant results?

Abstract

The historian's attitude to data and the way he uses it presents some thing of a challenge to the software writer and computer user. The histo rian's data is to be found in crumbling, smelly and dusty ledgers, buried in an obscure part of a crumbling, smelly and dusty archive. Opening one of these ledgers, he discovers that the entries are scrawled higgledy-piggledy over the page. The ink is faded: the language, if not if not in an obscure Latin, will consist of original clerk's shorthand and terminology, the mea ning of which was lost centuries ago. Our job is to translate all of this into authorative conclusions, gleaming graphics and credible generalisations. Before discussing potential ways of making the impossible merely dif ficult, it is worth reminding ourselves of why historians use the computer to produce results from this kind of data. Databases of historical records can act as a reference tool in other words we can use them to extract information on named individuals or houses or other items of interest. However, it is unlikely that many historians would be prepared to devote massive amounts of time to building a large database on the offchance that it contains references to specific individuals. Generally historians use the computer because they wish firstly to have the ability to pursue the parti cular, secondly to have the ability to identify groups according to particu lar shared characteristics, and thirdly in order to treat data relating to groups in some quantitative way. The historian wants to discover the make up of the early family, and to do this he loads a number of parish registers onto the computer. He asks questions of he data such as the number in each household, perhaps comparing one area with another, one social class with another or one period with another. Alternatively he might be in terested in patterns of trade, and so he loads shipping registers onto the computer and performs simple calculations on each of the relevant com modities. At the same time, in both cases, the original material is stored on the computer in a way that serves the enquirer searching for a particular early modern family, or a particular ship or commodity. These examples are used to emphasise the fact that historians frequently use the computer to store, retrieve or sort textual information which they wish to describe in a numeric way. The calculations may be simple (pos