Data mining for “big archives” analysis: A case study

Abstract

AbstractWe present a case of archival analysis using a combination of data mining methods. The team of researchers, composed by archivists and computer scientists, used a collection of declassified Department of State Cables as a case study. The methods implemented included Support Vector Machine (SVM) and Association Rule Mining. Combined in an analysis workflow, the results of the different methods allowed the team to identify the different security classes, understand how they changed over time and generate descriptions for the cables in each class. The interpretation of results also allowed understanding contextual aspects of the collection. Until now, the use of data mining for archival analysis and processing has not been thoroughly explored by the archival community. This study constitutes a seminal roadmap to understand how to apply, interpret and integrate data mining with the archivists experience and judgment in collaboration with computer scientists. It proposes an inductive approach to archives analysis and the possibility of verifying processing decisions.