Mass Digitization Projects Research Articles

Completing the Completeness Measure: the MIDS Specification

The need to rapidly digitise millions of specimens in natural history collections has seen the development of a staged approach for data capture by institutions that have been carrying out mass digitisation projects. The cost and difficulty of transcribing labels associated with specimens has necessitated an approach that prioritises parts of the data based on research and curatorial requirements as well as ease of transcription and methodology available. This approach means that there is currently a wide variation in the completeness of data capture and imaging of specimens within and between collections. The Biodiversity Information Standards (TDWG) Minimum Information about a Digital Specimen (MIDS) specification has been designed to provide a framework for measuring and monitoring the completeness of the data transcribed from the specimen and the presence of an image or other media file. This framework enables a consistent and standardised approach to the calculation of the level of digitisation of specimens within a collection and, by extension/extrapolation, to the collection itself. Four levels of digitisation have been defined in the MIDS specification including a pre-digitisation level. These have been developed to correspond to practical requirements and existing large-scale digitisation programmes. MIDS level 0 (Bare): A bare or skeletal record making the association between an identifier of a physical specimen and its digital representation, allowing for unambiguous attachment of all other information. MIDS level 1 (Basic): A basic record of specimen information enabling basic discoverability as well as how the user is permitted to use the data. MIDS level 2 (Regular): A regular level of information including data that have been agreed over time as essential for most scientific purposes. MIDS level 3 (Extended): An extended level of information about a specimen including identifiers enabling connections to be made to other data present or known about the specimen. The scope of MIDS does not directly include the quality of data entered. Instead, MIDS can be seen as a tool to support the continuing improvement of data access and quality. It provides guidance for prioritising the data to be captured as well as recommendations for data standards and mapping structures. Here we present the progress in the development of the MIDS specification and its implementation. The MIDS elements for each of the four levels have now been defined and a draft version is available here. A machine-readable mapping to Darwin Core and ABCD has been compiled following the Simple Standard for Sharing Ontological Mappings (SSSOM). The differing data relevances and priorities for different disciplines have been recognised and the inclusion of required elements at each level is discipline specific (biology, palaeontology and geology). In addition, MIDS level scores have been increasingly implemented in existing specimen management systems, allowing an evaluation of the levels achieved in various contexts. There are two implementations of MIDS calculators that have been developed and are available to test. One, developed at the Meise Botanic Garden, uses a zipped Global Biodiversity Information Facility (GBIF) annotated archive (a commonly used GBIF download format based on Darwin Core Archive) and is available here. The other, developed at The Natural History Museum London, interacts directly with the GBIF API and is available here. We encourage users to try these and provide feedback to the TDWG MIDS Task Group.

Digitizing the USPTO patent backfile

Abstract The digitization of the US Patent and Trademark Office’s (USPTO) backfile of six million patents undertaken between 1951 and 2001 was a five-decade struggle, featuring several media transitions from print and microfilm to CD-ROMs and, finally, the Web. This mass digitization project is on a similar scale to Google Books and the Internet Archive, but it is rarely discussed within critical digitization scholarship or for its significance as a tool for knowledge production. In this article, I focus on the USPTO’s patent document’s digital and physical material form and how the current paradigm of access and storage of the digital backfile emerged. Through this case study, I build upon Ian Milligan’s distinction between the ‘text’ and ‘platform’ layers of a digitization project to demonstrate how historical decisions regarding format and metadata continue to influence how users retrieve and interpret documents, such as patents, online.

Ontsluiting van het Amsterdamse notariële archief

The archives of the Amsterdam notaries (1578-1915) are an extremely rich but complex source for historical research. Although well known amongst researchers for decades, it was never possible to search the archive completely due to its enormous size (over 3.5 kilometers of shelve length). From 2016 onwards the Amsterdam City Archives have started a renewed attempt to disclosure this archive with new techniques such as crowd sourcing and Handwritten Text Recognition in the project Alle Amsterdamse Akten. This article provides insight in the way the Amsterdam City Archives unlocks the deeds of the seventeenth and eighteenth century notaries and how researchers can benefit from the results of this mass digitization project.

Herbarium Specimen Label Interpretation and Transcription: First steps used to clean digitized data

Herbarium Specimen Label Interpretation and Transcription: First steps used to clean digitized data

Copyright and information market today: European initiatives and library privileges

Copyright transformation in the digital world is going ahead. The paper describes new European initiatives in copyright regulation, particularly those that exert influence on information market and the work of libraries and other cultural heritage institutions as well as educational establishments. New European directive on copyright officially called The Directive on Copyright in the Digital Single Market allows for broader use of in -copyright material online and across borders within the European Union for the purpose of education, research and cultural heritage preservation. The Directive simplifies digitization and distribution of out-of-commerce works and gives legal certainty to online use of copies of works of art that are in the public domain. One of the Directive articles gives libraries, archives and museums the right to make digital copies of copyright-protected works in their collections but exclusively for preservation purpose and to the extent necessary for preservation. These important reforms incorporate copyright limitations and exceptions in European legislation and open the way to mass digitization projects in European libraries. The paper also briefly describes LIBER (The Association of European Research Libraries) activities to promote short-term and long-term provisions in international and national copyright legislation to take into account public interests during healthcare, environmental and economic crises.

Copyright and information market today: European initiatives and library privileges

Copyright transformation in the digital world is going ahead. The paper describes new European initiatives in copyright regulation, particularly those that exert influence on information market and the work of libraries and other cultural heritage institutions as well as educational establishments. New European directive on copyright officially called The Directive on Copyright in the Digital Single Market allows for broader use of in-copyright material online and across borders within the European Union for the purpose of education, research and cultural heritage preservation. The Directive simplifies digitization and distribution of out-of-commerce works and gives legal certainty to online use of copies of works of art that are in the public domain. One of the Directive articles gives libraries, archives and museums the right to make digital copies of copyright-protected works in their collections but exclusively for preservation purpose and to the extent necessary for preservation. These important reforms incorporate copyright limitations and exceptions in European legislation and open the way to mass digitization projects in European libraries. The paper also briefly describes LIBER (The Association of European Research Libraries) activities to promote short-term and long-term provisions in international and national copyright legislation to take into account public interests during healthcare, environmental and economic crises.

An exploratory prioritization of factors affecting current state of information security in Pakistani university libraries

Ever since the increased use of the internet in higher education, libraries have started mass digitisation programs to make their intellectual resources available on the World Wide Web. From the year 2000 onward, Pakistani universities’ libraries have been active in this mass digitisation projects. The purpose of this paper is to examine the data and information security measures taken by university libraries to safeguard the intellectual assets of their respective organisations. Researchers adopted a quantitative approach to conduct this study. Data were collected from 190 professional librarians around the country. The findings of the study revealed that majority of libraries are lagging in standardised measures to secure digital resources. It is hoped that the present study will help in suggesting appropriate standards of data and information security within universities’ libraries. Results can also be replicated in other developing countries to investigate data and information security in libraries.

Criminal Life Courses in Context

The future of the Historical Sample of the Netherlands (HSN) will certainly include the enrichment of the foundational database with additional, new sources of information. In general, the HSN would highly benefit from current mass digitization projects involving citizen science. This essay proposes a pilot in linking 19th- and early 20th-century criminal records to HSN. In spite of the extensive state and parish registration documenting individual and family lives in close systematic detail, life course approaches to historical crime are less common. The large datasets necessary to conduct longitudinal life course research into deviant behaviour will facilitate both the analysis of criminality as an event and the scrutiny of the trajectories of individuals' lives leading up to their involvement in crime.

Digital camera calibration for cultural heritage documentation: the case study of a mass digitization project of religious monuments in Cyprus

ABSTRACT The paper summarizes the methodology followed, to evaluate the accuracy of different digitization methods of ecclesiastical monuments in 3D computer vision form and stresses the importance of photographic equipment calibration. In this study, a set of images were taken using the CANON EOS M5 digital camera, while the internal calibration parameters – horizontal and vertical focal length (fx, fy), principal point coordinates (x0, y0), radial distortion coefficients (K1, K2, K3), tangential distortion coefficients (P1, P2) and the affinity and the shear terms (b1, b2) were estimated. These parameters were calculated using different software applications and then analyzed. For the calibration procedure, 3D texture models were built with the Agisoft commercial software based on: (a) the aforementioned calibration parameters and (b) the self-calibration process. The overall accuracy (Root Mean Square – RMS) between these models, by comparing known geo-referenced ground-control-points (GCP) is presented through the Cloud Compare software. The results indicate that the internal calibration parameters of the digital camera used for documentation purposes are essential and should be systematically implemented for documentation purposes.

LABOUR AND DIGITIZATION

In 2011, artist Andrew Norman Wilson was working as a subcontractor doing video work for Google at the Mountainview California Googleplex. He’d noticed that the workers entering and exiting the building next door had different working conditions to the majority of the workers at the site and discovered they were working on the Google Books mass digitization project. One day he took his camera down to film them leaving the building at the end of the shift and the next day he was fired. Wilson used the footage to create a video he calls “Workers Leaving the Googleplex” (2011), and projects such as Krissy Wilson’s “The Art of Google Books” destabilise the fictional veneer of smooth clean surfaces that the Google Books project projects into the world. The growing field of critical infrastructure studies helps us to unpack and understand the multiple dimensions operating This paper argues that notions of the particularities of digital labour of all kinds are central to developing a more comprehensive understanding of the multifaceted nature of digital cultural objects, both born-digital and digitised. The paper considers the often invisible, contingent, omitted or assumed labour involved in digitization projects, using thinking from critical infrastructure studies, new media, media archaeology and creative labour.

What 5 Years of Digitisation Teaches Us About What Comes Next

The Natural History Museum, London (NHM) has now carried out more than five years of digitisation under its Digital Collections Programme (DCP), working with peers from around the world as well as with industrial partners. Data from this and similar programmes are a key input to shared infrastructures and knowledge, for informing research and decision-making for a sustainable future. In keeping with the Biodiversity Next aims of sharing experiences, promoting innovation and inspiring the next leap forward, this presentation will share key lessons learned, including what has gone well and what we will change going into the next phase. To tackle the challenge of 80 million items, prioritisation is always at the forefront of NHM’s approach. Scientific and public value, as well as feasibility, have remained at the core, but we will share how and why our prioritisation methods have evolved over time, managing multiple sources of project ideas to develop a balanced portfolio of mass digitisation, pilot studies and bespoke projects for key specimens that do not lend themselves to a mass approach. The DCP is not just a set of digitisation projects or workflows – it has always been intended as a business transformation programme, catalysing much wider organisational and cultural changes. In key areas, including data standards and the Museum’s data portal, DCP has led the way and changes have matured into ‘business as usual’. The Programme and portal teams have been a focus for a much wider span of digitising and databasing activity, involving curators and others, and lead or input to a growing technology portfolio, e.g. influencing collections management systems and data storage. We will discuss our ‘culture is as culture does’ approach to change management. From an early stage, DCP has sought to work with industrial and commercial partners, benefiting from their expertise and resources. We have also worked continuously to raise additional funds by ‘selling’ the Programme. This comes with considerable challenges, including selecting the right ‘window’ on digitisation for each potential partner or funder; managing internal resource demands e.g. to give tours and respond to partner needs; and finding the right partners for different projects. When it works, however, this can give access to new resources and approaches, pushing digitisation forward. To make the case for continued investment in the Programme, including the digitiser team, NHM has worked to develop a data-driven approach, consistently measuring the time and cost of digitisation workflows, as well as continuously improving the wider case around the impact of digital collections data, drawing on sources such as citations collated by GBIF. Sometimes that data, and assessment of lessons learned, has led us to stop or change our approach, for example to crowdsourcing transcription. A key challenge going forward remains how to estimate economic value from collections, their data, and the research that uses them, to give us the strongest possible case for the next stages of global efforts to digitise natural history collections.

History playground: A tool for discovering temporal trends in massive textual corpora

AbstractRecent studies have shown that macroscopic patterns of continuity and change over the course of centuries can be detected through the analysis of time series extracted from massive textual corpora. Similar data-driven approaches have already revolutionized the natural sciences and are widely believed to hold similar potential for the humanities and social sciences, driven by the mass-digitization projects that are currently under way, and coupled with the ever-increasing number of documents which are ‘born digital’. As such, new interactive tools are required to discover and extract macroscopic patterns from these vast quantities of textual data. Here we present History Playground, an interactive web-based tool for discovering trends in massive textual corpora. The tool makes use of scalable algorithms to first extract trends from textual corpora, before making them available for real-time search and discovery, presenting users with an interface to explore the data. Included in the tool are algorithms for standardization, regression, change-point detection in the relative frequencies of n-grams, multi-term indices, and comparison of trends across different corpora.

Media Digitization and Preservation Initiative: A Case Study

Since its creation nearly a decade ago, the Digital Curation Centre (DCC) Curation Lifecycle Model has become the quintessential framework for understanding digital curation. Organizations and consortia around the world have used the DCC Curation Lifecycle Model as a tool to ensure that all the necessary stages of digital curation are undertaken, to define roles and responsibilities, and to build a framework of standards and technologies for digital curation. Yet, research on the application of the model to large-scale digitization projects as a way of understanding their efforts at digital curation is scant. This paper reports on findings of a qualitative case study analysis of Indiana University Bloomington’s multi-million-dollar Media Digitization and Preservation Initiative (MDPI), employing the DCC Curation Lifecycle Model as a lens for examining the scope and effectiveness of its digital curation efforts. Findings underscore the success of MDPI in performing digital curation by illustrating the ways it implements each of the model’s components. Implications for the application of the DCC Curation Lifecycle Model in understanding digital curation for mass digitization projects are discussed as well as directions for future research.

Managing a Mass Digitization Project at Meise Botanic Garden: From Start to Finish

Mass digitization is a large undertaking for a collection. It is disruptive of routine and can challenge long-held practises. Having been through the procedure and survived, we feel we have a lot of experience to share with other institutions who are considering taking on this challenge. The changes that digitization has made to our institution are positive and the digitization a success, but that is not to say that we would not have done some things differently, were we to repeat the exercise. In 2015 Meise Botanic Garden received a grant from the Flemish Government to upgrade its digitization infrastructure and mass digitize 1.2 million specimens from its African and Belgian Herbaria. The new infrastructure improved our workflow significantly, enabling us to digitize specimens five to ten times faster while also improving their quality. The mass digitization part of the project was split into two parts, imaging and transcription. The contract was awarded and out-sourced to Picturae, who started imaging in May 2016 using a conveyor belt installation. Prior to starting, a significant amount of preparation was required at the herbarium. Within one year, 1.2 million specimens were imaged. The images were captured as TIFF files and stored in triplicate at The Flemish Institute for Archiving (VIAA), while smaller derived JPEG 2000 and JPEG files were generated for day-to-day use. The second part of the project was label transcription. A third of the specimens were transcribed in-house for capturing minimal data (barcode, filing name, collector, collector number & country of origin). This was partly done to reduce costs, but also allowed us to compare in-house to out-sourced transcription. Some 500,000 specimens were transcribed, either completely or partially, by Alembo (subcontracted by Picturae).The remaining 200.000 specimens from our Belgian Herbarium are being transcribed using crowdsourcing. The latter is being realized through the citizen science platform DoeDat (www.doedat.be) that was launched in November 2017. Many lessons have been learnt with respect to implementing mass digitization, both practically and sociologically. Many of the problems encountered during the project could have been avoided by changing the workflow. The addition of extra control points during the process could have reduced problems encountered later in the data capture process. Solving these problems at a later stage was time consuming. Trying to “save money” can result in a disruptive workflow, which may lead to a number of costly errors. Mass digitization has fundamentally changed the workflow in our collections and the way in which our herbarium is managed. All images for the African and Belgian collections may be now found on our new virtual herbarium www.botanicalcollections.be.

An Evaluation of In-house versus Out-sourced Data Capture at the Meise Botanic Garden (BR)

There are many ways to capture data from herbarium specimen labels. Here we compare the results of in-house verses out-sourced data transcription with the aim of evaluating the pros and cons of each approach and guiding future projects that want to do the same. In 2014 Meise Botanic Garden (BR) embarked on a mass digitization project. We digitally imaged of some 1.2 million herbarium specimens from our African and Belgian Herbaria. The minimal data for a third of these images was transcribed in-house, while the remainder was out-sourced to a commercial company. The minimal data comprised the fields: specimen’s herbarium location, barcode, filing name, family, collector, collector number, country code and phytoregion (for the Democratic Republic of Congo, Rwanda & Burundi). The out-sourced data capture consisted of three types: additional label information for central African specimens having minimal data; complete data for the remaining African specimens; and, species filing name information for African and Belgian specimens without minimal data. As part of the preparation for out-sourcing, a strict protocol had to be established as to the criteria for acceptable data quality levels. Also, the creation of several lookup tables for data entry was necessary to improve data quality. During the start-up phase all the data were checked, feedback given, compromises made and the protocol amended. After this phase, an agreed upon subsample was quality controlled. If the error score exceeded the agreed level, the batch was returned for retyping. The data had three quality control checks during the process, by the data capturers, the contractor’s project managers and ourselves. Data quality was analysed and compared in-house versus out-sourced modes of data capture. The error rate by our staff versus the external company was comparable. The types of error that occurred were often linked to the specific field in question. These errors include problems of interpretation, legibility, foreign languages, typographic errors, etc. A significant amount of data cleaning and post-capture processing was required prior to import into our database, despite the data being of good quality according to protocol (error < 1%). By improving the workflow and field definitions a notable improvement could be made in the “data cleaning” phase. The initial motivation for capturing some data in-house was financial. However, after analysis, this may not have been the most cost effective approach. Many lessons have been learned from this first mass digitisation project that will implemented in similar projects in the future.

Work Ethics for the <italic> Digitizer</italic>. Opportunities and Best Practices for Production of Digital Archives: the Working Experience of the Photographic Archive of Pompeii

In this presentation, we intend to examine the ethical and workrelated issues relative to mass digitization projects of photografic archives. Our paper asks: who and what is there behind these productions? Our goal is to analyze and contextualize the activities of the digitizer, the frontline and central figure in the transformation of the archives from analogue to digital. Historically, the transmission of knowledge is based upon a continuous copying process; whereas monks once manually transferred content from manuscript to manuscript, contemporary digitizers follow in their footsteps, giving old analogue photographs a new digital life by utilizing contemporary transcription pathways. These aspects will be examined starting with our working experience within the photographic archive of the Pompeii excavations. During this experience we elaborate upon the idea that digitizers must no longer be considered as mere mechanical performers in the process of digital acquisition, but as consciously trained professionals, able to engage critically in the various processes involved in the creation of a digital archive.

The Impossible Quest \u2013 Problems with Diligent Search for Orphan Works

Digital technologies allow unprecedented preservation and sharing of world-wide cultural heritage. Public and private players are increasingly entering the scene with mass digitisation projects that will make this possible. In Europe, legislative action has been taken to allow cultural institutions to include in their online collections copyright works whose owners are either unknown or non-locatable (“orphan works”). However, according to the Orphan Works Directive, cultural institutions must attempt to locate the owner of a work before using it. This is the so-called “diligent search” requirement. This paper provides an empirical analysis of the conditions under which a diligent search can feasibly be carried out. The United Kingdom, the Netherlands, and Italy, all of which have implemented the Orphan Works Directive, have been selected as case studies. For each jurisdiction, this analysis determines what the requirements for a diligent search to locate copyright holders are, what the authoritative sources and databases to be consulted are in practice and, most importantly, to what extent these are freely accessible online. In doing so, our analysis provides insights into the two main issues affecting cultural heritage institutions: (1) how much legal certainty does the implementation provide, and (2) what is the practical burden of a diligent search. The analysis reveals that the jurisdictions have given different meanings to the term “diligent”. While the UK’s extensive guidance makes it unlikely that a search would not be deemed diligent, the search burden is onerous. On the other hand, Italy and especially the Netherlands have a lighter search burden, but in the absence of clear, definite guidance, the likelihood of accidental infringement by failing to meet the diligence standard is greater. In addition, all three jurisdictions have so far failed to take the accessibility of the sources into account, making the searches even more onerous than the numbers suggest at first sight. Therefore, it will be difficult for cultural institutions to clear the rights for their collections while fully complying with the requirements of the legislation. This article concludes that legislative action, official guidelines, or jurisprudence are needed to establish a different legal value of sources for a diligent search, with various degrees of optionality depending on data relevance and accessibility.

Out of Print

In the 2000s an interconnected set of elite projects in the United States sought to digitize “all books in all languages” and make them available online. These mass digitization projects were efforts to absorb the print book infrastructure into a new one centered in computer networks. Mass book digitization has now faded from view, and here I trace its setbacks through a curious figure—the “orphan”—that emerged from within these projects and acted ultimately as an agent of impasse. In legal policy debates, an “orphan” refers to a copyrighted work whose owner cannot be found, but its history, range of meanings, and deployments reveal it to be considerably more complex. Based on fieldwork conducted at a digital library engaged in mass digitization, this paper analyzes the “orphan” as a personifying metaphor that digital library activists embraced in order to challenge and/or disrupt the social relations that adhere in and around books. The figure of the orphan haunts the techno-cultural infrastructural project of mass digitization, and stands, I argue, as a disenchanted emblem for “the book” in suspension between a disfavored past and a fantasized future.

Quality Assurance in Mass Digitization Projects

QA can vary from simple procedures to highly developed workflows. Within one of the largest digitization programs in the Netherlands (Metamorfoze) the National Library has the task to ensure the quality of digitized images for preservation. To accomplish that task the National Library has taken several steps and is constantly improving its own process of quality management to ensure high speed, high volume and high standard controls for a huge amount of terabytes of data that will be stored permanently and made available for (online) use. In this paper you can find the questions we had to answer in order to set up the QA workflow, the workflow we did implement, the current status of our workflow and the lessons we learned along the way.

The Making of an 'Orphan': Cultural Heritage Digitisation in the EU

As the definition of accessibility of cultural heritage changes under the impact of new technologies, public and private actors seek new ways to enhance digital access. The digitisation of 20th Century cultural heritage, however, is severely restricted by the potential subsistence of copyright and related rights. Different solutions have been considered to deal with this problem and, in particular: the adoption of a specific exception to copyright for libraries and other institutions with similar purposes (as in the United Kingdom and Germany); the use of the mechanism of extended collective licensing (as in Nordic countries); the creation of collecting societies responsible for the licensing of orphan works; the introduction of a public body in charge of licensing in a system of compulsory licenses (as in Canada); the limitation on remedy rule (as in the United States); or, finally, the adoption of national laws that provide for the automatic recognition and accessibility of orphan works as indicated in the other Member States. The EU’s own response has decidedly bet on the introduction of a copyright exception, through the Orphan Works Directive (2012/28/EU, hereinafter the “Directive”), which is thought of as an instrument to allow libraries, museums and other cultural institutions to digitise their collections of 19th and 20th century works. The Directive could potentially allow to lawfully digitise and make available a large number of works also with uncertain copyright status. Pursuant to the Directive, a work can be considered an orphan if, after a diligent search, its legitimate right-holders cannot be identified or located. Then, the EU pursued the scope to facilitate the digitisation and, consequently, the dissemination of works qualified as orphans, after a consultation of a large number of sources. Despite the emphasis which has surrounded its adoption, preliminary evidence shows that cultural institutions do not consider the scheme introduced by the Directive as scalable; in other words, they do not plan to use it as basis for mass digitisation projects. On the one side, the diligent search requirement is considered too onerous; on the other side, little attention is devoted to the underlying business models that would encourage digitisation. The involvement of private actors appears indeed limited to non-for-profit interventions, which explicitly rules out the main and principal digitiser on the global scale – the Google Books project. In this paper, we analyse the impact of the permitted uses on the digitization process of cultural heritage that can be carried out by those cultural institution benefitting from the exception. We will focus on the activity of making available (section 2) and reproduction for the purposes of digitization, making available, indexing, cataloguing, preservation and restoration (section 3). The division at hand is useful because the orphan work issue asks for different treatment depending on the type of use that would be made of the concerned works. On this background, we move to analyse the interaction between private and public sector in the field. In particular, we firstly analyse the ways into which the current framework envisages the funding of digitization processes and its limits (section 4). Within this context we assess the impact of public-private partnerships (section 5) to conclude by highlighting the weaknesses of the current EU framework for orphan works and its difficulties in achieving the goal of contributing to the creation of a European digital cultural heritage (section 6).