Submission Information Packages Research Articles

Red Queen in the Repository


 One of the grand curation challenges is to secure metadata quality in the ever-changing environment of metadata standards and file formats. As the Red Queen tells Alice in Through the Looking-Glass: “Now, here, you see, it takes all the running you can do, to keep in the same place.” That is, there is some “running” needed to keep metadata records in a research data repository fit for long-term use and put in place. One of the main tools of adaptation and keeping pace with the evolution of new standards, formats – and versions of standards in this ever-changing environment are validation schemas. Validation schemas are mainly seen as methods of checking data quality and fitness for use, but are also important for long-term preservation. We might like to think that our present (meta)data standards and formats are made for eternity, but in reality we know that standards evolve, formats change (some even become obsolete with time), and so do our needs for storage, searching and future dissemination for re-use. Eventually, we come to a point where transformation of our archival records and migration to other formats will be necessary. This could also mean that even if the AIPs, the Archival Information Packages stay the same in storage, the DIPs, the Dissemination Information Packages that we want to extract from the archive are subject to change of format. Further, in order for archival information packages to be self-sustainable, as required in the OAIS model, it is important to take interdependencies between individual files in the information packages into account. This should be done already by the time of ingest and validation of the SIPs, the Submission Information Packages, and along the line at different points of necessary transformation/migration (from SIP to AIP, from AIP to DIP etc.), in order to counter obsolescence.
 This paper investigates possible validation errors and missing elements in metadata records from three general purpose, multidisciplinary research data repositories – Figshare, Harvard’s Dataverse and Zenodo, and explores the potential effects of these errors on future transformation to AIPs and migration to other formats within a digital archive.
 

Long-Term Reusability of Biodiversity and Collection Data using a National Federated Data Infrastructure

GFBio “German Federation for Biological Data” is a data infrastructure and network set up by several research institutions in Germany. It fosters archiving and long-term reusability of research data and provides open and free access via a joint web portal at www.gfbio.org. As part of the working procedures data are semantically enriched and provided via a visualization and analysis tool. The main aim of the infrastructure is to make research data from the biological domain reusable and accessible on the long run following FAIR principles. In order to achieve this, several workflows and best practices have been established. The archiving of biodiversity and collection research data follows the reference model (ISO 14721) for an Open Archival Information System (OAIS). The challenges for making data reusable is on the one hand the heterogeneity of this data, on the other hand the often implicit but differing semantics making data integration a hard and difficult process. The use of data management plans is one approach we run to face and solve the challenges. Data management plans contain recipes about the research data, the tools used to acquire data, the content- and exchange formats, the metadata required to describe the data, and finally the costs and resources needed by data providers to deliver structured “Submission Information Packages” (SIPs) in the sense of OAIS. The archiving of a data package as “Archival Information Package” (AIP) is not sufficient to make it reusable in the future. Changes in the semantic meaning over time (content obsolescence), changes in the formats (format obsolescence), and changes in the technology of storage media (hardware obsolescence) are the major factors to be considered here. According to the FAIR principles and to our understanding data is best preserved if it is visible and available for use. The biodiversity and collection data centers involved in GFBio therefore have a curation layer (cf. management aka OAIS) in the archiving pipeline assembling their in-house management systems for sample and observation data and their asset management systems for all kinds of multimedia. This layer allows a continuous quality control and review of the incoming information packages. Thus, data providers can continuously maintain their data if wished for. The data are stored as AIPs sensu OAIS at the specialized data centers and are accessed by GFBio's core system. Dissemination Information Packages (DIPs) can be generated continuously at every time from the data and disseminated using content standards for data and metadata, like EML, ABCD and MIxS. Data are available via the GFBio website and in parallel using other web portals from the biological domain, e.g. INSDC and GBIF. The GFBio data centers now strive for certification of their archiving processes using the Core Trust Seal and for the certification of the FAIRness of single data records. Established data flows and documentation on best practices are available under: www.gfbio.org/data-centers.

Validation for preservation: how sustainable are GBIF datasets?

Validation using schemas and tools like the Darwin Core Archive Validator from GBIF are mainly seen as methods of checking data quality and fitness for use, but are also important for long-term preservation. We may like to think that our present (meta)data standards and formats are made for eternity, but in reality we know that standards evolve, formats change (some even become obsolete with time), and so do our needs for storage, searching and future dissemination for re-use. So we might eventually come to a point where transformation of our archival records and migration to other formats will be necessary. This could also mean that even if the AIPs, the Archival Information Packages stay the same in storage, the DIPs, the Dissemination Information Packages that we want to extract from the archive are subject to change of format. Further, in order for archival information packages to be self-sustainable as required in the OAIS model, it is important to take interdependencies between individual files in the information packages into account, already by the time of ingest and validation of the SIPs, the Submission Information Packages, and along the line at different points of necessary transformation / migration (from SIP to AIP, from AIP to DIP etc.) to counter obsolecense. Validation schemas and transformation code should also be archived together with the AIPs. By ensuring compliance with standards these tools are essential in controlling uniformity of records in a collection, for future needs of transformation and migration to new, sustainable formats. An example is given of the problems encountered in transforming only a small, relatively well defined collection of about 1000 archival items but with substantial variations between them, due to a lack of effective input constraints and validation at ingest. A further assessment is made of validation errors encountered in some Darwin Core Archives comprising thousands of records from some hundred published datasets, and how these errors might affect a future potential transformation / migration effort. Migration efforts must necessarily be general in scope, while errors in datasets from non-compliance with standards risk being reinforced or aggravated in the transformation process, making the information contained in the resulting records more difficult to interpret. The conclusion is that efforts should be made, e.g. by means of embedded validation measures into upload forms and other methods of information transfer (e.g. ftp, oai-pmh) to ensure as close compliance as possible to standards, already at the time of ingest. https://orcid.org/0000-0001-5699-994X

Os padrões de metadados como recursos tecnológicos para a garantia da preservação digital

A garantia de preservação digital a longo prazo só será possível com a adoção efetiva de padrões ou esquemas de metadados, pois são eles que determinarão a descrição, a representação, a consistência e a persistência do recurso/objeto digital no ambiente informacional, além de definir a interoperabilidade entre sistemas. A adoção de metadados para preservação apoia a administração dos processos relativos ao arquivamento e à manutenção do acesso à informação digital a longo prazo, com garantias de autenticidade, de integridade e de confiabilidade. Neste cenário, foi objetivo do trabalho identificar e descrever padrões ou esquemas de metadados que poderiam ser considerados por instituições que estão desenvolvendo seus repositórios institucionais, para que possam contemplar a preservação digital. Para tanto, foi desenvolvida pesquisa bibliográfica e sistematização e análise de conteúdo. Foram selecionados e analisados os seguintes esquemas e padrões de metadados: DUBLIN CORE, MODS, EAD, ANSI/NISO Z39.87, METS e o PREMIS. A análise dos resultados indica que os metadados descritivos e técnicos do DC, MODS, EAD e ANSI/NISO Z39.87 ou MIX, possuem um papel mais direcionado a amparar o METS e o PREMIS, tanto na descoberta e no registro dos aspectos técnicos do objeto digital, como na comprovação de sua procedência, autenticidade, contexto e integridade. Englobando metadados descritivos, estruturais, administrativos e de preservação do PREMIS, o METS pode gerir os objetos digitais atuando como um Pacote de Submissão de Informação (PSI), Pacote de Arquivamento de Informação (PAI) ou Pacote de Disseminação de Informação (PDI) num Sistema Aberto de Arquivamento de Informação (SAAI).

Beyond TIFF and JPEG2000: PDF/A as an OAIS submission information package container

Purpose – The purpose of this paper is to introduce PDF/A to replace TIFF as the preferred file format for digitization of textual documents. In addition, PDF/A can be used as an open archival information system (OAIS) submission information package (SIP) container to reduce digitization and digital preservation costs. Design/methodology/approach – The author first reviewed the current digitization guidelines, the OAIS model and provides on an overview of the development PDF and PDF/A as international standards. Then literature review of the uses of PDF/A is presented. The author analyzed pitfalls of TIFFs as the preferred format for digitization, and showed how to use PDF/A to code digitization SIP. Findings – TIFF file format has been the preferred master file format by Federal Agency Digitization Guidelines Initiative digitization guidelines for the past 20 years. However, there are drawbacks of TIFF format. Literature reviews show that PDF/A has been the preferred standard for coding born-digital documents in court, government and business sectors. PDF/A-2 and PDF/A-3 are relatively new standards released after 2010. However, few understood the standards and have utilized the full potentials in digitization. The author shows that PDF/A can be used as an OAIS SIP container. Practical implications – In order to delivery OAIS SIPs, current practices require a combination of files, directories and various types of metadata. The author shows that PDF/A (PDF/A-2 and/or PDF/A-3) can be a better file format for textual document digitization with coding various types of metadata in extensible metadata platform and arbitrary file/data can be coded in PDF/A-3. These features in PDF/A provide much better ways to deliver SIPs in a cost-efficient manner. Originality/value – PDF/A has been recognized as the preferred standard for born-digital documents, but it has not been used as the preferred file format for digitized materials. The author recommends that: PDF/A with lossless JPX compressions as the preferred file format; and PDF/A with lossless JPX compressions along with metadata/data as the preferred OAIS SIP container. As a result, the uses reduce costs in digitization and digital preservation and also increase productivity. The author recommends to update the national and international digitization practices using PDF/A.

Developing an ontology for cloud-based archive systems

This paper presents a domain ontology for cloud archives, based in part on the PREMIS Editorial Committee ontology for the PREMIS Data Dictionary. Our ontology's design is based on a layered model of cloud computing where lower layers provide shared services to higher layers. It focuses on the submission of generic Submission Information Packages with PREMIS preservation metadata. An ontology is designed to define a common vocabulary for cloud archives, and define the roles and responsibilities for data creation and transfer, including the registration of cloud-based content creation systems. We define the classes, object properties, data properties and annotations necessary to describe the agents, objects, events and rights that comprise a cloud archive. We evaluated the ontology with a prototype system, using real-world examples of cloud systems, digital objects and metadata. We found that the ontology was able to describe the chosen components successfully, and that it improved metadata interoperability between content creating applications and the services providing preservation metadata.

Digital preservation ingest can be a “CINCH”

PurposeThe purpose of this paper is to introduce the “Capture, Ingest, and Checksum” (CINCH) tool, developed by the State Library of North Carolina with funding from a 2011 IMLS Sparks! Ignition grant.Design/methodology/approachThis article describes the needs assessment, development, and use cases for the CINCH tool in the digital preservation community. A special emphasis is put on the possible wider adoption of the tool for small and mid‐sized institutions and the call for community building around it.FindingsCINCH does not relate directly to storage or access, but it does address the need for appropriately formatted SIPs to ensure the fixity, authenticity, and long‐term manageability of our digital assets. As a broadly accessible tool with generalized purpose, its greatest impact will be on small and mid‐sized institutions and organizations that do not now have access to an application of this nature.Research limitationsCINCH 2.0 planning is already underway. Improving the quality and range of metadata, exploring DRM restrictions, and expanding the types of harvestable files are at the top of the development list.Practical implicationsCINCH will create a Submission Information Package (SIP) to certify that the files ingested are ready for whatever preservation action the institution is capable of performing on them.Originality/valueCINCH was developed by the State Library of North Carolina and is the only tool of its kind available to the digital preservation community.

Changing theory into practice: playing the Metadata Game

The international standard on metadata for records ISO 23081 has been incorporated into the Dutch legislation on archives. This paper gives an introduction to initiatives pursued by central government to develop metadata schemas and implement application profiles. The National Archives of the Netherlands has developed an e-Depot (Trusted Digital Repository). To transfer electronic records to the e-Depot, the records and the metadata linked to them are wrapped as a Submission Information Package (SIP) using an XML-schema called MeDuSa. This XML-schema defines the structure of the SIP as well as a set of metadata elements, and has been developed simultaneously and in compliance with the metadata schema for the Dutch government. To make colleagues, archivists, records managers, information architects and others familiar with the most important concepts concerning metadata, a ‘Metadata Game’ has also been developed.

Archiving in the networked world: preserving plagiarized works

PurposePlagiarism has become a salient issue for universities and thus for university libraries in recent years. This paper aims to discuss three interrelated aspects of preserving plagiarized works: collection development issues, copyright problems, and technological requirements. Too often these three are handled separately even though in fact each has an influence on the other.Design/methodology/approachThe paper looks first at the ingest process (called the Submission Information Package or SIP), then at storage management in the archive (the AIP or Archival Information Package), and finally at the retrieval process (the DIP or Distribution Information Package).FindingsThe chief argument of this paper is that works of plagiarism and the evidence exposing them are complex objects, technically, legally and culturally. Merely treating them like any other work needing preservation runs the risk of encountering problems on one of those three fronts.Practical implicationsThis is a problem, since currently many public preservation strategies focus on ingesting large amounts of self‐contained content that resembles print on paper, rather than on online works that need special handling. Archival systems also often deliberately ignore the cultural issues that affect future usability.Originality/valueThe paper discusses special handling and special considerations for archiving works of plagiarism.

Meeting the Preservation Demand Responsibly = Lowering the Ingest Bar?

This paper describes (1) a lessening of restrictions on the ingest acceptance criteria, and (2) a growing reliance on repository-supplied Submission Information Package (SIP) creation tools planned for the Harvard University Library's digital preservation repository (the DRS). Reasons for these changes include producer demand for the repository to accept new formats and genres, growing quantities of digital content producers want to preserve but can not adequately process, and the increase in the amount of content that is acquired by instead of created by DRS producers. In response to these needs the DRS will greatly change its SIP requirements and ingest process. Most notably files in any format will be accepted by the DRS. This paper describes steps taken to mitigate the effect of these changes so that the DRS can still meet its preservation commitments while broadening the range of acceptable content. One of these mitigating steps is the development of a new tool, FITS, which will be used to automate format identification and metadata extraction during SIP creation.

IJDL special issue on complex digital objects: Guest editors' introduction

Although the Web of today looks very different from the Web of 10 years ago, digital library (DL) file formats still look much the same. Using terminology from the Open Archival Information System (OAIS), the submission information packages (SIPs) and archival information packages (AIPs) were frequently the same application specific formats of the dissemination information packages (DIPs). Driven by the proliferation of enabling technologies such as XML, Resource Description Format (RDF) and Dublin Core, there has been increasing interest in the DL community to use complex digital objects: objects that aggregate data, metadata and sometimes services into a single, logical digital entity. For example, instead of just a bare PDF file, a complex object could aggregate the PDF, the descriptive metadata (perhaps in multiple formats), provenance and rights statements and links to format conversion services. While the end-user continues to receive the PDF, DLs would ingest and exchange the complex digital object, building extended services based on the additional encapsulated and aggregated data. In much the same way that metadata formats arise from different communities, so have complex digital objects. In the realm of scientific and high-performance computing, data formats such as NetCDF [1], HDF [2], SmartFiles [3] and ELFS [4] have been in widespread use. In the commercial content provider industry, complex objects can be traced to the persistent stores of object-oriented databases