Consistent Metadata Research Articles

Incremental Load in a Data Warehousing Environment

Incremental load is an important factor for successful data warehousing. Lack of standardized incremental refresh methodologies can lead to poor analytical results, which can be unacceptable to an organization’s analytical community. Successful data warehouse implementation depends on consistent metadata as well as incremental data load techniques. If consistent load timestamps are maintained and efficient transformation algorithms are used, it is possible to refresh databases with complete accuracy and with little or no manual checking. This paper proposes an Extract-Transform-Load (ETL) metadata model that archives load observation timestamps and other useful load parameters. The author also recommends algorithms and techniques for incremental refreshes that enable table loading while ensuring data consistency, integrity, and improving load performance. In addition to significantly improving quality in incremental load techniques, these methods will save a substantial amount of data warehouse systems resources.

Interoperability and Metadata Quality in Digital Video Repositories: A Study of Dublin Core

The purpose of this paper is to examine the interoperability of the Dublin Core metadata element set across a selection of digital video repositories. To explore how the metadata was applied and analyze its overall quality, 150 records were selected, 25 from each of 6 collections, and evaluated for use of the original 15 Dublin Core elements. Each record was examined for metadata completeness, accuracy, consistency, and usage of controlled vocabularies. Potential barriers to interoperability are identified, most notably the confusion caused by semantic ambiguity among DC elements.

MXF “Mit Out Essence”

The Material exchange Format (MXF) is now deployed extensively in acquisition, post-production, delivery, and archive, and recent systems make much greater use of the metadata capabilities of the MXF file format than in the early days. One innate capability of MXF allows the file that contains the metadata to make reference to essence that is stored externally. This feature can be used to optimize both small- and large-scale systems, so that one copy of the MXF metadata can be kept with the essence for library storage and disaster recovery, and a second copy can be kept within asset management databases, enabling rapid access and manipulation. The second copy is identical to the first, except that it is very lightweight, because it contains only the MXF headers, without the essence. This paper describes a systems architecture that benefits from this approach, together with practical examples drawn from current and new installations. The paper also shows how to provide for metadata growth and addresses questions of metadata consistency and reliability using the MXF data model.

Improving the quality of marine geophysical track line data: Along‐track analysis

We have examined 4918 track line geophysics cruises archived at the U.S. National Geophysical Data Center (NGDC) using comprehensive error checking methods. Each cruise was checked for observation outliers, excessive gradients, metadata consistency, and general agreement with satellite altimetry‐derived gravity and predicted bathymetry grids. Thresholds for error checking were determined empirically through inspection of histograms for all geophysical values, gradients, and differences with gridded data sampled along ship tracks. Robust regression was used to detect systematic scale and offset errors found by comparing ship bathymetry and free‐air anomalies to the corresponding values from global grids. We found many recurring error types in the NGDC archive, including poor navigation, inappropriately scaled or offset data, excessive gradients, and extended offsets in depth and gravity when compared to global grids. While ∼5–10% of bathymetry and free‐air gravity records fail our conservative tests, residual magnetic errors may exceed twice this proportion. These errors hinder the effective use of the data and may lead to mistakes in interpretation. To enable the removal of gross errors without over‐writing original cruise data, we developed an errata system that concisely reports all errors encountered in a cruise. With such errata files, scientists may share cruise corrections, thereby preventing redundant processing. We have implemented these quality control methods in the modified MGD77 supplement to the Generic Mapping Tools software suite.

Metadata hierarchy in integrated geoscientific database for regional mineral prospecting

One of the core developments in geomathematics in now days is the use of digital data processing in mineral prospecting and assessment. The information discovery is based on multidisciplinary geoscientific data and an integrated management approach is crucial. The lack of a standard description hinders interoperations in database search and discovery. Metadata hierarchy aims to provide a standard view of the geoscientific data, and facilitate data description and discovery. In the research of integrated geoscientific database, the metadata hierarchy used a standardized description for each collection in the content structure and realized in semantic structure. It recorded both dataset identification and inner structures and relationships of objects, thus differed from many other applications. There were four tiers in the content structure and three levels in the semantic structure. With its help, database users could determine how applicable a dataset is to a project, and improve their queries to the database. Effectiveness of data accessing is significantly enhanced through the rich, consistent metadata.

Storytelling in an automated environment

PurposeThe purpose of this study is to report the results of a project incorporating metadata analysis for the creation of curated guides to a digital library of historic photographic images.Design/methodology/approachThe general format and scope of the curated guides was determined by members of the project team. Automated processing was developed to analyze subject terms in metadata of digital images submitted to the digital library collection. The results of the metadata processing were used to narrow the themes of the curated guides, and to help create canned searches that would expose representative collection content. The curated guides were created and made available to users of the digital image collection.FindingsProcessing and analyzing metadata can be a useful tool in the creation of digital library collection guides. However, the usefulness of this method is highly dependent upon consistency and accuracy of the source metadata records.Practical implicationsCreators of digital library collections are shown a method of using existing resources to create collection guides. The value of metadata demonstrated by this application may affect considerations of the cost of metadata creation.Originality/valueThis activity discussed in this paper addresses an established need among digital library practitioners – the need to provide users with guided access to the content of their collections. The method discussed for creating curated guides is repeatable, and could be improved in further applications.

Bridging analog and digital assets with automated migration systems

Automated audiovisual migration systems are the only way to migrate large volumes of magnetic media to digital files. Historically, digital asset management (DAM) systems had no way to ingest analog assets en masse. By combining these two technologies, archives and commercial organizations can acquire digital versions of valuable content in a uniform, replicable manner. These legacy media assets can be monetized and repurposed, providing a lasting return on investment. Automated, consistent metadata collection further eases the transition from analog to digital, and enables quick discovery of assets within a DAM system. The evolution of audiovisual migration and restoration to automated systems positively impacts the DAM industry by leveling the field of assets, obviating the distinction between analog and digital, affording new business opportunities for DAM vendors.

Metadata Consistency in DCFS2

Metadata Consistency in DCFS2

Replica consistency in a Data Grid

A Data Grid is a wide area computing infrastructure that employs Grid technologies to provide storage capacity and processing power to applications that handle very large quantities of data. Data Grids rely on data replication to achieve better performance and reliability by storing copies of data sets on different Grid nodes. When a data set can be modified by applications, the problem of maintaining consistency among existing copies arises. The consistency problem also concerns metadata, i.e., additional information about application data sets such as indices, directories, or catalogues. This kind of metadata is used both by the applications and by the Grid middleware to manage the data. For instance, the Replica Management Service (the Grid middleware component that controls data replication) uses catalogues to find the replicas of each data set. Such catalogues can also be replicated and their consistency is crucial to the correct operation of the Grid. Therefore, metadata consistency generally poses stricter requirements than data consistency. In this paper we report on the development of a Replica Consistency Service based on the middleware mainly developed by the European Data Grid Project. The paper summarises the main issues in the replica consistency problem, and lays out a high-level architectural design for a Replica Consistency Service. Finally, results from simulations of different consistency models are presented.

Atomic Writes for data integrity and consistency in shared storage devices for clusters

With the recent trend to use storage area networks in distributed and cluster systems there is a need to improve the integrity and consistency guarantees of stored data in the presence of node or network failures. Currently, the main method for preserving data integrity and consistency is by logging techniques, e.g. journaling. This paper presents a new general method for preserving data consistency by Atomic multi-block Writes. The Atomic Writes method guarantees that either all the blocks in a write operation are written or no blocks are written at all. Its main advantage is that it does not require a recovery phase after a failure. The Atomic Writes method should be implemented in both the operating system and the storage system levels. It is easy to use and to implement. We present the method, its implementation and an example of its use for handling meta-data consistency in an existing file system that supports shared storage.

Metadata: A Fundamental Component of the Semantic Web

Metadata: A Fundamental Component of the Semantic Web

Ivy

Ivy is a multi-user read/write peer-to-peer file system. Ivy has no centralized or dedicated components, and it provides useful integrity properties without requiring users to fully trust either the underlying peer-to-peer storage system or the other users of the file system.An Ivy file system consists solely of a set of logs, one log per participant. Ivy stores its logs in the DHash distributed hash table. Each participant finds data by consuiting all logs, but performs modifications by appending only to its own log. This arrangement allows Ivy to maintain meta-data consistency without locking. Ivy users can choose which other logs to trust, an appropriate arrangement in a semi-open peer-to-peer system.Ivy presents applications with a conventional file system interface. When the underlying network is fully connected, Ivy provides NFS-like semantics, such as close-to-open consistency. Ivy detects conflicting modifications made during a partition, and provides relevant version information to application-specific conflict resolvers. Performance measurements on a wide-area network show that Ivy is two to three times slower than NFS.

DATA AND METADATA REPORTING STANDARDS FOR THE U.S. ENVIRONMENTAL PROTECTION AGENCY'S PM SUPERSITES RESEARCH PROGRAM

The EPA Supersites Research Program needs consistency of metadata and data structures to facilitate information sharing among investigators, analysts, and ultimately secondary data users. Under the auspices of NARSTO a successful mechanism was created to develop and implement reporting standards. The development effort included working closely with Supersites data coordinators, investigators, and technical experts, and also leveraging from existing data standards and practices. Overall, the standards are getting good acceptance from the atmospheric research community.

Bibliographic data mining: Automatically building component part records for E-Journal articles on the Internet

This paper is based on a presentation at the Medical Library Association Annual Meeting in May 2000. With the rapid expansion of fulltext on the World Wide Web, Internet navigation has become a serious concern to libraries. To increase access to fulltext digital resources and minimize staff effort, Stanford's Lane Medical Library began a Bibliographic Data Mining project in February 1999. With limited human editing, the PERL scripts captured relevant URLs from selected e-journal Web sites and automatically created basic MARC component part bibliographic records for the articles in those journals. This project demonstrated that bibliographic data mining is a feasible methodology for improving the effectiveness of access to Internet resources within the context of the online catalog. It provided valuable experience in managing digital resources and highlighted the need for more consistent metadata.

The OhioLINK approach: records and holdings for print and electronic serials in the OhioLINK central catalog

Since 1992, when OhioLINK’s central catalog was first implemented, the consortium’s union listing and serials holdings activities have shifted from an emphasis on print holdings in individual libraries for traditional inter‐library lending to dealing with holdings for electronic serials purchased with group licenses and with related issues for linking catalog records, abstracting and indexing data. Current work centers on using one record or separate records for multiple versions of a serial (especially print and electronic), the most comprehensible display for resources owned by the consortium as a group instead of by separate libraries, information about and display of holdings information for the electronic version of serial, and consistent metadata.

The OhioLINK approach ‐ Records and holdings for print and electronic serials in the OhioLINK central catalog

Since 1992, when OhioLINK’s central catalog was first implemented, the consortium’s union listing and serials holdings activities have shifted from an emphasis on print holdings in individual libraries for traditional inter‐library lending to dealing with holdings for electronic serials purchased with group licenses and with related issues for linking catalog records, abstracting and indexing data. Current work centers on using one record or separate records for multiple versions of a serial (especially print and electronic), the most comprehensible display for resources owned by the consortium as a group instead of by separate libraries, information about and display of holdings information for the electronic version of serials, and consistent metadata.