File Catalog Research Articles

Descriptive catalog of Homo naledi dental remains from the 2013 to 2015 excavations of the Dinaledi Chamber, site U.W. 101, within the Rising Star cave system, South Africa

More than 150 hominin teeth, dated to ∼330–241 thousand years ago, were recovered during the 2013–2015 excavations of the Dinaledi Chamber of the Rising Star cave system, South Africa. These fossils comprise the first large single-site sample of hominin teeth from the Middle Pleistocene of Africa. Though scattered remains attributable to Homo sapiens, or their possible lineal ancestors, are known from older and younger sites across the continent, the distinctive morphological feature set of the Dinaledi teeth supports the recognition of a novel hominin species, Homo naledi. This material provides evidence of African Homo lineage diversity that lasts until at least the Middle Pleistocene. Here, a catalog, anatomical descriptions, and details of preservation and taphonomic alteration are provided for the Dinaledi teeth. Where possible, provisional associations among teeth are also proposed. To facilitate future research, we also provide access to a catalog of surface files of the Rising Star jaws and teeth.

Lithuanian Riddles of the Pot as Allegories of the Human Life

Further analyzing the rather widespread international phenomenon – the so-called series of suffering in the riddles – this article focuses on the Lithuanian riddles of the pot, namely: ‘When I was young, I used to sit in a golden chair, when I grew old, even the dog does not eat my bones’ and ‘I was made like Adam from clay and annealed in the furnace like the three young people in Babylon. I was sold like Joseph to Egypt by his brothers. I drive a fiery chariot like Elijah. From my insides, the whole family eats, but when I die, they throw my bones over the fence (do not bury them).’ The analysis is based on all the variants of these riddles (both the published and the manuscript ones), their international parallels, and texts belonging to other genres. Despite the common answer to these riddles – pot – and its typical semantic field based on reality, the artistic expressions of these two riddles have little in common: they significantly differ in terms of structure, wording, and semantic. Thus, their recordings are reasonably attributed to different types in the current card file catalogue. The two riddles only share the common motive of bones at the end, which however has different context of the artistic expression and slightly different meaning. Except for some isolated cases and contaminations, the motives like dog does not eat my bones and they throw my bones over the fence (do not bury them) can be discerned only in these two riddle types respectively. The motive dog does not eat my bones is encountered in almost all the available variants of the first riddle and has the meaning of ‘being useless, idle’. However, the motives they throw the bones over the fence or do not bury them, found in the second riddle, also semantically include the aspect of disrespectful treatment of the deceased and neglecting of the proper funeral rites. Perhaps the main objective of this motive is prompting the listener towards the right answer and suggesting that the answer in question is not a human being. The first riddle seems to be of the Baltic origin, as it includes the typically Baltic motive of the golden chair and the common Baltic-Slavic expression dog does not eat (my) bones. Similar motives are encountered in the wedding songs, where they probably suggest the corresponding semantic field, which is narrowed by the pot as an answer to mean blossoming and withering of fertility. This classical riddle is likely to speak of the human life in general, albeit reducing it to the phases of valued youth and useless senility.The significantly more widespread version of the second riddle in Lithuania includes Biblical motives and has most probably found its way from the East, where it had been “Christianized” by a person well versed in the Old Testament and perhaps belonging to the Byzantine culture. Some variants of this riddle do not include Christian motives; such variants are few in Lithuania but abundant among the Eastern Slavs. Although such versions without Christian motives may be of common origins, however, the proportions of the materials suggest higher probability of the Slavic origin of this riddle. Attributing of these riddles to the series of suffering is also somewhat ambivalent: while the first riddle emphasizes the spiritual suffering, the totality of the variants of the second riddle may be rightfully summarized as the description of the pot’s life; descriptions of the suffering being more frequent in the riddles including the Christian motives. Both riddles tacitly compare the clay pot to the man, allegorically describing the shifts occurring in the human life, the inevitability of change and the instability of status. While reminding the perspective of eternity and the direction of the human life towards death, they allegedly fulfil the function of Memento mori and engage in a visual discussion with the Biblical phrase “from dust to dust”.

The Rucio File Catalog in DIRAC implemented for Belle II

DIRAC and Rucio are two standard pieces of software widely used in the HEP domain. DIRAC provides Workload and Data Management functionalities, among other things, while Rucio is a dedicated, advanced Distributed Data Management system. Many communities that already use DIRAC have expressed their interest in using DIRAC for Workload Management in combination with Rucio for Data Management. In this paper, we describe the integration of the Rucio File Catalog into DIRAC that was initially developed for the Belle II collaboration.

The Belle II Raw Data Management System

The Belle II experiment, a major upgrade of the previouse+e−asymmetric collider experiment Belle, is expected to produce tens of petabytes of data per year due to the luminosity increase from the upgraded SuperKEKB accelerator. The distributed computing system of the Belle II experiment plays a key role, storing and distributing data in a reliable way to be easily accessed and analyzed by more than 1000 collaborators. In particular, the Belle II Raw Data Management system has been developed with an aim to upload output files onto grid storage, register them into the file and metadata catalogs, and make two replicas of the full raw data set using the Belle II Distributed Data Management system. It has been implemented as an extension of DIRAC (Distributed Infrastructure with Remote Agent Control) and consists of a database, services, client and monitoring tools, and several agents that treat the data automatically. The first year of data taken with the Belle II full detector has been managed by the Belle II Raw Data Management system successfully. The design, current status, and performance are presented. Prospects for improvements towards the full luminosity data taking are also reviewed.

Distributed Computing for the Project 8 Experiment

The Project 8 collaboration aims to measure the absolute neutrino mass or improve on the current limit by measuring the tritium beta decay electron spectrum. We present the current distributed computing model for the Project 8 experiment. Project 8 is in its second phase of data taking with a near continuous data rate of 1Gbps. The current computing model uses DIRAC (Distributed Infrastructure with Remote Agent Control) for its workflow and data management. A detailed meta-data assignment using the DIRAC File Catalog is used to automate raw data transfers and subsequent stages of data processing. The DIRAC system is deployed on containers managed using a Kubernetes cluster to provide a scalable infrastructure. A modified DIRAC Site Director provides the ability to submit jobs using Singularity on opportunistic High-Performance Computing (HPC) sites.

Phenomenon of Popularity of the Lithuanian Folktale “The Sister as Duck”

The Lithuanian folktale “The Sister as Duck” (AT 452C*), most commonly known under the name of “Sigutė”, is generally regarded as a popular narrative. It is appreciated for the rich mythical imagery: after the female protagonist is burned and the cow licks the ashes, the duck appears; the snow is allegedly sparkling because of the witch’s brains (bones) scattered there, etc. Moreover, the tale abounds in expressive song insertions.This tale has been published both in academic and popular editions. Its images are widespread in popular culture, even becoming stereotypical. However, in the card file catalogue of the Lithuanian folk narratives, only 17 variants of this tale and 5 fragments recorded from oral tradition can be found.By using contextual analysis, the author of the article attempts establishing when, how, and due to what circumstances this tale has turned into a phenomenon of popular culture.Its first recording from oral tradition in the 19th century purported to serve as illustration of the peculiarities of a particular Lithuanian dialect. After the ban of the Lithuanian press in Latin alphabet was lifted in the beginning of the 20th century, this variant of the tale was included into a published collection of folktales for children, subsequently spreading to various schoolbooks, which commonly used it for didactic purposes, such as educating empathy in children.During Soviet times, the tale found its way into several academic editions, afterwards crossing into popular folktales’ collections for children published in thousands of copies. The tale also became popular in audial shape: its sound recordings were published both in Lithuania and abroad.The analysis of the tale’s variants recorded from the oral tradition highlighted certain characteristics indicating that its popularity must have depended on the written culture. In the five variants of the tale recorded in 1936–1959, the female protagonist is not once called Sigutė or turns into the duck. However, the eleven variants recorded in 1962–1980 include most of the structural components or certain peculiar details from the first published variant of the tale, e.g. the song insertions, the girl’s name, the duck motive. In most cases, the girl turns into duck here, and only once into a cuckoo. This testifies to the obvious influence of the published variant.The folktale “Sigutė” earned phenomenal popularity also due to its artistic interpretations. In the beginning of the 20th century, it inspired works by such Lithuanian neo-romanticists as Vydūnas, Salomėja Nėris and others. During Soviet times, this story served as basis for plays and theater performances for children, thus further paving its way into popular culture.The fate of the folktale “Sigutė” may be regarded as a certain case of folktale canon formation: the folk text originating in oral tradition is affected by instruments of the written culture, turning into a narrative of national renown and importance. Possibly such cannon formation was also determined by the decline of the traditional folk culture, as result of which the educated people selecting the folk texts to inspire their creativity particularly favored the ones characterized by a dramatic story line and impressive tragic developments, which therefore could be easily adapted into individual literary compositions.Thus, the literary and musical interpretations of the folktale “Sigutė” have significantly added to its popularity, turning it into a canonic childhood text. Lacking popularity in the oral tradition and recorded as just a few fragmented variants, most likely retold from the published folktale collections, the folktale “Sigutė” became a special phenomenon of the written Lithuanian culture.

WLCG space accounting in the SRM-less world

The WLCG computing infrastructure provides distributed storage capacity hosted at the geographically dispersed computing sites. In order to effectively organize storage and processing of the LHC data, the LHC experiments require a reliable and complete overview of the storage capacity in terms of the occupied and free space, the storage shares allocated to different computing activities, and the possibility to detect “dark” data that occupies space while being unknown to the experiment's file catalogue. The task of the WLCG space accounting activity is to provide such an overview and to assist LHC experiments and WLCG operations to manage storage space and to understand future requirements. Several space accounting solutions which have been developed by the LHC experiments are currently based on Storage Resource Manager (SRM). In the coming years SRM becomes an optional service for sites which do not provide tape storage. Moreover, already now some of the storage implementations do not provide an SRM interface. Therefore, the next generation of the space accounting systems should not be based on SRM. In order to enable possibility for exposing storage topology and space accounting information the Storage Resource Reporting proposal has been agreed between LHC experiments, sites and storage providers. This contribution describes the WLCG storage resource accounting system which is being developed based on Storage Resource Reporting proposal.

JAliEn: the new ALICE high-performance and high-scalability Grid framework

The ALICE experiment will undergo extensive hardware and software upgrades for the LHC Run3. This translates in significant increase of the CPU and storage resources required for data processing, and at the same time the data access rates will grow linearly with the amount of resources. JAliEn (Java ALICE Environment) is the new Grid middleware designed to scale-out horizontally to fulfil the computing needs of the upgrade, and at the same time to modernize all parts of the distributed system software. This paper will present the architecture of the JAliEn framework, the technologies used and performance measurements. This work will also describe the next generation solution that will replace our main database backend, the AliEn File Catalogue. The catalogue is an integral part of the system, containing the metadata of all files written to the distributed Grid storage and also provides powerful search and data manipulation tools. As for JAliEn, the focus has been put onto horizontal scalability, with the aim to handle near exascale data volumes and order of magnitude more workload than the currently used Grid middleware. Lastly, this contribution will present how JAliEn manages the increased complexity of the tasks associated with the new ALICE data processing and analysis framework (ALFA) and multi-core environments.

Aukusti Robert Niemi and His Collection of Folklore from Southern Lithuania Revisited. Textological Approach

The Lithuanian folklore collections by the Finnish folklorist Aukusti Robert Niemi seem symbolically to enframe his two decades of the Lithuanian cultural activity. Professor started from doing fieldwork (visiting northeastern Lithuania in 1910 and southern Lithuania in 1911), which subsequently formed the basis for his scholarly research. In 1931, just before Niemi died, the manuscripts of his folklore collections returned to Lithuania from Finland in order to get the fieldwork materials of 1911 finally edited and published. The two decades in-between served to form and establish in the Lithuanian historiography the pattern of particularly positive appreciation of professor’s Lithuanian activities, the basis for which was the collection produced as result of his fieldwork in 1910, entitled “Lithuanian Songs and Chants in the Northeastern Lithuania” and published in 1912.This article aims at revising the established mode of evaluating the Lithuanian traditional folklore collected by Niemi. The subject of analysis consists of the manuscript collection of the southern Lithuanian folklore, which, although edited for publication, remained nevertheless unpublished. This collection consists of two separate archived folders preserved at the Lithuanian Folklore Archives of the Institute of Lithuanian Literature and Folklore. It has been endowed status of a trustworthy source in the historiography of Lithuanian folklore research and the source editing tradition, without proper consideration of possible complications caused by the shape of this collection, the linguistic problems encountered by a foreign researcher recording oral folksongs, and the geographic area of recording, which significantly exceeded the boundaries of southern Lithuania.The manuscript folklore collection from southern Lithuania is peculiar in terms of one mode of evaluation required by the fieldwork notes and quite another – by the copy of the manuscript, which allows for reconstructing the collection as a whole, since part of the original autograph is lost. In the course of analysis, the texts from the copy and those from the autograph were thoroughly compared. This comparison highlighted certain changes having taken place in the copy, which perhaps resulted from carelessness, inexperience or neglect by the copyists or editors readying the texts for publication, as well as from specific problems inherent in the autograph and causing reasonable doubt in the trustworthiness of the copy. In the process of archiving, there occurred a mistake resulting in a sheaf of miscellaneous folklore from various places in Lithuania being added to the original collection. These texts subsequently found their way into the card file catalogues of Lithuanian riddles, proverbs and folksongs as authentic recordings; although according to the catalogues, they should have been regarded as copies from the folklore collections by the Lithuanian Science Society. So far, the fieldwork notes by Niemi were usually thought to have been produced by his assistants, who actually recorded the texts. The textological analysis of fieldwork notes from southern Lithuania enabled revision of such assumptions. Although Niemi’s assistants did lend him a hand in recording the folksong texts, certain peculiarities of the recordings allow us assuming that the majority of the material was noted down by the professor himself, who did not know the Lithuanian language sufficiently. This would clarify numerous inaccuracies that impede reading and understanding of the autograph: confused vowels and consonants, distorted words and lexical forms (these seem to be acoustic lapses resulting in random joining of separate syllables into words without considering the actual meaning of the text), as well as incoherent and fragmented character of the texts. However, unbiased appreciation of the folkloric legacy by the Finnish researcher Niemi is only possible with due consideration of the textological problems and quality issues inherent in his texts. Besides, sufficient elucidation of the circumstances under which this collection was compiled and the scholarly intentions of its collector could add attractiveness to it, particularly if the subtle textological nuances can be made relevant as peculiar marks left by a foreign speaking collector.

Scalable global grid catalogue for Run3 and beyond

The AliEn (ALICE Environment) file catalogue is a global unique namespace providing mapping between a UNIX-like logical name structure and the corresponding physical files distributed over 80 storage elements worldwide. Powerful search tools and hierarchical metadata information are integral parts of the system and are used by the Grid jobs as well as local users to store and access all files on the Grid storage elements. The catalogue has been in production since 2005 and over the past 11 years has grown to more than 2 billion logical file names. The backend is a set of distributed relational databases, ensuring smooth growth and fast access. Due to the anticipated fast future growth, we are looking for ways to enhance the performance and scalability by simplifying the catalogue schema while keeping the functionality intact. We investigated different backend solutions, such as distributed key value stores, as replacement for the relational database. This contribution covers the architectural changes in the system, together with the technology evaluation, benchmark results and conclusions.

Abstract 2602: The ICGC data portal and its underlying open source software architecture

Abstract The goal of the International Cancer Genome Consortium (ICGC) is to analyze the cancer genomes of at least 500 tumour samples with matched controls from 50 different cancer types and subtypes, building a comprehensive catalogue of somatic abnormalities for the benefit of the research community. The amount of data ICGC members will generate is close to that of 50,000 human genome projects and, to date, has received commitments for 107 projects to study more than 27,000 tumor genomes. The ICGC Data Coordination Center (DCC) is responsible for collecting, curating, aggregating, and disseminating the data generated by the consortium’s member projects. Given the size and the complexity of the ICGC data, these tasks represent significant scientific and technological challenges that require a performant, robust software infrastructure. Key to this infrastructure is the ability to scale as data grows. Using state-of-the-art Big Data, bioinformatics and cloud computing technologies, we developed a suite of web-based applications and microservices that enable member projects to first submit their data and validate their submissions according to the rules defined in the submission specification. Following validation, the data is processed, annotated and loaded into the data portal using a modular Extract-Transform-Load (ETL) pipeline. Submission, ETL and portal systems are built using scalable and distributed technologies such as Hadoop, Spark, MongoDB and ElasticSearch. Spark is used to validate, join, index, and harmonize annotations on submitted variants while ElasticSearch powers our variant query engine, API and portal displays. Here we present the ICGC Data Portal and describe both the current features and capabilities accessible to users along with the architecture of the underlying infrastructure. The portal provides scientists with powerful and unique tools for exploring and visualizing the millions of variants and annotations available. These include sophisticated, faceted search capabilities making data exploration extremely fast and easy, a suite of interactive Javascript components for in-depth analysis and visualization of specific genomic features, embedded genome and pathway browsers, synthetic cohorts comparisons and a streaming data download service. The portal integrates a large variety of annotations such as variant consequences and frequencies, functional impact factors and druggability. The portal also offers cloud-based tools for searching a catalog of raw ICGC data files stored in worldwide repositories and compute clouds. All source code is open to the community under the GPLv3 license. Citation Format: Junjun Zhang, Bob Tiernay, Dusan Andric, Phuong-My Do, Sid Joshi, Vitalii Slobodianyk, Chang Wang, Shane Wilson, Andy Yang, Vincent Ferretti. The ICGC data portal and its underlying open source software architecture [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2017; 2017 Apr 1-5; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2017;77(13 Suppl):Abstract nr 2602. doi:10.1158/1538-7445.AM2017-2602

The DIRAC Data Management System and the Gaudi dataset federation

The DIRAC Interware provides a development framework and a complete set of components for building distributed computing systems. The DIRAC Data Management System (DMS) offers all the necessary tools to ensure data handling operations for small and large user communities. It supports transparent access to storage resources based on multiple technologies, and is easily expandable. The information on data files and replicas is kept in a File Catalog of which DIRAC offers a powerful and versatile implementation (DFC). Data movement can be performed using third party services including FTS3. Bulk data operations are resilient with respect to failures due to the use of the Request Management System (RMS) that keeps track of ongoing tasks.In this contribution we will present an overview of the DIRAC DMS capabilities and its connection with other DIRAC subsystems such as the Transformation System. This paper also focuses on the DIRAC File Catalog, for which a lot of new developments have been carried out, so that LHCb could migrate its replica catalog from the LCG File Catalog to the DFC. Finally, we will present how LHCb achieves a dataset federation without the need of an extra infrastructure.

Extending DIRAC File Management with Erasure-Coding for efficient storage.

The state of the art in Grid style data management is to achieve increased resilience of data via multiple complete replicas of data files across multiple storage endpoints. While this is effective, it is not the most space-efficient approach to resilience, especially when the reliability of individual storage endpoints is sufficiently high that only a few will be inactive at any point in time. We report on work performed as part of GridPP[1], extending the Dirac File Catalogue and file management interface to allow the placement of erasure-coded files: each file distributed as N identically-sized chunks of data striped across a vector of storage endpoints, encoded such that any M chunks can be lost and the original file can be reconstructed. The tools developed are transparent to the user, and, as well as allowing up and downloading of data to Grid storage, also provide the possibility of parallelising access across all of the distributed chunks at once, improving data transfer and IO performance. We expect this approach to be of most interest to smaller VOs, who have tighter bounds on the storage available to them, but larger (WLCG) VOs may be interested as their total data increases during Run 2. We provide an analysis of the costs and benefits of the approach, along with future development and implementation plans in this area. In general, overheads for multiple file transfers provide the largest issue for competitiveness of this approach at present.

Distributed Computing for the Pierre Auger Observatory

Pierre Auger Observatory operates the largest system of detectors for ultra-high energy cosmic ray measurements. Comparison of theoretical models of interactions with recorded data requires thousands of computing cores for Monte Carlo simulations. Since 2007 distributed resources connected via EGI grid are successfully used. The first and the second versions of production system based on bash scripts and MySQL database were able to submit jobs to all reliable sites supporting Virtual Organization auger. For many years VO auger belongs to top ten of EGI users based on the total used computing time. Migration of the production system to DIRAC interware started in 2014. Pilot jobs improve efficiency of computing jobs and eliminate problems with small and less reliable sites used for the bulk production. The new system has also possibility to use available resources in clouds. Dirac File Catalog replaced LFC for new files, which are organized in datasets defined via metadata. CVMFS is used for software distribution since 2014. In the presentation we give a comparison of the old and the new production system and report the experience on migrating to the new system.

FAMoS - an information service on the usage of data files in AliEn

The File Access Monitoring Service (FAMoS) leverages the information stored in the central AliEn file catalogue, which describes every file in a Unix-like directory structure, as well as metadata on file location and its replicas. In addition, it uses the access information provided by a set of API servers, used by all Grid clients to access the catalogue. The main functions of FAMoS are to sort the file accesses by logical groups, access time, user and storage element. The collected data identifies rarely used groups of files, as well as those with high popularity over different time periods. This information can be further used to optimize file distribution and replication factors, thus increasing the data processing efficiency. The paper describes the FAMoS structure and user interface and presents the results obtained in one year of service operation.

A Flexible Monitoring Infrastructure for the Simulation Requests

Running and monitoring simulations usually involves several different aspects of the entire workflow: the configuration of the job, the site issues, the software deployment at the site, the file catalogue, the transfers of the simulated data. In addition, the final product of the simulation is often the result of several sequential steps. This project tries a different approach to monitoring the simulation requests. All the necessary data are collected from the central services which lead the submission of the requests and the data management, and stored by a backend into a NoSQL-based data cache; those data can be queried through a Web Service interface, which returns JSON responses, and allows users, sites, physics groups to easily create their own web frontend, aggregating only the needed information. As an example, it will be shown how it is possible to monitor the CMS services (ReqMgr, DAS/DBS, PhEDEx) using a central backend and multiple customized cross-language frontends.

MICE data handling on the Grid

The international Muon Ionisation Cooling Experiment (MICE) is designed to demonstrate the principle of muon ionisation cooling for the first time, for application to a future Neutrino factory or Muon Collider. The experiment is currently under construction at the ISIS synchrotron at the Rutherford Appleton Laboratory (RAL), UK. In this paper we present a system – the Raw Data Mover, which allows us to store and distribute MICE raw data – and a framework for offline reconstruction and data management. The aim of the Raw Data Mover is to upload raw data files onto a safe tape storage as soon as the data have been written out by the DAQ system and marked as ready to be uploaded. Internal integrity of the files is verified and they are uploaded to the RAL Tier-1 Castor Storage Element (SE) and placed on two tapes for redundancy. We also make another copy at a separate disk-based SE at this stage to make it easier for users to access data quickly. Both copies are check-summed and the replicas are registered with an instance of the LCG File Catalog (LFC). On success a record with basic file properties is added to the MICE Metadata DB. The reconstruction process is triggered by new raw data records filled in by the mover system described above. Off-line reconstruction jobs for new raw files are submitted to RAL Tier-1 and the output is stored on tape. Batch reprocessing is done at multiple MICE enabled Grid sites and output files are shipped to central tape or disk storage at RAL using a custom File Transfer Controller.

DDM Workload Emulation

Rucio is the successor of the current Don Quijote 2 (DQ2) system for the distributed data management (DDM) system of the ATLAS experiment. The reasons for replacing DQ2 are manifold, but besides high maintenance costs and architectural limitations, scalability concerns are on top of the list. Current expectations are that the amount of data will be three to four times as it is today by the end of 2014. Further is the availability of more powerful computing resources pushing additional pressure on the DDM system as it increases the demands on data provisioning. Although DQ2 is capable of handling the current workload, it is already at its limits. To ensure that Rucio will be up to the expected workload, a way to emulate it is needed. To do so, first the current workload, observed in DQ2, must be understood in order to scale it up to future expectations. The paper discusses how selected core concepts are applied to the workload of the experiment and how knowledge about the current workload is derived from various sources (e.g. analysing the central file catalogue logs). Finally a description of the implemented emulation framework, used for stress-testing Rucio, is given.

DIRAC File Replica and Metadata Catalog

File replica and metadata catalogs are essential parts of any distributed data management system, which are largely determining its functionality and performance. A new File Catalog (DFC) was developed in the framework of the DIRAC Project that combines both replica and metadata catalog functionality. The DFC design is based on the practical experience with the data management system of the LHCb Collaboration. It is optimized for the most common patterns of the catalog usage in order to achieve maximum performance from the user perspective. The DFC supports bulk operations for replica queries and allows quick analysis of the storage usage globally and for each Storage Element separately. It supports flexible ACL rules with plug-ins for various policies that can be adopted by a particular community. The DFC catalog allows to store various types of metadata associated with files and directories and to perform efficient queries for the data based on complex metadata combinations. Definition of file ancestor-descendent relation chains is also possible. The DFC catalog is implemented in the general DIRAC distributed computing framework following the standard grid security architecture. In this paper we describe the design of the DFC and its implementation details. The performance measurements are compared with other grid file catalog implementations. The experience of the DFC Catalog usage in the CLIC detector project are discussed.

ATLAS File and Dataset Metadata Collection and Use

The ATLAS Metadata Interface (“AMI”) was designed as a generic cataloguing system, and as such it has found many uses in the experiment including software release management, tracking of reconstructed event sizes and control of dataset nomenclature. The primary use of AMI is to provide a catalogue of datasets (file collections) which is searchable using physics criteria. In this paper we discuss the various mechanisms used for filling the AMI dataset and file catalogues. By correlating information from different sources we can derive aggregate information which is important for physics analysis; for example the total number of events contained in dataset, and possible reasons for missing events such as a lost file. Finally we will describe some specialized interfaces which were developed for the Data Preparation coordinators. These interfaces manipulate information from both the dataset domain held in AMI, and the run-indexed information held in the ATLAS COMA application (Conditions and Configuration MetadatA).