Existing Data Formats Research Articles

BAllC and BAllCools: efficient formatting and operating for single-cell DNA methylation data.

With single-cell DNA methylation studies yielding vast datasets, existing data formats struggle with the unique challenges of storage and efficient operations, highlighting a need for improved solutions. BAllC (Binary All Cytosines) emerges as a tailored format for methylation data, addressing these challenges. BAllCools, its complementary software toolkit, enhances parsing, indexing, and querying capabilities, promising superior operational speeds and reduced storage needs. https://github.com/jksr/ballcools.

Migration Misadventures: The Challenges of Creating an Inter-Institutional Migration Plan

Archival institution migration projects are planned data transfers from existing data formats to more accessible formats that assist in preserving transferred data, improving information retrieval, and preventing obsolescence. Various archival institutions are increasingly planning migration projects because current technology makes the process easier and more affordable than in the past. In this proceeding, I will walk through the planning stages and associated challenges of NASA’s inter-institutional migration plan, Archival Description Center Migration Plans Project, which is part of a larger reorganization of the archives’ functional area to an Enterprise program. For this project, we utilized the open-source, standards-based, web-based applications Access to Memory and Archivematica. At the current project stage, we have collaborated with archivists from seven of the eleven existing NASA archive centers. One of the goals of this Enterprise reorganization is to facilitate cross-collection search and discovery and establish cross-institution metadata standards.

Data format standards in analytical chemistry

Abstract Research data is an essential part of research and almost every publication in chemistry. The data itself can be valuable for reuse if sustainably deposited, annotated and archived. Thus, it is important to publish data following the FAIR principles, to make it findable, accessible, interoperable and reusable not only for humans but also in machine-readable form. This also improves transparency and reproducibility of research findings and fosters analytical work with scientific data to generate new insights, being only accessible with manifold and diverse datasets. Research data requires complete and informative metadata and use of open data formats to obtain interoperable data. Generic data formats like AnIML and JCAMP-DX have been used for many applications. Special formats for some analytical methods are already accepted, like mzML for mass spectrometry or nmrML and NMReDATA for NMR spectroscopy data. Other methods still lack common standards for data. Only a joint effort of chemists, instrument and software vendors, publishers and infrastructure maintainers can make sure that the analytical data will be of value in the future. In this review, we describe existing data formats in analytical chemistry and introduce guidelines for the development and use of standardized and open data formats.

The Notion of "Research Data": Types and Kinds of Research Data in the Context of Data Management Practice

With the rapid development of the concept of Open Science, the quantitative growth of data obtained during the research, scientific attention to the practice of research data management (research data management) increases, which actualizes the definition of “research data” and identifying types of research data within the practice of their management, justification and coverage of the specifics of such data. The methodological tools of the study are based on the terminological method, the use of which was due to the need to identify relevant interpretations of the concept of “research data”, as well as analysis of repositories for data from various fields of science, indexed by re3data.org., in the general areas presented in the register, namely: descriptions of repositories, including information on the types of data deposited by scientists and data curators. The analysis made it possible to define research data as materials obtained and collected to substantiate the scientific results of research in any field and in any form: numerical, textual, computer code, etc., as well as to identify types of data specific to different branches of science, which, in turn, allowed us to conclude the existing data formats, the most common among both natural and human sciences: text, numerical and graphic formats. As a result of the analysis, it was found that research data can be considered textual, numerical, software, archival, graphic and other objects (files) that serve as the basis of the study and the factual basis for scientific conclusions in a particular field of science. It was found that the type of data directly depends on the nature of the study and the characteristics of the discipline or field of research.

Balancing efficient analysis and storage of quantitative genomics data with the D4 format and d4tools.

Modern DNA sequencing is used as a readout for diverse assays, with the count of aligned sequences (read depth) representing the quantitative signal for each underlying cellular phenomena. Existing data formats for quantitative genomics assays are, however, limited in either the analysis speeds they enable, the disk space they require or both. We have developed the dense depth data dump (D4) format and tool suite, with the goal of balancing improved analysis speeds with file size. The D4 format is adaptive in that it profiles a random sample of aligned sequence depth from the input sequence file to determine an optimal encoding that enables fast data access. We demonstrate that the D4 format offers substantial speed improvements over existing formats for random access, aggregation and summarization, while also achieving better or comparable file sizes. This performance enables scalable downstream analyses that would be otherwise difficult.

Exploring Approaches for Large Data in Seismology: User and Data Repository Perspectives

Abstract New data acquisition techniques are generating data at much finer temporal and spatial resolution, compared to traditional seismic experiments. This is a challenge for data centers and users. As the amount of data potentially flowing into data centers increases by one or two orders of magnitude, data management challenges are found throughout all stages of the data flow. The Incorporated Research Institutions for Seismology—Réseau sismologique et géodésique français and GEOForschungsNetz data centers—carried out a survey and conducted interviews of users working with very large datasets to understand their needs and expectations. One of the conclusions is that existing data formats and services are not well suited for users of large datasets. Data centers are exploring storage solutions, data formats, and data delivery options to meet large dataset user needs. New approaches will need to be discussed within the community, to establish large dataset standards and best practices, perhaps through participation of stakeholders and users in discussion groups and forums.

CoPLINK: A complementary tool to PLINK.

In genome-wide association studies (GWAS), a wide variety of analysis tools have been designed, leading to various formats of GWAS data. How to convert a dataset in non-PLINK format into PLINK format to use its powerful analysis performance, or to convert a dataset in PLINK format into the format of other analysis tools, is a problem that needs to be faced and solved. To address this issue, we developed a tool called coPLINK, a complementary tool to PLINK, to cooperate with PLINK to implement the conversions of GWAS data formats and to provide some additional functions, such as data files comparison. The tool can implement mutual conversions not only between an existing data format and PLINK PED/BED, but also between a user-defined data format and PLINK PED. The usage and performance of the tool are similar to PLINK. The characteristics of the conversions of existing data formats and user-defined formats make it be a good assistant to PLINK or other tools and, have good potential for GWAS studies or other works.

Data Exchange for Process Engineering – Challenges and Opportunities

AbstractThe ideas and requirements for digitization in the process industry are manifold – the interoperability of the tools in use by engineers is an important prerequisite. The strong heterogeneity of existing data formats and standards as well as the variety of proprietary and company‐specific solutions confronts the process industry with huge challenges. This paper deals with the data exchange between process simulation, 2D CAD, and 3D CAD tools, provides an overview of existing standards and formats as well as initial experiences gained in a case study.

Processing of big heterogeneous genomic datasets for tertiary analysis of Next Generation Sequencing data.

We previously proposed a paradigm shift in genomic data management, based on the Genomic Data Model (GDM) for mediating existing data formats and on the GenoMetric Query Language (GMQL) for supporting, at a high level of abstraction, data extraction and the most common data-driven computations required by tertiary data analysis of Next Generation Sequencing datasets. Here, we present a new GMQL-based system with enhanced accessibility, portability, scalability and performance. The new system has a well-designed modular architecture featuring: (i) an intermediate representation supporting many different implementations (including Spark, Flink and SciDB); (ii) a high-level technology-independent repository abstraction, supporting different repository technologies (e.g., local file system, Hadoop File System, database or others); (iii) several system interfaces, including a user-friendly Web-based interface, a Web Service interface, and a programmatic interface for Python language. Biological use case examples, using public ENCODE, Roadmap Epigenomics and TCGA datasets, demonstrate the relevance of our work. The GMQL system is freely available for non-commercial use as open source project at: http://www.bioinformatics.deib.polimi.it/GMQLsystem/. Supplementary data are available at Bioinformatics online.

Archiving and exchange of digital ECGs: A review of existing data formats

Digital ECG is today a common practice but a universal format for its storage and exchange has never been widely implemented. The reason is linked on one side to the need of the manufacturing industry to (rightly) protect intellectual propriety and technology, but on the other to an inadequate effort of the research community to sufficiently enforce the use of digital ECG data. To some degree, and at least from a practical point of view, the problem is also linked to other factors, such as the need in some instances to protect patient-sensitive information, and whether digital exchanged data should also include annotations and measurements from an algorithm or by human intervention.As a result, after more than 30 years it is still common that the full ECG acquired information is not preserved, but only partially stored or saved as a PDF report. Paradoxically, the modern era of hospital information technology and the advent and large diffusion of electronic health record systems did not bring expected improvements: the process of digital ECG retrieval and management remains extremely complicated and cumbersome. The ultimate risk is that the ECG may end up being considered “just” an image rather than a voltage-versus-time signal as it has always been.A critical review of the most commonly used formats for digital ECG will be given, focusing in particular to those linked with DICOM, HL7 and SCP-ECG standards, and highlighting the respective advantages and limitations with special emphasis to the needs typically encountered by the research community. The goal is to provide a snapshot of the present, and to discuss mid- and long-term potential directions and changes, emphasizing what digital ECG organizations could do to “save” ECG information and facilitate its widespread exchange.

The Biological Object Notation (BON): a structured file format for biological data

The large size and high complexity of biological data can represent a major methodological challenge for the analysis and exchange of data sets between computers and applications. There has also been a substantial increase in the amount of metadata associated with biological data sets, which is being increasingly incorporated into existing data formats. Despite the existence of structured formats based on XML, biological data sets are mainly formatted using unstructured file formats, and the incorporation of metadata results in increasingly complex parsing routines such that they become more error prone. To overcome these problems, we present the “biological object notation” (BON) format, a new way to exchange and parse nearly all biological data sets more efficiently and with less error than other currently available formats. Based on JavaScript Object Notation (JSON), BON simplifies parsing by clearly separating the biological data from its metadata and reduces complexity compared to XML based formats. The ability to selectively compress data up to 87% compared to other file formats and the reduced complexity results in improved transfer times and less error prone applications.

Focus on software and data acquisition

Focus on software and data acquisition

Ontology-Based Modeling of Control Logic in Building Automation Systems

The control logic implemented in building automation systems (BAS) has a significant impact on the overall energy demand of the building. However, information on the control logic, if documented, is often concealed from further data integration and reuse in heterogeneous information silos using disparate data formats. In particular, existing data formats and information models offer limited support to describe control logic explicitly. Ontology-based modeling of the control logic of BAS can potentially result in a versatile source of information for information-driven processes to further increase the performance of technical equipment in a building. Therefore, we present a novel information model, CTRLont, which allows to formally specify the domain of control logic in BAS. We demonstrate the usefulness of the novel information model by using it as a knowledge base for automating rule-based verification of designed control logic in BAS. We successfully apply the methodology to a simple control of an air handling unit and indicate a number of future steps.

ElVisML: an open data format for the exchange and storage of electrophysiological data in ophthalmology.

The ISCEV standards and recommendations for electrophysiological recordings in ophthalmology define a set of protocols with stimulus parameters, acquisition settings, and recording conditions, to unify the data and enable comparability of results across centers. Up to now, however, there are no standards to define the storage and exchange of such electrophysiological recordings. The aim of this study was to develop an open standard data format for the exchange and storage of visual electrophysiological data (ElVisML). We first surveyed existing data formats for biomedical signals and examined their suitability for electrophysiological data in ophthalmology. We then compared the suitability of text-based and binary formats, as well as encoding in Extensible Markup Language (XML) and character/comma-separated values. The results of the methodological consideration led to the development of ElVisML with an XML-encoded text-based format. This allows referential integrity, extensibility, the storing of accompanying units, as well as ensuring confidentiality and integrity of the data. A visualization of ElVisML documents (ElVisWeb) has additionally been developed, which facilitates the exchange of recordings on mailing lists and allows open access to data along with published articles. The open data format ElVisML ensures the quality, validity, and integrity of electrophysiological data transmission and storage as well as providing manufacturer-independent access and long-term archiving in a future-proof format. Standardization of the format of such neurophysiology data would promote the development of new techniques and open software for the use of neurophysiological data in both clinic and research.

Multimodels — Instant nD-modeling using original data

Abstract The exchange and provision of interdisciplinary information is a key to leverage of sophisticated cross-domain BIM applications like project controlling, simulations, energy efficiency or building automation. Thereby existing data formats, domain applications and information processes must be retained as far as possible. As current interoperability methods cannot cope with this challenge satisfactorily, the multimodel method is presented here, which enables interdisciplinary construction information processes on the base of original data. A multimodel therefore combines heterogeneous models of different domains. The loose cross-model coupling of data elements is neutrally stored in external ID-based link models. As the single domain models remain unaffected, they can still be used by traditional applications. Because of the absence of a leading or integrating data scheme, no transformation processes are required, common established data formats can be reused and the linked domain models can be exchanged neutrally. The linked data in multimodels offers an additional value of information in contrast to the single domain models. Information belonging together can be automatically evaluated by the persistent links. Thus, a multimodel appears to a user as a single self-contained information space. While the method focuses on closed world data exchange scenarios, it is not intended to form a singular project information space.

An Adaptable Seismic Data Format

We present ASDF, the Adaptable Seismic Data Format, a modern and practical data format for all branches of seismology and beyond. The growing volume of freely available data coupled with ever expanding computational power opens avenues to tackle larger and more complex problems. Current bottlenecks include inefficient resource usage and insufficient data organization. Properly scaling a problem requires the resolution of both these challenges, and existing data formats are no longer up to the task. ASDF stores any number of synthetic, processed or unaltered waveforms in a single file. A key improvement compared to existing formats is the inclusion of comprehensive meta information, such as event or station information, in the same file. Additionally, it is also usable for any non-waveform data, for example, cross-correlations, adjoint sources or receiver functions. Last but not least, full provenance information can be stored alongside each item of data, thereby enhancing reproducibility and accountability. Any data set in our proposed format is self-describing and can be readily exchanged with others, facilitating collaboration. The utilization of the HDF5 container format grants efficient and parallel I/O operations, integrated compression algorithms and check sums to guard against data corruption. To not reinvent the wheel and to build upon past developments, we use existing standards like QuakeML, StationXML, W3C PROV and HDF5 wherever feasible. Usability and tool support are crucial for any new format to gain acceptance. We developed mature C/Fortran and Python based APIs coupling ASDF to the widely used SPECFEM3D_GLOBE and ObsPy toolkits.

Encoding Data Using Biological Principles: The Multisample Variant Format for Phylogenomics and Population Genomics.

Rapid progress in the fields of phylogenomics and population genomics has driven increases in both the size of multi-genomic datasets and the number and complexity of genome-wide analyses. We present the Multisample Variant Format, specifically designed to store multiple sequence alignments for phylogenomics and population genomic analysis. The signature feature of MVF is a distinctive encoding of aligned sites with specific biological information content (e.g., invariant, low-coverage). This biological pattern-based encoding of sequence data allows for rapid filtering and quality control of data and speeds up computation for many analyses. Similar to other modern formats, MVF has a simple data structure and flexible header structure to accommodate project metadata, allowing to also serve as an effective data publication and sharing format. We also propose several variants of the MVF format to accommodate protein and codon alignments, quality scores, and a mix of de novo and reference-aligned data. Using the MVFtools package, MVF files can be converted from other common sequence formats. MVFtools completes tasks ranging from simple transformation and filtering operations to complex genome-wide visualizations in only a few minutes, even on large datasets. In addition to presentation of MVF and MVFtools, we also discuss the application both in MVF and other existing data formats of the broader concept of using biological principles and patterns to inform sequence data encoding.

Corrigendum: Software and hardware infrastructure for research in electrophysiology

This article adds Yann Le Franc as a co-author of the technology report article “Software and Hardware Infrastructure for Research in Electrophysiology,” describes his individual contribution to the article, and presents changes in two paragraphs in Section 2.4, where an additional reference is also provided. Moreover, the members from the OEN group who work and cooperate on building OEN are personally acknowledged. Co-author: Yann Le Franc Affiliation: e-Science Data Factory S.A.S.U., Paris, France; Ludwig-Maximilians-Universitat Munchen, Planegg-Martinsried, Germany; University of Antwerp, Antwerpen, Belgium Authors' Contributions Yann Le Franc contributed to the Figures 3,4 and 8 and to the work on the Ontology for describing Experimental Neurophysiology (OEN). Section 2.4 Old version: The group follows the best practices for creating ontologies, for example, it cooperates with community of researchers who design and create ontologies, uses existing data formats and repositories (odML, HDF5), and reuses existing resources (terms, ontologies—NEMO, OBI). For the general description of experimental neurophysiology, the terms from ontologies NEMO and OBI are relevant. However, the set of the domain terms is still not complete in these ontologies (information stored in the EEG/ERP Portal cannot be fully described by these ontologies) and OEN will be finally an extension of OBI (e.g., the granularity of OBI for devices and related information will be extended). New version: The group follows the best practices for creating ontologies, for example, it cooperates with community of researchers who design and create ontologies, uses existing data formats and repositories (odML, HDF5), and reuses existing resources (terms, ontologies—NEMO, OBI). For the general description of experimental neurophysiology, the terms from ontologies NEMO and OBI are relevant. However, the set of the domain terms needed to describe the information stored in the EEG/ERP Portal is not yet complete in these ontologies. OEN aims at defining these missing terms and at term, should be used to propose an extension of OBI's neurophysiology model (e.g., the granularity of OBI for devices and related information will be extended). Old version: Terminologies within OEN have been primarily developed in the odML format. Subsequently, an OWL file has been constructed aided by Ontofox (Xiang et al., 2010). The current developer's version of OEN is available at https://github.com/G-Node/OEN. New version: The OEN device branch development is based on the odML terminology (Grewe et al., 2011), concepts gathered by the Neuroscience Information Framework (NIF) and concepts used in the EEGBase data model to describe setups and setup configurations. The gathered terms are currently mapped with the aforementioned ontologies. Subsequently, an OWL file has been constructed to contain OEN terms and the mapped terms. Existing terms in other ontologies will be imported using the MIREOT approach (Courtot et al., 2011), aided by Ontofox (Xiang et al., 2010). The current developer's version of OEN is available at https://github.com/G-Node/OEN.

Performance evaluation of 64 × 10 Gbps and 96 × 10 Gbps DWDM system with hybrid optical amplifier for different modulation formats

In this paper, we investigated the performance of multi terabits DWDM system consisting of hybrid optical amplifier RAMAN-EDFA for different data format such as non-return to zero (NRZ), return to zero (RZ) and differential phase shift keying (DPSK). We find that in 64×10 and 96×10Gbps, RZ is more adversely affected by nonlinearities, where as NRZ and DPSK is more affected by dispersion. We further show that RZ provide good quality factor (13.88dB and 15.93dB for 64 and 96 channels), less eye closure (2.609dB and 3.191dB for 64 and 96 channels) and acceptable bit error rate (3.89×108 and 1.24×109 for 64 and 96 channels) at the respective distance as compare to other existing modulation format. We further investigated the maximum single span distance covered by using existing data formats.

Effective Sparse Matrix Representation For The GPU Architectures

General purpose computation on graphics processing unit (GPU) is prominent in the high performance computing era of this time. Porting or accelerating the data parallel applications onto GPU gives the default performance improvement because of the increased computational units. Better performances can be seen if application specific fine tuning is done with respect to the architecture under consideration. One such very widely used computation intensive kernel is sparse matrix vector multiplication (SPMV) in sparse matrix based applications. Most of the existing data format representations of sparse matrix are developed with respect to the central processing unit (CPU) or multi cores. This paper gives a new format for sparse matrix representation with respect to graphics processor architecture that can give 2x to 5x performance improvement compared to CSR (compressed row format), 2x to 54x performance improvement with respect to COO (coordinate format) and 3x to 10 x improvement compared to CSR vector format for the class of application that fit for the proposed new format. It also gives 10% to 133% improvements in memory transfer (of only access information of sparse matrix) between CPU and GPU. This paper gives the details of the new format and its requirement with complete experimentation details and results of comparison .