Data Storage Format Research Articles

Optimizing Cloud Data Storage: Evaluating File Formats for Efficient Data Warehousing

This paper presents a detailed analysis of three widely-used data storage formats—Parquet, Avro, and ORC— evaluating their performance across key metrics such as query execution, compression efficiency, data skipping, schema evolution, and throughput. Each format offers distinct advantages depending on the nature of the workload. Parquet is optimized for read-heavy analytical queries, providing excellent compression and efficient query performance through its columnar structure. Avro excels in write-heavy, real-time data streaming scenarios, where schema flexibility and backward compatibility are crucial. ORC balances the two, offering strong support for analytical and transactional workloads, especially in handling complex queries and nested data structures. This comparative study highlights the contexts in which each format performs best, providing valuable insights into the trade-offs associated with their use in cloud data warehouses and large-scale data processing environments.

What does e-invoice data bring to SNA and real-time economy?

This paper explores the potential of electronic invoice data to enhance the System of National Accounts (SNA) and facilitate a Real-Time Economy (RTE). The rise of data science and advancements in information and communication technologies have increased the use of big data in economic statistics. E-invoice data is being digitized in Europe to detect VAT fraud, but it also contains a vast amount of information on economic entities that make up the SNA. This study focuses on integrating electronic accounting information to improve the coverage, granularity, and update speed of SNA. The paper examines the concept of RTE, which emphasizes real-time data processing and automation, particularly in Estonia and Finland. It also highlights the benefits and challenges of using invoice data for economic statistics, including issues related to data storage formats, legislative systems, and the coordination of data linkages. The research demonstrates invoice data’s theoretical and practical applications for constructing Input–Output Tables (IOTs) and monitoring financial situations. Despite the technical and institutional hurdles, the study suggests integrating invoice data can significantly transform economic statistics.

JCDC: A blockchain-based framework for secure data storage and circulation in JointCloud

JointCloud computing represents a new generation cloud computing paradigm, which deeply integrates the cloud resources of multiple Cloud Service Providers (CSPs) to offer tailored cloud services to users. In contrast to traditional multi-cloud environment, JointCloud environment involve data circulation among multiple CSPs. However, in JointCloud environment, CSPs are not always fully trustworthy and they may illegally infringe upon users’ data privacy and security for their own benefit. Additionally, the heterogeneity arising from different data storage formats, structures, access control, and permission management mechanisms adopted by various CSPs makes achieving unified data management in JointCloud challenging. Therefore, to ensure secure storage and efficient circulation of data within JointCloud, it is essential to prevent violations for user privacy and data ownership, shield the heterogeneity of underlying data management mechanisms across different CSPs, and establish trusted transactions between CSPs. In this paper, we propose a framework called JointCloud Data Chain (JCDC) based on JointCloud computing and blockchain for data storage and circulation, aiming to ensure secure data storage and trustworthy transactions. JCDC utilizes blockchain to record data ownership and control data circulation, while integrating storage resources from various CSPs to construct a distributed off-chain Personal Data Storage (PDS) for expanding system storage capacity. Additionally, JCDC employs Certificateless Public Key Cryptography (CL-PKC) and Proxy Re-encryption technologies for user identity management and secure data transactions. Furthermore, smart contracts are designed to enable automated data storage and sharing. We conduct a security analysis of JCDC and develop a prototype system to validate its performance and practicality. Finally, extensive experimentation and analysis demonstrate that JCDC exhibits low time latency and cost, which makes it practical.

Sentiment-driven Movie Recommendation System: A Machine Learning Approach

Pioneering method for enhancing movie recommendation systems through the integration of sentiment analysis of micro-blogging data. Leveraging natural language processing (NLP) techniques, the proposed system extracts sentiment from tweets and other micro-blogging sources pertinent to movies systematic approach encompassing the collection, preprocessing, and analysis of movie review data tailored for sentiment analysis tasks. Key aspects covered include the acquisition of publicly available datasets, methodologies for web scraping, preprocessing techniques, strategies for sentiment labeling, methods for data augmentation, procedures for data splitting, optimal data storage formats, and ethical considerations inherent in data collection and utilization. By offering a comprehensive guide, furnish both researchers and practitioners with the necessary tools to proficiently manage movie review data while navigating the ethical intricacies associated with its acquisition and application.

Distributed matrix computing system for big data

In order to solve the problem of low computing efficiency in big data analysis and model construction, this paper intended to deeply explore the big data analysis programming model, DAG (Directed Acyclic Graph) and other contents, and on this basis, it adopted a distributed matrix computing system Octopus for big data analysis. Octopus is a universal matrix programming framework that provides a programming model based on matrix operations, which can conveniently analyze and process large-scale data. By using Octopus, users can extract functions and data from multiple platforms and operate through a unified matrix operation interface. The distributed matrix representation and storage layer can design data storage formats for distributed file systems. Each computing platform in OctMatrix provides its own matrix library, and it provides a matrix library written in R language for the above users. SymboMatrix provides a matrix interface to OctMatrix that is consistent with OctMatrix. However, SymboMatrix also retains the flow diagram for matrix operations in the process, and it also supports logical and physical optimization of the flow diagram on a DAG. For the DAG computational flow graph generated by SymbolMatrix, this paper divided it into two parts: logical optimization and physical optimization. This paper adopted a distributed file system based on line matrix, and obtained the corresponding platform matrix by reading the documents based on line matrix. In the evaluation of system performance, it was found that the distributed matrix computing system had a high computing efficiency, and the average CPU (central processing unit) usage reached 70%. This system can make full use of computing resources and realize efficient parallel computing.

Data storage format for analytical systems based on metadata and dependency graphs between CSV and JSON

In the modern information society, the volume of data is constantly growing, and its effective processing is becoming key for enterprises. The transmission and storage of this data also plays a critical role. Big data used in analytics systems is most often transmitted in one of two popular formats: CSV for structured data and JSON for unstructured data. However, existing file formats may not be effective or flexible enough for certain data analysis tasks. For example, they may not support complex data structures or provide sufficient control over metadata. Alternatively, analytical tasks may require additional information about the data, such as metadata, data schema, etc. Based on the above, the subject of this study is a data format based on the combined use of CSV and JSON for processing and analyzing large amounts of information. The option of sharing the designated data types for the implementation of a new data format is proposed. For this purpose, designations have been introduced for the data structure, which includes CSV files, JSON files, metadata and a dependency graph. Various types of functions are described, such as aggregating, transforming, filtering, etc. Examples of the application of these functions to data are given. The proposed approach is a technique that can significantly facilitate the processes of information analysis and processing. It is based on a formalized approach that allows you to establish clear rules and procedures for working with data, which contributes to their more efficient processing. Another aspect of the proposed approach is to determine the criteria for choosing the most appropriate data storage format. This criterion is based on the mathematical principles of information theory and entropy. The introduction of a criterion for choosing a data format based on entropy makes it possible to evaluate the information content and compactness of the data. This approach is based on the calculation of entropy for selected formats and weights reflecting the importance of each data value. By comparing entropies, you can determine the required data transmission format. This approach takes into account not only the compactness of the data, but also the context of their use, as well as the possibility of including additional meta-information in the files themselves and supporting data ready for analysis.

Streamlined Algorithms for Large-Scale Matrix Computations in High-Performance Computing

In the realm of high-performance computing (HPC), large-scale matrix computations are fundamental to a myriad of scientific and engineering applications, from simulations in physics and engineering to data analysis in machine learning. This paper addresses the critical need for efficient and scalable algorithms capable of handling the growing computational demands of these applications. This study explores advanced algorithmic strategies that leverage the architecture of modern HPC systems, including parallelism, distributed computing, and optimized memory usage. The proposed algorithms are designed to minimize computational complexity and maximize performance, ensuring that largescale matrix operations can be executed swiftly and accurately. One of the key contributions of this work is the development of parallel algorithms that distribute the computational workload across multiple processors, significantly reducing execution time. By employing techniques such as data partitioning and workload balancing, the algorithms ensure optimal use of available computational resources, thus enhancing scalability. Furthermore, the paper introduces innovative approaches to memory management, which are crucial for handling large matrices that exceed the memory capacity of individual computing nodes. These approaches include efficient data storage formats and dynamic memory allocation strategies that reduce overhead and improve data access speeds. Another critical aspect of this research is the application of these algorithms to real-world problems. The paper presents case studies in fields such as climate modeling, structural analysis, and machine learning, demonstrating the practical benefits and performance gains achieved through the use of streamlined matrix computation algorithms. The results show substantial improvements in computation time and resource utilization, validating the effectiveness of the proposed methods. Additionally, the paper discusses the integration of these algorithms into existing HPC frameworks and libraries, making them accessible to a broader community of researchers and practitioners. By providing detailed implementation guidelines and performance benchmarks, the study ensures that the benefits of these advanced algorithms can be readily harnessed in various applications. The research also delves into the theoretical underpinnings of the proposed algorithms, offering a rigorous analysis of their computational complexity and scalability. In conclusion, this paper makes significant strides in addressing the challenges of large-scale matrix operations. Through the development of efficient, scalable, and practical algorithms, this research enhances the capability of HPC systems to tackle complex computational tasks, paving the way for advancements in various scientific and engineering domains. The proposed methods not only improve performance and resource utilization but also contribute to the broader goal of making high-performance computing more accessible and effective for large-scale data analysis and simulation.

Streamlined Algorithms for Large-Scale Matrix Computations in High-Performance Computing

In the realm of high-performance computing (HPC), large-scale matrix computations are fundamental to a myriad of scientific and engineering applications, from simulations in physics and engineering to data analysis in machine learning. This paper addresses the critical need for efficient and scalable algorithms capable of handling the growing computational demands of these applications. This study explores advanced algorithmic strategies that leverage the architecture of modern HPC systems, including parallelism, distributed computing, and optimized memory usage. The proposed algorithms are designed to minimize computational complexity and maximize performance, ensuring that largescale matrix operations can be executed swiftly and accurately. One of the key contributions of this work is the development of parallel algorithms that distribute the computational workload across multiple processors, significantly reducing execution time. By employing techniques such as data partitioning and workload balancing, the algorithms ensure optimal use of available computational resources, thus enhancing scalability. Furthermore, the paper introduces innovative approaches to memory management, which are crucial for handling large matrices that exceed the memory capacity of individual computing nodes. These approaches include efficient data storage formats and dynamic memory allocation strategies that reduce overhead and improve data access speeds. Another critical aspect of this research is the application of these algorithms to real-world problems. The paper presents case studies in fields such as climate modeling, structural analysis, and machine learning, demonstrating the practical benefits and performance gains achieved through the use of streamlined matrix computation algorithms. The results show substantial improvements in computation time and resource utilization, validating the effectiveness of the proposed methods. Additionally, the paper discusses the integration of these algorithms into existing HPC frameworks and libraries, making them accessible to a broader community of researchers and practitioners. By providing detailed implementation guidelines and performance benchmarks, the study ensures that the benefits of these advanced algorithms can be readily harnessed in various applications. The research also delves into the theoretical underpinnings of the proposed algorithms, offering a rigorous analysis of their computational complexity and scalability. In conclusion, this paper makes significant strides in addressing the challenges of large-scale matrix operations. Through the development of efficient, scalable, and practical algorithms, this research enhances the capability of HPC systems to tackle complex computational tasks, paving the way for advancements in various scientific and engineering domains. The proposed methods not only improve performance and resource utilization but also contribute to the broader goal of making high-performance computing more accessible and effective for large-scale data analysis and simulation.

PyDapsys: an open-source library for accessing electrophysiology data recorded with DAPSYS.

In the field of neuroscience, a considerable number of commercial data acquisition and processing solutions rely on proprietary formats for data storage. This often leads to data being locked up in formats that are only accessible by using the original software, which may lead to interoperability problems. In fact, even the loss of data access is possible if the software becomes unsupported, changed, or otherwise unavailable. To ensure FAIR data management, strategies should be established to enable long-term, independent, and unified access to data in proprietary formats. In this work, we demonstrate PyDapsys, a solution to gain open access to data that was acquired using the proprietary recording system DAPSYS. PyDapsys enables us to open the recorded files directly in Python and saves them as NIX files, commonly used for open research in the electrophysiology domain. Thus, PyDapsys secures efficient and open access to existing and prospective data. The manuscript demonstrates the complete process of reverse engineering a proprietary electrophysiological format on the example of microneurography data collected for studies on pain and itch signaling in peripheral neural fibers.

Estimating the Effect of Indices Compression in the CSR-like Data Storage Formats for Matrix-Vector Multiplications and Solving Linear Systems

Estimating the Effect of Indices Compression in the CSR-like Data Storage Formats for Matrix-Vector Multiplications and Solving Linear Systems

Neuroscience Data Formats, Models, Repositories and Analytics: A Review

As neurotechnologies have gotten better, a lot of neuroscientific research has been done using these new technologies. Even though labs all over the world produce a lot of neuro-data, most of this data has not been shared to help people from different fields understand neuroscience. The neuro-data sharing is essential because it accelerates the pace of discovery in neuroscience. Effective data sharing will depend on the standardized use of file or data formats, highly reusable data analytics tools, and data storage formats. In this review paper, we review the four domains (data format, data model, data repository, and data analytics) that are currently in use in the neuroscience community. In the end, we are discussing several challenges associated with data sharing.

MDIO: Open-source format for multidimensional energy data

MDIO is a fully open-source data storage format that enables computational workflows for various high-dimensional energy data sets including seismic data and wind models. Designed to be efficient and flexible, MDIO provides interoperable software infrastructure with existing energy data standards. It leverages an open-source format called Zarr to enable data usage in the cloud and on-premises file systems. An overview of the data model and schema for MDIO is provided, and an open-source Python library developed to work with MDIO data is demonstrated. We explain how MDIO supports different computational workflows and discuss applications for data management, seismic imaging, machine learning, wind resource assessment, and real-time seismic visualization. Overall, MDIO gives researchers, practitioners, and developers in the energy sector a standardized and open approach to managing and sharing multidimensional energy data.

BT07 Creation of a skin lesion image pipeline: key steps in developing a mobile application with DICOM® fields and metadata

Abstract A key issue facing artificial intelligence (AI) for dermatology is the lack of standardization around capture, transfer and storage of skin images and clinical data. To assist systematic capture of clinical data, we developed a nationally agreed proposed minimum dataset, based on Digital Imaging and Communications in Medicine (DICOM®) standards. Subsequently, we collaborated with an established image management system (IMS) supplier, Waba [supported by grants from our local National Health Service (NHS) innovation department], to incorporate these fields into a dermatology-specific mobile application (app), facilitating real-time capture and data transfer. We describe our steps in developing this app. The app integrates with the national Patient Administration System (SCI Store), allowing retrieval of real-time patient demographics linked to hospital or Community Health Index number. Each care episode captures a standardized dataset, including lesion views, clinical information and consent level. On episode completion, datasets are securely transferred to the IMS via WiFi or GPRS signal, following a fully auditable pathway. The IMS is a web-based app developed on a Microsoft Structured Query Language server, which allows authenticated users to access our trust’s centralized repository of nonradiological images. Adherence to industry-standard data storage format, patient safety reporting and specific access rights ensure compliance with Caldicott recommendations and data protection standards. For AI research, we are exploring conversion of datasets into DICOM format. Close collaboration between software developers and end users, including clinical photography, dermatology and general practice, was essential for app design. We worked on design architecture through fortnightly technical meetings, scheduled to progress development in months 1–8. Installation and functionality testing was performed in months 9 and 10, followed by comprehensive user testing in months 10–12. Live app test-of-change rollout was performed in months 13–16, with continued usability refinement during this period. To develop a standardized minimum dataset, we conducted a consensus exercise with the British Association of Dermatology AI working group. Based on DICOM standards, a comprehensive international standard for medical imaging, this could improve the quality, availability and interoperability of data for AI development. In summary, we have developed an app to standardize the capture, transmission and storage of skin lesion images and associated metadata. By collaborating with an industry partner with NHS experience, iterative user-led design and national engagement for standard-setting, we believe this is a useful step toward a national UK skin database for training and validating AI systems. Future work is planned to evaluate the output of this data pipeline.

A Comprehensive Evaluation of Data Quality in Nutrient Databases.

Nutrient databases are a critical component of nutrition science and the basis of exciting new research in precision nutrition (PN). To identify the most critical components needed for improvement of nutrient databases, food composition data were analyzed for quality, with completeness being the most important measure, and for FAIRness, how well the data conformed with the data science criteria of findable, accessible, interoperable, and reusable (FAIR). Databases were judged complete if they provided data for all 15 nutrition fact panel (NFP) nutrient measures and all 40 National Academies of Sciences, Engineering, and Medicine (NASEM) essential nutrient measures for each food listed. Using the gold standard the USDA standard reference (SR) Legacy database as surrogate, it was found that SR Legacy data were not complete for either NFP or NASEM nutrient measures. In addition, phytonutrient measures in the 4 USDA Special Interest Databases were incomplete. To evaluate data FAIRness, a set of 175 food and nutrient data sources were collected from worldwide. Many opportunities were identified for improving data FAIRness, including creating persistent URLs, prioritizing usable data storage formats, providing Globally Unique Identifiers for all foods and nutrients, and implementing citation standards. This review demonstrates that despite important contributions from the USDA and others, food and nutrient databases in their current forms do not yet provide truly comprehensive food composition data. We propose that to enhance the quality and usage of food and nutrient composition data for research scientists and those fashioning various PN tools, the field of nutrition science must step out of its historical comfort zone and improve the foundational nutrient databases used in research by incorporating data science principles, the most central being data quality and data FAIRness.

Acquisition and Processing Data from UAVs in the Process of Generating 3D Models for Solar Potential Analysis

UAVs have recently become a very popular tool for acquiring geospatial data. Photographs, films, images, and results of measurements of various sensors from them constitute source material for generating, among other things, photographic documentation, visualisation of places and objects, cartographic materials and 3D models. These models are not only material for the visualisation of objects but are also source material for spatial analysis, including the assessment and analyses of the solar potential of buildings. This research aims to benchmark the feasibility of using UAV-derived data acquired from three sensors, namely the DJI Zenmuse P1 camera, the Share PSDK102S v2 multi-lens camera and the DJI Zenmuse L1 laser scanner. The data from these were acquired for the construction of comprehensive and reliable 3D models, which will form the basis for generating solar potential maps. Various sensors, data storage formats, and geospatial data processing capabilities are analysed to determine the most optimal and efficient solution for providing accurate, complete and reliable 3D models of places and objects for the construction of solar potential maps. In this paper, the authors prepare a compilation of the results of the studies from different measurement combinations and analyse the strengths and weaknesses of the different solutions, as well as the integration of the results for an optimal 3D model, which was used to perform solar potential analyses for the selected built-up area. The results of the study show that the parameters for assessing the quality of a 3D model can be statistical parameters that determine the coplanarity of roof slope points (i.e., standard deviation, distances from the plane, and RMS value). The completeness of the model is defined as the percentage of the recorded area by sensors to the total area of the model.

Estimating the Values of Missing Data Related to Infrastructure Condition States Using Their Spatial Correlation

Infrastructure managers consistently monitor the condition of their assets to predict their deterioration speed and determine the optimal time to execute preventive interventions. However, despite the recent progress in more frequent and accurate monitoring of assets and storage of the related results, in practice, real-world data often contains errors and discrepancies such as missing data or faulty entries. This problem can happen owing to collection errors during routine inspections or inconsistency of data storage formats in different years. Because the quality of data plays a significant role in the accuracy of deterioration prediction and the resulting intervention programs, it is important to improve condition state predictions by imputing the values of missing information. This paper examines the efficiency of different models that use the spatial correlation of infrastructure assets in predicting the value of missing data. The models include univariate and multivariate Kriging, a hybrid artificial neural network (ANN)-Kriging model, and the bidirectional long short term memory (bi-LSTM) neural network, which can model the data with spatial correlation or a sequential relationship. The results confirm that the condition indicator values can be estimated with reasonably low levels of error.

MetaScenario: A Framework for Driving Scenario Data Description, Storage and Indexing

Autonomous driving related researches require the analysis and usage of massive amounts of driving scenario data. Compared to raw data collected by sensors, scenario data provide a preliminary abstraction of driving tasks and processes, explicitly integrate information about the road environment and the dynamic and static attributes of traffic participants, making it easier to conduct task understanding and decision making. However, many existing driving scenario datasets have the following two problems. First, it is not clear which data fields need to be recorded for driving scenarios. The data storage formats and organization standards are inconsistent. Second, the datasets cannot establish driving scenario indexing effectively. Existing datasets are sparsely annotated and difficult to index, which is detrimental to data sampling and extraction for machine learning process, thus hindering efficient fusion and reuse. In this paper, we propose MetaScenario, a framework for driving scenario data. We describe driving scenarios and design the centralized and unified data framework for the storage, processing, and indexing of scenario data based on relational database. The concept of atom scenario is proposed and characterized using semantic graphs. We also annotate and classify behaviors and interactions of traffic participants in atom scenarios by extracting the spatiotemporal evolution of semantic information. The annotation facilitates the indexing and extraction of data. The scenario datasets are further evaluated via the data distribution and annotation statistics. MetaScenario can provide researchers with convenient tools for scenario data extraction and important analytical references.

Пути организации параллельного доступа к структурированным данным

This paper explores ways to achieve the highest possible exchange performance with files containing structured data. The research was carried out on file systems with supercomputer systems parallel access designed to solve problems of physical and mathematical modeling of various processes and objects. For example, parallel access to raw data is considered using the Lustre file system. The article suggests a way to organize parallel access to structured data based on a specially developed PSIO storage format and the psio access library. A comparative analysis of the I/O performance of the developed data storage format and the HDF5 parallel version format is performed.

Designing privacy in personalized health: An empirical analysis

A crucial challenge for personalized health is the handling of individuals’ data and specifically the protection of their privacy. Secure storage of personal health data is of paramount importance to convince citizens to collect personal health data. In this survey experiment, we test individuals’ willingness to produce and store personal health data, based on different storage options and whether this data is presented as common good or private good. In this paper, we focus on the nonmedical context with two means to self-produce data: connected devices that record physical activity and genetic tests that appraise risks of diseases. We use data from a survey experiment fielded in Switzerland in March 2020 and perform regression analyses on a representative sample of Swiss citizens in the French- and German-speaking cantons. Our analysis shows that respondents are more likely to use both apps and tests when their data is framed as a private good to be stored by individuals themselves. Our results demonstrate that concerns regarding the privacy of personal heath data storage trumps any other variable when it comes to the willingness to use personalized health technologies. Individuals prefer a data storage format where they retain control over the data. Ultimately, this study presents results susceptible to inform decision-makers in designing privacy in personalized health initiatives.

Acceleration of structural topology optimization using symmetric element-by-element strategy for unstructured meshes on GPU

PurposeStructural topology optimization is computationally expensive due to the involvement of high-resolution mesh and repetitive use of finite element analysis (FEA) for computing the structural response. Since FEA consumes most of the computational time in each optimization iteration, a novel GPU-based parallel strategy for FEA is presented and applied to the large-scale structural topology optimization of 3D continuum structures.Design/methodology/approachA matrix-free solver based on preconditioned conjugate gradient (PCG) method is proposed to minimize the computational time associated with solution of linear system of equations in FEA. The proposed solver uses an innovative strategy to utilize only symmetric half of elemental stiffness matrices for implementation of the element-by-element matrix-free solver on GPU.FindingsUsing solid isotropic material with penalization (SIMP) method, the proposed matrix-free solver is tested over three 3D structural optimization problems that are discretized using all hexahedral structured and unstructured meshes. Results show that the proposed strategy demonstrates 3.1× –3.3× speedup for the FEA solver stage and overall speedup of 2.9× –3.3× over the standard element-by-element strategy on the GPU. Moreover, the proposed strategy requires almost 1.8× less GPU memory than the standard element-by-element strategy.Originality/valueThe proposed GPU-based matrix-free element-by-element solver takes a more general approach to the symmetry concept than previous works. It stores only symmetric half of the elemental matrices in memory and performs matrix-free sparse matrix-vector multiplication (SpMV) without any inter-thread communication. A customized data storage format is also proposed to store and access only symmetric half of elemental stiffness matrices for coalesced read and write operations on GPU over the unstructured mesh.