Storage Schema Research Articles

Deployment and Operation of the ATLAS EventIndex for LHC Run 3

The ATLAS Eventlndex is the global catalogue of all ATLAS real and simulated events. During the LHC long shutdown between Run 2 (20152018) and Run 3 (2022-2025) all its components were substantially revised and a new system was deployed for the start of Run 3 in Spring 2022. The new core storage system, based on HBase tables with a SQL interface provided by Phoenix, allows much faster data ingestion rates and scales much better than the old one to the data rates expected for the end of Run 3 and beyond. All user interfaces were also revised and a new command-line interface and web services were also deployed. The new system was initially populated with all existing data relative to Run 1 and Run 2 datasets, and then put online to receive Run 3 data in real time. After extensive testing, the old system, which ran in parallel to the new one for a few months, was finally switched off in October 2022. This paper describes the new system, the move of all existing data from the old to the new storage schemas and the operational experience gathered so far.

VRE

Managing massive trajectory data from various moving objects has always been a demanding task. A desired trajectory data system should be versatile in its supported query types and distance functions, of low storage cost, and be consistently efficient on processing trajectory data of different properties. Unfortunately, none of the existing systems can meet the above three criteria at the same time. To this end, we propose VRE, a v ersatile, r obust, and e conomical trajectory data system.VRE separates the storage from the processing. In the storage layer, we propose a novel segment-based storage model that takes advantage of the strengths of both point-based and trajectory-based storage models. VRE supports these three storage models and ten storage schemas upon them. With the secondary index, VRE reduces the storage cost up to 3x. In the processing layer, we first propose a two-stage processing framework and a pushdown strategy to alleviate full trajectory transmission cost. Then, we design a unified pruning strategy for five widely used trajectory distance functions and numerous tailored processing algorithms for five advanced queries. Extensive experiments are conducted to verify the design choice and efficiency of VRE, from which we present some key insights that are crucial to both VRE and future trajectory system's design.

TrajMesa: A Distributed NoSQL-Based Trajectory Data Management System

With the development of positioning technology, a large number of trajectories have been generated, which are very useful for many urban applications. However, it is challenging to manage trajectory data for its spatio-temporal dynamics and high-volume properties. Existing trajectory data management frameworks suffer from efficiency or scalability problem, and only support limited trajectory query types. This paper takes the first attempt to build a holistic distributed NoSQL trajectory storage engine, named TrajMesa, based on GeoMesa, an open-source indexing toolkit for spatio-temporal data. TrajMesa can manage a prohibitively large number of trajectories, and support plenty of query types efficiently. Specifically, we first design a novel trajectory storage schema, which reduces the storage size tremendously. We then devise a novel indexing key schema for time ranges, based on which ID temporal query can be supported efficiently. To reduce the amount of retrieved trajectory data for a spatial range query, we innovatively propose a position code to indicate the spatial location of trajectories accurately. We also propose a bunch of pruning strategies for similarity query and k-NN query in the NoSQL environment. Extensive experiments are conducted using two real datasets and one synthetic dataset, verifying the powerful query efficiency and scalability of TrajMesa.

The data storage system for SHINE

Shanghai HIgh repetition rate XFEL aNd Extreme light facility (SHINE) is a quasi-continuous wave hard X-ray free electron laser facility, which is currently under construction. The data storage system holds large volumes of data generated instantaneously during the operation. A combination architecture of relational database, non-relational database and the distributed file system is designed as the centralized data storage system serving to multiple application demands for SHINE. The storage of high-speed stream data such as waveform and image data is studied and relevant storage schemas are tested and compared. The optimized schema is identified finally.

Unsupervised Semantic Mapping for Healthcare Data Storage Schema

Data, information, and knowledge processing systems, in the domain of healthcare, are currently plagued by heterogeneity at various levels. Current solutions have focused on developing a standard-based, manual intervention mechanism, which requires a large number of human resources and necessitates the realignment of existing systems. State-of-the-art methodologies in the field of natural language processing and machine learning can help to partially automate this process, reducing the resource requirements and providing a relatively good multi-class-based classification algorithm. We present a novel methodology for bridging the gap between various healthcare data management solutions by leveraging the strength of transformer-based machine learning models, to create mappings between the data elements. Additionally, the annotated data, collected against five medical schemas and labeled by four annotators is made available for helping future researchers. Our results indicate, that for biased, dependent multi-class text classification, transformer-based models provide better results than linguistic and other classical models. In particular, the Robustly Optimized BERT Pretraining Approach (RoBERTa) provides the best schema matching performance by achieving a Cohen's kappa score of 0.47 and Matthews Correlation Coefficient (MCC) score of 0.48, with human-annotated data.

A Blockchain-Driven Electronic Contract Management System for Commodity Procurement in Electronic Power Industry

In the era of paper-based contract, a lot of time, human labor and expenses are required to handle the process of contract drafting, contract signing, contract execution, and payment settlement. The emerging of electronic contract enhances the tedious signing process of paper-based contract and improves the efficiency of contract management. However, due to the centralized system architecture and the database-based storage schema, the stored contract data is at high risk of information leaking, data tampering and hacker attacks. In this study, we introduce Blockchain technology to the contract management, and develop a process-oriented contract management system (BEcontractor) for a Hangzhou-located power grid enterprise X, aiming to solve a series of security issues existing in the traditional electronic contract system. By deploying BEcontractor, procurement activities could be resumed online among X and its nation-widely commodity suppliers during COVID-19 epidemic. Up to September 2020, 6336 electronic contracts have been signed, with an accumulated amount of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\yen ~6.5$ </tex-math></inline-formula> billion. It is showed that the cost for accomplishing the contract signing process was significantly reduced, and the payment period was shortened from three months to around one month.

A Secured and Efficient Privacy Preserving Cloud Storage and Fog Storage Schema with Privacy

A Secured and Efficient Privacy Preserving Cloud Storage and Fog Storage Schema with Privacy

Storing and querying fuzzy RDF(S) in HBase databases

The Resource Description Framework (RDF) and RDF Schema (RDFS) recommended by World Wide Web Consortium (W3C) have been used for meta-data management on the Web. With the rapid development of Web-based applications, the growth of RDF(S) data is increasing dramatically. In the context of open Web, uncertainty naturally arises in RDF and RDFS (RDF(S) in short). To manage large-scale fuzzy RDF(S) data efficiently and effectively, we propose, in this paper, a fuzzy RDF(S) storage schema with fuzzy HBase databases (FHDBs). On the basis, we investigate how to query in FHDBs. We propose a set of query algorithms. We implement a prototype system to demonstrate the feasibility of our approach.

How Do I Share My IoT Forensic Experience With the Broader Community? An Automated Knowledge Sharing IoT Forensic Platform

It is challenging for digital forensic practitioners to maintain skillset currency, for example knowing where and how to extract digital artifacts relevant to investigations from newer, emerging devices (e.g., due to the increased variety of data storage schemas across manufacturers and constantly changing models). This paper presents a knowledge sharing platform, developed and validated using an Internet of Things dataset released in the DFRWS 2017–2018 forensic challenge. Specifically, we present an automated knowledge-sharing forensic platform that automatically suggests forensic artifact schemas, derived from case data, but does not include any sensitive data in the final (shared) schema. Such artifact schemas are then stored in a schema pool and the platform presents candidate schemas for use in new cases based on the data presented. In this way, investigators need not learn the forensic profile of a new device from scratch, nor do they have to manually anonymize and share forensic knowledge obtained during the course of an investigation.

High density DNA data storage library via dehydration with digital microfluidic retrieval

DNA promises to be a high density data storage medium, but physical storage poses a challenge. To store large amounts of data, pools must be physically isolated so they can share the same addressing scheme. We propose the storage of dehydrated DNA spots on glass as an approach for scalable DNA data storage. The dried spots can then be retrieved by a water droplet using a digital microfluidic device. Here we show that this storage schema works with varying spot organization, spotted masses of DNA, and droplet retrieval dwell times. In all cases, the majority of the DNA was retrieved and successfully sequenced. We demonstrate that the spots can be densely arranged on a microfluidic device without significant contamination of the retrieval. We also demonstrate that 1 TB of data could be stored in a single spot of DNA and successfully retrieved using this method.

Improving NoSQL Storage Schema Based on Z-Curve for Spatial Vector Data

NoSQL database can provide massive, high concurrency, and scalable services for storing different types of data. HBase, a type of NoSQL database, in which columns are grouped into column families, is very suitable for storing semi-structured or unstructured spatial vector data. However, since there are few rules and constraints to be followed for the NoSQL database, the design of storage schema for spatial data based on NoSQL is difficult. In this paper, based on our early work, an improved Z-curve storage schema is proposed for spatial vector data. According to our new schema, row key of a geometric object is the Z-curve code of the spatial grids intersected with the geometric object. Moreover, geometric objects with the same row key are stored in a column family. Our proposed method has two features. First, geometric objects adjacent in the location are adjacent in physical storage. Second, redundant exists in storage for improving query accuracy. In our experiments, we compare the improved Z-curve storage schema with a Quadtree storage schema, an R-tree storage schema, and the previous Z-curve storage schema. Query response time, memory usage, and the query accuracy of spatial query on point and range are used to verify the validity of our proposed method. The experimental results show that the two storage schemas based on Z-curve achieve higher query efficiency than the two storage schemas based on tree-the Quadtree storage schema and the R-tree storage schema. More importantly, the query results of the improved Z-curve schema are completely correct, while the query results of the previous Z-curve schema are not.

Blockchain-Based Outsourced Storage Schema in Untrusted Environment

Outsourced data, as significant service offered by the cloud service provider (CSP), can effectively facilitate the data owner (DO) overcoming the storage limitations on massive data. To ensure the availability of data, DO usually outsources the data replications to multiple CSPs (multi-CSPs) and utilizes a third party metadata management (TPMM) to dominate the metadata of the corresponding replications. However, during the outsourced procedures, DO can hardly confirm the confidence of the TPMM who may take some malicious behaviors to affect the reliability of data. Thus, DO inevitably faces data security issues caused by the over-reliance on the semi-trusted TPMM to manage the metadata of replications. In this paper, we focus on the problem of reliable outsourced data service among multi-CSPs in untrusted environment, that is, how to reliably store and verify the metadata of the data replications in untrusted multi-CSPs environment. To address the problem, we use the novel blockchain technology as a medium to build a trusted outsourced service platform. Moreover, we fully consider the innovative characteristics of blockchain including decentralized architecture, redundancy storage, collective maintenance, and tamper resistant to ensure the data cannot be changed maliciously. We first design a blockchain-based outsourced service framework for storing data replications in untrusted environment, which contains three key layers, that is, storage layer, verification layer, and blockchain layer. Then, we devise a novel concept of verification peer (VP) for maintaining metadata stored by a form of blockchain, and each of which holds the entire blockchain locally to prevent metadata from being maliciously tampered with. Finally, based on the proposed model, we introduce a collaborative algorithm invoked by VPs to store and verify the metadata of replications. We present a completed analysis and conduct extensive experiments on multi-CSPs scenario. The evaluation results demonstrate that our proposed approach achieves superior performance.

Probabilistic data structure-based community detection and storage scheme in online social networks

With the wide popularity of various social network sites such as — Facebook, Twitter, and Instagram; processing, storage, and analysis of a large volume of data are becoming challenging issues. In general, social networks are assumed to be graphs having nodes for representation of a group of persons in order to explore the relationship between them for Social Network Analysis (SNA). Analysts claim that interconnections in these networks are the reflection of social structure of individual where personalities with common attributes often occupy similar positions. Such similarities are caused by the prospects, opportunities, sensitivities and perceptions created by these similar network positions. Thus, clustering of these individuals is necessary to analyse their common characteristics. However, most of the existing clustering algorithms considered for community detection in SNA have high memory requirements especially in online social networks. So, to mitigate these issues, this paper proposes a novel clustering scheme for community detection for fast access, storage and retrieval of data using Probabilistic Data Structures (PDS). In the proposed scheme, Bloom filter has been used for clustering and Quotient filter has been used for storage and access of cluster nodes. It has been experimentally proved that the proposed scheme provides significant improvement in computational time which is reduced by 64% and 79% respectively in comparison to the linked list and adjacency matrix. Moreover, Quotient filter based storage schema significantly enhances the effectiveness of the proposed scheme over conventional storage methods.

PlantES: A Plant Electrophysiological Multi-Source Data Online Analysis and Sharing Platform

At present, plant electrophysiological data volumes and complexity are increasing rapidly. It causes the demand for efficient management of big data, data sharing among research groups, and fast analysis. In this paper, we proposed PlantES (Plant Electrophysiological Data Sharing), a distributed computing-based prototype system that can be used to store, manage, visualize, analyze, and share plant electrophysiological data. We deliberately designed a storage schema to manage the multi-source plant electrophysiological data by integrating distributed storage systems HDFS and HBase to access all kinds of files efficiently. To improve the online analysis efficiency, parallel computing algorithms on Spark were proposed and implemented, e.g., plant electrical signals extraction method, the adaptive derivative threshold algorithm, and template matching algorithm. The experimental results indicated that Spark efficiently improves the online analysis. Meanwhile, the online visualization and sharing of multiple types of data in the web browser were implemented. Our prototype platform provides a solution for web-based sharing and analysis of plant electrophysiological multi-source data and improves the comprehension of plant electrical signals from a systemic perspective.

Vector Spatial Big Data Storage and Optimized Query Based on the Multi-Level Hilbert Grid Index in HBase

Faced with the rapid growth of vector data and the urgent requirement of low-latency query, it has become an important and timely challenge to effectively achieve the scalable storage and efficient access of vector big data. However, a systematic method is rarely seen for vector polygon data storage and query taking spatial locality into account in the storage schema, index construction and query optimization. In the paper, we focus on the storage and topological query of vector polygon geometry data in HBase, and the rowkey in the HBase table is the concatenation of the Hilbert value of the grid cell to which the center of the object entity’s MBR belongs, the layer identifier and the order code. Then, a new multi-level grid index structure, termed Q-HBML, that incorporates the grid-object spatial relationship and a new Hilbert hierarchical code into the multi-level grid, is proposed for improving the spatial query efficiency. Finally, based on the Q-HBML index, two query optimization strategies and an optimized topological query algorithm, ML-OTQ, are presented to optimize the topological query process and enhance the topological query efficiency. Through four groups of comparative experiments, it has been proven that our approach supports better performance.

A Three-Layer Privacy Preserving Cloud Storage Scheme Based on Computational Intelligence in Fog Computing

Recent years witness the development of cloud computing technology. With the explosive growth of unstructured data, cloud storage technology gets more attention and better development. However, in current storage schema, user's data is totally stored in cloud servers. In other words, users lose their right of control on data and face privacy leakage risk. Traditional privacy protection schemes are usually based on encryption technology, but these kinds of methods cannot effectively resist attack from the inside of cloud server. In order to solve this problem, we propose a three-layer storage framework based on fog computing. The proposed framework can both take full advantage of cloud storage and protect the privacy of data. Besides, Hash-Solomon code algorithm is designed to divide data into different parts. Then, we can put a small part of data in local machine and fog server in order to protect the privacy. Moreover, based on computational intelligence, this algorithm can compute the distribution proportion stored in cloud, fog, and local machine, respectively. Through the theoretical safety analysis and experimental evaluation, the feasibility of our scheme has been validated, which is really a powerful supplement to existing cloud storage scheme.

HBase storage schemas for massive spatial vector data

With the development of Geographic Information System (GIS), the storage requirement of spatial vector data is increasing dramatically. Nowadays, designing an efficient storage schema for massive spatial vector data becomes a key step for GIS. Cloud computing with NoSQL, such as HBase, can provide massive high-concurrent and scalable service for storage of spatial vector data. However, storage schemas in NoSQL for spatial vector data can be rarely seen. In this paper, two HBase storage schemas for spatial vector data are proposed. One is the storage schema with rowkeys based on Z curve, Z schema, and the other is the storage schema with rowkeys based on geometry objects identifiers, ID schema. In our experiments, the region query efficiency of the two storage schemas is tested on the cloud framework built by us. Different order Z curve and different query ranges are involved in the experiments. Experimental results show, for both schemas, the increase of query range leads to the growth of response time. More importantly, response time of Z schema is about one-fifth as long as that of ID schema in all cases. It can be seen that Z schema is a better solution for storing spatial vector data in HBase.

A multi-version document storage strategy based on chord in large-scale real time collaborative environments

In the research field of CSCW, the issue of data management is given little attention compared with other issues. Previous studies mainly focus on how to produce multiple versions. As for the large-scale real time collaborative environment, new storage strategy must be devised because: 1) There are many versions due to the multiple conflict operations, which should be saved to reflect the different intentions of the workers; 2) The new-comer may want to retrieve some old versions to catch up with the work process of the team. The paper devises a multi-version document storage model according to the characteristics of the large-scale real time collaborative environment. This strategy makes full use of the participating sites as the expandable storage resources, based on chord ring as the underlying storage structure and integrates the two storage modes: full storage and incremental storage. Besides, this strategy supports the dynamic change of the chord structure. The strategy realises the document access flexibility and reliability in large-scale collaborative environment, and ensures the storage load balancing between storage nodes. The storage schema is realised in the Co-AutoCAD system and experiments analysis is given to show the effectiveness of the schema.

A multi-version document storage strategy based on chord in large-scale real time collaborative environments

In the research field of CSCW, the issue of data management is given little attention compared with other issues. Previous studies mainly focus on how to produce multiple versions. As for the large-scale real time collaborative environment, new storage strategy must be devised because: 1) There are many versions due to the multiple conflict operations, which should be saved to reflect the different intentions of the workers; 2) The new-comer may want to retrieve some old versions to catch up with the work process of the team. The paper devises a multi-version document storage model according to the characteristics of the large-scale real time collaborative environment. This strategy makes full use of the participating sites as the expandable storage resources, based on chord ring as the underlying storage structure and integrates the two storage modes: full storage and incremental storage. Besides, this strategy supports the dynamic change of the chord structure. The strategy realises the document access flexibility and reliability in large-scale collaborative environment, and ensures the storage load balancing between storage nodes. The storage schema is realised in the Co-AutoCAD system and experiments analysis is given to show the effectiveness of the schema.

A SPARQL query processing system using map-phase-multi join for big data in clouds

Big data refers to large datasets which are huge, complex and hard to be stored and analysed by traditional data processing tools. Linked data is one of the approaches to deal with big data which are stored and processed in TripleStore. For huge dataset, TripleStore requires more scalable techniques. 'MapReduce' programming model is the most representative of cloud technology. There are several approaches using MapReduce to serve SPARQL query but still exhibit unacceptable performance in complex queries. In this paper, we propose a map-phase-multi-join algorithm for processing SPARQL queries. Using multi-join, job initialisation time is reduced by avoiding iterative of MapReduce jobs. Furthermore, map-phase join can save bandwidth by preventing join-less data to be transferred among computing nodes. We also design a storage schema and a join-order rule which enhance the performance of our system. The evaluation results show that our system outperforms traditional join approaches in most queries.