Volume Of Stored Data Research Articles

Organizing, Transmitting, and Accessing Vector Data in DGGS Based on Progressive Tile Structure

ABSTRACTIn order to solve the efficiency problem of massive multiresolution vector data organization in DGGS field, we propose a vector data organization method based on progressive tile structure in DGGS. The method is a combination of DGGS with tile structure, which are traditionally used in the field of GIS. That structure can support tile scheduling and express vector objects asymptotically in the inner pairs of tiles, which reduces the amount of data transfer and improves the access and construction efficiency of vector objects while guaranteeing the efficiency of data scheduling. To validate the above design, we first resampled the global 1:100,000 vector data containing multiple topological types to multiresolution vector data with a data resolution of 1:559,000–1:35,000, the data volume is 22,312 MB after converting to tile structure. Then we carried out three experiments: data volume, data construction, and data access. The results of the experiments show that compared with the organization method based on the traditional tile‐pyramid structure, the method is able to achieve higher construction efficiency and access efficiency, as well as the smaller data storage volume. Therefore, we believe that this method provides an efficient method for the organization of massive multiresolution vector data based on DGGS and also provides an idea for the organization of massive multiresolution data in the field of GIS.

A comparative study of feature selection and feature extraction methods for financial distress identification

Financial distress identification remains an essential topic in the scientific literature due to its importance for society and the economy. The advancements in information technology and the escalating volume of stored data have led to the emergence of financial distress that transcends the realm of financial statements and its’ indicators (ratios). The feature space could be expanded by incorporating new perspectives on feature data categories such as macroeconomics, sectors, social, board, management, judicial incident, etc. However, the increased dimensionality results in sparse data and overfitted models. This study proposes a new approach for efficient financial distress classification assessment by combining dimensionality reduction and machine learning techniques. The proposed framework aims to identify a subset of features leading to the minimization of the loss function describing the financial distress in an enterprise. During the study, 15 dimensionality reduction techniques with different numbers of features and 17 machine-learning models were compared. Overall, 1,432 experiments were performed using Lithuanian enterprise data covering the period from 2015 to 2022. Results revealed that the artificial neural network (ANN) model with 30 ranked features identified using the Random Forest mean decreasing Gini (RF_MDG) feature selection technique provided the highest AUC score. Moreover, this study has introduced a novel approach for feature extraction, which could improve financial distress classification models.

DATA FINDING, SHARING AND DUPLICATION REMOVAL IN THE CLOUD

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

CLOUD BASED DUPLICATION REMOVAL SYSTEM

Deduplication involves eliminating duplicate or redundant data to reduce stored data volume, commonly used in data backup, network optimization, and storage management. However, traditional deduplication methods have limitations with encrypted data and security. The primary objective of this project is to develop new distributed deduplication systems that offer increased reliability. In these systems, data chunks are distributed across the Hadoop Distributed File System (HDFS), and a robust key management system is utilized to ensure secure deduplication with slave nodes. Instead of having multiple copies of the same content, deduplication removes redundant data by retaining only one physical copy and referring other instances to that copy. The granularity of deduplication can vary, ranging from an entire file to a data block. The MD5 and 3DES algorithms are used to enhance the deduplication process. The proposed approach in this project is the Proof of Ownership (POF) of the file. With this method, deduplication can effectively address the issues of reliability and label consistency in HDFS storage systems. The proposed system has successfully reduced the cost and time associated with uploading and downloading data, while also optimizing storage space. Key Words: Cloud computing, data storage, file checksum algorithms, computational infrastructure, duplication.

Query Optimization in Distributed Database Based on Improved Artificial Bee Colony Algorithm

Query optimization is one of the key factors affecting the performance of database systems that aim to enact the query execution plan with minimum cost. Particularly in distributed database systems, due to the multiple copies of the data that are stored in different data nodes, resulting in the dramatic increase in the feasible query execution plans for a query statement. Because of the increasing volume of stored data, the cluster size of distributed databases also increases, resulting in poor performance of current query optimization algorithms. In this case, a dynamic perturbation-based artificial bee colony algorithm is proposed to solve the query optimization problem in distributed database systems. The improved artificial bee colony algorithm improves the global search capability by combining the selection, crossover, and mutation operators of the genetic algorithm to overcome the problem of falling into the local optimal solution easily. At the same time, the dynamic perturbation factor is introduced so that the algorithm parameters can be dynamically varied along with the process of iteration as well as the convergence degree of the whole population to improve the convergence efficiency of the algorithm. Finally, comparative experiments conducted to assess the average execution cost of Top-k query plans generated by the algorithms and the convergence speed of algorithms under the conditions of query statements in six different dimension sets. The results demonstrate that the Top-k query plans generated by the proposed method have a lower execution cost and a faster convergence speed, which can effectively improve the query efficiency. However, this method requires more execution time.

Big Data Analytics for the ATLAS EventIndex Project with Apache Spark

The ATLAS EventIndex was designed to provide a global event catalogue and limited event-level metadata for ATLAS experiment of the Large Hadron Collider (LHC) and their analysis groups and users during Run 2 (2015-2018) and has been running in production since. The LHC Run 3, started in 2022, has seen increased data-taking and simulation production rates, with which the current infrastructure would still cope but may be stretched to its limits by the end of Run 3. A new core storage service is being developed in HBase/Phoenix, and there is work in progress to provide at least the same functionality as the current one for increased data ingestion and search rates and with increasing volumes of stored data. In addition, new tools are being developed for solving the needed access cases within the new storage. This paper describes a new tool using Spark and implemented in Scala for accessing the big data quantities of the EventIndex project stored in HBase/Phoenix. With this tool, we can offer data discovery capabilities at different granularities, providing Spark Dataframes that can be used or refined within the same framework. Data analytic cases of the EventIndex project are implemented, like the search for duplicates of events from the same or different datasets. An algorithm and implementation for the calculation of overlap matrices of events across different datasets are presented. Our approach can be used by other higher-level tools and users, to ease access to the data in a performant and standard way using Spark abstractions. The provided tools decouple data access from the actual data schema, which makes it convenient to hide complexity and possible changes on the backed storage.

Geo-Distance Based 2- Replica Maintaining Algorithm for Ensuring the Reliability forever Even During the Natural Disaster on Cloud Storage System

In today's digitalized and globalized scenario, everyone has moved to cloud computing for storing their information on cloud storage to access their data from anywhere at any time. The most significant feature of cloud storage is its high availability and reliability then it has the capability of reducing management factors as well as incurred lower storage cost compared with some other storing methods, it is most suitable for a high volume of data storage. In order to meet the requirements of high availability and reliability, the system adopts a replication system concept. In replicating systems, the objects are replicated many times, with each copy residing in a different geographical location. Though it is beneficial to the users, it leads to some issues like security, integrity, consistency and hidden storage and maintenance cost, etc. Therefore, it is exposed to a few threats to the Cloud Storage System (CSS) user and the provider as well. So, this research seeks to explore the mechanisms to rectify the above-mentioned issues. Thus, the predecessor of the research work has proposed an algorithm named as 2-Replica Placing (2RP) algorithm which is used to reduce the storage cost, maintenance cost; and maintenance overheads as well as increase the available storage spaces for the providers by placing the data files on two locations based on Geo-Distance. But it fails to address the recovery mechanism when a natural disaster happens because providing reliability with less than 2 replicas is a challenging task for the providers. Thus, the research proposed Geo-distance based 2-Replica Maintaining (2RM) algorithm which is used to consider that issue for ensuring reliability forever even during natural disasters

An Optimized IoT-enabled Big Data Analytics Architecture for Edge-Cloud Computing.

The awareness of edge computing is attaining eminence and is largely acknowledged with the rise of Internet of Things (IoT). Edge-enabled solutions offer efficient computing and control at the network edge to resolve the scalability and latency-related concerns. Though, it comes to be challenging for edge computing to tackle diverse applications of IoT as they produce massive heterogeneous data. The IoT-enabled frameworks for Big Data analytics face numerous challenges in their existing structural design, for instance, the high volume of data storage and processing, data heterogeneity, and processing time among others. Moreover, the existing proposals lack effective parallel data loading and robust mechanisms for handling communication overhead. To address these challenges, we propose an optimized IoT-enabled big data analytics architecture for edge-cloud computing using machine learning. In the proposed scheme, an edge intelligence module is introduced to process and store the big data efficiently at the edges of the network with the integration of cloud technology. The proposed scheme is composed of two layers: IoT-edge and Cloud-processing. The data injection and storage is carried out with an optimized MapReduce parallel algorithm. Optimized Yet Another Resource Negotiator (YARN) is used for efficiently managing the cluster. The proposed data design is experimentally simulated with an authentic dataset using Apache Spark. The comparative analysis is decorated with existing proposals and traditional mechanisms. The results justify the efficiency of our proposed work.

Application of Compressive Sensing in the Presence of Noise for Transient Photometric Events

Compressive sensing is a simultaneous data acquisition and compression technique, which can significantly reduce data bandwidth, data storage volume, and power. We apply this technique for transient photometric events. In this work, we analyze the effect of noise on the detection of these events using compressive sensing (CS). We show numerical results on the impact of source and measurement noise on the reconstruction of transient photometric curves, generated due to gravitational microlensing events. In our work, we define source noise as background noise, or any inherent noise present in the sampling region of interest. For our models, measurement noise is defined as the noise present during data acquisition. These results can be generalized for any transient photometric CS measurements with source noise and CS data acquisition measurement noise. Our results show that the CS measurement matrix properties have an effect on CS reconstruction in the presence of source noise and measurement noise. We provide potential solutions for improving the performance by tuning some of the properties of the measurement matrices. For source noise applications, we show that choosing a measurement matrix with low mutual coherence can lower the amount of error caused due to CS reconstruction. Similarly, for measurement noise addition, we show that by choosing a lower expected value of the binomial measurement matrix, we can lower the amount of error due to CS reconstruction.

An Autonomous Storage Optimization LRM Generation Strategy Based on Cascaded NURBS

Automatic control of intelligent vehicles depends heavily on accurate lane-level location information. The traditional digital map cannot meet the automatic driving requirements, and it is very difficult for lane-level road maps (LRMs) to achieve a good balance between the representation accuracy and data storage volume. To tackle this problem, we obtain the recursive definition of a lane line by analyzing road lanes using the fractal geometry theory. Subsequently, we construct a recursive feedback system and model it. On this basis, an autonomous storage optimization LRM generation strategy based on cascaded NURBS is devised. According to the input data density, this framework generates an LRM with the minimum data storage volume within the error constraint. It is mainly composed of two modules: (1) a random fractal and (2) an adaptive iteration memory optimizer, based on cascaded NURBS error feedback. First, the whole lane is divided into several meta lane line modules. Second, data sampling/interpolation is carried out iteratively according to the NURBS fitting error and symmetry of road geometry. In this way, the storage space used to store lane lines and other information is automatically optimized. Last, a simulation experiment is carried out by using a dataset obtained from a park. The simulation results show that the proposed method can achieve high accuracy and high storage efficiency. Under the constraint of a 0.1 m curve fitting error, the average memory demand of the lane line elements is less than 2.4 bytes/m.

Coal Mine Personnel Safety Monitoring Technology Based on Uncooled Infrared Focal Plane Technology

In an effort to overcome the difficulty of real-time early warning via traditional infrared imaging technology caused by the complex working environment in coal mines, this paper proposes a mine early warning method based on uncooled infrared focal plane technology. The infrared thermal spectrogram of the detected object was visually displayed in a pseudo-color image with high resolution and high sensitivity, which can realize the real-time detection and early warning of personnel safety in modern mines. The multipoint compression correction algorithm based on human visual characteristics divided the response units of all acquisition units into gray intervals according to a threshold value, then the corresponding parameters were set in different intervals, and finally, each interval was compressed using a two-point correction algorithm. The volume of stored data was the sum of the calibration curve and the data from an encode table corrected by a MATLAB simulation, and the number of CPU cycles was run by a CCS 3.3 clock calculation algorithm. The results showed that when the temperature of the blackbody reached 115 °C, the nonuniformity before correction was 6.32%, and the nonuniformity after the multipoint correction of human eyes was 2.99%, which implied that the algorithm proposed in this paper had good denoising ability. The number of CPU cycles occupied by this algorithm was 18,257,363 cycles/frame with a frequency of 29.97 Hz. The sharpness of the compressed infrared images was obviously improved, and the uniformity was better. The method proposed in this paper can meet the need for modern mine personnel search and rescue, equipment supervision and dangerous area detection and other early warning requirements so as to achieve the goal of developing smart mines and ensuring safety in coal mine production.

Conceptual Design of Control System for the Prototype RF-Driven Negative Ion Source at ASIPP

To explore the generation and extraction of negative ions for neutral beam injection, a prototype radio-frequency (RF)–driven negative ion source is designed at the test facility, which is under construction at the Institute of Plasma Physics, Chinese Academy of Sciences (ASIPP). The control system provides beam pulse set up, remote supervision, plant control, timing synchronization, data management, and interlock and protection for the RF negative ion source. It plays an important role in negative ion source operation. The negative ion source prototype is currently in the development phase, involving more than 20 plant units. To match the requirements of control, data acquisition, and protection for different plant units, the plant control loop time is designed within the range of 10 μs to 100 ms, timing synchronization accuracy is 1 μs, the maximum sampling interval for data acquisition is 10 ms, the volume of data storage is tens of terabytes/year, and the interlock and protection response time is designed within the range of 10 μs to 100 ms. This paper describes the conceptual design of the control system for the prototype RF-driven negative ion source at the ASIPP, discusses the system requirements and the specifications for the control system, and shows the present status of system integration.

Translating a Distributed Relational Database to a Document Database

The problem of transferring data from a database of one type to a database of another type is relevant today due to the increase in data volumes and the complexity of tasks solved in various fields of human activity. The existing databases are less and less satisfying the needs of users. New types of databases appear that are more suitable for solving large-scale problems. This article proposes an algorithm for solving the problem of optimizing the document structure of a database based on metadata about the structure of a distributed relational database from which data are transferred. The approach also takes into account information about the structure of the priority database queries. The priority of database queries is user-defined. The system of automatic translation of the database, taking into account these metadata, allows the user to create a distributed document database that is optimal in two parameters: in terms of the volume of stored data and in terms of the execution time of priority database queries.

Optimal Data Placement and Replication Approach for SIoT with Edge

Social networks (SNs) are sources with extreme number of users around the world who are all sharing data like images, audio, and video to their friends using IoT devices. This concept is the so-called Social Internet of Things (SIot). The evolving nature of edge-cloud computing has enabled storage of a large volume of data from various sources, and this task demands an efficient storage procedure. For this kind of large volume of data storage, the usage of data replication using edge with geo-distributed cloud service area is suited to fulfill the user’s expectations with low latency. The major issue is the way to store the data and replicate these large data items optimally and allocate the request from the data center efficiently. For efficient storage of these data, we use edge server, which is part of the cloud server, in this study. Thus, the data are distributed and stored with quick access, which will reduce the latency with response. The proposed data placement approach learns with machine learning (ML) algorithm called radial basis kernel function assisted with support vector machine (RBF-SVM) to classify the data center for storing the user and friend’s data from the SIoT devices. These learning algorithms will be used to predict the workload of the data stored in the data center as either edge or cloud depending on the existing time slots. The data placement with dynamic nature is also optimized using the proposed dynamic graph partitioning (GP) method to meet the individual user’s demand of low latency with minimum costs. This way will keep the SIoT data placement efficient and effective over time. Accordingly, this proposed data placement and replication approach introduces three kinds of innovations compared with the existing data placement approach. (i) Rather than storing the user data in a single cloud, this study uses the edge server closest to the SIoT devices for faster access with reduced response time. (ii) The classification algorithm called RBF-SVM is used to find storage for user for reducing data replication. (iii) Dynamic GP is introduced for data placement with reduced latency and minimum cost to fulfil the dynamic nature of the SN. The simulation result of this approach obtains reduced latency of 130 ms and minimum cost compared with those of the existing data placement approaches. Therefore, our proposed data placement with ML-based learning on edge provides promising results in terms of efficiency, effectiveness, and performance with reduced latency and minimum cost.

Разработка устройств непрерывного контроля технического состояния контактной сети на основе современных технических решений

Представлено автономное малообслуживаемое устройство диагностирования, способное соединяться в единую информационную сеть из множества устройств по модульному принципу для создания распределенной системы диагностики. Описан алгоритм, выполнение которого позволяет установить предотказное состояние контактной сети. Разработаны технические решения, позволяющие достичь неограниченной автономности предлагаемого комплекса как по условиям электропитания, так и по объему хранилища данных. Приведены сведения о технологии обработки получаемых данных для формирования цифровых отпечатков. Описаны схемы установки предлагаемых устройств на проводах контактной сети и на консолях опорно-поддерживающих конструкций. Разработана функциональная схема программно-аппаратного комплекса мониторинга и диагностирования контактной сети.

Dynamic secure multi-keyword ranked search over encrypted cloud data

Due to increasing volume of stored data in organizations, data owners upload their data to cloud servers. Sensitive data has to be encrypted prior to uploading to the cloud server because of security requirements. Therefore, capable of searching over the encrypted cloud data is a very important issue. One of the most important challenges that must be addressed is to search whole documents to find related documents to a query. In this paper, a scheme is suggested to eliminate the need of searching for whole documents. A secure multi-keyword ranked search method over the encrypted cloud data is introduced that provides dynamic operation on the document collection. Also, access control mechanism is considered in the proposed scheme. The vector space model is employed to B+tree index and query creating. Due to generation of a B+tree-based index structure, search time is getting sub-linear. First, the cloud server is assumed to be “Honest-but-Curious”. Second, it is considered that the data are stored on an untrusted cloud server. The Merkle B-tree (MB-tree) is utilized to verify search results in the presence of access control rules when the cloud server is untrusted. Experiments are conducted using the real world dataset to confirm efficiency and precision of the offered scheme.

A novel blockchain protocol for selecting microservices providers and auditing contracts

Software architectures based on containers and microservices are often used to develop and manage large-scale distributed applications. Still, large vertical deployments spanning over multiple cloud and edge infrastructures are cumbersome to negotiate for, as each infrastructure provider is usually unique concerning prices, management strategies and Quality-of-Service (QoS) levels. In this scenario, Service Level Agreement (SLA) contracts are primarily crafted through pre-established templates and clients must trust providers to manage provisioned resources. The present paper proposes Dawn, a novel blockchain protocol for selecting microservice providers and auditing contracts. The protocol exploits the distributed and verifiable storage of a blockchain, as well as its decentralized consensus to enable contracts establishments in unreliable environments. Besides providing a formal definition of the protocol, this work discusses the possible threats to the correct operation of the network, originated by tenants and providers. We show that Dawn is secure under the evaluated terms, that it can efficiently help the contract establishment process as well as it guarantees a functional systematic way of auditing through monitoring. Finally, we studied both best and worst case scenarios regarding the number of issued messages, stored data volume and network traffic to execute Dawn with different numbers of clients and providers.

Discrimination of neutrons and gamma-rays in plastic scintillator based on falling-edge percentage slope method

With the development of neutron detection technology, the demand for real-time and fast neutron gamma discrimination has become of increasing importance. In this study, neutron and gamma pulse signals measured by an EJ299-33A plastic scintillator detector were analyzed via Matlab off-line. This study compares the effects of seven fast neutron and gamma (n/γ) discrimination methods; (i) charge comparison method, (ii) time-domain oriented method, (iii) amplitude oriented method, (iv) pulse time–amplitude two types of width method, (v) angle difference method, (vi) falling-edge numerical integral method, and, lastly, (vii) falling-edge percentage slope method proposed in this study. We verified that filtering and amplitude normalization improve the discrimination quality but consume much time that accounts for 99.6% of the total discrimination time. The proposed method was compared with other six methods in two aspects: (i) the figure of merit (FoM), (ii) the discrimination time. The FoM value and time consumption of the falling-edge percentage slope method with amplitude normalization are 1.26 and 0.0621s, respectively, while an unnormalized FoM value is 1.02. Experimental results showed that compared with all aforementioned methods, the FOM value of our proposed method is the second high followed the charge comparison method, retaining a short discrimination time which is only half of that of the charge comparison method. This relatively short time consuming is on account of its low volume of data storage and calculation. Generally, the falling-edge percentage slope method has a good discrimination performance while consumes little time, which is capable to be implemented on real-time discrimination applications.

A Hybrid Method for Energy Efficient Data Storage in the Internet Of Things

In the Internet of Things (IoT), to increase the volume of stored data, distributed systems have replaced by centralized systems, and high volume data is divided into smaller sections and each section is stored in a data registration center. The design of the system should be such that even with a number of unavailable data centers, all information is still retrievable, therefore it is necessary to store multiple copies of the information in the system so that the initial information is not lost, despite the loss of part of the data registration center. In the distribution data storage, node energy balance and reduce the cost of access to data is a major problem between sensor nodes. Sensor node clustering is a viable solution to solve this problem. In this paper, a distributed data storage method using PSO and the K-means clustering mechanism organized by the binary decision tree C4.5 in the Internet of Things environment was proposed. As the simulation results show, the proposed method has been able to reach increasing availability, decreasing communication costs, and decreasing energy usage.

Distributed Dual Coordinate Ascent in General Tree Networks and Communication Network Effect on Synchronous Machine Learning

Due to the big size of data and limited data storage volume of a single computer or a single server, data are often stored in a distributed manner. Thus, performing large-scale machine learning operations with the distributed datasets through communication networks is often required. In this paper, we study the convergence rate of the distributed dual coordinate ascent for distributed machine learning problems in a general tree-structured network. Since a tree network model can be understood as the generalization of a star network, our algorithm can be thought of as the generalization of the distributed dual coordinate ascent in a star network. We provide the convergence rate of the distributed dual coordinate ascent over a general tree network in a recursive manner and analyze the network effect on the convergence rate. Secondly, by considering network communication delays, we optimize the distributed dual coordinate ascent algorithm to maximize its convergence speed. From our analytical result, we can choose the optimal number of local iterations depending on the communication delay severity to achieve the fastest convergence speed. In numerical experiments, we consider machine learning scenarios over communication networks, where local workers cannot directly reach to a central node due to constraints in communication, and demonstrate that the usability of our distributed dual coordinate ascent algorithm in tree networks.