Massive Data Management Research Articles

The Intelligent Data Ecosystem: Uniting AI and Data Integration to Revolutionize Data Science

The Rise of Big Data: Today, the scale and domain of massive data management has prolonged to an unmatched state. In this paper, we will examine the process of data science developed over the recent past and argue that artificial intelligence (AI) and data integration converge to create intelligent data ecosystem reaching new heights. This paper combines AI's strength to automate analytics and data integration, which is able to harmonize different data sources and shows how businesses can benefit from a unified approach for innovation, operational efficiency, decision-making. We also cover some of the key use cases in the industry, and ways these technologies are gaining momentum and reshaping the future of data science.

A New Reversible Data Hiding Method Using a Proportional Relation between Peak Signal-to-Noise Ratio and Embedding Capacity on Convolutional Neural Network

In the rapidly evolving fields of artificial intelligence and various industries, the secure processing and management of massive data have become paramount. This paper introduces an innovative reversible data hiding (RDH) method that leverages a Convolutional Neural Network (CNN)-based predictor to generate a predicted image from a given cover image. The secret data are ingeniously embedded within the differences in pixel values between the cover and predicted images. Our experimental analysis reveals a notable reduction in image distortion with increasing secret data size, showcasing the method’s potential for diverse applications. The unique aspect of our approach lies in the proportional relation between the Peak Signal-to-Noise Ratio (PSNR) and Embedding Capacity, highlighting its efficacy and efficiency in reversible data hiding.

Lean resource management and reliable interaction for a low-carbon-oriented new grid

The lean resource management and reliable interaction of massive data are important components of a low-carbon-oriented new grid. However, with a high proportion of distributed low-carbon resources connected to a new grid, issues such as data anomalies, data redundancy, and missing data lead to inefficient resource management and unreliable interaction, affecting the accuracy of power grid decision-making, as well as the effectiveness of emission reduction and carbon reduction. Therefore, this paper proposes a lean resource management and reliable interaction framework of a middle platform based on distributed data governance. On this basis, a distributed data governance approach for the lean resource management method of the middle platform in the low-carbon new grid is proposed, which realizes anomalous data cleaning and missing data filling. Then, a data storage and traceability method for reliable interaction is proposed, which prevents important data from being illegally tampered with in the interaction process. The simulation results demonstrate that the proposed algorithm significantly enhances efficiency, reliability, and accuracy in anomalous data cleaning and filling, as well as data traceability.

Use of Elasticsearch-based business intelligence tools for integration and visualization of biological data.

The emergence of massive datasets exploring the multiple levels of molecular biology has made their analysis and knowledge transfer more complex. Flexible tools to manage big biological datasets could be of great help for standardizing the usage of developed data visualizations and integration methods. Business intelligence (BI) tools have been used in many fields as exploratory tools. They have numerous connectors to link numerous data repositories with a unified graphic interface, offering an overview of data and facilitating interpretation for decision makers. BI tools could be a flexible and user-friendly way of handling molecular biological data with interactive visualizations. However, it is rather uncommon to see such tools used for the exploration of massive and complex datasets in biological fields. We believe that two main obstacles could be the reason. Firstly, we posit that the way to import data into BI tools are not compatible with biological databases. Secondly, BI tools may not be adapted to certain particularities of complex biological data, namely, the size, the variability of datasets and the availability of specialized visualizations. This paper highlights the use of five BI tools (Elastic Kibana, Siren Investigate, Microsoft Power BI, Salesforce Tableau and Apache Superset) onto which the massive data management repository engine called Elasticsearch is compatible. Four case studies will be discussed in which these BI tools were applied on biological datasets with different characteristics. We conclude that the performance of the tools depends on the complexity of the biological questions and the size of the datasets.

Digital Hash Data Encryption for IoT Financial Transactions using Blockchain Security in the Cloud

Blockchain security via the Internet of Things (IoT) will reshape the decision-making function of the data-driven incumbent smart enterprise, providing the vision of the connected world of things. Enterprise IoT development of devices, personnel, and systems in such a way that they may connect and communicate with each other through the Internet. Blockchain is an enterprise financial transaction, and its digital network is distributed transaction ledger. Today, enterprises need the massive global data management and rapid trading volume to keep things going and growing. It creates enterprise business challenges of different types of security, transparency, and complexity of the problem. Enterprise architecture offers several advantages for the thief to obtain a specific user account, application, and access to the device. This is, will doesn't be to provide the necessities of security. The proposed Digital Hash Data Encryption (DHDE) is used to secure the transaction data-based embedded system people and blockchain. Blockchain and IoT technology integration may bring numerous benefits to mention. Therefore, the proposed DHDE algorithm comprehensively discusses the blockchain technology integration system. The proposed DHDE algorithm encrypts the transaction data for an unauthorized person who cannot access the enterprise transaction data based on embedded system people and blockchain.

W-Hilbert: A W-shaped Hilbert curve and coding method for multiscale geospatial data index

Space-filling curves (SFCs) are commonly used in discrete global grid systems (DGGSs) and applied to efficient indices and queries of massive geospatial data. Although Hilbert curves have the best locality preservation and clustering properties among SFCs, they have two major problems when applied to a multiscale data index or query. (1) Poor clustering in multiscale space reduces the index efficiency, and (2) the complexity of the child-code calculation reduces the query efficiency. We propose a multiscale W-shaped Hilbert curve (W-Hilbert) with its coding and calculation methods to solve these problems. We conducted three comparative experiments with the existing multiscale U-shaped Hilbert curve (U-Hilbert). (1) W-Hilbert demonstrated more satisfactory clustering properties in the multiscale grid space than U-Hilbert. (2) The average child-code query efficiency of W-Hilbert was 2.84 times that of U-Hilbert. (3) For the spatial index and query of multiscale urban building data, the average spatial query efficiency of W-Hilbert was 52.22% higher than that of U-Hilbert. Overall, W-Hilbert is an effective method for massive geospatial data index and management.

Research on Big Data Resource Management and Optimization Based on Cloud Computing Technology

This paper points out the problems of irrational, complex and low utilization rate of massive data storage in most industries at present, analyzes the meaning and characteristics of big data, thus determining the necessity of distribution and management of big data, providing reference ideas for the management of Chinese big data resources, and further promoting the big data resources to play its due role. To provide support for the transformation of science and technology management decision-making, to achieve social and technological progress. On this basis, from different aspects of the mass data management methods and strategies and the significance and efficiency of these methods to achieve, improve the utilization rate of cluster resources, increase the ability of data sharing, a variety of computing frameworks can also share a distributed storage data, finally through the experiment demonstrated the effectiveness and security of the above strategies.

GL-Tree: A Hierarchical Tree Structure for Efficient Retrieval of Massive Geographic Locations.

Location-based application services and location privacy protection solutions are often required for the storage, management, and efficient retrieval of large amounts of geolocation data for specific locations or location intervals. We design a hierarchical tree-like organization structure, GL-Tree, which enables the storage, management, and retrieval of massive location data and satisfies the user's location-hiding requirements. We first use Geohash encoding to convert the two-dimensional geospatial coordinates of locations into one-dimensional strings and construct the GL-Tree based on the Geohash encoding principle. We gradually reduce the location intervals by extending the length of the Geohash code to achieve geospatial grid division and spatial approximation of user locations. The hierarchical tree structure of GL-Tree reflects the correspondence between Geohash codes and geographic intervals. Users and their location relationships are recorded in the leaf nodes at each level of the hierarchical GL-Tree. In top-down order, along the GL-Tree, efficient storage and retrieval of location sets for specified locations and specified intervals can be achieved. We conducted experimental tests on the Gowalla public dataset and compared the performance of the B+ tree, R tree, and GL-Tree in terms of time consumption in three aspects: tree construction, location insertion, and location retrieval, and the results show that GL-Tree has good performance in terms of time consumption.

A Map Tile Data Access Model Based on the Jump Consistent Hash Algorithm

Tiled maps are one of the key GIS technologies used in the development and construction of WebGIS in the era of big data; there is an urgent need for high-performance tile map services hosted on big data GIS platforms. To address the current inefficiency of massive tile map data management and access, this paper proposes a massive tile map data access model that utilizes the jump consistent hash algorithm. Via the uniformity and consistency of a certain seed of a pseudo-random function, the algorithm can generate a storage slot for each tile data efficiently. By recording the slot information in the head of a row key, a uniform distribution of the tiles on the physical cluster nodes is achieved. This effectively solves the problem of hotspotting caused by the monotonicity of tile row keys in the data access process, thereby maximizing the random-access performance of a big data platform and greatly improving concurrent database access. Experiments show that this model can significantly improve the efficiency of tile map data access by more than 39% compared to a direct storage method, thereby confirming the model’s advantages in accessing massive tile map data on a big data GIS platform.

La Mediación Mercantil Ante los Nuevos Retos de la Inteligencia Artificial: Una Visión Comparada Entre los Ordenamientos Jurídicos Español e Italiano

El presente artículo se centra en el análisis de la aplicación y posibilidades de uso que ofrecen las herramientas basadas en inteligencia artificial y gestión de datos en masa (big data) en la tramitación de los procesos de mediación, concretamente los de naturaleza mercantil, desde una comparativa ítalo-española. Para alcanzar este fin el método utilizado se ha centrado en el análisis doctrina y jurisprudencial sobre la materia, con especial referencia al tratamiento de los procedimientos extrajudiciales de resolución de conflictos en la Ley 7/2017 y el Anteproyecto de Ley de Medidas de Eficiencia Procesal del Servicio Público de Justicia (2020), así como a lo dispuesto en los Decretos Leyes italianos núm. 28/2010 y 69/2013. Entre los resultados alcanzados, puede anticiparse que la incorporación instrumental de la inteligencia artificial a la mediación, con las debidas garantías de funcionamiento, puede proporcionar una herramienta más de soporte y ayuda para el mediador, pero sin llegar en ningún caso a sustituirlo o reemplazarlo.

Construction and Optimization of Distributed Electronic Archives System Merging Control Messages

This paper analyzes the whole of the electronic archives management system, describes the basic features and functions of the system, and gives the flow of each functional module of the system. Techniques for implementing important functions in the system are discussed and compared. Using the object-oriented idea, this paper designs the electronic archives management system and gives the hybrid structure model of the system. According to the needs of the electronic office, an official document transmission module is established to realize the electronic transmission of official documents, and the specific implementation process is given. According to the characteristics of archives information management and utilization, SQL Server is used as the background database and the database design is carried out. This paper introduces the full-text retrieval technology that realizes the query and retrieval function of electronic archives and gives how to configure the full-text retrieval service in SQL Server. This paper proposes a K-means-based merging control message clustering algorithm to solve the problem of communication topology changes caused by large-scale, high-mobility merging control messages. Based on the idea of K-means clustering, the calculation method of distance in K-means is improved by combining the distribution of the initial electronic archives message, and the fitness function of the electronic archives is designed by using the information of the electronic archives in the cluster and the average distance to the neighboring electronic archives. According to the maximum fitness value, the cluster head is selected to establish clustering. The file management system is tested and analyzed in a simulated environment. Based on the functional positioning of the system and the business environment, the functional performance and performance of the analysis system are examined and analyzed, and the research and development results are inspected according to the system development goals to check whether the development results of the system have reached the advanced stage. At the functional level, it can automate the management of the core business of the distributed electronic archives management department and support the distributed storage management of massive archive data, which effectively improves the informatization level and development efficiency of archives management.

Implementation of Distributed Operation and Maintenance of Cross-Border e-Commerce Platform Based on Cloud Native Architecture

The traditional architecture of e-commerce data management needs to adapt to the new and more complex environment, which needs to provide massive data management, compatibility of different types of data, and better user experience. Cloud computing technology is a synthesis of parallel, distributed, and grid computing and is one of the future directions of information technology development. In this paper, we use cloud computing data management technology to study the data management of e-commerce. It is hoped that the current e-commerce data management can be improved with the help of cloud computing technology. This paper starts from analyzing the current e-commerce representatives Taobao.com and Jingdong Mall to find out the data characteristics of these two enterprises, analyze the existing data problems, and find the aspects that can be improved. Using this open-source cloud computing implementation solution, we solve the storage problem of large files and unstructured small files in the e-commerce system.

Cloud Computing Cloud Computing in Remote Sensing : High Performance Remote Sensing Data Processing in a Big data Environment

Multi-area and multi-faceted remote sensing (SAR) datasets are widely used due to the increasing demand for accurate and up-to-date information on resources and the environment for regional and global monitoring. In general, the processing of RS data involves a complex multi-step processing sequence that includes several independent processing steps depending on the type of RS application. The processing of RS data for regional disaster and environmental monitoring is recognized as computationally and data demanding.Recently, by combining cloud computing and HPC technology, we propose a method to efficiently solve these problems by searching for a large-scale RS data processing system suitable for various applications. Real-time on-demand service. The ubiquitous, elastic, and high-level transparency of the cloud computing model makes it possible to run massive RS data management and data processing monitoring dynamic environments in any cloud. via the web interface. Hilbert-based data indexing methods are used to optimally query and access RS images, RS data products, and intermediate data. The core of the cloud service provides a parallel file system of large RS data and an interface for accessing RS data from time to time to improve localization of the data. It collects data and optimizes I/O performance. Our experimental analysis demonstrated the effectiveness of our method platform.

Constructing a prior-dependent graph for data clustering and dimension reduction in the edge of AIoT

The Artificial Intelligence Internet of Things (AIoT) is an emerging concept aiming to perceive, understand and connect the ‘intelligent things’ to make the intercommunication of various networks and systems more efficient. A key step in achieving this goal is to carry out high-precision data analysis at the edge and cloud level. Clustering and dimensionality reduction in AIoT can facilitate efficient data management, storage, computing, and transmission of various data-driven AIoT applications. For high-efficiency data clustering and dimensionality reduction, this paper develops a prior-dependent graph (PDG) construction method to model and discover the complex relations of data. With the proper utilization and incorporation of data priors, i.e., (a) element local sparsity; (b) pair-wise symmetry; (c) multi-instance manifold smoothness; and (d) matrix low-rankness, the obtained graph has the characteristics of local sparsity, symmetry, low-rank, and can well reveal the complex multi-instance proximity among data points. The developed PDG model is then applied for two typical data analysis tasks, i.e., unsupervised data clustering and dimensionality reduction. Experimental results on multiple benchmark databases verify that, compared with some existing graph learning models, the PDG model can achieve substantial performance, which can be deployed in edge computing modules to provide efficient solutions for massive data management and applications in AIoT.

DeFiHap

The emergence of Hive greatly facilitates the management of massive data stored in various places. Meanwhile, data scientists face challenges during HiveQL programming - they may not use correct and/or efficient HiveQL statements in their programs; developers may also introduce anti-patterns indeliberately into HiveQL programs, leading to poor performance, low maintainability, and/or program crashes. This paper presents an empirical study on HiveQL programming, in which 38 HiveQL anti-patterns are revealed. We then design and implement DeFiHap, the first tool for automatically detecting and fixing HiveQL anti-patterns. DeFiHap detects HiveQL anti-patterns via analyzing the abstract syntax trees of HiveQL statements and Hive configurations, and generates fix suggestions by rule-based rewriting and performance tuning techniques. The experimental results show that DeFiHap is effective. In particular, DeFiHap detects 25 anti-patterns and generates fix suggestions for 17 of them.

A Blockchain-Assisted Massive IoT Data Collection Intelligent Framework

Due to the vigorous development of wireless communication technology, massive sensors have been gradually connected to the Internet of Things (IoT) and generate a massive quantity of valuable IoT data from large-scale wireless sensor networks (WSNs) controlled by different owners. Massive IoT data need to be collected and circulated among multiple data owners and data users. However, existing data collection frameworks may cause heavy computational overhead or rely on trusted third parties, since sensors have constrained resources. Consequently, massive IoT data are transformed among different parties, causing severe trust and security issues. In this article, we propose a blockchain-assisted massive IoT data collection (MIDC) intelligent framework to support the security, trust and efficiency of massive data collection for large-scale heterogeneous WSNs. In particular, we propose a series of novel technologies for the framework: 1) we design a large-scale heterogeneous WSNs collaborative identity verification protocol to ensure reliable data sources; 2) we build a hierarchical massive data aggregation scheme to collect massive IoT data efficiently and securely; and 3) we depict a blockchain-based massive IoT data management method to construct trust among different parties. Extensive simulation and prototype experimental results prove the effectiveness of our framework.

Data distribution and secure data transmission using IANFIS and MECC in IoT

IoT entities amass a massive quantity of sensing data and transmit it to the cloud for processing as well as reasoning. Such disparate networks raise numerous research questions, say processing, storage, and management of massive data. Though numerous prevailing methodologies are presented, it brings about restricted memory, battery-centered processes, privacy, safety, low processing ability, along with compatibility that directed them to be exposed to disparate attacks, particularly during the data transmission process. To surmount such disadvantages, ‘two’ methods are proposed aimed at IoT data distribution to the cloud utilizing IANFIS in addition to secure transmission of those distributed data utilizing MECC. Initially, the IoT device is authenticated by means of performing registration, login, as well as verification for preventing the device as of unauthorized utilization. Subsequently, the device is accessed to sense the information as of associated objects by utilizing smart sensors; in addition, the communication of disparate IoT sensors is performed by employing the MQTT protocol. Utilizing the IANFIS, this sensed information as of diverse IoT devices is distributed to the specific cloud server. Then by employing the MECC, the data is sent securely to the user. Lastly, the LC is executed by means of HFPGA. To corroborate the proposed system’s effectiveness and correctness, it is compared with existing methodologies and evaluated against relevant metrics.

An improved tile-based scalable distributed management model of massive high-resolution satellite images

The amount of remote sensing (RS) data has increased at an unexpected scale, due to the rapid progress of earth-observation and the growth of satellite RS and sensor technologies. Traditional relational databases attend their limit to meet the needs of high-resolution and large-scale RS Big Data management. As a result, massive RS data management is currently one of the most imperative topics. To address this problem, this paper describes a distributed architecture for big RS data storage based on a unified metadata file, pyramid model, and Hilbert curve for data composition and indexing using NoSQL databases (i.e, Apache Hbase). In this paper, a Hadoop-based framework in AzureInsight cloud platform is designed to manage massive RS data in a parallel and distributed way. Experimental results prove that our method has the potential to overcome the weakness of traditional methods. The proposed model is suitable for massive high-resolution image data management.

Machine Learning Models and Big Data Tools for Evaluating Kidney Acceptance

The rise of on-demand healthcare and the unprecedented growth of electronic health records has given rise to big data opportunities and data analysis using machine learning. The massive and disparate data management using conventional databases is incredibly challenging and expensive to manage. It often requires specialized analytical tools for developing advanced data-driven capabilities and performing data analytics. This paper explores the capability of an open-source framework ‘Apache Spark’ capable of processing large amounts of data on clusters of nodes to analyze Big data and integrate technologies to provide decision support systems in healthcare settings. Next, we propose machine learning models on top of Apache Spark to expedite the decision-making in allocating organs such as kidney selection for the right candidate, thus increasing donor utilization by locating a recipient within the allotted time. The proposed models help in identifying waitlisted candidates willing to accept kidneys that may otherwise be discarded.

Research on seismic technology of civil engineering structure based on GIS system

The development and application of seismic technology in civil engineering structures is of great significance to extend the service life of buildings and improve the overall quality of buildings, so it is necessary to further strengthen the research on it. Geographic Information System (GIS) is a new subject integrating computer science, informatics, geography and other sciences. Because of its rapid and convenient management of massive data, GIS has been widely used in the fields of resource development, environmental protection, urban planning and construction, disaster monitoring and evaluation, etc. GIS is used to manage and analyze the data, and the damage detection module in the system is used to realize the structural damage identification. And use that special thematic analysis function of GIS to display the data on the map in the form of statistical graph, so that users can find the damage position more directly and clearly.