Large-scale Data Management Research Articles

Database Construction and Application of Large-scale Geospatial Data

Abstract. Geospatial data refers to data that related to spatial location. It is the abstract representation of objects, events, or other characteristic information located on or near the surface of the Earth. It characterizes attribute features, spatial features, and associated information of geospatial features. Geospatial data commonly include various data types such as point, polyline, polygon, and so on. Geospatial database is the basic method to store and manage geospatial data. It is also the important component of geographic information systems (GIS). Geospatial database generally stores data in a specific structure in a medium, which can achieve efficient management of geospatial data. Users can add, modify, and delete data efficiently. Furthermore, query retrieval and statistical analysis can be quickly performed based on geospatial database. In this paper, distributed geospatial database is constructed by using database partition tables storage technology. Meanwhile, database construction software is developed to achieve the construction, management, and application of large-scale geospatial data. It can improve data import, query and retrieval efficiency significantly. The database application service system can directly connect to the database for displaying statistical analysis results. At the same time, geospatial data are processed to vector tiles, which stored in the tile database. Furthermore, standard online service is then published, which can be used for database management and application service system. It can be used for natural resource management work such as basic surveying, survey and monitoring, and so on.

Optimization and performance improvement of distributed data storage in hybrid storage systems

With the rapid development of information technology, the storage and processing of massive data has become one of the important challenges facing the current computing field. Traditional centralized storage systems have been unable to meet the needs of big data applications, while distributed storage systems have become the infrastructure of modern data centers because of their high scalability and high availability. However, in practical applications, a single distributed storage model is often difficult to balance cost-effectiveness and performance requirements. Therefore, this paper proposes a hybrid storage system model combining local storage and cloud storage resources and discusses a series of optimization strategies for this model. The focus of the research is to improve the overall performance of the system through the design of intelligent data sharding, redundancy and fault tolerance mechanisms, and the application of effective load balancing technology. The effectiveness and superiority of the proposed method are verified by testing and analyzing several typical application scenarios on the experimental platform. The experimental results show that the optimized hybrid storage system not only significantly improves the speed of data access, but also effectively reduces the storage cost, demonstrating its potential in future large-scale data management.

A Comprehensive Survey of Artificial Intelligence and Machine Learning Application in Healthcare

The ongoing technological transformation, driven by rapid advancements in machine learning (ML), holds the potential to profoundly impact our lives and redefine the essence of humanity. Recent years have witnessed significant progress in ML, fuelled by the widespread adoption of artificial intelligence (AI) due to machines' capacity for large-scale data processing and management. The intersection of ML and healthcare encompasses both computer science along with medical science. Within medical science, ML techniques have enabled the examination of complex medical data, marking a significant advancement. In the healthcare industry, ML functions as the cognitive and knowledge counterpart of healthcare professionals. This paper explores the benefits of ML-based solutions and their applications in healthcare, emphasizing their role in automating data analysis for patients' health records along with making predictions per the extracted insights. The integration of ML in healthcare promises transformative outcomes, shaping the future of medical diagnostics and decision-making.

Innovative Approaches to Full-Text Search with Solr and Lucene

Full-text search engines help efficiently process large volumes of textual material and provide appropriate results. Apache Solr and Apache Lucene are popular full-text search tools for indexing and querying huge datasets. This research study examines Solr and Lucene's strengths, weaknesses, and unique methods for improving full-text search efficiency and accuracy. Apache Lucene, the fundamental full-text search framework, has extensive indexing and querying features. Developers may tailor the search process using its flexible and extendable framework. Advanced indexing algorithms like inverted indices and tokenization underpin Lucene's search capabilities. However, complicated query needs and efficient large-scale data management remain problems. Solr, founded on Lucene, adds faceting, distributed searching, and rich text analysis to its search engine. Enterprise applications may use Solr's high availability, fault tolerance, and large-scale deployments. Solr has performance tuning and setup complexity issues despite these benefits. This study explores novel solutions to these issues and improves full-text search. Advanced tokenization and normalization may improve indexing tactics. Machine learning algorithms increase search relevancy, providing more accurate and contextual results. Query processing optimization is another invention. Caching, query rewriting, and parallel processing may minimize query latency and boost throughput. GPUs are also used to improve query execution. The article also discusses integrating Solr and Lucene with big data platforms and cloud services. Distributed computing frameworks and cloud storage may improve scalability and real-time search. How Solr and Lucene may incorporate AI and NLP to improve search accuracy and user experience is also investigated.

Design and Application of Legal Information Systems Based on Big Data Technology

China's exploration of the legal application of big data is still far from thorough enough. This article proposes a universal suitable for hierarchical big data storage systems, which can quickly add new cache policies when needed and provide cache scheduling strategies. The neural training algorithm used in legal information systems implements a complete parallel computing framework for large-scale neural network training, supporting distributed storage and management of large-scale sample data. The experimental results show that the framework has good scalability and fault tolerance, and can quickly train legal information systems, improving their efficiency and response speed. This provides new ideas and methods for the design and development of legal information systems.

Advances in Molecular Breeding of Forage Crops: Technologies, Applications and Prospects

Molecular breeding has revolutionized the improvement of forage crops by offering precise tools to enhance the yield, quality, and environmental resilience. This review provides a comprehensive overview of the current technologies, applications, and future directions in the field of forage crop molecular breeding. Technological advancements in the field, including Quantitative Trait Loci (QTL) mapping, Genome-Wide Association Studies (GWASs), genomic selection (GS), and genome-editing tools such as CRISPR-Cas9, have significantly advanced the identification and incorporation of beneficial traits into forage species. These approaches have dramatically shortened the breeding cycles and increased the efficiency of developing cultivars with improved yield, disease resistance, stress tolerance, and nutritional profiles. The implementation of these technologies has led to notable successes, as demonstrated by case studies on various forage crops, showcasing enhanced forage quality and adaptability to challenging environmental conditions. Furthermore, the integration of high-throughput phenotyping with advanced bioinformatics tools has streamlined the management of large-scale genomic data, facilitating more precise selection and breeding decisions. Looking ahead, this review explores the potential of emerging technologies, such as the application of artificial intelligence in predictive breeding, along with the associated ethical and regulatory considerations. While we stand to gain benefit from these innovations, the future of molecular breeding in forage crops must also confront the challenges posed by climate change and the imperative of sustainable agricultural practices. This review concludes by emphasizing the transformative impact of molecular breeding on the improvement of forage crop and the critical need for ongoing research and collaboration to fully realize its potential.

Research on the Utilization Path of Ideological and Political Education Data based on the DIKW Model

Aiming at the large scale and dynamic iteration, systematic and comprehensive, and multiple synergistic characteristics of the current ideological and political education data, the "Data-Wisdom" decision-making model based on the ideological and political data of colleges and universities is proposed in order to enrich the decision-making theoretical system of the existing colleges and universities and provide references for ideological and political decision-making. The core element of the "Data-Wisdom" decision-making model in colleges and universities is the organic fusion of data, computation, algorithm, and platform. Through rooting the new concept of big data ideological and political education, improving the ability of participants in ideological and political education to use big data, perfecting the mechanism of big data management, protection, and monitoring, and standardizing the use of data, the model can be used in large-scale data management. application purposes and other suggestions, on the basis of large-scale data collection, the wisdom of big data integration is brought into play in order to solve the problem of realizing the personalized customization of ideological and political education and nurturing services, and realizing the high-quality planning and development of ideological and political education.

Concurrent Data Processing in Microsoft Dynamics CRM Using Python

The realm of Customer Relationship Management (CRM) has seen significant improvements with the integration of automation and data analytics. Python, known for its robust data manipulation libraries, offers a seamless experience for handling data in Microsoft Dynamics CRM. This paper aims to serve as a comprehensive guide on how Python can be employed to perform CRUD operations— Read, Update, Insert—on data in Microsoft Dynamics CRM. We delve into the intricacies of using Python's 'requests' library for API calls and 'concurrent futures' for parallel processing, thereby optimizing data manipulation tasks. The paper also presents a performance evaluation showcasing the efficiency gains achieved through these methods. Furthermore, the paper highlights the challenges associated with large-scale data management in CRM systems and proposes Python-based solutions as a scalable and effective approach. The paper concludes with a discussion on the scope of this approach in the broader context of CRM data analytics and automation. The methods and findings presented herein are expected to be of particular interest to data engineers, software developers, and CRM administrators.

Massive Data Connection Architecture Design of Web Platform Based on Blockchain

In order to improve the universality and security of the data management of the Internet of Things and avoid the single point of failure problem in the traditional management architecture, this paper proposes a massive connection management architecture based on blockchain. The blockchain is reconstructed according to the hierarchical structure of blockchain technology, and with reference to the attribute-based access control mechanism, a data exchange model is designed using gossip mode and TLS networking mode to ensure the safe transmission of data. The peer computer network technology, data signature, and consensus mechanism are used to complete the design of the data exchange network layer and realize the design of the blockchain-based massive connection management architecture, so as to effectively avoid the single point of failure problem in the traditional management architecture and greatly reduce the burden of the management center. The access requests of massive devices in 5G application scenarios were simulated, and the results showed that the blockchain-based access control architecture designed in this article can quickly process received requests, with information leakage below 10%, improving the security and processing performance of data connections, achieving efficient processing and management of large-scale data, and providing reliable guarantees for the development of the Internet era.

Modernizing the Legacy Healthcare System to Decentralize Platform Using Blockchain Technology

The use of blockchain technology is expanding in various industries, including finance, supply chain management, food, energy, IoT, and healthcare. The article aims to address the challenges of complex medical procedures, large-scale medical data management, and cost optimization in the healthcare industry. By employing blockchain technology, the article aims to enhance data security and privacy while ensuring the integrity and efficiency of the healthcare system. This article focuses on the application of blockchain technology in the healthcare system by reviewing the existing literature and proposing multiple workflows for better data management. These workflows were implemented using the Ethereum blockchain platform and involve complex medical procedures such as surgery and clinical trials, as well as managing a large amount of medical data. The feasibility of the proposed system is analyzed in terms of associated costs, and a model-driven engineering approach is used to recover the architecture of traditional healthcare systems. The aim is to provide stakeholders in the healthcare system with better healthcare services and cost optimization. The solution being proposed automates interactions between different parties involved. Smart contracts were created using Solidity language, and their functions were tested using the Remix IDE. This paper illustrates that our smart contract code was designed to avoid common security vulnerabilities and attacks. To test the framework, a prototype of the smart contract was deployed on an Ethereum TESTNET blockchain in a Windows environment. This study found that the proposed approach is both practical and efficient.

Recent Updates on Intelligent System for Talent Management: Does That Become a Helpful System?

The use of information technology in human management today has increased, along with the implementation of intelligent systems that generally use artificial intelligence (AI) in their application. One of the areas of human management that has adopted AI is talent management (TM). TM is crucial for companies to identify, manage, determine, assess, and recommend talent (in this case, it can be employees) for their company's sustainability. The application of family planning in TM is not as extensive as thought, but this study tries to review the latest research that adapts AI to a very complex TM process. The results of this review are at least 11 articles involved in the use of family planning. These 11 articles certainly discuss one or more processes in TM, such as talent identification, talent matching or mapping, and talent recommendations. Some critical studies in the future are that in practice, AI needs to be widely used, especially to handle large-scale data management (data intelligence), in addition to intelligent system methods, and AI can be used for all processes in TM, proven to be accurate, efficient, safe, and fast in practice.

Igneous: Distributed dense 3D segmentation meshing, neuron skeletonization, and hierarchical downsampling.

Three-dimensional electron microscopy images of brain tissue and their dense segmentations are now petascale and growing. These volumes require the mass production of dense segmentation-derived neuron skeletons, multi-resolution meshes, image hierarchies (for both modalities) for visualization and analysis, and tools to manage the large amount of data. However, open tools for large-scale meshing, skeletonization, and data management have been missing. Igneous is a Python-based distributed computing framework that enables economical meshing, skeletonization, image hierarchy creation, and data management using cloud or cluster computing that has been proven to scale horizontally. We sketch Igneous's computing framework, show how to use it, and characterize its performance and data storage.

Scalable training of graph convolutional neural networks for fast and accurate predictions of HOMO-LUMO gap in molecules

Graph Convolutional Neural Network (GCNN) is a popular class of deep learning (DL) models in material science to predict material properties from the graph representation of molecular structures. Training an accurate and comprehensive GCNN surrogate for molecular design requires large-scale graph datasets and is usually a time-consuming process. Recent advances in GPUs and distributed computing open a path to reduce the computational cost for GCNN training effectively. However, efficient utilization of high performance computing (HPC) resources for training requires simultaneously optimizing large-scale data management and scalable stochastic batched optimization techniques. In this work, we focus on building GCNN models on HPC systems to predict material properties of millions of molecules. We use HydraGNN, our in-house library for large-scale GCNN training, leveraging distributed data parallelism in PyTorch. We use ADIOS, a high-performance data management framework for efficient storage and reading of large molecular graph data. We perform parallel training on two open-source large-scale graph datasets to build a GCNN predictor for an important quantum property known as the HOMO-LUMO gap. We measure the scalability, accuracy, and convergence of our approach on two DOE supercomputers: the Summit supercomputer at the Oak Ridge Leadership Computing Facility (OLCF) and the Perlmutter system at the National Energy Research Scientific Computing Center (NERSC). We present our experimental results with HydraGNN showing (i) reduction of data loading time up to 4.2 times compared with a conventional method and (ii) linear scaling performance for training up to 1024 GPUs on both Summit and Perlmutter.

Qanaat

Today's large-scale data management systems need to address distributed applications' confidentiality and scalability requirements among a set of collaborative enterprises. This paper presents Qanaat , a scalable multi-enterprise permissioned blockchain system that guarantees the confidentiality of enterprises in collaboration workflows. Qanaat presents data collections that enable any subset of enterprises involved in a collaboration workflow to keep their collaboration private from other enterprises. A transaction ordering scheme is also presented to enforce only the necessary and sufficient constraints on transaction order to guarantee data consistency. Furthermore, Qanaat supports data consistency across collaboration workflows where an enterprise can participate in different collaboration workflows with different sets of enterprises. Finally, Qanaat presents a suite of consensus protocols to support intra-shard and cross-shard transactions within or across enterprises.

Cross-Domain Transfer Learning for Demand Forecasting: Using Social Media Sentiment from Related Industries

This study examines various window-based techniques, including time-based, size-based, and hybrid approaches, and evaluates their effectiveness in improving extract performance. Through extensive analysis and empirical testing, we demonstrate that window-based strategies can significantly reduce processing time and resource utilization while maintaining data consistency and integrity. This research paper investigates the application of window-based refresh strategies to enhance the performance of data extracts in large-scale data management systems. Traditional extract, transform, load (ETL) processes often struggle with the increasing volume and velocity of data in modern environments. Window-based refresh strategies offer a promising solution by focusing on specific subsets of data during each refresh cycle. This paper shall be devoted to assessing the efficiency of window-based refresh strategies related to the issues described above. The primary research goals are: Propose a general framework with which to apply window-based refresh strategies during the data extract process. Assess the performance benefits derived from applying different types of approaches based on window-based forms as opposed to conventional full and incremental extracts.

Linked Data Fusion Based on Similarity Calculation and K-Nearest Neighbor

The development of semantic web technology supports the continuous development of linked data and its applications. In order to make effective use of the growing linked data on the web, multi-source data should be fused, which is a key step in multi-source large-scale data analysis and management. Currently, merging these data can be difficult, because various sources usually provide multiple conflict descriptions for entities in the same real world. To complete the fusion, we need to solve the problem of object conflict. This paper proposes a linked data fusion method based on similarity calculation and k-nearest neighbor. This method has two contributions. Firstly, a similarity calculation method of linked data is proposed, which can effectively integrate URI nodes and blank nodes in linked data; Secondly, a literal type node fusion strategy based on k-nearest neighbor classification method is proposed, which realizes the automation of fusion and has the independence of data source. The results show that compared with other methods, this method can improve the conciseness and consistency and precision by up to 12.9%, 30.6% and 12%.

Network biology and artificial intelligence drive the understanding of the multidrug resistance phenotype in cancer.

Globally with over 10 million deaths per year, cancer is the most transversal disease across countries, cultures, and ethnicities, affecting both developed and developing regions. Tumorigenesis is dynamically altered by distinct events and can be lethal when untreated. Despite the innovative therapeutics available, multidrug resistance (MDR) to chemotherapy remains the major hindrance to the success of cancer therapy. The multiple mechanisms by which cancer cells evade cell death are diverse, indicating that MDR involves complex interconnected biological networks. Molecular profiling is currently able to stratify cancer into its distinct subtypes and help identify the best therapeutics, leading to "translational systems medicine". Highly specialized methodologies are generating a large amount of "omics" data - including epigenetics, genomics, transcriptomics, proteomics, metabolomics, as well as pharmacogenomics. Many of the resulting databases store data in non-standard formats, which need to be converted, interpreted, and merged into readable formats. The latest development of artificial intelligence (AI) methodologies and tools, coupled with advancements in large-scale data management and powerful graphic processing computing units, potentiate the integration of these large data sources into relevant biological networks, which will enhance our understanding of cancer MDR. In this review, we revisit common MDR mechanisms and compile a list of the most relevant "omics" public databases. We highlight examples of AI methods that are now decisively contributing to clear advances in cancer research, such as identification of new drugs from large databases and prediction of relevant drug, target, and system properties. An overview of several freely available "ready-to-use" algorithms is also provided. The described molecular scale AI algorithms and tools will undoubtedly guide important improvements in efficiency and efficacy of traditional methods of cancer diagnostics and treatment.

INCloud: integrated neuroimaging cloud for data collection, management, analysis and clinical translations

BackgroundNeuroimaging techniques provide rich and accurate measures of brain structure and function, and have become one of the most popular methods in mental health and neuroscience research. Rapidly growing neuroimaging...

Mineral quantification at deposit scale using drill-core hyperspectral data: A case study in the Iberian Pyrite Belt

Drill-core analysis is paramount for the characterization of deposits in mineral exploration. Over the past years, the use of hyperspectral (HS) sensors has rapidly increased to improve the reliability and efficiency of core logging. However, scanning drill-core samples of an entire mineral deposit entails several complex challenges, from transport logistics to large scale data management and analysis. Hence, academic studies on new applications of drill-core HS data at a mineral deposit scale remain rare.We present a semi-automated workflow for large scale interpretation of HS data, founded on a novel approach of mineral mapping based on a supervised dictionary learning technique. This approach exploits the complementary information from scanning electron microscopy based automated mineralogy and hyperspectral imaging techniques for estimating mineral quantities along all boreholes. We propose that it is effectively possible to propagate the mineral quantification to the entire borehole from small samples with high resolution mineralogical information strategically selected throughout the deposit.We showcase this approach on data acquired in the Elvira shale-hosted volcanogenic massive sulphide (VMS) deposit located at the Iberian Pyrite Belt (IPB), where 7000 m of drill-core were acquired along 80 boreholes. Resulting maps provide insights on the controls on the mineral assemblages and chemical composition of specific minerals across the whole volume at several spatial scales, from large scale variations within apparently homogeneous black shales to small scale mineral composition variations, of potential use as vectors towards mineralization. This approach adds value to the core data, allowing for a better understanding of the geological setting of the Elvira deposit and providing valuable insights for future exploration targeting in the region.This approach based on machine learning can easily be transposed to different ore deposits with a limited input from a geologist.

Improving Performance of Data Extracts Using Window-Based Refresh Strategies

This research paper investigates the application of window-based refresh strategies to enhance the performance of data extracts in large-scale data management systems. Traditional extract, transform, load (ETL) processes often struggle with the increasing volume and velocity of data in modern environments. Window-based refresh strategies offer a promising solution by focusing on specific subsets of data during each refresh cycle. This study examines various window-based techniques, including time-based, size-based, and hybrid approaches, and evaluates their effectiveness in improving extract performance. Through extensive analysis and empirical testing, we demonstrate that window-based strategies can significantly reduce processing time and resource utilization while maintaining data consistency and integrity. The paper also explores optimization techniques, challenges, and future research directions in this field.