Massive Data Analysis Research Articles

Research on Predicting Stock Market Profitability Changes Based on Machine Learning

The expansion of the market in our country has led to rapid economic growth, and many companies have begun to seek foreign investment for themselves. What investors are given is an invisible but very useful thing - stocks. However, stocks are unpredictable and subject to many influences, making it difficult for investors to predict whether the stock trend will decline or rise. It is uncertain whether investors will make a profit or lose money. The development of artificial intelligence technology has played a crucial role in driving stock model prediction. With the development and gradual maturity of machine learning technology, machine learning models are used for efficient and accurate analysis of massive data in the stock market, thereby achieving data mining in the stock market, analyzing changes in stock returns, and predicting them to improve investor returns.

CAMDA 2023: Finding patterns in urban microbiomes.

The Critical Assessment of Massive Data Analysis (CAMDA) addresses the complexities of harnessing Big Data in life sciences by hosting annual competitions that inspire research groups to develop innovative solutions. In 2023, the Forensic Challenge focused on identifying the city of origin for 365 metagenomic samples collected from public transportation systems and identifying associations between bacterial distribution and other covariates. For microbiome classification, we incorporated both taxonomic and functional annotations as features. To identify the most informative Operational Taxonomic Units, we selected features by fitting negative binomial models. We then implemented supervised models conducting 5-fold cross-validation (CV) with a 4:1 training-to-validation ratio. After variable selection, which reduced the dataset to fewer than 300 OTUs, the Support Vector Classifier achieved the highest F1 score (0.96). When using functional features from MIFASER, the Neural Network model outperformed other models. When considering climatic and demographic variables of the cities, Dirichlet regression over Escherichia, Enterobacter, and Klebsiella bacteria abundances suggests that population increase is indeed associated with a rise in the mean of Escherichia while decreasing temperature is linked to higher proportions of Klebsiella. This study validates microbiome classification using taxonomic features and, to a lesser extent, functional features. It shows that demographic and climatic factors influence urban microbial distribution. A Docker container and a Conda environment are available at the repository: GitHub facilitating broader adoption and validation of these methods by the scientific community.

Using PySpark to accelerate batch data point rotation for paleogeographic reconstruction

ABSTRACT Batch paleogeographic point rotation (BPPR) is a PySpark-based extensible batch data point rotation method that accelerates rotation during paleogeographic reconstruction. Data point rotation is an important part of paleogeographic reconstruction and a significant tool for exploring the co-evolution of Earth and life. However, current point rotation techniques have challenges with processing speeds when handling extensive paleogeographic data. Therefore, this study introduced a parallel-computing framework to construct a BPPR. This method combines PySpark and PyGPlates, which can partition points and compute them simultaneously in multiple threads. The rotation of 232,277 fossil occurrences from the Cretaceous Period in the Paleobiology Database (PBDB) was completed within 26 s. By contrast, an alternative GPlates method completed the same task within 96 s. The proposed method supports CSV, EXCEL, SHP, and other data formats, thereby avoiding possible software switching requirements when using methods associated with GPlates. Using synthetic and real paleontological data as experimental datasets, BPPR proved to be nine times more efficient than GPlates when rotating 900,000 points. This efficiency improvement significantly enhanced data-driven paleogeographic analysis. The parallel strategy employed can be broadly applied to massive data analysis in geoscience.

Additive Hazards Regression Analysis of Massive Interval-Censored Data via Data Splitting

With the rapid development of data acquisition and storage space, massive datasets exhibited with large sample size emerge increasingly and make more advanced statistical tools urgently need. To accommodate such big volume in the analysis, a variety of methods have been proposed in the circumstances of complete or right censored survival data. However, existing development of big data methodology has not attended to interval-censored outcomes, which are ubiquitous in cross-sectional or periodical follow-up studies. In this work, we propose an easily implemented divide-and-combine approach for analyzing massive interval-censored survival data under the additive hazards model. We establish the asymptotic properties of the proposed estimator, including the consistency and asymptotic normality. In addition, the divide-and-combine estimator is shown to be asymptotically equivalent to the full-data-based estimator obtained from analyzing all data together. Simulation studies suggest that, relative to the full-data-based approach, the proposed divide-and-combine approach has desirable advantage in terms of computation time, making it more applicable to large-scale data analysis. An application to a set of interval-censored data also demonstrates the practical utility of the proposed method.

Closha 2.0: a bio-workflow design system for massive genome data analysis on high performance cluster infrastructure

BackgroundThe explosive growth of next-generation sequencing data has resulted in ultra-large-scale datasets and significant computational challenges. As the cost of next-generation sequencing (NGS) has decreased, the amount of genomic data has surged globally. However, the cost and complexity of the computational resources required continue to be substantial barriers to leveraging big data. A promising solution to these computational challenges is cloud computing, which provides researchers with the necessary CPUs, memory, storage, and software tools.ResultsHere, we present Closha 2.0, a cloud computing service that offers a user-friendly platform for analyzing massive genomic datasets. Closha 2.0 is designed to provide a cloud-based environment that enables all genomic researchers, including those with limited or no programming experience, to easily analyze their genomic data. The new 2.0 version of Closha has more user-friendly features than the previous 1.0 version. Firstly, the workbench features a script editor that supports Python, R, and shell script programming, enabling users to write scripts and integrate them into their pipelines. This functionality is particularly useful for downstream analysis. Second, Closha 2.0 runs on containers, which execute each tool in an independent environment. This provides a stable environment and prevents dependency issues and version conflicts among tools. Additionally, users can execute each step of a pipeline individually, allowing them to test applications at each stage and adjust parameters to achieve the desired results. We also updated a high-speed data transmission tool called GBox that facilitates the rapid transfer of large datasets.ConclusionsThe analysis pipelines on Closha 2.0 are reproducible, with all analysis parameters and inputs being permanently recorded. Closha 2.0 simplifies multi-step analysis with drag-and-drop functionality and provides a user-friendly interface for genomic scientists to obtain accurate results from NGS data. Closha 2.0 is freely available at https://www.kobic.re.kr/closha2.

Distributed Least Product Relative Error estimation for semi-parametric multiplicative regression with massive data

Distributed systems have been widely used for massive data analysis, but few studies focus on multiplicative regression models. We consider a communication-efficient surrogate likelihood method using the Least Product Relative Error criterion for semi-parametric multiplicative models on massive datasets. The non-parametric component is efficiently handled via B-spline approximation. We derive the asymptotic properties for both parametric and non-parametric components, while the SCAD and adaptive Lasso penalty functions are developed and their oracle properties for variable selection are validated. Simulation studies and an application to an energy prediction dataset are used to demonstrate the effectiveness and practical utility of the proposed method.

Response-Aided Score-Matching Representative Approaches for Big Data Analysis and Model Selection under Generalized Linear Models

In this paper, we propose an efficient method called the response-aided score-matching representative (RASMR) approach to facilitate massive data model selection and data analysis with generalized linear models (GLMs) and a predetermined data partition due to data localization. Similar to the original score-matching representative (SMR) approach, RASMR constructs an artificial data point, called the representative, for each data block. It then fits a GLM on the representative dataset, which provides not only an efficient approach for massive data analysis but also an ideal solution in response to privacy concerns by avoiding the transfer of sensitive data. By further splitting the data blocks according to the values of the response variables, RASMR can obtain more accurate parameter estimates than SMR. Furthermore, by theoretical justifications and simulation studies, we show that RASMR can be more efficiently utilized for model selection and variable selection for a massive dataset by approximating the Akaike information criterion (AIC) and the aggregated prediction errors for cross-validation, which are commonly used for choosing the most appropriate statistical model and drawing reliable conclusions. We also apply the proposed RASMR approach to the airline on-time performance data, which consists of 371 data files labeled by month, and show that RASMR can be successfully used for selecting the most appropriate model for real massive data analysis.

Application of Real-Time Data Service Technology in Smart oilfield

Abstract Real time data services play an important role in the construction of Smart oilfields, which can help oilfield production managers better understand the production situation and make more accurate decisions. Real time data services based on the Data Lake platform will bring more efficient and safer production methods to oil and gas fields, promoting the upgrading and sustainable development of the Smart oilfield industry. The technical architecture of Smart oilfield real-time data service mainly includes data lake platform, big data processing, edge computing and other core technologies. The data lake platform provides infrastructure, platform services and application services for Smart oilfield. Big data processing technology is responsible for rapid and accurate analysis of massive data. edge computing refers to computing at the data source to reduce data transmission and improve computing efficiency. The key technologies include data collection, transmission, storage, analysis, etc. Among them, data collection mainly uses sensors, PLCs and other equipment to collect production data. Data transmission uses technologies such as the Internet of Things and industrial Ethernet to transmit data to the data lake platform. Data storage adopts distributed storage technology to ensure the safety and reliability of data. Data analysis uses machine learning, deep learning and other technologies to analyze data, Provide decision support.

Power cable monitoring method based on UHF‐RFID and deep learning in edge computing environment

AbstractThis research addresses the challenge faced by most existing prediction methods in handling nonlinear data of cables. Furthermore, it proposes a novel power cable monitoring method utilizing UHF‐RFID and deep learning within an edge computing environment, specifically targeting the currently suboptimal wireless monitoring of cables. First, based on edge computing, a power cable monitoring system is designed to migrate the analysis of massive data to the edge of the network to improve the monitoring efficiency. Then, the temperature sensing chip and RFID chip were integrated to design a UHF‐RFID temperature tag, which was fixed at the cable temperature measurement point to achieve passive wireless monitoring of the cable. Finally, the parameters of the GRNN model are optimized using the beetle antennae search algorithm, and the EEMD decomposed data is input into the BAS‐GRNN model for learning to output temperature prediction results. Based on the establishment of an experimental platform, the method was demonstrated, and results showed that the maximum error between the UHF‐RFID temperature tag temperature measurement results and the thermocouple was within 0.3°C, and the average relative error of the proposed method was only 0.01, which can meet the accuracy requirements of actual monitoring of power cables.

Design and Implementation of AI-based Judicial Big Data Intelligent Service Platform

Based on big data and artificial intelligence technology, the judicial big data intelligent service platform provides users with efficient and accurate legal consulting services. It has many functions such as intelligent question and answer, case analysis, and law inquiry. It also analyzes the application value of the intelligent legal consulting service platform in the field of judicial big data. By building an intelligent service platform, the rapid processing and analysis of massive judicial data has been realized, and the efficiency and quality of judicial work have been significantly improved. In addition, it emphasizes the importance of data management and quality control in the application of judicial big data, and puts forward specific measures to strengthen data management and quality control. With the continuous advancement of technology and the expansion of application fields, the platform is expected to achieve deep integration with other legal services, expand new application fields, and provide users with more comprehensive and efficient legal services. At the same time, it points out the problems that need to be paid attention to in the future development, such as data security, privacy protection and so on. Through in-depth analysis and research on judicial big data, the platform can provide strong support for judicial decision-making and promote the improvement of judicial justice and efficiency

Automatic text classification of drug-induced liver injury using document-term matrix and XGBoost.

Regulatory agencies generate a vast amount of textual data in the review process. For example, drug labeling serves as a valuable resource for regulatory agencies, such as U.S. Food and Drug Administration (FDA) and Europe Medical Agency (EMA), to communicate drug safety and effectiveness information to healthcare professionals and patients. Drug labeling also serves as a resource for pharmacovigilance and drug safety research. Automated text classification would significantly improve the analysis of drug labeling documents and conserve reviewer resources. We utilized artificial intelligence in this study to classify drug-induced liver injury (DILI)-related content from drug labeling documents based on FDA's DILIrank dataset. We employed text mining and XGBoost models and utilized the Preferred Terms of Medical queries for adverse event standards to simplify the elimination of common words and phrases while retaining medical standard terms for FDA and EMA drug label datasets. Then, we constructed a document term matrix using weights computed by Term Frequency-Inverse Document Frequency (TF-IDF) for each included word/term/token. The automatic text classification model exhibited robust performance in predicting DILI, achieving cross-validation AUC scores exceeding 0.90 for both drug labels from FDA and EMA and literature abstracts from the Critical Assessment of Massive Data Analysis (CAMDA). Moreover, the text mining and XGBoost functions demonstrated in this study can be applied to other text processing and classification tasks.

Research on Enterprise Human Resource Management Reform Strategy in the Era of Big Data

In the 21st century, with the development of the Internet, mobile devices, and information technology, society has entered a new era: the era of big data. With the help of big data technology, enterprises can obtain massive market and consumer data, realize in-depth analysis of business and market, and enable enterprises to have a deeper understanding of consumer needs, preferences, and behaviors. At the same time, big data technology can also help enterprises carry out human resource management innovation and improve the performance and competitiveness of enterprises. Of course, from another perspective, enterprises in this era are also facing severe challenges. In the face of massive data processing and analysis, it requires superb data processing and analysis capabilities. Secondly, enterprises need to reconstruct their management system to adapt to the changes in the era of big data. Enterprises must treat data as assets and establish a perfect data management system. In addition, enterprises also need to pay attention to protecting customer privacy and data security to avoid data leakage and abuse. In this context, this paper will explore the thinking of enterprise human resource management innovation in the era of big data, and put forward some suggestions on enterprise human resource management innovation.

Communication spectrum prediction method based on convolutional gated recurrent unit network

In modern wireless communication systems, the scarcity of spectrum resources poses challenges to the performance and efficiency of the system. Spectrum prediction technology can help systems better plan and schedule resources to respond to the dynamic changes in spectrum. Dynamic change in the spectrum refers to the changes in the radio spectrum in a wireless communication system. It means that the available spectrum resources may change at different times and locations. In response to this current situation, this study first constructs a communication collaborative spectrum sensing model using channel aliasing dense connection networks. Then, combining convolutional neural network and gated cyclic unit network in deep learning technology, a communication spectrum prediction model is built. It aims to achieve accurate perception and prediction of spectrum resources through the aforementioned spectrum sensing and prediction models. The results confirm that the proposed perception model has inconsistent perception accuracy under different number of secondary users, with a maximum of 0.99. It is verified that the proposed spectrum prediction model achieves a high prediction accuracy of 0.95 within 208 s and its performance outperforms current similar models. The results are based on the model's deep learning analysis of massive historical communication data, in which the optimized shuffle dense net model plus convolutional gated recurrent unit model is the key to achieve fast and accurate prediction. On the contrary, the highest spectrum prediction accuracy of Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Convolutional Neural Networks-Long Short-Term Memory (ConvLSTM) models are 0.86, 0.90, and 0.85, respectively. And the model needs to run for a longer period of time, up to 324, for ConvLSTM to reach the prediction accuracy value of 0.95. In summary, the perception and prediction model built by this research has good performance, and its application in the field of wireless communication can assist staff in better monitoring spectral changes, thereby making more efficient use of spectral resources.

AI-enabled Rendering Technology for the Construction and Development of Dunhuang Virtual Pavilion

This paper focuses on the use of AI technology to assist in the construction of digital scenes in Dunhuang Research Institute, which realizes the roaming of tourists in virtual scenes through image recognition and semantic analysis of massive digital image data, and explores the possibility of AI-assisted rendering of virtual museums. Using AI technology, tourists can observe the uniqueness of each cave within the cavern and its rich content from any angle, and obtain an immersive travel experience. The study fully utilizes the new generation of information technology to build a virtual scene platform of Dunhuang Mogao Grottoes, which comprehensively presents the history, cultural value and artistic value of Dunhuang Mogao Grottoes through the comprehensive integration and display of a variety of digital resources. Focusing on digital media and virtual reality technology, through the use of artificial intelligence technology and the comprehensive integration of related digital resources, to build a complete set of virtual reality ecosystem, and explore the realization of multi-source data fusion of Dunhuang cultural communication.

Leveraging AI and Blockchain for Privacy Preservation and Security in Fog Computing

INTRODUCTION: Cloud computing's offshoot, fog computing, moves crucial data storage, processing, and networking capabilities closer to the people who need them. There are certain advantages, such improved efficiency and lower latency, but there are also some major privacy and security concerns. For these reasons, this article presents a new paradigm for fog computing that makes use of blockchain and Artificial Intelligence (AI).
 OBJECTIVES: The main goal of this research is to create and assess a thorough framework for fog computing that incorporates AI and blockchain technology. With an emphasis on protecting the privacy and integrity of data transactions and streamlining the management of massive amounts of data, this project seeks to improve the security and privacy of Industrial Internet of Things (IIoT) systems that are cloud-based.
 METHODS: Social network analysis methods are utilised in this study. The efficiency and accuracy of data processing in fog computing are guaranteed by the application of artificial intelligence, most especially Support Vector Machine (SVM), due to its resilience in classification and regression tasks. The network's security and reliability are enhanced by incorporating blockchain technology, which creates a decentralised system that is tamper resistant. To make users' data more private, zero-knowledge proof techniques are used to confirm ownership of data without actually disclosing it.
 RESULTS: When applied to fog computing data, the suggested approach achieves a remarkable classification accuracy of 99.8 percent. While the consensus decision-making process of the blockchain guarantees trustworthy and secure operations, the support vector machine (SVM) efficiently handles massive data analyses. Even in delicate situations, the zero-knowledge proof techniques manage to keep data private. When these technologies are integrated into the fog computing ecosystem, the chances of data breaches and illegal access are greatly reduced.
 CONCLUSION: Fog computing, which combines AI with blockchain, offers a powerful answer to the privacy and security issues with cloud centric IIoT systems. Combining SVM with AI makes data processing more efficient, while blockchain's decentralised and immutable properties make it a strong security measure. Additional security for user privacy is provided via zero-knowledge proofs. Improving the privacy and security of fog computing networks has never been easier than with this novel method.

Sizing of Battery Energy Storage Systems for Firming PV Power including Aging Analysis

The variability of solar radiation presents significant challenges for the integration of solar photovoltaic (PV) energy into the electrical system. Incorporating battery storage technologies ensures energy reliability and promotes sustainable growth. In this work, an energy analysis is carried out to determine the installation size and the operating setpoint with optimal constant monthly power through an iterative calculation process, considering various operating setpoints and system parameters. A degradation model is integrated according to the curves offered by battery manufacturers and the charge–discharge cycles are calculated using the rainflow method to guarantee a reliable analysis of the plant. Through massive data analysis in a long-term simulation, indicators are generated that allow for establishing a relationship between the energy unavailability of the system and the BESS dimensions.

A step-by-step guide to include key electroencephalography (EEG) parameters in the study of human performance applied to air traffic control

The study of human performance of air traffic controllers (ATCOs) is an interesting line of research to improve operational safety. In recent years, there has been an increase in the number of techniques available to develop this research based on massive data analysis. This study presents the use of certain electroencephalography (EEG) parameters to study the human performance of ATCOs. Although software and applications are now available to calculate these parameters, there are often problems in understanding the detailed process used to calculate them. The parameters presented in this study are intended to overcome this limitation and are applicable in real air traffic control (ATC) situations. Six parameters are analysed: excitement, stress, relaxation, boredom, engagement, and attention. As an application case for the parameters, a total of 50 data samples obtained during the development of real-time simulations on a highly realistic ATC platform are analysed. From these data, the above-mentioned EEG parameters and their trends are calculated. In addition, the evolution of these parameters is studied in relation to two other variables that characterise the operational situation of the sector during the simulations: the taskload based on ATC events and the number of simultaneous aircraft in the sector per minute.

RT-CBCH: Real-Time VPN Traffic Service Identification Based on Sampled Data in High-Speed Networks

Virtual Private Network (VPN) technology can bypass censorship and access geographically locked services. Some harmful information may be hidden in VPN traffic and circumvent the surveillance systems, bringing a significant challenge to network security. Considering the increasing richness of service types in VPN traffic, identifying traffic service facilitates further targeting harmful VPN traffic. Therefore, VPN traffic service identification is critical in network management. The existing identification methods use complete traffic for analysis. However, massive data analysis in high-speed networks consumes enormous resources, limiting the real-time processing of traffic identification. This paper proposes a real-time VPN traffic service identification method named RT-CBCH. We construct features that are still available after sampling and design a fast traffic processing structure based on Counting Bloom Filter and Chained Hash Table (CBCH). Experimental results validate the real-time capability, stability and accuracy of our method. At the sampling ratio of 1/256, it takes only 23.63 seconds to process the mixed traffic of 900-second traffic generated on a 10 Gbps link and our collected V2Ray traffic, which is increasingly common in VPN traffic. Under different sampling ratios, the identification results remain respectable, with an overall accuracy of about 90% for application service and over 99% for V2Ray proxy service. Furthermore, comparisons with similar work illustrate the high accuracy and low resource consumption of RT-CBCH. Experimental results show that our method can stably implement real-time VPN traffic service identification from sampled data in high-speed networks.

CluBear: a subsampling package for interactive statistical analysis with massive data on a single machine

This article introduces CluBear, a Python-based open-source package for interactive massive data analysis. The key feature of CluBear is that it enables users to conduct convenient and interactive statistical analysis of massive data with only a traditional single-computer system. Thus, CluBear provides a cost-effective solution when mining large-scale datasets. In addition, the CluBear package integrates many commonly used statistical and graphical tools, which are useful for most commonly encountered data analysis tasks.

A study of the transaction volume prediction problem based on recurrent neural networks

With the rapid development of artificial intelligence technology, intelligent fintech scenarios based on big data are receiving more and more attention, and through the analysis of massive financial class data, accurate decision support can be provided for its various scenarios. By predicting the transaction volume of a financial product of a bank, abnormal transaction flow and gradual change trend can be found 1 day in advance to provide decision support for business department program development, and provide decision support for system expansion and contraction, thus reducing system online pressure or releasing unnecessary system resources. Linear algorithms such as AR model, MA model, ARMA model, etc. have poor prediction results for transaction volumes during holidays in the non-stationary dataset handled in this study due to strong assumptions on historical data. In this paper, we design and implement an LSTM-based trading volume prediction model LSTM-WP (LSTM-WebPredict) using deep learning algorithm, which can improve the accuracy of prediction of holiday trading volume by about 8% based on the linear algorithm by discovering and learning the features of historical data, and the learning ability of the model will gradually increase with the increasing of training data; Not only that, the research of this algorithm also provides corresponding technical accumulation for other business scenarios of time series problems, such as trend prediction and capacity assessment.