Big Data Framework Research Articles

Application of Statistical Models in Air Quality Monitoring

Currently, statistical models are vital instruments for examining and forecasting diverse environmental variables, including air quality analysis. With the development of big data, foundational statistical techniquesincluding regression and time series analysisare adapting to manage larger datasets and complex environmental dynamics. This paper examines the recent applications of statistical models used to assess and forecast air quality in Chinese cities based on existing literature and data research. By focusing on various modeling approaches, such as Poisson regression, grey correlation, and neural networks, it discusses the advantages and drawbacks of each model. Furthermore, it examines the enhancement of classical analytical methodologies inside a big data framework to augment the accuracy of air quality analysis and forecasting. The result shows that although datasets have been more extensive, to increase the accuracy of pollution forecasting, diverse data and more excellent computational capabilities are still needed. Advances in machine learning and optimisation algorithms show promise for overcoming these challenges in the future

Analyzing the Effectiveness of Machine Learning Techniques in Detecting Attacks in a Big Data Environment

Protecting big data has become an extremely vital necessity in the context of cybersecurity, given the significant impact that this data has on institutions and clients. The importance of this type of data is highlighted as a basis for decision-making processes and policy guidance. Therefore, attacks on this data can lead to serious losses through illicit access, resulting in a loss of integrity, reliability, confidentiality, and availability of this data. The second problem in this context arises from the necessity of reducing the attack detection period and its vital importance in classifying malicious and non-harmful patterns. Structured Query Language Injection Attack (SQLIA) is among the common attacks targeting data, which is the focus of interest in the proposed model. The aim of this research revolves around developing an approach aimed at detecting and distinguishing patterns of loads sent by the user. The proposed method is based on training a model using random forest technology, which is considered one of the machine learning (ML) techniques while taking advantage of the Spark ML library that interacts effectively with big data frameworks. This is accompanied by a comprehensive analysis of the effectiveness of ML techniques in monitoring and detecting SQLIA. The study was conducted using the SQL dataset available on the Kaggle platform and showed promising results as the proposed method achieved an accuracy of 98.12%. While the proposed approach takes 0.046 seconds to determine the SQL type. It is concluded from these results that using the Spark ML library based on ML techniques contributes to achieving higher accuracy and requires less time to identify the class of request sent due to its ability to be distributed in memory.

Intelligent decision-making framework for big data using enhanced honey badger-based adaptive hybrid deep learning network

Intelligent decision-making framework for big data using enhanced honey badger-based adaptive hybrid deep learning network

Intelligent decision-making framework for big data using enhanced honey badger-based adaptive hybrid deep learning network

Intelligent decision-making framework for big data using enhanced honey badger-based adaptive hybrid deep learning network

Suitability analysis of ecological park site selection in small and medium-sized cities based on multi-source geographic big data

In the context of rapid urbanization, the scientific planning and development of ecological parks in small- to medium-sized cities are essential for enhancing the well-being of city dwellers and optimizing the ecological quality of urban environments. Traditional site suitability methods, however, are increasingly limited due to data resolution constraints and the challenges of balancing ecological protection, urban development, and socio-economic interests, which hardly provide insights for ecological park planning in small and medium-sized cities. This study addresses these limitations by employing a multi-source geographic big data framework, and applying it in Yongji City, Shanxi Province, by integrating remote sensing, Points of Interest, and land use data, a detailed 10m × 10m spatial grid was used for analysis. A comprehensive three-dimensional evaluation framework was developed, comprising ecological, infrastructural, and socio-economic factors, represented by 11 indicators. Points of Interest data have effectively assessed infrastructural configurations necessary for urban ecological parks, while remote sensing data provides critical insights into surface temperature and vegetation cover, which provide high-resolution indicators of environmental quality in urban and sub-urban regions. The site suitability analysis results reveal that hydrophilic landscapes, such as the Yellow River mudflats and Wuxing Lake wetlands, were identified as optimal areas for ecological park development due to their cooling effects and ecological benefits. Zone C, located southwest of Wuxing Lake, emerged as the ideal site due to its abundant natural assets and high accessibility, with a phased development strategy of other recommended sites based on their suitability. This research establishes a data-integrated framework that advances the methodology for urban ecological park planning in small- to medium-sized cities, providing both theoretical and practical insights that can inform similar projects in other urban contexts.

Cloud data centers and networks: Applications and optimization techniques

As the volume and complexity of big data continue to escalate, optimizing the performance, scalability, and energy efficiency of big data applications within cloud data centers has become increasingly crucial. This journal presents a comprehensive survey of current optimization techniques, focusing on data placement, job scheduling, and network configurations tailored for cloud environments. We explore the impact of various data center topologies on the performance of big data frameworks like Hadoop, emphasizing the trade-offs between performance and energy efficiency. Advanced methodologies, including dynamic data placement strategies, locality-aware scheduling, and innovative reduce task placement techniques, are reviewed in depth. Additionally, we highlight the importance of network power effectiveness (NPE) and examine the role of optical and electronic switching technologies in enhancing data center efficiency. By synthesizing findings from recent studies, this paper provides valuable insights into the optimization of cloud data centers, offering recommendations for improving resource utilization and reducing job completion times while maintaining energy efficiency. The findings contribute to the ongoing efforts to scale and adapt cloud data infrastructures for the rapidly growing demands of big data applications.

Optimized robust learning framework based on big data for forecasting cardiovascular crises

Numerous Deep Learning (DL) scenarios have been developed for evolving new healthcare systems that leverage large datasets, distributed computing, and the Internet of Things (IoT). However, the data used in these scenarios tend to be noisy, necessitating the incorporation of robust pre-processing techniques, including data cleaning, preparation, normalization, and addressing imbalances. These steps are crucial for generating a robust dataset for training. Designing frameworks capable of handling such data without compromising efficiency is essential to ensuring robustness. This research aims to propose a novel healthcare framework that selects the best features and enhances performance. This robust deep learning framework, called (R-DLH2O), is designed for forecasting cardiovascular crises. Unlike existing methods, R-DLH2O integrates five distinct phases: robust pre-processing, feature selection, feed-forward neural network, prediction, and performance evaluation. This multi-phase approach ensures superior accuracy and efficiency in crisis prediction, offering a significant advancement in healthcare analytics. H2O is utilized in the R-DLH2O framework for processing big data. The main improvement of this paper lies in the unique form of the Whale Optimization Algorithm (WOA), specifically the Modified WOA (MWOA). The Gaussian distribution approach for random walks was employed with the diffusion strategy to choose the optimal MWOA solution during the growth phase. To validate the R-DLH2O framework, six performance tests were conducted. Surprisingly, the MWOA-2 outperformed other heuristic algorithms in speed, despite exhibiting lower accuracy and scalability. The suggested MWOA was further analyzed using benchmark functions from CEC2005, demonstrating its advantages in accuracy and robustness over WOA. These findings highlight that the framework’s processing time is 436 s, mean per-class error is 0.150125, accuracy 95.93%, precision 92.57%, and recall 93.6% across all datasets. These findings highlight the framework’s potential to produce significant and robust results, outperforming previous frameworks concerning time and accuracy.

Integrating Big Data, Artificial Intelligence, and motion analysis for emerging precision medicine applications in Parkinson’s Disease

One of the key challenges in Big Data for clinical research and healthcare is how to integrate new sources of data, whose relation to disease processes are often not well understood, with multiple classical clinical measurements that have been used by clinicians for years to describe disease processes and interpret therapeutic outcomes. Without such integration, even the most promising data from emerging technologies may have limited, if any, clinical utility. This paper presents an approach to address this challenge, illustrated through an example in Parkinson’s Disease (PD) management. We show how data from various sensing sources can be integrated with traditional clinical measurements used in PD; furthermore, we show how leveraging Big Data frameworks, augmented by Artificial Intelligence (AI) algorithms, can distinctively enrich the data resources available to clinicians. We showcase the potential of this approach in a cohort of 50 PD patients who underwent both evaluations with an Integrated Motion Analysis Suite (IMAS) composed of a battery of multimodal, portable, and wearable sensors and traditional Unified Parkinson's Disease Rating Scale (UPDRS)-III evaluations. Through techniques including Principal Component Analysis (PCA), elastic net regression, and clustering analysis we demonstrate how this combined approach can be used to improve clinical motor assessments and to develop personalized treatments. The scalability of our approach enables systematic data generation and analysis on increasingly larger datasets, confirming the integration potential of IMAS, whose use in PD assessments is validated herein, within Big Data paradigms. Compared to existing approaches, our solution offers a more comprehensive, multi-dimensional view of patient data, enabling deeper clinical insights and greater potential for personalized treatment strategies. Additionally, we show how IMAS can be integrated into established clinical practices, facilitating its adoption in routine care and complementing emerging methods, for instance, non-invasive brain stimulation. Future work will aim to augment our data repositories with additional clinical data, such as imaging and biospecimen data, to further broaden and enhance these foundational methodologies, leveraging the full potential of Big Data and AI.

Spark: A Statistical Comparison and Evaluation of Classification Algorithms for Fault Prediction in Electrical Secondary Distribution Network

Managing faults in the electrical secondary distribution network is a challenging task given the nature, size, and complexity. Predicting faults early before they occur helps in increasing the safety and reliability of the power distribution system. Various statistical and machine learning techniques are being used to predict different types of faults. This study applies classification algorithms available in the big data framework Apache Spark through its python interface PySpark to predict electrical secondary distribution network faults. The study evaluates and compares nine algorithms: Decision tree, Gradient-boosted tree, Logistic regression, Naïve Bayes, Multilayer perceptron, Random forest, Linear Support Vector Machine, One-versus-rest and Factorization machines. The research uses Friedman’s test followed by the Nemenyi post hoc test to find the significance of performance differences among the algorithms. The results show significant differences among the algorithms. Gradient-boosted tree and One-versus-rest with Gradient-boosted tree had the best performance for binary and multiclass classification, respectively, while Naïve Bayes had the worst performance.

TeraHeap: Exploiting Flash Storage for Mitigating DRAM Pressure in Managed Big Data Frameworks

Big data analytics frameworks, such as Spark and Giraph, need to process and cache massive datasets that do not always fit on the managed heap. Therefore, frameworks temporarily move long-lived objects outside the heap (off-heap) on a fast storage device. However, this practice results in (1) high serialization/deserialization (S/D) cost and (2) high memory pressure when off-heap objects are moved back for processing. In this article, we propose TeraHeap , a system that eliminates S/D overhead and expensive GC scans for a large portion of objects in analytics frameworks. TeraHeap relies on three concepts: (1) It eliminates S/D by extending the managed runtime (JVM) to use a second high-capacity heap (H2) over a fast storage device. (2) It offers a simple hint-based interface, allowing analytics frameworks to leverage object knowledge to populate H2. (3) It reduces GC cost by fencing the collector from scanning H2 objects while maintaining the illusion of a single managed heap, ensuring memory safety. We implement TeraHeap in OpenJDK8 and OpenJDK17 and evaluate it with fifteen widely used applications in two real-world big data frameworks, Spark and Giraph. We find that for the same DRAM size, TeraHeap improves performance by up to 73% and 28% compared to native Spark and Giraph. Also, it can still provide better performance by consuming up to \(4.6\times\) and \(1.2\times\) less DRAM than native Spark and Giraph, respectively. TeraHeap can also be used for in-memory frameworks and applying it to the Neo4j Graph Data Science library improves its performance by up to 26%. Finally, it outperforms Panthera, a state-of-the-art garbage collector for hybrid DRAM-NVM memories, by up to 69%.

How to Realize the Collaborative Supply of Cultural Resource Big Data with Government Participation: Experiences from China

To foster the sustainable development of culture, particularly focusing on the preservation of cultural heritage, encompassing relics, intangible cultural heritage, and historical sites, China has launched a strategy for the digitalization of culture, with the goal of establishing a holistic national big data framework for cultural resources. To improve the efficiency of collaborative supply of cultural resource big data among various parties and to further advance the sustainable development of culture, this research has created a cooperative model that includes cultural institutions, a cultural resource big data service platform, and government participation. Our research findings, based on prospect theory and evolutionary game theory combined with Chinese practice, are presented below. (1) Various factors, including the coefficient of digital infrastructure empowerment, access charges for digital infrastructure, government penalties, and the probability of data leakage, have varying effects on the system in different states. (2) Once the industry has developed, the government can increase the impact of digital infrastructure empowerment to create stronger incentives, rather than relying solely on rewards or penalties. (3) When the value level of cultural resource big data is high, the benefit distribution coefficient does not affect the system evolution results. Finally, we offer practical insights for the government, cultural organizations, and cultural resource big data service platforms based on our research results. Our research offers Chinese insights for global cultural sustainable development.

Big Data Framework for Predicting Infectious Diseases to Improve Healthcare by Discovering New Symptom Patterns

Big Data Framework for Predicting Infectious Diseases to Improve Healthcare by Discovering New Symptom Patterns

Toward High-Quality Real-World Laboratory Data in the Era of Healthcare Big Data.

With Industry 4.0, big data and artificial intelligence have become paramount in the field of medicine. Electronic health records, the primary source of medical data, are not collected for research purposes but represent real-world data; therefore, they have various constraints. Although structured, laboratory data often contain unstandardized terminology or missing information. The major challenge lies in the lack of standardization of test results in terms of metrology, which complicates comparisons across laboratories. In this review, we delve into the essential components necessary for integrating real-world laboratory data into high-quality big data, including the standardization of terminology, data formats, equations, and the harmonization and standardization of results. Moreover, we address the transference and adjustment of laboratory results, along with the certification for quality of laboratory data. By discussing these critical aspects, we seek to shed light on the challenges and opportunities inherent to utilizing real-world laboratory data within the framework of healthcare big data and artificial intelligence.

Field scale wheat yield prediction using ensemble machine learning techniques

Wheat is crucial for global food security and plays significant role in achieving United Nations Sustainable Development Goal 2 (Zero Hunger). In India, wheat accounts for 33.5 % of total cereal production. Accurate and cost effective yield predictions are essential for maintaining food security. Wheat yield forecasting is influenced by various factors, such as genotype and environmental conditions. Among these, the effect of morpho-physiological traits in the field is important for predicting yield but hasn't been studied much using ensemble machine learning methods. This study aims to bridge this gap by evaluating 29 morpho-physiological traits to predict site specific wheat yield. Big data framework was used to develop and refine several ensemble machine learning models based on field trial datasets. The developed models are optimized to reduce errors and prevent overfitting and underfitting, boosting their predictive precision. Each model's efficiency was evaluated using performance metrics such as mean absolute error, mean absolute deviation, root mean square error, R2, and overall accuracy. The ensemble model, which combines random forest (RF) and artificial neural networks (ANN), demonstrated better performance by achieving a mean absolute percentage error of 4.65 %, and R2 value of 98.48 % and 98.18 % accuracy on test data.Our results demonstrate that ensemble models combining RF with support vector regression (SVR) outperformed individual models such as RF, SVR, ANN, multivariate adaptive regression splines (MARS). These findings are a promising step towards future research focused on creating more advanced ensemble methods with finely tuned hyperparameters for improving the accuracy of large-scale wheat yield prediction.

Performance aware algorithm design for elastic resource workflow management of cluster consolidation to handle enterprise big data

<span lang="EN-US">Integration and deployment of big data and business analytics application with cloud computing are more attractive as a service and are trending practice. This hybrid workflow is rapidly increasing and will trigger a revolution for enterprise data handling, information retrieval and computing. This paper presents hybrid workflow management framework for big data and multi cloud computing systems in a two-step approach. Linear optimization-based resource assessment algorithm is planned in the first step. Cluster oriented elastic resource allocation and workflow management techniques are concentrated in the second step. This paper also focus on performance evaluation parameters includes execution time, through put with multi task work flow optimization model. The proposed framework is efficiently managed the implementation of hybrid workflows by finetuning the evaluation attributes and provides improvement in terms of response time an average of 6%.</span>

Retraction Note: A hybrid group search optimization: firefly algorithm-based big data framework for ancient script recognition

Retraction Note: A hybrid group search optimization: firefly algorithm-based big data framework for ancient script recognition

Implementing Privacy Preserving Techniques in Distributed Big Data Framework on Cloud Computing

Cloud-based deployment architectures have emerged as the preferred model for Big Data operations due to their scalability, flexibility, and cost-effectiveness. However, this shift raises new security concerns as data is no longer directly under the user's control. Securing Big Data in cloud environments is crucial for widespread adoption, but developing a comprehensive security plan is challenging without a thorough analysis of potential vulnerabilities.To address this, a novel security-by-design framework for Big Data deployment on cloud computing, termed Big Cloud, is proposed in this article. The framework leverages systematic security analysis methodologies and automated assessment tools to map domain-specific security knowledge to design best practices. Validation of the framework was conducted through an Apache Hadoop stack use case, demonstrating its effectiveness in enhancing security awareness and reducing design time.The study also evaluates the framework's strengths and limitations, identifying key challenges in the Big Cloud domain.

An Effective Metaheuristic Based Dynamic Fine Grained Data Security Framework for Big Data

An Effective Metaheuristic Based Dynamic Fine Grained Data Security Framework for Big Data

Application of Clustering Algorithm in the Evaluation of College Students' English Reading Ability Under the Framework of Big Data

In recent years, China has accelerated the process of internationalization and made more and more achievements in transnational communication and cooperation. English learning is very important for contemporary college students. And English reading is an important means to acquire English language knowledge, understand external information and improve English language practice ability. Therefore, cultivating college students' English reading ability is a pivotal element in college English teaching. This study will focus on the application of clustering algorithm based on big data framework in the evaluation of college students' English reading ability, aiming at exploring and establishing an effective evaluation model to promote the improvement of college students' English reading ability and academic achievement.

International Law Protection of Cross-Border Transfers of Personal Information Based on Cloud Computing and Big Data

Cross-border data is crucial for the digital economy s growth, but uncontrolled transfers risk national security, public interests, corporate interests, and data sovereignty. Currently, cross-border data regulation lacks unified global rules, with existing ones dominated by Europe and the U.S. China needs to improve its international legal protection for cross-border personal information transfer. This paper examines China s mechanisms for personal data privacy protection in digital trade, including frameworks for big data, cloud computing, and cloud service selection for data lifecycle privacy protection. It suggests clarifying data controllers and exporters obligations, strengthening regulatory responsibilities, improving cross-border data legislation, enhancing law operability, and advocating for stronger international cooperation and participation in global rule-making to protect personal privacy information rights.