Big Data Processing Research Articles

Standardization of studies on physical development of children in the Russian Federation

BACKGROUND:The importance of studying the growth and development processes of the child population is due to the significance of the pre-clinical approach aimed at primary disease prevention. AIM:To provide the constituent entities of the Russian Federation with standards for the physical development of children and adolescents aged 7–17years. MATERIAL AND METHODS:An analytical study was conducted, big data from the database “Physical Development of Children, Adolescents, and Youth of the Russian Federation in 2000–2021” were analyzed, containing 395,956observations of physical development (length and body weight indicators) of children, adolescents, and young people living in 49constituent entities of the Russian Federation. Priority was given to research in the constituent entities for which regional standards had never been developed or for which they were developed 15–20years or more ago. As a result, at least 2constituent entities were analyzed in each federal district. Big data processing using machine learning was carried out on the basis of the PostgreSQL database. A web application was created that implemented the functions of tabular display of data, construction of histograms with the ability to select parameters and subsequent data publication on the map of the Russian Federation. RESULTS:In Russia at the beginning of the 21st century, the characteristics of physical development of students living in diffe­rent constituent entities of the Russian Federation were established, which allowed us to talk about the presence of “regional scenarios” that must be taken into account when developing standards. At the age of 15, the range of average values of growth indicators for boys in the constituent entities ranged from 161.70±0.20cm in the Kursk Region to 174.55±0.40cm in the Krasnodar Territory (the difference was 12.85cm, p=0.001), for girls— from 154.60±0.50cm in the Republic of Tyva to 165.69±0.55cm in St.Petersburg (the difference was 11.09cm, p=0.001). The range of body weight of 15-year-old boys in the regions was from 50.17±0.70kg in the Kursk region to 66.00±0.80kg in the Khabarovsk region (difference 15.83kg, p=0.001), girls— from 46.90±0.60kg in the Chelyabinsk region to 58.40±0.70kg in the Khabarovsk region (difference 8.50kg, p=0.001). CONCLUSION:40regional standards for the physical development of the child population have been developed in a standar­dized manner, which allows taking into account regional characteristics and makes it possible to use them in practical healthcare in the constituent entities of the Russian Federation.

Research on the Impact of Big Data and Enterprise Digitalization on the E-commerce Industry and Coping Strategies

Abstract: Big data, enterprise digitalization, and e-commerce all have one thing in common: they were born from the Internet. Since the onset of the 21st century, the global Internet industry has experienced rapid growth, with e-commerce emerging as a key player. Participants in e-commerce engage with both the global online marketplace and the physical market, utilizing modern communication networks and information technology to conduct business activities through e-commerce platforms or service platforms. E-commerce embodies characteristics such as market globalization, rapid transactions, virtualization, and cost-effectiveness, making it highly favored by enterprises. Presently, e-commerce relies heavily on robust databases, as well as the data collection, processing, and analysis capabilities of big data. This reliance compels traditional enterprises to undergo digital transformation. This study delves into the impact of big data and enterprise digitalization on the e-commerce industry, exploring the mechanisms underlying their influence on cross-border e-commerce. By examining these dynamics, the study aims to provide insights into the evolving landscape of e-commerce and the transformative potential of big data and digitalization in shaping its future trajectory.

Storage and Processing of Big Data for Geomagnetic Support of Directional Drilling

Storage and Processing of Big Data for Geomagnetic Support of Directional Drilling

A review of load forecasting technology based on artificial intelligence

Abstract. Driven by the "dual carbon" goals and the construction of new-type power systems, the importance of power load forecasting technology has become increasingly prominent. Accurate power load forecasting can not only ensure the safe and stable operation of the power system, but also promote the efficient use of clean energy and reduce carbon emissions. Traditional power load forecasting methods have become insufficient in dealing with the complexity and nonlinear growth of power systems, and the development of artificial intelligence technology has brought new possibilities for power load forecasting. Power load forecasting methods based on machine learning and deep learning have gradually become a hot topic of research due to their powerful nonlinear mapping capabilities and advantages in processing big data. These methods not only perform well in short-term forecasting, but also show potential in medium- and long-term forecasting. This paper discusses the application of different artificial intelligence technologies in power load forecasting by sorting out the development status of related technologies, and provides a reference direction for future research.

Optimizing Business Operations through Edge Computing: Advancements in Real-Time Data Processing for the Big Data Era

The question of big data brought new opportunities and threats to business environment primarily in terms of processing increasingly large amounts of data in realtime. This paper examines the way in which edge computing can /should benefit business processes by offering improvements to real-time data analysis beyond what is provided by cloud-based systems while overcoming the challenge of latency, bandwidth and scalability. The study uses both primary and secondary data collected from 50 companies that use edge computing for processing of big data in industries including manufacturing, health, and retail. It involves cases where the factors are manifested, practices involving changes in numerical values of operations and other models involving the management of real-time data. Studies show that edge computing cuts down data processing delay by 40-60% on average, cuts data utilization and improves decision-making. This work also examines the use-cases of edge computing in order to demonstrate its potential across industries that must analyze data in near realtime, including self-driving automobiles, IoT gadgets, and logistics. Toward this end, this research provides an analysis of the ethical issues of decentralized data handling especially focusing on data security and data privacy. In response to the research questions of the paper, this study provides practical recommendations to help the companies in pursuing edge computing in the big data environment for the sake of sustaining market competitiveness. The research enriches the existing knowledge in business technology by presenting the possibility of change that edge computing brings.

Course Reform and Practice of "Python Programming" for Artificial Intelligence and Big Data Applications

With the rapid development of artificial intelligence (AI) and big data technologies, Python has become a widely used programming language in related fields. However, the traditional "Python Programming" course has been unable to meet the demands of applications in AI and big data in terms of content and teaching models. Therefore, this paper presents a comprehensive reform of the "Python Programming" course, optimizing course content by incorporating modules on big data processing and artificial intelligence. Additionally, a project-driven teaching model and a diversified evaluation system have been implemented. By introducing innovative teaching tools and resources, students' practical abilities and innovative thinking have been enhanced. The practical outcomes of the course reform are significant, with students showing notable improvements in their programming skills in AI and big data. In the future, this study will continue to optimize course content to address the ever-changing technological demands.

Research on Ship Data Analysis Platform Development and Its Application in Teaching

This paper delves into the significant importance of developing a ship data analysis platform, which aims to enhance the operational efficiency and safety of the shipping industry by integrating big data processing technologies. The development of the platform centers on crucial aspects such as data collection, preprocessing, analysis, and visualization, enabling real-time monitoring and intelligent prediction of ship operational status. This platform not only assists in optimizing route planning and reducing energy consumption costs but also provides robust support for navigational safety. In project-based teaching, this platform serves as a bridge between theory and practice, cultivating students' abilities in big data processing and analysis through case studies, practical exercises, and other forms, thereby promoting the cultivation of high-quality talents and technological innovation in the field of shipping.

Application Status of Hadoop in Data Cloud Computing

This paper summarizes the Hadoop platform and its application in network public opinion analysis, and emphasizes the advantages of Hadoop in big data storage and processing, as well as the development of cloud computing technology and its importance in data collection, processing and analysis. This paper also discusses the core technologies based on Hadoop and Spark, including keyword retrieval, web crawler and text mining, which together constitute a comprehensive toolkit for analyzing and managing online public opinion.

DIAERESIS: RDF data partitioning and query processing on SPARK

The explosion of the web and the abundance of linked data demand effective and efficient methods for storage, management, and querying. Apache Spark is one of the most widely used engines for big data processing, with more and more systems adopting it for efficient query answering. Existing approaches exploiting Spark for querying RDF data, adopt partitioning techniques for reducing the data that need to be accessed in order to improve efficiency. However, simplistic data partitioning fails, on one hand, to minimize data access and on the other hand to group data usually queried together. This is translated into limited improvement in terms of efficiency in query answering. In this paper, we present DIAERESIS, a novel platform that accepts as input an RDF dataset and effectively partitions it, minimizing data access and improving query answering efficiency. To achieve this, DIAERESIS first identifies the top-k most important schema nodes, i.e., the most important classes, as centroids and distributes the other schema nodes to the centroid they mostly depend on. Then, it allocates the corresponding instance nodes to the schema nodes they are instantiated under. Our algorithm enables fine-tuning of data distribution, significantly reducing data access for query answering. We experimentally evaluate our approach using both synthetic and real workloads, strictly dominating existing state-of-the-art, showing that we improve query answering in several cases by orders of magnitude.

Exploring deep learning models for 4D-STEM-DPC data processing

For the study of magnetic materials at the nanoscale, differential phase contrast (DPC) imaging is a potent tool. With the advancements in direct detector technology, and consequent popularity gain for four-dimensional scanning transmission electron microscopy (4D-STEM), there has been an ongoing development of new and enhanced ways for STEM-DPC big data processing. Conventional algorithms are experimentally tailored, and so in this article we explore how supervised learning with convolutional neural networks (CNN) can be utilized for automated and consistent processing of STEM-DPC data. Two different approaches are investigated, one with direct tracking of the beam with regression analysis, and one where a modified U-net is used for direct beam segmentation as a pre-processing step. The CNNs are trained on experimentally obtained 4D-STEM data, enabling them to effectively handle data collected under similar instrument acquisition parameters. The model outputs are compared to conventional algorithms, particularly in how they process data in the presence of strong diffraction contrast, and how they affect domain wall profiles and width measurement.

Stream ETL framework for twitter-based sentiment analysis: Leveraging big data technologies

Twitter has emerged as a rich source of real-time data, providing valuable insights across various domains such as healthcare and business. Sentiment analysis is crucial in understanding public reactions and sentiments expressed on Twitter, empowering organizations to make informed decisions. However, efficiently analyzing sentiment from social media data, presents a challenge for real-time streaming. Conventional Extract-Transform-Load (ETL) methods, inadequate for this challenge, limit their applicability in processing vast volumes of Twitter data. Big data technologies like Kafka, Spark, Hadoop HDFS, Hive, and HBase have become indispensable in addressing this challenge. To address this, we propose a stream ETL framework for Twitter-based sentiment analysis, leveraging Kafka, Spark, Cassandra, HBase, Hive, and HDFS. Our framework enables data stream processing, bias detection and correction, sentiment-based analysis, and visualization of tweets’ geospatial distribution. We present a set of use case studies to illustrate the applicability of the proposed framework comprises of sentiment classification, and bias detection and correction capability. We also present a comparative study to demonstrate the performance of data streaming processing, analysis, and visualization that has been implemented using multiple big data technologies under different parameter settings. Experimental results demonstrate the framework scalability and trade-off factors of the data stream processing pipeline in the execution of big data processing tasks.

LEVERAGING DATA-DRIVEN TECHNIQUES FOR EFFICIENT DATA MINING IN CLOUD COMPUTING ENVIRONMENTS

The capacity to efficiently use big data and analytics is becoming a critical differentiator for company growth in today's data-driven environment. Using important trends, obstacles, and best practices as a framework, this article investigates how to promote company growth via the use of big data and analytics. An important issue in cloud computing is deciding on an acceptable amount and location of data. Decisions about resource management are based on data aspects and operations in data-driven infrastructure management (DDIM), a novel solution to this problem. It is critical to have a unified system that can manage various forms of big data and the analysis of that data, as well as common knowledge management functions. The approach stated in this research is DD-DM-CCE, or Data-Driven Methods for Efficient Data Mining in Cloud Computing Environments. Improving data using derived information from maximum frequent correlated pattern mining is the main focus of the work. By considering the centrality factor, the DD-DM-CCE method may help choose the best locations to store data in order to reduce access latency. In order to gain a competitive edge, this study offers a cloud-based conceptual framework that can analyze large data in real time and improve decision making. Efficient big data processing is possible with cloud computing infrastructures that can store and analyze massive amounts of data, as this reduces the upfront cost of the massively parallel computer infrastructure needed for big data analytics. According to simulations run on cloud computing, the DD-DM-CCE approach does better than the status quo regarding hit ratio, effective network utilization, and average response time. According to this study, data mining methods are valuable and successful in predicting how consumers will utilize cloud services.

Penerapan Algoritma Clustering untuk Segmentasi Pelanggan E-commerce berdasarkan Data Pembelian dan Aktivitas

In the current digital era, e-commerce has become one of the main pillars of global trade. With the ever-increasing amount of transaction and user activity data, e-commerce companies are faced with the challenge of understanding and managing diverse customer segments more effectively. This paper discusses the application of clustering algorithms for e-commerce customer segmentation based on purchasing data and user activity. The aim of this research is to identify homogeneous customer groups to support more targeted marketing strategies and increase customer retention. The problem faced is how to process big data originating from user transactions and activities on e-commerce platforms, as well as how to identify patterns that are useful for customer segmentation. The data used in this research includes purchase history, frequency of visits, length of time spent on the site, and interactions with certain products. The solution method applied in this research is the clustering algorithm, especially K-Means and DBSCAN. K-Means is used to group data into a predetermined number of clusters based on the Euclidean distance between data points. Meanwhile, DBSCAN is used to identify clusters with high density and separate data that is considered noise or outliers. Data preprocessing is carried out to clean and normalize the data before being applied to the clustering algorithm. Validation of clustering results is carried out using metrics such as Silhouette Score and Davies-Bouldin Index. The research results show that by applying the clustering algorithm, customers can be grouped into several segments that have similar characteristics. For example, we found groups of customers with high purchase frequency but low transaction value, as well as other groups with high transaction value but low purchase frequency. This information is very useful for companies to design more effective marketing strategies, such as special offers for customers with high transaction values ​​or loyalty programs for customers with high purchasing frequency. The conclusion of this research is that clustering algorithms can be a very effective tool in e-commerce customer segmentation, allowing companies to understand customer behavior patterns and develop more targeted and effective marketing strategies. Thus, implementing this method is expected to improve business performance and overall customer satisfaction.

Enabling Adaptive Sampling for Intra-Window Join: Simultaneously Optimizing Quantity and Quality

>Sampling is one of the most widely employed approximations in big data processing. Among various challenges in sampling design, sampling for join is particularly intriguing yet complex. This perplexing problem starts with a classical case where the join of two Bernoulli samples shrinks its output size quadratically and exhibits a strong dependency on the input data, presenting a unique challenge that necessitates adaptive sampling to guarantee both the quantity and quality of the sampled data. The community has made strides in achieving this goal by constructing offline samples and integrating support from indexes or key frequencies. However, when dealing with stream data, due to the need for real-time processing and high-quality analysis, methods developed for processing static data become unavailable. Consequently, a fundamental question arises: Is it possible to achieve adaptive sampling in stream data without relying on offline techniques? To address this problem, we propose FreeSam, which couples hybrid sampling with intra-window join, a key stream join operator. Our focus lies on two widely used metrics: output size, ensuring quantity, and variance, ensuring quality. FreeSam enables adaptability in both the desired quantity and quality of data sampling by offering control on the two-dimensional space spanned by these metrics. Meanwhile, adjustable trade-offs between quality and performance make FreeSam practical for use. Our experiments show that, for every 1% increase in latency limitation, FreeSam can yield a 3.83% increase in the output size while maintaining the level of the estimator's variance. Additionally, we give FreeSam a multi-core implementation and ensure predictability of its latency through both an analytic model and a neural network model. The accuracy of these models is 88.05% and 96.75% respectively.

Efficient Big Data Processing Framework for Real-Time Analytics

In today's fast-changing digital landscape, massive amounts of data are generated every second. This "big data" explosion comes from a variety of sources, including social media, IoT devices, financial systems, and healthcare platforms. Managing, processing, and analyzing this data in real-time has become critical for firms looking to gain actionable insights and preserve a competitive advantage. "Real-time analytics" refers to the practice of analyzing data as it is generated, allowing organizations to respond quickly to changing conditions. An effective big data processing system is required to enable such analytics, ensuring that large amounts of information are not only stored but also processed quickly enough to drive prompt decisions. Big data is often defined by its volume, velocity, and variety, known as the "three Vs" of big data. A vast amount of data can be petabytes or exabytes in size, with velocity referring to the quick rate at which new data is created and diversity indicating the various forms that big data can take, such as structured, unstructured, or semi-structured. Real-time analytics refers to the instantaneous analysis of data as it enters a system, with results produced rapidly enough to impact present activities.

371 Applying precision rangeland grazing management systems in western South Dakota

Abstract Western rangelands represent approximately 58% of the total arable land in the U.S. and are used primarily for cow-calf production, which has the largest greenhouse gas (GHG) emission footprint of all beef production phases. Further, beef production sustainability concerns involve climate mitigation (reducing GHG output) and adaptation (climate-resilient soil-plant-animals). Precision livestock farming (PLF) may help address sustainability concerns by providing innovative solutions and new opportunities for extensive rangeland production. The use of precision measurement and management tools with precision system models can connect measurement data to inform management capabilities. For example, real-time individual weighing and remote sensing (precision measurement tools) can be used to inform and implement dynamic rotational grazing management using virtual fencing technology (precision management; Menendez et al., 2022, 2023). Recent USDA investment in climate-smart agriculture (CSA) commodities has provided over $3 billion in funding to implement practices to reduce GHG emissions in agriculture production, for which PLF may be one approach to accomplish these goals. The overall purpose is to develop commodities produced using NRCS climate-smart practices (e.g., prescribed grazing, 528) and document reductions to provide market opportunities associated with inset GHG. Currently, South Dakota State University is leading two simultaneous programs, which include precision ranching (virtual fencing, precision weighing, and GHG evaluation) and a beef and bison CSA program (GHG evaluation; 7 producer-owned research ranches, representing more than 1.09 million ha). The scale of the CSA program has revealed that current PLF research has only been a prelude to providing the precision tools necessary to successfully implement CSA practices and document their impact through monitoring, measuring, reporting, and verification (MMRV). This presentation will include rangeland grazing case studies that cover the application of virtual fencing, animal location, behavior tracking, remote sensing, precision weighing, feeding, supplementation, and enteric emissions and soil moisture monitoring on extensive rangelands. Case studies will include an overview of big data processing and precision system model development methods. Increasing awareness of available PLF tools for optimizing grazing management, animal performance, productivity, and associated challenges (maintenance, costs) is essential for meeting livestock and sustainability goals. PLF and data-driven approaches aid in the creation of scalable, cost-effective MMRV protocols and models (soil carbon and enteric GHG) for extensive rangelands (20 to 70,000 ha areas) that allow producers to realize potential CSA market incentives. Further, MMRV will likely help guide management decisions by identifying CSA strategies that build climate-resilient landscapes for sustainable livestock production and other environmental synergies (soil microbiome, bird and insect habit, water retention).

Enhancing Data Pipeline Efficiency Using Cloud-Based Big Data Technologies: A Comparative Analysis of AWS and Microsoft Azure

This study conducts a comprehensive comparative analysis of data pipeline efficiency between Amazon Web Services (AWS) Glue and Microsoft Azure Data Factory, two leading cloud-based big data technologies. As organizations increasingly rely on data-driven decision-making, optimizing data pipeline performance is crucial for processing large volumes of information from diverse sources. The research evaluates AWS Glue and Azure Data Factory based on key metrics such as processing speed, scalability, cost efficiency, and fault tolerance, using synthetic datasets ranging from 10GB to 500GB. The results indicate that AWS Glue consistently outperforms Azure Data Factory in processing speed and scalability, particularly for larger data sets, while Azure Data Factory offers greater cost efficiency for smaller workloads. Additionally, AWS Glue demonstrated superior fault tolerance, recovering more quickly from simulated errors compared to Azure Data Factory. These findings provide valuable insights for businesses and data professionals seeking to select the most suitable cloud platform for efficient data pipeline management. This study contributes to the growing body of knowledge on cloud-based big data technologies by offering an up-to-date evaluation of AWS and Azure's data pipeline efficiency, helping organizations optimize their big data processing strategies.

Analyzing Service Models in Cloud Computing and Networking through Big Data Lens

Cloud computing provides small and medium-sized businesses with the assurance of significant data application. The MapReduce programming standard is used for Big Data processing. However, the application of the MapReduce standard generally requires online connected storage space and similar processing capabilities. The computing requirements of MapReduce programming are typically beyond the capabilities of small and medium-sized businesses. In this paper, we briefly discussed cloud computing service models and network infrastructure driven by big data.

Image Processing of Big Data for Plant Diseases of Four Different Plant Categories

Image Processing of Big Data for Plant Diseases of Four Different Plant Categories

Evaluation technologies for assessing drought tolerance of Kimchi cabbage seedlings using hyperspectral imaging and principal component analysis

Conventionally, methods for evaluating the tolerance of kimchi cabbage have primarily relied on visual inspection. The development of evaluation criteria based on hyperspectral imaging may help address the problem of decision-making errors arising from visual inspection. However, to evaluate the tolerance of kimchi cabbage using hyperspectral imaging, not only are hyperspectral big data processing and sophisticated exponential model development required, but also real-time evaluation technology with high field usability. This study presents an evaluation method for extracting vegetation information from hyperspectral data, logically analyzing the extracted data, validating the developed model, and ensuring field utility. The red-edge band between 680 and 700 nm is for evaluating the drought tolerance of Kimchi cabbage. The proposed method allows for assessing the drought tolerance of Kimchi cabbage seedlings within a range of 1 to 30, using a robust index with a variance of approximately 2.25 and a sensitivity index with a variance of around 8.4. Since the proposed nonlinear drought tolerance evaluation method must strongly correlate with the chlorophyll index, its performance was verified using the newly proposed cross-point analyzer. While nonlinear drought evaluation technology can serve as a robust analysis method based on extensive data, it has the disadvantage of requiring a long evaluation time. However, the additionally developed cross-point analyzer can improve the real-time capability of testing. Proper utilization of these two analytical technologies enhances the performance of robust and rapid drought tolerance evaluation for Kimchi cabbage seedlings. Therefore, this study introduces two techniques for non-destructively and quantitatively explaining plant water stress.