Big Data Tools Research Articles

Aquaponics Revolution: Reinforcing performance by means of Apache Spark and Apache Kafka

Many studies provide ways to monitor and manage the aquaponics system, a closed-loop ecosystem that combines hydroponics—growing plants without soil—and aquaculture—raising fish—using cloud computing, the Internet of Things (IoT), and smart sensing technologies. A system operating at peak efficiency yields organic, healthful food with little water and no chemical fertilizer usage, therefore contributing to food security and sustainability. In order to develop our proposed aquaponic system based on Internet of Things technologies, this paper will centralize our endeavor to integrate big data tools, namely Apache Spark and Kafka, into the previous architecture for data collection and processing, as well as for the prediction of any anomalies in the system.

Awaking deep emotional needs of consumers based on big data and emotional analysis

Consumer service and customer satisfaction are related to meeting the needs and demands of consumers while retaining them for lifelong. In an effort towards customer satisfaction, the customer’s emotional needs are taken into consideration for the formulation of future strategies and the development of products. The analysis of such information can be performed with the aid of big data tools. The study highlights the factors of emotional analysis and provided strategies for enhancing emotional marketing. The background of the topic was jotted down, and thematic analysis was conducted for the interpretation of the core variables in the study. The limitations and future scope of such a topic were also laid down.

Разработка инструментов поддержки системы управления проектами цифровой трансформации предприятия на основе больших данных

In the context of the digital economy development and the transition to the data economy, it is necessary to work out iteratively the strategy of digital transformation of the enterprise on an ongoing basis, as well as optimize the digital project portfolio to achieve strategic business goals and obtain new quality and value of the business model and business processes due to the digitalization advantages. The study identifies the directions for the formation of a management system model that correspond to modern trends in enterprise digitalization and intellectualization based on approaches, tools and standards of project management. The conducted theoretical research allowed developing a set of tools that includes a conceptual model of the digital transformation project management system, a methodology for evaluating the effectiveness of the analytical subsystem for managing digital transformation projects based on big data and a generalized model for forming a strategy for managing digital transformation projects of an enterprise. The proposed authors’ methodology for evaluating the effectiveness of the analytical subsystem for managing digital transformation projects based on big data can be used as one of the tools for assessing the digital maturity of an enterprise and the level of use of big data tools in management. This will allow identifying problem areas for the development of a management system based on big data, as well as reasonably forming an analytical infrastructure for digital transformation corresponding to the transition towards the data economy. The developed conceptual model of the digital transformation project management system is aimed at forming an effective portfolio taking into account risks, the external and internal context of the enterprise, and includes a mechanism for choosing an approach (flexible, classic, hybrid) to managing digital transformation projects.

Current challenges and possible big data solutions for the use of web data as a source for official statistics

Web scraping has become popular in scientific research, especially in statistics. Preparing an appropriate IT environment for web scraping is currently not difficult and can be done relatively quickly. Extracting data in this way requires only basic IT skills. This has resulted in the increased use of this type of data, widely referred to as big data, in official statistics. Over the past decade, much work was done in this area both on the national level within the national statistical institutes, and on the international one by Eurostat. The aim of this paper is to present and discuss current problems related to accessing, extracting, and using information from websites, along with the suggested potential solutions. For the sake of the analysis, a case study featuring large-scale web scraping performed in 2022 by means of big data tools is presented in the paper. The results from the case study, conducted on a total population of approximately 503,700 websites, demonstrate that it is not possible to provide reliable data on the basis of such a large sample, as typically up to 20% of the websites might not be accessible at the time of the survey. What is more, it is not possible to know the exact number of active websites in particular countries, due to the dynamic nature of the Internet, which causes websites to continuously change.

Big Data in strategic management accounting

The importance of the influence of Industry 4.0 and the digitalization of the economy on the development of the accounting system in modern conditions has been grounded. The role of Big Data in solving key accounting problems and increasing the relevance of accounting information has been grounded. The reasons for the relevance of using Big Data in management accounting have been analyzed. The problems caused by using Big Data in management accounting have been identified and analyzed. The need to develop management accountants’ abilities and skills in data processing and presentation using Big Data tools has been grounded. The role of big data in ensuring the digital transformation of the enterprise’s strategic management system has been analyzed. Directions for using Big Data in strategic management accounting have been identified and described (innovative development of the system; improvement of strategic control; improvement of particular accounting methods; improvement of accounting methodology; change in the role of strategic management accountants).

Strategies, methods and policy support to improve students' academic achievement and career development across academic periods

This study explores the strategies, methods and impacts of personalized education implemented at different school stages (from kindergarten to high school). Aiming to solve the problem that the traditional education model ignores individual differences among students and based on empirical analysis, this study explores the effect of personalized education on students' academic achievement, career planning and personal development through cross-stage long-term tracking methods. Using cross-stage long-term tracking methods and experimental grouping of the entire K12 school, combined with artificial intelligence and big data tools, quantitative and qualitative data are analyzed to develop new assessment tools and scales. Data analysis will cover descriptive statistics, multivariate analysis, content analysis, and case studies to evaluate the impact of personalized education on students and test multiple hypotheses. The study also highlights the importance of ethical standards to ensure informed consent and privacy protection of participants.

Digital business foresight: Keyword-based analysis and CorEx topic modeling

While previous research develops big data tools to identify weak signals of technological and business changes, we lack an understanding of how to link weak signals with new business concepts and use them to evaluate performance of startups in an automated way. To address this research gap, we analyze the business description of digital companies extracted from the CrunchBase metadata and use a keyword-based text mining approach to collect business-related weak and strong signals for the period 2010–2016. We, then, employ term frequency-inverse document frequency (TF-IDF) to measure the importance and relevance of weak/strong signal keywords for subsequent business development and utilize Correlation Explanation (CorEx) topic modelling to link weak/strong signals with business concepts of startup companies founded from 2017 until 2020 time period. The ANOVA statistical method is also employed to study the relationship between weak/strong signals and the performance of new businesses. The results show no significant difference between startup companies operating in weak and strong signal related business areas in terms of acquiring venture capital funding and estimated revenue range. Moreover, we examine industry and technology profiles of startup companies by using word co-occurrence analysis and study in which sub-sectors they bring digital technologies across selected European regions. Finally, we discuss the implications of the study for strategic planning and investment policy at the firm and regional levels.

Editorial: Are machine learning, AI, and big data tools ready to be used for sustainable development? Challenges, and limitations of current approaches

Editorial: Are machine learning, AI, and big data tools ready to be used for sustainable development? Challenges, and limitations of current approaches

Multi-cluster high performance computing method based on multimodal tensor in enterprise resource planning system

The big data representation and processing method based on multimodal tensors can achieve the fusion representation of different types of data, and perform correlation analysis on the fusion data through multimodal tensor decomposition, providing accurate data services for practical applications. However, in the process of big data analysis and processing based on multimodal tensors, there are still some urgent problems to be solved, such as repeated calculations caused by streaming data, time-consuming calculations caused by large scale, ultra-high energy consumption caused by complex calculations, and poor analysis results caused by complex relationships. This article uses big data tools Hadoop and Spark to build a data mining platform to analyze and study the data of traditional manufacturing enterprises' enterprise resource planning (ERP) systems. In order to improve the accuracy of data analysis, the Stacking fusion framework was also introduced. Furthermore, We transform the data generated by the enterprise into valuable information, making more accurate decisions for enterprise personnel. This article proposes a distributed parallel multi-mode tensor chain decomposition method. In the distributed parallel multi-mode tensor chain decomposition method, a method and theoretical proof for solving tensor chain decomposition are proposed based on the original tensor chain decomposition results and tensor chain decomposition results. The multi kernel clustering algorithm based on K-Means aims to improve the performance of kernel K-clustering by integrating a set of kernels. Deep learning model is also proposed to fuse these multilinear attributes, a clustering method based on weighted average similarity matrix and a clustering method based on optional weighted Euclidean distance were proposed. The distributed parallel multimodal method proposed in this article can effectively achieve tensor chain decomposition, thereby improving the efficiency of big data processing. © 2001 Elsevier Science. All rights reserved

Dynamics of C-reactive protein level after orthopedic surgeries

BackgroundOnly two studies present the results of the dynamics of C-reactive protein (CRP) after a large number of orthopedic surgeries. The aim of the study was to investigate the dynamics of CRP levels and determine the influence of various factors on the CRP level after various orthopedic surgeries using big data tools. MethodsA total of 16042 operated patients were included in the study. ResultsThe tendency of peak values to increase by 2–3 days after surgery with subsequent decrease was the same in all subgroups without surgical site infections (SSI). In patients with SSI, the higher preoperative CRP level changed to a peak on day 3 after surgery with a subsequent drop and increase after day 6 in case of probably unsuccessful sanitation. The peak value of CRP significantly correlates with the duration of surgery (r = 0.1072, p < 0,0001). Older patients tend to exhibit higher CRP value (r = 0.2219, p < 0.001). The peak values of CRP in women were significantly greater than those in men. ConclusionIn cases without SSI, the peak of CRP values occurs on the second or third day after surgery, a secondary increase in CRP levels may indicate a risk of complications. The peak CRP value correlates with the duration of surgery. In clinical practice, relationship between gender, age and the level of CRP should not be one of the indicators influencing the assessment of the risk of a threat to the patient's health, since the results of various studies are contradictory.

Big Data Analytics-based life cycle sustainability assessment for sustainable manufacturing enterprises evaluation

Recently, governments and organizations have repeatedly pressed manufacturing enterprises to promote the ethical and transparent use of natural resources, lessen their negative effects on national and international ecosystems, and safeguard people and the environment. In this context, enhancing the various stages of the product/service life cycle to fulfill sustainability requirements and foster sustainable value creation is a key area of interest for researchers and professionals. This emphasis reflects the growing recognition of the importance of minimizing the environmental impact of products and services, while also maximizing their positive contributions to society, economy, and environment. To this end, this research work addresses how manufacturing enterprises benefit from life cycle sustainability assessment (LCSA) thinking to incorporate the environmental and social criteria into the product/service life cycle strategies. To do so, a novel approach based on environmental priority strategy (EPS) as an LCSA method for impacts monetization coupling with Big Data Analytics (BDA) techniques and tools is developed to evaluate and analyze the manufacturing enterprises’ impacts on the environment and society. Moreover, the developed approach evaluates manufacturing enterprises’ progress toward sustainable development goals (SDGs). Finally, to demonstrate the applicability of the developed approach, a case study from the corporate environmental impact database is used, and the obtained numerical results are analyzed showing its efficiency and added value.

Online Education after the Pandemic: Student Problems and Opportunities Research Using Big Data Tools

This paper presents a scientifically based approach to analyzing large volumes of data from digital traces of students on social networks, which allows you to effectively identify emerging and most discussed problems among students, as well as highlight pain points that provide opportunities for growth, development of universities and improvement of the characteristics of the educational process, support for students etc. The study is based on a thematic analysis of messages published in university communities on the VKontakte social network using big data tools. The study results showed that Russian university students still face a number of challenges, including weak technical infrastructure at universities, a digital divide in access to online education, and negative attitudes towards distance learning.The scientific problem of the study is the contradiction between the existing volume of unstructured data of students’ digital traces in social networks and the lack of a scientifically-based and proven methodological approach to the analysis and evaluation of this voluminous data, which creates obstacles to fundamental research into the relationship between students’ activity in social networks and their satisfaction quality of the educational process. The practical focus is determined in conducting data analysis using big data tools. The findings and evidence-based implications are useful for developing innovative strategies and tools for assessing and supporting students.The results show that the use of big data tools for tracking trends based on digital traces of students on social networks provides highly accurate analytical data and can become the basis for identifying problematic situations in individual universities and the industry as a whole, for data-driven decision-making and management .

SINIRLARIN ÖTESİNDEKİ DİJİTAL İZLER: GÖÇ KRİZİ ÜZERİNE SİSTEMATİK BİR DERLEME

This study aimed to systematically examine the studies conducted and published on immigrants, asylum seekers, and refugees by using big data written in English. Articles were searched on Scholar, The Web of Science, ProQuest, Science Direct, PubMed and Scopus databases. The concept set centered around the concepts of immigration and big data was used in the surveys. In accordance with the PRISMA protocol principles, 49 articles were examined according to the inclusion and exclusion criteria among 258 articles obtained from the relevant databases until the end of December 2022. The reviewed articles were categorized under the headings of “topics examined”, “dataset”, “analyses”, “software used” and “key findings”. The studies provide indications on how to obtain information about this population, which is difficult to reach group especially due to its massiveness, using big data tools. In the findings, it has been seen that studies based on big data on immigrants, asylum seekers and refugees contribute to facilitating the integration of these groups into the target country. Also, it has been revealed that these studies may lead to undesirable results in terms of violating the confidentiality of research groups, producing labeling, and increasing surveillance for these groups. In addition to these, it has been found that these studies have methodological handicaps in terms of representativeness, accuracy, excessive homogenization, and easy generalization. It is thought that the findings of the study will shed light on the international migration and refugee policies to be carried out using big data analysis tools.

Big Data Adoption in Project Management: Insights From French Organizations

Big data have the potential to revolutionize the project management, but it is not clear how. Despite the growing interest, there is a paucity of exploratory research to assess the practice and the impact of big data tools and technologies on project management approaches and practices. To address this gap, in this article, we initially embrace the technological&#x2013;organizational&#x2013;environmental (TOE) framework as the research basis for the adoption process and conducts in-depth interviews with project managers from French organizations in different sectors. The synthesis of interviews reveals that many organizations are still in the early stage of the adoption process, mainly due to a lack of resources, especially expertise. Besides the factors of the three contexts of TOE, the project-level factors are also found to be critical for the adoption of big data in project management. Most of them adopted big data solutions to support them in the conception, definition, and execution phases of project management. Drawing on the findings, this article also provides guidelines to broaden the understanding of big data applications and their role in project management. Based on these results, we present a model with testable propositions and discuss insights that arise for organizations and project managers regarding how to apply big data tools and technologies to create value and overcome the related challenges.

Challenges and prospects in big data analytics: a comprehensive review of developments, hurdles, and future research directions

Big data and business analytics are trends that are positively affecting the business world. This comprehensive review article explores the shifting paradigms and dynamic trends within Big Data Technology (BDT), predominantly for last 5 years, based on an extensive literature review and comparative analysis methodology. It elucidates the transformative influence of big data analytics (BDA) in various sectors, emphasizing the rapid ascendance of cloud computing, Artificial Intelligence (AI) integration, and development of sophisticated analytics tools. The review leverages a wealth of academic literature and market research to underscore the predicted expansion of the big data market. This projected growth indicates the widespread adoption of BDT across industries, with healthcare becoming a significant consumer, motivated by the demand for personalized medicine and improved patient care. The review then navigates emerging trends such as open data usage and ethical concerns surrounding big data, indicating the increasing necessity for stringent guidelines for data use and robust individual data control mechanisms. This is derived from a methodical analysis of recent scholarly articles and industry reports. The article also scrutinizes the evolving definition of "big data" through comparative study of the 3V model and the expanded 7V model in various literature sources, reflecting the evolving nature of data and the unique challenges introduced by modern big data analytics. The review also outlines the challenges for successful implementation of big data projects and highlights the current open research directions of big data analytics. The reviewed areas of big data suggest that good management and manipulation of the large data sets using the techniques and tools of big data can deliver actionable insights that create business values.

A Novel Data Management Scheme in Cloud for Micromachines

In cyber-physical systems (CPS), micromachines are typically deployed across a wide range of applications, including smart industry, smart healthcare, and smart cities. Providing on-premises resources for the storage and processing of huge data collected by such CPS applications is crucial. The cloud provides scalable storage and computation resources, typically through a cluster of virtual machines (VMs) with big data tools such as Hadoop MapReduce. In such a distributed environment, job latency and makespan are highly affected by excessive non-local executions due to various heterogeneities (hardware, VM, performance, and workload level). Existing approaches handle one or more of these heterogeneities; however, they do not account for the varying performance of storage disks. In this paper, we propose a prediction-based method for placing data blocks in virtual clusters to minimize the number of non-local executions. This is accomplished by applying a linear regression algorithm to determine the performance of disk storage on each physical machine hosting a virtual cluster. This allows us to place data blocks and execute map tasks where the data blocks are located. Furthermore, map tasks are scheduled based on VM performance to reduce job latency and makespan. We simulated our ideas and compared them with the existing schedulers in the Hadoop framework. The results show that the proposed method improves MapReduce performance in terms of job latency and makespan by minimizing non-local executions compared to other methods taken for evaluation.

Prevalence of cancer in patients with hypothyroidism: Analysis using big data tools

ObjectiveTo evaluate the frequency of different types of cancer in patients diagnosed with hypothyroidism using big data methodology on the Savana Manager platform. MethodsAn observational, retrospective study was carried out using electronic medical record (EMR) data from the Hospital Universitario Puerta de Hierro Majadahonda (Madrid). Information from the EMRs was extracted using artificial intelligence techniques and analysed using the Savana Manager v3.0 software. Searches were performed using the term “hypothyroidism” and the terms corresponding to the tumours analysed. ResultsOf a total population of 506,749 patients, 23,570 (4.7%) were diagnosed with hypothyroidism. Patients with this diagnosis had a significantly higher frequency of cancer than that found in non-hypothyroid subjects (OR 2.09, 95% confidence interval [CI] 2.01−2.17). This higher frequency was found both in women (OR 1.99, 95% CI 1.90−2.08) and in men (OR 2.83, 95% CI 2.63−3.05). However, this higher frequency of cancer was not observed in hypothyroid patients older than 60 years (OR 0.97, 95% CI 0.92−1.02). Although the frequency of most of the neoplasms studied individually was higher in the population with hypothyroidism, we observed that hypothyroid patients over 60 years of age had a significant decrease in the frequency of prostate, lung, colorectal, and liver cancer. ConclusionData from this hospital cohort suggest that there is a significant association between the diagnosis of hypothyroidism and cancer. However, this association is less evident in hypothyroid patients older than 60 years.

Modified Masking-Based Federated Singular Value Decomposition Method for Fast Anomaly Detection in Smart Grid Systems

The digitalization of production in smart grids entails challenges related to data collection, coordination, privacy protection, and anomaly detection. Machine learning techniques offer effective tools for processing Big Data, but identifying critical system states amidst vast amounts of data remains a challenge. To expedite data analysis, preprocessing through machine learning algorithms becomes essential. This paper introduces the advanced FedSVD algorithm, utilizing Singular Value Decomposition (SVD), which efficiently decomposes large datasets, establishes relationships, and identifies irrelevant data. The algorithm operates in federated machine learning systems, enabling local data processing on private devices while sharing only results with the global learning model. This approach enhances information processing confidentiality and facilitates the exchange of anomaly detection outcomes among network devices. The results of the study demonstrate that the modified FedSVD processing is 5 ms faster on average in comparison to the non-modified one. The proposed FedSVD algorithm calculates anomaly detection with higher accuracy by an average of 1–3% compared to the non-modified FedSVD and SVD ones. The advanced FedSVD algorithm proves to be a decentralized, confidential, and efficient solution for anomaly detection in smart grid systems.

Feature-based real-time distributed denial of service detection in SDN using machine learning and Spark

Recently, software defined networking (SDN) has been deployed extensively in diverse practical domains, providing a new direction in network management by separating the control plane from the data plane. Nevertheless, SDN is vulnerable to distributed denial of service (DDoS) attacks resulting from its centralized controller. Several studies have been suggested to address the DDoS attacks in SDN utilizing machine learning approaches. However, these approaches are resource-intensive and cause performance degradation since they cannot perform effectively in large-scale SDN networks that generate vast traffic statistics. To handle all these challenges, we build a DDoS attack detection model in SDN using Spark as a big data tool to overcome the limitations of conventional data processing methods. Four machine learning algorithms are employed. The decision tree (DT) is elected to be used for real-time deployment based on the performance results, which indicates that it has the best accuracy of 0.936. The model performance is compared with state-of-the-art and shows an overall better performance.

Feature-based real-time distributed denial of service detection in SDN using machine learning and Spark

Recently, software defined networking (SDN) has been deployed extensively in diverse practical domains, providing a new direction in network management by separating the control plane from the data plane. Nevertheless, SDN is vulnerable to distributed denial of service (DDoS) attacks resulting from its centralized controller. Several studies have been suggested to address the DDoS attacks in SDN utilizing machine learning approaches. However, these approaches are resource-intensive and cause performance degradation since they cannot perform effectively in large-scale SDN networks that generate vast traffic statistics. To handle all these challenges, we build a DDoS attack detection model in SDN using Spark as a big data tool to overcome the limitations of conventional data processing methods. Four machine learning algorithms are employed. The decision tree (DT) is elected to be used for real-time deployment based on the performance results, which indicates that it has the best accuracy of 0.936. The model performance is compared with state-of-the-art and shows an overall better performance.