Big Data Architecture Research Articles

ETL in Big Data Architectures: Challenges and Solutions

ETL in Big Data Architectures: Challenges and Solutions

THE ROLE OF BIG DATA IN SYSTEMS ENGINEERING: A REVIEW

This research review explores the transformative intersection of Big Data and Systems Engineering (SE), a convergence that leverages the vast capabilities of Big Data analytics to enhance the design, analysis, and management of complex systems. As the digital era ushers in unprecedented volumes of data, traditional systems engineering approaches are challenged to adapt, necessitating a paradigm shift towards integrating Big Data technologies. This review draws on recent scholarly contributions to highlight the implications, challenges, and innovative solutions emerging at this intersection. The integration of Big Data into Systems Engineering introduces significant opportunities for innovation. By processing and analyzing vast datasets, systems engineers can uncover hidden patterns and insights, leading to more efficient, reliable, and adaptable systems. However, this integration is not without its challenges. Information security emerges as a paramount concern, with the risk of insider attacks necessitating the development of new architectural solutions. Furthermore, the benchmarking and evaluation of Big Data systems present unique challenges due to the diversity of data and workloads involved. To address these challenges, this review examines several key contributions to the field. It discusses a formal framework for designing information systems that handle heterogeneous data, emphasizing the role of ontological models in separating system architecture from its implementation. The review also highlights the importance of security in Big Data systems, proposing a novel architecture for detecting insider attacks through data replication. Additionally, it explores the development of BigDataBench, a benchmark suite that facilitates the comprehensive evaluation of Big Data systems and architectures. Moreover, the review delves into the design of Big Data analytics architectures, focusing on goal-oriented modeling and the resolution of obstacles to quality goal achievement. It also introduces datar, a unified framework for Big Data Management Systems, showcasing a solution that manages Big Data in a pluggable, automatic, and intelligent manner. In conclusion, the intersection of Big Data and Systems Engineering heralds a new era of system design and management. By addressing the inherent challenges and leveraging innovative solutions, this convergence holds the potential to significantly enhance the capabilities of Systems Engineering, driving forward the development of complex, data-driven systems.
 Keywords: Big Data, Systems Engineering, Information Security, Data Management Systems, Benchmarking Big Data Systems.

A Review of Environmental Factors for an Ontology-Based Risk Analysis for Pandemic Spread

Contact tracing is a method used to control the spread of a pandemic. The objectives of this research are to conduct an empirical review and content analysis to identify the environmental factors causing the spread of the pandemic and to propose an ontology-based big data architecture to collect these factors for prediction. No research studies these factors as a whole in pandemic prediction. The research method used was an empirical study and content analysis. The keywords contact tracking, pandemic spread, fear, hygiene measures, government policy, prevention programs, pandemic programs, information disclosure, pandemic economics, and COVID-19 were used to archive studies on the pandemic spread from 2019 to 2022 in the EBSCOHost databases (e.g., Medline, ERIC, Library Information Science & Technology, etc.). The results showed that only 84 of the 588 archived studies were relevant. The risk perception of the pandemic (n = 14), hygiene behavior (n = 7), culture (n = 12), and attitudes of government policies on pandemic prevention (n = 25), education programs (n = 2), business restrictions (n = 2), technology infrastructure, and multimedia usage (n = 24) were the major environmental factors influencing public behavior of pandemic prevention. An ontology-based big data architecture is proposed to collect these factors for building the spread prediction model. The new method overcomes the limitation of traditional pandemic prediction model such as Susceptible-Exposed-Infected-Recovered (SEIR) that only uses time series to predict epidemic trend. The big data architecture allows multi-dimension data and modern AI methods to be used to train the contagion scenarios for spread prediction. It helps policymakers to plan pandemic prevention programs.

A Big Data architecture for early identification and categorization of dark web sites

The dark web has become notorious for its association with illicit activities and there is a growing need for systems to automate the monitoring of this space. This paper proposes an end-to-end scalable architecture for the continuous early identification of new Tor sites and the daily analysis of their content. The solution is built using an Open Source Big Data stack for data serving with Kubernetes, Kafka, Kubeflow, and MinIO, continuously discovering onion addresses in different sources (threat intelligence, code repositories, web-Tor gateways, and Tor repositories), downloading the HTML from Tor and deduplicating the content using MinHash LSH, and categorizing with the BERTopic modeling (SBERT embedding, UMAP dimensionality reduction, HDBSCAN document clustering and c-TF-IDF topic keywords). In 93 days, the system identified 80,049 onion services and characterized 90% of them, addressing the challenge of Tor volatility. A disproportionate amount of repeated content is found, with only 6.1% unique sites. From the HTML files of the dark sites, 31 different low-topics are extracted, manually labeled, and grouped into 11 high-level topics. The five most popular included sexual and violent content, repositories and search engines, carding, cryptocurrencies, and marketplaces. During the experiments, we identified 14 sites with 13,946 clones that shared a suspiciously similar mirroring rate per day, suggesting an extensive common phishing network. Among the related works, this study is the most representative characterization of onion services based on topics to date.

Examining the Trend of Research on Big Data Architecture: Bibliometric Analysis using Scopus Database

Big Data Architecture refers to the infrastructure and processes used to store, manage and analyse large, complex data sets. This topic is widely considered to be challenging due to the ever-increasing volume of data and the need for scalable infrastructure to accommodate it. The aim of this study is to illustrate the current trends in Big Data Architecture through a bibliometric analysis. A comprehensive analysis of Big Data Architecture articles was conducted. Based on the keywords "Big Data Architecture", 151 articles were compiled for further analysis. Between 2014 and 2022, the growth rate of Big Data Architecture literature has increased.

An architecture for COVID-19 analysis and detection using big data, AI, and data architectures.

The COVID-19 epidemic is affecting individuals in many ways and continues to spread all over the world. Vaccines and traditional medical techniques are still being researched. In diagnosis and therapy, biological and digital technology is used to overcome the fear of this disease. Despite recovery in many patients, COVID-19 does not have a definite cure or a vaccine that provides permanent protection for a large number of people. Current methods focus on prevention, monitoring, and management of the spread of the disease. As a result, new technologies for combating COVID-19 are being developed. Though unreliable due to a lack of sufficient COVID-19 datasets, inconsistencies in the datasets availability, non-aggregation of the database because of conflicting data formats, incomplete information, and distortion, they are a step in the right direction. Furthermore, the privacy and confidentiality of people's medical data are only partially ensured. As a result, this research study proposes a novel, cooperative approach that combines big data analytics with relevant Artificial Intelligence (AI) techniques and blockchain to create a system for analyzing and detecting COVID-19 instances. Based on these technologies, the reliability, affordability, and prominence of dealing with the above problems required time. The architecture of the proposed model will analyze different data sources for preliminary diagnosis, detect the affected area, and localize the abnormalities. Furthermore, the blockchain approach supports the decentralization of the central repository so that it is accessible to every stakeholder. The model proposed in this study describes the four-layered architecture. The purpose of the proposed architecture is to utilize the latest technologies to provide a reliable solution during the pandemic; the proposed architecture was sufficient to cover all the current issues, including data security. The layers are unique and individually responsible for handling steps required for data acquisition, storage, analysis, and reporting using blockchain principles in a decentralized P2P network. A systematic review of the technologies to use in the pandemic covers all possible solutions that can cover the issue best and provide a secure solution to the pandemic.

Hierarchical Hybrid Networks for Automatic Pulmonary Blood Vessel Segmentation in Computed Tomography Images.

Pulmonary arterial hypertension (PAH) is considered the third most common cardiovascular disease after coronary heart disease and hypertension. The diagnosis of PAH is mainly based on the comprehensive judgment of computed tomography and other medical image examinations. Medical image processing based on deep learning has achieved significant success. However, the data belongs to the patient's privacy; therefore, the medical institutions as data custodians have the responsibility to protect the security of their data privacy. This situation makes medical institutions face a dilemma when building data-driven deep learning-assisted medical diagnosis methods. On the one hand, they need to pursue more high-quality data based on Big Data architecture for deep learning; on the other hand, they need to protect patient privacy to avoid data leakage. In response to the above challenges, we propose a hierarchical hybrid automatic segmentation model for pulmonary blood vessels based on local learning and federated learning approaches for segmenting the pulmonary blood vessels. The experiments prove the proposal could automatically segment the vessels from the original CT. It also indicates that the model based on a federated learning approach can achieve impressive performance under the premise of protecting data privacy for Big Data.

A Big Data Architecture for Digital Twin Creation of Railway Signals Based on Synthetic Data

A Big Data Architecture for Digital Twin Creation of Railway Signals Based on Synthetic Data

Supply Chain Management for Pre-Teacher Preparation of Higher Education in Thailand Model

The article is in the second phase of research is about “the big data architecture for pre-teacher preparation supply chain with prescriptive analytics of higher education in Thailand”. The objectives of the study were (1) to study the pre-teacher preparation supply chain in Thailand, (2) to develop a model the big data system for the pre-teacher preparation supply chain management with prescriptive analytics of higher education in Thailand, (3) to design the big data architecture for the pre-teacher preparation supply chain management with prescriptive analytics of higher education in Thailand, (4) to develop the big data system for the pre-teacher preparation supply chain prescriptive of higher education in Thailand, (5) to assess accuracy of the predictive analytics in the pre-teachers needs of higher education in Thailand, and (6) to assess accuracy of the prescriptive analytics in the pre-teacher preparation of higher education in Thailand. In the study, the research procedures were divided into 6 steps according to the objectives, and all steps were assessed on the system suitability by 25 experts throughout the procedures.

Contemporary High-Performance Computing for Big Data Applications

High-performance computing (HPC) involves leveraging parallel data processing to enhance computer performance and handle difficult tasks. HPC meets these aims by pooling computing capacity, enabling efficient, reliable, and prompt execution of even complex programs according to user demands and expectations. The rapid growth of HPDA in many sectors has led to the extension of the HPC market into new territory. HPC as well as Big Data systems differ not just in terms of technology but also in ecosystems. Extensive research in this sector has led to the emergence of various Big Data analytics models in recent years. As Big Data analytics spreads across several fields, new challenges about the usefulness of analytical paradigms also emerge. This article discusses the key analytical models, as well as the difficulties and challenges associated with high-performance data analytics. This research work aims to identify the factors influencing the integration of HPC with big data, including present and future trends. The study also proposes an architecture for big data with HPC convergence based on design principles.

Intelligent biomedical image classification in a big data architecture using metaheuristic optimization and gradient approximation

Intelligent biomedical image classification in a big data architecture using metaheuristic optimization and gradient approximation

A big data association rule mining based approach for energy building behaviour analysis in an IoT environment

The enormous amount of data generated by sensors and other data sources in modern grid management systems requires new infrastructures, such as IoT (Internet of Things) and Big Data architectures. This, in combination with Data Mining techniques, allows the management and processing of all these heterogeneous massive data in order to discover new insights that can help to reduce the energy consumption of the building. In this paper, we describe a developed methodology for an Internet of Things (IoT) system based on a robust big data architecture. This innovative approach, combined with the power of Spark algorithms, has been proven to uncover rules representing hidden connections and patterns in the data extracted from a building in Bucharest. These uncovered patterns were essential for improving the building’s energy efficiency.

Semantic segmentation and classification of polycystic ovarian disease using attention UNet, Pyspark, and ensemble learning model

AbstractOvarian abnormality like polycystic ovarian disease (PCOD) is one of the most common diseases among women worldwide. PCOD not only has an impact on infertility but also hurts the psychological well‐being of women affecting their quality of life. In this study, a two‐class pattern learning problem is designed for the classification of PCOD. In total, 37 clinical parameters and abdominal ultrasound images of women are collected under the proper ethical protocol. Using only clinical data, an accuracy of 93.7% is obtained using Random Forest as the classifier which is further improved to 95.54% by using a Randomized Search CV during Random Forest classification. The ultrasound images are classified using the proposed Attention‐UNet architecture and a mean Dice score of 0.945 is obtained indicating more accurate segmentation. The segmented images are passed through the state‐of‐the‐art EfficientNet B6 for the classification of PCOS and non‐PCOS and recorded an accuracy of 95.47%. Using big data architecture Pyspark, the performance is further enhanced to 96.8% and 96.3% for clinical and ultrasound images respectively along with the reduced computational speed. The results of these classifiers are then used to create metadata and a customized Artificial Neural Network is applied for the final prediction of PCOD and non‐PCOD. From the results, it can be seen that the stacking model outperformed with an accuracy of 98.12% when compared to the single classifier. Our proposed method has very good performance with less computation, contributing a new architecture to evaluate PCOD and hence helping to improve the wellness of women.

A Model of Big Data Architecture on the Base of FIWARE Components

The intensive development of hardware and software tools provides many opportunities for solving various real-world tasks, such as working with large data sets. This, in turn, is a powerful incentive and opportunity for the development of different approaches and tools for solving them. After discussing several well-known BigData architectures a model of a BigData architecture is designed. The architecture is organized in three sections: edge section, processing-and-storage section and application section. The model will be implemented using FIWARE components.

Web-based E-learning in Elementary School: A Systematic Literature Review

This article presents literature review on web-based e-learning in elementary school in the latest literature. SLR method and PRISMA protocol with the stages of identification, screening, eligibility, inclusion, and abstraction, data analysis assisted by the Publish or Perish 7 application, VOSviewer, and NVIVO 12 Plus. The results of searching for articles on Scopus through the Publish or Perish 7 application are 507. Then the articles were filtered according to compatible themes into 50 articles. The topic findings are web-based e-learning, elementary school, the impact of web-based e-learning and web-based e-learning concept, academic performance, teaching/learning strategies, online learning, Covid-19, HPC database, web-based applications, distance learning, 3D visualization, automation, strategic learning, semantic web, technology, education, linguistic content, big data architecture, learning setting, e-readiness, linguistic content, STEM, etc., that are directly or indirectly connected. The 50 articles were analyzed according to the specified topics through the NVIVO 12 Plus application, and the results were described according to the research questions. The findings in this article explain that web-based e-learning integrates pedagogy and technology and becomes part of digital multimedia implemented in e-learning, blended learning, and face-to-face that impacts elementary school students and teachers directly or indirectly. Future research needs to explore web-based e-learning in schools that is current, safe, and needed by students and teachers.

Big data: "Navigating the Hadoop ecosystem: Unraveling the potential of big data”

Big data refers to extremely large data sets that can be analyzed to reveal patterns, trends, and associations, particularly those relating to hu-man behavior and interactions. This data is too large and complex to be processed using traditional data processing methods, necessitating the use of specialized software and systems to manage and analyze it. This paper discusses about the Big Data Architecture, Hadoop Ecosystem, HDFS, Map reduce, Yarn, Hive, and many other components of Hadoop ecosystem.

Performance Analysis of Lambda Architecture-Based Big-Data Systems on Air/Ground Surveillance Application with ADS-B Data.

This study introduces a novel methodology designed to assess the accuracy of data processing in the Lambda Architecture (LA), an advanced big-data framework qualified for processing streaming (data in motion) and batch (data at rest) data. Distinct from prior studies that have focused on hardware performance and scalability evaluations, our research uniquely targets the intricate aspects of data-processing accuracy within the various layers of LA. The salient contribution of this study lies in its empirical approach. For the first time, we provide empirical evidence that validates previously theoretical assertions about LA, which have remained largely unexamined due to LA's intricate design. Our methodology encompasses the evaluation of prospective technologies across all levels of LA, the examination of layer-specific design limitations, and the implementation of a uniform software development framework across multiple layers. Specifically, our methodology employs a unique set of metrics, including data latency and processing accuracy under various conditions, which serve as critical indicators of LA's accurate data-processing performance. Our findings compellingly illustrate LA's "eventual consistency". Despite potential transient inconsistencies during real-time processing in the Speed Layer (SL), the system ultimately converges to deliver precise and reliable results, as informed by the comprehensive computations of the Batch Layer (BL). This empirical validation not only confirms but also quantifies the claims posited by previous theoretical discourse, with our results indicating a 100% accuracy rate under various severe data-ingestion scenarios. We applied this methodology in a practical case study involving air/ground surveillance, a domain where data accuracy is paramount. This application demonstrates the effectiveness of the methodology using real-world data-intake scenarios, therefore distinguishing this study from hardware-centric evaluations. This study not only contributes to the existing body of knowledge on LA but also addresses a significant literature gap. By offering a novel, empirically supported methodology for testing LA, a methodology with potential applicability to other big-data architectures, this study sets a precedent for future research in this area, advancing beyond previous work that lacked empirical validation.

Preface

The 2023 6th International Symposium on Big Data and Applied Statistics (ISBDAS 2023) was held in Shanghai, China from March 10th to 12th, 2023. The purpose of ISBDAS 2023 was to discuss the present status and future perspectives of big data and applied statistics technology. The following topics were selected in order to cover most of the scientific program and highlight an area where new ideas have emerged over recent years:(1) Big Data Mining and Visualization;(2) Big Data Security and Privacy;(3) Computational Mathematics;(4) Probability Theory;(5) Statistical Methodology, etc.There were about 60 participants. In addition to the keynote speeches, we had many talks selected from the contributed papers. There were invited, oral, and poster sessions. This proceedings contains the papers presented in invited and selected talks together with those presented in poster sessions.At the Conference, the participants discussed a wide range of issues affecting the theoretical and computational aspects of research problems in the big data and applied statistics technology. Besides, we had a comprehensive overview of this fascinating field and of future scenarios thanks to the participation of leaders of the most important projects. Professor Zhihui Lv from Fudan University, China performed a keynote speech and shared with us his own research experience and professional view. His main research fields include: network multimedia technology, cloud computing and service computing technology, big data architecture, edge computing, blockchain distributed system, network security technology, cloud data center management technology. And still another speaker was Jingsong Yang, a Professor at Shanghai Jiao Tong University, China. As a third-level researcher, he has undertaken more than 20 national, provincial and international cooperation projects, and has made many innovative achievements in the mechanism of marine microwave remote sensing, remote sensing technology and application, and remote sensing equipment development.We wish to acknowledge our host institutions, sponsors, and the many people whose work made this Conference possible. We would like to express our gratitude to the members of the International Technical Program Committee and those of the Organizing Committee for their efforts which made this Workshop successful. Particularly we would like to present our great thanks to the editors and other colleagues of Journal of Physics: Conference Series for their patience and kind assistance in publishing this paper volume.The Committee of ISBDAS 2023list of Committee Member is available in this pdf.

Design an efficient data driven decision support system to predict flooding by analysing heterogeneous and multiple data sources using Data Lake

Floods are the most common natural disaster in several countries throughout the world. Flooding has a major impact on people's lives and livelihoods. The impact of flood disasters on human lives can be mitigated by developing effective flood forecasting and prediction models. The majority of flood prediction models do not take all flood-causing factors into account when they are designed. It is difficult to collect and handle some of these flood-causing variables since they are heterogeneous in nature. This paper presents a new big data architecture called Data Lake, which can ingest and store all important flood-causing heterogeneous data sources in their raw format for machine learning model creation. The statistical relevance of important flood producing factors on flood prediction outcome is determined utilizing inferential statistical approaches. The outcome of this research is to create flood warning systems that can alert the public and government officials so that they can make decisions in the event of a severe flood, reducing socioeconomic loss.•Flood causing factors are from heterogeneous sources, so there is no big data architecture for handling variety of data sources.•To provide data architectural solution using data lake for collecting and analysing heterogeneous flood causing factors.•Uses inferential statistical approach to determine importance of different flood causing factors in design of efficient flood prediction models.

TRENDS AND APPLICATIONS OF BIG DATA IN EDUCATION

Big data technologies have facilitated innovation across various industries, including healthcare, technology, and education. The utilization of this innovation is now required within the education sector at all levels. It is a progressing framework named as "Education 4.0" to accommodate the numerous needs of this sector. Specifically, this article first describes the architecture of big data and its properties compatible to Education sector . The process of utilizing large educational data and data mining techniques in education are explained. To find solutions to diverse educational problems, an ever-increasing variety of mining tools have been assembled. This study also provides insights into these techniques and other data sources for gathering educational data. Several applications have been developed to promote the use of big data techniques in higher education and highlight its benefits for the sector, but there are still a number of issues that need to be considered. We will highlight the obstacles of using big data techniques in higher education