Big Data Architecture Research Articles

Making Sense of Big Data in Intelligent Transportation Systems: Current Trends, Challenges and Future Directions

Intelligent Transportation Systems (ITS) generate massive amounts of Big Data through both sensory and non-sensory platforms. The data support batch processing as well as stream processing, which are essential for reliable operations on the roads and connected vehicles in ITS. Despite the immense potential of Big Data intelligence in ITS, autonomous vehicles are largely confined to testing and trial phases. The research community is working tirelessly to improve the reliability of ITS by designing new protocols, standards and connectivity paradigms. In the recent past, several surveys have been conducted that focus on Big Data Intelligence for ITS, yet none of them have comprehensively addressed the fundamental challenges hindering the widespread adoption of autonomous vehicles on the roads. Our survey aims to help readers better understand the technological advancements by delving deep into Big Data architecture, focusing on data acquisition, data storage and data visualization. We reviewed sensory and non-sensory platforms for data acquisition, data storage repositories for archival and retrieval of large datasets, and data visualization for presenting the processed data in an interactive and comprehensible format. To this end, we discussed the current research progress by comprehensively covering the literature and highlighting challenges that urgently require the attention of research community. Based on the concluding remarks, we argued that these challenges hinder the widespread presence of autonomous vehicles on the roads. Understanding these challenges is important for a more informed discussion on the future of self-driven technology. Moreover, we acknowledge that these challenges not only affect individual layers but also impact the functionality of subsequent layers. Finally, we outline our future work that explores how resolving these challenges could enable the realization of innovations such as smart charging systems on the roads and data centers on wheels.

E-solar: a tool for solar resource assessment based on a Big Data architecture in a PySpark environment

Con el tiempo, diversos investigadores han creado modelos matemáticos, estadísticos y predictivos para evaluar el recurso solar. Sin embargo, su implementación en herramientas técnicas limita su utilización por usuarios no técnicos. Además, el procesamiento de datos para estimar la radiación solar suele requerir hardware potente. Este estudio presenta una herramienta basada en Big data que utiliza archivos planos e imágenes de satélite para estimar la radiación solar en Colombia. Se desarrolló un modelo con técnicas de aprendizaje automático y varios lenguajes de programación. Se ejecuta en MapR, una distribución del ecosistema Hadoop con un amplio conjunto de capacidades Big data y emplea la API de PySpark para procesar datos en paralelo en un clúster de computadoras. La herramienta E-Solar implementada en un servidor web fue evaluada por profesionales del sector energético. Se analizó la usabilidad, se verificó la conformidad con estándares de programación recientes y se identificaron perfiles de usuarios interesados. Los datos de radiación solar generados por la herramienta son fundamentales para proyectos solares. Además, la herramienta proporciona apoyo a investigadores y organizaciones; y facilita la toma de decisiones en la implementación de sistemas fotovoltaicos al ofrecer información relevante sobre el comportamiento del recurso solar en Colombia.

Evaluating external load responses to cumulative playing time and position in the European Handball Federation Women’s Euro 2022 through an IoT and Big Data architecture approach

Evaluating external load responses to cumulative playing time and position in the European Handball Federation Women’s Euro 2022 through an IoT and Big Data architecture approach

Enhancing financial portfolio management with predictive analytics and scalable data modeling techniques

The financial industry faces an increasingly complex landscape for portfolio management, where data-driven insights are crucial for optimizing asset allocation and managing risk. This paper explores the integration of predictive analytics and scalable data modeling techniques in enhancing financial portfolio management. By leveraging machine learning algorithms, big data architectures, and real-time data processing, predictive analytics can forecast asset trends, detect market anomalies, and assess portfolio risk with high precision. We evaluate several predictive models, including time-series forecasting, neural networks, and ensemble methods, for their efficacy in financial prediction. The study also discusses the role of scalable data modeling frameworks, such as Apache Spark and cloud-based data lakes, in handling vast volumes of unstructured data across different markets and asset classes. Findings indicate that predictive analytics, when paired with robust data models, can deliver real-time, actionable insights, enhancing decision-making for fund managers and institutional investors. Furthermore, this paper highlights best practices for implementing scalable models in financial institutions, addressing challenges like data latency, model interpretability, and system scalability. By adopting these advanced analytics frameworks, portfolio managers can achieve improved risk-adjusted returns, better asset diversification, and enhanced adaptability to market volatility.

Human interaction with structure in designing big data architecture

Background: There is an increasing interest in big data. However, challenges shape and affect the gathering, retrieval, use and management of big data in many organisations. Some of the challenges are linked to a lack of architecture that is specific to big data. Attempts have been made from both business and academic fronts, yet the challenges persist. The challenges are attributed to a lack of an understanding of the factors that influence the design of architecture for big data in an organisation.Objectives: The study aims to propose big data architecture for enterprises.Method: We employed the qualitative method, using document analysis to gather data. Activity theory (AT) was employed in the analysis of the data.Results: From the analysis, governance, interactions, relationships and allocative were found to be the factors that influence the design of big data architecture. An interpretation was conducted following the inductive reasoning approach to gain a deeper insight of how the factors manifest themselves.Conclusion: Big data architecture is proposed. The architecture is intended to address some of the challenges encountered in gathering, retrieving, using or managing big data in organisations.Contribution: This study advances our understanding of the complex interplay of factors influencing the architecture of big data. Applying AT, the study fortifies our understanding of complex interactions between humans and big data including the architecture design.

EHKP-Res: An explainable dynamic security solution for medical healthcare

An increasing number of medical institutions and patients are adopting the practice of storing their data on medical cloud servers, which greatly facilitates the sharing of medical data. However, when many nodes are connected to the medical big data architecture, the overall system's security can be compromised. To address this, an explainable dynamic security protection scheme for medical healthcare, named EHKP-Res, has been proposed. This scheme aggregates data relevance features and integrates them with a deep residual network to predict the behavior of medical staff, addressing the issue of sudden changes in their behavioral credibility. To tackle the black-box problem, a model post-interpretation scheme using Bayesian networks to generate perturbed datasets has been proposed. It calculates sample weights and performs nonlinear fitting, deriving model interpretation through the eigenvalues of the instances. Experimental results show that the proposed Hidden KP-ABE method reduces time overhead by 14.95% compared to other access control methods and achieves a prediction accuracy of 98.34% for doctor trust metrics, effectively preventing malicious behavior by doctors.

ETL in Big Data Architectures: Challenges and Solutions

ETL in Big Data Architectures: Challenges and Solutions

THE ROLE OF BIG DATA IN SYSTEMS ENGINEERING: A REVIEW

This research review explores the transformative intersection of Big Data and Systems Engineering (SE), a convergence that leverages the vast capabilities of Big Data analytics to enhance the design, analysis, and management of complex systems. As the digital era ushers in unprecedented volumes of data, traditional systems engineering approaches are challenged to adapt, necessitating a paradigm shift towards integrating Big Data technologies. This review draws on recent scholarly contributions to highlight the implications, challenges, and innovative solutions emerging at this intersection. The integration of Big Data into Systems Engineering introduces significant opportunities for innovation. By processing and analyzing vast datasets, systems engineers can uncover hidden patterns and insights, leading to more efficient, reliable, and adaptable systems. However, this integration is not without its challenges. Information security emerges as a paramount concern, with the risk of insider attacks necessitating the development of new architectural solutions. Furthermore, the benchmarking and evaluation of Big Data systems present unique challenges due to the diversity of data and workloads involved. To address these challenges, this review examines several key contributions to the field. It discusses a formal framework for designing information systems that handle heterogeneous data, emphasizing the role of ontological models in separating system architecture from its implementation. The review also highlights the importance of security in Big Data systems, proposing a novel architecture for detecting insider attacks through data replication. Additionally, it explores the development of BigDataBench, a benchmark suite that facilitates the comprehensive evaluation of Big Data systems and architectures. Moreover, the review delves into the design of Big Data analytics architectures, focusing on goal-oriented modeling and the resolution of obstacles to quality goal achievement. It also introduces datar, a unified framework for Big Data Management Systems, showcasing a solution that manages Big Data in a pluggable, automatic, and intelligent manner. In conclusion, the intersection of Big Data and Systems Engineering heralds a new era of system design and management. By addressing the inherent challenges and leveraging innovative solutions, this convergence holds the potential to significantly enhance the capabilities of Systems Engineering, driving forward the development of complex, data-driven systems.
 Keywords: Big Data, Systems Engineering, Information Security, Data Management Systems, Benchmarking Big Data Systems.

A Review of Environmental Factors for an Ontology-Based Risk Analysis for Pandemic Spread

Contact tracing is a method used to control the spread of a pandemic. The objectives of this research are to conduct an empirical review and content analysis to identify the environmental factors causing the spread of the pandemic and to propose an ontology-based big data architecture to collect these factors for prediction. No research studies these factors as a whole in pandemic prediction. The research method used was an empirical study and content analysis. The keywords contact tracking, pandemic spread, fear, hygiene measures, government policy, prevention programs, pandemic programs, information disclosure, pandemic economics, and COVID-19 were used to archive studies on the pandemic spread from 2019 to 2022 in the EBSCOHost databases (e.g., Medline, ERIC, Library Information Science & Technology, etc.). The results showed that only 84 of the 588 archived studies were relevant. The risk perception of the pandemic (n = 14), hygiene behavior (n = 7), culture (n = 12), and attitudes of government policies on pandemic prevention (n = 25), education programs (n = 2), business restrictions (n = 2), technology infrastructure, and multimedia usage (n = 24) were the major environmental factors influencing public behavior of pandemic prevention. An ontology-based big data architecture is proposed to collect these factors for building the spread prediction model. The new method overcomes the limitation of traditional pandemic prediction model such as Susceptible-Exposed-Infected-Recovered (SEIR) that only uses time series to predict epidemic trend. The big data architecture allows multi-dimension data and modern AI methods to be used to train the contagion scenarios for spread prediction. It helps policymakers to plan pandemic prevention programs.

A Big Data architecture for early identification and categorization of dark web sites

The dark web has become notorious for its association with illicit activities and there is a growing need for systems to automate the monitoring of this space. This paper proposes an end-to-end scalable architecture for the continuous early identification of new Tor sites and the daily analysis of their content. The solution is built using an Open Source Big Data stack for data serving with Kubernetes, Kafka, Kubeflow, and MinIO, continuously discovering onion addresses in different sources (threat intelligence, code repositories, web-Tor gateways, and Tor repositories), downloading the HTML from Tor and deduplicating the content using MinHash LSH, and categorizing with the BERTopic modeling (SBERT embedding, UMAP dimensionality reduction, HDBSCAN document clustering and c-TF-IDF topic keywords). In 93 days, the system identified 80,049 onion services and characterized 90% of them, addressing the challenge of Tor volatility. A disproportionate amount of repeated content is found, with only 6.1% unique sites. From the HTML files of the dark sites, 31 different low-topics are extracted, manually labeled, and grouped into 11 high-level topics. The five most popular included sexual and violent content, repositories and search engines, carding, cryptocurrencies, and marketplaces. During the experiments, we identified 14 sites with 13,946 clones that shared a suspiciously similar mirroring rate per day, suggesting an extensive common phishing network. Among the related works, this study is the most representative characterization of onion services based on topics to date.

Examining the Trend of Research on Big Data Architecture: Bibliometric Analysis using Scopus Database

Big Data Architecture refers to the infrastructure and processes used to store, manage and analyse large, complex data sets. This topic is widely considered to be challenging due to the ever-increasing volume of data and the need for scalable infrastructure to accommodate it. The aim of this study is to illustrate the current trends in Big Data Architecture through a bibliometric analysis. A comprehensive analysis of Big Data Architecture articles was conducted. Based on the keywords "Big Data Architecture", 151 articles were compiled for further analysis. Between 2014 and 2022, the growth rate of Big Data Architecture literature has increased.

An architecture for COVID-19 analysis and detection using big data, AI, and data architectures.

The COVID-19 epidemic is affecting individuals in many ways and continues to spread all over the world. Vaccines and traditional medical techniques are still being researched. In diagnosis and therapy, biological and digital technology is used to overcome the fear of this disease. Despite recovery in many patients, COVID-19 does not have a definite cure or a vaccine that provides permanent protection for a large number of people. Current methods focus on prevention, monitoring, and management of the spread of the disease. As a result, new technologies for combating COVID-19 are being developed. Though unreliable due to a lack of sufficient COVID-19 datasets, inconsistencies in the datasets availability, non-aggregation of the database because of conflicting data formats, incomplete information, and distortion, they are a step in the right direction. Furthermore, the privacy and confidentiality of people's medical data are only partially ensured. As a result, this research study proposes a novel, cooperative approach that combines big data analytics with relevant Artificial Intelligence (AI) techniques and blockchain to create a system for analyzing and detecting COVID-19 instances. Based on these technologies, the reliability, affordability, and prominence of dealing with the above problems required time. The architecture of the proposed model will analyze different data sources for preliminary diagnosis, detect the affected area, and localize the abnormalities. Furthermore, the blockchain approach supports the decentralization of the central repository so that it is accessible to every stakeholder. The model proposed in this study describes the four-layered architecture. The purpose of the proposed architecture is to utilize the latest technologies to provide a reliable solution during the pandemic; the proposed architecture was sufficient to cover all the current issues, including data security. The layers are unique and individually responsible for handling steps required for data acquisition, storage, analysis, and reporting using blockchain principles in a decentralized P2P network. A systematic review of the technologies to use in the pandemic covers all possible solutions that can cover the issue best and provide a secure solution to the pandemic.

Hierarchical Hybrid Networks for Automatic Pulmonary Blood Vessel Segmentation in Computed Tomography Images.

Pulmonary arterial hypertension (PAH) is considered the third most common cardiovascular disease after coronary heart disease and hypertension. The diagnosis of PAH is mainly based on the comprehensive judgment of computed tomography and other medical image examinations. Medical image processing based on deep learning has achieved significant success. However, the data belongs to the patient's privacy; therefore, the medical institutions as data custodians have the responsibility to protect the security of their data privacy. This situation makes medical institutions face a dilemma when building data-driven deep learning-assisted medical diagnosis methods. On the one hand, they need to pursue more high-quality data based on Big Data architecture for deep learning; on the other hand, they need to protect patient privacy to avoid data leakage. In response to the above challenges, we propose a hierarchical hybrid automatic segmentation model for pulmonary blood vessels based on local learning and federated learning approaches for segmenting the pulmonary blood vessels. The experiments prove the proposal could automatically segment the vessels from the original CT. It also indicates that the model based on a federated learning approach can achieve impressive performance under the premise of protecting data privacy for Big Data.

A Big Data Architecture for Digital Twin Creation of Railway Signals Based on Synthetic Data

A Big Data Architecture for Digital Twin Creation of Railway Signals Based on Synthetic Data

Supply Chain Management for Pre-Teacher Preparation of Higher Education in Thailand Model

The article is in the second phase of research is about “the big data architecture for pre-teacher preparation supply chain with prescriptive analytics of higher education in Thailand”. The objectives of the study were (1) to study the pre-teacher preparation supply chain in Thailand, (2) to develop a model the big data system for the pre-teacher preparation supply chain management with prescriptive analytics of higher education in Thailand, (3) to design the big data architecture for the pre-teacher preparation supply chain management with prescriptive analytics of higher education in Thailand, (4) to develop the big data system for the pre-teacher preparation supply chain prescriptive of higher education in Thailand, (5) to assess accuracy of the predictive analytics in the pre-teachers needs of higher education in Thailand, and (6) to assess accuracy of the prescriptive analytics in the pre-teacher preparation of higher education in Thailand. In the study, the research procedures were divided into 6 steps according to the objectives, and all steps were assessed on the system suitability by 25 experts throughout the procedures.

Contemporary High-Performance Computing for Big Data Applications

High-performance computing (HPC) involves leveraging parallel data processing to enhance computer performance and handle difficult tasks. HPC meets these aims by pooling computing capacity, enabling efficient, reliable, and prompt execution of even complex programs according to user demands and expectations. The rapid growth of HPDA in many sectors has led to the extension of the HPC market into new territory. HPC as well as Big Data systems differ not just in terms of technology but also in ecosystems. Extensive research in this sector has led to the emergence of various Big Data analytics models in recent years. As Big Data analytics spreads across several fields, new challenges about the usefulness of analytical paradigms also emerge. This article discusses the key analytical models, as well as the difficulties and challenges associated with high-performance data analytics. This research work aims to identify the factors influencing the integration of HPC with big data, including present and future trends. The study also proposes an architecture for big data with HPC convergence based on design principles.

Intelligent biomedical image classification in a big data architecture using metaheuristic optimization and gradient approximation

Medical imaging has experienced significant development in contemporary medicine and can now record a variety of biomedical pictures from patients to test and analyze the illness and its severity. Computer vision and artificial intelligence may outperform human diagnostic ability and uncover hidden information in biomedical images. In healthcare applications, fast prediction and reliability are of the utmost importance parameters to assure the timely detection of disease. The existing systems have poor classification accuracy, and higher computation time and the system complexity is higher. Low-quality images might impact the processing method, leading to subpar results. Furthermore, extensive preprocessing techniques are necessary for achieving accurate outcomes. Image contrast is one of the most essential visual parameters. Insufficient contrast may present many challenges for computer vision techniques. Traditional contrast adjustment techniques may not be adequate for many applications. Occasionally, these technologies create photos that lack crucial information. The primary contribution of this work is designing a Big Data Architecture (BDA) to improve the dependability of medical systems by producing real-time warnings and making precise forecasts about patient health conditions. A BDA-based Bio-Medical Image Classification (BDA-BMIC) system is designed to detect the illness of patients using Metaheuristic Optimization (Genetic Algorithm) and Gradient Approximation to improve the biomedical image classification process. Extensive tests are conducted on publicly accessible datasets to demonstrate that the suggested retrieval and categorization methods are superior to the current methods. The suggested BDA-BMIC system has average detection accuracy of 94.6% and a sensitivity of 97.3% in the simulation analysis.

A big data association rule mining based approach for energy building behaviour analysis in an IoT environment

The enormous amount of data generated by sensors and other data sources in modern grid management systems requires new infrastructures, such as IoT (Internet of Things) and Big Data architectures. This, in combination with Data Mining techniques, allows the management and processing of all these heterogeneous massive data in order to discover new insights that can help to reduce the energy consumption of the building. In this paper, we describe a developed methodology for an Internet of Things (IoT) system based on a robust big data architecture. This innovative approach, combined with the power of Spark algorithms, has been proven to uncover rules representing hidden connections and patterns in the data extracted from a building in Bucharest. These uncovered patterns were essential for improving the building’s energy efficiency.

Semantic segmentation and classification of polycystic ovarian disease using attention UNet, Pyspark, and ensemble learning model

AbstractOvarian abnormality like polycystic ovarian disease (PCOD) is one of the most common diseases among women worldwide. PCOD not only has an impact on infertility but also hurts the psychological well‐being of women affecting their quality of life. In this study, a two‐class pattern learning problem is designed for the classification of PCOD. In total, 37 clinical parameters and abdominal ultrasound images of women are collected under the proper ethical protocol. Using only clinical data, an accuracy of 93.7% is obtained using Random Forest as the classifier which is further improved to 95.54% by using a Randomized Search CV during Random Forest classification. The ultrasound images are classified using the proposed Attention‐UNet architecture and a mean Dice score of 0.945 is obtained indicating more accurate segmentation. The segmented images are passed through the state‐of‐the‐art EfficientNet B6 for the classification of PCOS and non‐PCOS and recorded an accuracy of 95.47%. Using big data architecture Pyspark, the performance is further enhanced to 96.8% and 96.3% for clinical and ultrasound images respectively along with the reduced computational speed. The results of these classifiers are then used to create metadata and a customized Artificial Neural Network is applied for the final prediction of PCOD and non‐PCOD. From the results, it can be seen that the stacking model outperformed with an accuracy of 98.12% when compared to the single classifier. Our proposed method has very good performance with less computation, contributing a new architecture to evaluate PCOD and hence helping to improve the wellness of women.

A Model of Big Data Architecture on the Base of FIWARE Components

The intensive development of hardware and software tools provides many opportunities for solving various real-world tasks, such as working with large data sets. This, in turn, is a powerful incentive and opportunity for the development of different approaches and tools for solving them. After discussing several well-known BigData architectures a model of a BigData architecture is designed. The architecture is organized in three sections: edge section, processing-and-storage section and application section. The model will be implemented using FIWARE components.