Big Data Challenges Research Articles

Dealing with the big data challenges in AI for thermoelectric materials

Dealing with the big data challenges in AI for thermoelectric materials

Parallel Fractional Stochastic Gradient Descent With Adaptive Learning for Recommender Systems

The structural change toward the digital transformation of online sales elevates the importance of parallel processing techniques in recommender systems, particularly in the pandemic and post-pandemic era. Matrix factorization (MF) is a popular and scalable approach in collaborative filtering (CF) to predict user preferences in recommender systems. Researchers apply Stochastic Gradient Descent (SGD) as one of the most famous optimization techniques for MF. Paralleling SGD methods help address Big Data challenges due to the wide range of products and the sparsity in user ratings. However, these methods’ convergence rate and accuracy are affected by the dependency between the user and item latent factors, specifically in large-scale problems. Besides, the performance is sensitive to the applied learning rates. This paper proposes a new parallel method to remove dependencies and boost speed-up by using fractional calculus to improve accuracy and convergence rate. We also apply adaptive learning rates to enhance the performance of our proposed method. The proposed method is based on Compute Unified Device Architecture (CUDA) platform. We evaluate the performance of our proposed method using real-world data and compare the results with the close baselines. The results show that our method can obtain high accuracy and convergence rate in addition to high parallelism.

Big data challenges and opportunities in Internet of Vehicles: a systematic review

PurposeBig data challenges and opportunities on the Internet of Vehicles (IoV) have emerged as a transformative paradigm to change intelligent transportation systems. With the growth of data-driven applications and the advances in data analysis techniques, the potential for data-adaptive innovation in IoV applications becomes an outstanding development in future IoV. Therefore, this paper aims to focus on big data in IoV and to provide an analysis of the current state of research.Design/methodology/approachThis review paper uses a systematic literature review methodology. It conducts a thorough search of academic databases to identify relevant scientific articles. By reviewing and analyzing the primary articles found in the big data in the IoV domain, 45 research articles from 2019 to 2023 were selected for detailed analysis.FindingsThis paper discovers the main applications, use cases and primary contexts considered for big data in IoV. Next, it documents challenges, opportunities, future research directions and open issues.Research limitations/implicationsThis paper is based on academic articles published from 2019 to 2023. Therefore, scientific outputs published before 2019 are omitted.Originality/valueThis paper provides a thorough analysis of big data in IoV and considers distinct research questions corresponding to big data challenges and opportunities in IoV. It also provides valuable insights for researchers and practitioners in evolving this field by examining the existing fields and future directions for big data in the IoV ecosystem.

Marketing and psychological analysis based on big data perspective

In the contemporary world, as technology keeps advancing, there exists a trend that big data is becoming increasingly prevalent. Under the current circumstance, the influence of big data can be easily found in various aspects, thus, leading the public to a more convenient and efficient society. There is a rising number of firms and universities that utilize such techniques in the process of monitoring. This paper draws particular attention to the advantages and challenges of big data in the perspective of marketing regarding data regulatory transparency and market sales together with the psychology of consumers and undergraduates so as to stress the importance and drawbacks of big data by reviewing prevenient studies. This research has concluded that big data reveal its benefits and shortcomings from the perspective of marketing, and transparency control could be a very useful point to build customers trust in the future. Also, consumers and college students psychology also require big data and also the control after using them.

Evaluating the understanding of the ethical and moral challenges of Big Data and AI among Jordanian medical students, physicians in training, and senior practitioners: a cross-sectional study

AimsTo examine the understanding of the ethical dilemmas associated with Big Data and artificial intelligence (AI) among Jordanian medical students, physicians in training, and senior practitioners.MethodsWe implemented a literature-validated questionnaire to examine the knowledge, attitudes, and practices of the target population during the period between April and August 2023. Themes of ethical debate included privacy breaches, consent, ownership, augmented biases, epistemology, and accountability. Participants’ responses were showcased using descriptive statistics and compared between groups using t-test or ANOVA.ResultsWe included 466 participants. The greater majority of respondents were interns and residents (50.2%), followed by medical students (38.0%). Most participants were affiliated with university institutions (62.4%). In terms of privacy, participants acknowledged that Big Data and AI were susceptible to privacy breaches (39.3%); however, 59.0% found such breaches justifiable under certain conditions. For ethical debacles involving informed consent, 41.6% and 44.6% were aware that obtaining informed consent posed an ethical limitation in Big Data and AI applications and denounced the concept of “broad consent”, respectively. In terms of ownership, 49.6% acknowledged that data cannot be owned yet accepted that institutions could hold a quasi-control of such data (59.0%). Less than 50% of participants were aware of Big Data and AI’s abilities to augment or create new biases in healthcare. Furthermore, participants agreed that researchers, institutions, and legislative bodies were responsible for ensuring the ethical implementation of Big Data and AI. Finally, while demonstrating limited experience with using such technology, participants generally had positive views of the role of Big Data and AI in complementing healthcare.ConclusionJordanian medical students, physicians in training and senior practitioners have limited awareness of the ethical risks associated with Big Data and AI. Institutions are responsible for raising awareness, especially with the upsurge of such technology.

Hierarchical-Based Dynamic Scenario-Adaptive Risk Assessment for Power Data Lifecycle

In an era marked by rapid advancements in information technology, the task of risk assessment for data security within the complex infrastructure of the power grid has become increasingly vital. This paper introduces a novel methodology for dynamic, scenario-adaptive risk assessment, specifically designed to address the entire lifecycle of power data. Integrating hierarchical analysis with fuzzy comprehensive evaluation, our approach provides a flexible and robust framework for assessing and managing risks in various scenarios. This method enables the generation of adaptive weight matrices and precise risk level determinations, ensuring a detailed and responsive analysis of data security at each lifecycle stage. In our study, we applied predictive analytics and anomaly detection to conduct a thorough examination of diverse data scenarios within the power grid, aiming to proactively identify and mitigate potential security threats. The results of this research demonstrate a significant enhancement in the effectiveness of risk detection and management, leading to improved data protection and operational efficiency. This study contributes a scalable, adaptable model for data security risk assessment, meeting the challenges of big data and complex information systems in the power sector.

Quality control of manufacturing enterprises based on computer and big data technology

AbstractThe traditional manufacturing industry is facing the impact of Big Data, and all aspects of product research and development, process design, quality management, production and operation are urgently looking forward to the birth of innovative methods to cope with the challenges of big data in the industrial background. Although traditional manufacturing enterprises have initially established quality control information system, there are still many problems and limitations in the existing quality control system, which cannot meet the operation needs of manufacturing enterprises. This paper chooses the quality management of manufacturing enterprises as the research object, and uses advanced technologies such as computer, internet, big data and cloud computing to build a data system model of quality control of manufacturing enterprises. this paper optimizes the quality data processing process by establishing the quality monitoring model, and builds an intelligent quality supervision platform, and sets up the quality alarm rules for manufacturing enterprises. If the deviation between the actual value and the predicted value of the quality monitoring index is mapped between [0,1], the quality monitoring system can generate an outlier probability score. In this study, the traditional manual quality management is transformed into the management mode of the “internet + quality data” in order to realize the information, digital and intelligent quality management of manufacturing enterprises. This comprehensive research method combines modern digital technology and relevant theories of quality management to explore the optimization scheme of quality management of manufacturing enterprises, and also provides reference experience for the information construction of other similar enterprises.

Parallel power load abnormalities detection using fast density peak clustering with a hybrid canopy-K-means algorithm

Parallel power loads anomalies are processed by a fast-density peak clustering technique that capitalizes on the hybrid strengths of Canopy and K-means algorithms all within Apache Mahout’s distributed machine-learning environment. The study taps into Apache Hadoop’s robust tools for data storage and processing, including HDFS and MapReduce, to effectively manage and analyze big data challenges. The preprocessing phase utilizes Canopy clustering to expedite the initial partitioning of data points, which are subsequently refined by K-means to enhance clustering performance. Experimental results confirm that incorporating the Canopy as an initial step markedly reduces the computational effort to process the vast quantity of parallel power load abnormalities. The Canopy clustering approach, enabled by distributed machine learning through Apache Mahout, is utilized as a preprocessing step within the K-means clustering technique. The hybrid algorithm was implemented to minimise the length of time needed to address the massive scale of the detected parallel power load abnormalities. Data vectors are generated based on the time needed, sequential and parallel candidate feature data are obtained, and the data rate is combined. After classifying the time set using the canopy with the K-means algorithm and the vector representation weighted by factors, the clustering impact is assessed using purity, precision, recall, and F value. The results showed that using canopy as a preprocessing step cut the time it proceeds to deal with the significant number of power load abnormalities found in parallel using a fast density peak dataset and the time it proceeds for the k-means algorithm to run. Additionally, tests demonstrate that combining canopy and the K-means algorithm to analyze data performs consistently and dependably on the Hadoop platform and has a clustering result that offers a scalable and effective solution for power system monitoring.

ALGORITHMIC LITERACY: Generative Artificial Intelligence Technologies for Data Librarians

INTRODUCTION: Artificial intelligence (AI) is a novel type of library technology. AI technologies and the needs of data librarians are hybrid and symbiotic, because academic libraries must insert AI technologies into their information and data services. Library services need AI to interpret the context of big data.OBJECTIVES: In this context, we explore the use of the the OpenAI Codex, a deep learning model trained on Python code from repositories, to generate code scripts for data librarians. This investigation examines the practices, models, and methodologies for obtaining code script insights from complex code environments linked to AI GPT technologies. METHODS: The proposed AI-powered method aims to assist data librarians in creating code scripts using Python libraries and plugins such as the integrated development environment PyCharm, with additional support from the Machinet AI and Bito AI plugins. The process involves collaboration between the data librarian and the AI agent, with the librarian providing a natural language description of the programming problem and the OpenAI Codex generating the solution code in Python.RESULTS: Five specific web-scraping problems are presented. The scripts demonstrate how to extract data, calculate metrics, and write the results to files.CONCLUSION: Overall, this study highlights the application of AI in assisting data librarians with code script creation for web scraping tasks. AI may be a valuable resource for data librarians dealing with big data challenges on the Web. The possibility of creating Python code with AI is of great value, as AI technologies can help data librarians work with various types of data sources. The Python code in Data Science web scraping projects uses a machine-learning model that can generate human-like code to help create and improve the library service for extracting data from a web collection. The ability of nonprogramming data librarians to use AI technologies facilitates their interactions with all types and data sources. The Python programming language has artificial intelligence modules, packages, and plugins such as the OpenAI Codex, which serialises automation and navigation in web browsers to simulate human behaviour on pages by entering passwords, selecting captcha options, collecting data, and creating different collections of datasets to be viewed.

CloudProteoAnalyzer: scalable processing of big data from proteomics using cloud computing.

Shotgun proteomics is widely used in many system biology studies to determine the global protein expression profiles of tissues, cultures, and microbiomes. Many non-distributed computer algorithms have been developed for users to process proteomics data on their local computers. However, the amount of data acquired in a typical proteomics study has grown rapidly in recent years, owing to the increasing throughput of mass spectrometry and the expanding scale of study designs. This presents a big data challenge for researchers to process proteomics data in a timely manner. To overcome this challenge, we developed a cloud-based parallel computing application to offer end-to-end proteomics data analysis software as a service (SaaS). A web interface was provided to users to upload mass spectrometry-based proteomics data, configure parameters, submit jobs, and monitor job status. The data processing was distributed across multiple nodes in a supercomputer to achieve scalability for large datasets. Our study demonstrated SaaS for proteomics as a viable solution for the community to scale up the data processing using cloud computing. This application is available online at https://sipros.oscer.ou.edu/ or https://sipros.unt.edu for free use. The source code is available at https://github.com/Biocomputing-Research-Group/CloudProteoAnalyzer under the GPL version 3.0 license.

INTEGRATING ARTIFICIAL INTELLIGENCE AND BIG DATA IN MOBILE HEALTH: A SYSTEMATIC REVIEW OF INNOVATIONS AND CHALLENGES IN HEALTHCARE SYSTEMS

This systematic review explores the integration of Artificial Intelligence (AI) and Big Data in mobile health (mHealth) and their transformative impact on healthcare systems. Analysing 25 peer-reviewed articles, the review delves into the utilisation, innovations, and inherent challenges of AI and Big Data in various healthcare settings. The findings reveal that AI and Big Data significantly enhance diagnostic precision, personalise treatment strategies, and streamline healthcare operations. However, this technological advancement is not without its complexities. Key challenges identified include ethical dilemmas, data privacy and security issues, and preserving the human element within healthcare. The review underscores the risk of an over-reliance on AI and the criticality of using unbiased, representative data sets. The integration of AI into clinical practices, while promising, demands rigorous oversight and ethical governance. The review concludes that the successful adoption of AI and Big Data in healthcare hinges on a balanced approach that harmonises technological innovation with ethical, equitable, and human-centred healthcare practices. If managed effectively, this integration offers the potential for significant improvements in patient outcomes and operational efficiency yet requires vigilant management of the challenges to realise its benefits fully.

Big data challenge for monitoring quality in higher education institutions using business intelligence dashboards

As big data becomes an apparent challenge to handle when building a business intelligence (BI) system, there is a motivation to handle this challenging issue in higher education institutions (HEIs). Monitoring quality in HEIs encompasses handling huge amounts of data coming from different sources. This paper reviews big data and analyses the cases from the literature regarding quality assurance (QA) in HEIs. It also outlines a framework that can address the big data challenge in HEIs to handle QA monitoring using BI dashboards and a prototype dashboard is presented in this paper. The dashboard was developed using a utilisation tool to monitor QA in HEIs to provide visual representations of big data. The prototype dashboard enables stakeholders to monitor compliance with QA standards while addressing the big data challenge associated with the substantial volume of data managed by HEIs’ QA systems. This paper also outlines how the developed system integrates big data from social media into the monitoring dashboard

Curriculum Teaching and Research Based on Cloud Computing and Big Data

This study investigates the integrated application of cloud computing and big data in the field of education and its impact on curriculum teaching. Through a comprehensive review of the fundamental principles, current applications, and relevant literature, the study reveals the advantages and challenges of cloud computing and big data in aspects such as curriculum design, learning analytic, and personalized teaching. It also explores their influence on aspects like the sharing of teaching resources, the construction of online learning environments, and the evaluation of student learning outcomes. Furthermore, through case analysis and empirical studies, this paper provides a more in-depth validation of the practical effects of cloud computing and big data in improving teaching effectiveness, promoting educational innovation, and cultivating students’ information literacy. In summary, the integration of cloud computing and big data provides new perspectives and solutions for building smart education, while also laying a solid foundation for nurturing talents capable of meeting the demands of future society.

An Overview of Big Data Challenges, Tools and Techniques

An Overview of Big Data Challenges, Tools and Techniques

A Machine Learning Based Decision Support Framework For Big Data Pipeline Modeling And Design

The data warehousing process requires an architectural revolution to settle big data challenges and address new data sources such as social networks, recommendation systems, smart cities, and the web to extract value from shared data. In this respect, the pipeline modeling community for the acquisition, storage, and processing of data for analysis purposes is enacting a wide range of technological solutions that present significant challenges and difficulties. More specifically, the choice of the most appropriate tool for the user's specific business needs and the interoperability between the different tools have become a primary challenges. From this perspective, we propose in this paper a new interactive framework based on machine learning techniques (ML) to assist experts in the process of modeling a customized pipeline for data warehousing. More precisely, we elaborate first (i) an analysis of the experts’ requirements and the characteristics of the data to be processed, then (ii) we propose the most appropriate architecture to their requirements from a multitude of specific architectures instantiated from a generic one, by using (iii) several ML methods to predict the most suitable tool for each phase and task within the architecture. Additionally, our framework is validated through two real-world use cases and user feedback.

NA

NA

Case T-578/22: Supervising Europol’s Big Data Challenge – making the best of it by means of (limited) administrative enforcement powers

Case T-578/22: Supervising Europol’s Big Data Challenge – making the best of it by means of (limited) administrative enforcement powers

Review on audit data visualization method based on R language

To be competitive in today's market, companies must constantly assess and improve their operations, and financial data analysis has emerged as a crucial tool for this purpose. Executives can utilize data analysis to gain a deeper understanding of the underlying facts in their data and make better informed decisions about their company and the market. Multiple data models in the language make it possible to have a fully functional language environment with tools for statistical analysis and visual visualization of data. With its ability to enhance the quality of work in statistical computations and graphical analysis, the R language is ideally suited to the industrial data analysis environment. In this paper, we use a literature review methodology to examine the domestic and international literature on R language big data visualization auditing, analyze the visualization application areas of R language big data auditing, discuss the benefits and challenges of big data auditing, demonstrate how R language can be adapted to real-world auditing scenarios, and offer reasonable recommendations and optimistic outlooks for the field's future.

Overview of Big Data Analytics in Modern Astronomy

Astronomers are increasingly compelled to chart the universe with ever greater precision. Projects like the Sloan Digital Sky Survey (SDSS), Pan-STARRS, and the Large Synoptic Survey Telescope (LSST) generate approximately 100-200 Petabytes of data annually, presenting significant big data challenges in terms of storage, processing, and data transfer. The Square Kilometer Array (SKA), an ambitious project involving 130,000 antennas and 200 dishes spanning two continents, is scheduled to become operational in 2028. It will collect 160 terabytes of data per second, translating to 1 petabyte of data daily. Coping with this immense volume of data necessitates real-time processing and analysis, driving the need for efficient machine learning and image analysis algorithms. Astronomy stands as an ideal domain for big data analytics, pushing the boundaries of data analysis. This review paper will present intriguing applications for data scientists, exploring the challenges and recent technological advancements in big data analytics concerning astronomy. The paper will also critically assess the strengths and weaknesses of various approaches, methodologies, or tools used in big data analytics within the context of astronomy, supported by relevant case studies.

REVIEW OF OPPORTUNITIES AND CHALLENGES REPRESENTED BY BIG DATA FOR FINTECH IN MALAYSIA

This paper explores the potential of big data for driving economic growth and innovation within the fintech sector in Malaysia. By using systematic literature review, it investigates the opportunities of big data in the fintech sector and its current utilisation in Malaysia. Firstly, the paper will outline current trends in fintech within the Malaysian context. This will include examining the current state of big data utilization, prevalent initiatives, and current industry players. Secondly, it will identify challenges for fintech development by exploring technological, organisational, and environmental factors that might affect the utilisation of big data in Malaysia. Finally, it will suggest approaches for organisations to adopt in overcoming these challenges. The paper hopes to enrich the understanding of the opportunities and challenges for big data in fintech in Malaysia and the ways it can contribute to the domestic economy.