Big Data Revolution Research Articles

Private commercial companies sharing health-relevant consumer data with health researchers in sub-Saharan Africa: an ethical exploration

ABSTRACT Sharing large digital-first datasets, including for purposes for which they were not originally intended, is a hallmark of the “big data revolution”. Through their routine operations, private commercial companies collect massive amounts of diverse data from their customers, some of which may interest those working in the public sector, such as health researchers. Researchers and government agencies worldwide have been increasingly using data from commercial entities (such as Google, Microsoft, Apple, Facebook/Meta, Twitter/X and Amazon, among others) to generate health-related insights. This article explores ethical issues raised by the practice of commercial companies sharing consumer data with third parties for the purposes of promoting health in the sub-Saharan African (SSA) context. First, as an illustrative example, it examines some of the ways telecommunication (telecom) companies in SSA shared mobility data from cellphone users with public health researchers during the COVID-19 pandemic. Second, it examines a recent debate about the ethical responsibilities of companies that collect, process and share user-generated data, drawing implications for the SSA context. Finally, since this is a relatively understudied subject, we point out some areas where future conceptual and empirical work could contribute to the development of relevant ethics guidance and regulatory governance in SSA.

Big Data Analytics and Management Forecasting Behavior

SYNOPSIS This paper investigates whether the use of Big Data analytics by firms has a spillover effect on management forecasting behavior. Insights provided by Big Data could potentially improve firms’ ability to forecast earnings (supply channel) and investor demand for earnings information is likely higher for firms engaging in data analytics (demand channel). Using a text-based measure of firms’ commitments to and usage of Big Data analytics, we find that Big Data analytics usage is positively associated with the propensity to issue management earnings forecasts. Consistent with the “supply channel” explanation, we find that Big Data analytics usage is positively associated with management forecast accuracy as well. Also, supporting the “demand channel” explanation, we find that Big Data analytics usage is associated with greater analyst following. Our findings of improved disclosure following commitments to Big Data analytics highlight a potentially unintended benefit of the Big Data revolution.

The Potential of Evidence-Based Clinical Intake Tools to Discover or Ground Prevalence of Symptoms Using Real-Life Digital Health Encounters: Retrospective Cohort Study.

Evidence-based clinical intake tools (EBCITs) are structured assessment tools used to gather information about patients and help health care providers make informed decisions. The growing demand for personalized medicine, along with the big data revolution, has rendered EBCITs a promising solution. EBCITs have the potential to provide comprehensive and individualized assessments of symptoms, enabling accurate diagnosis, while contributing to the grounding of medical care. This work aims to examine whether EBCITs cover data concerning disorders and symptoms to a similar extent as physicians, and thus can reliably address medical conditions in clinical settings. We also explore the potential of EBCITs to discover and ground the real prevalence of symptoms in different disorders thereby expanding medical knowledge and further supporting medical diagnoses made by physicians. Between August 1, 2022, and January 15, 2023, patients who used the services of a digital health care (DH) provider in the United States were first assessed by the Kahun EBCIT. Kahun platform gathered and analyzed the information from the sessions. This study estimated the prevalence of patients' symptoms in medical disorders using 2 data sets. The first data set analyzed symptom prevalence, as determined by Kahun's knowledge engine. The second data set analyzed symptom prevalence, relying solely on data from the DH patients gathered by Kahun. The variance difference between these 2 prevalence data sets helped us assess Kahun's ability to incorporate new data while integrating existing knowledge. To analyze the comprehensiveness of Kahun's knowledge engine, we compared how well it covers weighted data for the symptoms and disorders found in the 2019 National Ambulatory Medical Care Survey (NMCAS). To assess Kahun's diagnosis accuracy, physicians independently diagnosed 250 of Kahun-DH's sessions. Their diagnoses were compared with Kahun's diagnoses. In this study, 2550 patients used Kahun to complete a full assessment. Kahun proposed 108,523 suggestions related to symptoms during the intake process. At the end of the intake process, 6496 conditions were presented to the caregiver. Kahun covered 94% (526,157,569/562,150,572) of the weighted symptoms and 91% (1,582,637,476/173,4783,244) of the weighted disorders in the 2019 NMCAS. In 90% (224/250) of the sessions, both physicians and Kahun suggested at least one identical disorder, with a 72% (367/507) total accuracy rate. Kahun's engine yielded 519 prevalences while the Kahun-DH cohort yielded 599; 156 prevalences were unique to the latter and 443 prevalences were shared by both data sets. ECBITs, such as Kahun, encompass extensive amounts of knowledge and could serve as a reliable database for inferring medical insights and diagnoses. Using this credible database, the potential prevalence of symptoms in different disorders was discovered or grounded. This highlights the ability of ECBITs to refine the understanding of relationships between disorders and symptoms, which further supports physicians in medical diagnosis.

Big Data Revolution in Finance: Opportunities, Challenges, and Future Trends

In todays digital era, big data has emerged as one of the most exciting and potentially transformative technological innovations in the financial sector. Its rise has not only altered our understanding of financial markets and institutions but also presented unprecedented opportunities and challenges to the financial industry. As financial transactions, customer interactions, and market behaviors continue to digitize, financial institutions are actively exploring how to harness the power of big data to gain a competitive edge and enhance decision-making. This paper delves into the significance of big data in the financial sector and effective development strategies, analyzing its real-world impact on the industry. It aims to provide valuable insights to financial practitioners, decision-makers, and researchers on how to leverage big data to achieve financial objectives.

Ghosts in the Machine: How Big Data Analytics Can Be Used to Strengthen Online Public Procurement Accountability

The core of sustainable public procurement lies in its ability to stem uneconomical public expenditures that waste taxpayer money and stifle social trust and development. The external audit of public procurement proves problematic since current research fails to provide sufficient empirical studies aimed at identifying procurement fraud. The development of online portals with embedded e-procurement solutions, along with the big data revolution, open new horizons and allow us to reveal trends otherwise impossible to spot, such as transactions achieved in an exclusive commercial relationship, in which a vendor engages only with a single public entity. By using innovative data acquisition techniques, our research encompasses 2.25 million online direct public procurement procedures conducted in 2023 using the Romanian portal for public procurement, totaling EUR 3.22 billion. By aggregating databases obtained from various public sources, our analysis achieved remarkable granularity, using over 112 million data elements—50 pertaining to each transaction. Research results indicate a unique sub-population of public procurement procedures—those conducted with “in-house” vendors totaling 14.28% of all direct public acquisitions and which is significantly differentiated along the entire list of analyzed criteria—financial, geographical, statistical, or risk-wise—illustrating a troubling phenomenon: possible gerrymandering of the online public procurement landscape, which, at least in theory, resembles a perfect market, by cultivating preferential commercial relations, thus affecting the legality, regularity, and economical aspects of public procurement.

Big Data Education Landscape for Graduates in Morocco: Insights from 2022 Offerings

The main objective of this research was to analyze graduate programs in Big Data in Morocco in 2022, focusing on the technical skills, tools and technologies, and functional skills taught in the programs. In the digital age, the Big Data revolution has generated massive amounts of data, transforming the way businesses and organizations around the world make decisions and innovate. In this ever-changing context, Training of Big Data professionals in Morocco in 2022 is crucial. The expansion of Big Data has created a growing need for skilled professionals to manage these immense data sets. Higher education programs play a key role in preparing these future experts. This study adopted a content analysis methodology, examining the curricula of 16 public engineering schools, and 21 accredited public schools and science faculties in Morocco. The technical skills, tools and technologies, and functional skills taught were analyzed and compared with existing literature on Big Data skills. The programs focus on technical skills such as programming, machine learning, statistics, and mathematics, with Python as the main tool. However, certain skills, such as predictive analysis, are under-represented. Functional skills in communication and management are essential; however, skills in agile methods and change management vary in coverage. The results offer an insight into the alignment between higher education programs and the needs of the Big Data market. Educational institutions can adjust their programs to better meet emerging needs, focusing on under-represented skills, and ensuring the continued relevance of training.

Big Data Revolution: A Comprehensive Exploration of its Transformative Impact on Government Sectors in Railway Sectors

Big Data plays a crucial role on governance and railway operations, navigating the intricate web of applications that define this paradigm shift. From data-driven decision-making and predictive analytics shaping government policies to real-time operational efficiency and personalized passenger experiences revolutionizing railways, the impact of Big Data is far-reaching. As a powerful catalyst for change, Big Data not only enhances efficiency but also fuels innovation, steering both governance and railway systems toward a future characterized by citizen-centric services. Keywords: Data, Governance, Railway Operations, Predictive analytics, Government sectors, Resource Optimization, Sustainability.

A detailed database of sub-annual Spanish demographic statistics: 2005–2021

The big data revolution has made it possible to collect, transmit and exploit huge amounts of data. The potential this offer for data analysis, however, clashes with the limitations imposed by laws on protection of personal data. This paper details a new database (DEMOSPA0521) made after processing and summarising more than 868 million demographic records from Spain, corresponding to a period of seventeen years (2005–2021). DEMOSPA0521 is composed of fifteen files: a group of (monthly and daily moving averages) datasets derived from population stocks and a collection of (daily, monthly and quarterly) datasets obtained from population, death, migration and birth statistics. The intra-annual distributions were calculated by exploiting both the temporal dimensions of age and calendar. DEMOSPA0521 also includes eleven R-Code files that enables the summary datasets to be derived from the raw microdata. DEMOSPA0521 can be used to confirm established results and employed to answer new research questions.

Application of Multi-Omics Technologies to the Study of Phytochromes in Plants.

Phytochromes (phy) are distributed in various plant organs, and their physiological effects influence plant germination, flowering, fruiting, and senescence, as well as regulate morphogenesis throughout the plant life cycle. Reactive oxygen species (ROS) are a key regulatory factor in plant systemic responses to environmental stimuli, with an attractive regulatory relationship with phytochromes. With the development of high-throughput sequencing technology, omics techniques have become powerful tools, and researchers have used omics techniques to facilitate the big data revolution. For an in-depth analysis of phytochrome-mediated signaling pathways, integrated multi-omics (transcriptomics, proteomics, and metabolomics) approaches may provide the answer from a global perspective. This article comprehensively elaborates on applying multi-omics techniques in studying phytochromes. We describe the current research status and future directions on transcriptome-, proteome-, and metabolome-related network components mediated by phytochromes when cells are subjected to various stimulation. We emphasize the importance of multi-omics technologies in exploring the effects of phytochromes on cells and their molecular mechanisms. Additionally, we provide methods and ideas for future crop improvement.

The Open Prison of the Big Data Revolution: False Consciousness, Faustian Bargains, and Digital Entrapment

Although some scholars raise alarm about societal harm emerging from Big Data practices, critical social theory (CST) Information Systems research on the structures and dynamics driving Big Data practices is rare. In this research commentary, we interrogate how tech firms use social practices and platform design to strategically manipulate individuals into accepting datafication and data assetization that accrue positive data network effects for themselves and mostly negative data network effects (economic loss, social and privacy harm) for individuals. We draw on the ideas of Heidegger and Marcuse to critically question the Big Data paradigm in order to develop better understanding of the social implications for individuals and society. Using the concepts of false consciousness, digital entrapment, and Faustian bargains, we critically inquire into the Big Data practices that keep us tethered to digital platforms. Specifically, we interrogate sociomaterial structures that socially condition individuals into a digital habitus and to identify themselves as homo digitalis, who view all their “relations” (social and economic) as digital. This social conditioning reproduces a false consciousness that constricts our worldview, undermines our rational choices, and enables the risky compromises we make with tech companies that manipulate and exploit us with their increasingly oppressive Big Data practices and related dark patterns. We critically analyze the case of Microsoft Viva to provide an illustration of how mundane digital tools can condition our reality and entrap us into an open prison. We argue that if we do not critically interrogate our false consciousness of the digital and understand how digital giants colonize our social systems by structurally embedding Big Data practices, we will continue to be susceptible to manipulation and digital entrapment. Ongoing risky compromises with tech firms will erode the very foundations of the “good life,” freedom, liberty, and personal privacy, and they will institutionalize the open prison. The CST explanation we propose and the research agenda we outline are meant to encourage research into solutions to the digital entrapment problem.History: Suprateek Sarker, Senior Editor; Robert Gregory, Associate Editor.Supplemental Material: The online appendix is available at https://doi.org/10.1287/isre.2020.0588 .

A comparative analysis of big data processing paradigms: Mapreduce vs. apache spark

The paper addresses a highly relevant and contemporary topic in the field of data processing. Big data is a crucial aspect of modern computing, and the choice of processing framework can significantly impact performance and efficiency. The technical revolution of big data has changed how organizations handle and value large databases. As data quantities expand quickly, effective and scalable data processing systems are essential. MapReduce and Apache Spark are two of the most popular large data processing techniques. This study compares these two frameworks to determine their merits, shortcomings, and applicability for big data applications. Nearly quintillion bytes of data are created daily. Approximately 90% of data was produced in the previous two years. At this stage, data comes from temperature sensors, social media, movies, photographs, transaction records (like banking records), mobile phone conversations, GPS signals, etc. In this article, all key big data technologies are introduced. This document compares all big data technologies and discusses their merits and downsides. Run trials using multiple data sets of varying sizes to validate and explain the study. Graphical depiction shows how one tool outperforms others for given data. Big Data is data generated by the rapid usage of the internet, sensors, and heavy machinery, with great volume, velocity, variety, and veracity. Numbers, photos, videos, and text are omnipresent in every sector. Due to the pace and amount of data generation, the computing system struggles to manage large data. Data is stored in a distributed architectural file system due to its size and complexity. Big distributed file systems, which must be fault-tolerant, adaptable, and scalable, make complicated data analysis dangerous and time-consuming. Big data collection is called ‘datafication’. Big data is ‘datafied’ for productivity. Organisation alone does not make Big Data valuable; we must choose what we can do with it.

Making Ethical Decisions Is Hard!

In the early years of the “big data revolution,” big data was described by 4 or 5 v’s (or more) - volume, velocity, variety, veracity, and value. Although intended to describe data, the 5 v’s can provide insights into what makes ethical decision making hard. The volume of work, the expectation of speed, the variety of problems, the veracity or maybe more explicitly the provenance of the data, and the value of the work viewed from the diverse and sometimes competing perspectives of stakeholders can make ethically navigating the data science landscape challenging. As the field grows, the need for resources and tools has become more urgent. In this article, we will briefly examine the history of several ethical guidelines and frameworks.

Smart batch process: The evolution from 1D and 2D to new 3D perspectives in the era of Big Data

Big Data will revolutionize modern industry by improving process optimization, facilitating insight discovery, and improving decision-making. This big data revolution presents a multitude of possibilities and challenges in evolving from traditional batch processes to smart batch processes. This tremendous potential requires the ability to extract value from vast amounts of industrial process data. Using a new three-dimensional (3D) perspective of time, batch, and operational context, this paper explores smart batch processes with higher efficiency, greater profitability, and longer sustainability. First, we review the traditional one-dimensional (1D) perspective on batch processes and summarize the existing two-dimensional (2D) perspectives on batch processes, i.e., modeling, monitoring, control, and optimization methods. Based on those results, the spotlight will focus on how big data can be used to achieve smart batch processes using the 3D perspective. This will include detailed discussions of definitions and concepts, operational mechanisms, and the benefits and advantages of smart batch processes. For further implementation of the 3D perspective, we present several monitoring and control methodologies. Next, we analyze several challenges and issues in implementing smart batch processes in the era of Big Data. In conclusion, we provide both a novel viewpoint and encouragement for future research into batch process automation from the 3D perspective.

The Transformation from Text Big Data to Visual Short Video Big Data

The traditional text era has gradually declined, replaced by the boom of short video. In the rapid development of the Internet information age, people's interests and habits have begun to quietly change. Text newspapers and periodicals have become the past, and industry reform has become an irreversible trend. In this process, how to realize the visualization of text data and the transformation of big data is a topic that countless practitioners have been struggling to explore. In order to realize the transformation from text data to short video, the focus of the problem is to overturn the working mode of text data and build a new architecture based on the new trend of short video big data. Because of the limitations of text big data, there are inevitably many drawbacks in today's Internet society. Not only is the media backward, but also the content can no longer meet the needs of the majority of people. It has gradually been recognized as a "sunset industry" in society. On the other hand, short video big data not only caters to the needs and hobbies of the majority of people in the current trend, Moreover, it has made great progress on the development of technological innovation and mode innovation. Compared with each other, short video big data has incomparable advantages over text big data. It is not so much a competition between them as a new revolution of short video big data for text big data.

In-depth analysis on parallel processing patterns for high-performance Dataframes

The Data Science domain has expanded monumentally in both research and industry communities during the past decade, predominantly owing to the Big Data revolution. Artificial Intelligence (AI) and Machine Learning (ML) are bringing more complexities to data engineering applications, which are now integrated into data processing pipelines to process terabytes of data. Typically, a significant amount of time is spent on data preprocessing in these pipelines, and hence improving its efficiency directly impacts the overall pipeline performance. The community has recently embraced the concept of Dataframes as the de-facto data structure for data representation and manipulation. However, the most widely used serial Dataframes today (R, pandas) experience performance limitations while working on even moderately large data sets. We believe that there is plenty of room for improvement by taking a look at this problem from a high-performance computing point of view. In a prior publication, we presented a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon [1]. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. Furthermore, we evaluate the performance of Cylon on the ORNL Summit supercomputer.

The evolution of Big Data in neuroscience and neurology

Neurological diseases are on the rise worldwide, leading to increased healthcare costs and diminished quality of life in patients. In recent years, Big Data has started to transform the fields of Neuroscience and Neurology. Scientists and clinicians are collaborating in global alliances, combining diverse datasets on a massive scale, and solving complex computational problems that demand the utilization of increasingly powerful computational resources. This Big Data revolution is opening new avenues for developing innovative treatments for neurological diseases. Our paper surveys Big Data’s impact on neurological patient care, as exemplified through work done in a comprehensive selection of areas, including Connectomics, Alzheimer’s Disease, Stroke, Depression, Parkinson’s Disease, Pain, and Addiction (e.g., Opioid Use Disorder). We present an overview of research and the methodologies utilizing Big Data in each area, as well as their current limitations and technical challenges. Despite the potential benefits, the full potential of Big Data in these fields currently remains unrealized. We close with recommendations for future research aimed at optimizing the use of Big Data in Neuroscience and Neurology for improved patient outcomes.

A social and ethical framework for providing health information obtained from combining genetics and fitness tracking data

In this paper we explore a new technological intersection in the “big data revolution”: the integration of two forms of data, genetic data and fitness tracking data. For example, a small number of direct-to-consumer (DTC) genetic testing companies have recently begun offering customers the ability to link their fitness tracking data with their genetic profile to get personalized diet and exercise recommendations. In this paper we put forward four ethical considerations that should inform potential uses of this health information. Those considerations are: the heightened risks to privacy resulting from combining sensitive data sets; the poor quality of health information that is possible at present in the realm of precision DTC genomics; the limited usefulness of the recommendations; and finally, the cultural assumptions about health and personal responsibility that are embedded within fitness genetic testing and fitness tracking. To conclude, we offer some guidance on how the benefits and risks of returning this type of health information can be weighed.

A novel bidirectional DiPLS based LSTM algorithm and its application in industrial process time series prediction

The development of industry has led to the big data revolution in manufacturing industry, which has greatly improved production quality and efficiency. A accuracy prediction model is established by using deep learning method, especially short and long term memory (LSTM). However, the application field of LSTM is usually very limited, and the interpretability of LSTM is weak. In this paper, a novel bidirectional DiPLS based LSTM algorithm (BiDiPLS-LSTM) is proposed to solve this problem. First, DiPLS is used to process the forward time series data to obtain the most predictable dynamic latent variable (DLV) of the target variable. At the same time, DiPLS method is also used to process the reverse time series data and extract the reverse DLV. Then, the forward and reverse dynamic features of the same location will form new input data, and this new input data will be input into the LSTM network to realize the prediction of time series data. In order to verify the effectiveness of the proposed BiDiPLS-LSTM method, the real 660 MW coal-fired boiler process data is used, and the emission of oxides of nitrogen (NOx) is regarded as the output. The results show that the extracted dynamic features can be used to build a very accuracy model, and the results outperform the traditional LSTM method, the PLS-LSTM method, the unidirectional DiPLS-LSTM method, and traditional RNN method in the comparison. For the first time, we used the bidirectional dynamic features of the input data as the real input to predict the output, which will obtain a good prediction effect. What is more, the use of dynamic latent variables made the result more explanatory.

The many faces of the big data revolution in health for sub-Saharan Africa

The many faces of the big data revolution in health for sub-Saharan Africa

Using Web-Data to Estimate Spatial Regression Models

Macro econometrics has been recently affected by the so-called ‘Google Econometrics’. Comparatively less attention has been paid to the subject by the regional and spatial sciences where the Big Data revolution is challenging the conventional econometric techniques with the availability of a variety of non- traditionally collected data (such as, e. g., crowdsourcing, web scraping, etc) which are almost invariably geo-coded. However, these unconventionally collected data represent only what in statistics is known as a “convenience sample” that does not allow any sound probabilistic inference. This paper aims at making aware researchers of the consequence of the unwise use of such data in the applied work and to propose a technique to minimize such the negative effects in the estimation of spatial regression. The method consists of manipulating the data prior their use in an inferential context.