Big Data Repositories Research Articles

Exploring statistical and machine learning methods for modeling probability distribution parameters in downtime length analysis: a paper manufacturing machine case study

Manufacturing companies focus on improving productivity, reducing costs, and aligning performance metrics with strategic objectives. In industries like paper manufacturing, minimizing equipment downtime is essential for maintaining high throughput. Leveraging the extensive data generated by these facilities offers opportunities for gaining competitive advantages through data-driven insights, revealing trends, patterns, and predicting future performance indicators like unplanned downtime length, which is essential in optimizing maintenance and minimizing potential losses. This paper explores statistical and machine learning techniques for modeling downtime length probability distributions and correlation with machine vibration measurements. We proposed a novel framework, employing advanced data-driven techniques like artificial neural networks (ANNs) to estimate parameters of probability distributions governing downtime lengths. Our approach specifically focuses on modeling parameters of these distribution, rather than directly modeling probability density function (PDF) values, as is common in other approaches. Experimental results indicate a significant performance boost, with the proposed method achieving up to 30% superior performance in modeling the distribution of downtime lengths compared to alternative methods. Moreover, this method facilitates unsupervised training, making it suitable for big data repositories of unlabelled data. The framework allows for potential expansion by incorporating additional input variables. In this study, machine vibration velocity measurements are selected for further investigation. The study underscores the potential of advanced data-driven techniques to enables companies to make better-informed decisions regarding their current maintenance practices and to direct improvement programs in industrial settings.

From Syntactic to Semantic Interoperability Using a Hyperontology in the Oncology Domain.

Interoperability is crucial to overcoming various challenges of data integration in the healthcare domain. While OMOP and FHIR data standards handle syntactic heterogeneity among heterogeneous data sources, ontologies support semantic interoperability to overcome the complexity and disparity of healthcare data. This study proposes an ontological approach in the context of the EUCAIM project to support semantic interoperability among distributed big data repositories that have applied heterogeneous cancer image data models using a semantically well-founded Hyperontology for the oncology domain.

Calls to action on lung cancer management and research.

Lung cancer, the leading cause of cancer-related deaths globally, remains a pressing health issue despite significant medical advances. The New York Lung Cancer Foundation brought together experts from academia, the pharmaceutical and biotech industries as well as organizational leaders and patient advocates, to thoroughly examine the current state of lung cancer diagnosis, treatment, and research. The goal was to identify areas where our understanding is incomplete and to develop collaborative public health and scientific strategies to generate better patient outcomes, as highlighted in our "Calls to Action." The consortium prioritized 8 different calls to action. These include (1) develop strategies to cure more patients with early-stage lung cancer, (2) investigate carcinogenesis leading to lung cancers in patients without a history of smoking, (3) harness precision medicine for disease interception and prevention, (4) implement solutions to deliver prevention measures and effective therapies to individuals in under-resourced countries, (5) facilitate collaborations with industry to collect and share data and samples, (6) create and maintain open access to big data repositories, (7) develop new immunotherapeutic agents for lung cancer treatment and prevention, and (8) invest in research in both the academic and community settings. These calls to action provide guidance to representatives from academia, the pharmaceutical and biotech industries, organizational and regulatory leaders, and patient advocates to guide ongoing and planned initiatives.

Enhancing Big Data Performance Through Graph Coloring-Based Locality of Reference

Efficiency is a crucial factor when handling the retrieval and storage of data from vast amounts of records in a Big Data repository. These systems require a subset of data that can be accommodated within the combined physical memory of a cluster of servers. It becomes impractical to analyze all of the data if its size exceeds the available memory capacity. Retrieving data from virtual storage, primarily hard disks, is significantly slower compared to accessing data from main memory, resulting in increased access time and diminished performance. To address this, a proposed model aims to enhance performance by identifying the most suitable data locality structure within a big data set and reorganizing the data schema accordingly; by locality, it has been referred to as a particular access pattern. This allows transactions to be executed on data residing in the fastest memory layer, such as cache, main memory, or disk cache

Artificial intelligence applied to MRI data to tackle key challenges in multiple sclerosis.

Artificial intelligence (AI) is the branch of science aiming at creating algorithms able to carry out tasks that typically require human intelligence. In medicine, there has been a tremendous increase in AI applications thanks to increasingly powerful computers and the emergence of big data repositories. Multiple sclerosis (MS) is a chronic autoimmune condition affecting the central nervous system with a complex pathogenesis, a challenging diagnostic process strongly relying on magnetic resonance imaging (MRI) and a high and largely unexplained variability across patients. Therefore, AI applications in MS have the great potential of helping us better support the diagnosis, find markers for prognosis to eventually design more powerful randomised clinical trials and improve patient management in clinical practice and eventually understand the mechanisms of the disease. This topical review aims to summarise the recent advances in AI applied to MRI data in MS to illustrate its achievements, limitations and future directions.

Using Area Under the Precision Recall Curve to Assess the Effect of Random Undersampling in the Classification of Imbalanced Medicare Big Data

In this paper, we investigate the impact of Random Undersampling (RUS) on a supervised Machine Learning task involving highly imbalanced Big Data. We present the results of experiments in Medicare Fraud detection. To the best of our knowledge, these experiments are conducted with the largest insurance claims datasets ever used for Medicare Fraud detection. We obtain two datasets from two Big Data repositories provided by the United States government’s Centers for Medicare and Medicaid Services. The larger of the two datasets contains nearly 174 million instances, with a minority to majority class ratio of approximately 0.0039. Our contribution is to show that RUS has a detrimental effect on a Medicare Fraud detection task when performed on large scale, imbalanced data. The effect of RUS is apparent in the Area Under the Precision Recall Curve (AUPRC) scores recorded from experimental outcomes. We use four popular, open-source classifiers in our experiments to confirm the negative impact of RUS on their AUPRC scores.

Lessons From The Glaucoma Foundation Think Tank 2023: A Patient-Centric Approach to Glaucoma.

The main takeaways also included that BIG DATA repositories and AI are important combinatory tools to foster novel strategies to prevent and stabilize glaucoma and, in the future, recover vision loss from the disease. To summarize the main topics discussed during the 28th Annual Glaucoma Foundation Think Tank Meeting "A Patient-Centric Approach to Glaucoma" held in New York on June 9 and 10, 2023. The highlights of the sessions on BIG DATA, genetics, modifiable lifestyle risk factors, female sex hormones, and neuroprotection in the field of primary open angle glaucoma (POAG) were summarized. The researchers discussed the importance of BIG DATA repositories available at national and international levels for POAG research, including the United Kingdom Biobank. Combining genotyped large cohorts worldwide, facilitated by artificial intelligence (AI) and machine-learning approaches, led to the milestone discovery of 312 genome-wide significant disease loci for POAG. While these loci could be combined into a polygenic risk score with clinical utility, Think Tank meeting participants also provided analytical epidemiological evidence that behavioral risk factors modify POAG polygenetic risk, citing specific examples related to caffeine and alcohol use. The impact of female sex hormones on POAG pathophysiology was discussed, as was neuroprotection and the potential use of AI to help mitigate specific challenges faced in clinical trials and speed approval of neuroprotective agents. The experts agreed on the importance of genetics in defining individual POAG risk and highlighted the additional crucial role of lifestyle, gender, blood pressure, and vascular risk factors. The main takeaways also included that BIG DATA repositories and AI are important combinatory tools to foster novel strategies to prevent and stabilize glaucoma and, in the future, recover vision loss from the disease.

A Reference Architecture for Supporting Multidimensional Big Data Analytics over Big Web Knowledge Bases: Definitions, Implementation, Case Studies

This paper focuses attention on the problem of supporting multidimensional big data analytics tools over Big Web Knowledge Bases. These big data repositories are generated from web systems built on top of big datasets, as in several real-life examples such as web interfaces to social networks, e-government systems, smart city tools (e.g., for weather forecasting), and so forth. The problem of extracting actionable knowledge insights from big web knowledge bases is a relevant challenge now. In line with this emerging research field, the paper proposes models, issues and research trends in the context of multidimensional big data analytics over big web knowledge bases, along with a reference architecture that impersonates this paradigm.

A Novel Cloud Enabled Access Control Model for Preserving the Security and Privacy of Medical Big Data

In the context of healthcare, big data refers to a complex compilation of digital medical data collected from many sources that are difficult to manage with normal technology and software due to its size and complexity. These big data are useful in various aspects of healthcare, such as disease diagnosis, early prevention of diseases, and predicting epidemics. Even though medical big data has many advantages and a lot of potential for revolutionizing healthcare, it also has a lot of drawbacks and problems, of which security and privacy are of the utmost concern, owing to the severity of the complications once the medical data is compromised. On the other hand, it is evident that existing security and privacy safeguards in healthcare organizations are insufficient to protect their massive, big data repositories and ubiquitous environment. Thus, motivated by the synthesizing of the current knowledge pertaining to the security and privacy of medical big data, including the countermeasures, in the study, firstly, we provide a comprehensive review of the security and privacy of medical big data, including countermeasures. Secondly, we propose a novel cloud-enabled hybrid access control framework for securing the medical big data in healthcare organizations, and the result of this research indicates that the proposed access control model can withstand most cyber-attacks, and it is also proven that the proposed framework can be utilized as a primary base to build secure and safe medical big data solutions. Thus, we believe this research would be useful for future researchers to comprehend the knowledge on the security and privacy of medical big data and the development of countermeasures.

The role of government and research organisations in the development of logistics networks as an integral area of physical internet

Digital technologies are fundamentally changing the development of economies and societies, affecting transport and logistics. Data and data-driven innovations are at the centre of this transformation. At the same time, the role of government and research organisations in the development of the physical internet is not sufficiently clear. This investigation has been performed to research the current limitations of cloud-based big data repositories, business attitudes toward sharing the data and identify priority areas for governmental and public research institutions. The study revealed the interest and desire of companies to cooperate while identifying the essential conditions for cooperation - data security, maintaining competitiveness, the mechanism of data preparation and provision, and possible pricing.

Big Data Bot with a Special Reference to Bioinformatics

There are quintillions of data on deoxyribonucleic acid (DNA) and protein in publicly accessible data banks, and that number is expanding at an exponential rate. Many scientific fields, such as bioinformatics and drug discovery, rely on such data; nevertheless, gathering and extracting data from these resources is a tough undertaking. This data should go through several processes, including mining, data processing, analysis, and classification. This study proposes software that extracts data from big data repositories automatically and with the particular ability to repeat data extraction phases as many times as needed without human intervention. This software simulates the extraction of data from web-based (point-and-click) resources or graphical user interfaces that cannot be accessed using command-line tools. The software was evaluated by creating a novel database of 34 parameters for 1360 physicochemical properties of antimicrobial peptides (AMP) sequences (46240 hits) from various MARVIN software panels, which can be later utilized to develop novel AMPs. Furthermore, for machine learning research, the program was validated by extracting 10,000 protein tertiary structures from the Protein Data Bank. As a result, data collection from the web will become faster and less expensive, with no need for manual data extraction. The software is critical as a first step to preparing large datasets for subsequent stages of analysis, such as those using machine and deep-learning applications.

Body Mass Index and Caries: Machine Learning and Statistical Analytics of the Dental, Oral, Medical Epidemiological (DOME) Nationwide Big Data Study.

The objectives of the research were to analyze the association between Body Mass Index (BMI) and dental caries using novel approaches of both statistical and machine learning (ML) models while adjusting for cardiovascular risk factors and metabolic syndrome (MetS) components, consequences, and related conditions. This research is a data-driven analysis of the Dental, Oral, Medical Epidemiological (DOME) big data repository, that integrates comprehensive socio-demographic, medical, and dental databases of a nationwide sample of dental attendees to military dental clinics for 1 year aged 18−50 years. Obesity categories were defined according to the World Health Organization (WHO): under-weight: BMI < 18.5 kg/m2, normal weight: BMI 18.5 to 24.9 kg/m2, overweight: BMI 25 to 29.9 kg/m2, and obesity: BMI ≥ 30 kg/m2. General linear models were used with the mean number of decayed teeth as the dependent variable across BMI categories, adjusted for (1) socio-demographics, (2) health-related habits, and (3) each of the diseases comprising the MetS definition MetS and long-term sequelae as well as associated illnesses, such as hypertension, diabetes, hyperlipidemia, cardiovascular disease, obstructive sleep apnea (OSA) and non-alcoholic fatty liver disease (NAFLD). After the statistical analysis, we run the XGBoost machine learning algorithm on the same set of clinical features to explore the features’ importance according to the dichotomous target variable of decayed teeth as well as the obesity category. The study included 66,790 subjects with a mean age of 22.8 ± 7.1. The mean BMI score was 24.2 ± 4.3 kg/m2. The distribution of BMI categories: underweight (3113 subjects, 4.7%), normal weight (38,924 subjects, 59.2%), overweight (16,966, 25.8%), and obesity (6736, 10.2%). Compared to normal weight (2.02 ± 2.79), the number of decayed teeth was statistically significantly higher in subjects with obesity [2.40 ± 3.00; OR = 1.46 (1.35−1.57)], underweight [2.36 ± 3.04; OR = 1.40 (1.26−1.56)] and overweight [2.08 ± 2.76, OR = 1.05 (1.01−1.11)]. Following adjustment, the associations persisted for obesity [OR = 1.56 (1.39−1.76)] and underweight [OR = 1.29 (1.16−1.45)], but not for overweight [OR = 1.11 (1.05−1.17)]. Features important according to the XGBoost model were socioeconomic status, teeth brushing, birth country, and sweetened beverage consumption, which are well-known risk factors of caries. Among those variables was also our main theory independent variable: BMI categories. We also performed clinical features importance based on XGBoost with obesity set as the target variable and received an AUC of 0.702, and accuracy of 0.896, which are considered excellent discrimination, and the major features that are increasing the risk of obesity there were: hypertension, NAFLD, SES, smoking, teeth brushing, age as well as our main theory dependent variable: caries as a dichotomized variable (Yes/no). The study demonstrates a positive association between underweight and obesity BMI categories and caries, independent of the socio-demographic, health-related practices, and other systemic conditions related to MetS that were studied. Better allocation of resources is recommended, focusing on populations underweight and obese in need of dental care.

Maritime information sharing environment deployment using the advanced multilayered Data Lake capabilities

Establishing an efficient information sharing network among national agencies in maritime domain is of essential importance in enhancing the operational performance, increasing the situational awareness and enabling interoperability among all involved maritime surveillance assets. Based on various data-driven technologies and sources, the EU initiative of Common Information Sharing Environment (CISE), enables the networked participants to timely exchange information concerning vessel traffic, joint SAR &amp; operational missions, emergency situations and other events at sea. In order to host and process vast amounts of vessels and related maritime data consumed from heterogeneous sources (e.g. SAT-AIS, UAV, radar, METOC), the deployment of big data repositories in the form of Data Lakes is of great added value. The different layers in the Data Lakes with capabilities for aggregating, fusing, routing and harmonizing data are assisted by decision support tools with combined reasoning modules with semantics aiming at providing a more accurate Common Operational Picture (COP) among maritime agencies. Based on these technologies, the aim of this paper is to present an end-to-end interoperability framework for maritime situational awareness in strategic and tactical operations at sea, developed in EFFECTOR EU-funded project, focusing on the multilayered Data Lake capabilities. Specifically, a case study presents the important sources and processing blocks, such as the SAT-AIS, CMEMS, UAV components, enabling maritime information exchange in CISE format and communication patterns. Finally, the technical solution is validated in the project’s recently implemented maritime operational trials and the respective results are documented.

Challenges Faced by the Traditional Medical Practitioners from Siddha System towards Transition to Information Technology-based Approaches at Chennai, India

Traditional System of Medicine has its strong roots in human life since eons. The wellbeing and health care needs of the Human kind has been taken care consistently by this profound historical medical system. One such system of medicine which has its origin from Southern part of India is Siddha System of Medicine. Siddha framework is a treasure-house of mystery science, epitomizing the consequences of the ardent pursuit thereof by the old Siddhars. The registered and traditional Siddha Doctors practice these ancient medical services - based on the treasured Palm scripts of Siddhars. A major part of these manuscripts are more defenseless than the recorded conventional information. In this manner, the current review was planned to report and quantitatively dissect the neighborhood information on the customary Siddha practitioners in Chennai. This article highlights the challenges encountered by the Siddha practitioners towards the transition to an information technology-based approach in the Siddha system of medicine at Chennai. In this study, a questionnaire was provided to 56 Siddha practitioners in Chennai and the data were evaluated based on a questionnaire pertaining to utilisation of big data repository, use of electronic reporting, personalized patient care management, evidence-based decision supporting system using Artificial Intelligence and Machine Learning and data on preventive diseases. The study observed that the practitioners felt a strong need for change to IT-based systems for recording the personalized patient care management and utilising the databases for reporting the various aspects observed.

Lifting the Veil on the Use of Big Data News Repositories: A Documentation and Critical Discussion of A Protest Event Analysis

ABSTRACT This paper presents a critical discussion of the processing, reliability and implications of free big data repositories. We argue that big data is not only the starting point of scientific analyses but also the outcome of a long string of invisible or semi-visible tasks, often masked by the fetish of size that supposedly lends validity to big data. We unpack these notions by illustrating the process of extracting protest event data from the Global Database of Events, Language and Tone (GDELT) in six European countries over a period of seven years. To stand up to rigorous scientific scrutiny, we collected additional data by computational means and undertook large-scale neural-network translation tasks, dictionary-based content analyses, machine-learning classification tasks, and human coding. In a documentation and critical discussion of this process, we render visible opaque procedures that inevitably shape any dataset and show how this type of freely available datasets require significant additional resources of knowledge, labor, money, and computational power. We conclude that while these processes can ultimately yield more valid datasets, the supposedly free and ready-to-use big news data repositories should not be taken at face value.

Controlled outputs, full data: A privacy‐protecting infrastructure for MOOC data

AbstractLearning analytics research presents challenges for researchers embracing the principles of open science. Protecting student privacy is paramount, but progress in increasing scientific understanding and improving educational outcomes depends upon open, scalable and replicable research. Findings have repeatedly been shown to be contextually dependent on personal and demographic variables, so how can we use this data in a manner that is ethical and secure for all involved? This paper presents ongoing work on the MOOC Replication Framework (MORF), a big data repository and analysis environment for Massive Open Online Courses (MOOCs). We discuss MORF's approach to protecting student privacy, which allows researchers to use data without having direct access. Through an open API, documentation and tightly controlled outputs, this framework provides researchers with the opportunity to perform secure, scalable research and facilitates collaboration, replication, and novel research. We also highlight ways in which MORF represents a solution template to issues surrounding privacy and security in the age of big data in education and key challenges still to be tackled. Practitioner notesWhat is already known about this topic Personal Identifying Information (PII) has many valid and important research uses in education. The ability to replicate or build on analyses is important to modern educational research, and is usually enabled through sharing data. Data sharing generally does not involve PII in order to protect student privacy. MOOCs present a rich data source for education researchers to better understand online learning. What this paper adds The MOOC replication framework (MORF) 2.1 is a new infrastructure that enables researchers to conduct analyses on student data without having direct access to the data, thus protecting student privacy. Detail of the MORF 2.1 structure and workflow. Implications for practice and/or policy MORF 2.1 is available for use by practitioners and research with policy implications. The infrastructure and approach in MORF could be applied to other types of educational data.

Digital Pathology and Artificial Intelligence Applications in Pathology

Digital pathology is revolutionizing pathology. The introduction of digital pathology made it possible to comprehensively change the pathology diagnosis workflow, apply and develop pathological artificial intelligence (AI) models, generate pathological big data, and perform telepathology. AI algorithms, including machine learning and deep learning, are used for the detection, segmentation, registration, processing, and classification of digitized pathological images. Pathological AI algorithms can be helpfully utilized for diagnostic screening, morphometric analysis of biomarkers, the discovery of new meanings of prognosis and therapeutic response in pathological images, and improvement of diagnostic efficiency. In order to develop a successful pathological AI model, it is necessary to consider the selection of a suitable type of image for a subject, utilization of big data repositories, the setting of an effective annotation strategy, image standardization, and color normalization. This review will elaborate on the advantages and perspectives of digital pathology, AI-based approaches, the applications in pathology, and considerations and challenges in the development of pathological AI models.

Identification, semantic annotation and comparison of combinations of functional elements in multiple biological conditions.

Approaches such as chromatin immunoprecipitation followed by sequencing (ChIP-seq) represent the standard for the identification of binding sites of DNA-associated proteins, including transcription factors and histone marks. Public repositories of omics data contain a huge number of experimental ChIP-seq data, but their reuse and integrative analysis across multiple conditions remain a daunting task. We present the Combinatorial and Semantic Analysis of Functional Elements (CombSAFE), an efficient computational method able to integrate and take advantage of the valuable and numerous, but heterogeneous, ChIP-seq data publicly available in big data repositories. Leveraging natural language processing techniques, it integrates omics data samples with semantic annotations from selected biomedical ontologies; then, using hidden Markov models, it identifies combinations of static and dynamic functional elements throughout the genome for the corresponding samples. CombSAFE allows analyzing the whole genome, by clustering patterns of regions with similar functional elements and through enrichment analyses to discover ontological terms significantly associated with them. Moreover, it allows comparing functional states of a specific genomic region to analyze their different behavior throughout the various semantic annotations. Such findings can provide novel insights by identifying unexpected combinations of functional elements in different biological conditions. The Python implementation of the CombSAFE pipeline is freely available for non-commercial use at: https://github.com/DEIB-GECO/CombSAFE. Supplementary data are available at Bioinformatics online.

Big Data Mining and Classification of Intelligent Material Science Data Using Machine Learning

There is a high need for a big data repository for material compositions and their derived analytics of metal strength, in the material science community. Currently, many researchers maintain their own excel sheets, prepared manually by their team by tabulating the experimental data collected from scientific journals, and analyzing the data by performing manual calculations using formulas to determine the strength of the material. In this study, we propose a big data storage for material science data and its processing parameters information to address the laborious process of data tabulation from scientific articles, data mining techniques to retrieve the information from databases to perform big data analytics, and a machine learning prediction model to determine material strength insights. Three models are proposed based on Logistic regression, Support vector Machine SVM and Random Forest Algorithms. These models are trained and tested using a 10-fold cross validation approach. The Random Forest classification model performed better on the independent dataset, with 87% accuracy in comparison to Logistic regression and SVM with 72% and 78%, respectively.

Gaps in standards for integrating artificial intelligence technologies into ophthalmic practice.

The purpose of this review is to provide an overview of healthcare standards and their relevance to multiple ophthalmic workflows, with a specific emphasis on describing gaps in standards development needed for improved integration of artificial intelligence technologies into ophthalmic practice. Healthcare standards are an essential component of data exchange and critical for clinical practice, research, and public health surveillance activities. Standards enable interoperability between clinical information systems, healthcare information exchange between institutions, and clinical decision support in a complex health information technology ecosystem. There are several gaps in standards in ophthalmology, including relatively low adoption of imaging standards, lack of use cases for integrating apps providing artificial intelligence -based decision support, lack of common data models to harmonize big data repositories, and no standards regarding interfaces and algorithmic outputs. These gaps in standards represent opportunities for future work to develop improved data flow between various elements of the digital health ecosystem. This will enable more widespread adoption and integration of artificial intelligence-based tools into clinical practice. Engagement and support from the ophthalmology community for standards development will be important for advancing this work.