Data-driven Pipeline Research Articles

Nucleosome placement and polymer mechanics explain genomic contacts on 100kbp scales.

The 3d organization of the genome - in particular, which two regions of DNA are in contact with each other - plays a role in regulating gene expression. Several factors influence genome 3d organization. Nucleosomes (where ~ 100 basepairs of DNA wrap around histone proteins) also bend, twist and compactify chromosomal DNA, altering its polymer mechanics. How much does the positioning of nucleosomes between gene loci influence contacts between those gene loci? And, to what extent is polymer mechanics responsible for this? To address this question, we combine a stochastic polymer mechanics model of chromosomal DNA including twists and wrapping induced by nucleosomes with two data-driven pipelines. The first estimates nucleosome positioning from ATACseq data in regions of high accessibility. Most of the genome is low-accessibility, so we combine this with a novel image analysis method that estimates the distribution of nucleosome spacing from electron microscopy data. There are no free parameters in the biophysical model. We apply this method to IL6, IL15, CXCL9, and CXCL10, inflammatory marker genes in macrophages, before and after immune stimulation, and compare the predictions with contacts measured by conformation capture experiments (4C-seq). We find that within a 500 kilo-basepairs genomic region, polymer mechanics with nucleosomes can explain 71% of close contacts. These results suggest that, while genome contacts on 100kbp-scales are multifactorial, they may be amenable to mechanistic, physical explanation. Our work also highlights the role of nucleosomes, not just at the loci of interest, but between them, and not just the total number of nucleosomes, but their specific placement. The method generalizes to other genes, and can be used to address whether a contact is under active regulation by the cell (e.g., a macrophage during inflammatory stimulation). Importantly, our findings suggest that gene function may have evolved through selective pressures that co-opted contact-mediated regulatory mechanisms reliant largely on polymer mechanics.

Exploring the feasibility of Support Vector Machine for short-term hydrological forecasting in South Tyrol: challenges and prospects

Short-term hydrological forecasting is crucial for suitable multipurpose water resource management involving water uses, hydrological security, and renewable production. In the Alpine Regions such as South Tyrol, characterized by several small watersheds, quick information is essential to feed the decision processes in critical cases such as flood events. Predicting water availability ahead is equally crucial for optimizing resource utilization, such as irrigation or snow-making. The increasing data availability and computational power led to data-driven models becoming a serious alternative to physically based hydrological models, especially in complex conditions such as the Alpine Region and for short predictive horizons. This paper proposes a data-driven pipeline to use the local ground station data to infer information in a Support Vector Regression model, which can forecast streamflow in the main closure points of the area at hourly resolution with 48 h of lead time. The main steps of the pipeline are analysed and discussed, with promising results that depend on available information, watershed complexity, and human interactions in the catchment. The presented pipeline, as it stands, offers an accessible tool for integrating these models into decision-making processes to guarantee real-time streamflow information at several points of the hydrological network. Discussion enhances the potentialities, open challenges, and prospects of short-term streamflow forecasting to accommodate broader studies.

Inference of highly time-resolved melt pool visual characteristics and spatially-dependent lack-of-fusion defects in laser powder bed fusion using acoustic and thermal emission data

With a growing demand for high-quality fabrication, the interest in real-time process and defect monitoring of laser powder bed fusion (LPBF) has increased, leading manufacturers to incorporate a variety of online sensing methods including acoustic sensing, photodiode sensing, and high-speed imaging. However, real-time acquisition of high-resolution melt pool images in particular remains computationally demanding in practice due to the high variability of melt pool morphologies and the limitation of data caching and transfer, making it challenging to detect the local lack-of-fusion (LOF) defect occurrences. In this work, we propose a new acoustic and thermal information-based monitoring method that can robustly infer critical LPBF melt pool morphological features in image forms and detect spatially-dependent LOF defects within a short time period. We utilize wavelet scalogram matrices of acoustic and photodiode clip data to identify and predict highly time-resolved (within a 1.0 ms window) visual melt pool characteristics via a well-trained data-driven pipeline. With merely the acoustic and photodiode-collected thermal emission data as the input, the proposed pipeline enables data-driven inference and tracking of highly variable melt pool visual characteristics with R2≥0.8. We subsequently validate our proposed approach to infer local LOF defects between two adjacent scanlines, showing that our proposed approach can outperform our selected baseline theoretical model based on previous literature. Revealing the physical correlation between airborne acoustic emission, thermal emission, and melt pool morphology, our work demonstrates the feasibility of creating an efficient and cost-effective acoustic- and thermal-based approach to facilitate online visual melt pool characterization and LOF defect detection. We believe that our work can further contribute to the advances in quality control for LPBF.

Development of machine learning-based models to predict 10-year risk of cardiovascular disease: a prospective cohort study

BackgroundPrevious prediction algorithms for cardiovascular diseases (CVD) were established using risk factors retrieved largely based on empirical clinical knowledge. This study sought to identify predictors among a comprehensive variable space,...

Parallel Optimization for Large Scale Interferometric Synthetic Aperture Radar Data Processing

Interferometric synthetic aperture radar (InSAR) has developed rapidly over the past years and is considered as an important method for surface deformation monitoring, benefiting from growing data quantities and improving data quality. However, the handing of SAR big data poses significant challenges for related algorithms and pipeline, particularly in large-scale SAR data processing. In addition, InSAR algorithms are highly complex, and their task dependencies are intricate. There is a lack of efficient optimization models and task scheduling for InSAR pipeline. In this paper, we design parallel time-series InSAR processing models based on multi-thread technology for high efficiency in processing InSAR big data. These models concentrate on parallelizing critical algorithms that have high complexity, with a focus on deconstructing two computationally intensive algorithms through loop unrolling. Our parallel models have shown a significant improvement of 10–20 times in performance. We have also developed a parallel optimization tool, Simultaneous Task Automatic Runtime (STAR), which utilizes a data flow optimization strategy with thread pool technology to address the problem of low CPU utilization resulting from multiple modules and task dependencies in the InSAR processing pipeline. STAR provides a data-driven pipeline and enables concurrent execution of multiple tasks, with greater flexibility to keep the CPU busy and further improve CPU utilization through predetermined task flow. Additionally, a supercomputing-based system has been constructed for processing massive InSAR scientific big data and providing technical support for nationwide surface deformation measurement, in accordance with the framework of time series InSAR data processing. Using this system, we processed InSAR data with the volumes of 500 TB and 700 TB in 5 and 7 days, respectively. Finally we generated two maps of land surface deformation all over China.

Automatic identification of urban functions via social mining

Social network users upload billions of geo-referenced data daily, such as photos, videos, and descriptions. While sharing the users' daily moments, these contents can provide distributed and, compared to traditional collections, real-time insights into the interactions between humans and the territory. This paper presents a new methodology to leverage geo-referenced data, linking latent technology data to the municipalities' need for feedback and guidance on past and new investments supporting urban function identification via unsupervised algorithms. We propose a 3-step data-driven pipeline. In the first step, we collect geotagged images posted on Flickr and Instagram and treat them as points on a map. Then, through clustering techniques, we explore a city's geographic contours and investigate how the users' interaction with the city evolved. Eventually, we exploit classification tools to determine what primarily characterizes the city areas. The pipeline, which can be generalized to any geographical area at any level of detail, is verified for the city of Turin through peak detection and zoning tasks. The results demonstrate that the pipeline's outcomes are coherent with the city's traditional characterizations and that social data can provide valid support for the urban experts' analysis.

A Cognitive Model for Technology Adoption

The widespread adoption of advanced technologies, such as Artificial Intelligence (AI), Machine Learning, and Robotics, is rapidly increasing across the globe. This accelerated pace of change is drastically transforming various aspects of our lives and work, resulting in what is now known as Industry 4.0. As businesses integrate these technologies into their daily operations, it significantly impacts their work tasks and required skill sets. However, the approach to technological transformation varies depending on location, industry, and organization. However, there are no published methods that can adequately forecast the adoption of technology and its impact on society. It is essential to prepare for the future impact of Industry 4.0, and this requires policymakers and business leaders to be equipped with scientifically validated models and metrics. Data-driven scenario planning and decision-making can lead to better outcomes in every area of the business, from learning and development to technology investment. However, the current literature falls short in identifying effective and globally applicable strategies to predict the adoption rate of emerging technologies. Therefore, this paper proposes a novel parametric mathematical model for predicting the adoption rate of emerging technologies through a unique data-driven pipeline. This approach utilizes global indicators for countries to predict the technology adoption curves for each country and industry. The model is thoroughly validated, and the paper outlines highly promising evaluation results. The practical implications of this proposed approach are significant because it provides policymakers and business leaders with valuable insights for decision-making and scenario planning.

Data-driven selection and spectral classification of white dwarf stars

ABSTRACT The next generation of spectroscopic surveys is expected to provide spectra for hundreds of thousands of white dwarf (WD) candidates in the upcoming years. Currently, spectroscopic classification of white dwarfs is mostly done by visual inspection, requiring substantial amounts of expert attention. We propose a data-driven pipeline for fast, automatic selection, and spectroscopic classification of WD candidates, trained using spectroscopically confirmed objects with available Gaia astrometry, photometry, and Sloan Digital Sky Survey (SDSS) spectra with signal-to-noise ratios ≥9. The pipeline selects WD candidates with improved accuracy and completeness over existing algorithms, classifies their primary spectroscopic type with ${\gtrsim}90\ \hbox{per cent}$ accuracy, and spectroscopically detects main sequence companions with similar performance. We apply our pipeline to the Gaia Data Release 3 cross-matched with the SDSS Data Release 17 (DR17), identifying 424 096 high-confidence WD candidates and providing the first catalogue of automated and quantifiable classification for 36 523 WD spectra. Both the catalogue and pipeline are made available online. Such a tool will prove particularly useful for the undergoing SDSS-V survey, allowing for rapid classification of thousands of spectra at every data release.

Data-driven design of the Belle II track segment finder

The Belle II experiment relies on a level-1 trigger system to reduce noise background and preselect events of interest for particle physics. The Central Drift Chamber is the main track detector which makes its trigger system important for online track reconstruction. To improve its hit efficiency, an extension of the track segment finder for low angle tracks is proposed. By combining hardware and software development flows, an automated data-driven pipeline is created and three different-sized hardware concepts are implemented. The operation point is adjustable to balance hit efficiency against hit purity in the trigger system.

Naturally-meaningful and efficient descriptors: machine learning of material properties based on robust one-shot ab initio descriptors

Establishing a data-driven pipeline for the discovery of novel materials requires the engineering of material features that can be feasibly calculated and can be applied to predict a material’s target properties. Here we propose a new class of descriptors for describing crystal structures, which we term Robust One-Shot Ab initio (ROSA) descriptors. ROSA is computationally cheap and is shown to accurately predict a range of material properties. These simple and intuitive class of descriptors are generated from the energetics of a material at a low level of theory using an incomplete ab initio calculation. We demonstrate how the incorporation of ROSA descriptors in ML-based property prediction leads to accurate predictions over a wide range of crystals, amorphized crystals, metal–organic frameworks and molecules. We believe that the low computational cost and ease of use of these descriptors will significantly improve ML-based predictions.

Semi-supervised learning framework for oil and gas pipeline failure detection

Quantifying failure events of oil and gas pipelines in real- or near-real-time facilitates a faster and more appropriate response plan. Developing a data-driven pipeline failure assessment model, however, faces a major challenge; failure history, in the form of incident reports, suffers from limited and missing information, making it difficult to incorporate a persistent input configuration to a supervised machine learning model. The literature falls short on the development of appropriate solutions to utilize incomplete databases and incident reports in the pipeline failure problem. This work proposes a semi-supervised machine learning framework which mines existing oil and gas pipeline failure databases. The proposed cluster-impute-classify (CIC) approach maps a relevant subset of the failure databases through which missing information in the incident report is reconstructed. A classifier is then trained on the fly to learn the functional relationship between the descriptors from a diverse feature set. The proposed approach, presented within an ensemble learning architecture, is easily scalable to various pipeline failure databases. The results show up to 91% detection accuracy and stable generalization ability against increased rate of missing information.

Automated quantification of brain connectivity in Alzheimer's disease using ClusterMetric

Diffusion magnetic resonance imaging tractography allows investigating brain structural connections in a noninvasive way and has been widely used for understanding neurological disease. Quantification of brain connectivity along with its length by dividing a fiber bundle into multiple segments (node) is a powerful approach to assess biological properties, which is termed as tractometry. However, current tractometry methods face challenges in node identification along with the length of complex bundles whose morphology is difficult to summarize. In addition, the anatomic measure reflecting the macroscopic fiber cross-section has not been followed in previous tractometry. In this paper, we propose an automated fiber bundle quantification, which we refer to as ClusterMetric. The ClusterMetric uses a data-driven approach to identify fiber clusters corresponding to subdivisions of the white matter anatomy and identify consistent space nodes along the length of clusters across individuals. The proposed method is demonstrated by applicating to our collected dataset including 23 Alzheimer's disease (AD) patients and 22 healthy controls (HCs) and a public dataset of ADNI including 53 AD patients and 85 HCs. The altered white matter tracts in AD group are observed using both datasets, which involve several major fiber tracts including the corpus callosum, corona-radiata-frontal, arcuate fasciculus, inferior occipito-frontal fasciculus, uncinate fasciculus, thalamo-frontal, superior longitudinal fasciculus, inferior cerebellar peduncle, cingulum bundle, and extreme capsule. These fiber clusters represent the white matter connections that could be most affected in AD, suggesting the ability of our method in identifying potential abnormalities specific to local regions within a fiber cluster.

A data-driven pipeline to extract potential adverse drug reactions through prescription, procedures and medical diagnoses analysis: application to a cohort study of 2,010 patients taking hydroxychloroquine with an 11-year follow-up.

ContextReal-life data consist of exhaustive data which are not subject to selection bias. These data enable to study drug-safety profiles but are underused because of their temporality, necessitating complex models (i.e., safety depends on the dose, timing, and duration of treatment). We aimed to create a data-driven pipeline strategy that manages the complex temporality of real-life data to highlight the safety profile of a given drug.MethodsWe proposed to apply the weighted cumulative exposure (WCE) statistical model to all health events occurring after a drug introduction (in this paper HCQ) and performed bootstrap to select relevant diagnoses, drugs and interventions which could reflect an adverse drug reactions (ADRs). We applied this data-driven pipeline on a French national medico-administrative database to extract the safety profile of hydroxychloroquine (HCQ) from a cohort of 2,010 patients.ResultsThe proposed method selected eight drugs (metopimazine, anethole trithione, tropicamide, alendronic acid & colecalciferol, hydrocortisone, chlormadinone, valsartan and tixocortol), twelve procedures (six ophthalmic procedures, two dental procedures, two skin lesions procedures and osteodensitometry procedure) and two medical diagnoses (systemic lupus erythematous, unspecified and discoid lupus erythematous) to be significantly associated with HCQ exposure.ConclusionWe provide a method extracting the broad spectrum of diagnoses, drugs and interventions associated to any given drug, potentially highlighting ADRs. Applied to hydroxychloroquine, this method extracted among others already known ADRs.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12874-022-01628-3.

Automated soccer head impact exposure tracking using video and deep learning

Head impacts are highly prevalent in sports and there is a pressing need to investigate the potential link between head impact exposure and brain injury risk. Wearable impact sensors and manual video analysis have been utilized to collect impact exposure data. However, wearable sensors suffer from high deployment cost and limited accuracy, while manual video analysis is a long and resource-intensive task. Here we develop and apply DeepImpact, a computer vision algorithm to automatically detect soccer headers using soccer game videos. Our data-driven pipeline uses two deep learning networks including an object detection algorithm and temporal shift module to extract visual and temporal features of video segments and classify the segments as header or nonheader events. The networks were trained and validated using a large-scale professional-level soccer video dataset, with labeled ground truth header events. The algorithm achieved 95.3% sensitivity and 96.0% precision in cross-validation, and 92.9% sensitivity and 21.1% precision in an independent test that included videos of five professional soccer games. Video segments identified as headers in the test data set correspond to 3.5 min of total film time, which can be reviewed through additional manual video verification to eliminate false positives. DeepImpact streamlines the process of manual video analysis and can help to collect large-scale soccer head impact exposure datasets for brain injury research. The fully video-based solution is a low-cost alternative for head impact exposure monitoring and may also be expanded to other sports in future work.

Mining Electronic Health Records for Drugs Associated With 28-day Mortality in COVID-19: Pharmacopoeia-wide Association Study (PharmWAS).

BackgroundPatients hospitalized for a given condition may be receiving other treatments for other contemporary conditions or comorbidities. The use of such observational clinical data for pharmacological hypothesis generation is appealing in the context of an emerging disease but particularly challenging due to the presence of drug indication bias.ObjectiveWith this study, our main objective was the development and validation of a fully data-driven pipeline that would address this challenge. Our secondary objective was to generate pharmacological hypotheses in patients with COVID-19 and demonstrate the clinical relevance of the pipeline.MethodsWe developed a pharmacopeia-wide association study (PharmWAS) pipeline inspired from the PheWAS methodology, which systematically screens for associations between the whole pharmacopeia and a clinical phenotype. First, a fully data-driven procedure based on adaptive least absolute shrinkage and selection operator (LASSO) determined drug-specific adjustment sets. Second, we computed several measures of association, including robust methods based on propensity scores (PSs) to control indication bias. Finally, we applied the Benjamini and Hochberg procedure of the false discovery rate (FDR). We applied this method in a multicenter retrospective cohort study using electronic medical records from 16 university hospitals of the Greater Paris area. We included all adult patients between 18 and 95 years old hospitalized in conventional wards for COVID-19 between February 1, 2020, and June 15, 2021. We investigated the association between drug prescription within 48 hours from admission and 28-day mortality. We validated our data-driven pipeline against a knowledge-based pipeline on 3 treatments of reference, for which experts agreed on the expected association with mortality. We then demonstrated its clinical relevance by screening all drugs prescribed in more than 100 patients to generate pharmacological hypotheses.ResultsA total of 5783 patients were included in the analysis. The median age at admission was 69.2 (IQR 56.7-81.1) years, and 3390 (58.62%) of the patients were male. The performance of our automated pipeline was comparable or better for controlling bias than the knowledge-based adjustment set for 3 reference drugs: dexamethasone, phloroglucinol, and paracetamol. After correction for multiple testing, 4 drugs were associated with increased in-hospital mortality. Among these, diazepam and tramadol were the only ones not discarded by automated diagnostics, with adjusted odds ratios of 2.51 (95% CI 1.52-4.16, Q=.01) and 1.94 (95% CI 1.32-2.85, Q=.02), respectively.ConclusionsOur innovative approach proved useful in generating pharmacological hypotheses in an outbreak setting, without requiring a priori knowledge of the disease. Our systematic analysis of early prescribed treatments from patients hospitalized for COVID-19 showed that diazepam and tramadol are associated with increased 28-day mortality. Whether these drugs could worsen COVID-19 needs to be further assessed.

Knowledge-Driven Data Ecosystems Toward Data Transparency

A data ecosystem (DE) offers a keystone-player or alliance-driven infrastructure that enables the interaction of different stakeholders and the resolution of interoperability issues among shared data. However, despite years of research in data governance and management, trustability is still affected by the absence of transparent and traceable data-driven pipelines. In this work, we focus on requirements and challenges that DEs face when ensuring data transparency. Requirements are derived from the data and organizational management, as well as from broader legal and ethical considerations. We propose a novel knowledge-driven DE architecture, providing the pillars for satisfying the analyzed requirements. We illustrate the potential of our proposal in a real-world scenario. Last, we discuss and rate the potential of the proposed architecture in the fulfillmentof these requirements.

Optimized data-driven pipeline for digital mapping of quantitative and categorical properties of soils in Colombia

Soil maps provide a method for graphically communicating what is known about the spatial distribution of soil properties in nature. We proposed an optimized pipeline, named dino-soil toolbox, programmed in the R software for mapping quantitative and categorical properties of legacy soil data. The pipeline, composed of four main modules (data preprocessing, covariates selection, exploratory data analysis and modeling), was tested across a study area of 14,537 km 2 located between the departments of Cesar and Magdalena, Colombia. We [...]

Data-driven strategies for predictive maintenance: Lesson learned from an automotive use case

Predictive maintenance is an ever-growing topic of interest, spanning different fields and approaches. In the automotive domain, thanks to on-board sensors and the possibility to transmit collected data to the cloud, car manufacturers can deploy predictive maintenance solutions to prevent components malfunctioning and eventually recall to the service the vehicle before the customer experiences the failure. In this paper we present PREPIPE, a data-driven pipeline for predictive maintenance. Given the raw time series of signals recorded by the on-board engine control unit of diesel engines, we exploit PREPIPE to predict the clogging status of the oxygen sensor, a key component of the exhaust system to control combustion efficiency and pollutant emissions. In the design of PREPIPE we deeply investigate: (i) how to choose the best subset of signals to best capture the sensor status, (ii) how much data needs to be collected to make the most accurate prediction, (iii) how to transform the original time series into features suitable for state-of-art classifiers, (iv) how to select the most important features, (v) how to include historical features to predict the clogging status of the sensor. We thoroughly assess PREPIPE performance and compare it with state-of-art deep learning architectures. Our results show that PREPIPE correctly identifies critical situations before the sensor reaches critical conditions. Furthermore, PREPIPE supports domain experts in optimizing the design of data-driven predictive maintenance pipelines with performance comparable to deep learning methodologies while keeping a degree of interpretability.

Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments

The control of modern buildings is a complex multi-loop problem due to the integration of renewable energy generation, storage devices, and electric vehicles (EVs). Additionally, it is a complex multi-criteria problem due to the need to optimize overall energy use while satisfying users’ comfort. Both conventional rule-based (RB) controllers, which are difficult to apply in multi-loop settings, and advanced model-based controllers, which require an accurate building model, cannot fulfil the requirements of the building automation industry to solve this problem optimally at low development and commissioning costs. This work presents a fully data-driven pipeline to obtain an optimal control policy from historical building and weather data, thus avoiding the need for complex physics-based modelling. We demonstrate the potential of this method by jointly controlling a room temperature and an EV to minimize the cost of electricity while retaining the comfort of the occupants. We model the room temperature with a recurrent neural network and use it as a simulation environment to learn a deep reinforcement learning (DRL) control policy. It achieves on average 17% energy savings and 19% better comfort satisfaction than a standard RB room temperature controller. When a bidirectional EV is connected additionally and a two-tariff electricity pricing is applied, it successfully leverages the battery and decreases the overall cost of electricity. Finally, we deployed it on a real building, where it achieved up to 30% energy savings while maintaining similar comfort levels compared to a conventional RB room temperature controller.

Seeing the forest for the trees: Predicting attendance in trials for co-occurring PTSD and substance use disorders with a machine learning approach.

Objective: High dropout rates are common in randomized clinical trials (RCTs) for comorbid posttraumatic stress disorder and substance use disorders (PTSD + SUD). Optimizing attendance is a priority for PTSD + SUD treatment development, yet research has found few consistent associations to guide responsive strategies. In this study, we employed a data-driven pipeline for identifying salient and reliable predictors of attendance. Method: In a novel application of the iterative Random Forest algorithm (iRF), we investigated the association of individual level characteristics and session attendance in a completed RCT for PTSD + SUD (n = 70; women = 22 [31.4%]). iRF identified a group of potential predictor candidates for the total trial sessions attended; then, a Poisson regression model assessed the association between the iRF-identified factors and attendance. As a validation set, a parallel regression of significant predictors was conducted on a second, independent RCT for PTSD + SUD (n = 60; women = 48 [80%]). Results: Two testable hypotheses were derived from iRF's variable importance measures. Faster within-treatment improvement of PTSD symptoms was associated with greater session attendance with age moderating this relationship (p = .01): faster PTSD symptom improvement predicted fewer sessions attended among younger patients and more sessions among older patients. Full-time employment was also associated with fewer sessions attended (p = .02). In the validation set, the interaction between age and speed of PTSD improvement was significant (p = .05) and the employment association was not. Conclusions: Results demonstrate the potential of data-driven methods to identifying meaningful predictors as well as the dynamic contribution of symptom change during treatment to understanding RCT attendance. (PsycInfo Database Record (c) 2021 APA, all rights reserved).