Processing Pipeline Research Articles

Automated Image Segmentation and Processing Pipeline Applied to X‐Ray Computed Tomography Studies of Pitting Corrosion in Aluminum Wires

Understanding pitting corrosion is critical, yet its kinetics and morphology remain challenging to study from X‐ray computed tomography (XCT) due to manual segmentation barriers. To address this, an automated pipeline leveraging deep learning for efficient large‐scale XCT analysis is developed, revealing new corrosion insights. The pipeline enables pit segmentation, 3D reconstruction, statistical characterization, and a topological transformation for visualization. The pipeline is applied to 87 648 XCT images capturing commercial purity aluminum (1100 Al) wire exposed to sodium chloride (NaCl) salt particles over a period of 122 h. The pipeline achieves complete feature extraction and statistical quantification across the entire XCT dataset, leveraging distributed computing environment for high efficiency. Global growth kinetics such as high‐level stepwise sigmoidal volume loss patterns and granular individual pit developments are both captured for 36 detected pits. By combining automation, computer vision, and extensive XCT datasets, this research accelerates precise corrosion assessment to enable materials science discoveries at scale.

FlowCron - Increasing access to HPC by wrapping Globus into a function-as-a-service

Despite significant investment in High-Performance Computing (HPC) clusters by funding councils, there are still many researchers whose workflows could not benefit from the computation speed that is provided by these clusters. Reducing barriers to entry for these researchers would accelerate their scientific throughput, since they will be able to respond to results in a timely fashion, improving either their protocols or correcting any problems that might have arisen. This improves the quality of science, and therefore the return on investment, in computationally-intensive areas such as Cryogenic Electron Microscopy (cryo-EM). This paper outlines a technique, FlowCron, for users to analyse their data on a HPC facility with minimal training, increasing accessibility. FlowCron transfers the responsibilities of installation and upkeep of data processing pipelines from users to HPC systems administrators, simplifies the set up of HPC pipelines, and makes pipelines as reliable as possible once set up. The work described here has software dependencies that are common to the majority of HPC clusters. We achieve this by linking Globus and cron to produce an open-source system that requires little administrative support but provides a very easy way of running an analysis on a HPC system. The user starts the analysis through the Globus website and, when started, the data will be encrypted, uploaded to the HPC, analysed, and returned to the originating machine, along with a record of the analysis.

Pipeline ShiftAddNet: An FPGA‐Based CNN Implementation With Low Hardware Consumption Targeting Constrained Devices

ABSTRACTShiftAddNet is a recently proposed multiplier‐less CNN that replaces conventional multiplication with cheaper shift and add operations, which makes it suitable for hardware implementation. In this paper, we present the first implementation of ShiftAddNet FPGA inference core, which achieves low area consumption and fast computation. ShiftAddNet combined the convolutional layer of the DeepShift‐PS (denoted as Shift‐Accumulate, sac) and AdderNet (denoted as Add‐Accumulate, aac) into a single computational stage. Due to this reason, there are data dependencies between the sac and aac, which prohibits them from being executed in parallel, resulting in more operations compared to other multiplier‐less CNNs like DeepShift‐PS and AdderNet. To overcome this performance bottleneck, we proposed a novel technique to allow pipeline processing between sac and aac, effectively reducing the latency. The proposed ShiftAddNet‐18 was evaluated on a small ResNet‐18, achieving 11.37 ms of latency per image, which is 69.21% faster than the original version that takes 19.24 ms. On a denser network, the proposed pipeline ShiftAddNet‐101 requires only 61.92 ms as compared to the original version of 98.85 ms, showing a latency reduction of 37.1%. Compared to the state‐of‐the‐art multiplier‐less CNN core (e.g., AdderNet), our work is 20% slower in latency but provides higher accuracy and consumes less DSP.

A Pilot Report on Extracting Symptom Onset Date and Time from Clinical Notes in Patients Presenting with Chest Pain.

Acute coronary syndrome (ACS) is an acute heart disease that often evolves rapidly. In ACS patients presenting with no-ST-segment elevation (NSTE-ACS), the timing of symptom onset pre-hospital may inform the disease stage and prognosis. We pilot-tested two off-the-shelf natural language processing (NLP) pipelines, namely parsedatetime and regular expression ( regex ), to extract date and time (DateTime) information of patient-reported chest pain symptoms from electronic health records (EHR) clinical notes. We included three types of clinical notes (N=71): History and Physical (n=49), Emergency Department Screening (n=3), and Triage Notes (n=19). All notes were manually annotated for the true DateTime of symptom onset. Parsedatetime returned matching DateTime outputs in 36 notes (50.7%), while regex returned zero matched outputs. Parsedatetime performed better than regex , although it was still suboptimal. Both pipelines require constant refinement and custom improvements. Methods for a large-scale, automated DateTime extraction from EHR clinical notes further investigation.

Quantifying Similarity between Graph-Theoretic Resting-State fMRI Data Processing Pipelines for Efficient Multiverse Analysis

Quantifying Similarity between Graph-Theoretic Resting-State fMRI Data Processing Pipelines for Efficient Multiverse Analysis

Reverse Engineering GitHub CoPilot: Creating an OpenAI-Compatible Endpoint for Enhanced Developer Integration

This paper presents the reverse engineering of GitHub CoPilot to develop an OpenAI-compatible endpoint, enabling broader access and integration possibilities for AI-assisted code completion. By analyzing CoPilot's communication protocols and creating a proxy server that translates OpenAI API requests to CoPilot's internal API, we bridge the gap between proprietary tools and open standards. The implementation, allows developers to utilize CoPilot's capabilities within their preferred environments using the familiar OpenAI API interface. We detail the system architecture, authentication mechanisms, request processing pipeline, and performance optimization techniques. Our results demonstrate successful integration, with robust performance metrics, including low response times and high compatibility rates. This work opens avenues for enhanced developer productivity and flexibility in AI-assisted coding tools.

Using machine learning to predict disease outbreaks and enhance public health surveillance

Disease outbreaks pose significant challenges to public health systems, often requiring rapid response strategies to mitigate widespread health and economic impacts. Traditional methods of outbreak prediction and surveillance, while effective, often lack the capacity to process and analyse the vast quantities of heterogeneous data generated in modern healthcare ecosystems. Machine learning (ML) offers transformative potential in this domain, leveraging its ability to process large datasets, identify complex patterns, and provide real-time insights. By integrating diverse data sources such as electronic health records (EHRs), social media feeds, climate data, and genomic sequences, ML algorithms can predict disease outbreaks with unprecedented accuracy. Supervised learning models, for instance, have been successfully applied to forecast influenza trends, while unsupervised clustering techniques have been employed to detect anomalies indicative of emerging infectious diseases. Moreover, ML facilitates advanced public health surveillance by automating data processing pipelines, enhancing real-time monitoring capabilities, and enabling resource optimization for outbreak responses. Despite these advances, the adoption of ML in public health surveillance is not without challenges. Issues related to data privacy, ethical considerations, algorithm interpretability, and integration with existing public health infrastructures remain significant hurdles. Addressing these challenges requires a multidisciplinary approach, incorporating robust data governance frameworks, improved algorithm transparency, and collaborations between technology developers and public health stakeholders. This paper highlights the critical role of ML in transforming public health surveillance, focusing on its application in disease outbreak prediction. It underscores the importance of continued innovation, regulatory support, and ethical considerations in advancing ML-driven solutions for global health security.

A Systematic Search for Redback and Black Widow Candidates Based on the 4FGL-DR3 Unassociated Sources and the Zwicky Transient Facility Data

Spider pulsars represent a unique subclass of radio millisecond pulsars in binaries, and are further categorized into black widows and redbacks according to the mass of the low-mass companion. These pulsars, observable across multiple wavelengths, exhibit periodic variability in optical. The discovery and study of additional spider-type pulsars are crucial for a fuller understanding the evolution of binary stars in close orbits and the recycling theory of millisecond pulsars. In this work, we systematically searched for spider pulsar binary systems using time-domain variability data from the Zwicky Transient Facility and unassociated gamma-ray sources from the Fermi 4FGL-DR3 catalog. We developed a time-domain data processing pipeline that employs the Lomb–Scargle periodogram algorithm. As a result, we identified a total of 194 ellipsoidal variables and irradiation-type binary stars. Further refinement using the Gaia Hertzsprung–Russell diagram resulted in a selection of 24 spider pulsar candidates. Incorporating the 4FGL 95% confidence error ellipse reduced the sample to 19 candidates. An additional filter using the Gaia color-reduced proper motion diagram yielded nine “gold sample” candidates. These newly identified spider pulsar candidates will guide future observational campaigns in radio, X-ray, and optical spectroscopy, aiding in the comprehensive validation of their nature.

A mini review of leveraging biobanking in the identification of novel biomarkers in neurological disorders: insights from a rapid single-cell sequencing pipeline.

Recent successes in the identification of biomarkers and therapeutic targets for diagnosing and managing neurological diseases underscore the critical need for cutting-edge biobanks in the conduct of high-caliber translational neuroscience research. Biobanks dedicated to neurological disorders are particularly timely, given the increasing prevalence of neurological disability among the rising aging population. Translational research focusing on disorders of the central nervous system (CNS) poses distinct challenges due to the limited accessibility of CNS tissue pre-mortem. Nevertheless, technological breakthroughs, including single-cell and single-nucleus methodologies, offer unprecedented insights into CNS pathophysiology using minimal input such as cerebrospinal fluid (CSF) cells and brain biopsies. Moreover, assays designed to detect factors that are released by CNS resident cells and diffuse into the CSF and/or bloodstream (such as neurofilament light chain [NfL], glial fibrillar acidic protein [GFAP] and amyloid beta peptides), and systemic factors that cross the blood-brain barrier to target CNS-specific molecules (e.g., autoantibodies that bind either the NMDA receptor [NMDAR] or myelin oligodendrocyte glycoprotein [MOG]), are increasingly deployed in clinical research and practice. This review provides an overview of current biobanking practices in neurological disorders and discusses ongoing challenges to biomarker discovery. Additionally, it outlines a rapid consenting and processing pipeline ensuring fresh paired blood and CSF specimens for single-cell sequencing that might more accurately reflect in vivo pathways. In summary, augmenting biobank rigor and establishing innovative research pipelines using patient samples will undoubtedly accelerate biomarker discovery in neurological disorders.

AI-Driven Cloud Platform for Optimizing Short-Term Rental Property Furnishings

Abstract—Short-term rental platforms like Airbnb and VRBO have revolutionized the hospitality industry, creating a pressing need for efficient property setup tools tailored to diverse rental audiences. This paper introduces Hostly, an innovative AI-powered cloud platform that provides personalized recommendations for furnishing short- term rental properties. Built on Amazon Web Services (AWS), Hostly incorporates AI recommendation systems, real-time vendor integration, cost-ROI analytics, and secure payment processing. The platform simplifies property setup by curating tailored furnishing solutions, maximizing aesthetic appeal, functionality, and return on investment (ROI). Technical details of the AI architecture, cloud infrastructure, and payment processing pipelines are discussed. The paper also evaluates business impacts and future scalability Keywords—AI recommendation systems, AWS cloud services, Analytics dashboards, Cloud infrastructure, E- commerce, PCI DSS compliance, Payment processing, Short-term rental optimization, Vendor API integration.

Development of an Optimized LC-MS Workflow for Host Cell Protein Characterization to Support Upstream Process Development.

Host cell proteins (HCPs) coexpressed during the production of biotherapeutics can affect the safety, efficacy, and stability of the final product. As such, monitoring HCP populations and amounts throughout the production and purification process is an essential part of the overall quality control framework. Mass spectrometry (MS) is used as an orthogonal method to enzyme-linked immunosorbent assays (ELISA) for the simultaneous identification and quantification of HCPs, particularly for the analysis of downstream processes. In this study, we present an MS-based analytical protocol with improvements in both speed and identification performance that can be implemented for routine analysis to support upstream process development. The protocol adopts a streamlined sample preparation strategy, combined with a high-throughput MS analysis pipeline. The developed method identifies and quantifies over 1000 HCPs, including 20 proteins listed as high risk in the literature, in a clarified cell culture sample with repeatability and precision shown for digest replicates. In addition, we explore the effects of varying standard spike-ins and changes to the data processing pipeline on absolute quantification estimates of the HCPs, which highlight the importance of standardization for wider use in the industry. Data are available via ProteomeXchange with the identifier PXD053035.

The Effect of Processing Techniques on the Classification Accuracy of Brain-Computer Interface Systems.

Background/Objectives: Accurately classifying Electroencephalography (EEG) signals is essential for the effective operation of Brain-Computer Interfaces (BCI), which is needed for reliable neurorehabilitation applications. However, many factors in the processing pipeline can influence classification performance. The objective of this study is to assess the effects of different processing steps on classification accuracy in EEG-based BCI systems. Methods: This study explores the impact of various processing techniques and stages, including the FASTER algorithm for artifact rejection (AR), frequency filtering, transfer learning, and cropped training. The Physionet dataset, consisting of four motor imagery classes, was used as input due to its relatively large number of subjects. The raw EEG was tested with EEGNet and Shallow ConvNet. To examine the impact of adding a spatial dimension to the input data, we also used the Multi-branch Conv3D Net and developed two new models, Conv2D Net and Conv3D Net. Results: Our analysis showed that classification accuracy can be affected by many factors at every stage. Applying the AR method, for instance, can either enhance or degrade classification performance, depending on the subject and the specific network architecture. Transfer learning was effective in improving the performance of all networks for both raw and artifact-rejected data. However, the improvement in classification accuracy for artifact-rejected data was less pronounced compared to unfiltered data, resulting in reduced precision. For instance, the best classifier achieved 46.1% accuracy on unfiltered data, which increased to 63.5% with transfer learning. In the filtered case, accuracy rose from 45.5% to only 55.9% when transfer learning was applied. An unexpected outcome regarding frequency filtering was observed: networks demonstrated better classification performance when focusing on lower-frequency components. Higher frequency ranges were more discriminative for EEGNet and Shallow ConvNet, but only when cropped training was applied. Conclusions: The findings of this study highlight the complex interaction between processing techniques and neural network performance, emphasizing the necessity for customized processing approaches tailored to specific subjects and network architectures.

Model of reed valve dynamics in unsteady flow, solution algorithm, and validation on a single-cycle rig

The results of studying an application model of unsteady flow through the reed valve assembly of a two-stroke piston engine are presented. The main goal was to develop and validate a model based on the check valve calculation scheme in the ALLBEA software package. The principles of modeling the valve motion dynamics in conjunction with a one-dimensional model of gas dynamics, as well as the algorithmic presentation of calculations for their implementation in the package, are briefly discussed. An approach to valve model identification and the proposed method for adjusting the model coefficients are considered. For the identification and validation of the model the data obtained from a single-cycle experimental setup were used, in which flow through the reed valve with finite amplitude waves in the pipeline was organized. The conducted research contributes to the analysis and optimization of processes in pipelines, pumps, and engines, promotes the development of applied models, software tools, and calculation methods in the specified areas.

Genome-Scale DNA Methylome and Transcriptome Profiles of Prostate Cancer Recurrence After Prostatectomy

Prostate cancer (PCa) is a major health burden worldwide, and despite early treatment, many patients present with biochemical recurrence (BCR) post-treatment, reflected by a rise in prostate-specific antigen (PSA) over a clinical threshold. Novel transcriptomic and epigenomic biomarkers can provide a powerful tools for the clinical management of PCa. Here, we provide matched RNA sequencing and array-based genome-wide DNA methylome data of PCa patients (n = 17) with or without evidence of BCR following radical prostatectomy. Formalin-fixed paraffin-embedded (FFPE) tissues were used to generate these data, which included technical replicates to provide further validity of the data. We describe the sample features, experimental design, methods and bioinformatic pipelines for processing these multi-omic data. Importantly, comprehensive clinical, histopathological, and follow-up data for each patient were provided to enable the correlation of transcriptome and methylome features with clinical features. Our data will contribute towards the efforts of developing epigenomic and transcriptomic markers for BCR and also facilitate a deeper understanding of the molecular basis of PCa recurrence.

Detecting flying objects in synthetic aperture radar images using Moving Target Indicator methods

AbstractThe growing proliferation of synthetic aperture radar (SAR) sensors brings the tantalising prospect of extending their utility into ‘novel’ applications. One potential extension is the detection of fast moving and accelerating flying objects in SAR imagery. However, since SAR image formation typically assumes the scene to be static over the coherent processing interval, moving objects give rise to blurred point spread functions, significant range migration and even potential aliasing of target signatures. The result is reduced target to clutter ratio (TCR) and poor detection performance. Successful detection of airborne targets thus requires compensation for potentially large target acceleration and velocity values observed over the comparatively long dwell times typical of practical SAR collection paradigms. This paper considers this problem and presents two main ideas to achieve this goal: a carefully constructed Moving Target Indicator (MTI) detection method implemented using real‐world Ingara SAR data, and a theoretical ground clutter suppression method. The MTI detection method combines several well‐known techniques for the flying target detection problem: interferometric processing, clutter suppression, and autofocus, and provides an extended acceleration phase compensation technique for highly accelerating targets such as planes. This proposed processing pipeline has been applied to experimental data of a plane during take off (a challenging Doppler unambiguous moving target), with the goal of continued detecting and tracking of this target. A generalised SAR signal model is presented that parameterises a flying moving target signature in terms of range and azimuthal target velocities and accelerations. Data driven approaches for estimating these motion parameters are examined and applied to experimental data acquired with the Ingara SAR sensor. The detection method was found to improve TCR by around 6 dB, along with superior detection and tracking performance. Following this, a theoretical study into suppressing ground clutter via multi‐channel cross‐track interferometry is investigated. Three separate ground clutter suppression methods, coherent subtraction, conventional beamforming, and minimum variance distortionless response (MVDR) beamformer, are presented then analysed using stochastic simulations. The MVDR adaptive beamformer method was found to provide the best performance for the scenario simulated.

Segmentation of Foreground Row Trees in Apple Orchard Images Collected by Ground Vehicles

Abstract. This paper proposes a fully automatic method for the segmentation of foreground row trees in industrial apple orchard images. The segmentation is based on analyzing a combination of a depth map constructed by the Marigold diffusion model and a model depth map created using automatically detected vanishing lines. The output of the method is a binary mask of the selected foreground trees. These masks can be used in subsequent stages of the image processing pipeline to discard false detections in the fruit counting module. The proposed method was evaluated as a preprocessing step for an apple detection method using the OrchardAppleDet- MSU dataset. Experiments showed that the proposed method can improve the quality of apple detection by 1–3%.

Versatile MRI acquisition and processing protocol for population-based neuroimaging.

Neuroimaging has an essential role in studies of brain health and of cerebrovascular and neurodegenerative diseases, requiring the availability of versatile magnetic resonance imaging (MRI) acquisition and processing protocols. We designed and developed a multipurpose high-resolution MRI protocol for large-scale and long-term population neuroimaging studies that includes structural, diffusion-weighted and functional MRI modalities. This modular protocol takes almost 1 h of scan time and is, apart from a concluding abdominal scan, entirely dedicated to the brain. The protocol links the acquisition of an extensive set of MRI contrasts directly to the corresponding fully automated data processing pipelines and to the required quality assurance of the MRI data and of the image-derived phenotypes. Since its successful implementation in the population-based Rhineland Study (ongoing, currently more than 11,000 participants, target participant number of 20,000), the proposed MRI protocol has proved suitable for epidemiological and clinical cross-sectional and longitudinal studies, including multisite studies. The approach requires expertise in magnetic resonance image acquisition, in computer science for the data management and the execution of processing pipelines, and in brain anatomy for the quality assessment of the MRI data. The protocol takes ~1 h of MRI acquisition and ~20 h of data processing to complete for a single dataset, but parallelization over multiple datasets using high-performance computing resources reduces the processing time. By making the protocol, MRI sequences and pipelines available, we aim to contribute to better comparability, interoperability and reusability of large-scale neuroimaging data.

Building a Cloud-Based Data Engineering Pipeline for AI-Powered Customer Analytics

This article comprehensively examines implementing a cloud-based data engineering pipeline for AI-powered customer analytics in the retail sector. It explores the architecture of advanced analytics solutions, their implementation strategies, and their impact on modern retail environments. The article details the development of a sophisticated system architecture encompassing data integration, processing pipelines, and machine learning capabilities, along with best practices for data quality, security, and scalability. It demonstrates how cloud-based analytics platforms can significantly improve customer engagement, reduce churn, enhance marketing effectiveness, and drive operational efficiency through real-world implementation examples and performance metrics. The article provides valuable insights into the transformative potential of AI-powered analytics in retail operations, highlighting key considerations for organizations undertaking similar digital transformation initiatives.

Reinforcement Learning for Intra- & Inter-Node Recommender Data Pipeline Optimization

Deep learning-based recommender models (DLRMs) have become an essential component of many modern recommender systems. Several companies are now building large compute clusters reserved for DLRM training, driving new interest in cost- & time- saving optimizations. The systems challenges faced in this setting are unique; while typical deep learning (DL) training jobs are dominated by model execution times, the most important factor in DLRM training performance is often online data ingestion. In this paper, we study real-world DLRM data processing pipelines taken from our compute cluster at Netflix to observe the performance impacts of online ingestion and identify shortfalls in existing data pipeline optimizers. Our studies lead us to design a new solution for data pipeline optimization, InTuneX . InTuneX is designed for production-scale multi-node recommender data pipelines. It unifies & tackles the challenges of both intra- and inter- node pipeline optimization. We achieve this with a multi-agent reinforcement learning (RL) design, simultaneously optimizing node assignments at the cluster level & CPU assignments within nodes. Our experiments show that InTuneX can build optimized data pipeline configurations within minutes. We apply InTuneX to our cluster, and find that it increases single-node data ingestion throughput by as much as 2.29X versus state-of-the-art optimizers, while improving the cost-efficiency of multi-node pipelines by 15-25%.

Toward generalizable phenotype prediction from single-cell morphology representations

BackgroundFunctional cell processes (e.g., molecular signaling, response to stimuli, mitosis, etc.) impact cell phenotypes, which scientists can measure with cell morphology. However, linking these measurements with phenotypes remains challenging because it requires manually annotated labels. We propose that nuclear morphology can be a predictive marker for cell phenotypes that are generalizable across contexts.MethodsWe reanalyzed a pre-labeled, publicly-available nucleus microscopy dataset from the MitoCheck consortium. We extracted single-cell morphology features using CellProfiler and DeepProfiler, which provide robust processing pipelines. We trained multinomial, multi-class elastic-net logistic regression models to classify nuclei into one of 15 phenotypes such as ‘Anaphase,’ ‘Apoptosis’, and ‘Binuclear’. We rigorously assessed performance using F1 scores, precision-recall curves, and a leave-one-image-out (LOIO) cross-validation analysis. In LOIO, we retrained models using cells from every image except one and predicted phenotype in the held-out image, repeating this procedure for all images. We evaluated each morphology feature space, a concatenated feature space, and several feature space subsets (e.g., nuclei AreaShape features only). We applied models to the Joint Undertaking in Morphological Profiling (JUMP) data to assess performance using a different dataset.ResultsIn a held-out test set, we observed an overall F1 score of 0.84. Individual phenotype scores ranged from 0.64 (moderate performance) to 0.99 (high performance). Phenotypes such as ‘Elongated’, ‘Metaphase’, and ‘Apoptosis’ showed high performance. While CellProfiler and DeepProfiler features were generally equally effective, concatenation yielded the best results for 9/15 phenotypes. LOIO showed a performance decline, indicating our model could not reliably predict phenotypes in new images. Poor performance was unrelated to illumination correction or model selection. Applied to the JUMP data, models trained using nuclear AreaShape features only increased alignment with the annotated MitoCheck data (based on UMAP space). This approach implicated many chemical and genetic perturbations known to be associated with specific phenotypes.DiscussionPoor LOIO performance demonstrates challenges of single-cell phenotype prediction in new datasets. We propose several strategies that could pave the way for more generalizable methods in single-cell phenotype prediction, which is a step toward morphology representation ontologies that would aid in cross-dataset interpretability.