Scalable Data Analysis Research Articles

Scalable and Resolution Data Analysis of Image and Video Compression using DL-CNNS Neural Network

Scalable and Resolution Data Analysis of Image and Video Compression using DL-CNNS Neural Network

Constructing phylogenetic trees for microbiome data analysis: A mini-review

As next-generation sequencing technologies advance rapidly and the cost of metagenomic sequencing continues to decrease, researchers now face an unprecedented volume of microbiome data. This surge has stimulated the development of scalable microbiome data analysis methods and necessitated the incorporation of phylogenetic information into microbiome analysis for improved accuracy. Tools for constructing phylogenetic trees from 16S rRNA sequencing data are well-established, as the highly conserved regions of the 16S gene are limited, simplifying the identification of marker genes. In contrast, metagenomic and whole genome shotgun (WGS) sequencing involve sequencing from random fragments of the entire gene, making identification of consistent marker genes challenging owing to the vast diversity of genomic regions, resulting in a scarcity of robust tools for constructing phylogenetic trees. Although bacterial sequence tree construction tools exist for upstream bioinformatics, many downstream researchers—those integrating these trees into statistical models or machine learning—are either unaware of these tools or find them difficult to use due to the steep learning curve of processing raw sequences. This is compounded by the fact that public datasets often lack phylogenetic trees, providing only abundance tables and taxonomic classifications. To address this, we present a comprehensive review of phylogenetic tree construction techniques for microbiome data (16S rRNA or whole-genome shotgun sequencing). We outline the strengths and limitations of current methods, offering expert insights and step-by-step guidance to make these tools more accessible and widely applicable in quantitative microbiome data analysis.

ScParser: sparse representation learning for scalable single-cell RNA sequencing data analysis

The rapid rise in the availability and scale of scRNA-seq data needs scalable methods for integrative analysis. Though many methods for data integration have been developed, few focus on understanding the heterogeneous effects of biological conditions across different cell populations in integrative analysis. Our proposed scalable approach, scParser, models the heterogeneous effects from biological conditions, which unveils the key mechanisms by which gene expression contributes to phenotypes. Notably, the extended scParser pinpoints biological processes in cell subpopulations that contribute to disease pathogenesis. scParser achieves favorable performance in cell clustering compared to state-of-the-art methods and has a broad and diverse applicability.

A robust CETSA data analysis automation workflow for routine screening

The Cellular Thermal Shift Assay (CETSA) enables the study of protein-ligand interactions in a cellular context. It provides valuable information on the binding affinity and specificity of both small and large molecule ligands in a relevant physiological context, hence forming a unique tool in drug discovery. Though high-throughput lab protocols exist for scaling up CETSA, subsequent data analysis and quality control remain laborious and limit experimental throughput. Here, we introduce a scalable and robust data analysis workflow which allows integration of CETSA into routine high throughput screening (HT-CETSA). This new workflow automates data analysis and incorporates quality control (QC), including outlier detection, sample and plate QC, and result triage. We describe the workflow and show its robustness against typical experimental artifacts, show scaling effects, and discuss the impact of data analysis automation by eliminating manual data processing steps.

Custom Biomedical FAIR Data Analysis in the Cloud Using CAVATICA.

The historically fragmented biomedical data ecosystem has moved towards harmonization under the findable, accessible, interoperable, and reusable (FAIR) data principles, creating more opportunities for cloud-based research. This shift is especially opportune for scientists across diverse domains interested in implementing creative, nonstandard computational analytic pipelines on large and varied datasets. However, executing custom cloud analyses may present difficulties, particularly for investigators lacking advanced computational expertise. Here, we present an accessible, streamlined approach for the cloud compute platform CAVATICA that offers a solution. We outline how we developed a custom workflow in the cloud, for analyzing whole genome sequences of case-parent trios to detect sex-specific genetic effects on orofacial cleft risk, which required several programming languages and custom software packages. The approach involves just three components: Docker to containerize software environments, tool creation for each analysis step, and a visual workflow editor to weave the tools into a Common Workflow Language (CWL) pipeline. Our approach should be accessible to any investigator with basic computational skills, is readily extended to implement any scalable high-throughput biomedical data analysis in the cloud, and is applicable to other commonly used compute platforms such as BioData Catalyst. We believe our approach empowers versatile data reuse and promotes accelerated biomedical discovery in a time of substantial FAIR data.

RADAR-Pipeline: Scalable Feature Generation for Mobile Health Data

Introduction & BackgroundRADAR-Pipeline is an open-source Python framework designed to simplify and enhance mobile health data analysis. It has been designed to efficiently read and process the large amount of data generated through the RADAR-Base platform. RADAR-base is a scalable, real-time streaming and analytics open-source platform to facilitate research access and customisation requirements. Studies using the Radar-base platform have collected fine-grained longitudinal data from wearables and phones. The data can potentially create multitudes of digital biomarkers, which can be used to inform us greatly about the disease condition. Due to the sheer size of the data, it can be difficult for researchers to read and process those data -- a common task is identifying useful features and common data processing/analysis steps previously used by the community. Up to now, these have been hand-crafted by individual data scientists, often lacking the capability to be easily reused by the community without author-specific knowledge. Furthermore, generating variables based on already established research on a larger scale can be challenging and could hinder replication. Hence, we have designed RADAR-Pipeline to help researchers overcome these challenges. It empowers them to create and share their data analysis and visualisation pipelines, fostering collaboration and knowledge sharing within the research community. Objectives & ApproachThe primary objective of RADAR-Pipeline is to offer researchers a user-friendly and powerful platform to develop and share their research. Researchers can build reusable analysis and visualisation pipelines to ensure consistent and reliable results. It simplifies big data analysis by leveraging Apache Spark to handle large and complex mobile health datasets efficiently. Researchers can also save time and effort by reusing and extending existing pipelines built by others. Finally, the RADAR-Pipeline promotes collaboration and recognition by allowing researchers to share their work through the RADAR-base Analytics Catalogue, making their pipelines citable and accessible to the wider research community. Whilst Radar-pipeline has been designed to read data from Radar-base, it can also be used to read data from any dataset which uses Hadoop Distributed File System (HDFS) file system namespace. Relevance to Digital FootprintsMobile health data is rich and valuable for understanding human behaviour and health. RADAR-Pipeline addresses the challenges associated with analysing large and complex mobile health datasets, enabling researchers to extract valuable insights that can be used to (1) Improve public health: By enabling efficient analysis of large-scale mobile health data, RADAR-Pipeline can contribute to research efforts aimed at improving population health outcomes and developing effective interventions; (2) Personalised healthcare: By facilitating the extraction of individual-level features from mobile health data, RADAR-Pipeline can seamlessly be integrated with Kafka data streams and machine learning pipelines to process the data in real-time, which can then be utilised to create more effective and targeted real-time interventions. (3) Promote reproducible research: The framework's emphasis on transparency and reproducibility in research aligns with the conference's focus on the responsible use of digital mobile health data. Conclusions & ImplicationsRADAR-Pipeline is a valuable tool for researchers, offering them the means to harness the potential of mobile health data. By adopting this framework, researchers can achieve efficient and scalable data analysis, thereby streamlining the extracting insights from digital footprints. This efficiency enables researchers to delve deeper into the data and uncover valuable patterns and trends. Furthermore, RADAR-Pipeline promotes collaboration and knowledge sharing within the research community. By providing a standardised framework for data analysis, RADAR-Pipeline facilitates collaboration among researchers, leading to the sharing of best practices and the dissemination of knowledge.

BANKSY unifies cell typing and tissue domain segmentation for scalable spatial omics data analysis.

Spatial omics data are clustered to define both cell types and tissue domains. We present Building Aggregates with a Neighborhood Kernel and Spatial Yardstick (BANKSY), an algorithm that unifies these two spatial clustering problems by embedding cells in a product space of their own and the local neighborhood transcriptome, representing cell state and microenvironment, respectively. BANKSY's spatial feature augmentation strategy improved performance on both tasks when tested on diverse RNA (imaging, sequencing) and protein (imaging) datasets. BANKSY revealed unexpected niche-dependent cell states in the mouse brain and outperformed competing methods on domain segmentation and cell typing benchmarks. BANKSY can also be used for quality control of spatial transcriptomics data and for spatially aware batch effect correction. Importantly, it is substantially faster and more scalable than existing methods, enabling the processing of millions of cell datasets. In summary, BANKSY provides an accurate, biologically motivated, scalable and versatile framework for analyzing spatially resolved omics data.

A crowdsensing-based framework for sound and vibration data analysis in smart urban environments

In this paper, we introduce a framework tailored to the precise analysis of sound and vibration data in the dynamic context of smart cities. By harnessing the power of crowdsensing data, our system offers a robust and highly adaptable solution for the real-time monitoring and comprehensive assessment of acoustic and vibrational parameters within urban environments. The rise of smart cities, fueled by sensor tech and data analysis, demands advanced tools to tackle urban environmental issues and improve residents’ quality of life. Our framework empowers city stakeholders, offering insights for informed decisions in urban planning, transportation, infrastructure maintenance, and public health. Key features of our system include data acquisition through mobile crowdsensing application, advanced signal processing algorithms for noise and vibration identification, and the integration of geospatial information to provide location-specific context. Furthermore, our framework supports scalable and adaptable data analytics, ensuring the efficient utilization of resources and the effective management of urban environments. The presented framework addresses the challenges of noise pollution, structural health monitoring, transportation optimization, and public safety in smart cities. By exploiting crowdsourced data, it promotes a collaborative approach to data collection, analysis, and decision-making, fostering an environment where cities can continuously evolve and adapt to the evolving needs of their residents. This paper offers a detailed exploration of the system’s architecture, and showcases its practical implementation, affirming its potential as a powerful starting point for advancing the science of smart cities.

Adaptive Model Selection in Stock Market Prediction: A Modular and Scalable Big Data Analytics Approach

Adaptive Model Selection in Stock Market Prediction: A Modular and Scalable Big Data Analytics Approach

Zooming in on Urban Life in Early Modern Amsterdam. Time Machine as a Tool for Researching Everyday Urban Experience Across Space and Time

How do we make sense of urban life in the past? What do we do when we study urban history, and to what extent do our methods fully capture the complexities of historical city living? These are crucial questions for any scholar interested in the historical dimensions of urban experience. Notwithstanding the interest of most urban historians in the relationship between the physical form of urban space and its experience by inhabitants and visitors, very few scholars have written histories that systematically integrate these two areas of inquiry. In this article the authors argue that such research requires a method and an accompanying digital tool that can analyze historical urban life in a more integrated, holistic way. The authors propose a way forward by introducing the Time Machine platform as a scalable data visualization and analysis tool for researching everyday urban experience across space and time. To illustrate the potential, the authors focus on a case study: the area of the Bloemstraat in early modern Amsterdam. Unpacking a section of the Bloemstraat, house by house and room by room, this case shows how the Time Machine forms an instrument to connect spatial layouts to the arrangement of objects and to the practical and social use of the space by the inhabitants and visitors. The authors also sketch how this tool illuminates more dynamic spatial and temporal practices such as how people, goods and activities are connected to locations in the wider city and beyond.

Motorized three-wheelers and their potential for just mobility in Caribbean urban areas

Abstract This paper investigates the role of motorized three-wheelers (MTW) in urban mobility within popular transport, a demand-responsive and unscheduled mode of transportation provided by self-organized small operators frequently operating in grey areas of regulation. Although popular transport is the primary mobility option for millions worldwide, knowledge about its users, operation, and environmental and social impacts remains scarce. This paper sheds light on some of the features and impacts of popular MTW, focusing on two case studies in the Caribbean with different scales and urban trajectories: Puerto Viejo, Costa Rica, and Soledad in Colombia. We explored the relationship between MTW and fragmentation–(in)accessibility–exclusion in these cities, drawing on a framework connecting these concepts in the Latin American and Caribbean context. Using primary data from qualitative and quantitative methods, the paper examines the distribution of inhibitors or enablers of accessibility within the context of unequal, splintered, and fragmented transport and communication infrastructures. Additionally, the environmental impact of MTW in terms of CO2 and PM2.5 emissions is assessed using field data from low-cost sensors. The paper argues that planning for just urban mobility necessitates considering the ecological consequences of various transportation modes and their social consequences and potential for participation and inclusion. The applied methodology introduces low-cost, replicable, and scalable data production and analysis techniques, contributing to future research on sustainable and just mobility in resource-limited urban areas.

A Microservices-based Framework for Scalable Data Analysis in Agriculture with IoT Integration

We propose a microservices-based framework for scalable data analysis in agriculture with IoT integration, leveraging the flexibility and modularity of microservices architecture to build a highly adaptable, maintainable, and efficient data analysis system. This framework allows for faster data processing and carry a diversity of agricultural data analysis tasks while maintaining scalability and fault tolerance. Despite the potential benefits, several challenges and obstacles need to be addressed, such as data integration and standardization, the development of agricultural-specific analytical microservices, and ensuring data security and privacy. Practical application and real-world validation are required to assess the impact of the proposed framework on the agricultural sector and inform future research directions.

Scalable Infrastructure Supporting Reproducible Nationwide Healthcare Data Analysis toward FAIR Stewardship

Transparent and FAIR disclosure of meta-information about healthcare data and infrastructure is essential but has not been well publicized. In this paper, we provide a transparent disclosure of the process of standardizing a common data model and developing a national data infrastructure using national claims data. We established an Observational Medical Outcome Partnership (OMOP) common data model database for national claims data of the Health Insurance Review and Assessment Service of South Korea. To introduce a data openness policy, we built a distributed data analysis environment and released metadata based on the FAIR principle. A total of 10,098,730,241 claims and 56,579,726 patients’ data were converted as OMOP common data model. We also built an analytics environment for distributed research and made the metadata publicly available. Disclosure of this infrastructure to researchers will help to eliminate information inequality and contribute to the generation of high-quality medical evidence.

MetaPro: a scalable and reproducible data processing and analysis pipeline for metatranscriptomic investigation of microbial communities

BackgroundWhole microbiome RNASeq (metatranscriptomics) has emerged as a powerful technology to functionally interrogate microbial communities. A key challenge is how best to process, analyze, and interpret these complex datasets. In a typical application, a single metatranscriptomic dataset may comprise from tens to hundreds of millions of sequence reads. These reads must first be processed and filtered for low quality and potential contaminants, before being annotated with taxonomic and functional labels and subsequently collated to generate global bacterial gene expression profiles.ResultsHere, we present MetaPro, a flexible, massively scalable metatranscriptomic data analysis pipeline that is cross-platform compatible through its implementation within a Docker framework. MetaPro starts with raw sequence read input (single-end or paired-end reads) and processes them through a tiered series of filtering, assembly, and annotation steps. In addition to yielding a final list of bacterial genes and their relative expression, MetaPro delivers a taxonomic breakdown based on the consensus of complementary prediction algorithms, together with a focused breakdown of enzymes, readily visualized through the Cytoscape network visualization tool. We benchmark the performance of MetaPro against two current state-of-the-art pipelines and demonstrate improved performance and functionality.ConclusionsMetaPro represents an effective integrated solution for the processing and analysis of metatranscriptomic datasets. Its modular architecture allows new algorithms to be deployed as they are developed, ensuring its longevity. To aid user uptake of the pipeline, MetaPro, together with an established tutorial that has been developed for educational purposes, is made freely available at https://github.com/ParkinsonLab/MetaPro. The software is freely available under the GNU general public license v3.9LNNdXLsCA96o8Y-9qmMqKVideo

Sequential Data Processing for IMERG Satellite Rainfall Comparison and Improvement Using LSTM and ADAM Optimizer

This study introduces a systematic methodology whereby different technologies were utilized to download, pre-process, and interactively compare the rainfall datasets from the Integrated Multi-Satellite Retrievals for Global Precipitation Mission (IMERG) satellite and rain gauges. To efficiently handle the large volume of data, we developed automated shell scripts for downloading IMERG data and storing it, along with rain gauge data, in a relational database system. Hypertext pre-processor (pHp) programs were built to visualize the result for better analysis. In this study, the performance of IMERG estimations over the east coast of Peninsular Malaysia for the duration of 10 years (2011–2020) against rain gauge observation data is evaluated. Moreover, this study aimed to improve the daily IMERG estimations with long short-term memory (LSTM) developed with Python. Findings show that the LSTM with Adaptive Moment Estimation (ADAM) optimizer trained against the mean square error (MSE) loss enhances the accuracy of satellite estimations. At the point-to-pixel scale, the correlation between satellite estimations and ground observations was increased by about 15%. The bias was reduced by 81–118%, MAE was reduced by 18–59%, the root-mean-square error (RMSE) was reduced by 1–66%, and the Kling–Gupta efficiency (KGE) was increased by approximately 200%. The approach developed in this study establishes a comprehensive and scalable data processing and analysis pipeline that can be applied to diverse datasets and regions encountering similar domain-specific challenges.

Scalable Data Ingestion and Analytics: Leveraging Azure Data Explorer for IoT Performance Optimization

Scalable Data Ingestion and Analytics: Leveraging Azure Data Explorer for IoT Performance Optimization

Design and Development of a Big Data Platform for Disease Burden Based on the Spark Engine.

This study attempts to build a big data platform for disease burden that can realize the deep coupling of artificial intelligence and public health. This is a highly open and shared intelligent platform, including big data collection, analysis, and result visualization. Based on data mining theory and technology, the current situation of multisource data on disease burden was analyzed. Putting forward the disease burden big data management model, functional modules, and technical framework, Kafka technology is used to optimize the transmission efficiency of the underlying data. This will be an efficient and highly scalable data analysis platform through embedding embedded Sparkmlib in the Hadoop ecosystem. With the concept of "Internet + medical integration," the overall architecture design of the big data platform for disease burden management was proposed based on the Spark engine and Python language. The main system composition and application scenarios are given at four levels: multisource data collection, data processing, data analysis, and the application layer, according to application scenarios and use requirements. The big data platform of disease burden management helps to promote the multisource convergence of disease burden data and provides a new path for the standardized paradigm of disease burden measurement. Provide methods and ideas for the deep integration of medical big data and the formation of a broader standard paradigm.

Information enhancement or hindrance? Unveiling the impacts of user-generated photos in online reviews

PurposeThis study aims to investigate the promoting effects of the quantity and quality of online review user-generated photos (UGPs) on perceived review usefulness. The research further tests the hindering effect of human facial presence in review photos on review usefulness.Design/methodology/approachBased on review samples of restaurants in a tourist destination Las Vegas, this study used an integrated method combining a machine learning algorithm and econometric modeling.FindingsResults indicate that the number of UGPs depicting a restaurant’s food, drink, menu and physical environment has positive impacts on perceived review usefulness. The quality of online review UGPs can also enhance perceived review usefulness, whereas facial presence in these UGPs hinders perceived review usefulness.Practical implicationsFindings suggest that practitioners can implement certain tactics to potentially improve consumers’ willingness to share more UGPs and UGPs with higher quality. Review websites could develop image-processing algorithms for identifying and presenting UGPs containing core attributes in prominent positions on the site.Originality/valueTo the best of the authors’ knowledge, this study is the first to present a comprehensive analytical framework investigating the enhancing or hindering roles of review photo quantity, photo quality and facial presence in online review UGPs on review usefulness. Using the heuristic-systematic model as a theoretical foundation, this study verifies the additivity effect and attenuation effect of UGPs’ visual elements on judgements of online review usefulness. Furthermore, it extends scalable image data analysis by adopting a deep transfer learning algorithm in hospitality and tourism.

Scalable transcriptomics analysis with Dask: applications in data science and machine learning

BackgroundGene expression studies are an important tool in biological and biomedical research. The signal carried in expression profiles helps derive signatures for the prediction, diagnosis and prognosis of different diseases. Data science and specifically machine learning have many applications in gene expression analysis. However, as the dimensionality of genomics datasets grows, scalable solutions become necessary.MethodsIn this paper we review the main steps and bottlenecks in machine learning pipelines, as well as the main concepts behind scalable data science including those of concurrent and parallel programming. We discuss the benefits of the Dask framework and how it can be integrated with the Python scientific environment to perform data analysis in computational biology and bioinformatics.ResultsThis review illustrates the role of Dask for boosting data science applications in different case studies. Detailed documentation and code on these procedures is made available at https://github.com/martaccmoreno/gexp-ml-dask.ConclusionBy showing when and how Dask can be used in transcriptomics analysis, this review will serve as an entry point to help genomic data scientists develop more scalable data analysis procedures.

PyTCR: A comprehensive and scalable solution for TCR-Seq data analysis to facilitate reproducibility and rigor of immunogenomics research.

T cell receptor (TCR) studies have grown substantially with the advancement in the sequencing techniques of T cell receptor repertoire sequencing (TCR-Seq). The analysis of the TCR-Seq data requires computational skills to run the computational analysis of TCR repertoire tools. However biomedical researchers with limited computational backgrounds face numerous obstacles to properly and efficiently utilizing bioinformatics tools for analyzing TCR-Seq data. Here we report pyTCR, a computational notebook-based solution for comprehensive and scalable TCR-Seq data analysis. Computational notebooks, which combine code, calculations, and visualization, are able to provide users with a high level of flexibility and transparency for the analysis. Additionally, computational notebooks are demonstrated to be user-friendly and suitable for researchers with limited computational skills. Our tool has a rich set of functionalities including various TCR metrics, statistical analysis, and customizable visualizations. The application of pyTCR on large and diverse TCR-Seq datasets will enable the effective analysis of large-scale TCR-Seq data with flexibility, and eventually facilitate new discoveries.