Python Ecosystem Research Articles

A Multi-Method Data Science Pipeline for Analyzing Police Service

Despite the fact that most police departments in the U.S. serve jurisdictions with fewer than 10,000 residents, policing practices in small towns are understudied. This is due in part to data limitations and technological barriers that exist in the small-town context. In this article we focus on one small town police department in New England with a history of misconduct, and develop a comprehensive data science pipeline that addresses the stages from design and collection to reporting. We present the reader with specific tools in the open-source Python ecosystem for replicating this pipeline. Once these data are processed, we perform two statistical analyses in an attempt to better understand the provisions of service by the small-town police department of focus. First, we perform ecological inference to estimate the rate at which residents are placing calls for service. Second, we model wait times using a negative binomal regression model to account for overdispersion in the data. We discuss data and model limitations arising through the pipeline creation and analysis process.

Advancing scanning probe microscopy simulations: A decade of development in probe-particle models

The Probe-Particle Model combine theories designed for the simulation of scanning probe microscopy experiments, employing non-reactive, flexible tip apices to achieve sub-molecular resolution. In the article we present the latest version of the Probe-Particle Model implemented in the open-source ppafm package, highlighting substantial advancements in accuracy, computational performance, and user-friendliness. To demonstrate this we provide a comprehensive review of approaches for simulating non-contact Atomic Force Microscopy. They vary in complexity from simple Lennard-Jones potential to the latest full density-based model. We compared those approaches with ab initio calculated references, showcasing their respective merits. All parts of the ppafm package have undergone acceleration by 1-2 orders of magnitude using OpenMP and OpenCL technologies. The updated package includes an interactive graphical user interface and seamless integration into the Python ecosystem via pip, facilitating advanced scripting and interoperability with other software. This adaptability positions ppafm as an ideal tool for high-throughput applications, including the training of machine learning models for the automatic recovery of atomic structures from nc-AFM measurements. We envision significant potential for this application in future single-molecule analysis, synthesis, and advancements in surface science in general. Additionally, we discuss simulations of other sub-molecular scanning-probe imaging techniques, such as bond-resolved scanning tunneling microscopy and kelvin probe force microscopy, all built on the robust foundation of the Probe-Particle Model. Altogether this demonstrates the broad impact of the model across diverse domains of on-surface science and molecular chemistry.

Poincaré and SimBio: a versatile and extensible Python ecosystem for modeling systems.

Chemical reaction networks (CRNs) play a pivotal role in diverse fields such as systems biology, biochemistry, chemical engineering, and epidemiology. High-level definitions of CRNs enables to use various simulation approaches, including deterministic and stochastic methods, from the same model. However, existing Python tools for simulation of CRN typically wrap external C/C++ libraries for model definition, translation into equations and/or numerically solving them, limiting their extensibility and integration with the broader Python ecosystem. In response, we developed Poincaré and SimBio, two novel Python packages for simulation of dynamical systems and CRNs. Poincaré serves as a foundation for dynamical systems modeling, while SimBio extends this functionality to CRNs, including support for the Systems Biology Markup Language (SBML). Poincaré and SimBio are developed as pure Python packages enabling users to easily extend their simulation capabilities by writing new or leveraging other Python packages. Moreover, this does not compromise the performance, as code can be just-in-time compiled with Numba. Our benchmark tests using curated models from the BioModels repository demonstrate that these tools may provide a potentially superior performance advantage compared to other existing tools. In addition, to ensure a user-friendly experience, our packages use standard typed modern Python syntax that provides a seamless integration with integrated development environments. Our Python-centric approach significantly enhances code analysis, error detection, and refactoring capabilities, positioning Poincaré and SimBio as valuable tools for the modeling community. Poincaré and SimBio are released under the MIT license. Their source code is available on GitHub (https://github.com/maurosilber/poincare and https://github.com/hgrecco/simbio) and can be installed from PyPI or conda-forge.

Utilizing Python for Scalable Data Processing in Cloud Environments

In the age of big data and cloud computing, enterprises need to effectively analyze enormous datasets to get meaningful insights and stay ahead. Python, a popular programming language, is a strong tool for cloud-scale data processing. This research study examines Python's integration with cloud platforms and its effects on performance and efficiency in scalable data processing. The study introduces scalable data processing and cloud computing. It then discusses Python's ecosystem, including Dask, Apache Spark with PySpark, TensorFlow, and PyTorch for data processing and machine learning. The study also examines Python's interoperability with cloud services like AWS, Google Cloud Platform, and Microsoft Azure in data input, transformation, and analysis. Many case studies and real-world applications demonstrate how Python has been used in banking, healthcare, and e-commerce. Python is useful for managing massive amounts of data, streamlining processing processes, and scaling cloud applications, as shown in the case studies. The report also analyzes Python-based cloud systems' performance indicators and cost consequences, revealing best practices and possible issues. The article explores Python's involvement in cloud computing trends and technology. Serverless architectures, Docker and Kubernetes, and Python interaction with cloud-native tools and services are examples. These patterns show how data processing is changing and how Python is improving to meet current data needs. This study concludes that Python is a reliable and scalable cloud data processing option. The language's strengths, alignment with cloud technologies, and practical applications in many areas are covered in detail. The results indicate that Python's versatility and cloud scalability provide a robust foundation for handling and analyzing massive datasets, enabling better decision-making and innovation across fields.

Intelligent control of combined sewer systems using PySWMM—A Python wrapper for EPA’s Stormwater Management Model

Wastewater utilities face competing priorities as they work to protect human health and water quality, and to maintain infrastructure in their communities. Budgetary constraints can be especially pronounced among small to medium-sized utilities. Utilities are increasingly turning to so-called intelligent water approaches as a cost-effective alternative to upgrading aging infrastructure. Intelligent water encompasses automated control and real-time decision support technologies and can be applied at scale to large and small utilities alike accommodating differences in needs, capabilities, and funds. Intelligent water upgrades can be designed to optimize existing conveyance, storage, and treatment during storms to help mitigate flooding and combined sewer overflows. The most promising real-time control algorithms coordinate control of upstream and downstream assets and are designed using urban hydrologic and hydraulic modeling software. The capabilities of legacy software, however, can sometimes inhibit the creation of sophisticated control algorithms. In this paper, we present PySWMM — an open-source Python wrapper developed for the EPA Storm Water Management Model (SWMM). PySWMM enables runtime interactions with the SWMM computational engine to flexibly read, modify system parameters, and control digital infrastructure during a simulation. Crucially, it allows modelers to easily combine SWMM with the rich set of scientific computing, big data, and machine learning modules found in the Python ecosystem. We highlight two real-world intelligent water case studies utilizing PySWMM in the cities of Cincinnati and Columbus, Ohio where it has helped to eliminate tens of millions of gallons of combined sewer overflows annually.

Common data models to streamline metabolomics processing and annotation, and implementation in a Python pipeline.

To standardize metabolomics data analysis and facilitate future computational developments, it is essential to have a set of well-defined templates for common data structures. Here we describe a collection of data structures involved in metabolomics data processing and illustrate how they are utilized in a full-featured Python-centric pipeline. We demonstrate the performance of the pipeline, and the details in annotation and quality control using large-scale LC-MS metabolomics and lipidomics data and LC-MS/MS data. Multiple previously published datasets are also reanalyzed to showcase its utility in biological data analysis. This pipeline allows users to streamline data processing, quality control, annotation, and standardization in an efficient and transparent manner. This work fills a major gap in the Python ecosystem for computational metabolomics.

Heavy Rainfall Events in Selected Geographic Regions of Mexico, Associated with Hail Cannons

In this article, we document the use of hail cannons in Mexico to dispel or suppress heavy rain episodes, a common practice among farmers, without scientific evidence to support its effectiveness. This study uses two rain databases: one compiled from the Global Precipitation Measurement (GPM) mission and the other generated with the implementation of the Weather Research and Forecasting (WRF) model. The aim is to explore the association between heavy rain episodes and hail cannon locations. The analysis includes two geographic features: a pair of coordinates and a 3 km radius area of influence around each hail cannon. This dimension is based on the size and distribution of the heavy rainfall events. This study analyzes four years of half-hourly rain data using the Python ecosystem environment with machine learning libraries. The results show no relationship between the operation of hail cannons and the dissipation or attenuation of heavy rainfall events. However, this study highlights that the significant differences between the GPM and WRF databases in registering heavy rain events may be attributable to their own uncertainty. Despite the unavailability of ground-based observations, the inefficiency of hail cannons in affecting the occurrence of heavy rain events is evident. Overall, this study provides scientific evidence that hail cannons are inefficient in preventing the occurrence of heavy rain episodes.

MUFFIN: a suite of tools for the analysis of functional sequencing data.

The large diversity of functional genomic assays allows for the characterization of non-coding and coding events at the tissue level or at a single-cell resolution. However, this diversity also leads to protocol differences, widely varying sequencing depths, substantial disparities in sample sizes, and number of features. In this work, we have built a Python package, MUFFIN, which offers a wide variety of tools suitable for a broad range of genomic assays and brings many tools that were missing from the Python ecosystem. First, MUFFIN has specialized tools for the exploration of the non-coding regions of genomes, such as a function to identify consensus peaks in peak-called assays, as well as linking genomic regions to genes and performing Gene Set Enrichment Analyses. MUFFIN also possesses a robust and flexible count table processing pipeline, comprising normalization, count transformation, dimensionality reduction, Differential Expression, and clustering. Our tools were tested on three widely different scRNA-seq, ChIP-seq and ATAC-seq datasets. MUFFIN integrates with the popular Scanpy ecosystem and is available on Conda and at https://github.com/pdelangen/Muffin.

PyPWA: A software toolkit for parameter optimization and amplitude analysis

PyPWA is a toolkit designed to optimize parametric models describing data and generate simulated distributions according to a model. Its software has been written within the python ecosystem with the goal of performing Amplitude or Partial Wave Analysis (PWA) in nuclear and particle physics experiments. We briefly describe the general features of amplitude analysis and we provide a description of the PyPWA software design and usage. We also provide benchmarks of the scaling and an example of its application.

Thebeat: A Python package for working with rhythms and other temporal sequences

thebeat is a Python package for working with temporal sequences and rhythms in the behavioral and cognitive sciences, as well as in bioacoustics. It provides functionality for creating experimental stimuli, and for visualizing and analyzing temporal data. Sequences, sounds, and experimental trials can be generated using single lines of code. thebeat contains functions for calculating common rhythmic measures, such as interval ratios, and for producing plots, such as circular histograms. thebeat saves researchers time when creating experiments, and provides the first steps in collecting widely accepted methods for use in timing research. thebeat is an open-source, on-going, and collaborative project, and can be extended for use in specialized subfields. thebeat integrates easily with the existing Python ecosystem, allowing one to combine our tested code with custom-made scripts. The package was specifically designed to be useful for both skilled and novice programmers. thebeat provides a foundation for working with temporal sequences onto which additional functionality can be built. This combination of specificity and plasticity should facilitate research in multiple research contexts and fields of study.

Physics analysis for the HL-LHC: Concepts and pipelines in practice with the Analysis Grand Challenge

Realistic environments for prototyping, studying and improving analysis workflows are a crucial element on the way towards user-friendly physics analysis at HL-LHC scale. The IRIS-HEP Analysis Grand Challenge (AGC) provides such an environment. It defines a scalable and modular analysis task that captures relevant workflow aspects, ranging from large-scale data processing and handling of systematic uncertainties to statistical inference and analysis preservation. By being based on publicly available Open Data, the AGC provides a point of contact for the broader community. Multiple different implementations of the analysis task that make use of various pipelines and software stacks already exist. This contribution presents an updated AGC analysis task. It features a machine learning component and expanded analysis complexity, including the handling of an extended and more realistic set of systematic uncertainties. These changes both align the AGC further with analysis needs at the HL-LHC and allow for probing an increased set of functionality. Another focus is the showcase of a reference AGC implementation, which is heavily based on the HEP Python ecosystem and uses modern analysis facilities. The integration of various data delivery strategies is described, resulting in multiple analysis pipelines that are compared to each other.

Pyqcm: An open-source Python library for quantum cluster methods

Pyqcm is a Python/C++ library that implements a few quantum cluster methods with an exact diagonalization impurity solver. Quantum cluster methods are used in the study of strongly correlated electrons to provide an approximate solution to Hubbard-like models. The methods covered by this library are Cluster Perturbation Theory (CPT), the Variational Cluster Approach (VCA) and Cellular (or Cluster) Dynamical Mean Field Theory (CDMFT). The impurity solver (the technique used to compute the cluster’s interacting Green function) is exact diagonalization from sparse matrices, using the Lanczos algorithm and variants thereof. The core library is written in C++ for performance, but the interface is in Python, for ease of use and inter-operability with the numerical Python ecosystem. The library is distributed under the GPL license.

Codebase release 2.2 for Pyqcm

Pyqcm is a Python/C++ library that implements a few quantum cluster methods with an exact diagonalization impurity solver. Quantum cluster methods are used in the study of strongly correlated electrons to provide an approximate solution to Hubbard-like models. The methods covered by this library are Cluster Perturbation Theory (CPT), the Variational Cluster Approach (VCA) and Cellular (or Cluster) Dynamical Mean Field Theory (CDMFT). The impurity solver (the technique used to compute the cluster’s interacting Green function) is exact diagonalization from sparse matrices, using the Lanczos algorithm and variants thereof. The core library is written in C++ for performance, but the interface is in Python, for ease of use and inter-operability with the numerical Python ecosystem. The library is distributed under the GPL license.

Exploring Applications and Practical Examples by Streamlining Material Requirements Planning (MRP) with Python

Background: Material Requirements Planning (MRP) is critical in Supply Chain Management (SCM), facilitating effective inventory management and meeting production demands in the manufacturing sector. Despite the potential benefits of automating the MRP tasks to meet the demand for expedited and efficient management, the field appears to be lagging behind in harnessing the advancements offered by Artificial Intelligence (AI) and sophisticated programming languages. Consequently, this study aims to address this gap by exploring the applications of Python in simplifying the MRP processes. Methods: This article offers a twofold approach: firstly, it conducts research to uncover the potential applications of the Python code in streamlining the MRP operations, and the practical examples serve as evidence of Python’s efficacy in simplifying the MRP tasks; secondly, this article introduces a conceptual framework that showcases the Python ecosystem, highlighting libraries and structures that enable efficient data manipulation, analysis, and optimization techniques. Results: This study presents a versatile framework that integrates a variety of Python tools, including but not limited to Pandas, Matplotlib, and Plotly, to streamline and actualize an 8-step MRP process. Additionally, it offers preliminary insights into the integration of the Python-based MRP solution (MRP.py) with Enterprise Resource Planning (ERP) systems. Conclusions: While the article focuses on demonstrating the practicality of Python in MRP, future endeavors will entail empirically integrating MRP.py with the ERP systems in small- and medium-sized companies. This integration will establish real-time data synchronization between the Python and ERP systems, leading to accurate MRP calculations and enhanced decision-making processes.

Intelligent Jammer on Mobile Network LTE Technology: A Study Case in Bucharest

The purpose of this study was to develop a laboratory model that enables the monitoring of communications carried out through mobile phones and their blocking in cases where it is prohibited. The main goal was to realise an intelligent jammer that blocks only illicit communications. The jammer was built with a software-defined radio (SDR) that can be found on the market and is accessible from a financial point of view. This study consisted of an analysis of the behaviour of mobile phones and mobile networks using the long-term evolution (LTE) of UMTS technologies so that the jamming technique can disrupt the communication of the cellular mobile system by using the software-defined radio and Python ecosystem. Because the 5G standalone (5G SA) is not yet implemented in Romania, we could not start developing a laboratory model for jamming this technology. When 5G SA is implemented, we will adapt this intelligent jamming solution to the new technology.

Pattern Matching in Python: Expanding the Horizons of Engineering Applications

Pattern matching is a powerful programming construct that simplifies code, enhances readability, and enables efficient handling of complex data structures. This paper introduces the pattern matching feature newly introduced in Python 3.10 and explores its applications in various engineering domains. The aim of this research is to showcase how Python's pattern matching capability can be leveraged for parsing and analyzing data, structural matching in data analysis, model and system validation, and signal processing. Through illustrative examples and case studies, we demonstrate the versatility and effectiveness of Python pattern matching in solving real-world engineering problems. By introducing pattern matching in Python, this research opens new avenues for engineers and scientists to tackle complex data processing tasks, enhance system validation techniques, and streamline algorithmic implementations. With the integration of pattern matching into Python's ecosystem, the language becomes even more powerful and expressive, empowering practitioners to write cleaner, more concise, and efficient code. This research lays the foundation for the adoption and exploration of pattern matching techniques in Python, highlighting its potential impact on engineering applications and providing a roadmap for future research and development in this field.

WebNORM: a web application for calculating normative mineralogy

Normative mineralogy is a valuable tool for interpreting the mineralogical composition of rocks based on their bulk rock geochemical data. Despite the development of various computational tools for normative mineralogy calculations, the accessibility and continued usage of many existing tools is limited by the use of older languages, licensing constraints, and restricted access to the source code. In this paper, we introduce a Python-based implementation of the normative mineralogy algorithm, which is accessible through a user-friendly web application webNORM, or as part of the pyrolite Python package. The algorithm uses major, minor, and trace element oxides as input and returns up to 31 normative minerals. It offers two methods for estimating the Fe2O3:FeO ratio. The web application offers a convenient interface for users without programming experience, while the pyrolite integration allows more advanced users to leverage the Python ecosystem for end-to-end geochemical data analysis. Comparison between our implementation of the normative calculation and previous implementations shows that our version is accurate and reliable. webNORM offers the geoscience community an accessible solution for calculating normative mineralogy. The algorithm and web application are open-source, and we encourage contributions and adaptations by the community for specific needs and improvements.

Pyglotaran: a lego-like Python framework for global and target analysis of time-resolved spectra

The dynamics of molecular systems can be studied with time-resolved spectroscopy combined with model-based analysis. A Python framework for global and target analysis of time-resolved spectra is introduced with the help of three case studies. The first study, concerning broadband absorption of intersystem crossing in 4-thiothymidine, demonstrates the framework's ability to resolve vibrational wavepackets with a time resolution of ≈10 fs using damped oscillations and their associated spectra and phases. Thereby, a parametric description of the “coherent artifact” is crucial. The second study addresses multichromophoric systems composed of two perylene bisimide chromophores. Here, pyglotaran's guidance spectra and lego-like model composition enable the integration of spectral and kinetic properties of the parent chromophores, revealing a loss process, the undesired production of a radical pair, that reduces the light harvesting efficiency. In the third, time-resolved emission case study of whole photosynthetic cells, a megacomplex containing ≈500 chromophores of five different types is described by a combination of the kinetic models for its elements. As direct fitting of the data by theoretical simulation is unfeasible, our global and target analysis methodology provides a useful ‘middle ground’ where the theoretical description and the fit of the experimental data can meet. The pyglotaran framework enables the lego-like creation of kinetic models through its modular design and seamless integration with the rich Python ecosystem, particularly Jupyter notebooks. With extensive documentation and a robust validation framework, pyglotaran ensures accessibility and reliability for researchers, serving as an invaluable tool for understanding complex molecular systems.Graphical abstract

CyRSoXS: a GPU-accelerated virtual instrument for polarized resonant soft X-ray scattering.

Polarized resonant soft X-ray scattering (P-RSoXS) has emerged as a powerful synchrotron-based tool that combines the principles of X-ray scattering and X-ray spectroscopy. P-RSoXS provides unique sensitivity to molecular orientation and chemical heterogeneity in soft materials such as polymers and biomaterials. Quantitative extraction of orientation information from P-RSoXS pattern data is challenging, however, because the scattering processes originate from sample properties that must be represented as energy-dependent three-dimensional tensors with heterogeneities at nanometre to sub-nanometre length scales. This challenge is overcome here by developing an open-source virtual instrument that uses graphical processing units (GPUs) to simulate P-RSoXS patterns from real-space material representations with nanoscale resolution. This computational framework - called CyRSoXS (https://github.com/usnistgov/cyrsoxs) - is designed to maximize GPU performance, including algorithms that minimize both communication and memory footprints. The accuracy and robustness of the approach are demonstrated by validating against an extensive set of test cases, which include both analytical solutions and numerical comparisons, demonstrating an acceleration of over three orders of magnitude relative to the current state-of-the-art P-RSoXS simulation software. Such fast simulations open up a variety of applications that were previously computationally unfeasible, including pattern fitting, co-simulation with the physical instrument for operando analytics, data exploration and decision support, data creation and integration into machine learning workflows, and utilization in multi-modal data assimilation approaches. Finally, the complexity of the computational framework is abstracted away from the end user by exposing CyRSoXS to Python using Pybind. This eliminates input/output requirements for large-scale parameter exploration and inverse design, and democratizes usage by enabling seamless integration with a Python ecosystem (https://github.com/usnistgov/nrss) that can include parametric morphology generation, simulation result reduction, comparison with experiment and data fitting approaches.

Pyerrors: A python framework for error analysis of Monte Carlo data

We present the pyerrors python package for statistical error analysis of Monte Carlo data. Linear error propagation using automatic differentiation in an object oriented framework is combined with the Γ-method for a reliable estimation of autocorrelation times. Data from different sources can easily be combined, keeping the information on the origin of error components intact throughout the analysis. pyerrors can be smoothly integrated into the existing scientific python ecosystem which allows for efficient and compact analyses. Program summaryProgram Title: pyerrorsCPC Library link to program files:https://doi.org/10.17632/7ncw242ymh.1Developer's repository link:https://github.com/fjosw/pyerrorsLicensing provisions: MITProgramming language: pythonNature of problem: Data obtained from Markov chain Monte Carlo simulations exhibits autocorrelations. These become particularly severe when approaching the continuum limit of lattice discretized quantum field theories which becomes more and more relevant in modern day large scale simulations. In order to obtain reliable error estimates these autocorrelations have to be taken into account in complex data analysis workflows.Solution method: Linear error propagation in combination with automatic differentiation is implemented in a new python data type which keeps track of statistical errors across multiple sources of uncertainty. Operator overloading allows for a seamless integration into the scientific python ecosystem and into existing workflows. The Γ-method facilitates a controlled estimate of integrated autocorrelation times at any stage of the analysis and provides reliable error estimates without numerical overhead.