Scientific Python Research Articles

Interoperable and scalable echosounder data processing with Echopype

Abstract Echosounders are high-frequency sonar systems used to sense fish and zooplankton underwater. Their deployment on a variety of ocean observing platforms is generating vast amounts of data at an unprecedented speed from the oceans. Efficient and integrative analysis of these data, whether across different echosounder instruments or in combination with other oceanographic datasets, is crucial for understanding marine ecosystem response to the rapidly changing climate. Here we present Echopype, an open-source Python software library designed to address this need. By standardizing data as labeled, multi-dimensional arrays encoded in the widely embraced netCDF data model following a community convention, Echopype enhances the interoperability of echosounder data, making it easier to explore and use. By leveraging scientific Python libraries optimized for distributed computing, Echopype achieves computational scalability, enabling efficient processing in both local and cloud computing environments. Echopype’s modularized package structure further provides a unified framework for expanding support for additional instrument raw data formats and incorporating new analysis functionalities. We plan to continue developing Echopype by supporting and collaborating with the echosounder user community, and envision that the growth of this package will catalyze the integration of echosounder data into broader regional and global ocean observation strategies.

Blik is an extensible 3D visualisation tool for the annotation and analysis of cryo-electron tomography data.

Powerful, workflow-agnostic and interactive visualisation is essential for the ad hoc, human-in-the-loop workflows typical of cryo-electron tomography (cryo-ET). While several tools exist for visualisation and annotation of cryo-ET data, they are often integrated as part of monolithic processing pipelines, or focused on a specific task and offering limited reusability and extensibility. With each software suite presenting its own pros and cons and tools tailored to address specific challenges, seamless integration between available pipelines is often a difficult task. As part of the effort to enable such flexibility and move the software ecosystem towards a more collaborative and modular approach, we developed blik, an open-source napari plugin for visualisation and annotation of cryo-ET data (source code: https://github.com/brisvag/blik). blik offers fast, interactive, and user-friendly 3D visualisation thanks to napari, and is built with extensibility and modularity at the core. Data is handled and exposed through well-established scientific Python libraries such as numpy arrays and pandas dataframes. Reusable components (such as data structures, file read/write, and annotation tools) are developed as independent Python libraries to encourage reuse and community contribution. By easily integrating with established image analysis tools-even outside of the cryo-ET world-blik provides a versatile platform for interacting with cryo-ET data. On top of core visualisation features-interactive and simultaneous visualisation of tomograms, particle picks, and segmentations-blik provides an interface for interactive tools such as manual, surface-based and filament-based particle picking, and image segmentation, as well as simple filtering tools. Additional self-contained napari plugins developed as part of this work also implement interactive plotting and selection based on particle features, and label interpolation for easier segmentation. Finally, we highlight the differences with existing software and showcase blik's applicability in biological research.

Pyerrors: A python framework for error analysis of Monte Carlo data

We present the pyerrors python package for statistical error analysis of Monte Carlo data. Linear error propagation using automatic differentiation in an object oriented framework is combined with the Γ-method for a reliable estimation of autocorrelation times. Data from different sources can easily be combined, keeping the information on the origin of error components intact throughout the analysis. pyerrors can be smoothly integrated into the existing scientific python ecosystem which allows for efficient and compact analyses. Program summaryProgram Title: pyerrorsCPC Library link to program files:https://doi.org/10.17632/7ncw242ymh.1Developer's repository link:https://github.com/fjosw/pyerrorsLicensing provisions: MITProgramming language: pythonNature of problem: Data obtained from Markov chain Monte Carlo simulations exhibits autocorrelations. These become particularly severe when approaching the continuum limit of lattice discretized quantum field theories which becomes more and more relevant in modern day large scale simulations. In order to obtain reliable error estimates these autocorrelations have to be taken into account in complex data analysis workflows.Solution method: Linear error propagation in combination with automatic differentiation is implemented in a new python data type which keeps track of statistical errors across multiple sources of uncertainty. Operator overloading allows for a seamless integration into the scientific python ecosystem and into existing workflows. The Γ-method facilitates a controlled estimate of integrated autocorrelation times at any stage of the analysis and provides reliable error estimates without numerical overhead.

The SunPy Project: An interoperable ecosystem for solar data analysis

The SunPy Project is a community of scientists and software developers creating an ecosystem of Python packages for solar physics. The project includes the sunpy core package as well as a set of affiliated packages. The sunpy core package provides general purpose tools to access data from different providers, read image and time series data, and transform between commonly used coordinate systems. Affiliated packages perform more specialized tasks that do not fall within the more general scope of the sunpy core package. In this article, we give a high-level overview of the SunPy Project, how it is broader than the sunpy core package, and how the project curates and fosters the affiliated package system. We demonstrate how components of the SunPy ecosystem, including sunpy and several affiliated packages, work together to enable multi-instrument data analysis workflows. We also describe members of the SunPy Project and how the project interacts with the wider solar physics and scientific Python communities. Finally, we discuss the future direction and priorities of the SunPy Project.

An array-oriented Python interface for FastJet

Analysis on HEP data is an iterative process in which the results of one step often inform the next. In an exploratory analysis, it is common to perform one computation on a collection of events, then view the results (often with histograms) to decide what to try next. Awkward Array is a Scikit-HEP Python package that enables data analysis with array-at-a-time operations to implement cuts as slices, combinatorics as composable functions, etc. However, most C++ HEP libraries, such as FastJet, have an imperative, one-particle-at-a-time interface, which would be inefficient in Python and goes against the grain of the array-at-a-time logic of scientific Python. Therefore, we developed fastjet, a pip-installable Python package that provides FastJet C++ binaries, the classic (particle-at-a-time) Python interface, and the new array-oriented interface for use with Awkward Array. The new interface streamlines interoperability with scientific Python software beyond HEP, such as machine learning. In one case, adopting this library along with other array-oriented tools accelerated HEP analysis code by a factor of 20. It was designed to be easily integrated with libraries in the Scikit-HEP ecosystem, including Uproot (file I/O), hist (histogramming), Vector (Lorentz vectors), and Coffea (high-level glue). We discuss the design of the fastjet Python library, integrating the classic interface with the array oriented interface and with the Vector library for Lorentz vector operations. The new interface was developed as open source.

Pyrate: a novel system for data transformations, reconstruction and analysis

The Pyrate framework provides a dynamic, versatile, and memory-efficient approach to data format transformations, object reconstruction and data analysis in particle physics. The system is implemented with the Python programming language, allowing easy access to the scientific Python ecosystem and commodity big data technologies. Developed within the context of the SABRE South experiment for dark matter direct detection, Pyrate relies on a blackboard design pattern where algorithmic trees are dynamically generated throughout a run where root nodes are managed by a central control unit. The system guarantees an economical usage of memory allocated by algorithms where individual algorithmic instances can be reused for multiple objects. The framework is intended to improve upon the user experience, portability and scalability of offline software systems currently available in the particle physics community with particular attention to medium to small-scale experiments.

Breast cancer prediction using supervised machine learning techniques

Breast cancer is one of the most prevalent diseases in India’s urban regions and the second most common in the country’s rural parts. In India, a woman is diagnosed with breast cancer growth every four minutes, and a woman dies from breast cancer sickness every thirteen minutes. Over half of breast cancer patients in India are diagnosed with stage 3 or 4 illness, which has extremely low survival rates; hence, an urgent need exists for a rapid detection strategy. To forecast if a patient is at risk for breast cancer, we utilise the classification techniques of machine learning, in which the machine learning model learns from the previous information and can anticipate on the new information that is generated by the data. To create a model using Logistic Regression, Support Vector Machines, and Random Forests, this dataset was collected from the UCI repository and studied in this study. The primary goal is to improve the accuracy, precision, and sensitivity of all the algorithms that are used to categorise data in terms of the competency and viability of each and every algorithm. Random Forest has been shown to be the most accurate in classifying breast cancer, with a precision of 98.60 percent in tests. The Scientific Python Development Environment is used to carry out this machine learning study, which is written in the python programming language.

Crowd Face Detection with Naive Bayes in Attendance System Using Raspberry Pi

PT. Restu Agung Narogong is a company with a total of 176 employees, queues often occur in the attendance process, both incoming and outgoing attendance. The employee needs to register their attendance. It is time consuming during the shift change. Therefore, a biometric system is needed to support the attendance system to identify employee without registering themselves. One of the alternative biometric systems is face recognition by using a computer vision. The purpose is to implement a crowd face detection with Raspberry Pi using the Naïve Bayes classifier. This system uses an algorithm to extract facial characteristics into mathematical data. Then the data is compared with data from other facial characteristics collected in the database. This device uses Python as a programming language with some of the scientific Python libraries. The testing of the Naïve Bayes method was conducted using a sample of dataset of 370 augmented facial imagery. The accuracy of this implementation is 76.31%, the precision is 78.25% and recall 81.25%. The background and lighting of the captured image affect the accuracy of this device.

The SpacePy space science package at 12 years

For over a decade, the SpacePy project has contributed open-source solutions for the production and analysis of heliophysics data and simulation results. Here we introduce SpacePy’s functionality for the scientific user and present relevant design principles. We examine recent advances and the future of SpacePy in the broader scientific Python ecosystem, concluding with some of the work that has used SpacePy.

Pyleoclim: Paleoclimate Timeseries Analysis and Visualization With Python

AbstractWe present a Python package geared toward the intuitive analysis and visualization of paleoclimate timeseries, Pyleoclim. The code is open‐source, object‐oriented, and built upon the standard scientific Python stack, allowing users to take advantage of a large collection of existing and emerging techniques. We describe the code's philosophy, structure, and base functionalities and apply it to three paleoclimate problems: (a) orbital‐scale climate variability in a deep‐sea core, illustrating spectral, wavelet, and coherency analysis in the presence of age uncertainties; (b) correlating a high‐resolution speleothem to a climate field, illustrating correlation analysis in the presence of various statistical pitfalls (including age uncertainties); (c) model‐data confrontations in the frequency domain, illustrating the characterization of scaling behavior. We show how the package may be used for transparent and reproducible analysis of paleoclimate and paleoceanographic datasets, supporting Findable, Accessible, Interoperable, and Reusable software and an open science ethos. The package is supported by an extensive documentation and a growing library of tutorials shared publicly as videos and cloud‐executable Jupyter notebooks, to encourage adoption by new users.

MetPy: A Meteorological Python Library for Data Analysis and Visualization

Abstract MetPy is an open-source, Python-based package for meteorology, providing domain-specific functionality built extensively on top of the robust scientific Python software stack, which includes libraries like NumPy, SciPy, Matplotlib, and xarray. The goal of the project is to bring the weather analysis capabilities of GEMPAK (and similar software tools) into a modern computing paradigm. MetPy strives to employ best practices in its development, including software tests, continuous integration, and automated publishing of web-based documentation. As such, MetPy represents a sustainable, long-term project that fills a need for the meteorological community. MetPy’s development is substantially driven by its user community, both through feedback on a variety of open, public forums like Stack Overflow, and through code contributions facilitated by the GitHub collaborative software development platform. MetPy has recently seen the release of version 1.0, with robust functionality for analyzing and visualizing meteorological datasets. While previous versions of MetPy have already seen extensive use, the 1.0 release represents a significant milestone in terms of completeness and a commitment to long-term support for the programming interfaces. This article provides an overview of MetPy’s suite of capabilities, including its use of labeled arrays and physical unit information as its core data model, unit-aware calculations, cross sections, skew T and GEMPAK-like plotting, station model plots, and support for parsing a variety of meteorological data formats. The general road map for future planned development for MetPy is also discussed.

Modflow-setup: Robust automation of groundwater model construction

In an age of both big data and increasing strain on water resources, sound management decisions often rely on numerical models. Numerical models provide a physics-based framework for assimilating and making sense of information that by itself only provides a limited description of the hydrologic system. Often, numerical models are the best option for quantifying even intuitively obvious connections between human activities and water resource impacts. However, despite many recent advances in model data assimilation and uncertainty quantification, the process of constructing numerical models remains laborious, expensive, and opaque, often precluding their use in decision making. Modflow-setup aims to provide rapid and consistent construction of MODFLOW groundwater models through robust and repeatable automation. Common model construction tasks are distilled in an open-source, online code base that is tested and extensible through collaborative version control. Input to Modflow-setup consists of a single configuration file that summarizes the workflow for building a model, including source data, construction options, and output packages. Source data providing model structure and parameter information including shapefiles, rasters, NetCDF files, tables, and other (geolocated) sources to MODFLOW models are read in and mapped to the model discretization, using Flopy and other general open-source scientific Python libraries. In a few minutes, an external array-based MODFLOW model amenable to parameter estimation and uncertainty quantification is produced. This paper describes the core functionality of Modflow-setup, including a worked example of a MODFLOW 6 model for evaluating pumping impacts to a lake in central Wisconsin, United States.

Machine learning models for predicting the number of COVID-19 patients in Ukraine and India

Models for predicting the number of patients with COVID-19 using machine learning methods have been built. The data for models are obtained from various official sources, including the World Health Organization, from the beginning of the epidemic to the present time. The data in Ukraine and India were selected to teach models for predicting the number of patients with COVID-19. Algorithms of linear regression for Ukraine and gradient boosting for India proved to be the methods that provided high accuracy of the forecast for the existing data. Data analysis was performed using the Python programming language with Sklearn library which is based on SciPy (Scientific Python). In addition, the XGboost gradient boost algorithm library was used. To develop the model, multifactor prediction of time series with the delays as predictors was chosen. It is established that the such characteristics as the date of the event, day of the week, week number, month affect to the model. Model errors are smallest and forecast accuracy were estimated with the best values of 0.83 for Ukraine and 0.75 for India. The built models allow to predict the epidemiological situation in the future, to coordinate actions in different areas of health care and to carry out reasonable preventive measures at the state level.

OpenEP-Py and OpenEP-GUI: An interactive platform for the visualisation and analysis of clinical and simulated cardiac electrophysiology data

Abstract Funding Acknowledgements Type of funding sources: Public grant(s) – National budget only. Main funding source(s): 1) King’s Health Partnership Research & Development Challenge Fund Award (funded through Medical Research Council, UK) 2) British Heart Foundation, UK Background Electroanatomic mapping data is typically stored in proprietary formats and is difficult to access and analyse. This presents a barrier to electrophysiology research. OpenEP has been developed over the past decade to address this issue, with the first public release of the software in late 2020. OpenEP provides a standard format for storing electrophysiology data as well as an extensive suite of analysis tools. However, the current implementation of OpenEP has several limitations. Firstly, it is written in Matlab and thus not accessible to researchers without a Matlab license. Secondly, it lacks a graphical interface mandating a level of programming expertise to use. Finally, it lacks interfaces for computational modelling data, limiting its use to clinical rather than simulated data. Purpose In this work we address these limitations by developing (1) a Python-based implementation of OpenEP; (2) a graphical interface for interacting with OpenEP data and (3) interfaces for computational modelling environments. Methods Following review of the OpenEP source code, we developed use-case documentation for each OpenEP function and the proposed graphical user interface. By inspection of simulation outputs from an open-source simulation environment (openCARP) we designed new interfaces for modelling data. The implementation of OpenEP-py and OpenEP-GUI is based on the standard scientific Python stack with a PyQt front-end. Results OpenEP-Py is an open-source Python implementation for manipulation of electrophysiology data. The feature set is under active development and aims to match that of the Matlab implementation of OpenEP. In addition, OpenEP-Py can read data from cardiac electrophysiology simulations. The simulated data is stored in the standard OpenEP data structure, meaning analysis tools for clinical data can be used on the simulated data. Further, clinical data can be exported into the openCARP format, streamlining the creation of patient-specific models for simulations. Finally, because the software is open-source, OpenEP provides a transparent process for analysing data and thereby improves the reproducibility of studies. OpenEP-GUI (Figure 1) is a cross-platform desktop application for the visualisation and analysis of clinical and simulated cardiac electrophysiology data. It uses OpenEP-Py for loading, visualising, analysis and exporting electroanatomic mapping data. Multiple cases can be loaded via the System Manager, facilitating qualitative and quantitative comparison between cases. OpenEP-Py and OpenEP-GUI are licensed under the GNU-GPL v3 [WS1] and available online. Conclusions OpenEP-Py and OpenEP-GUI provide a simple yet effective way to perform cardiac electrophysiology research. The software is both open-source and under active development. We welcome feedback, feature requests, and contributions from the wider electrophysiology research community.

Collaboration in software ecosystems: A study of work groups in open environment

As a particular type of software ecosystem, an open source software ecosystem (OSSECO) is a collection of interdependent open source software (OSS) projects which are developed and evolve together. Events happening within an OSSECO inherently involve the collaboration of participants from multiple OSS projects, forming a temporary work group. However, it is still unclear how different members of a work group collaborate to fix cross-project bugs, a typical event in the maintenance of OSSECOs. This study aims to investigate the characteristics of collaboration within a work group when fixing cross-project bugs in an OSSECO. It involves the participants from the upstream (which caused the bugs) and the downstream (which were affected by the bugs) OSS projects. We conducted our study on 236 cross-project bugs from the scientific Python ecosystem, involving 571 participants and 91 OSS projects, to understand open collaboration within a work group. We established a quantitative analysis to investigate the members of a work group, along with a qualitative analysis to understand the roles of the members from different OSS communities. The results show that: (1) A typical work group is constituted of four to eight members from the core development teams of the two OSS communities. More members concern with the upstream OSS projects and few can make active contributions to both sides; (2) Distinct responsibilities are taken by the two OSS communities, with the downstream members as the problem-finders and the upstream members as the decision-makers or gatekeepers. Our findings reveal the collaborative mechanism and the responsibility allocation between the upstream and downstream OSS communities in the ecosystems. • A study on 236 cross-project bug fixing activities to understand open collaboration in open source software ecosystems. • Six findings that reveal the members of a cross-project bug fixing work group. • Five findings that summarize the roles of cross-project bug fixing group members. • Recommendations to the researchers, developers, and practitioners.

Free, flexible and fast: Orientation mapping using the multi-core and GPU-accelerated template matching capabilities in the Python-based open source 4D-STEM analysis toolbox Pyxem

This work presents the new template matching capabilities implemented in Pyxem, an open source Python library for analyzing four-dimensional scanning transmission electron microscopy (4D-STEM) data. Template matching is a brute force approach for deriving local crystal orientations. It works by comparing a library of simulated diffraction patterns to experimental patterns collected with nano-beam and precession electron diffraction (NBED and PED). This is a computationally demanding task, therefore the implementation combines efficiency and scalability by utilizing multiple CPU cores or a graphical processing unit (GPU). The code is built on top of the scientific Python ecosystem, and is designed to support custom and reproducible workflows that combine the image processing, template library generation, indexation and visualization all in one environment. The tools are agnostic to file size and format, which is significant in light of the increased adoption of pixelated detectors from different manufacturers. This paper details the implementation and validation of the method. The method is illustrated by calculating orientation maps of nanocrystalline materials and precipitates embedded in a crystalline matrix. The combination of speed and flexibility opens the door for automated parameter studies and real-time on-line orientation mapping inside the TEM.

PyOpenSci: Open and reproducible research, powered by Python

pyOpenSci (short for Python Open Science), funded by the Alfred P. Sloan Foundation, is building a diverse community that supports well documented, open source Python software that enables open reproducible science. pyOpenSci will work with the community to openly develop best practice guidelines and open standards for scientific Python software, which will be reinforced through a community-led peer review process and training. Packages that complete the peer review process become a part of the pyOpenSci ecosystem, where maintenance can be shared to ensure longevity and stability in code. pyOpenSci packages are also eligible for a “fast tracked” acceptance to JOSS (Journal of Open Source Software). In addition, we provide review for open science tools that would be of interest to TDWG members but are not within scope for JOSS, such as API (Application Programming Interface) wrappers. pyOpenSci is built on top of the successful model of rOpenSci, founded in 2011, which has fostered the development of several useful biodiversity informatics R packages. The pyOpenSci team looks to following the lessons learned by rOpenSci, to create a similarly successful community. We invite TDWG members developing open source software tools in Python to become part of the pyOpenSci community.

Prophecy on Programming Language using Machine Learning Algorithms

A programming language is a computer language developers use to develop software programs, scripts, or other sets of instruction for computers to execute. It is difficult to determine which programming language is widely used. In our work, I have analyzed and compared the classification results of various machine learning models and find out which programming language is widely used by developers. I have used Support Vector Machine (SVM), K neighbor classifier (KNN),Decision Tree Classifier(CART) for our comparative study. My task is to analyze different data and to classify them for the efficiency of each algorithm in terms of accuracy, precision, recall, and F1 Score. My best accuracy was 94.29% percent which was found using SVM. These techniques are coded in python and executed in Jupyter NoteBook, the Scientific Python Development Environment. Our experiments have shown that SVM is the best for predictive analysis and from our study that SVM is the well-suited algorithm for the prediction of the most widely used programming language.

PyMoDAQ: An open-source Python-based software for modular data acquisition.

Thanks to the recent multiplication of scientific Python packages in the open-source software landscape, Data Acquisition frameworks (DAQ-Fs) appear as versatile replacements of custom-made or costly commercial solutions. PyMoDAQ is a DAQ-F focusing on easy-to-use graphical user interfaces allowing a simple control and automation of a large variety of experimental setups. Its development included a highly modular structure allowing any experimental data acquisition as a function of multiple varying parameters. It offers numerous additional functionalities: instrument and setup configuration, plotting, saving, logging, etc. Live visual feedback is available at all times to monitor the ongoing experiment. Flexibility of its user interfaces is the key advantage of PyMoDAQ allowing also its integration as the core of more focused applications. Its hierarchical binary format data saving mechanism includes experimental metadata highly compatible with FAIR (Findable, Accessible,Interoperable, Reusable) data. Among the presented characteristics, seven criteria have been chosen to judge the pertinence of PyMoDAQ as a versatile DAQ-F. They are also the basis for a comparison with other existing frameworks highlighting the novelty of PyMoDAQ.

Echopype: Enhancing the interoperability and scalability of ocean sonar data processing

The recent expanded availability of autonomous ocean sonar systems has created a data deluge. Despite their potential in advancing our understanding of the marine ecosystems, a large proportion of these new data remains significantly under-utilized. Specifically, the sheer tedium in data wrangling has significantly diverted efforts of the research community away from answering scientific questions. One of the root causes of this problem is the lack of an interoperable and scalable analysis workflow that adapts well with the rapidly increasing data volume and can be easily integrated with other oceanographic observations. To address this challenge, we developed an open-source software package “Echopype” that leverages the power of existing distributed computing libraries in the scientific Python ecosystem to directly interface data storage and computation on the cloud. Echopype provides tools to convert manufacturer-specific data files to a standardized, labeled multi-dimensional format that is familiar to the wider oceanography and geoscience communities, and offers a uniform computational interface for data originating from heterogeneous instrument sources. We envision that the continued development of Echopype will catalyze broader use of sonar data in oceanographic studies.