Pandas DataFrames Research Articles

Blik is an extensible 3D visualisation tool for the annotation and analysis of cryo-electron tomography data.

Powerful, workflow-agnostic and interactive visualisation is essential for the ad hoc, human-in-the-loop workflows typical of cryo-electron tomography (cryo-ET). While several tools exist for visualisation and annotation of cryo-ET data, they are often integrated as part of monolithic processing pipelines, or focused on a specific task and offering limited reusability and extensibility. With each software suite presenting its own pros and cons and tools tailored to address specific challenges, seamless integration between available pipelines is often a difficult task. As part of the effort to enable such flexibility and move the software ecosystem towards a more collaborative and modular approach, we developed blik, an open-source napari plugin for visualisation and annotation of cryo-ET data (source code: https://github.com/brisvag/blik). blik offers fast, interactive, and user-friendly 3D visualisation thanks to napari, and is built with extensibility and modularity at the core. Data is handled and exposed through well-established scientific Python libraries such as numpy arrays and pandas dataframes. Reusable components (such as data structures, file read/write, and annotation tools) are developed as independent Python libraries to encourage reuse and community contribution. By easily integrating with established image analysis tools-even outside of the cryo-ET world-blik provides a versatile platform for interacting with cryo-ET data. On top of core visualisation features-interactive and simultaneous visualisation of tomograms, particle picks, and segmentations-blik provides an interface for interactive tools such as manual, surface-based and filament-based particle picking, and image segmentation, as well as simple filtering tools. Additional self-contained napari plugins developed as part of this work also implement interactive plotting and selection based on particle features, and label interpolation for easier segmentation. Finally, we highlight the differences with existing software and showcase blik's applicability in biological research.

The FAST-HEP toolset: Using YAML to make tables out of trees

The Faster Analysis Software Taskforce (FAST) is a small, European group of HEP researchers that have been investigating and developing modern software approaches to improve HEP analyses. We present here an overview of the key product of this effort: a set of packages that allows a complete implementation of an analysis using almost exclusively YAML files. Serving as an analysis description language (ADL), this toolset builds on top of the evolving technologies from the Scikit-HEP and IRIS-HEP projects as well as industry-standard libraries such as Pandas and Matplotlib. Data processing starts with event-level data (the trees) and can proceed by adding variables, selecting events, performing complex user-defined operations and binning data, as defined in the YAML description. The resulting outputs (the tables) are stored as Pandas dataframes which can be programmatically manipulated and converted to plots or inputs for fitting frameworks. No longer just a proof-of-principle, these tools are now being used in CMS analyses, the LUX-ZEPLIN experiment, and by students on several other experiments. In this talk we will showcase these tools through examples, highlighting how they address the different experiments’ needs, and compare them to other similar approaches.

Pandas DataFrames for a FAST binned analysis at CMS

Binned data frames are a generalisation of multi-dimensional histograms, represented in a tabular format with one category per row containing the labels, bin contents, uncertainties and so on. Pandas is an industry-standard tool, which provides a data frame implementation complete with routines for data frame manipultion, persistency, visualisation, and easy access to “big data” scientific libraries and machine learning tools. FAST (the Faster Analysis Software Taskforce) has developed a generic approach for typical binned HEP analyses, driving the summary of ROOT Trees to multiple binned DataFrames with a yaml-based analysis description. Using Continuous Integration to run subsets of the analysis, we can monitor and test changes to the analysis itself, and deploy documentation automatically. This report describes this approach using examples from a public CMS tutorial and details the benefit over traditional methods.

BioPandas: Working with molecular structures in pandas DataFrames

BioPandas: Working with molecular structures in pandas DataFrames

Gadfly: A pandas-based Framework for Analyzing GADGET Simulation Data

We present the first public release (v0.1) of the open-source gadget Dataframe Library: gadfly. The aim of this package is to leverage the capabilities of the broader python scientific computing ecosystem by providing tools for analyzing simulation data from the astrophysical simulation codes gadget and gizmo using pandas, a thoroughly documented, open-source library providing high-performance, easy-to-use data structures that is quickly becoming the standard for data analysis in python. Gadfly is a framework for analyzing particle-based simulation data stored in the HDF5 format using pandas DataFrames. The package enables efficient memory management, includes utilities for unit handling, coordinate transformations, and parallel batch processing, and provides highly optimized routines for visualizing smoothed-particle hydrodynamics data sets.

Pypet: A Python Toolkit for Data Management of Parameter Explorations.

pypet (Python parameter exploration toolkit) is a new multi-platform Python toolkit for managing numerical simulations. Sampling the space of model parameters is a key aspect of simulations and numerical experiments. pypet is designed to allow easy and arbitrary sampling of trajectories through a parameter space beyond simple grid searches. pypet collects and stores both simulation parameters and results in a single HDF5 file. This collective storage allows fast and convenient loading of data for further analyses. pypet provides various additional features such as multiprocessing and parallelization of simulations, dynamic loading of data, integration of git version control, and supervision of experiments via the electronic lab notebook Sumatra. pypet supports a rich set of data formats, including native Python types, Numpy and Scipy data, Pandas DataFrames, and BRIAN(2) quantities. Besides these formats, users can easily extend the toolkit to allow customized data types. pypet is a flexible tool suited for both short Python scripts and large scale projects. pypet's various features, especially the tight link between parameters and results, promote reproducible research in computational neuroscience and simulation-based disciplines.

Pypet: a python toolkit for simulations and numerical experiments

(python parameter exploration toolkit [1]) is a new multi-platform python toolkit for management of simulations and storage of numerical data. Exploring or sampling the space of model parameters is one key aspect of simulations and numerical experiments. pypet was especially designed to allow easy and arbitrary sampling of trajectories through a parameter space beyond simple grid searches. Moreover, special focus is put on managing different neuron models in python network simulations like BRIAN [2]. Simulation parameters as well as the obtained results are collected by pypet and stored in the widely used HDF5 file format [3]. This allows fast and convenient loading of data for further analyses. Furthermore, pypet provides an environment with various features. For example, among these are multiprocessing for fast parallel simulations, dynamic loading of data, integration of Git version control, and supervision of experiments via the electronic lab notebook Sumatra [4]. A rich set of data formats is supported, encompassing native python types, numpy and scipy data, pandas DataFrames [5], and data from BRIAN [2]. Moreover, the toolkit is easily extendible to allow the user to add customized data formats. pypet is a very flexible tool and suited for short python scripts as well as large scale projects. Thus, pypet supports reproducible research in computational neuroscience and other disciplines that involve simulations and numerical experiments.

AtomPy: An Open Atomic Data Curation Environment for Astrophysical Applications

We present a cloud-computing environment, referred to as AtomPy, based on Google-Drive Sheets and Pandas (Python Data Analysis Library) DataFrames to promote community-driven curation of atomic data for astrophysical applications, a stage beyond database development. The atomic model for each ionic species is contained in a multi-sheet workbook, tabulating representative sets of energy levels, A-values and electron impact effective collision strengths from different sources. The relevant issues that AtomPy intends to address are: (i) data quality by allowing open access to both data producers and users; (ii) comparisons of different datasets to facilitate accuracy assessments; (iii) downloading to local data structures (i.e., Pandas DataFrames) for further manipulation and analysis by prospective users; and (iv) data preservation by avoiding the discard of outdated sets. Data processing workflows are implemented by means of IPython Notebooks, and collaborative software developments are encouraged and managed within the GitHub social network. The facilities of AtomPy are illustrated with the critical assessment of the transition probabilities for ions in the hydrogen and helium isoelectronic sequences with atomic number Z ≤ 10.