Open-source Software Framework Research Articles

Rucio: Scientific Data Management

Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. The data can be distributed across heterogeneous data centers at widely distributed locations. Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. In this article, we detail the fundamental concepts of Rucio, describe the architecture along with implementation details, and report operational experience from production usage.

Numerical Analysis of Electromechanically Driven Bone Remodeling Using the Open-source Software Framework.

Natural bone remodeling is the mechanism that regulates the relationship between bone morphology and external mechanical loads applied to it. This phenomenon has been studied extensively, including multiple numerical models that have been formulated to predict the density distribution and its evolution in several bone types. However, despite these models, bone remodeling mechanism under different stimuli is still not well understood. We implemented a recently proposed electromechanically driven bone remodeling model that encompasses both mechanical and therapeutic electrical stimuli using an open-source software framework, and studied a two-dimensional (2D) plate model and a femur bone model, respectively. For discretization, we employed the finite element method (FEM) for the spatial quantities and Euler scheme for the time derivatives. The simulation results demonstrate that the density distribution is changed under electrical stimulation, generally resulting in a greater mass deposition. This study supports the possibility of enhancing and accelerating the bone remodeling process via simultaneous application of electrical and mechanical stimulus.

The Additivity Project - Use Cases and User Interface

Herbarium specimens are central to botanical science and of rising importance thanks to increasing accessibility and broadened usability. Alongside the many new uses of specimen data, sit a range of traditional uses supporting the collection of morphological data and their application to taxonomy and systematics. (Henning et al. 2018). Technical workflows are needed to support the sustainable collection of this traditional information and maintain the high quality of the morphological data. Data exchange and re-usability requires the use of accepted controlled vocabularies (community approved) that are accessible (web-based ontologies and term vocabularies) and reliable (long-term availability/unique identifiers). The same applies to datasets that must be stored accessibly and sustainably by maintaining all data relationships that would facilitate convenient re-use. This project aims to construct a comprehensive workflow to optimise the delimitation and characterisation (“descriptions”) of taxa (see complementary talk by Plitzner et al.). It is implemented on the open-source software framework of the EDIT Platform for Cybertaxonomy (http://www.cybertaxonomy.org, Ciardelli et al. 2009) extending the workflow for sample data processing developed in a preceding project (Kilian et al. 2015). The principal goals of this new software component are: specimen-level recording and storage of character data in structured character matrices generating taxon characterisations by aggregating the individual specimen-based datasets using and developing community-coordinated, ontology-based exemplar vocabularies persistently linking character datasets with source specimens for high visibility and re-usability specimen-level recording and storage of character data in structured character matrices generating taxon characterisations by aggregating the individual specimen-based datasets using and developing community-coordinated, ontology-based exemplar vocabularies persistently linking character datasets with source specimens for high visibility and re-usability The angiosperm order, Caryophyllales, provides an exemplar use case through cooperation with the Global Caryophyllales Initiative (Borsch et al. 2015). A basic set of morphological terms and vocabularies has been obtained from various online sources (ontologies, glossaries) and can be used, searched and expanded in the EDIT platform. The terms are categorised into: structures, properties and states. Different editors have been developed to combine structure and property terms to characters and assign a customised state vocabulary (categorical) or suitable values and units (numerical) to them. The workflow is built around a data set defining the taxonomic environment of individual use cases. A data set is specified by the characters and a taxonomic group, which can be filtered by area or rank. The dataset can be opened in a tabular representation (character matrix) to enter preselected state terms or values for the individual specimen. The matrix provides several features for basic comparison and analysis and allows the entry of alternative datasets (e.g. literature). Finally, the aggregation of data subsets to potential taxonomic units by adding up the values and summarising character states, allows the convenient test of taxonomic hypotheses. The term additivity is used here to describe this set of workflows and processes adding value to herbarium specimens and accumulating the specimen data for a taxon description.

The Platform for Cybertaxonomy: Standards, services and tools

The Platform for Cybertaxonomy (http://www.cybertaxonomy.org) is a standards-based open-source software framework covering the breadth of the taxonomic workflow, from fieldwork to publication (Ciardelli et al. 2009). It provides coupled tools for full, customized access to taxonomic data, editing and management, and collaborative team work. At the core of the platform is the Common Data Model (CDM, Müller et al. 2017), offering a comprehensive information model covering all relevant data domains: names and classifications, descriptive data (morphological and molecular), media, geographic information, literature, specimens, types, persons, and external resources. Platform compliant software interacts via services and includes the following components: CDM Server Taxonomic Editor Rich Client Web-based editors Drupal-based and highly configurable portal software Map services and map viewer Xper2 descriptive data editor Specimen search tool Import and export modules CDM Server Taxonomic Editor Rich Client Web-based editors Drupal-based and highly configurable portal software Map services and map viewer Xper2 descriptive data editor Specimen search tool Import and export modules Recent platform-based developments include software components for deriving formal species-level descriptions from measurements on individual specimens (Henning et al. 2018) as well as a registration system for nomenclatural acts of algae (Phycobank, https://www.phycobank.org/). Currently, about 30 portals with regional and taxonomic foci are using the Platform for Cybertaxonomy as their technical basis for capturing, managing, and publishing biodiversity data over the World Wide Web. Prominent examples are the Euro+Med Plantbase, the International Caryophyllales Network, and the Flora of Greece.

Rocketsled: a software library for optimizing high-throughput computational searches

A major goal of computation is to optimize an objective for which a forward calculation is possible, but no inverse solution exists. Examples include tuning parameters in a nuclear reactor design, optimizing structures in protein folding, or predicting an optimal materials composition for a functional application. In such instances, directing calculations in an optimal manner is important to obtaining the best possible solution within a fixed computational budget. Here, we introduce Rocketsled, an open-source Python-based software framework to help users optimize arbitrary objective functions. Rocketsled is built upon the existing FireWorks workflow software, which allows its computations to scale to supercomputing centers and for its objective functions to be complex, long-running, and error-prone workflows. Other unique features of Rocketsled include its ability to easily swap out the underlying optimizer, the ability to handle multiple competing objectives, the possibility to inject domain knowledge into the optimizer through feature engineering, incorporation of uncertainty estimates, and its parallelization scheme for running in high-throughput at massive scale. We demonstrate the generality of Rocketsled by applying it to optimize several common test functions (Branin-Hoo, Rosenbrock 2D, and Hartmann 6D). We highlight its potential impact through two example use cases for computational materials science. In a search for photocatalysts for hydrogen production among 18 928 perovskites previously calculated with density functional theory, the untuned Rocketsled Random Forest optimizer explores the search space with approximately 6–28 times fewer calculations than random search. In a search among 7394 materials for superhard candidates, Rocketsled requires approximately 61 times fewer calculations than random search to discover interesting candidates. Thus, Rocketsled provides a practical framework for establishing complex optimization schemes with minimal code infrastructure and enables the efficient exploration of otherwise prohibitively large search spaces.

An online mapping database of molecular markers of drug resistance in Plasmodium falciparum: the ACT Partner Drug Molecular Surveyor

BackgroundPrior to this project, only a handful of online visualizations existed for exploring the published literature on molecular markers of antimalarial drug resistance, and none specifically for the markers associated with Plasmodium falciparum resistance to the partner drugs in artemisinin-based combination therapy (ACT). Molecular information is collected in studies with different designs, using a variety of molecular methodologies and data analysis strategies, making it difficult to compare across studies. The purpose of this project was to develop a free online tool, which visualizes the widely published data on molecular markers of antimalarial drug resistance, starting with the two genes pfcrt and pfmdr-1, associated with resistance to the three most common partner drugs; amodiaquine, lumefantrine and mefloquine.MethodsA literature review was conducted, and a standardized method was used to extract data from publications, and critical decisions on visualization were made. A global geospatial database was developed of specific pfmdr1 and pfcrt single nucleotide polymorphisms and pfmdr1 copy number variation. An informatics framework was developed that allowed flexibility in development of the tool over time and efficient adaptation to different source data.ResultsThe database discussed in this paper has pfmdr1 and pfcrt marker prevalence information, from 579 geographic sites in 76 different countries, including results from over 86,000 samples from 456 articles published January 2001–May 2017. The ACT Partner Drugs Molecular Surveyor was launched by the WorldWide Antimalarial Resistance Network (WWARN) in March 2015 and it has attracted over 3000 unique visitors since then. Presented here is a demonstration of how the Surveyor database can be explored to monitor local, temporal changes in the prevalence of molecular markers. Here publications up to May 2017 were included, however the online ACT partner drug Molecular Surveyor is continuously updated with new data and relevant markers.ConclusionsThe WWARN ACT Partner Drugs Molecular Surveyor summarizes data on resistance markers in the pfmdr1 and pfcrt genes. The database is fully accessible, providing users with a rich resource to explore and analyze, and thus utilize a centralized, standardized database for different purposes. This open-source software framework can be adapted to other data, as demonstrated by the subsequent launch of the Artemisinin Molecular Surveyor and the Vivax Surveyor.

Using Lustre and Slurm to process Hadoop workloads and extending to the WLCG

The Queen Mary University of London Grid site has investigated the use of its Lustre file system to support Hadoop work flows. Lustre is an open source, POSIX compatible, clustered file system often used in high performance computing clusters and is often paired with the Slurm batch system. Hadoop is an open-source software framework for distributed storage and processing of data normally run on dedicated hardware utilising the HDFS file system and Yarn batch system. Hadoop is an important modern tool for data analytics used by a large range of organisation including CERN. By using our existing Lustre file system and Slurm batch system, the need to have dedicated hardware is removed and a single platform only has to be maintained for data storage and processing. The motivation and benefits of using Hadoop with Lustre and Slurm are presented. The installation, benchmarks, limitations and future plans are discussed. We also investigate using the standard WLCG Grid middleware Cream-CE service to provide a Grid enabled Hadoop service.

Benchmarking treewidth as a practical component of tensor network simulations.

Tensor networks are powerful factorization techniques which reduce resource requirements for numerically simulating principal quantum many-body systems and algorithms. The computational complexity of a tensor network simulation depends on the tensor ranks and the order in which they are contracted. Unfortunately, computing optimal contraction sequences (orderings) in general is known to be a computationally difficult (NP-complete) task. In 2005, Markov and Shi showed that optimal contraction sequences correspond to optimal (minimum width) tree decompositions of a tensor network’s line graph, relating the contraction sequence problem to a rich literature in structural graph theory. While treewidth-based methods have largely been ignored in favor of dataset-specific algorithms in the prior tensor networks literature, we demonstrate their practical relevance for problems arising from two distinct methods used in quantum simulation: multi-scale entanglement renormalization ansatz (MERA) datasets and quantum circuits generated by the quantum approximate optimization algorithm (QAOA). We exhibit multiple regimes where treewidth-based algorithms outperform domain-specific algorithms, while demonstrating that the optimal choice of algorithm has a complex dependence on the network density, expected contraction complexity, and user run time requirements. We further provide an open source software framework designed with an emphasis on accessibility and extendability, enabling replicable experimental evaluations and future exploration of competing methods by practitioners.

Fast and Robust Flight Altitude Estimation of Multirotor UAVs in Dynamic Unstructured Environments Using 3D Point Cloud Sensors

This paper presents a fast and robust approach for estimating the flight altitude of multirotor Unmanned Aerial Vehicles (UAVs) using 3D point cloud sensors in cluttered, unstructured, and dynamic indoor environments. The objective is to present a flight altitude estimation algorithm, replacing the conventional sensors such as laser altimeters, barometers, or accelerometers, which have several limitations when used individually. Our proposed algorithm includes two stages: in the first stage, a fast clustering of the measured 3D point cloud data is performed, along with the segmentation of the clustered data into horizontal planes. In the second stage, these segmented horizontal planes are mapped based on the vertical distance with respect to the point cloud sensor frame of reference, in order to provide a robust flight altitude estimation even in presence of several static as well as dynamic ground obstacles. We validate our approach using the IROS 2011 Kinect dataset available in the literature, estimating the altitude of the RGB-D camera using the provided 3D point clouds. We further validate our approach using a point cloud sensor on board a UAV, by means of several autonomous real flights, closing its altitude control loop using the flight altitude estimated by our proposed method, in presence of several different static as well as dynamic ground obstacles. In addition, the implementation of our approach has been integrated in our open-source software framework for aerial robotics called Aerostack.

FI-AR learning: a web-based platform for augmented reality educational content

The use of Augmented Reality (AR) offers great learning opportunities and challenges for educators. There are many examples of the application of AR to education. Furthermore, different libraries and frameworks have been proposed for the development of AR learning content, some of them based on the use of authoring tools. In this context, the present work introduces and describes FI-AR Learning, a web-based platform for the development of AR educational content, the distribution of which is accomplished through standard Learning Management Systems (LMS) by means of SCORM packages. Contrary to most existing solutions, the visualization of the AR learning content is performed without using plugins or specific applications. This is accomplished because FI-AR is based on the European open-source software framework FIWARE. Therefore, FI-AR uses standard web technologies, providing potential universal access. FIWARE also eases many non-educational management tasks, such as registration, security, marketing or billing, providing several important advantages to the FI-AR learning framework compared to other works. The paper describes the architecture and implementation of FI-AR, and several tests intended to measure its usability and performance. In addition, a specific example of application is presented, although the platform is designed to support AR educational content of any subject.

CasADi: a software framework for nonlinear optimization and optimal control

We present CasADi, an open-source software framework for numerical optimization. CasADi is a general-purpose tool that can be used to model and solve optimization problems with a large degree of flexibility, larger than what is associated with popular algebraic modeling languages such as AMPL, GAMS, JuMP or Pyomo. Of special interest are problems constrained by differential equations, i.e. optimal control problems. CasADi is written in self-contained C++, but is most conveniently used via full-featured interfaces to Python, MATLAB or Octave. Since its inception in late 2009, it has been used successfully for academic teaching as well as in applications from multiple fields, including process control, robotics and aerospace. This article gives an up-to-date and accessible introduction to the CasADi framework, which has undergone numerous design improvements over the last 7 years.

Multimodal microscopy test standard for scanning microwave, electron, force and optical microscopy

We report on measurement results of a test standard suitable for different microscopic modalities. These findings were obtained by a multimodal hybrid microscope, which requires various calibration methods, also in terms of its further use as a tool in a nanorobotic environment. A Scanning Probe Microscopy (SPM)-Controller based on an FPGA (Field Programmable Gate Array) enables the submicrometer imaging for atomic force and microwave microscopic modalities. It is embedded in an open source software framework for nanorobotics and -automation and is described in this report.

Allpix2: A modular simulation framework for silicon detectors

Allpix2 (read: Allpix Squared) is a generic, open-source software framework for the simulation of silicon pixel detectors. Its goal is to ease the implementation of detailed simulations for both single detectors and more complex setups such as beam telescopes from incident radiation to the digitised detector response. Predefined detector types can be automatically constructed from simple model files describing the detector parameters.The simulation chain is arranged with the help of intuitive configuration files and an extensible system of modules, which implement separate simulation steps such as realistic charge carrier deposition with the Geant4 toolkit or propagation of charge carriers in silicon using a drift–diffusion model. Detailed electric field maps imported from TCAD simulations can be used to precisely model the drift behaviour of charge carriers within the silicon, bringing a new level of realism to Monte Carlo based simulations of particle detectors.This paper provides an overview of the framework and a selection of different simulation modules, and presents a comparison of simulation results with test beam data recorded with hybrid pixel detectors. Emphasis is placed on the performance of the framework itself, using a first-principles simulation of the detectors without addressing secondary ASIC-specific effects.

Robust Insider Attacks Countermeasure for Hadoop: Design and Implementation

Hadoop is an open source software framework for storage and processing of large-scale datasets. The proliferation of cloud services and its corresponding increasing number of users lead to a larger attack surface, especially for internal threats. Therefore, in corporate data centers, it is essential to ensure the security, authenticity, and integrity of all the entities of Hadoop. The current secure implementations of Hadoop mainly utilize Kerberos, which is known to suffer from many security and performance issues, including the concentration of authentication credentials, single point of failure, and online availability. Most importantly, these Kerberos-based implementations do not guard against insider threats. In this paper, we propose an authentication framework for Hadoop that utilizes trusted platform module technology. The proposed approach provides significant security guarantees against insider threats, which manipulate the execution environment without the consent of legitimate clients. We have conducted extensive experiments to validate the performance and the security properties of our approach. The results demonstrate that the proposed approach alleviates many of the shortcomings of Kerberos-based state-of-the-art protocols and provides unique security guarantees with acceptable overhead. Moreover, we have formally proved the correctness and the security guarantees of our protocol via Burrows–Abadi–Needham logic.

Nontargeted Identification of Tracer Incorporation in High-Resolution Mass Spectrometry.

"Fluxomics" refers to the systematic analysis of metabolic fluxes in a biological system and may uncover novel dynamic properties of metabolism that remain undetected in conventional metabolomic approaches. In labeling experiments, tracer molecules are used to track changes in the isotopologue distribution of metabolites, which allows one to estimate fluxes in the metabolic network. Because unidentified compounds cannot be mapped on pathways, they are often neglected in labeling experiments. However, using recent developments in de novo annotation may allow to harvest the information present in these compounds if they can be identified. Here, we present a novel tool (HiResTEC) to detect tracer incorporation in high-resolution mass spectrometry data sets. The software automatically extracts a comprehensive, nonredundant list of all compounds showing more than 1% tracer incorporation in a nontargeted fashion. We explain and show in an example data set how mass precision and other filter heuristics, calculated on the raw data, can efficiently be used to reduce redundancy and noninformative signals by 95%. Ultimately, this allows to quickly investigate any labeling experiment for a complete set of labeled compounds (here 149) with acceptable false positive rates. We further re-evaluate a published data set from liquid chromatography-electrospray ionization (LC-ESI) to demonstrate broad applicability of our tool and emphasize importance of quality control (QC) tests. HiResTEC is provided as a package in the open source software framework R and is freely available on CRAN.

TASI: A software tool for spatial-temporal quantification of tumor spheroid dynamics

Spheroid cultures derived from explanted cancer specimens are an increasingly utilized resource for studying complex biological processes like tumor cell invasion and metastasis, representing an important bridge between the simplicity and practicality of 2-dimensional monolayer cultures and the complexity and realism of in vivo animal models. Temporal imaging of spheroids can capture the dynamics of cell behaviors and microenvironments, and when combined with quantitative image analysis methods, enables deep interrogation of biological mechanisms. This paper presents a comprehensive open-source software framework for Temporal Analysis of Spheroid Imaging (TASI) that allows investigators to objectively characterize spheroid growth and invasion dynamics. TASI performs spatiotemporal segmentation of spheroid cultures, extraction of features describing spheroid morpho-phenotypes, mathematical modeling of spheroid dynamics, and statistical comparisons of experimental conditions. We demonstrate the utility of this tool in an analysis of non-small cell lung cancer spheroids that exhibit variability in metastatic and proliferative behaviors.

SaLEM (v1.0) – the Soil and Landscape Evolution Model (SaLEM) for simulation of regolith depth in periglacial environments

Abstract. We propose the implementation of the Soil and Landscape Evolution Model (SaLEM) for the spatiotemporal investigation of soil parent material evolution following a lithologically differentiated approach. Relevant parts of the established Geomorphic/Orogenic Landscape Evolution Model (GOLEM) have been adapted for an operational Geographical Information System (GIS) tool within the open-source software framework System for Automated Geoscientific Analyses (SAGA), thus taking advantage of SAGA's capabilities for geomorphometric analyses. The model is driven by palaeoclimatic data (temperature, precipitation) representative of periglacial areas in northern Germany over the last 50 000 years. The initial conditions have been determined for a test site by a digital terrain model and a geological model. Weathering, erosion and transport functions are calibrated using extrinsic (climatic) and intrinsic (lithologic) parameter data. First results indicate that our differentiated SaLEM approach shows some evidence for the spatiotemporal prediction of important soil parental material properties (particularly its depth). Future research will focus on the validation of the results against field data, and the influence of discrete events (mass movements, floods) on soil parent material formation has to be evaluated.

CS::APEX: A Framework for Algorithm Prototyping and Experimentation with Robotic Systems

Robotic systems differ drastically in their sensory capabilities, their computational power and their designated tasks. For efficient algorithm development, however, we need to have a common modeling framework that enables us to generalize and re-use existing solutions. A modular approach, which is coherent across different platforms, also allows faster prototyping of new systems, given that existing functionality can be reused from already implemented modules. In this paper we develop a modeling framework based on data flow graphs that achieves the following goal: We first merge synchronous data flow and reactive programming into hybrid flow graphs, where we explicitly model synchronous and asynchronous data flow. Then we transfer concepts from finite-state machines to achieve a coherent framework which we call Activity Flow Graphs. The flow of activity enables us to model high level states directly in the data flow graph. The result is a single computation graph that can express both perception and high level control aspects of any robotic system. This theoretical foundation is the core of our open-source software framework CS::APEX, which allows the creation, manipulation and evaluation of Activity Flow Graphs and enables rapid prototyping and experimentation and can be used with any robot supporting the Robot Operating System (ROS). We then demonstrate the framework with two high level models for a fetch-and-delivery robot and a person following robot.

Implementation of cost effective hierarchical Hadoop cluster–a case study for education

To equip the younger generation of the province with computing skills and provide them access to a wide variety of modern educational resources such as multimedia based on educational content in government schools and colleges to set a strong foundation at an early stage of their education. Educational Media help to empower educational institutions by altering the way of using Information Communication Technology (ICT). To put this into practice, several challenges need to be addressed. It requires a scalable technological architecture and algorithms to form a cluster in which effective resource sharing like CPU, Memory among multiple nodes, tools to monitor, assess and evaluate data under hierarchical Hadoop cluster is needed. By analyzing text, audios, videos information, Periodic reports will be generated to assist the students, teachers and Government. In this case, a software framework is required to process big data stored in hierarchical nodes. As the architecture of Hadoop, an open source software framework doesn't support processing the data stored in hierarchical nodes. This case study proposes the Hierarchical Hadoop cluster to alter the way of using ICT. The proposed work helps in monitoring, reporting the usage of ICT and also acts a help desk to address the issues of the educational institutions. This establishes a novel communication media by generating reports on text, audio, video information based on analysis.

Design and Implementation of Data-at-Rest Encryption for Hadoop

The manuscript should contain an abstract. The security aspects in Cloud computing is paramount in order to ensure high quality of Service Level Agreement (SLA) to the cloud computing customers. This issue is more apparent when very large amount of data is involved in this emerging computing environment. Hadoop is an open source software framework that supports large data sets storage and processing in a distributed computing environment and well-known implementation of Map Reduce. Map Reduce is one common programming model to process and handle a large amount of data, specifically in big data analysis. Further, Hadoop Distributed File System (HDFS) is a distributed, scalable and portable file system that is written in java for Hadoop framework. However, the main problem is that the data at rest is not secure where intruders can steal or converts the data stored in this computing environment. Therefore, the AES encryption algorithm has been implemented in HDFS to ensure the security of data stored in HDFS. It is shown that the implementation of AES encryption algorithm is capable to secure data stored in HDFS to some extent.