General-purpose Software Framework Research Articles

FDup: a framework for general-purpose and efficient entity deduplication of record collections.

Deduplication is a technique aiming at identifying and resolving duplicate metadata records in a collection. This article describes FDup (Flat Collections Deduper), a general-purpose software framework supporting a complete deduplication workflow to manage big data record collections: metadata record data model definition, identification of candidate duplicates, identification of duplicates. FDup brings two main innovations: first, it delivers a full deduplication framework in a single easy-to-use software package based on Apache Spark Hadoop framework, where developers can customize the optimal and parallel workflow steps of blocking, sliding windows, and similarity matching function via an intuitive configuration file; second, it introduces a novel approach to improve performance, beyond the known techniques of “blocking” and “sliding window”, by introducing a smart similarity matching function T-match. T-match is engineered as a decision tree that drives the comparisons of the fields of two records as branches of predicates and allows for successful or unsuccessful early-exit strategies. The efficacy of the approach is proved by experiments performed over big data collections of metadata records in the OpenAIRE Research Graph, a known open access knowledge base in Scholarly communication.

Unified Software Platform for Intelligent Home Service Robots

Although the mobile manipulation capability is crucial for a service robot to perform physical work without human support, the long-term autonomous operation of such a mobile manipulation robot in a real environment is still a tremendously difficult task. In this paper, we present a modular, general purpose software framework for intelligent mobile manipulation robots that can interact with humans using complex human speech commands; navigate smoothly in tight indoor spaces; and finally detect and manipulate various household objects and pieces of furniture autonomously. The suggested software framework is designed to be easily transferred to different home service robots, which include the Toyota Human Support Robot (HSR) and our Modular Service Robot-1 (MSR-1) platforms. It has successfully been used to solve various home service tasks at the RoboCup@Home and World Robot Summit international service robot competitions with promising results.

Augmenting Operating Systems with OpenCL Accelerators

Heterogeneous computing leverages more than one kind of processors to boost the performance of user-space applications with the heterogeneous programming languages, e.g., OpenCL. While some works have been done to accelerate the computations required by Linux kernel software, they are either application-specific solutions or tightly coupled with the certain computing platforms and are not able to support the general-purpose in-kernel accelerations using different types of processors. In this article, the general-purpose software framework called Kernel acceleration with OpenCL (KOCL), is proposed to tackle the problem. KOCL exposes a set of the high-level programming interfaces for the Linux kernel module developers to offload compute-intensive tasks on different hardware accelerators without managing and coordinating the platform-specific computing and memory resources. The simplified programming efforts are achieved by the developed platform management and memory models, which provide a systematic means of managing the heterogeneous hardware resources. In addition, the one- and zero-copy data-buffering schemes are offered by KOCL, so that the offloaded tasks deliver high performance on the platforms with different memory architectures. We have developed the prototype system to accelerate the Network-Attached Storage server applications. Significant performance improvements are achieved with the three different types of accelerators, i.e., the multicore processor, the integrated GPU, and the discrete GPU, respectively. We believe that KOCL is useful for the design of embedded appliances to evaluate the performance of design alternatives.

DictyExpress: a web-based platform for sequence data management and analytics in Dictyostelium and beyond

BackgroundDictyostelium discoideum, a soil-dwelling social amoeba, is a model for the study of numerous biological processes. Research in the field has benefited mightily from the adoption of next-generation sequencing for genomics and transcriptomics. Dictyostelium biologists now face the widespread challenges of analyzing and exploring high dimensional data sets to generate hypotheses and discovering novel insights.ResultsWe present dictyExpress (2.0), a web application designed for exploratory analysis of gene expression data, as well as data from related experiments such as Chromatin Immunoprecipitation sequencing (ChIP-Seq). The application features visualization modules that include time course expression profiles, clustering, gene ontology enrichment analysis, differential expression analysis and comparison of experiments. All visualizations are interactive and interconnected, such that the selection of genes in one module propagates instantly to visualizations in other modules. dictyExpress currently stores the data from over 800 Dictyostelium experiments and is embedded within a general-purpose software framework for management of next-generation sequencing data. dictyExpress allows users to explore their data in a broader context by reciprocal linking with dictyBase—a repository of Dictyostelium genomic data. In addition, we introduce a companion application called GenBoard, an intuitive graphic user interface for data management and bioinformatics analysis.ConclusionsdictyExpress and GenBoard enable broad adoption of next generation sequencing based inquiries by the Dictyostelium research community. Labs without the means to undertake deep sequencing projects can mine the data available to the public. The entire information flow, from raw sequence data to hypothesis testing, can be accomplished in an efficient workspace. The software framework is generalizable and represents a useful approach for any research community. To encourage more wide usage, the backend is open-source, available for extension and further development by bioinformaticians and data scientists.

A General-Purpose Microcontroller-Based Framework for Integrating Oceanographic Sensors, Instruments, and Peripherals

AbstractSensors and instruments for basic oceanographic properties are becoming increasingly sophisticated, which both simplifies and complicates their use in field studies. This increased sophistication disproportionately affects smaller-scale observational efforts that are less likely to be well supported technically but which need to integrate instruments, sensors, and commonly needed peripheral devices in ways not envisioned by their manufacturers. A general-purpose hardware and software framework was developed around a widely used family of low-power microcontrollers to lessen the technical expertise and customization required to integrate sensors, instruments, and peripherals, and thus simplify such integration scenarios. Both the hardware and associated firmware development tools provide a range of features often required in such scenarios: serial data interfaces, analog inputs and outputs, logic lines and power-switching capability, nonvolatile storage of data and parameters for sampling or configuration, and serial communication interfaces to supervisory or telemetry systems. The microcontroller and additional components needed to implement this integration framework are small enough to encapsulate in standard cable splices, creating a small form factor “smart cable” that can be readily wired and programmed for a range of integration needs. An application programming library developed for this hardware provides skeleton code for functions commonly desired when integrating sensors, instruments, and peripherals. This minimizes the firmware programming expertise needed to apply this framework in many integration scenarios and thus streamlines the development of firmware for different field applications. Envisioned applications are in field programs where significant technical instrumentation expertise is unavailable or not cost effective.

ParadisEO-MO: from fitness landscape analysis to efficient local search algorithms

This paper presents a general-purpose software framework dedicated to the design, the analysis and the implementation of local search metaheuristics: ParadisEO-MO. A substantial number of single solution-based local search metaheuristics has been proposed so far, and an attempt of unifying existing approaches is here presented. Based on a fine-grained decomposition, a conceptual model is proposed and is validated by regarding a number of state-of-the-art methodologies as simple variants of the same structure. This model is then incorporated into the ParadisEO-MO software framework. This framework has proven its efficiency and high flexibility by enabling the resolution of many academic and real-world optimization problems from science and industry.

A Flexible Software Framework for Transportation Maintenance Scheduling

This paper presents a general-purpose software framework dedicated to the evolutionary transportation scheduling, which is one of the most challenging areas in transportation engineering. Against evolution, the most popular approach is manual process. Manual process in here is to describe the system business using the most appropriate modeling language(s) at the most appropriate level(s) of abstraction. To demonstrate the applicability of the framework, we report our experience in using the software to generate pavement maintenance report under manual business model.

A software framework based on a conceptual unified model for evolutionary multiobjective optimization: ParadisEO-MOEO

This paper presents a general-purpose software framework dedicated to the design and the implementation of evolutionary multiobjective optimization techniques: ParadisEO-MOEO. A concise overview of evolutionary algorithms for multiobjective optimization is given. A substantial number of methods has been proposed so far, and an attempt of conceptually unifying existing approaches is presented here. Based on a fine-grained decomposition and following the main issues of fitness assignment, diversity preservation and elitism, a conceptual model is proposed and is validated by regarding a number of state-of-the-art algorithms as simple variants of the same structure. This model is then incorporated into the ParadisEO-MOEO software framework. This framework has proven its validity and high flexibility by enabling the resolution of many academic, real-world and hard multiobjective optimization problems.

Dataflow Engine in DAQ Backbone DABC

New experiments at FAIR require new concepts of data acquisition systems. So rather than building hardware trigger configurations with strict latency limitations, self-triggered electronic systems will be utilized. Their front-end components will be time synchronized and will provide data furnished with time stamps. Data streams from a multitude of such components will be forwarded via a powerful sorting network to an event building farm. The Data Acquisition Backbone Core (DABC) is designed as a general purpose software framework for the implementation of such data acquisition systems. It is written in C++. Recently, we made its first version available. In this paper we describe the mechanisms of the data flow engine of DABC.

Data Acquisition Backbone Core DABC release v1.0

The Data Acquisition Backbone Core (DABC) is a general purpose software framework designed for the implementation of a wide-range of data acquisition systems – from various small detector test beds to high performance systems. DABC consists of a compact data-flow kernel and a number of plug-ins for various functional components like data inputs, device drivers, user functional modules and applications. DABC provides configurable components for implementing event building over fast networks like InfiniBand or Gigabit Ethernet. A generic Java GUI provides the dynamic control and visualization of control parameters and commands, provided by DIM servers. A first set of application plug-ins has been implemented to use DABC as event builder for the front-end components of the GSI standard DAQ system MBS (Multi Branch System). Another application covers the connection to DAQ readout chains from detector front-end boards (N-XYTER) linked to read-out controller boards (ROC) over UDP into DABC for event building, archiving and data serving. This was applied for data taking in the September 2008 test beamtime for the CBM experiment at GSI. DABC version 1.0 is released and available from the website.

General-purpose modular hardware and software framework for mobile outdoor augmented reality applications in engineering

This paper presents a reusable, general-purpose, mobile augmented reality (AR) framework developed to address the critical and repetitive challenges specific to visualization in outdoor AR. In all engineering applications of AR developed thus far, basic functionality that supports accurate user registration, maximizes the range of user motion, and enables data input and output has had to be repeatedly re-implemented. This is primarily due to the fact that designed methods have been traditionally custom created for their respective applications and are not generic enough to be readily shared and reused by others. The objective of this research was to remedy this situation by designing and implementing a reusable and pluggable hardware and software framework that can be used in any AR application without the need to re-implement low-level communication interfaces with selected hardware. The underlying methods of hardware communication as well as the object-oriented design (OOD) of the reusable interface are presented. Details on the validation of framework reusability and pluggability are also described.

Advances in mathematical modeling of biofilm structure

Mathematical modeling of spatial biofilm structure has been in development for the past 10 years, its main goal being to derive the dynamics of biofilm structure from first-principle descriptions of the various physical, chemical and biological processes involved in biofilm formation. Early efforts described development of unrestricted monospecies consortia, often considering diffusion and reaction of a single solute species. Multi-dimensional modeling of biofilms has presently reached a stage where multi-species systems with any number of bacterial and solute species, reactions and arbitrary detachment scenarios may be readily implemented using a general-purpose software framework introduced recently. The present work presents motivations for the mathematical modeling of biofilm structure and provides an overview on major contributions to this field from pioneering efforts using cellular automata (CA) to more recent methods using the preferred individual-based modeling (IbM). Recent examples illustrate how biofilm models can be used to study the microbial ecology in: (a) development of multi-species nitrifying biofilms with anammox bacteria, (b) interspecies hydrogen transfer in anaerobic digestion methanogenic consortia, (c) competition between flock-formers and filamentous bacteria influenced by environmental conditions and its effect on morphology of activated sludge flocs, and (d) a two-species biofilm system with structured biomass describing extracellular polymeric substances (EPS) and internal storage compounds. As recent efforts from direct comparison of structure predicted by three-dimensional modeling with that observed by confocal laser scanning microscopy imaging of biofilms grown in laboratory flow cells show a good agreement of predicted structures, multi-dimensional modeling approaches presently constitute a mature and established methodology to enhance our understanding of biofilm systems.

Role-Based Access Controls: Status, Dissemination, and Prospects for Generic Security Mechanisms

E-commerce applications have diverse security requirements ranging from business-to-business over business-to-consumer to consumer-to-consumer types of applications. This range of requirements cannot be handled adequately by one single security model although role-based access controls (RBAC) depict a promising fundament for generic high-level security. Furthermore, RBAC is well researched but rather incompletely realized in most of the current backend as well as business layer systems. Security mechanisms have often been added to existing software causing many of the well-known deficiencies found in most software products. However, with the rise of component-based software development security models can also be made available for reuse. Therefore, we present a general-purpose software framework providing security mechanisms such as authentication, access controls, and auditing for Java software development. The framework is called GAMMA (Generic Authorization Mechanisms for Multi-Tier Applications) and offers multiple high-level security models (including the aforementioned RBAC) that may even be used concurrently to cover such diverse security requirements as found within e-commerce environments.