Distributed Software Components Research Articles

Data Fabric and Datafication

This paper is a response to the invitation by Forti et al. [1] to contribute to the discussion of trending topics in software engineering. The main focus of this contribution is the growing importance of data in software engineering, that is the data fabric and the datafication. Based on some (personal) observations, it is tried to refine the research questions raised by Forti et al. in more detail, in particular with respect to sharing and processing data in an environment of highly interacting distributed software components.

Integration of Autonomous Vehicle Frameworks for Software-in-the-Loop Testing

<div class="section abstract"><div class="htmlview paragraph">This paper presents an approach for performing software in the loop testing of autonomous vehicle software developed in the Autoware framework. Autoware is an open source software for autonomous driving that includes modules such as localization, detection, prediction, planning and control [<span class="xref">8</span>]. Multitudes of autonomous driving frameworks exist today, each having its own pros and cons. Often, MATLAB-Simulink is used for rapid prototyping, system modeling and testing, specifically for the lower-level vehicle dynamics and powertrain control features. For the autonomous software, the Robotic Operating System (ROS) is more commonly used for integrating distributed software components so that they can easily share information through a publish and subscribe paradigm. Thorough testing and evaluation of such complex, distributed software, implemented on a physical vehicle poses significant challenges in terms of safety, time, and cost, especially when considering rare edge cases. Virtual prototyping is therefore a crucial enabler in the development of autonomous software. In a simulated environment, many traffic scenarios under a variety of environmental conditions can be quickly evaluated, at low cost, without safety concerns. In this paper, we report on a simulation environment consisting of three simulation tools. PreScan (by Siemens/TASS) combined with Simulink (by Mathworks) is used for simulating how the vehicle interacts with the environment: sensors, actuators, the vehicle dynamics and powertrain. The autonomy software is emulated directly in Autoware.AI on top of ROS. To evaluate the autonomous software, synthetic data from the sensors simulated in PreScan are published to ROS where they are processed by the autonomy stack. Similarly, the control signals generated by the autonomy stack in Autoware are subscribed to by PreScan where they serve as input to the virtual vehicle model. The paper describes in detail the integration of PreScan and Autoware, illustrates this integration for object detection using a monocular camera, and characterizes the performance in terms of message transfer speed.</div></div>

Shared Memory Transport for ALFA

The high data rates expected for the next generation of particle physics experiments (e.g.: new experiments at FAIR/GSI and the upgrade of CERN experiments) call for dedicated attention with respect to design of the needed computing infrastructure. The common ALICE-FAIR framework ALFA is a modern software layer, that serves as a platform for simulation, reconstruction and analysis of particle physics experiments. Beside standard services needed for simulation and reconstruction of particle physics experiments, ALFA also provides tools for data transport, configuration and deployment. The FairMQ module in ALFA offers building blocks for creating distributed software components (processes) that communicate between each other via message passing.The abstract "message passing" interface in FairMQ has at the moment three implementations: ZeroMQ, nanomsg and shared memory. The newly developed shared memory transport will be presented, that provides significant per-formance benefits for transferring large data chunks between components on the same node. The implementation in FairMQ allows users to switch between the different transports via a trivial configuration change. The design decisions, im-plementation details and performance numbers of the shared memory transport in FairMQ/ALFA will be highlighted.

Obtaining and Managing Answer Quality for Online Data-Intensive Services

Online data-intensive (OLDI) services use anytime algorithms to compute over large amounts of data and respond quickly. Interactive response times are a priority, so OLDI services parallelize query execution across distributed software components and return best effort answers based on the data so far processed. Omitted data from slow components could lead to better answers, but tracing online how much better the answers could be is difficult. We propose Ubora, a design approach to measure the effect of slow-running components on the quality of answers. Ubora randomly samples online queries and executes them a second time. The first online execution omits data from slow components and provides interactive answers. The second execution uses mature results from intermediate components completed after the online execution finishes. Ubora uses memoization to speed up mature executions by replaying network messages exchanged between components. Our systems-level implementation works for a wide range of services, including Hadoop/Yarn, Apache Lucene, the EasyRec Recommendation Engine, and the OpenEphyra question-answering system. Ubora computes answer quality with more mature executions per second than competing approaches that do not use memoization. With Ubora, we show that answer quality is effective at guiding online admission control. While achieving the same answer quality on high-priority queries, our adaptive controller had 55% higher peak throughput on low-priority queries than a competing controller guided by the rate of timeouts.

Scalable Component-Based Manzai Robots as Automated Funny Content Generators

[abstFig src='/00280006/10.jpg' width='300' text='Tabletop component-based Manzai robots' ] This manuscript describes a scalable tabletop Manzai robot system that has been developed using distributed software components. Manzai is a style of traditional Japanese stand-up comedy that is typically performed by two comedians – a stooge and a straight man. Manzai script refers to the dialogues exchanged between the two comedians. Manzai robots automatically generate their Manzai scripts from web news articles based on keywords provided by audiences and search results on the Internet. Then, the robots perform according to these Manzai scripts. This study focuses on the flexibility and scalability of a robot system based on distributed Robot Technology (RT) components. The results of the implementation experiments demonstrate the flexibility of the Manzai performing robots and the scalability of the functions of the robot system.

Failure Diagnosis for Distributed Systems using Targeted Fault Injection

This paper introduces a novel approach to automating failure diagnostics in distributed systems by combining fault injection and data analytics. We use fault injection to populate the database of failures for a target distributed system. When a failure is reported from production environment, the database is queried to find “matched” failures generated by fault injections. Relying on the assumption that similar faults generate similar failures, we use information from the matched failures as hints to locate the actual root cause of the reported failures. In order to implement this approach, we introduce techniques for (i) reconstructing end-to-end execution flows of distributed software components, (ii) computing the similarity of the reconstructed flows, and (iii) performing precise fault injection at pre-specified executing points in distributed systems. We have evaluated our approach using an OpenStack cloud platform, a popular cloud infrastructure management system. Our experimental results showed that this approach is effective in determining the root causes, e.g., fault types and affected components, for 71-100 percent of tested failures. Furthermore, it can provide fault locations close to actual ones and can easily be used to find and fix actual root causes. We have also validated this technique by localizing real bugs that occurred in OpenStack.

Search GenBank: interactive orchestration and ad-hoc choreography of Web services in the exploration of the biomedical resources of the National Center For Biotechnology Information

BackgroundDue to the growing number of biomedical entries in data repositories of the National Center for Biotechnology Information (NCBI), it is difficult to collect, manage and process all of these entries in one place by third-party software developers without significant investment in hardware and software infrastructure, its maintenance and administration. Web services allow development of software applications that integrate in one place the functionality and processing logic of distributed software components, without integrating the components themselves and without integrating the resources to which they have access. This is achieved by appropriate orchestration or choreography of available Web services and their shared functions. After the successful application of Web services in the business sector, this technology can now be used to build composite software tools that are oriented towards biomedical data processing.ResultsWe have developed a new tool for efficient and dynamic data exploration in GenBank and other NCBI databases. A dedicated search GenBank system makes use of NCBI Web services and a package of Entrez Programming Utilities (eUtils) in order to provide extended searching capabilities in NCBI data repositories. In search GenBank users can use one of the three exploration paths: simple data searching based on the specified user’s query, advanced data searching based on the specified user’s query, and advanced data exploration with the use of macros. search GenBank orchestrates calls of particular tools available through the NCBI Web service providing requested functionality, while users interactively browse selected records in search GenBank and traverse between NCBI databases using available links. On the other hand, by building macros in the advanced data exploration mode, users create choreographies of eUtils calls, which can lead to the automatic discovery of related data in the specified databases.Conclusionssearch GenBank extends standard capabilities of the NCBI Entrez search engine in querying biomedical databases. The possibility of creating and saving macros in the search GenBank is a unique feature and has a great potential. The potential will further grow in the future with the increasing density of networks of relationships between data stored in particular databases. search GenBank is available for public use at http://sgb.biotools.pl/.

Distributed control for the networks of adaptive software components

As a result of technological advances, a typical type of software systems has emerged. A large number of distributed software components are networked together through a task flow structure, and each component may have alternative algorithms among which it can choose to process tasks. However, the increased complexity and vulnerability to adverse events of such systems give rise to the need for more sophisticated yet scalable control mechanisms. In this study a control mechanism is designed to meet the need. First, stress environments are implicitly modeled by quantifying the resource availability of the system through sensors. Second, a mathematical programming model is built with the resource availability incorporated and with the stability in system behavior assured. Third, a multi-tier auction market is designed to solve the programming model by distributing computation and communication overheads. By periodically opening the auction market, the system can achieve desirable performance adaptively to changing stress environment while assuring stability and scalability properties. The control mechanism devised in this paper contributes to the efforts of managing the ever-increasing complexity of modern software systems.

Mixing Workflows and Components to Support Evolving Services

Composite distributed services involve local and remote services that get orchestrated according to specific business logic. This logic can be programmed by applying a traditional general-purpose programming language, but is generally described using a workflow language that coordinates a set of given services. The services involved in the composition, or the composition may need to evolve both at the business logic level (workflow level) and the global architecture level. This paper presents a solution to ease such evolution for compound distributed services and the authors’ proposal enables the evolution of both the business logic and the underlying architecture. This paper suggests relying on a distributed software component model to represent and easily manage the set of local or remote software entities (services) involved in the composition. Composite services are represented in a model that combines the use of a distributed and hierarchical software component model and new timed-automata based workflow language. This combination makes explicit the separation between functional and non-functional concerns, and as a consequence this approach helps in defining the required and various evolution procedures in context to compound services.

DDDAS-based multi-fidelity simulation framework for supply chain systems

Dynamic-Data-Driven Application Systems (DDDAS) is a new modeling and control paradigm which adaptively adjusts the fidelity of a simulation model. The fidelity of the simulation model is adjusted against available computational resources by incorporating dynamic data into the executing model, which then steers the measurement process for selective date update. To this end, comprehensive system architecture and methodologies are first proposed, where the components include a real-time DDDAS simulation, grid modules, a web service communication server, databases, various sensors and a real system. Abnormality detection, fidelity selection, fidelity assignment, and prediction and task generation are enabled through the embedded algorithms developed in this work. Grid computing is used for computational resources management and web services are used for inter-operable communications among distributed software components. The proposed DDDAS is demonstrated on an example of preventive maintenance scheduling in a semiconductor supply chain.

XenLoop: a transparent high performance inter-VM network loopback

Advances in virtualization technology have focused mainly on strengthening the isolation barrier between virtual machines (VMs) that are co-resident within a single physical machine. At the same time, a large category of communication intensive distributed applications and software components exist, such as web services, high performance grid applications, transaction processing, and graphics rendering, that often wish to communicate across this isolation barrier with other endpoints on co-resident VMs. State of the art inter-VM communication mechanisms do not adequately address the requirements of such applications. TCP/UDP based network communication tends to perform poorly when used between co-resident VMs, but has the advantage of being transparent to user applications. Other solutions exploit inter-domain shared memory mechanisms to improve communication latency and bandwidth, but require applications or user libraries to be rewritten against customized APIs--something not practical for a large majority of distributed applications. In this paper, we present the design and implementation of a fully transparent and high performance inter-VM network loopback channel, called XenLoop, in the Xen virtual machine environment. XenLoop does not sacrifice user-level transparency and yet achieves high communication performance between co-resident guest VMs. XenLoop intercepts outgoing network packets beneath the network layer and shepherds the packets destined to co-resident VMs through a high-speed inter-VM shared memory channel that bypasses the virtualized network interface. Guest VMs using XenLoop can migrate transparently across machines without disrupting ongoing network communications, and seamlessly switch between the standard network path and the XenLoop channel. In our evaluation using a number of unmodified benchmarks, we observe that XenLoop can reduce the inter-VM round trip latency by up to a factor of 5 and increase bandwidth by a up to a factor of 6.

Galapagos: Model-driven discovery of end-to-end application-storage relationships in distributed systems

Modern business information systems are typically multitiered distributed systems comprising Web services, application services, databases, enterprise information systems, file systems, storage controllers, and other storage systems. In such environments, data is stored in different forms at multiple tiers, with each tier associated with some level of data abstraction. An information entity owned by an application generally maps to several data entities, logically associated across tiers and related to the application. Discovery of such relationships in a distributed system is a challenging problem, complicated by the widespread adoption of virtualization technologies and by the traditional tendency to manage each tier as an independent domain. In this paper, we present a system and methodology for model-driven discovery of end-to-end application-data relationships spanning multiple tiers, from the applications to the lowest levels of the storage hierarchy. The key to our methodology involves modeling how data is used and transformed by distributed software components. An important benefit of our system, which we call Galapagos, is the ability to reflect business decisions expressed at the application level to the level of storage.

AqquaScan: design and implementation of an Internet-based service for the remote monitoring and management of decentralised WWTPs

This paper describes the design and implementation of AqquaScan, an Internet-based service for remote monitoring and integrated management of decentralised WWTPs. AqquaScan is a multi-user and multi-WWTP service. It has been built according to criteria such as flexibility, scalability and interoperability with the idea of providing an open environment suited to quickly accommodate future scenarios (e.g. incorporation of new plants or upgrading of existing installations). Both, the management of plant information and users interfaces have been implemented in distributed software components that communicate with one another via web services. The implemented web services can be exploited to develop customised user interfaces for visualising the monitored data. By default, a customised web-based client module has been programmed in order for users to be able to exploit the facilities offered within AqquaScan: (1) real-time monitoring of on-line signals; (2) visualisation of historical data; (3) changing operational parameters; (4) notification of time-event information; and (5) storage of measurements from laboratory analysis. At present, AqquaScan is fully operative and is offering supervision services to eleven industrial WWTPs distributed around Northern Spain.

Component Contracts in Service-Oriented Architectures

For SOAs to reach their full potential, the basic interoperable framework must accommodate meaningful quality-of-service contracts. Work on both industry-specific standards and semantic Web services is still needed to fully meet that goal. At the core of service-oriented architectures (SOAs) are distributed software components provided or accessed by independent third parties. Because access is not limited to a specific organization, explicit component contracts and universally adopted standards must support third-party access. Although such contracts could cover any technical or business aspect of service interaction, the current focus is on quality-of-service (QoS) policies. From an SOA point of view, we must consider two separate aspects of the use of QoS policies: interoperability between components, which is the subject of the Web services specifications stack; and composition, which composition models, such as the service component architecture (SCA).

Enhancing automated process design with cognitive agents, distributed software components and web repositories

A novel problem-solving environment for automated process design, which integrates cognitive agents, web repositories and distributed software components with an existing automated process design tool, is presented. It is shown how the approach offers the potential for significant improvements in two key aspects of design, namely, problem formulation and innovation. Note that by improving the quality of the solutions space, the approach directly supports optimization.

An architecture for an Internet Marketing Multi-Agent System using AUML

An Internet Marketing Multi-Agent System (IMMAS) has been proposed to enhance traditional Internet marketing mechanism through the mobile agent technique [3]. A critical design of the proposed system is the introduction of two software agents, namely a Sales agent and a Personal agent, for performing the delegated jobs of the designated marketing activity helping customers to assist his own job administration respectively. This paper is aimed to report the architecture for IMMAS represented in form of Agent UML (AUML) specifications. The strength of a multi-agent architecture is highly dependent on the coordination of multiple distributed software components. Therefore, the description of IMMAS architecture will be focused on the coordination and interaction patterns among multiple agents in a three-layer presentation. This architecture could be helpful to and used as the reference for later software engineers in agent-based software systems.

Extending the OBJECTIVE Benchmark for Evaluation of Active Rules in a Distributed Component Integration Environment

This research presents a benchmark for the evaluation of active rules within a distributed component integration environment. The benchmark is based on the OBJECTIVE active database benchmark. Our work was specifically performed in the context of the Integration Rules (IRules) project that uses active rules for the integration of distributed software components, which are based on the Enterprise JavaBeans component model. In addition to mapping the test rules of the OBJECTIVE benchmark to the integration rules of the IRules system, we have extended the OBJECTIVE benchmark to test additional features of a component-based active rule system, such as the event detection time, the rule execution time related to the invocation of methods on components, asynchronous coupling modes, and the rule processor execution time under a heavy event load. Although the research presents specific performance evaluation results for the IRules project, the approach more generally demonstrates the manner in which an existing rule processing benchmark for a centralized environment can be adapted and extended for use in a rule-based approach to distributed component integration.

A Grid-Based Image Archival and Analysis System

Here the authors present a Grid-aware middleware system, called GridPACS, that enables management and analysis of images in a massive scale, leveraging distributed software components coupled with interconnected computation and storage platforms. The need for this infrastructure is driven by the increasing biomedical role played by complex datasets obtained through a variety of imaging modalities. The GridPACS architecture is designed to support a wide range of biomedical applications encountered in basic and clinical research, which make use of large collections of images. Imaging data yield a wealth of metabolic and anatomic information from macroscopic (e.g., radiology) to microscopic (e.g., digitized slides) scale. Whereas this information can significantly improve understanding of disease pathophysiology as well as the noninvasive diagnosis of disease in patients, the need to process, analyze, and store large amounts of image data presents a great challenge.

Software Components: a Formal Semantics Based on Coloured Petri Nets

This paper proposes a component model compliant with the current practice of Software Engineering, yet provided with a sound formal semantics based on Coloured Petri nets. Our proposal is structured as follows: 1) Define a component model. We have chosen a component model inspired by the CORBA Component Model (CCM), yet simpler and more precise. 2) Propose a notation to formally specify the internal behaviour of a software component. Our formal approach is based on Coloured Petri nets which makes it well suited to the modelling of concurrent, distributed or event-driven systems, and amenable to formal verification. 3) Define a mapping from the constructs of the component model (facets, receptacles, event sources and sinks) to the constructs of the Petri-net based behavioural specification (e.g. places, transitions, etc.). 4) Provide a formal definition of inter-components communication primitives, (invocation of methods, event-based communication). This definition is also given in terms of Petri nets. 5) Provide a denotational semantics of an assembly of components, in order to define the behaviour of such a system in terms of the individual behaviour of each component and of the formal definition of inter-component communication primitives.The expected benefits of such an approach are threefold: 1) Offer a convenient notation for describing the internal behaviour of concurrent and distributed software components, 2) Provide a formal, unambiguous semantics of component features such as event multicast or service invocation, 3) And, with the previous two being necessary conditions, offer some means to reason about assemblies of components designed with this approach, in particular to mathematically verify properties on them.

Throughput performance of popular JMS servers

The Java Messaging Service (JMS) facilitates communication among distributed software components according to the publish/subscribe principle. If the subscribers install filter rules on the JMS server, JMS can be used as a message routing platform, but it is not clear whether its message throughput is sufficiently high to support large-scale systems. In this paper, we investigate the capacity of three high performance JMS server implementations: FioranoMQ, SunMQ, and WebsphereMQ. In contrast to other studies, we focus on the message throughput in the presence of filters and show that filtering reduces the performance significantly. We present models for the message processing time of each server and validate them by measurement.