Software Architecture Evaluation Research Articles

LPMSAEF: Lightweight process mining-based software architecture evaluation framework for security and performance analysis

The article discusses the need for a lightweight software architecture evaluation framework that can address practitioners' concerns. Specifically, the proposed framework uses process mining and Petri nets to analyze security and performance in software development's early and late stages. Moreover, the framework has been implemented in six case studies, and the results show that it is a feasible and effective solution that can detect security and performance issues in complex and heterogeneous architecture with less time and effort. Furthermore, the article provides a detailed explanation of the framework's features, factors, and evaluation criteria. Additionally, this article discusses the challenges associated with traditional software architecture documentation methods using Unified Modeling Language diagrams and the limitations of code alone for creating comprehensive Software Architecture models. Various methods have been developed to extract implicit Software Architecture from code artifacts, but they tend to produce code-oriented diagrams instead of Software Architecture diagrams. Therefore, to bridge the model-code gap, the article proposes a framework that considers existing Software Architecture in the source code as architectural components and focuses on Software Architecture behaviors for analyzing performance and security. The proposed framework also suggests comparing Software Architecture extracted by different Process Mining algorithms to achieve consensus on architecture descriptions, using visualizations to understand differences and similarities. Finally, the article suggests that analyzing the previous version of a system's Software Architecture can lead to improvements and deviations from planned Software Architecture can be detected using traceability approaches to aid software architects in detecting inconsistencies.

Comprehensive Survey of Cutting-Edge Methods for Software Architecture Evaluation

The crucial responsibility of assessing software architecture is of utmost importance in ensuring that a software system conforms to superior qualities. It is a crucial tool for cutting expenses and labor during the course of the software development lifecycle. The main goal of software architecture evaluation is to provide reliable methods for determining and improving the quality characteristics that are innate in software. This study is deeply devoted to investigating the wide range of techniques used in software architecture assessment. One of the main focuses of these evaluation techniques is scenario-based assessments, which offer a comprehensive picture of the software's behavior under different circumstances. Within the scope of this study, a thorough analysis is carried out on eighteen different methods. Thirteen of these fall into the category of early evaluation techniques, which are positioned to detect and address architectural problems at the outset of development. The other five strategies are classified as late assessment techniques and focus on validating and optimizing the software architecture in the latter stages of the development process.

Trends in continuous evaluation of software architectures

Trends in continuous evaluation of software architectures

Process Mining Discovery Techniques for Software Architecture Lightweight Evaluation Framework

This research recognizes the limitation and challenges of adapting and applying Process Mining as a powerful tool and technique in the Hypothetical Software Architecture (SA) Evaluation Framework with the features and factors of lightweightness. Process mining deals with the large-scale complexity of security and performance analysis, which are the goals of SA evaluation frameworks. As a result of these conjectures, all Process Mining researches in the realm of SA are thoroughly reviewed, and nine challenges for Process Mining Adaption are recognized. Process mining is embedded in the framework and to boost the quality of the SA model for further analysis, the framework nominates architectural discovery algorithms Flower, Alpha, Integer Linear Programming (ILP), Heuristic, and Inductive and compares them vs. twelve quality criteria. Finally, the framework’s testing on three case studies approves the feasibility of applying process mining to architectural evaluation. The extraction of the SA model is also done by the best model discovery algorithm, which is selected by intensive benchmarking in this research. This research presents case studies of SA in service-oriented, Pipe and Filter, and component-based styles, modeled and simulated by Hierarchical Colored Petri Net techniques based on the cases’ documentation. Process mining within this framework deals with the system’s log files obtained from SA simulation. Applying process mining is challenging, especially for a SA evaluation framework, as it has not been done yet. The research recognizes the problems of process mining adaption to a hypothetical lightweight SA evaluation framework and addresses these problems during the solution development.

Continuous and Proactive Software Architecture Evaluation: An IoT Case

Design-time evaluation is essential to build the initial software architecture to be deployed. However, experts’ assumptions made at design-time are unlikely to remain true indefinitely in systems that are characterized by scale, hyperconnectivity, dynamism, and uncertainty in operations (e.g. IoT). Therefore, experts’ design-time decisions can be challenged at run-time. A continuous architecture evaluation that systematically assesses and intertwines design-time and run-time decisions is thus necessary. This paper proposes the first proactive approach to continuous architecture evaluation of the system leveraging the support of simulation. The approach evaluates software architectures by not only tracking their performance over time, but also forecasting their likely future performance through machine learning of simulated instances of the architecture. This enables architects to make cost-effective informed decisions on potential changes to the architecture. We perform an IoT case study to show how machine learning on simulated instances of architecture can fundamentally guide the continuous evaluation process and influence the outcome of architecture decisions. A series of experiments is conducted to demonstrate the applicability and effectiveness of the approach. We also provide the architect with recommendations on how to best benefit from the approach through choice of learners and input parameters, grounded on experimentation and evidence.

Lightweight Software Architecture Evaluation for Industry: A Comprehensive Review.

Processes for evaluating software architecture (SA) help to investigate problems and potential risks in SA. It is derived from many studies that proposed a plethora of systematic SA evaluation methods, while industrial practitioners currently refrain from applying them since they are heavyweight. Nowadays, heterogeneous software architectures are organized based on the new infrastructure. Hardware and associated software allow different systems, such as embedded, sensor-based, modern AI, and cloud-based systems, to cooperate efficiently. It brings more complexities to SA evaluation. Alternatively, lightweight architectural evaluation methods have been proposed to satisfy the practitioner’s concerns, but practitioners still do not adopt these methods. This study employs a systematic literature review with a text analysis of SA’s definitions to propose a comparison framework for SA. It identifies lightweight features and factors to improve the architectural evaluation methods among industrial practitioners. The features are determined based on the practitioner’s concerns by analyzing the architecture’s definitions from stakeholders and reviewing architectural evaluation methods. The lightweight factors are acquired by studying the five most commonly used lightweight methods and the Architecture-based Tradeoff Analysis Method (ATAM), the most well-known heavyweight method. Subsequently, the research addresses these features and factors.

Evaluation of Software Architectures under Uncertainty

Context: Evaluating software architectures in uncertain environments raises new challenges, which require continuous approaches. We define continuous evaluation as multiple evaluations of the software architecture that begins at the early stages of the development and is periodically and repeatedly performed throughout the lifetime of the software system. Numerous approaches have been developed for continuous evaluation; to handle dynamics and uncertainties at run-time, over the past years, these approaches are still very few, limited, and lack maturity. Objective: This review surveys efforts on architecture evaluation and provides a unified terminology and perspective on the subject. Method: We conducted a systematic literature review to identify and analyse architecture evaluation approaches for uncertainty including continuous and non-continuous, covering work published between 1990–2020. We examined each approach and provided a classification framework for this field. We present an analysis of the results and provide insights regarding open challenges. Major results and conclusions: The survey reveals that most of the existing architecture evaluation approaches typically lack an explicit linkage between design-time and run-time. Additionally, there is a general lack of systematic approaches on how continuous architecture evaluation can be realised or conducted. To remedy this lack, we present a set of necessary requirements for continuous evaluation and describe some examples.

Kurumsal Ölçekte Mikroservis Tabanlı Bir Mimarinin Tasarlanması

Building or transformation of an enterprise software system is an onerous process which requires a precise definition of business demands. Then to enable the satisfaction of business requirements, the well-thought-of and convenient software architecture must be determined and designed. According to common sense, there are two methods to be followed in order to find the right solution for a problem. One is to handle the problem as a whole; like the traditional monolith architecture. The second method is to divide the problem into easily understandable and soluble fine-grains. If the second path is chosen in the software world, microservices architecture can be shown. When the entire enterprise-level system design is considered, to the best of our knowledge, there is no any leading empirical research on the evaluation of software architectures, selection of communication protocol, data formats, and database. In this paper, an easily scalable, maintainable, highly-available, reliable and observable software system is designed by comparing variant architectures, communication methods, and data models that would help to choose the most appropriate architecture or model for the right purpose. All the paper is about designing a backend API system. The client types or technologies are out of scope.

A fuzzy-AHP-based approach to select software architecture based on quality attributes (FASSA)

The software system design phase has recently received increasing attention due to continuous growth in both the size and complexity of software systems. As a key concept of this phase, software architecture plays an important role in the software extension cycle to the extent that the success of a software project is often determined by the degree of its design efficiency. In addition, software architecture evaluation is a fundamental step toward its subsequent validation. This paper is an attempt to propose an innovative method, based on fuzzy logic, to evaluate software architecture that addresses the inherent problems of existing methods found in the literature. The method can be used for complete design or even reconstruction of the architecture. Given the multi-faceted nature of the problem of evaluation and selection of an optimal architecture, we have employed a multi-objective decision technique, namely fuzzy hierarchical analysis process, which solves the problems associated with uncertainties and inaccuracies by incorporating fuzzy logic.

An Evaluation of Quantitative Non-Functional Requirements Assurance Using ArchiMate

Goal-oriented NFR (Non-Functional Requirement) assurance approaches were used to qualitatively evaluate software architectures. Assurance cases using quantitative method have not been applied to evaluate NFR assurance for software architectures. This paper presents a system architecture evaluation method which is able to conduct quantitative NFR assurance evaluation for system architecture through ArchiMate. The paper also proposes an algorithm to automate the quantitative evaluation process. A questionnaire survey among software engineers and a case study on a vehicular safety monitor system were carried out to verify the necessity of the method. Additionally, we conducted an experimental design with 18 samples divided into 2 groups with the goal of comparing how the independent variables affect the dependent variables. The results of the experiment demonstrate that the proposed method achieves better NFR evaluation effect than the traditional approach. Moreover, compared with the traditional approach, the proposed method shortens the time for NFR evaluation. The proposed method is expected to be used at the early stage of software development projects for system NFR development, such as requirements analysis, system architecture design and system modeling. At present, the method has been applied by software engineers in a practical software project.

Run-time evaluation of architectures: A case study of diversification in IoT

Run-time properties of modern software system environments, such as Internet of Things (IoT), are a challenge for existing software architecture evaluation methods. Such systems are largely data-driven, characterized by their dynamism, unpredictability in operation, hyper-connectivity, and scale. Properties, such as performance, delayed delivery, and scalability, are acknowledged to pose great risk and are difficult to evaluate at design-time. Run-time evaluation could potentially be used to complement design-time evaluation, enabling significant deviations from the expected performance values to be captured. However, there are no systematic software architecture evaluation methods that intertwine and interleave design-time and run-time evaluation. This paper addresses this gap by proposing a novel run-time architecture evaluation method suited for systems that exhibit uncertainty and dynamism in their operation. Our method uses machine learning and cost-benefit analysis at run-time to continuously profile the architecture decisions made, to assess their added value. We demonstrate the applicability and effectiveness of this approach in the context of an IoT system architecture, where some architecture design decisions were diversified to meet Quality of Service (QoS) requirements. Our approach provides run-time assessment for these decisions which can inform deployment, refinement, and/or phasing-out decisions.

An Empirical Study of the Effectiveness of Software Architecture Evaluation Meetings

One of the most important issues in scenario-based software architecture evaluation is the development of scenarios for characterizing a desired set of quality attributes by holding meetings of stakeholders. As such meetings are costly to arrange, studying the effectiveness of such meetings is an important research question. In this paper, we report on the findings from a software architecture evaluation empirical study aimed at investigating the effectiveness of scenario development meetings in terms of the quantity of gained and lost scenarios during the evaluation process. The findings from the experiment data analysis raise the question about the effectiveness of holding meetings for developing scenarios as more existing scenarios were lost than new scenarios were gained as a result of these meetings.

Un Enfoque Inteligente para Soporte a la Toma de Decisiones de Diseño Arquitectónicas en el Contexto de la Evaluación de Arquitecturas de Software

La Ingeniería de Software necesita herramientas novedosas para alcanzar alta calidad en el software, enfrentando el rol cambiante del mismo. La Arquitectura de Software es clave, ya que afecta directamente a la calidad final. La Evaluación de Arquitecturas de Software valida si la arquitectura cumple con los requerimientos de calidad, implicando decisiones de diseño. La toma de decisiones es un proceso complejo conducido por factores humanos, donde la Inteligencia Artificial puede asistir. Entonces, se propone un enfoque basado en Inteligencia Artificial para ayudar a arquitectos en el proceso de toma de decisiones de diseño conducido por atributos de calidad. Esta versión combina modelos de atributos de calidad y un Agente inteligente, utilizando Aprendizaje por Refuerzo para obtener una política de aplicación de patrones arquitectónicos secuencial mediante simulación. Un estudio de caso y una serie de experimentos ilustran la propuesta con patrones comúnmente utilizados en la industria del software.

Software architecture design, implementation, and evaluation of DCF-based MAC protocol on a low-power transceiver

The carrier-sensing multiple access with collision avoidance (CSMA/CA) protocol is the most well-known medium access control (MAC) protocol for wireless networks. Both the distributed coordination function (DCF) defined in IEEE 802.11 and the MAC layer defined in IEEE 802.15.4 are based on the CSMA/CA protocol. Nevertheless, these two standards have quite different carrier-sensing mechanisms. Different to continuous carrier sensing in DCF, an IEEE 802.15.4 node only senses the channel once just after a backoff. Sensing-once mechanism can reduce the computation loading on the CPU. However, it significantly increases the probability of failure transmission because a node is not fully aware of channel activity. This paper first proposes a software architecture integrating proper hardware features for designing a DCF-based MAC protocol and then successfully implements it on a low-power transceiver. In addition, this paper conducts experiments in a star topology network to compare the performance of the above DCF-MAC protocol with the IEEE 802.15.4 MAC protocol. Experimental results show that, without continuous sensing, the IEEE 802.15.4 network suffers a high transmission failure probability as the network size increases. Consequently, the proposed DCF-based MAC protocol outperforms the IEEE 802.15.4 MAC protocol in terms of packet loss probability and throughput.

A method for dependability evaluation of software architectures

Dependability is an important quality attribute for today’s software systems, and the decisions made at the architectural design stage play a key role in the dependability of software products. Architecture-level dependability evaluation techniques are effective only if they support a practical architecture modelling notation and consider the impact of execution environment, error propagation and usage profile on dependability. Regarding these factors, the objective of this paper is to introduce a new method for evaluating the dependability of software architectures. UML is the de facto standard for modelling software architectures, and DAM is the most comprehensive UML profile supporting dependability annotations. In this paper, UML–DAM models are transformed into a hierarchical and coloured extension of stochastic activity networks and the dependability measures are quantified using the activity-marking oriented reward structures. The transformation and evaluation processes are detailed and implemented in a software tool. Also, to validate the applicability of the method, a case study is presented.

Toward an analytical method for SLA validation

Quantitative properties of modern software systems are often defined as a part of a service-level agreement (SLA) that fixes the maximal load to be submitted to a system and guarantees bounds for the response time or delay. The evaluation of software architectures in order to validate SLAs is a challenging task since the systems tend to be complex, highly dynamic and to some extent unpredictable. Thus, there is a need for fast and abstract techniques to evaluate the performance of modern software architectures based on the information available in the SLAs. The paper presents an efficient approach to compute bounds on the delay of composed systems based on available bounds for the load and the response times of components. The technique can be used by a user of a software architecture to validate SLAs of composed services based on SLAs of the components. It can also be used by a provider of a software architecture to validate whether additional users can be accepted or to compute required service capacities to fulfill an SLA.

A Verification‐Based Approach to Evaluate Software Architecture Evolution

As the development and wide usage of Software architecture (SA), SA evolution becomes one of the hotspots of current research in modern software engineering domain. Most current researches concentrate on the modeling of SA evolution and lack the verification and evaluation of SA evolution. We proposed a verificationbased approach to evaluate SA evolution. The basic process includes:1) using Unified modeling language (UML) sequence diagram to model the interaction of components and study different types of evolution in practical examples; 2) using SPIN-based model checking to model and verify SA evolution; 3) comparing various verification outcomes and analyzing the influence of SA evolution on SA correctness and temporal properties. Both theory analysis and an experiment on a real evolution example from Model-view controller (MVC) to Spring web MVC (SWMVC) show that the verification-based approach to evaluate SA is significant.

Bio-inspired benchmark generator for extracellular multi-unit recordings

The analysis of multi-unit extracellular recordings of brain activity has led to the development of numerous tools, ranging from signal processing algorithms to electronic devices and applications. Currently, the evaluation and optimisation of these tools are hampered by the lack of ground-truth databases of neural signals. These databases must be parameterisable, easy to generate and bio-inspired, i.e. containing features encountered in real electrophysiological recording sessions. Towards that end, this article introduces an original computational approach to create fully annotated and parameterised benchmark datasets, generated from the summation of three components: neural signals from compartmental models and recorded extracellular spikes, non-stationary slow oscillations, and a variety of different types of artefacts. We present three application examples. (1) We reproduced in-vivo extracellular hippocampal multi-unit recordings from either tetrode or polytrode designs. (2) We simulated recordings in two different experimental conditions: anaesthetised and awake subjects. (3) Last, we also conducted a series of simulations to study the impact of different level of artefacts on extracellular recordings and their influence in the frequency domain. Beyond the results presented here, such a benchmark dataset generator has many applications such as calibration, evaluation and development of both hardware and software architectures.

Architecting cloud‐enabled systems: a systematic survey of challenges and solutions

SummaryThe literature on the challenges of and potential solutions to architecting cloud‐based systems is rapidly growing but is scattered. It is important to systematically analyze and synthesize the existing research on architecting cloud‐based software systems in order to build a cohesive body of knowledge of the reported challenges and solutions. We have systematically identified and reviewed 133 papers that report architecture‐related challenges and solutions for cloud‐based software systems. This paper reports the methodological details, findings, and implications of a systematic review that has enabled us to identify 44 unique categories of challenges and associated solutions for architecting cloud‐based software systems. We assert that the identified challenges and solutions classified into the categories form a body of knowledge that can be leveraged for designing or evaluating software architectures for cloud‐based systems. Our key conclusions are that a large number of primary studies focus on middleware services aimed at achieving scalability, performance, response time, and efficient resource optimization. Architecting cloud‐based systems presents unique challenges as the systems to be designed range from pervasive embedded systems and enterprise applications to smart devices with Internet of Things. We also conclude that there is a huge potential of research on architecting cloud‐based systems in areas related to green computing, energy efficient systems, mobile cloud computing, and Internet of Things. Copyright © 2016 John Wiley & Sons, Ltd.

Metric-based Evaluation of Software Architecture for an Engine Management System

Metric-based Evaluation of Software Architecture for an Engine Management System