Model-based Testing Techniques Research Articles

Multi-Level Testing Approach for Multi-Agent Systems

The software development cycles need product testing. There is a crucial lack in testing phases of multi-agent systems. To this end, a call for an investigation of appropriate testing techniques is necessary to provide adequate software development processes and supporting tools. Among all existing solutions of the test, the model-based testing technique –MBT) has gained attention with the popularization of models both in software design and development. This technique uses a so-called abstract test model to generate abstract test cases. After their concretization, concrete test cases are submitted to the system under test. The systems outputs are finally compared to the abstract test model expected results. In this context, a model-based testing approach for multi-agent systems based on the Reference net paradigm is proposed in this paper. A running example supported by a multi-agent testing prototype, which aims at simplifying and providing a uniform and automated way for multi-agent systems testing is presented and discussed.

An approach for performance requirements verification and test environments generation

Model-based testing (MBT) is a method that supports the design and execution of test cases by models that specify the intended behaviors of a system under test. While systematic literature reviews on MBT in general exist, the state of the art on modeling and testing performance requirements has seen much less attention. Therefore, we conducted a systematic mapping study on model-based performance testing. Then, we studied natural language software requirements specifications in order to understand which and how performance requirements are typically specified. Since none of the identified MBT techniques supported a major benefit of modeling, namely identifying faults in requirements specifications, we developed the Performance Requirements verificatiOn and Test EnvironmentS generaTion approach (PRO-TEST). Finally, we evaluated PRO-TEST on 149 requirements specifications. We found and analyzed 57 primary studies from the systematic mapping study and extracted 50 performance requirements models. However, those models don’t achieve the goals of MBT, which are validating requirements, ensuring their testability, and generating the minimum required test cases. We analyzed 77 Software Requirements Specification (SRS) documents, extracted 149 performance requirements from those SRS, and illustrate that with PRO-TEST we can model performance requirements, find issues in those requirements and detect missing ones. We detected three not-quantifiable requirements, 43 not-quantified requirements, and 180 underspecified parameters in the 149 modeled performance requirements. Furthermore, we generated 96 test environments from those models. By modeling performance requirements with PRO-TEST, we can identify issues in the requirements related to their ambiguity, measurability, and completeness. Additionally, it allows to generate parameters for test environments.

Automatic Generation and Optimization of Combinatorial Test Cases from UML Activity Diagram Using Particle Swarm Optimization

Generation of test cases is one of the essential activities of the software testing process. The process of executing a programme to identify defects to improve the system's quality is known as software testing. Manually writing test cases takes time, effort, and money. On the other hand, generating test cases automatically is the solution to this problem. For this automation process, a model-based test case generation technique would be acceptable. A model is usually required to generate test cases in the model-based testing technique. Nowadays, researchers have relied on the activity diagram to generate test cases. Test cases for combinatorial logic systems are required. Combinatorial testing is essential for producing a small number of test cases and identifying errors occurred by interactions between system input parameters. Information about constraints, parameters and its values are required for generation of test cases. It is difficult to extract information regarding constraints, parameters, its values, and interactions between parameters from an Activity Diagram. A novel approach is proposed to extract this information from an Activity Diagram. The authors created a tool that automatically generates combinatorial test cases using UML Activity Diagrams. The proposed tool has two main parts. First, the combinatorial test design model is developed for extraction of input parameters. Second part is generation of optimized number of combinatorial test cases using Particle Swarm Optimization algorithm. Finally, the authors experimented on a real-world case study namely viz. Railway Reservation using the proposed tool, and it is shown that the proposed tool generated optimum number of combinatorial test cases.

Adaptive or embedded software testing and mutation testing

This issue contains four papers. The first paper provides a survey of work on testing adaptive and context-aware systems, while the second one concerns testing embedded systems. The remaining two papers explore particular problems associated with an area well known to most STVR readers: mutation testing. The first paper, ‘Testing of adaptive and context-aware systems: Approaches and challenges’, by Bento R. Siqueira, Fabiano C. Ferrari, Kathiani E. Souza, Valter V. Camargo and Rogério de Lemos, introduces a systematic literature review and a thematic analysis of studies to characterize the state of the art in testing adaptive systems (ASs) and context-aware systems (CASs) and discuss approaches, challenges, observed trends and research limitations and directions. The authors discover recurring research concerns related to AS and CAS testing (such as generation of test cases and built-in tests), recurring testing challenges (such as context monitoring and runtime decisions), some trends (such as model-based testing and hybrid techniques) and some little investigated issues (such as uncertainty and prediction of changes). (Recommended by T.Y. Chen) The second paper, ‘Remote embedded devices test framework on the cloud’, by Il-Seok (Benjamin) Choi and Chang-Sung Jeong, introduces a remote embedded device test framework on the cloud named RED-TFC, whose reliability test manager component can automatically perform various tests for evaluating reliability and performance of distributed shared devices by utilizing the cloud concept. RED-TFC includes two major techniques: the adaptive sample scale for reliability test (ASRT) and the mass sample reliability test (MSRT). The authors analyse two Android smartphone models that include many embedded components and show that RED-TFC can help detect a high number of reliability problems in smartphones. (Recommended by Tanja Vos) The third paper, ‘Analysing the combination of cost reduction techniques in Android mutation testing’, by Macario Polo-Usaola and Isyed Rodríguez-Trujillo, concerns the use of mutation testing when testing mobile apps. As the authors note, when testing an app, one typically deploys the app and its mutants on mobile devices or executes them on an emulator. Doing so increases the test execution time. Naturally, it can also significantly increase the cost of mutation testing, especially when there are many mutants. The authors investigate several techniques that have been devised for reducing execution time in mutation testing and produce a mathematical model with the aim of predicting the time taken when some combination of these techniques is used. (Recommended by Mike Papadakis) The final paper is ‘An ensemble-based predictive mutation testing approach that considers impact of unreached mutants’ by Alireza Aghamohammadi and Seyed-Hassan Mirian-Hosseinabadi. This paper also concerns both mutation testing and prediction. However, the authors look at a different prediction problem: that of predicting whether a mutant will be killed. The authors note that previous work did not consider the impact of unreachable mutants: those where the mutation point is not covered by any of the test cases used. It is argued that since many mutation tools exclude unreachable mutants, such mutants should also be removed from any empirical evaluation. The authors report the results of replicating previous studies but also eliminating unreachable mutants, finding that the resultant performance of prediction techniques is far lower than that reported. The authors then propose an alternative prediction model, which is shown to be effective when unreachable mutants are removed. (Recommended by Tanja Vos)

A Formal Testing Model for Operating Room Control System using Internet of Things

Technological advances in recent years have significantly changed the way an operating room works. This work aims to create a platform to solve the problems of operating room occupancy and prepare the rooms with an environment that is favorable for all operations. Using this system, a doctor can control all operation rooms, especially before an operation, and monitor their temperature and humidity to prepare for the operation. Also, in the event of a problem, an alert is sent to the nurse responsible for the room and medical stuff so that the problem can be resolved. The platform is tested using a Raspberry PI card and sensors. The sensors are connected to a cloud layer that collects and analyzes the temperature and humidity values obtained from the environment during an operation. The result of experimentations is visualized through a web application and an Android application. The platform also considers the security aspects such as authorization to access application functionalities for the Web and the mobile applications. We can also test and evaluate the system’s existing problems and vulnerabilities using the IEEE and owasp IoT standards. Finally, the proposed framework is extended with a model based testing technique that may be adopted for validating thesecurity aspects.

Extending MUD Profiles Through an Automated IoT Security Testing Methodology

Defining the intended behaviour of IoT devices is considered as a key aspect to detect and mitigate potential security attacks. In this direction, the Manufacturer Usage Description (MUD) has been recently standardised to reduce the attack surface of a certain device through the definition of access control policies. However, the semantic model is only intended to provide network level restrictions for the communication of such device. In order to increase the expressiveness of this approach, we propose the use of an automated IoT security testing methodology, so that testing results are used to generate augmented MUD profiles, in which additional security aspects are considered. For the enforcement of these profiles, we propose the use of different access control technologies addressing application layer security concerns. Furthermore, the methodology is based on the use of Model-Based Testing (MBT) techniques to automate the generation, design and implementation of security tests. Then, we describe the application of the resulting approach to the Elliptic Curve Diffie-Hellman over COSE (EDHOC) protocol, which represents a standardisation effort to build a lightweight authenticated key exchange protocol for IoT constrained scenarios.

Hierarchical featured state machines

Variants of the Finite State Machine (FSM) model have been extensively used to describe the behavior of reactive systems. In particular, several model-based testing techniques have been developed to support test case generation from FSMs and test case execution. Most of such techniques require several validation properties to hold for the underlying test models. Featured Finite State Machine (FFSM) is an extension of the FSM model proposed in our earlier publication that represents the abstract behavior of an entire Software Product Line (SPL). By validating an FFSM, we validate all valid products configurations of the SPL looking forward configurable test suites. However, modeling a large SPL using flat FFSMs may lead to a huge and hard-to-maintain specification. In this paper, we propose an extension of the FFSM model, named Hierarchical Featured State Machine (HFSM). Inspired by Statecharts and UML state machines, we introduce the HFSM model to improve model readability by grouping up FFSM conditional states and transitions into abstracted entities. Our ultimate goal is to use HFSMs as test models. To this end, we first define some syntactic and semantical validation criteria for HFSMs as prerequisites for using them as test models. Moreover, we implement an adapted graphical Eclipse-based editor from the Yakindu Project for modeling, derivation, and checking feature-oriented properties using Satisfiability Modulo Theory (SMT) solver tools. We investigate the applicability of our approach by applying it to an HFSM for a realistic case study (the Body Comfort System). The results indicate that HFSMs can be used to compactly represent and efficiently validate the behavior of parallel components in SPLs. • We propose an extension of the FFSM model, named Hierarchical Featured State Machine (HFSM). • A graphical Eclipse-based editor was implemented for modeling, validation, and derivation of HFSMs. • A case study from the automotive domain concerning the Body Comfort System was explored.

Test case prioritization techniques for model-based testing: a replicated study

Recently, several test case prioritization (TCP) techniques have been proposed to order test cases for achieving a goal during test execution, particularly, revealing faults sooner. In the model-based testing (MBT) context, such techniques are usually based on heuristics related to structural elements of the model and derived test cases. In this sense, techniques’ performance may vary due to a number of factors. While empirical studies comparing the performance of TCP techniques have already been presented in literature, there is still little knowledge, particularly in the MBT context, about which factors may influence the outcomes suggested by a TCP technique. In a previous family of empirical studies focusing on labeled transition systems, we identified that the model layout, i.e., amount of branches, joins, and loops in the model, alone may have little influence on the effectiveness of TCP techniques investigated, whereas characteristics of test cases that actually fail definitely influences this aspect. However, we considered only synthetic artifacts in the study, which reduced the ability of representing properly the reality. In this paper, we present a replication of one of these studies, now with a larger and more representative selection of techniques and considering test suites from industrial systems as experimental objects. Our objective is to find out whether the results remain while increasing the validity in comparison to the original study. Results reinforce that there is no best performer among the investigated techniques and characteristics of test cases that fail represent an important factor, although adaptive random-based techniques are less affected by it.

An Automated Functional Testing Framework for Context-Aware Applications

In the modern era of mobile computing, context-aware computing is an emerging paradigm due to its widespread applications. Context-aware applications are gaining increasing popularity in our daily lives since these applications can determine and react according to the situational context and help users to enhance usability experience. However, testing these applications is not straightforward since it poses several challenges, such as generating test data, designing context-coupled test cases, and so on. However, the testing process can be automated to a greater extent by employing model-based testing technique for context-aware applications. To achieve this goal, it is necessary to automate model transformation, test data generation, and test case execution processes. In this paper, we propose an approach for behavior modeling of context-aware application by extending the UML activity diagram. We also propose an automated model transformation approach to transform the development model, i.e., extended UML activity diagram into the testing model in the form of function nets. The objective of this paper is to automate the context-coupled test case generation and execution. We propose a functional testing framework for automated execution of keyword-based test cases. Our functional testing framework can reduce the testing time and cost, thus enabling the test engineers to execute more testing cycles to attain a higher degree of test coverage.

A Review of Meta-Model Designed for the Model-Based Testing Technique

Recently, software engineering research has focused on IT system testing. Due to its practicality, academic and corporate research has presented various model-based testing methodologies. Model-based testing is a popular software technique that ensures system performance and evaluation. Automatic test case generation from requirements is intended to replace manual testing. Most of these approaches differ in critical areas like the specification paradigm and the model used to define requirements. Still, they share common principles, problems, and characteristics inherited from model-based testing. This paper proposes a meta-model that represents model-based testing principles and attributes to define the structure and entities of any new approach.

Uncertainty-Wise Cyber-Physical System test modeling

It is important that a Cyber-Physical System (CPS) with uncertainty in its behavior caused by its unpredictable operating environment, to ensure its reliable operation. One method to ensure that the CPS will handle such uncertainty during its operation is by testing the CPS with model-based testing (MBT) techniques. However, existing MBT techniques do not explicitly capture uncertainty in test ready models, i.e., capturing the uncertain expected behavior of a CPS in the presence of environment uncertainty. To fill this gap, we present an Uncertainty-Wise test-modeling framework, named as UncerTum, to create test ready models to support MBT of CPSs facing uncertainty. UncerTum relies on the definition of a UML profile [the UML Uncertainty Profile (UUP)] and a set of UML Model Libraries extending the UML profile for Modeling and Analysis of Real-Time and Embedded Systems (MARTE). UncerTum also benefits from the UML Testing Profile V.2 to support standard-based MBT. UncerTum was evaluated with two industrial CPS case studies, one real-world case study, and one open-source CPS case study from the following four perspectives: (1) Completeness and Coverage of the profiles and Model Libraries in terms of concepts defined in their underlying uncertainty conceptual model for CPSs, i.e., U-Model and MARTE, (2) Effort required to model uncertainty with UncerTum, and (3) Correctness of the developed test ready models, which was assessed via model execution. Based on the evaluation, we can conclude that we were successful in modeling all the uncertainties identified in the four case studies, which gives us an indication that UncerTum is sufficiently complete. In terms of modeling effort, we concluded that on average UncerTum requires 18.5% more time to apply stereotypes from UUP on test ready models.

Cost and Effectiveness of Search-Based Techniques for Model-Based Testing: An Empirical Analysis

Model-based testing (MBT) seems to be gaining interest in industry and academia due to its provision of systematic, automated and comprehensive testing. The challenge in MBT is to generate optimal test data to execute test cases. Recently, researchers have successfully applied search-based techniques (SBTs) by automating the search for an optimal set of test data at reasonable cost compared to other more expensive techniques. In real complex systems, effectiveness and cost of SBTs for MBT in industrial context are little known. The objective of this study is to empirically evaluate the cost and the effectiveness of SBTs for MBT on industrial case studies. We applied a model-driven approach and SBTs to automatically generate executable feasible test cases. The results show that the model-driven approach generated high number of infeasible test cases with less time while genetic algorithm (GA) and simulating annealing (SA) outperformed significantly random search (RS) with high generation time. We concluded that local SBTs are more appropriate to generate test data when the type of the constraints is simple. Current work on analyzing the cost and effectiveness on SBTs for MBT indicates possible enhancement using the model-driven approach to detect the infeasible paths and SBTs to achieve optimal success rate.

How to manage a conference special issue

STVR publishes a small number of special issues each year and has associated policies. However, simply following these policies/requirements does not ensure that all will run smoothly. In this editorial, I will focus on a very particular type of special issue: those that contain extended versions of conference papers. The editors of a special issue have many responsibilities. The first is to determine which authors should be invited to submit extended papers. This is not as simple as looking at the review scores from the conference and choosing the top few papers. For example, I have seen strong papers that have excellent reviews but that also are “complete” and so there is little scope to extend the work. Editors also have to consider the quality of the reviews, the contents of the reviewer comments, and also read the paper. Only then are they able to make a good assessment. The next stage of the process involves inviting authors and then advising those who accept the invitation. An extended paper should have significant additional content and not, for example, be the union of conference papers. It is also normal for an extended paper to have a different abstract and to carefully explain how the paper extends the original conference paper; I find that it is often best to place such an explanation in the introduction. Reviewers will be expected to comment on whether the extension is sufficient and authors should be encouraged to make it easy for the reviewers to make this assessment. The editors also have to choose reviewers. Ideally, this is done well in advance of the papers being received, thus limiting the scope for delays. We normally advise that, where possible, each paper should have one of the original reviewers and at least 1 reviewer who did not review the conference paper. In choosing one of the original reviewers, it is good to look at the quality of the reviews and the confidence scores. I would also recommend that the initial list of possible reviewers includes more than the minimum (3 reviewers per paper); in reality, not all those invited to review papers will accept. Once the reviews are in, the editors make recommendations (accept, minor revisions, major revisions, or reject). The recommendation should normally be made as soon as possible—there is usually no reason to make the authors wait longer than necessary. Making prompt recommendations can help speed up the overall process since, for example, reviews for 1 paper might arrive quite early but the authors might take longer to revise the paper. I certainly would advise against waiting for all of the reviews of all of the papers. The recommendations/decisions should be independent—we might have an expected number of papers for a special issue, but a paper will not be rejected because there is “no room.” As we have seen, there are many things to consider when organising a special issue—the above simply describes a few of these. However, there should be real satisfaction in seeing the final issue and hopefully, this makes the work worthwhile for the editors as well as the authors and readers. Now to this issue of STVR, which contains 2 papers. Both take a general area (testing, model checking) and explore this within a particular context. The first paper is “Pattern-Based GUI Testing: Bridging the Gap Between Design & Quality Assurance” by Moreira, Rodrigo; Paiva, Ana; Nabuco, Miguel; Memon, Atif (recommended by Le Traon). The aim of the paper is to develop new testing techniques, for graphical user interfaces, that take advantage of classic generic solutions (patterns). The authors thus develop a specialised testing technique that takes advantage of such generic solutions having been used. Essentially, this is a model-based testing technique that incorporates reuse around the test patterns identified. The work was evaluated through a case study with real faults and a second study that used seeded faults. The second paper is “Bounded Model Checking of C++ Programs based on the Qt Cross-Platform Framework” by Rodrigues Monteiro Sousa, Felipe; Praia Garcia, Mário Angel; Cordeiro, Lucas; Batista de Lima Filho, Eddie (recommended by Müller). This looks at the problem of verifying embedded systems where the software has been developed using a cross-platform software development framework (Qt) and the C++ programming language. Essentially, the authors modelled the Qt libraries. The approach was evaluated using a benchmark with 711 C++/Qt programs (many extracted from Qt documentation) plus 2 real applications.

The experimental applications of search-based techniques for model-based testing: Taxonomy and systematic literature review

ContextModel-based testing (MBT) aims to generate executable test cases from behavioral models of software systems. MBT gains interest in industry and academia due to its provision of systematic, automated, and comprehensive testing. Researchers have successfully applied search-based techniques (SBTs) by automating the search for an optimal set of test cases at reasonable cost compared to other more expensive techniques. Thus, there is a recent surge toward the applications of SBTs for MBT because the generated test cases are optimal and have low computational cost. However, successful, future SBTs for MBT applications demand deep insight into its existing experimental applications that underlines stringent issues and challenges, which is lacking in the literature. ObjectiveThe objective of this study is to comprehensively analyze the current state-of-the-art of the experimental applications of SBTs for MBT and present the limitations of the current literature to direct future research. MethodWe conducted a systematic literature review (SLR) using 72 experimental papers from six data sources. We proposed a taxonomy based on the literature to categorize the characteristics of the current applications. ResultsThe results indicate that the majority of the existing applications of SBTs for MBT focus on functional and structural coverage purposes, as opposed to stress testing, regression testing and graphical user interface (GUI) testing. We found research gaps in the existing applications in five areas: applying multi-objective SBTs, proposing hybrid techniques, handling complex constraints, addressing data and requirement-based adequacy criteria, and adapting landscape visualization. Only twelve studies proposed and empirically evaluated the SBTs for complex systems in MBT. ConclusionThis extensive systematic analysis of the existing literature based on the proposed taxonomy enables to assist researchers in exploring the existing research efforts and reveal the limitations that need additional investigation.

Model-based testing for building reliable realtime interactive music systems

The role of an Interactive Music System (IMS) is to accompany musicians during live performances, acting like a real musician. It must react in realtime to audio signals from musicians, according to a timed high-level requirement called mixed score, written in a domain specific language. Such goals imply strong requirements of temporal reliability and robustness to unforeseen errors in input, yet not much addressed by the computer music community.We present the application of Model-Based Testing techniques and tools to a state-of-the-art IMS, including in particular: offline and on-the-fly approaches for the generation of relevant input data for testing (including timing values), with coverage criteria, the computation of the corresponding expected output, according to the semantics of a given mixed score, the black-box execution of the test data on the System Under Test and the production of a verdict. Our method is based on formal models in a dedicated intermediate representation, compiled directly from mixed scores (high-level requirements), and either passed, to the model-checker Uppaal (after conversion to Timed Automata) in the offline approach, or executed by a virtual machine in the online approach. Our fully automatic framework has been applied to real mixed scores used in concerts and the results obtained have permitted to identify bugs in the target IMS.

Testing HMAS-based applications: An ASPECS-based approach

Organization-Based Multi-Agent Systems are receiving increasingly more attention in the field of complex and distributed systems development. Holonic Multi-Agent Systems (HMAS) are considered as well known organization allowing the construction of complex systems that are efficient in terms of use of resources, highly resilient to disturbances and adaptable to changes in their environment. However, HMAS-based applications development is not mature enough yet. Particularly, testing activities, which represent an important task in their quality assurance, are not well covered. This paper aims at proposing a new model-based testing technique for holonic agents. The technique uses genetic algorithms and takes into account the evolution (successive versions) of an agent. The approach is organized in two main phases that are conducted iteratively. The first phase focuses on the detection of a new version of an agent under test. The second phase addresses the testing of each new detected version. The new version of the agent is analyzed in order to generate a behavioral model on which is based the generation of test cases. The test cases generation process focuses on the new (and/or changed) parts of the agent behavior. In this way, the technique supports an incremental update of the test cases, which is a crucial issue. A software tool that we developed supports the proposed technique. The approach and associated tool are illustrated using a concrete case study.

Introduction to the special issue on Mutation Testing

Introduction to the special issue on Mutation Testing

An Integration of UML Sequence Diagram with Formal Specification Methods—A Formal Solution Based on Z

UML Diagrams are considered as a main component in requirement engineering process and these become an industry standard in many organizations. UML diagrams are useful to show an interaction, behavior and structure of the system. Similarly, in requirement engineering, formal specification methods are also being used in crucial systems where precise information is required. It is necessary to integrate System Models with such formal methods to overcome the requirements errors i.e. contradiction, ambiguities, vagueness, incompleteness and mixed values of abstraction. Our objective is to integrate the Formal Specification Language (Z) with UML Sequence diagram, as sequence diagram is an interaction diagram which shows the interaction and proper sequence of components (Methods, procedures etc.) of the system. In this paper, we focus on components of UML Sequence diagram and then implement these components in formal specification language Z. And the results of this research papers are complete integrated components of Sequence diagram with Z schemas, which are verified by using tools and model based testing technique of Formal Specifications. Results can be more improved by integrating remaining components of Sequence and other UML diagrams into Formal Specification Language.

Supporting the Combined Selection of Model-Based Testing Techniques

The technical literature on model-based testing (MBT) offers us several techniques with different characteristics and goals. Contemporary software projects usually need to make use of different software testing techniques. However, a lack of empirical information regarding their scalability and effectiveness is observed. It makes their application difficult in real projects, increasing the technical difficulties to combine two or more MBT techniques for the same software project. In addition, current software testing selection approaches offer limited support for the combined selection of techniques. Therefore, this paper describes the conception and evaluation of an approach aimed at supporting the combined selection of MBT techniques for software projects. It consists of an evidence-based body of knowledge with 219 MBT techniques and their corresponding characteristics and a selection process that provides indicators on the level of adequacy (impact indicator) amongst MBT techniques and software projects characteristics. Results from the data analysis indicate it contributes to improve the effectiveness and efficiency of the selection process when compared to another selection approach available in the technical literature. Aiming at facilitating its use, a computerized infrastructure, evaluated into an industrial context and evolved to implement all the facilities needed to support such selection approach, is presented.

Achieving scalable model-based testing through test case diversity

The increase in size and complexity of modern software systems requires scalable, systematic, and automated testing approaches. Model-based testing (MBT), as a systematic and automated test case generation technique, is being successfully applied to verify industrial-scale systems and is supported by commercial tools. However, scalability is still an open issue for large systems, as in practice there are limits to the amount of testing that can be performed in industrial contexts. Even with standard coverage criteria, the resulting test suites generated by MBT techniques can be very large and expensive to execute, especially for system level testing on real deployment platforms and network facilities. Therefore, a scalable MBT technique should be flexible regarding the size of the generated test suites and should be easily accommodated to fit resource and time constraints. Our approach is to select a subset of the generated test suite in such a way that it can be realistically executed and analyzed within the time and resource constraints, while preserving the fault revealing power of the original test suite to a maximum extent. In this article, to address this problem, we introduce a family of similarity-based test case selection techniques for test suites generated from state machines. We evaluate 320 different similarity-based selection techniques and then compare the effectiveness of the best similarity-based selection technique with other common selection techniques in the literature. The results based on two industrial case studies, in the domain of embedded systems, show significant benefits and a large improvement in performance when using a similarity-based approach. We complement these analyses with further studies on the scalability of the technique and the effects of failure rate on its effectiveness. We also propose a method to identify optimal tradeoffs between the number of test cases to run and fault detection.