Runtime testing of context-aware variability in adaptive systems

Abstract

Abstract Context: A Dynamically Adaptive System (DAS) supports runtime adaptations to handle changes in the operational environment. These adaptations can change the system’s structure or behavior and even the logic of its adaptation mechanism. However, these adaptations may insert defects, leading the system to fail at runtime. Objective: Aiming to identify these failures, testing can be executed to verify the system at runtime. Studies in the literature mostly focus on testing to verify the adaptations at design-time or functionalities at runtime, rather than exercising the adaptation mechanism at runtime. So, we propose RETAkE (RuntimE Testing of dynamically Adaptive systEms). Method: RETAkE is an approach to perform the runtime testing based on the system’s context variability and feature modeling. RETAkE tests the adaptation mechanism, enabling the verification of its adaptation rules with the system’s variability model. The runtime testing is supported by the verification of behavioral properties. For the evaluation, we used the mutation testing technique with two DAS. We also conducted an evaluation to measure the overhead introduced when RETAkE is integrated to the DAS. Results: RETAkE identified the mutants in the two mobile DAS, but the results vary due to the probabilistic nature of the approach to generate test sequences. Regarding the overhead, test sequences of size 30 had a low impact. However, bigger test sequences increase the overhead. Conclusion: The integration of RETAkE to the DAS adaptation mechanism can support the discovery of adaptation failures that occur at runtime. Furthermore, the results of the evaluation suggest its feasibility to perform runtime testing.