Towards the Systematic Testing of Virtual Reality Programs

Abstract

Background: Software testing is a critical activity to ensure that software complies with its specification. However, current software testing activities tend not to be completely effective when applied in specific software domains in Virtual Reality (VR) that has several new types of features such as images, sounds, videos, and differentiated interaction, which can become sources of new kinds of faults. Aims: This paper presents an overview of the main VR characteristics that can have an impact on verification, validation, and testing (VV&T). Furthermore, it analyzes some of the most successful VR open-source projects to draw a picture concerning the danger of the lack of software testing activities. Method: We compared the current state of software testing practice in open-source VR projects and evaluate how the lack of testing can be damaging to the development of a product. We assessed the incidence of code smells and verified how such projects behave concerning the tendency to present faults. We also perform the same analyses on projects that are not VR related to have a better understanding of these results. Results: The results showed that the practice of software testing is not yet widespread in the development of VR applications. It was also found that there is a high incidence of code smells in VR projects. Analyzing Non-VR projects we noticed that classes that have test cases tend to produce fewer smells compared to classes that were not tested. Regarding fault-proneness analysis, we used an unsupervised approach to VR and Non-VR projects. Results showed that about 12.2% of the classes analyzed in VR projects are fault-prone, while Non-VR projects presented a lower fault-proneness rate (8.9%). Conclusions: Regarding the application of software testing techniques on VR projects, it was observed that only a small number of projects are concerned about developing test cases for VR projects, perhaps because we still do not have the necessary tools to help in this direction. Concerning smells, we concluded that there is a high incidence in VR projects, especially regarding implementing smells and this high incidence can have a significant influence on faults. Finally, the study related to fault proneness pointed out that the lack of software testing activity is a significant risk to the success of the projects.