Exploring the challenges in software testing of the 5G system at Nokia: A survey

Abstract

The ever-growing size and complexity of industrial software products pose significant quality assurance challenges to engineering researchers and practitioners, despite the constant effort to increase knowledge and improve the processes. 5G technology developed by Nokia is one example of such a grand and highly complex system with improvement potential. The following paper provides an overview of the current quality assurance processes used by Nokia to develop the 5G technology and provides insight into the most prominent challenges by an evaluation of perceived importance, urgency, and difficulty to understand the future opportunities. Nokia mode of operation, briefly introduced in this paper, has been subjected to extensive analysis by a selected group of experienced test-oriented professionals to define the most critical areas of concern. Secondly, the identified problems were evaluated by Nokia gNB system-level test professionals in a dedicated survey. The questionnaire was completed by 312 out of 2935 (10.63%) possible respondents. The challenges are seen as the most important and urgent: customer scenario testing, performance testing, and competence ramp-up. Challenges seen as the most difficult to solve are low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. Our research identified several improvement areas in the quality assurance processes used to develop the 5G technology by determining the most important and urgent problems that at the same time have a low perceived difficulty. Such initiatives are attractive from a business perspective. On the other hand, challenges seen as the most impactful yet difficult may be of interest to the academic research community. • Our survey, designed to elicit understanding of the most significant challenges in testing large and complex 5G gNB software system, received 312 out of 2935 (10.63%) system-level test practitioners in Nokia. • Challenges seen as having the highest importance are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as having the highest urgency are: customer scenario testing, performance testing, and competence ramp-up. • Challenges seen as being the most difficult are: low occurrence failures, hidden feature dependencies, and hardware configuration-specific problems. • Obtained results imply which future improvement opportunities can have the most significant impact on the quality and cost of the software testing processes in the company. • Interestingly, respondents still seem to be not well familiar with software defect prediction models.