Software Testing Practices Research Articles

Examining the Impact of Software Testing Practices on Software Quality in Batam Software Houses

This research aimed to investigate the impact of software testing practices on software quality in software companies in Batam, Indonesia. It focused on identifying key factors such as software testing knowledge, software testing approach, and software testing complexity and analysing their correlation with software quality. Data was collected from 48 respondents, including project managers, developers, and QA teams, using a questionnaire distributed via Google Forms and convenience sampling. The questionnaire was designed based on related studies to ensure relevance to the respondents’ roles. Regression analysis identified significant impacts of testing complexity, approach (p = 0.000), and knowledge (p = 0.003) on software quality. The F-test result (F = 32.622) confirmed a strong relationship between testing practices and software quality. These findings emphasise the critical role of robust testing strategies in enhancing software quality. For companies in Batam, the study offers actionable insights, including adopting structured frameworks and preferable action on testing approaches. Implementing these strategies can help organisations improve software outcomes and maintain competitiveness in the evolving software development landscape.

Enhancing Software Testing Practices in Tanzanian Software Development Companies: A Case Study Approach

Purposes: The primary objective of this conducted research is to investigate and propose strategies for improving software testing practices in Tanzanian software development companies. Specifically, the study identify the current state of software testing practices, understand challenges faced by software development companies in Tanzania, propose effective solutions, and evaluate their impact. Methodology: A mixed-methods approach employed to achieve the research objectives. Qualitative and quantitative data collection methods, including surveys, interviews, observations, documentation analysis, and experimentation, utilized to gather comprehensive insights into software testing practices. Purposive sampling employed to select diverse software development companies across different regions in Tanzania. Thematic analysis and statistical analysis applied to analyze qualitative and quantitative data, respectively, ensuring a robust examination of software testing practices. Findings: The research findings reveal the prevailing software testing practices in Tanzanian software development companies. Challenges such as resource constraints, inadequate test coverage, and limited collaboration between developers and testers are identified. Additionally, the study identifies best practices and proposes context-specific solutions to enhance software testing practices in Tanzanian companies. Statistical analysis provides quantitative insights into the effectiveness of proposed solutions. Unique Contribution to Theory, Practices and Policy: The study contributes to bridging the gap between academic research and industrial practices in software testing. Through addressing the unique challenges and opportunities in the Tanzanian context, the research provides actionable recommendations for improving software testing practices. The findings underscore the importance of tailored strategies and collaboration between academia and industry to enhance software quality and reliability in Tanzanian software development companies.

An empirical study of testing machine learning in the wild

Background: Recently, machine and deep learning (ML/DL) algorithms have been increasingly adopted in many software systems. Due to their inductive nature, ensuring the quality of these systems remains a significant challenge for the research community. Traditionally, software systems were constructed deductively, by writing explicit rules that govern the behavior of the system as program code. However, ML/DL systems infer rules from training data i.e., they are generated inductively). Recent research in ML/DL quality assurance has adapted concepts from traditional software testing, such as mutation testing, to improve reliability. However, it is unclear if these proposed testing techniques are adopted in practice, or if new testing strategies have emerged from real-world ML deployments. There is little empirical evidence about the testing strategies. Aims: To fill this gap, we perform the first fine-grained empirical study on ML testing in the wild to identify the ML properties being tested, the testing strategies, and their implementation throughout the ML workflow. Method: We conducted a mixed-methods study to understand ML software testing practices. We analyzed test files and cases from 11 open-source ML/DL projects on GitHub. Using open coding, we manually examined the testing strategies, tested ML properties, and implemented testing methods to understand their practical application in building and releasing ML/DL software systems. Results: Our findings reveal several key insights: 1.) The most common testing strategies, accounting for less than 40%, are Grey-box and White-box methods, such as Negative Testing , Oracle Approximation , and Statistical Testing . 2.) A wide range of \(17\) ML properties are tested, out of which only 20% to 30% are frequently tested, including Consistency , Correctness , and Efficiency . 3.) Bias and Fairness is more tested in Recommendation (6%) and CV (3.9%) systems, while Security & Privacy is tested in CV (2%), Application Platforms (0.9%), and NLP (0.5%). 4.) We identified 13 types of testing methods, such as Unit Testing , Input Testing , and Model Testing . Conclusions: This study sheds light on the current adoption of software testing techniques and highlights gaps and limitations in existing ML testing practices.

Foreign experience of software testing engineers training: analysis of educational standards and programs of American universities

The article aims to explore the U.S. educational system's approach to training software testing specialists, highlighting specific features and methodologies. The study involves analyzing key standards for IT engineer training in the USA, such as the ACM/IEEE Computer Science Curricula, ABET Accreditation Criteria, ISTE Standards for Educators, and CSTA K-12 Computer Science Standards, their overall context and targeting, as well as the inclusion of both fundamental and IT-specific disciplines into their content. In the study of each standard, particular emphasis was placed on defining the place and teaching methods of theory and practice of software testing in the university curriculum guidelines. As it was revealed in the analysis, despite the extremely similar interpretation of basic concepts and practices in the context of software testing, opinions about the methodology of their teaching and the need to study as a separate discipline, differ in a set of reviewed standards. Additionally, the research examines training programs at leading American universities, including MIT, Stanford, and Carnegie Mellon University, to identify how software testing and quality assurance are reflected in their curricula, covering full-time training programs for students in computer science, software engineering and similar related fields. The analysis examined programs in both general and basic disciplines in the context of information technology, and specific to a certain technological or organizational area, such as mobile device development or IT project management. The content of the subjects, the list of modules and lectures, as well as tasks for extracurricular work, available in open sources, was also analyzed and searched for topics related to quality assurance or software testing. As revealed in the study, in most technical universities testing is not covered within a separate discipline, but is taught as a part of ones related to development practices. However, some training programs, in particular the Master's level, still have disciplines, devoted exclusively to quality control, testing management and automation

Sentiment overflow in the testing stack: Analyzing software testing posts on Stack Overflow

Software testing is an integral part of modern software engineering practice. Past research has not only underlined its significance, but also revealed its multi-faceted nature. The practice of software testing and its adoption is influenced by many factors that go beyond tools or technology. This paper sets out to investigate the context of software testing from the practitioners’ point of view by mining and analyzing sentimental posts on the widely used question and answer website Stack Overflow. By qualitatively analyzing sentimental expressions of practitioners, which we extract from the Stack Overflow dataset using sentiment analysis tools, we discern factors that help us to better understand the lived experience of software engineers with regards to software testing. Grounded in the data that we have analyzed, we argue that sentiments like insecurity, despair and aspiration, have an impact on practitioners’ attitude towards testing. We suggest that they are connected to concrete factors like the level of complexity of projects in which software testing is practiced.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Exploring granular test coverage and its evolution with matrix visualizations

Current software-development tools that are used in practice make understanding the test execution of software difficult, for both granular tasks ( e.g., answering questions such as, “which test cases execute this method?”) and global tasks ( e.g., answering questions such as, “what is the proportion of unit tests to system tests?”). Current tools typically support local, file-based views of a project’s test suite and its execution data, and rarely offer a global overview. Even more rarely, do they provide access to historical information of this nature. Such global overviews can provide a larger context for a method’s execution by test cases; help identify other similar, or related methods; and even reveal similarity between individual tests. This work approaches such challenges with a novel, interactive, matrix-based visual interface that provides a global overview of a software project’s test suite, specifically in the context of the methods available in the project’s codebase. Through a series of interactive functions to sort, filter, query, and explore a test-matrix visualization, we evaluate how developers can effectively answer questions about their project’s test suite, and the code executed by such tests. We built a dynamic test-suite analysis and software-visualization tool that implements our designed interface to address the challenges of understanding the testing of software systems. With this implementation, we conducted a user study of 20 software developers to assess their ability to understand and report test execution information and measured accuracy and time. Additionally, we present a series of case studies to demonstrate a number of insights that our tool reveals. Our evaluations, performed on 26 real-world software systems, show that the interactive visualization assisted developers to answer questions about software tests and the code they execute. Further, the visualization consistently outperforms traditional development tools, both in accuracy and time taken to complete software-engineering tasks. Global-overview test matrices offer novel perspectives on test-suite composition, which can guide software development and testing practices.

Software Testing Instructional Design and Practice Based on BOPPPS Model

To address the problems in traditional teaching of Software Testing course, the BOPPPS model is integrated into its instructional design. And the boundary value analysis in black-box testing is used as an example. Some methods and specific operations are given for each instructional module, so as to improve students' learning autonomy and classroom efficiency. The feedback shows that the BOPPPS model is significantly helpful to improve the teaching effect.

Alternatives for testing of context-aware software systems in non-academic settings: results from a Rapid Review

Context-awareness challenges the engineering of contemporary software systems and jeopardizes their testing. The variation of context represents a relevant behavior that deepens the limitations of available software testing practices and technologies. However, such software systems are mainstream. Therefore, researchers in non-academic settings also face challenges when developing and testing contemporary software systems. To understand how researchers deal with the variation of context when testing context-aware software systems developed in non-academic settings. To undertake a secondary study ( Rapid Review ) to uncover the necessary evidence from primary sources describing the testing of context-aware software systems outside academia. The current testing initiatives in non-academic settings aim to generate or improve test suites that can deal with the context variation and the sheer volume of test input possibilities. They mostly rely on modeling the systems' dynamic behavior and increasing computing resources to generate test inputs to achieve this. We found no evidence of test results aiming at managing context variation through the testing lifecycle process. So far, the identified testing initiatives and strategies are not ready for mainstream adoption. They are all domain-specific, and while the ideas and approaches can be reproduced in distinct settings, the technologies are to be re-engineered and tailored to the context-awareness of contemporary software systems in different problem domains. Further and joint investigations in academia and experiences in non-academic settings can evolve the body of knowledge regarding the testing of contemporary software systems in the field.

A survey on the practices of software testing: a look into Brazilian companies

[Context:] Software testing is essential for all software development, and techniques and criteria have been proposed to ensure its quality in different application domains. [Objective:] This survey aims at the identification of software testing practices in Brazilian industries, towards an overview of the latest testing techniques, selection processes, challenges faced, tools and metrics used by testers. [Methodology:] Survey questions were carefully designed for providing relevant information to both industry and academy and evaluated by testers for improving the quality of the survey. [Results and Conclusions:] Our study provides insights into the current software testing practices in Brazilian software companies. The results show testers select a testing technique according to the project scope under development; however, most companies have shown a lack of importance and priority regarding to the testing activity. Some challenges raised will foster new research topics, outlined by the needs faced by testers in practice.

How are test smells treated in the wild? A tale of two empirical studies

Developing test code may be a time­-consuming process that requires much effort and cost, especially when done manually. In addition, during this process, developers and testers are likely to adopt bad design choices, which may lead to introducing the so­-called test smells in the test code. As the test code with test smells size increases, these tests might become more complex, and as a consequence, much more challenging to understand and evolve them correctly. Therefore, test smells may harm the test code quality and maintenance and break the whole software testing activities. In this context, this study aims to understand whether software testing practitioners unintentionally insert test smells when they implement test code. We first carried out an expert survey to analyze the usage frequency of a set of test smells and then interviews to reach a deeper understanding of how practitioners deal with test smells. Sixty professionals participated in the survey, and fifty professionals participated in the interviews. The yielded results indicate that experienced professionals introduce test smells during their daily programming tasks, even when using their companies’ standardized practices. Additionally, tools support test development and quality improvement, but most interviewees are not aware of test smells’ concepts.

SWFC-ART: A cost-effective approach for Fixed-Size-Candidate-Set Adaptive Random Testing through small world graphs

Adaptive random testing (ART) improves the failure-detection effectiveness of random testing by leveraging properties of the clustering of failure-causing inputs of most faulty programs: ART uses a sampling mechanism that evenly spreads test cases within a software’s input domain. The widely-used Fixed-Sized-Candidate-Set ART (FSCS-ART) sampling strategy faces a quadratic time cost, which worsens as the dimensionality of the software input domain increases. In this paper, we propose an approach based on small world graphs that can enhance the computational efficiency of FSCS-ART: SWFC-ART. To efficiently perform nearest neighbor queries for candidate test cases, SWFC-ART incrementally constructs a hierarchical navigable small world graph for previously executed, non-failure-causing test cases. Moreover, SWFC-ART has shown consistency in programs with high dimensional input domains. Our simulation and empirical studies show that SWFC-ART reduces the computational overhead of FSCS-ART from quadratic to log-linear order while maintaining the failure-detection effectiveness of FSCS-ART, and remaining consistent in high dimensional input domains. We recommend using SWFC-ART in practical software testing scenarios, where real-life programs often have high dimensional input domains and low failure rates.

Test Maturity Model Integration: Trends of Worldwide Test Maturity and Certifications

Test maturity model integration (TMMi) is popular for the maturity assessment and capability improvement of software testing practices. We present a status report for TMMi, including motivations and benefits, and discuss how companies have been ranked in its process areas.

Quadruple Factors Interference and Its Effects on Quality of Outsourcing Testing: A Case Study

The share of IT companies in the global market has been increasing and many organizations are moving to outsource their software development projects to off-shore vendors. Recently, the IT sector in Palestine has been gaining much ground with many software development companies specializing in software development outsourcing services. Maintaining high-quality and robust software products has always been a key success factor in the outsourcing sector. Software testing is considered to be the main phase in which teams can assure high quality of software products.Therefore, we did an exploratory multiple-case study and this study involvesfour Palestinian software outsourcing companiesof different domain. We provide insights about factors that affect the quality of software testing practice, and explore applied testing techniques, methods and confronted challenges. The results reveal that there is big attention toward software testing assoftware development companies are investing more effort, time, and technical resources into their quality assurance teams. Additionally, among the applied software testing methods, more attention needs to be directed towards test automation. Moreover, more theoretical and technical courses targeting software quality assurance should be included in national Palestinian universities’ curriculum. Finally, we propose a new model that can help practitioners to enhance the quality of their outsourced software products

Simulated Software Testing Process and Its Optimization Considering Heterogeneous Debuggers and Release Time

Most traditional software reliability growth models (SRGMs) assume immediate fault correction upon detection and therefore only consider fault detection process (FDP). In order to be more realistic, some researchers have tried to incorporate fault correction process (FCP) and fault introduction process (FIP) into the software reliability models. However, it is still difficult to incorporate into the analytical software reliability models some other factors, such as the different fault detection and correction capabilities of debuggers. In this paper, a simulation approach is proposed to model FDP, FIP, and FCP together considering debuggers with different contributions to fault detection rate, different fault correction rate and different fault introduction rate. Besides, this paper also constructed a cost calculation method to optimize the testing design including debuggers assignment and software release time. Some numerical examples are provided to illustrate the proposed model. The results show that the trends of FDP, FCP and FIP are consistent with the intuition to the practice of software testing, and the optimal testing resources allocation and the optimal release time can be obtained according to the proposed model.

Practices in software testing in Cameroon challenges and perspectives

AbstractMany software projects in Cameroon and elsewhere fail to produce the desired quality product due to low adoption of good software testing practices, including test automation. The fundamental questions are what are the local barriers to adopting structured approaches to specifying test cases? And what obstacles are preventing software testing from being automated? To answer these questions, this study analyzes software testing practices in Cameroon. The ultimate objective is to issue recommendations on the directions in which to direct research on software testing, in order to provide solutions promoting the adoption of good testing practices in projects, taking into account limited financial and human resources in developing countries. To do this, a survey was carried out among companies that carry out software development activities. The analysis of the results obtained reveals several interesting elements among which we can cite: (a) more than 80% of the respondents do not ensure that there has been a test other than that of the developer who does not follow a structured approach, (b) automated tests represent less than 8% of the number of tests carried out, (c) the most recurrent factors against test automation are the time required to configure or adapt the tools, the acquisition and implementation costs, costs of design, development, and maintenance of test cases.

Fast Track Technique for Software Testing and Quality Assurance Practice in Project Development Life Cycle

Fast Track Technique for Software Testing and Quality Assurance Practice in Project Development Life Cycle

On testing machine learning programs

Nowadays, we are witnessing a wide adoption of Machine learning (ML) models in many software systems. They are even being tested in safety-critical systems, thanks to recent breakthroughs in deep learning and reinforcement learning. Many people are now interacting with systems based on ML every day, e.g., voice recognition systems used by virtual personal assistants like Amazon Alexa or Google Home. As the field of ML continues to grow, we are likely to witness transformative advances in a wide range of areas, from finance, energy, to health and transportation. Given this growing importance of ML-based systems in our daily life, it is becoming utterly important to ensure their reliability. Recently, software researchers have started adapting concepts from the software testing domain (e.g., code coverage, mutation testing, or property-based testing) to help ML engineers detect and correct faults in ML programs. This paper reviews current existing testing practices for ML programs. First, we identify and explain challenges that should be addressed when testing ML programs. Next, we report existing solutions found in the literature for testing ML programs. Finally, we identify gaps in the literature related to the testing of ML programs and make recommendations of future research directions for the scientific community. We hope that this comprehensive review of software testing practices will help ML engineers identify the right approach to improve the reliability of their ML-based systems. We also hope that the research community will act on our proposed research directions to advance the state of the art of testing for ML programs.

A preliminary survey on software testing practices in Khyber PakhtunKhwa region of Pakistan

Conducted to ensure the quality of software products, the software testing process has a great significance in the software development and is the vital step of the verification and validation process. For conforming a software feature to the end user requirements, organizations rely on extensive testing procedures. Despite being the key factor, many of the software development industries/companies do not define/follow a systematic testing process. In this paper, we analyze/learn from the conducted surveys in the past and formulate a questionnaire for a survey in the northern region of Pakistan. To the best of our knowledge, no such survey has ever been conducted in Pakistan. Thus, our work can be used as a baseline for such surveys in other regions or for a survey at the country level. This survey focuses on various aspects of testing methodologies and techniques. With the analyses of the survey findings, we conclude that a testing "maturity" model (based on the cultural/organizational/political values/constraints) should be devised for the improvement of current testing practices to ensure the production of quality software. The survey is regional but the results and knowledge obtained are in line with the related surveys conducted in other regions of the globe.

Toward Understanding Students’ Learning Performance in an Object-Oriented Programming Course: The Perspective of Program Quality

This pilot study examines how students’ performance has evolved in an Object-oriented (OO) programming course and contributes to the learning analytic framework for similar programming courses in university curriculum. First, we briefly introduce the research background, a novel OO teaching practice with consecutive and iterative assignments consisting of programming and testing assignments. We propose a planned quantitative method for assessing students’ gains in terms of programming performance and testing performance. Based on real data collected from students who engaged in our course, we use trend analysis to observe how students’ performance has improved over the whole semester. By using correlation analysis, we obtain some interesting findings on how students’ programming performance correlates with testing performance, which provides persuasive empirical evidence in integrating software testing practices into an Object-oriented programming curriculum. Then, we conduct an empirical study on how students’ design competencies are represented by their program code quality changes over consecutive assignments by analyzing their submitted source code in the course system and the GitLab repository. Three different kinds of profiles are found in the students’ program quality in the OO design level. The group analysis results reveal several significant differences in their programming performance and testing performance. Moreover, we conduct systematical explanations on how students’ programming skill improvement can be attributed to their object-oriented design competency. By performing principal component analysis on software statistical data, a predictive OO metrics suite for both students’ programming performance and their testing performance is proposed. The results show that these quality factors can serve as useful predictors of students’ learning performance and can provide effective feedback to the instructors in the teaching practices.

A fault-based testing approach for VR applications

Technologies such as Virtual Reality (VR) have emerged, allowing the development of three-dimensional environments with real-time interaction. VR systems lack similarities between traditional programs, which makes it ineffective to apply traditional software testing criteria in then. Considering this motivation and the acceptance between researchers and engineers that quality is an essential factor in software development, in this paper we examine software testing practices available for the VR domain and present the possibilities for improvement to provide an automated software testing approach that can contribute to the quality assurance of VR applications.