Test Case Design Research Articles

FSECAM: A contextual thematic approach for linking feature to multi-level software architectural components

Linking software features to code components is commonly performed during software development and maintenance, including to implement a feature, document code, design test cases, trace requirements, track changes, and support inspection of safety–critical software by government and other third parties. However, manually mapping features to code is error-prone and time consuming, even for developers familiar with a system. To overcome these challenges several studies proposed automated techniques to reduce human intervention when linking features to code components. Nonetheless, three challenges remain: (i) accuracy, (ii) cost, and (iii) explainability. Linking of irrelevant code snippets causes an extra burden of analyses. If the approach lacks explainability, then a tool is less useful for many crucial systems such as safety–critical software. Moreover, heavyweight techniques such as those that require generating execution traces of every scenario or require training deep-learning models are costly and limit small companies from integrating them into their development process.We propose a contextual thematic approach that extracts the most relevant theme properties of the feature/requirement to address the aforementioned challenges. Our experiments with two proprietary projects reveal significant enhancement of performance (precision and F1 scores are more than 50% in ideal cases) in linking features to three abstractions of code components, i.e., modules, classes, and methods. Our approach is also capable of linking commits to issues in a promising way. Contextual theme extraction enhances the subjective explainability which has not yet been solved with existing approaches. Moreover, we extract several critical characteristics of the feature documents and code structures that are important to consider in both manual and automated techniques. Finally, we present the FSECAM tool for linking features to code components, which can be immediately deployed within the development process and used without much effort and cost in linking code components and commits.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

LITERATURE REVIEW: ADVANTAGES AND DISADVANTAGES OF BLACK BOX AND WHITE BOX TESTING METHODS

Software testing methods play a crucial role in ensuring the quality, security, and performance of applications. The two main approaches often used are black box and white box testing. This article presents a comprehensive literature review of the characteristics, advantages and disadvantages of both testing methods with the aim of providing in-depth insights for practitioners and researchers in the field of software engineering. The research method used is Systematic Literature Review (SLR), which is a technique of collecting information through the process of searching for literature based on research questions, filtering search results, evaluating, and analyzing relevant research from various academic database sources to gain a comprehensive understanding. Through the analysis of various literature studies, the results of this research are the characteristics of each testing method that is more effectively used in best practice to achieve efficient testing with the coverage of literature review sources in the range of 2020 to 2024. The review of these characteristics is based on seven criteria, namely: test focus, test base, tester's point of view, test case design, program error identification method, program code knowledge, and test effectiveness. In addition, this article presents a comparison of the two methods so that the differences in testing methods can be clearly recognised in terms of how they are implemented. Based on the results of review literature, both methods are better used together in order to ensure that the functional and internal code structure of the software is functioning properly.

Impact of Critical Success Factors on Productivity Gain during Automation Testing

Software testing, a linchpin of quality assurance in software development, encompasses both manual and automation approaches. While manual testing ensures meticulous scrutiny, automation accelerates efficiency, reducing time and costs in dynamic landscape of software development. This research investigates the impact of critical success factors on productivity gain in software testing, with a focus on automation, within CMMI Level 5 companies. Drawing from a dataset comprising thirty real-time projects spanning banking, retail, and industrial applications, study explores dynamics of software development under agile and waterfall models. Interviews with developers and testers reveal insights into key aspects influencing productivity, including skills, experiences, and training. Emphasizing the need for strategic automation in projects subject to changes and maintenance, the research analyzes over thirty projects, scrutinizing variables such as manually designed test cases, execution time, and subsequent automation of test cases. Strategic selection of test cases for automation emerges as a cost-efficient practice. Results highlight a correlation between manual and automated test cases, indicating productivity gains. Research introduces a productivity gain metric, showcasing a break-even point where automation significantly reduces testing time. Overall, the findings offer a comprehensive understanding of software testing within CMMI Level 5, guiding organizations toward efficient automation practices and improved productivity.

Construction method of fault function test case for Primary-Secondary fusion switch in distribution network

Test case design is the key to effective quality inspection of electrical equipment. To tackle the challenges of inadequate scenario coverage and disparate difficulty levels in the detection of single-phase ground faults in power distribution primary and secondary fusion switch, we propose a case construction method utilizing Latin hypercube sampling (LHS) based on NSGA-II double-objective optimization. Initially, the method investigates the primary factors influencing waveform detection difficulty and determines their respective weights through measurement and calculation, thus establishing a comprehensive model for calculating difficulty coefficients. Then, based on the dual objectives of uniformity and difficulty balance, we introduce a double-objective optimization LHS generation algorithm to construct a multi-dimensional detection cases of single-phase ground fault in power distribution primary and secondary fusion switch. Finally, the difficulty coefficient calculation model and the constructed test case undergo validation using actual test results, demonstrating a 43.59% efficiency improvement on the premise of ensuring the detection efficacy.

Multi‐objective optimization‐based and fault localization‐oriented test case generation for novice programs

SummaryOnline judgment (OJ) systems are capable of evaluating program results by automatically executing test cases, significantly improving the efficiency of traditional guidance approaches. Moreover, existing studies attempt to assist novices through automated fault localization techniques to provide feedback to novices, which can help them quickly find the location of faulty statements. Among them, spectrum‐based fault localization (SBFL) techniques have been widely used for their lightweight and efficiency, which only requires coverage information and test results of test cases to conduct fault localization. However, manually constructing high‐quality test cases for a large number of OJ questions is tough work to complete. To solve this problem, we propose the novice program‐oriented Multi‐Objective Optimization‐Based Fault Localization‐Oriented Test Case Generation (MFTCG) for automatically generating test inputs. Specifically, we use multi‐objective optimization algorithms to evolve the test case in terms of both fault localization and faulty code detection capability. We conduct experiments with 8911 programs from the well‐known public OJ platform AtCoder. The results show that our proposed approach MFTCG can achieve the best fault localization performance compared with existing automated test case generation approaches in most cases and can achieve the similar faulty code detection capability compared to manually designed test cases.

C-ShipGen: a conditional guided diffusion model for parametric ship hull design

Ship design is a complex design process that may take a team of naval architects many years to complete. Improving the ship design process can lead to significant cost savings, while still delivering high-quality designs to customers. A new technology for ship hull design is diffusion models, a type of generative artificial intelligence. Prior work with diffusion models for ship hull design created high-quality ship hulls with reduced drag and larger displaced volumes. However, the work could not generate hulls that meet specific design constraints. This paper proposes a conditional diffusion model that generates hull designs given specific constraints, such as the desired principal dimensions of the hull. In addition, this diffusion model leverages the gradients from a total resistance regression model to create low-resistance designs. Five design test cases compared the diffusion model to a design optimization algorithm to create hull designs with low resistance. In all five test cases, the diffusion model was shown to create diverse designs with a total resistance less than the optimized hull, having resistance reductions over 25%. The diffusion model also generated these designs without retraining. This work can significantly reduce the design cycle time of ships by creating high-quality hulls that meet user requirements with a data-driven approach.

Optimization of White Box Testing by Utilizing Branching and Repeating Structures in Java Programs Using Base Path

Software testing is a process created to detect anomalies in the operation of a program or system in order to achieve the expected results. White box testing is a software testing method that tests the internal structure, design and program code. This research aims to produce an optimization method for white box testing in Java programs by utilizing branching and repetition structures using the basis path method, as well as analyzing the effectiveness of the proposed method in generating test cases. The Java program tested in this research includes input function calls, loops, branching, and exception handling. Test case design is carried out by applying the basis path method to achieve comprehensive coverage. The test results show that the base path method is able to produce effective test cases for testing control structures without redundancy. Test case design is assisted by flowgraph and matrix graph modeling. It is hoped that this research can contribute to the optimization of white box testing techniques.

Analysis Testing Black Box and White Box on Application To-Do List Based Web

The rapid development of information technology has led to the creation of numerous web-based applications designed to assist human activities and work. One such application is the To-Do List, which helps users manage their tasks and increase productivity. This study aims to analyze the quality of web-based To-Do List applications through black box and white box testing. The research focuses on the login and main pages of the application, where various scenarios are tested to ensure that the system functions as intended. The testing process includes designing test scenarios, creating test cases, executing the test cases, and collecting and processing test result data. The study also includes an analysis of the program's source code using flowcharts and flowgraphs to identify the number of independent logic execution paths and design test cases for white box testing. The results of the testing will help identify errors and weaknesses in the application, ensuring that the final product is of high quality.

Report from the 14th International Workshop on Automating Test Case Design, Selection, and Evaluation (A-TEST 2023)

For the past fourteen years, the Workshop on Automating Test Case Design, Selection and Evaluation (A-TEST)1 has provided a venue for researchers and industry members alike to exchange and discuss trending views, ideas, state-of-the-art, work-in-progress, and scientific results on automated testing. The previous thirteen editions of A-TEST were co-located with several conferences over time: CISTI2 (first two editions), FEDCSIS 3 (three editions), and ESEC/FSE4 (seven editions). The 14th edition took place on September 15, 2023, in Kirchberg, Luxembourg, co-located with ASE 2023 (38th IEEE/ACM International Conference on Automated Software Engineering5). The A-TEST 2023 program featured an enlightening keynote on quantum software testing, followed by three technical sessions in which the authors of 6 full papers and 4 short papers dived into various aspects of automated testing, from neuroevolution and fuzzing techniques to GUI-based software testing and test case recommendations. This short report summarizes the keynote and paper presentations at A-TEST 2023.

PENGUJIAN BLACKBOX PADA SISTEM PEMESANAN UNTUK SALES ORDER DI PT BUKIT MURIA JAYA BERBASIS EQUIVALENCE PARTITIONS

This research aims to conduct blackbox testing on the sales order booking system at PT. BMJ based on the concept of equivalence partitions. The blackbox testing method is employed to assess the functionality of the system without considering its internal implementation. The study identifies representative equivalence partitions within the input and output domains of the sales order booking system and utilizes these partitions to design test cases. Through a series of tests, the system's performance is analyzed under various input conditions representing each partition. The testing results provide a better understanding of the system's reliability, identify potential defects, and offer recommendations for further improvement. This study contributes to enhancing the quality of the sales order booking system at PT. BMJ through a systematic and structured blackbox testing approach.

Investigating the graphical IEC 61131-3 language impact on test case design and evaluation of mechatronic apprentices

Abstract Mechatronic on-site technicians are responsible for maintenance and thus adequate testing of automated production systems. Hence, they must derive test cases from the IEC 61131-3 control code, which requires systematic testing skills. But despite the industry’s high demand for such skills, testing is not established in mechatronic apprentices’ educational curriculum. For the design of a teaching strategy for mechatronic apprentices on testing, this paper investigates the impact of the IEC 61131-3 language on test case design and evaluation. Comparing Sequential Function Chart (SFC) and Function Block Diagram, a trend towards SFC in apprentices’ self-perceived and real competence is shown.

Combinatorial Test Case Generation Based on ROBDD and Improved Particle Swarm Optimization Algorithm

In applications of software testing, the cause–effect graph method is an approach often used to design test cases by analyzing various combinations of inputs with a graphical approach. However, not all inputs have equal impacts on the results, and approaches based on exhaustive testing are generally time-consuming and laborious. As a statute-based software-testing method, combinatorial testing aims to select a small but effective number of test cases from the large space of all possible combinations of the input values for the software to be tested, and to generate a set of test cases with a high degree of coverage and high error detection capability. In this paper, the reduced ordered binary decision diagram is utilized to simplify the cause–effect graph so as to reduce the numbers of both the inputs and test cases, thereby saving the testing cost. In addition, an improved particle swarm optimization algorithm is proposed to significantly reduce the computation time needed to generate test cases. Experiments on several systems show that the proposed method can generate excellent results for test case generation.

Integrated Project Development through Combined Theory and Practices of Core Courses focusing on Software Development Skills: Integrated Learning Framework

The National Education Policy promotes moving from the conventional content-heavy and memorization learning practice towards holistic learning/integrated learning. It imparts a creative and multidisciplinary curriculum that focuses equally on curriculum and assessment. All educational establishments assess students using written examinations, quizzes, seminars, term paper writing, and course projects. A semester typically includes 4-5 courses, and students must earn credits for these courses by scoring a good Cumulative Grade Point Average (CGPA). As course projects offer depth knowledge/holistic learning/lifelong learning of a course for the student, many courses include course projects as one of the activities in the course. If all the courses are intended to include course projects as a mandatory pedagogy, it will be difficult for students to acquire in-depth knowledge and required skills while also dealing with stress. So we are proposing an integrated learning framework by applying the theory and practices of two core courses- Software Engineering and Web Technologies to develop a web application. This integrated learning focuses on developing software development and software testing skills in computer science for undergraduate students pursuing a bachelor of engineering degree. This framework alleviated the pressure on students during placement and created job opportunities in software development. The framework consists of three important phases- The first phase includes the identification of the problem as a need for customers, writing requirements and analyzing the same. Students apply modular design principles and break down the codebase into distinct modules. This technique enhances code organization, reusability, and maintainability. The second phase focused on developing the front end by harnessing the power of Angular, a leading web framework, to craft a sleek and interactive user interface. The backend is built using Node.js, which serves as the foundation, enabling the software system to cater to highperformance server environments. These modules communicated seamlessly through well-defined APIs, facilitating the integration of various components within the application, ultimately delivering a seamless and responsive user experience. An industry expert conducted a workshop on Angularsoftware testing workshop was conducted for students by industry experts to expose the students to designing test cases, test plans, and testing strategies. The hands-on experience on testing tools was provided during the workshop. Faculty reviews are conducted on each phase, and rubrics-based assessment is done on each phase. Approximately sixty teams created web-based applications for real-world scenarios. Positive aspects of the framework in feedback indicated that more than 87% of the students agreed that they could apply Software engineering principles and practices such as requirements management. modular design and testing in web applications. Also, more than 85% of students acquire skills from code-to-web design mastery by developing web applications in Angular Node.js and backend implementation. This framework helped to improve teamwork, presentation and communication skills. Confidence in software development improved to a greater extent. The design and implementation of the framework met the stated outcome of the courses. The student's academic performance improved by 10% compared to the previous year when students were not involved in the integrated project development. Keywords— Framework, learning, practices, project, skills, technology, testing, web

Effects of Peer Teaching Method in Software Testing Course in Comparison with Traditional Teaching Method

This action research project incorporated a peer teaching model for creating an active learning environment for teaching the course “Software Verification, Validation, and Testing”. Topics covered in the course included an overview of software testing, software quality, types of software testing, test case design techniques, test documentation, software test frameworks; model-based testing, software verification, and validation. The research aimed to explore to what extent peer teaching would increase students’ learning motivation, and engagement, develop critical thinking skills, and their performance. The study was conducted for one semester and data were generated using questionnaires, test scores, and student activity records as instruments. The test score data is collected from 10 students taking the course. The data were analyzed using qualitative (textual data obtained from open-ended responses) and quantitative (numerical data from questionnaires, and test scores) methods. The result showed that the peer teaching approach has a positive effect on the attitudes of students toward software testing courses compared to lecture mode instruction.

Innovative Integration of Phase Change Materials and Conceptional Design of Test Cases – New Products for the Building Envelope

The building sector contributes to a great part of energy consumption and has a major impact on climate in Europe. Development of climate-friendly and energy-efficient products is required to reach low and net-zero energy buildings and to facilitate the energy transition. The building envelope is critical in defining the demand for space heating and cooling as well as ensuring thermal comfort and indoor environmental quality through proper ventilation. This paper presents the conceptual test cases integrating phase change materials (PCMs) in the building envelope. PCM absorbs, stores, and releases the heat energy in a controlled manner taking advantage of the temperature difference between night and day. Hence PCM can support the thermal management of the building by minimizing unwanted heat gains and losses while integrated into walls or support the energy efficiency of ventilation while supplying fresh air. Smart ventilated heat harvesting windows and multifunctional sandwich panels are developed within the EU-funded iclimabuilt project. Additionally, commercial air-guided PCM storages are presented as ready-to-install products for decentralized ventilation and heat recovery systems.

Automated and Efficient Test-Generation for Grid-Based Multiagent Systems

Automatic generation of random test inputs is an approach that can alleviate the challenges of manual test case design. However, random test cases may be ineffective in fault detection and increase testing cost, especially in systems where test execution is resource- and time-consuming. To remedy this, the domain knowledge of test engineers can be exploited to select potentially effective test cases. To this end, test selection constraints suggested by domain experts can be utilized either for filtering randomly generated test inputs or for direct generation of inputs using constraint solvers. In this paper, we propose a domain specific language (DSL) for formalizing locality-based test selection constraints of autonomous agents and discuss the impact of test selection filters, specified in our DSL, on randomly generated test cases. We study and compare the performance of filtering and constraint solving approaches in generating selective test cases for different test scenario parameters and discuss the role of these parameters in test generation performance. Through our study, we provide criteria for suitability of the random data filtering approach versus the constraint solving one under the varying size and complexity of our testing problem. We formulate the corresponding research questions and answer them by designing and conducting experiments using QuickCheck for random test data generation with filtering and Z3 for constraint solving. Our observations and statistical analysis indicate that applying filters can significantly improve test efficiency of randomly generated test cases. Furthermore, we observe that test scenario parameters affect the performance of the filtering and constraint solving approaches differently. In particular, our results indicate that the two approaches have complementary strengths: random generation and filtering works best for large agent numbers and long paths, while its performance degrades in the larger grid sizes and more strict constraints. On the contrary, constraint solving has a robust performance for large grid sizes and strict constraints, while its performance degrades with more agents and long paths.

Maximizing Test Coverage for Security Threats Using Optimal Test Data Generation

As time continues to advance, the need for robust security threat mitigation has become increasingly vital in software. It is a constant struggle to maximize test coverage through optimal data generation. We conducted explanatory research to maximize test coverage of security requirements as modeled in the structured misuse case description (SMCD). The acceptance test case is designed through the structured misuse case description for mitigation of security threats. Mal activity is designed from SMCD upon which constraints are specified in object constraint language (OCL) in order to minimize human dependency and improve consistency in the optimal test case design. The study compared two state-of-the-art test coverage maximization approaches through optimal test data generation. It was evident through the results that MC/DC generated optimal test data, i.e., n + 1 test conditions in comparison to the decision coverage approach, i.e., 2n test conditions for security threats. Thus, MC/DC resulted in a significantly lower number of test cases yet maximized test coverage of security threats. We, therefore, conclude that MC/DC maximizes test coverage through optimal test data in comparison to decision coverage at the design level for security threat mitigation.

User acceptance test for software development in the agricultural domain using natural language processing

ABSTRACT Software test case design is one of the most challenging activities since many actors with different backgrounds must cover most of the user’s needs and expectations. In the agricultural domain tasks can be done in very different ways since practices vary worldwide. Thus, in this context, it is very hard to design test cases to validate requested functionality that automatizes some farm tasks. This paper proposes an approach to make the testing step easier, designing User Acceptance Tests (UATs) from requirements captured through scenarios. The scenarios capture the knowledge of different stakeholders (farmers) and using natural language processing tools, the approach proposed to consolidate the set of scenarios in a consistent and coherent base of knowledge organised in a tree, from where the design of test cases is extracted using the Task/Method model, a tool from the Artificial Intelligence.

Test Case Design Of Security Threats Through State Machine OCL

Software testing is one activity for Software Quality Assurance (SQA). One way to test is through Black box test cases at design level. Since UML is a de-facto design language and is one of the most used design language now a day. Therefore, there is need to design black box test cases using UML behavioral diagrams. Object Constraint Language (OCL) is a formal language to write constraints on UML diagrams as it is unambiguous. Although test cases against robustness have been designed through OCL expressions but no test cases as yet have been designed against security specific authentication, access control and availability. Specification and testing of security specific authentication, access Control and availability through OCL. An experiment is performed with a hypothesis ‘if Security Specific Authentication, Access Control and Availability is specified through OCL in state diagram then Test cases can be designed’. We have specifiedsecurity in OCL then same OCL expressions are used to model security in UML state diagram. Furthermore, mutation testing is performed on OCL expressions. Then ECP is done using same extracted mutants. Security specific authentication, access control and availability is specified in OCL and then test cases are designed using a black box testing technique Equivalence Class Partitioning. Hypothesis results shows that one can design test cases through OCL. This thesis shows that Specification and Testing of Security specific authentication, access control and availability through OCL.

Towards understanding students’ sensemaking of test case design

Context:Software testing is the most used technique for quality assurance in industry. However, in computer science education software testing is still treated as a second-class citizen and students are unable to test their software well enough. One reason for this is that teaching the subject of software testing is difficult as it is a complex intellectual activity for which students need to allocate multiple cognitive resources at the same time. A myriad of primary and secondary studies have tried to solve this problem in education, however still with very limited results. Objective:Before we can design interventions to improve our pedagogical approaches, we need to gain more in-depth understanding and recognition of sensemaking as it is happening when students design test cases. Method:An initial exploratory study identified four different sensemaking approaches used by students while creating test models. In this paper we present a follow-up study with 50 students from a large university in Spain. The used methodology was based on the previous study with the improvements that originated from its evaluation. We asked the participants to create a test model based on a description of a test problem using a specialized web-based tool for modeling test cases. We measured how well these models fit the test problem, the sensemaking process that students went through when creating the models, and the students’ perception of the modeling task. The participants received no compensation for their efforts, and we scheduled the experiment during a regular class. Apart from the created models and their metadata, we also collected recordings of the students’ computer screens made during the experiment and used a questionnaire to study their perspectives on the assignment. All the collected textual, graphical, and video data was analyzed using an iterative inductive analysis process to allow new information about the different sensemaking approaches to emerge. Results:We gained better insights into the sensemaking processes of students while modeling test cases for a problem. The results enabled us to refine our previous findings, and we identified new sensemaking approaches. Conclusions:Based on these results, we can further investigate ways to influence the sensemaking process in education, the possible misconceptions that have a negative influence on it, and the desired mental model we want our students to have to design test cases.