Automated Testing Approach Research Articles

Revolutionizing DevOps Security: AI and ML-Enabled Automated Testing Approaches

In modern DevOps environments, the integration of security practices poses significant challenges due to the fast-paced nature of Continuous Integration/Continuous Deployment (CI/CD) pipelines. Traditional security testing methods are usually too slow and reactive to address vulnerabilities effectively in such dynamic settings. To overcome these challenges, organizations are increasingly adopting automated security testing solutions that leverage Artificial Intelligence (AI) and Machine Learning (ML). This paper discusses AI and ML capabilities in automating security testing during DevOps. It talks about how these technologies can improve security by enabling real-time threat detection, reducing false positives, and adapting to new vulnerabilities through continuous learning. Key AI/ML-based tools and techniques, along with their integration into DevOps workflows, are also discussed in detail. It also covers the integration challenges and the potential of AI/ ML in security testing in the coming years.

Finding Cross-Rule Optimization Bugs in Datalog Engines

Datalog is a popular and widely-used declarative logic programming language. Datalog engines apply many cross-rule optimizations; bugs in them can cause incorrect results. To detect such optimization bugs, we propose an automated testing approach called Incremental Rule Evaluation (IRE), which synergistically tackles the test oracle and test case generation problem. The core idea behind the test oracle is to compare the results of an optimized program and a program without cross-rule optimization; any difference indicates a bug in the Datalog engine. Our core insight is that, for an optimized, incrementally-generated Datalog program, we can evaluate all rules individually by constructing a reference program to disable the optimizations that are performed among multiple rules. Incrementally generating test cases not only allows us to apply the test oracle for every new rule generated—we also can ensure that every newly added rule generates a non-empty result with a given probability and eschew recomputing already-known facts. We implemented IRE as a tool named Deopt, and evaluated Deopt on four mature Datalog engines, namely Soufflé, CozoDB, μZ, and DDlog, and discovered a total of 30 bugs. Of these, 13 were logic bugs, while the remaining were crash and error bugs. Deopt can detect all bugs found by queryFuzz, a state-of-the-art approach. Out of the bugs identified by Deopt, queryFuzz might be unable to detect 5. Our incremental test case generation approach is efficient; for example, for test cases containing 60 rules, our incremental approach can produce 1.17× (for DDlog) to 31.02× (for Soufflé) as many valid test cases with non-empty results as the naive random method. We believe that the simplicity and the generality of the approach will lead to its wide adoption in practice.

An integrated and automated testing approach on Inception Restnet-V3 based on convolutional neural network for leukocytes image classification.

The leukocyte is a specialized immune cell that functions as the foundation of the immune system and keeps the body healthy. The WBC classification plays a vital role in diagnosing various disorders in the medical area, including infectious diseases, immune deficiencies, leukemia, and COVID-19. A few decades ago, Machine Learning algorithms classified WBC types required for image segmentation, and the feature extraction stages, but this new approach becomes automatic while existing models can be fine-tuned for specific classifications. The inception architecture and deep learning model-based Resnet connection are integrated into this article. Our proposed method, inception Resnet-v3, was used to classify WBCs into five categories using 15.7k images. Pathologists made diagnoses of all images so a model could be trained to classify five distinct types of cells. After implementing the proposed architecture on a large dataset of 5 categories of human peripheral white blood cells, it achieved high accuracy than VGG, U-Net and Resnet. We tested our model with WBC images from additional public datasets such as the Kaagel data sets and Raabin data sets of which the accuracy was 98.80% and 98.95%. Considering the large sample sizes, we believe the proposed method can be used for improving the diagnostic performance of clinical blood examinations as well as a promising alternative for machine learning. Test results obtained with the system have been satisfying, with outstanding values for Accuracy, Precision, Recall, Specificity and F1 Score.

Visual Identification of Mobile App GUI Elements for Automated Robotic Testing.

Automated robotic testing is an emerging testing approach for mobile apps that can afford complete black-box testing. Compared with other automated testing approaches, automatic robotic testing can reduce the dependence on the internal information of apps. However, capturing GUI element information accurately and effectively from a black-box perspective is a critical issue in robotic testing. This study introduces object detection technology to achieve the visual identification of mobile app GUI elements. First, we consider the requirements of test implementation, the feasibility of visual identification, and the external image features of GUI comprehensively to complete the reasonable classification of GUI elements. Subsequently, we constructed and optimized an object detection dataset for the mobile app GUI. Finally, we implement the identification of GUI elements based on the YOLOv3 model and evaluate the effectiveness of the results. This work can serve as the basis for vision-driven robotic testing for mobile apps and presents a universal approach that is not restricted by platforms to identify mobile app GUI elements.

Model-based automated testing of JavaScript Web applications via longer test sequences

JavaScript has become one of the most widely used languages for Web development. Its dynamic and event-driven features make it challenging to ensure the correctness of Web applications written in JavaScript. A variety of dynamic analysis techniques have been proposed which are, however, limited in either coverage or scalability. In this paper, we propose a simple, yet effective, model-based automated testing approach to achieve a high code-coverage within the time budget via testing with longer event sequences. We implement our approach as an open-source tool LJS, and perform extensive experiments on 21 publicly available benchmarks. On average, LJS is able to achieve 86.5% line coverage in 10 minutes. Compared with JSDEP, a state-of-the-art breadth-first search based automated testing tool enriched with partial order reduction, the coverage of LJS is 11%–19% higher than that of JSDEP on real-world large Web applications. Our empirical findings support that proper longer test sequences can achieve a higher code coverage in JavaScript Web application testing.

Automated Visual Testing of Application User Interfaces Using Static Analysis of Screenshots

Mobile and web applications must operate and be displayed correctly on many different devices and browsers. The visual testing of web or mobile applications is usually a manual process that requires a significant amount of testing time, meaning that applications are tested only on a few devices. It is then assumed that the applications will be displayed correctly on other compatible or similar devices. This paper presents an automated visual testing method for user interfaces. The main contributions of this paper are a classification scheme for visual defects of user interfaces and the definition of an automatic visual testing method that tests applications on many different devices with varying hardware and software parameters. The method is based on an automated search for defects using heuristic and expected state prediction algorithms, which involves analyzing the resources used by applications and screenshots. The testing method works by executing applications on a full set of devices, taking a screenshot at every execution step, and analyzing each of these screenshots. The manual as well as automated testing approaches were validated on 781 of Android applications. The experimental results show that the proposed method has advantages over manual testing.

Video Game Automated Testing Approaches: An Assessment Framework

The video-game industry has recently grown from focused markets to mainstream. The advancements the industry has been enjoying motivated researchers to propose techniques and tools to support the activities across the different phases of the game development lifecycle. Game testing is one of the crucial activities within the game development process. Due to the nature of game testing, many automated game testing techniques have been proposed in the literature. However, there is no framework that could be used to aid practitioners in selecting appropriate techniques suitable for their particular development efforts. In this article, we present an attribute-based framework to classify and compare these techniques and provide such aid to practitioners. The framework is also meant to guide researchers interested in proposing new game testing techniques. The article discusses a number of prominent techniques against the framework. Analysis of the discussion reveals gaps and suggests open points for future research.

A Review of Automated Testing Approach for Software Regression Testing

One of the important parts of software development life-cycle is software testing and one of which is regression testing. Regression testing is done to verify that previously detected errors have been corrected correctly and no new errors arise as a result of incorrect corrections so that the quality of the software being tested is maintained. It is important and must be done. However, the conventional method is not efficient as it is time-consuming, not reusable, and prone to error. In this paper, a comprehensive review of the automated testing approach is presented to be used by other researchers in this field of study. The review result shows that the automated testing approach is suitable to enhance the regression testing with some plausible options of tools e.g. Selenium, SAHI, and Robot-framework. Furthermore, some plausible options of execution methods are also discussed in this paper. This review further concludes that the parallel execution method is considered as a promising choice to conduct the most efficient regression testing process.

Automated Testing Approach for Generation and Optimization of Test Cases using Hybrid Bat Algorithm

Automated Testing Approach for Generation and Optimization of Test Cases using Hybrid Bat Algorithm

An Automated Testing Approach in Data Mining System using Genetic Algorithm Framework

An Automated Testing Approach in Data Mining System using Genetic Algorithm Framework

An Empirical Analysis of Android Apps Bug and Automated Testing Approach for Android Apps

Android platforms and its applications (apps) have gained tremendous popularity recently, hence the reliability of Android apps is becoming increasingly important. Due to the novelty of the Android platform, apps are prone to errors, which would affect user experience and requires frequent bug fixes. In this paper, an empirical study on bugs in some widely-used open-source Android apps from diverse categories such as media, tools and communication were performed. Based on the findings of the empirical study, an approach for automating the testing process for detecting Android apps GUI bugs was presented. We show how the approach helped to re-discover existing bugs and find new bugs, and how it could be used to prevent certain bug categories. Our empirical study and automated testing approach have the potential to help developers increase the quality of Android apps.

CenUMs – concurrency enhanced usage models for statistical testing of complex systems with concurrent streams of use

Model-based statistical testing with Markov chain usage model (MCUMs) represents a highly automated test approach. It also boosts test planning by providing relevant parameters that aid decision making. However, MCUMs are not suitable for testing modern systems that allow concurrent streams of use. Concurrent aspects result in a state space explosion in the model. They complicate the model specification and increase the calculation time for parameters. Therefore, we introduce concurrency enhanced usage model (CenUMs). They represent a natural extension to MCUMs by composite states with regions, which are used to formalize concurrent streams of use. This allows a simplified and reasonable specification as it hides the state space explosion. We also provide configurable algorithms for test case generation that take composite states into account. Additionally, a new analysis method is outlined that leads to a significant improvement of the calculation time for parameters. An analysis of its arithmetic complexity and results from experiments are provided.

Automated Test Approach Based on All Paths Covered Optimal Algorithm and Sequence Priority Selected Algorithm

A timely and complete test is an important factor to assure the functionality and safety of a railway signal system before it is put into service. With the development of rail transportation in China, the traditional semiautomatic test methods cannot satisfy the timely and complete test requirements any longer. This paper proposes an automated model-based test method. First, colored Petri nets are used as a formal language to describe the system specification. Second, the all paths covered optimal algorithm and the sequence priority selected algorithm are proposed to generate the test cases and sequences automatically. Third, taking a typical radio block center (RBC) handover scenario as an example, the generated test cases and sequences are applied into the RBC functionality test platform. The testing result validated the feasibility and efficiency of the proposed automated test method. Compared with the random-walk-based test sequence generation algorithm, the repeatability rate of the generated test sequences is reduced by 46%. The test sequences can cover all the generated test cases, and the cases can cover all the related criteria in the function requirements specification of the Chinese Train Control System Level 3.

Achieving scalable model-based testing through test case diversity

The increase in size and complexity of modern software systems requires scalable, systematic, and automated testing approaches. Model-based testing (MBT), as a systematic and automated test case generation technique, is being successfully applied to verify industrial-scale systems and is supported by commercial tools. However, scalability is still an open issue for large systems, as in practice there are limits to the amount of testing that can be performed in industrial contexts. Even with standard coverage criteria, the resulting test suites generated by MBT techniques can be very large and expensive to execute, especially for system level testing on real deployment platforms and network facilities. Therefore, a scalable MBT technique should be flexible regarding the size of the generated test suites and should be easily accommodated to fit resource and time constraints. Our approach is to select a subset of the generated test suite in such a way that it can be realistically executed and analyzed within the time and resource constraints, while preserving the fault revealing power of the original test suite to a maximum extent. In this article, to address this problem, we introduce a family of similarity-based test case selection techniques for test suites generated from state machines. We evaluate 320 different similarity-based selection techniques and then compare the effectiveness of the best similarity-based selection technique with other common selection techniques in the literature. The results based on two industrial case studies, in the domain of embedded systems, show significant benefits and a large improvement in performance when using a similarity-based approach. We complement these analyses with further studies on the scalability of the technique and the effects of failure rate on its effectiveness. We also propose a method to identify optimal tradeoffs between the number of test cases to run and fault detection.

Automated PLC Software Testing using adapted UML Sequence Diagrams

Current production plants are highly customizable and flexible in their processes. This flexibility is mainly realized by software. It causes an increasing complexity of control software components and the need of new methods for comprehensive and automated testing approaches to ensure a requested level of quality at high efficiency. A survey among mechanical engineering companies of the industries aerospace, automotive and machine/plant automation was conducted. Aspects of requirements engineering, testing, simulation, processes, tools, etc. were addressed to reveal research demands of PLC (Programmable Logic Controller) control software development in machine/plant automation. A system architecture embedding UML sequence diagrams for testing is proposed. We further contribute transformation rules of semantically adapted UML sequence diagrams to the cyclic execution logic of PLCs for reactive systems. The approach can be applied to any IEC 61131-3 programming language of the automation control software. A prototypical realization shows proof of concept and reveals the potential for future work.

A document driven methodology for developing a high quality Parallel Mesh Generation Toolbox

This paper motivates the value of using a document driven methodology to improve the quality of scientific computing applications by illustrating the design and documentation of a Parallel Mesh Generation Toolbox (PMGT). Formal mathematical specification is promoted for writing unambiguous requirements, which can be used to judge the correctness and thus the reliability of PMGT. Mathematics is also shown to improve understandability, reusability and maintainability through modelling software modules as finite state machines. The proposed methodology includes explicit traceability between requirements, design, implementation and test cases. Traceability improves the verification of completeness and consistency and it allows for proper change management. To improve the reliability of PMGT, given the challenge that the correct solution is unknown a priori, an automated testing approach is adopted to verify the known properties of a correct solution, such as conformality and counterclockwise vertex numbering.

Investigations for minimum invasion digital only built-in "ramp" based test techniques for charge pump PLL's

Due to a number of desirable operational and design characteristics, CP-PLL's (Charge Pump Phase Locked Loops) have, in recent years become a pervasive PLL architecture. CP-PLL architectures are exploited for a variety of applications such as on chip clock generation, CRC (clock recovery circuits) and radio frequency synthesis applications. This paper describes a simple, digital only, minimally invasive and fully automated test approach for high performance CP-PLL's that can be used to provide more information about the CP-PLL function beyond that obtained through the commonly used FLT (Frequency Lock Test). The test strategy described here allows the estimation of forward path gain and relative leakage in the forward path loop components. Applications of the test are focussed towards digital only testing of fully embedded CP-PLL's, however further test modifications could yield marked test time improvements for embedded and board level CP-PLL's incorporating multiple CP currents and or multiple loop filter configurations.

Simple digital test approach for embedded charge-pumpphase-locked loops

Techniques for a simple automated test approach for high performance fully embedded charge-pump phase-locked loops (CP-PLLs) are explained. The test approach is focused towards non-invasive high volume production testing of PLLs using digital only testers in conjunction with additional on-chip circuitry.

An automated test approach for US Air Force fighter engines

In an effort to support the Air Force's new F-22 Fighter aircraft with its dual F119-PW-100 jet engines, a new approach has been taken to realize the benefits of automated jet engine testing. This paper describes an automated test approach that is currently under development and will be used by the Air Force to test the F119 jet engine; It also identifies; the equipment, methods and techniques that are being employed to accomplish this task.

Evaluation and design of an ultra-reliable distributed architecture for fault tolerance

The issues related to the experimental evaluation of an early conceptual prototype of the MAFT (multicomputer architecture for fault tolerance) architecture are discussed. A completely automated testing approach was designed to allow fault-injection experiments to be performed, including stuck-at and memory faults. Over 2000 injection tests were run and the system successfully tolerated all faults. Concurrent with the experimental evaluation, an analytic evaluation was carried out to determine if higher levels of reliability could be achieved. The lessons learned in the evaluation phase culminated in a new design of the MAFT architecture for applications needing ultrareliability. The design uses the concept of redundantly self-checking functions to address the rigid requirements proposed for a future generation of mission-critical avionics. The testing of three subsystems critical to the operation of the new MAFT is presented with close to 50-k test cycles performed over 51 different IC devices to verify the designs.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>