Test Automation Framework Research Articles

Behaviour Driven Testing as an Effective Approach for Automated Testing in Continuous Integration

The global digitalization has transformed business dramatically, making it respond and adapt fast to the changing world and new challenges. The development teams face a global challenge of how to speed up product delivery time to the end user without losing product quality. Due to this, the demand for automation of routine tasks has significantly increased, because this process helps to minimize and optimize time and money for digital products delivery.Functional automation testing plays a very important role in the process of faster and more effective product delivery to the end users in a limited time. For example, test automation plays a key role when you need to test periodically the same functionality over and over again to check whether the product is ready for delivery to the end user or not, especially, when working with very complex systems.The development of the automated testing system architecture is influenced by such factors as the analysis of the possibility of coverage by automated tests, the creation of a strategy to build the automation testing on the project and the development of the automated testing system at its own. The test approach itself also plays an important role and could vary dramatically. For instance, Test-Driven Development, Behaviour-Driven Development, Keyword-Driven Testing, and Data-Driven Testing approaches.The behavioural approach (BDD) has been used to develop an automation testing system described in this work. This approach is a set of practices aimed to reduce the risks with communication between the development team and the client. There is a wide range of tools for BDD-based testing, namely Cucumber, JBehave, Gauge, and more.As a result, the automated testing system, described in this paper, has been built using Cucumber practices, which in turn allows functional verifications to be performed simultaneously at such levels as graphical interface (UI) and web service (testing API). Cucumber recognizes the specification for the tested functionality, which is described in a simple language, then looks whether testing scenarios are present in the feature files and only then runs test scripts. Each scenario consists of a set of steps that Cucumber processes and verifies against a given expected business logic.It should also be noted that this system has been integrated into Continuous Integration, based on the Jenkins service. Test automation framework integration with CI tool helps to constantly monitor the project source code repository for changes, and this enables fast feedback and high quality of the product in the process of continuous development.Manuscript received 04.06.2020

Automatic penetration testing framework based on campus network

The traditional penetration test is using the tools of scan and penetration by professional security personnels to simulate hacker attacks and find the vulnerable points in the network. It depends on the ability of personnels, especially their own experience, and may take a longtime. There are many systems in campus networks, and they are becoming more and more complex. One needs to take a long period and cost to carry out penetration tests for these services, so it is difficult to complete detections throughfully. Automatic penetration test is easy to operate, and can be carried out continuously. However, most penetration test platforms are single direction penetration or semi-automatic, lack comprehensive and flexible. In this paper, we propose a new automatic penetration testing framework (AAPF), which is loosely coupled and easily expanded. Using deep learning and artificial intelligence, AAPF can solve many key problems in the process of penetration automation. The tests in campus network show that the platform can improve the success rate of penetration test and significantly shorten the penetration time.

Automated Testing Framework for Functionality of Configuration of Complex Modern Architectures

Automated Testing Framework for Functionality of Configuration of Complex Modern Architectures

Evaluation of the Automated Testing Framework: A Case Study

Developing mobile applications without defects or in a minimum number is an important challenge for programmers and quality assurance teams. Automated software test can be the key to improve the traditional manual testing, often time consuming and repetitive. Mobile applications support user-interaction characteristics, whichareindependentfromtheapplicationlogic.Theyincludecontentpresentationornavigationfeatures,such as scroll or zoom into screens, and a rotating device. In this paper, an automated testing framework is proposed and evaluated. This framework integrates user- interaction features, historical bug information, and an interest points detector and descriptor to identify new bugs. It has shown that it works well detecting bugs associated with user-interactions.

Reinforcement Learning for Optimizing Test Case Execution in Automated Testing

This study explores the use of reinforcement learning (RL) techniques to optimize test case execution in automated testing frameworks, addressing the inefficiencies of traditional testing methods. The primary research problem involves enhancing testing efficiency, improving coverage, and reducing redundancy through intelligent RL-based optimization. The study employed a design that integrated RL algorithms into automated testing frameworks, involving the development and training of an RL model, followed by empirical evaluation. ajor findings indicate that RL-based optimization significantly reduced test case execution time, improved test coverage, and minimized redundancy compared to conventional methods. The RL model dynamically adjusted test case sequences based on real-time feedback, leading to enhanced efficiency and more comprehensive testing. The study concludes that RL techniques offer a promising approach to overcoming traditional testing limitations, demonstrating tangible benefits in real-world scenarios. To put in a nutshell, RL-based optimization effectively addresses key challenges in automated testing, offering a more adaptive and efficient strategy for test case execution.

IoT-PEN: An E2E Penetration Testing Framework for IoT

The lack of inbuilt security protocols in cheap and resource-constrained Internet of Things (IoT) devices give privilege to an attacker to exploit these device's vulnerabilities and break into the target device. Attacks like Mirai, Wannacry, Stuxnet, etc. show that a cyber-attack often comprises of a series of exploitations of victim device's vulnerabilities. Timely detection and patching of these vulnerabilities can avoid future attacks. Penetration testing helps to identify such vulnerabilities. However, traditional penetration testing methods are not End-to-End, which fail to detect multi-hosts and multi-stages attacks. Even if an individual system is secure under some threat model, the attacker can use a kill-chain to reach the target system. In this paper, we introduced first-of-its-kind, IoT-PEN, a Penetration Testing Framework for IoT. The framework follows a client-server architecture wherein all IoT nodes act as clients and “a system with resources” as a server. IoT-PEN is an End-to-End, scalable, flexible and automatic penetration testing framework for discovering all possible ways an attacker can breach the target system using target-graphs. Finally, the paper recommends patch prioritization order by identifying critical nodes, critical paths for efficient patching. Our analysis shows that IoT-PEN is easily scalable to large and complex IoT networks.

Strategies for Developing and Deploying Enterprise-Level Mobile Applications on a Large Scale: A Comprehensive Analysis

The object of the exploration in this study is the approach and methods within the building of large-scale enterprise mobile applications. The primary aim is to design a pragmatic model that will be helpful for any organization engaging the services of applications to improve its operational excellence. This capture of scholarly insight, however, goes beyond the detailed description of the particular needs and intricate challenges, such as stringent security, additional scalability, prevalent device fragmentation, and the requirement of producing consistent interaction with the current enterprise environment. The development discourse is expanded further to layer upon the manifold methodologies and technological frameworks that coordinate the large-scale development of mobile applications, in particular the agile development paradigm, the DevOps principles, and the utilization of versatile cross-platform development strategy. This process involves strategies to promote advanced cloud infrastructures and the newer containerization technologies to scale up the deployment and management with great accuracy of the mobile applications, in addition to the development of methods to maximize the performance of such applications working on several devices and operating systems. These will cover the major aspects of creating a resilient app building pipeline such as solutions towards project management processes, introducing tools that will facilitate collaboration, implementing automated testing frameworks and applying CI/CD (continuous integration and continuous deployment) methodologies. Furthermore, these questions are related to the new methods in App development, for example, serverless computing, microservices architectures and AI usage for making the decision making process better along with predictive analytics in the App lifecycle. By way of this academic investigation, we propose to provide all relevant players, namely developers, marketers, as well as venture capitalists with a deep knowledge and a set of practical solutions for building enterprise mobile applications at scale.

Towards an Automated Testing Framework for Big Data

Big data testing services are to deliver end to end testing methodologies which address our big data challenges. The testing module includes two types of functionalities. One is functional testing and second is non- functional testing. The functional testing should be accomplished at every stage of big data processing. Functional testing is nothing but the big data sources extraction testing, data migration testing and big data ecosystem. Testing which completes ETL test strategy, Map job reduce validation, multicore Data integration validation and data duplication check. On the other side the non-functional testing is to ensure that there are no quality defeat in data and no performance related issues. It covers the area for security testing, performance testing which solve the problem of monitoring and identify bottlenecks.

MAESTRO: Automated test generation framework for high test coverage and reduced human effort in automotive industry

Abstract Context The importance of automotive software has been rapidly increasing because software controls many components of motor vehicles such as smart-key system, tire pressure monitoring system, and advanced driver assistance system. Consequently, the automotive industry spends a large amount of human effort to test automotive software and is interested in automated testing techniques to ensure high-quality automotive software with reduced human effort. Objective Applying automated test generation techniques to automotive software is technically challenging because of false alarms caused by imprecise test drivers/stubs and lack of tool supports for symbolic analysis of bit-fields and function pointers in C. To address such challenges, we have developed an automated testing framework MAESTRO. Method MAESTRO automatically builds a test driver and stubs for a target task (i.e., a software unit consisting of target functions). Then, it generates test inputs to a target task with the test driver and stubs by applying concolic testing and fuzzing together in an adaptive way. In addition, MAESTRO transforms a target program that uses bit-fields into a semantically equivalent one that does not use bit-fields. Also, MAESTRO supports symbolic function pointers by identifying the candidate functions of a symbolic function pointer through static analysis. Results MAESTRO achieved 94.2% branch coverage and 82.3% MC/DC coverage on the four target modules (238 KLOC) developed by Hyundai Mobis. Furthermore, it significantly reduced the cost of coverage testing by reducing the manual effort for coverage testing by 58.8%. Conclusion By applying automated testing techniques, MAESTRO can achieve high test coverage for automotive software with significantly reduced manual testing effort.

IEC 61131-3 Software Testing: A Portable Solution for Native Applications

Programmable logic controllers (PLCs) are the most used digital systems in manufacturing industry, but there is little support for test automation of such systems. As net result, testing is mostly done manually or not at all despite the recommendations of the IEC 61131-3 Standard. Attempts to provide an automated testing framework for PLCs have been recently performed with first successful results. The most advanced and promising framework proposes an approach close to object orientation that relies on nonnative language and platform. In this article, we propose a testing library called Advanced Program Organization Unit Testing Framework written in native language, built according to the unit testing paradigm, and supporting automated testing for simple and complex scenarios of IEC 61131-3—compliant PLCs. In this article, we present such library, discuss its performance and advantages, and illustrate its application to a real case study.

A Model based Test Pattern Generation and Testing Framework for IoT Applications

The whole world is entering towards the trend of smart technology. Internet of Things ( IoT) is an important domain behind this enormous growth. A simple IoT system consists of a device or actuators or sensors, which are connected to software with the help of an internet. The embedded sensors can be monitored and managed from remote place through the network from anywhere in the world internet. There are various applications that are supported in this domain due to this feature. They are smart agriculture, Home automation, been applied in various domains like Home automation, Patient Health Monitoring, Smart City, Smart Agriculture and much more. The usage of these applications is increasing day by day, so there arises a need for verifying and validating the IoT devices in all aspects. The test automation framework that generates test pattern for various testing of IoT application domains that deploys in a sequence process of test patterns which can be easily started for the development of IoT scenarios described. To test their IoT device, there is a need for proper testing techniques for IoT applications through different IoT developers follow their own strategy. The main goal of the automation framework is to reduce the effort in the testing process and to make the test process easier for testing the IoT applications by generating various test patterns various tests depends on a number of IoT test patterns, which allows the process of various operations as the future extension.

How to Ensure Testing Robustness in Microservice Architectures and Cope with Test Smells

This paper presents the most common test smells and prevention methods against them in test automation frameworks which are used to test microservice architectures. In this scope; the necessity for test automation is discussed, and the most probable test smells in a test automation framework are listed. Finally, applied solutions to handle them are told and advantages are analyzed by investigating the results.

Automated Testing of Simulation Software in the Aviation Industry: An Experience Report

An industry-academia collaboration developed a test automation framework for aviation simulation software. The technology has been successfully deployed in several test teams.

Data residency as a service: A secure mechanism for storing data in the cloud

Recently, researchers have been working on cloud data assurance models to ensure that the data is in compliance with the policies. In such a model, the data is placed in the intended data centres without focusing on data residency issues. Thus, in order to solve such issues, we have proposed a framework known as the data resident storage (DRS) which is intended to achieve data residency protection, thus providing data residency as a service (DRaaS). The finite state machine model is developed for the purpose of verifying the methodology in terms of data location and data access. The validation is carried out in the CloudSim toolkit with the help of defined services. The developed test automation framework places the data only in data centres that are in compliance with data residency laws. Finally, this framework can be hosted by the cloud service provider to provide DRaaS to the end users.

The Design and Realization of One Automatic Test Framework for TVOS

With the development of software industry, the framework for system software becomes more and more complex, and moves in the directions towards hierarchy, modularization and flexibility. The ensure of function and performance of the whole system through traditional manual test is low efficient and high cost. The software testing is in the process of transition from manual testing to automatic testing. This study designed and realized one automatic testing frame for TVOS which could fulfill the demand of many manufacturers and cooperative research and development. Moreover, we analyzed the testing model and designed the test cases accordingly based on the characteristics of framework of TVOS.

Automatic white-box testing of first-order logic ontologies

AbstractFormal ontologies are axiomatizations in a logic-based formalism. The development of formal ontologies is generating considerable research on the use of automated reasoning techniques and tools that help in ontology engineering. One of the main aims is to refine and to improve axiomatizations for enabling automated reasoning tools to efficiently infer reliable information. Defects in the axiomatization cannot only cause wrong inferences, but can also hinder the inference of expected information, either by increasing the computational cost of or even preventing the inference. In this paper, we introduce a novel, fully automatic white-box testing framework for first-order logic (FOL) ontologies. Our methodology is based on the detection of inference-based redundancies in the given axiomatization. The application of the proposed testing method is fully automatic since (i) the automated generation of tests is guided only by the syntax of axioms and (ii) the evaluation of tests is performed by automated theorem provers (ATPs). Our proposal enables the detection of defects and serves to certify the grade of suitability—for reasoning purposes—of every axiom. We formally define the set of tests that are (automatically) generated from any axiom and prove that every test is logically related to redundancies in the axiom from which the test has been generated. We have implemented our method and used this implementation to automatically detect several non-trivial defects that were hidden in various FOL ontologies. Throughout the paper we provide illustrative examples of these defects, explain how they were found and how each proof—given by an ATP—provides useful hints on the nature of each defect. Additionally, by correcting all the detected defects, we have obtained an improved version of one of the tested ontologies: Adimen-SUMO.

Adversarial Samples on Android Malware Detection Systems for IoT Systems.

Many IoT (Internet of Things) systems run Android systems or Android-like systems. With the continuous development of machine learning algorithms, the learning-based Android malware detection system for IoT devices has gradually increased. However, these learning-based detection models are often vulnerable to adversarial samples. An automated testing framework is needed to help these learning-based malware detection systems for IoT devices perform security analysis. The current methods of generating adversarial samples mostly require training parameters of models and most of the methods are aimed at image data. To solve this problem, we propose a testing framework for learning-based Android malware detection systems (TLAMD) for IoT Devices. The key challenge is how to construct a suitable fitness function to generate an effective adversarial sample without affecting the features of the application. By introducing genetic algorithms and some technical improvements, our test framework can generate adversarial samples for the IoT Android application with a success rate of nearly 100% and can perform black-box testing on the system.

Development of Test Automation Framework for REST API Testing

This work proposes a generalized excel based test automation framework for REST API testing using groovy script. Testing is an important process in software development process, which helps to find unknown errors, failure and bugs in a software application and increases quality of the software application. With the existing process of testing is manual, where the inputs are given manually and the actual outcome or output is validated for different test case scenarios. This manual testing leads to increase in time and effort of testing, as there will be different scenarios to be tested in which different test data has to be prepared to test a particular functionality. As the functionalities in an application changes or added up, over the software development phase, there is a need to test the old functionalities after and before, integrating the new or changed functionality to the existing system. This results in testing the already tested functionalities to test again manually, which increases effort and time for the testing process. The manual testing process has to be automated, so that the testing time, effort and money can be reduced. The test automation framework is implemented using a groovy compiling tool in which the functionalities such as, read the data from input test data excel file, make a REST API request, validate the data and writing the result to the excel file has been developed. Since groovy is a flavor of java, several libraries have been imported to implement the test automation framework.

Test case generation for production systems with model-implemented fault injection consideration

Complex production systems, which have to handle product diversity and short product life cycles, can only be stable and efficient when successfully tested prior to their start-up. Also, changes in automation software within the life cycle of a production system should be verified through repetitive testing before applying the new software version. Therefore, test automation in combination with fault injection on a virtual model of the production system is seen as the next step to improve the verification and validation of a production system. This work presents an approach to achieve negative test cases for a test automation framework.

Integrated Test Automation for Evaluating a Motion-Based Image Capture System Using a Robotic Arm

The problem discussed in this paper is how to create an integrated test automation setup, in which the test engineer does not have to use a pendant to communicate with the robotic arm and a separate test script to trigger on-device test automation for testing computer vision algorithms, such as motion-based image capture. There is no communication between the pendant to control the robotic arm and the test script that launches the on-device test automation for the motion-based image capture system executing on a mobile phone. Therefore, the test engineer needs to manually trigger the programs for the robot pendant to start the robotic arm, the automation, and launch times to keep them in sync. The solution to this problem is to design a test automation program using a middleware software (such as ORiN2 middleware software provided by Denso Robotics) to bypass the dependency on the robotic pendant, which must be manually started. The programmatic usage of the ORiN2 software also helps to establish a communication between the lab computer and a robot controller. The controller access object is used by the ORiN2 middleware software to communicate with the robotic controller of a Denso robot. It directly controls the robotic arm motion by accessing the programming variables. The image capture accuracy is calculated based on the number of snapshots and the similarity of the snapshots taken if the mobile phone under test is stationary and a motion is injected in the test subject. We compared the test results of the hardware-accelerated solution with a third-party solution, and found that the hardware accelerated solution is 12% more accurate for image capture accuracy at a medium panning speed. This paper discusses the integrated test automation framework for testing motion-based image capture system using a robotic arm.