Continuous Integration Testing Research Articles

Security Development Lifecycle-based Adaptive Reward Mechanism for Reinforcement Learning in Continuous Integration Testing Optimization

Security Development Lifecycle-based Adaptive Reward Mechanism for Reinforcement Learning in Continuous Integration Testing Optimization

Challenges and Solutions in Implementing Continuous Integration and Continuous Testing for Agile Quality Assurance

Challenges and Solutions in Implementing Continuous Integration and Continuous Testing for Agile Quality Assurance

Comparative study of machine learning test case prioritization for continuous integration testing

Comparative study of machine learning test case prioritization for continuous integration testing

Just-in-time defect prediction for mobile applications: using shallow or deep learning?

Just-in-time defect prediction (JITDP) research is increasingly focused on program changes instead of complete program modules within the context of continuous integration and continuous testing paradigm. Traditional machine learning-based defect prediction models have been built since the early 2000s, and recently, deep learning-based models have been designed and implemented. While deep learning (DL) algorithms can provide state-of-the-art performance in many application domains, they should be carefully selected and designed for a software engineering problem. In this research, we evaluate the performance of traditional machine learning algorithms and data sampling techniques for JITDP problems and compare the model performance with the performance of a DL-based prediction model. Experimental results demonstrated that DL algorithms leveraging sampling methods perform significantly worse than the decision tree-based ensemble method. The XGBoost-based model appears to be 116 times faster than the multilayer perceptron-based (MLP) prediction model. This study indicates that DL-based models are not always the optimal solution for software defect prediction, and thus, shallow, traditional machine learning can be preferred because of better performance in terms of accuracy and time parameters.

A Bibliometric Analysis of DevOps Metrics

DevOps has become an important set of organisational and technical practices in software development under the agile umbrella. Many efforts are still being made in this field, mainly focused on the inclusion of metrics for monitoring progress. Gathering metrics is a difficult task, but it can provide insight into the performance of the software delivery process. The current status of the definition, application and implementation of metrics in DevOps projects and processes is of interest to software practitioners. Thus, the objective of this article is to analyze documents regarding the impact of metrics in DevOps projects. 103 documents were obtained from the Scopus database to analyze them through the bibliometric method considering several aspects. The bibliometric analysis performed included author analysis, author affiliation, authors’ countries, keyword analysis, citation analysis, and network analysis. The results indicate that DevOps research is not centralized in a specific group of researchers. Moreover, the most significant contributions of DevOps are related to continuous integration, software design, and software testing. The bibliometric analysis presented in this article helps to identify the current state of the DevOps literature and provides an insightful discussion of future research trends.

Comparative Study of Machine Learning Test Case Prioritization for Continuous Integration Testing

Comparative Study of Machine Learning Test Case Prioritization for Continuous Integration Testing

Sparse reward for reinforcement learning‐based continuous integration testing

AbstractReinforcement learning (RL) has been used to optimize the continuous integration (CI) testing, where the reward plays a key role in directing the adjustment of the test case prioritization (TCP) strategy. In CI testing, the frequency of integration is usually very high, while the failure rate of test cases is low. Consequently, RL will get scarce rewards in CI testing, which may lead to low learning efficiency of RL and even difficulty in convergence. This paper introduces three rewards to tackle the issue of sparse rewards of RL in CI testing. First, the historical failure density‐based reward (HFD) is defined, which objectively represents the sparse reward problem. Second, the average failure position‐based reward (AFP) is proposed to increase the reward value and reduce the impact of sparse rewards. Furthermore, a technique based on additional reward is proposed, which extracts the test occurrence frequency of passed test cases for additional rewards. Empirical studies are conducted on 14 real industry data sets. The experiment results are promising, especially the reward with additional reward can improve NAPFD (Normalized Average Percentage of Faults Detected) by up to 21.97%, enhance Recall with a maximum of 21.87%, and increase TTF (Test to Fail) by an average of 9.99 positions.

Adaptive Reward Computation in Reinforcement Learning-Based Continuous Integration Testing

Reinforcement learning (RL) has been applied to prioritizing test cases in Continuous Integration (CI) testing, where the reward plays a crucial role. It has been demonstrated that historical information-based reward function can improve the effectiveness of the test case prioritization (TCP). However, the inherent character of frequent iterations in CI can produce a considerable accumulation of historical information, which may decrease TCP efficiency and result in slow feedback. In this paper, the partial historical information is considered in the reward computation, where sliding window techniques are adopted to capture the possible efficient information. Firstly, the fixed-size sliding window is introduced to set a fixed length of recent historical information for each CI test. Then dynamic sliding window techniques are proposed, where the window size is continuously adaptive to each CI testing. Two methods are proposed, the test suite-based dynamic sliding window and the individual test case-based dynamic sliding window. The empirical studies are conducted on fourteen industrial-level programs, and the results reveal that under limited time, the sliding window-based reward function can effectively improve the TCP effect, where the NAPFD (Normalized Average Percentage of Faults Detected) and Recall of the dynamic sliding windows are better than that of the fixed-size sliding window. In particular, the individual test case-based dynamic sliding window approach can rank 74.18% failed test cases in the top 50% of the sorting sequence, with 1.35% improvement of NAPFD and 6.66 positions increased in TTF (Test to Fail).

Monitoring reconstruction software in LHCb

The LHCb detector at the LHC is currently undergoing a major upgrade to increase its full detector read-out rate to 30 MHz. In addition to the detector hardware modernisation, the new trigger system will be software-only. The code base of the new trigger system must be thoroughly tested for data flow, functionality and physics performance. Currently, the testing procedure is based on a system of nightly builds and continuous integration tests of each new code development. The continuous integration tests are now extended to test and evaluate high-level quantities related to LHCb’s physics program, such as track reconstruction and particle identification, which is described in this paper. Before each merge request, the differences after the change in code are shown and automatically compared using an interactive visualisation tool, allowing easy verification of all relevant quantities. This approach gives an extensive control over the physics performance of the new code resulting into better preparation for data taking with the upgraded LHCb detector at Run 3.

Behaviour Driven Testing as an Effective Approach for Automated Testing in Continuous Integration

The global digitalization has transformed business dramatically, making it respond and adapt fast to the changing world and new challenges. The development teams face a global challenge of how to speed up product delivery time to the end user without losing product quality. Due to this, the demand for automation of routine tasks has significantly increased, because this process helps to minimize and optimize time and money for digital products delivery.Functional automation testing plays a very important role in the process of faster and more effective product delivery to the end users in a limited time. For example, test automation plays a key role when you need to test periodically the same functionality over and over again to check whether the product is ready for delivery to the end user or not, especially, when working with very complex systems.The development of the automated testing system architecture is influenced by such factors as the analysis of the possibility of coverage by automated tests, the creation of a strategy to build the automation testing on the project and the development of the automated testing system at its own. The test approach itself also plays an important role and could vary dramatically. For instance, Test-Driven Development, Behaviour-Driven Development, Keyword-Driven Testing, and Data-Driven Testing approaches.The behavioural approach (BDD) has been used to develop an automation testing system described in this work. This approach is a set of practices aimed to reduce the risks with communication between the development team and the client. There is a wide range of tools for BDD-based testing, namely Cucumber, JBehave, Gauge, and more.As a result, the automated testing system, described in this paper, has been built using Cucumber practices, which in turn allows functional verifications to be performed simultaneously at such levels as graphical interface (UI) and web service (testing API). Cucumber recognizes the specification for the tested functionality, which is described in a simple language, then looks whether testing scenarios are present in the feature files and only then runs test scripts. Each scenario consists of a set of steps that Cucumber processes and verifies against a given expected business logic.It should also be noted that this system has been integrated into Continuous Integration, based on the Jenkins service. Test automation framework integration with CI tool helps to constantly monitor the project source code repository for changes, and this enables fast feedback and high quality of the product in the process of continuous development.Manuscript received 04.06.2020

A systematic study of reward for reinforcement learning based continuous integration testing

Continuous integration(CI) testing is characterized by continually changing test cases, limited execution time, and fast feedback, where the classical test prioritization approaches are no longer suitable. Based on the essence of continuous decision mechanism, reinforcement learning(RL) is suggested for prioritizing test cases in CI testing, in which the reward plays a crucial role. In this paper, we conducted a systematic study of the reward function and reward strategy in CI testing. In terms of reward function, the whole historical execution information of test cases is used with the consideration of the failure times and failure distribution. Further considering the validity of historical information, partial historical information is used by proposing a time-window based approach. In terms of reward strategy which means how to reward, three strategies are introduced, i.e., total reward, partial reward, and fuzzy reward. The empirical study is conducted on four industrial-level programs, and the results reveal that using the reward function with historical information improves the Recall by on average 13.21% when compared with existing TF(Test Case Failure) reward function, and the fuzzy reward strategy is more flexible and improve the NAPFD(Normalized Average Percentage of Faults Detected) by on average 3.43% when compared with the other two strategies.

A Framework for Continuous Regression and Integration Testing in IoT Systems Based on Deep Learning and Search-Based Techniques

Tremendous systems are rapidly evolving based on the trendy Internet of Things (IoT) in various domains. Different technologies are used for communication between the massive connected devices through all layers of the IoT system, causing many security and performance issues. Regression and integration testing are considered repeatedly, in which the vast costs and efforts associated with the frequent execution of these inflated test suites hinder the adequate testing of such systems. This necessitates the focus on exploring innovative scalable testing approaches for large test suites in IoT-based systems. In this paper, a scalable framework for continuous integration and regression testing in IoT-based systems (IoT-CIRTF) is proposed, based on IoT-related criteria for test case prioritization and selection. The framework utilizes search-based techniques to provide an optimized prioritized set of test cases to select from. The selection is based on a trained prediction model for IoT standard components using supervised deep learning algorithms to continuously ensure the overall reliability of IoT-based systems. The experiments are held on two GSM datasets. The experimental results achieved prioritization accuracy up to 90% and 92% for regression testing and integration testing respectively. This provides an enhanced and efficient framework for continuous testing of IoT-based systems, as per IoT-related criteria for the prioritization and selection purposes.

CPL library — A minimal framework for coupled particle and continuum simulation

We present an open-source library for coupling particle codes, such as molecular dynamics (MD) or the discrete element method (DEM), and grid based computational fluid dynamics (CFD). The application is focused on domain decomposition coupling, where a particle and continuum software model different parts of a single simulation domain with information exchange. This focus allows a simple library to be developed, with core mapping and communication handled by just four functions. Emphasis is on scaling on supercomputers, a tested cross-language library, deployment with containers and well-documented simple examples. Building on this core, a template is provided to facilitate the user development of common features for coupling, such as averaging routines and functions to apply constraint forces. The interface code for LAMMPS and OpenFOAM is provided to both include molecular detail in a continuum solver and model fluids flowing through a granular system. Two novel development features are highlighted which will be useful in the development of the next generation of multi-scale software: (i) The division of coupled code into a smaller blocks with testing over a range of processor topologies. (ii) The use of coupled mocking to facilitate coverage of various parts of the code and allow rapid prototyping. These two features aim to help users develop coupled models in a test-driven manner and focus on the physics of the problem instead of just software development. All presented code is open-source with detailed documentation on the dedicated website (cpl-library.org) permitting useful aspects to be evaluated and adopted in other projects. Program summaryProgram Title:CPLLIBRARYProgram Files doi:http://dx.doi.org/10.17632/9dh8w97d2x.1Licensing provisions: GPLv3Programming languages: Fortran/C/C++/PythonExternal routines/libraries: Message Passing Interface (MPI)Nature of problem: Coupling of particle and continuum software to enable simulations not possible with either code alone. In particular, handling communication and interaction of computational fluid dynamics (CFD) software with either molecular dynamics (MD) or discrete element method (DEM) solvers on high performance computing (HPC) platforms.Solution method: A shared library with a minimal set of functions to enable coupling, together with an entire infrastructure to facilitate development of validated coupled software. This includes a minimal Python interface to encourage mock testing, libraries to help develop coupled tools along with pre-coupled examples including OpenFOAM, LAMMPS, Flowmol (Smith, 2014) and interactive plotting using wxPython and matplotlib.Unusual features: Minimal interface with simple setup. A CPL_Mocks framework to facilitate debugging and test driven development. Communication established between independent executables using MPI_Open_Port, reducing required changes to core source code. Coupled codes track the git repository with validation through continuous integration testing and deployment on DockerHub. All communication based on MPI_Cart and MPI_Graph so optimisation is possible through MPI implementation on supercomputers.

Design and Implementation of SFCI: A Tool for Security Focused Continuous Integration

Software security is a component of software development that should be integrated throughout its entire development lifecycle, and not simply as an afterthought. If security vulnerabilities are caught early in development, they can be fixed before the software is released in production environments. Furthermore, finding a software vulnerability early in development will warn the programmer and lessen the likelihood of this type of programming error being repeated in other parts of the software project. Using Continuous Integration (CI) for checking for security vulnerabilities every time new code is committed to a repository can alert developers of security flaws almost immediately after they are introduced. Finally, continuous integration tests for security give software developers the option of making the test results public so that users or potential users are given assurance that the software is well tested for security flaws. While there already exists general-purpose continuous integration tools such as Jenkins-CI and GitLab-CI, our tool is primarily focused on integrating third party security testing programs and generating reports on classes of vulnerabilities found in a software project. Our tool performs all tests in a snapshot (stateless) virtual machine to be able to have reproducible tests in an environment similar to the deployment environment. This paper introduces the design and implementation of a tool for security-focused continuous integration. The test cases used demonstrate the ability of the tool to effectively uncover security vulnerabilities even in open source software products such as ImageMagick and a smart grid application, Emoncms.

Terrainbento 1.0: a Python package for multi-model analysis in long-term drainage basin evolution

Abstract. Models of landscape evolution provide insight into the geomorphic history of specific field areas, create testable predictions of landform development, demonstrate the consequences of current geomorphic process theory, and spark imagination through hypothetical scenarios. While the last 4 decades have brought the proliferation of many alternative formulations for the redistribution of mass by Earth surface processes, relatively few studies have systematically compared and tested these alternative equations. We present a new Python package, terrainbento 1.0, that enables multi-model comparison, sensitivity analysis, and calibration of Earth surface process models. Terrainbento provides a set of 28 model programs that implement alternative transport laws related to four process elements: hillslope processes, surface-water hydrology, erosion by flowing water, and material properties. The 28 model programs are a systematic subset of the 2048 possible numerical models associated with 11 binary choices. Each binary choice is related to one of these four elements – for example, the use of linear or nonlinear hillslope diffusion. Terrainbento is an extensible framework: base classes that treat the elements common to all numerical models (such as input/output and boundary conditions) make it possible to create a new numerical model without reinventing these common methods. Terrainbento is built on top of the Landlab framework such that new Landlab components directly support the creation of new terrainbento model programs. Terrainbento is fully documented, has 100 % unit test coverage including numerical comparison with analytical solutions for process models, and continuous integration testing. We support future users and developers with introductory Jupyter notebooks and a template for creating new terrainbento model programs. In this paper, we describe the package structure, process theory, and software implementation of terrainbento. Finally, we illustrate the utility of terrainbento with a benchmark example highlighting the differences in steady-state topography between five different numerical models.

MAUS: the MICE analysis user software

The Muon Ionization Cooling Experiment (MICE) collaboration has developed the MICE Analysis User Software (MAUS) to simulate and analyze experimental data. It serves as the primary codebase for the experiment, providing for offline batch simulation and reconstruction as well as online data quality checks. The software provides both traditional particle-physics functionalities such as track reconstruction and particle identification, and accelerator physics functions, such as calculating transfer matrices and emittances. The code design is object orientated, but has a top-level structure based on the Map-Reduce model. This allows for parallelization to support live data reconstruction during data-taking operations. MAUS allows users to develop in either Python or C++ and provides APIs for both. Various software engineering practices from industry are also used to ensure correct and maintainable code, including style, unit and integration tests, continuous integration and load testing, code reviews, and distributed version control. The software framework and the simulation and reconstruction capabilities are described.

Continuous Performance Benchmarking Framework for ROOT

Foundational software libraries such as ROOT are under intense pressure to avoid software regression, including performance regressions. Continuous performance benchmarking, as a part of continuous integration and other code quality testing, is an industry best-practice to understand how the performance of a software product evolves. We present a framework, built from industry best practices and tools, to help to understand ROOT code performance and monitor the efficiency of the code for several processor architectures. It additionally allows historical performance measurements for ROOT I/O, vectorization and parallelization sub-systems.

A learning algorithm for optimizing continuous integration development and testing practice

SummaryContinuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large‐scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault‐detection effectiveness of continuous integration tests using historical test records and combining this information with coverage‐based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest‐all approach. The results further demonstrate no significant reduction in fault‐detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.

An Introduction to Applied Bioinformatics: a free, open, and interactive text.

Statement of need: Due to the increasing rate of biological data generation, bioinformatics is rapidly growing as a field and is now an essential part of scientific advances in human health and environmental sciences. Online and publicly accessible resources for learning bioinformatics exist (e.g., Rosalind, (Searls, 2012, 2014)), and there are excellent textbooks and courses in the area, some focused heavily on theory (Durbin, Eddy, Krogh, & Mitchison, 1998; Felsenstein, 2003), and others geared toward learning specific skills such as Python programming or the Unix shell (Dunn & Haddock, 2010; Wilson, 2016). An Introduction to Applied Bioinformatics (IAB) is a free, online bioinformatics text that bridges the gap between theory and application by teaching fundamentals of bioinformatics in the context of their implementation, using an interactive framework based on highly relevant tools including Python 3, Jupyter Notebooks, and GitHub. IAB is geared toward students who are completely new to bioinformatics, though having completed an introductory course (or book) in both Computer Science and Biology are useful prerequisites. IAB readers begin on the project website. While it is possible to view the content statically from this page, we recommend that readers work interactively by installing IAB. Readers progress through chapters that introduce fundamental topics, such as sequence homology searching and multiple sequence alignment, and presents their Python 3 implementation. Because the content is presented in Jupyter Notebooks, students can edit and execute the code, for example to explore how changing k-word size or an alignment gap penalty might impact the results of a database search. The Python code that readers interact with is intended for educational purposes, where the implementation is made as simple as possible, sometimes at the cost of computational efficiency. Chapters therefore also include examples of performing the same analyses with scikit-bio, a production-quality bioinformatics Python 3 library. This enables a rapid transition from learning theory, or how an algorithm works, to applying techniques in a real-world setting. IAB additionally contains Wikipedia-style “Edit” links in each section of the text. When one of these links is followed, the reader is taken to the GitHub online editor where they can submit a pull request to modify content or code. Readers are therefore introduced to GitHub through a user-friendly web interface, and can begin building their GitHub activity history (commonly reviewed by bioinformatics hiring managers). Finally, every time a change is proposed via GitHub, all of the executable content of IAB is automatically tested. This continuous integration testing ensures that IAB example code remains functional as changes are introduced, solving an issue that plagues printed applied computational texts (for example because they describe an outdated software interface). IAB evolved from lecture materials developed by Dr. Caporaso for an introductory bioinformatics course targeted toward computer science and biology undergraduates (typically juniors or seniors) at Northern Arizona University. Since the early stages of its development, it has been used to teach at least ten courses and short (e.g., one day) bioinformatics workshops. As it became clear that the content and format was useful for teaching bioinformatics, Dr. Caporaso applied for and received grants from the Arizona Technology and Research Initiative and the Alfred P Sloan Foundation to further develop the resource.The content was originally written in Jupyter Notebooks, but as the project grew, it became difficult to maintain the notebooks and in particular to review submissions from others. The Jupyter Notebooks were transitioned to markdown files which are now the source for static HTML and Jupyter Notebook renderings of the content. The current version of IAB contains six chapters covering fundamental concepts and their applications. It is a dynamic resource that will be expanded, revised and updated over time. Its lifecycle is thus more similar to an active software project than a textbook: a practical approach to education in a rapidly changing field.

Pixel readout chip software emulators for the YARR DAQ system upgrade

The Yet Another Rapid Readout (YARR) system is a DAQ system designed for pixel readout chips including the current generation ATLAS FE-I4 chip and the next generation RD53A chip, which is part of the development of new Pixel detector technology to be implemented in High-Luminosity Large Hadron Collider experiments. YARR utilises a PCI-e FPGA card which acts as a simple gateway to pipe all data from pixel readout chips via the high speed PCI-e connection into the host system’s memory. All data processing is done on a software level in the host CPU(s), utilising a data-driven, multi-threaded, parallel processing paradigm. YARR has recently been upgraded to interface with software emulators of pixel readout chips. These emulators offer many benefits: quick development of DAQ software; expansion of the developer base to users without access to readout hardware; preparation of DAQ software for upcoming readout chips, such as RD53A; implementation of Continuous Integration and unit tests to ensure code quality and maintainability. The design and capabilities of the FE-I4 and RD53A software emulators will be presented.