Test Suite Research Articles

Symmetric projection optimizer: concise and efficient solving engineering problems using the fundamental wave of the Fourier series

The fitness function value is a kind of important information in the search process, which can be more targeted according to the guidance of the fitness function value. Most existing meta-heuristic algorithms only use the fitness function value as an indicator to compare the current variables as good or bad but do not use the fitness function value in the search process. To address this problem, the mathematical idea of the fitting is introduced into the meta-heuristic algorithm, and a symmetric projection optimizer (SPO) is proposed to solve numerical optimization and engineering problems more efficiently. The SPO algorithm mainly utilizes a new search mechanism, the symmetric projection search (SP) method. The SP method quickly completes the fitting of the projection plane, which is located through the symmetry of the two points and finds the minima in the projection plane according to the fitting result. Fitting by using the fitness function values allows the SP to find regions where extreme values may exist more quickly. Based on the SP method, exploration and exploitation strategies are constructed, respectively. The exploration strategy is used to find better regions, and the exploitation strategy is used to optimize the discovered regions continuously. The timing of the use of the two strategies is designed so that the SPO algorithm can converge faster while avoiding falling into local optima. The effectiveness of the SPO algorithm is extensively evaluated using seven test suites, including CEC2017, CEC2019, CEC2020, and CEC2022. It is also compared with two sets of 19 recent competitive algorithms. Statistical analyses are performed using five metrics such as the Wilcoxon test, the Friedman test, and variance. Finally, the practicality of the SPO algorithm is verified by four typical engineering problems and a real spacecraft trajectory optimization problem. The results show that the SPO algorithm can find superior results in 94.6% of the comparison tests and is a promising alternative for solving real-world problems.

Hardware study and implementation of image encryption algorithm based on a hyperchaotic key generator

In today’s interconnected world, digital data transmissions are an integral part of our communication systems. Connected objects are commonly used in digital life, however, they can be vulnerable to hacking and personal data theft due to inadequate security. To address the size and capability limitations of existing embedded devices, lightweight cryptography has been developed. This paper presents a hardware implementation of a lightweight cryptographic method using VHDL and FPGAs to secure digital information, specifically images. The proposed algorithm is a block cipher scheme that operates on blocks of 128-bit length and employs chaos based generator-based. The cryptosystem includes a whitening operation, random permutation, and key-dependent Substitution box. The proposed hardware implementation of the algorithm was validated using a Xilinx FPGA-Zynq platform and can secure different gray-scale images while maintaining low power usage, a high frequency of 160.6098 MHz, and high throughput of 11314.917 Mbps. The cryptosystem is shown to be effective against statistical and differential attacks, and key sequences are tested using the NIST SP 800–22 test suite. The algorithm’s robustness with respect to Shannon’s theory and Chaos is discussed, emphasizing its high level of security and performance.

A Framework and Toolkit for Testing the Correctness of Recommendation Algorithms

Evaluating recommender systems adequately and thoroughly is an important task. Significant efforts are dedicated to proposing metrics, methods, and protocols for doing so. However, there has been little discussion in the recommender systems’ literature on the topic of testing. In this work, we adopt and adapt concepts from the software testing domain, e.g., code coverage, metamorphic testing, or property-based testing, to help researchers to detect and correct faults in recommendation algorithms. We propose a test suite that can be used to validate the correctness of a recommendation algorithm, and thus identify and correct issues that can affect the performance and behavior of these algorithms. Our test suite contains both black box and white box tests at every level of abstraction, i.e., system, integration, and unit. To facilitate adoption, we release RecPack Tests , an open-source Python package containing template test implementations. We use it to test four popular Python packages for recommender systems: RecPack , PyLensKit , Surprise , and Cornac . Despite the high test coverage of each of these packages, we find that we are still able to uncover undocumented functional requirements and even some bugs. This validates our thesis that testing the correctness of recommendation algorithms can complement traditional methods for evaluating recommendation algorithms.

Real world performance of the 21st Century Cures Act population-level application programming interface.

To evaluate the real-world performance of the SMART/HL7 Bulk Fast Health Interoperability Resources (FHIR) Access Application Programming Interface (API), developed to enable push button access to electronic health record data on large populations, and required under the 21st Century Cures Act Rule. We used an open-source Bulk FHIR Testing Suite at 5 healthcare sites from April to September 2023, including 4 hospitals using electronic health records (EHRs) certified for interoperability, and 1 Health Information Exchange (HIE) using a custom, standards-compliant API build. We measured export speeds, data sizes, and completeness across 6 types of FHIR. Among the certified platforms, Oracle Cerner led in speed, managing 5-16 million resources at over 8000 resources/min. Three Epic sites exported a FHIR data subset, achieving 1-12 million resources at 1555-2500 resources/min. Notably, the HIE's custom API outperformed, generating over 141 million resources at 12000 resources/min. The HIE's custom API showcased superior performance, endorsing the effectiveness of SMART/HL7 Bulk FHIR in enabling large-scale data exchange while underlining the need for optimization in existing EHR platforms. Agility and scalability are essential for diverse health, research, and public health use cases. To fully realize the interoperability goals of the 21st Century Cures Act, addressing the performance limitations of Bulk FHIR API is critical. It would be beneficial to include performance metrics in both certification and reporting processes.

Dynamic constrained evolutionary optimization based on deep Q-network

Dynamic constrained optimization problems (DCOPs) are common and important optimization problems in real-world, which have great difficulty to solve. Dynamic constrained evolutionary algorithms (DCEAs) are widely used methods for solving DCOPs. However, existing DCEAs often struggle with convergence, particularly for DCOPs with drastic dynamic changes or intricate constraints. To address this issue, this paper proposes a novel DCEA called DCEA-DQN, which leverages the powerful perception and decision-making capabilities of Deep Q-Network (DQN). DCEA-DQN integrates two DQNs to enhance its performance. The first DQN is designed to adaptively respond to dynamic changes, enabling effective handling of DCOPs with various types and degrees of changes. It provides a high-quality re-initialized population for subsequent static optimization, resulting in faster and improved convergence. The second DQN is introduced to guide the mutation direction during offspring generation. It steers the population towards better feasible regions or directs it towards the optimal individual within the current feasible region. Moreover, a penalty mechanism is employed to handle constraints during offspring generation. To evaluate the performance of DCEA-DQN, comprehensive empirical studies are conducted using a new test suite called C-GMPB and a dynamic flexible job-shop scheduling problem. The experimental results, using two commonly used metrics EB and EO in the field of DCOPs, demonstrate that DCEA-DQN outperforms six state-of-the-art DCEAs and achieved optimal performance on 80% and 75% of all 24 test problems, respectively. The source code for DCEA-DQN is available at https://github.com/CIA-SZU/YRT.

A method to identify overfitting program repair patches based on expression tree

The primary aim of Automatic Program Repair (APR) is to automatically repair defective programs, with the intention of reducing the amount of effort required by developers. However, APR techniques may produce overfitting patches that do not truly repair the program, allowing the program to pass all test cases. This paper provides a comprehensive review of the overfitting problem and adds to the existing research on overfitting in conditional statements. Our proposed method, ETPAT (Expression Tree-based Patch Assessment Technique), implements expression trees and targeted coverage criteria to identify differences between the original and the patched program. We utilize ETPAT to verify test case adequacy. In parallel, ETPAT also guides the generation of corresponding test cases via equivalence class information, which may be added to the original test suite, making it more robust while also preventing the repair technique from generating comparable overfitting patches. With reference to the patch set in the BuggyJavaJML benchmark, ETPAT recognized 77/82 (93.9%) overfitting patches out of 120 patches related to conditional constraints, displaying superior accuracy rates and fewer test cases required than the original repair tool.

Usage of cause‐effect graphs for reliability analysis of an embedded IoT alarm software system

AbstractSoftware is a part of most complex systems used for different purposes in today's world. The reliability of these systems must be verified before putting them to use, because their malfunction may lead to catastrophic failure, the loss of human lives, or huge costs. Software reliability can be verified by performing black‐box testing, without knowledge of the complex internal structure of these systems. In this work, an embedded IoT alarm system for detection of intruders and potential robbery is proposed. It is composed of three subsystems: embedded IoT, web‐server, and mobile phone subsystems. Cause‐effect graphing technique was applied for generating three types of black‐box test suites, by using the forward‐propagation and two different minimization algorithms. These test suites were used for black‐box testing the system, which was performed after injecting a maximum of 12 faults into the system in 100 iterations. The resulting software quality and reliability metrics showed that the forward‐propagation algorithm always achieves maximum fault detection rate (FDR), and this algorithm should be used when the highest demand for reliability must be met. The optimized minimization should only be used in cases where testing is very expensive, because it did not find faults in all testing iterations. The basic optimization retained high FDR, while reducing the impact of the minimized test suite and achieved a significant increase of test effect coverage values.

A High-Quality Random Number Generator Using Multistage Ring Oscillators and Fast Fourier Transform-Based Noise Extraction

Random Numbers are widely employed in cryptography and security applications. This paper presents a novel approach to generate high-quality random bitstreams by harnessing the inherent noise properties of ring oscillators. We implemented ring oscillators with varying numbers of stages (3, 5, and 7), different geometries and different startup voltages in Cadence and recorded their total output power, which includes the cumulative noise effects. Subsequently, we exported these power measurements to MATLAB, where we applied a Fast Fourier Transform (FFT)-based technique to extract the total noise characteristics for each ring oscillator. Using the obtained noise data, we generated separate random bitstreams of 10 million bits for the 3-stage, 5-stage, and 7-stage ring oscillators. The final random bitstream, consisting of 10 million bits, was created by performing a bitwise XOR operation on the bitstreams generated by each ring oscillator. The degree of randomness of the generated bitstreams was assessed using the NIST 800-22 statistical test suite. Remarkably, the final random bitstream exhibited strong robustness and suitability for cryptographic applications. This innovative approach leverages the noise properties of ring oscillators to create reliable random bitstreams, offering potential applications in secure communications and cryptography. The results highlight the feasibility of using ring oscillators as noise sources for random bit generation and underscore their effectiveness in meeting stringent randomness criteria.

Differential evolution with proration-based mutation strategy and multi-segment mixed parameter setting for numerical optimization

In this paper, a new differential evolution (DE) algorithm is presented for solving global optimization problems by guiding the individuals adaptively to explore different decision regions and setting the control parameters properly for individuals. To consolidate algorithm efficiency and preserve population diversity, we first introduce a proration-based mutation strategy. This strategy dynamically selects promising individuals from population as guiders and proportionately allocates the individuals following them based on their fitness values and unsuccessful updates. Besides, we design a multi-segment mixed parameter setting to provide suitable parameters for individuals, considering feedback information from the population, individual requirements, and parameters' interaction. Additionally, we devise an adaptive population maintaining mechanism to refresh the population by replacing individuals with higher unsuccessful updates and removing those with poor fitness values. Unlike previous DE versions, our new algorithm dynamically and appropriately explores promising areas, refines parameter suitability, and accelerates convergence by updating and removing unpromising individuals. This enhances the algorithm search efficiency and achieves a good balance between exploration and exploitation. Finally, we evaluate the proposed algorithm's performance by comparing it with 17 typical or state-of-the-art algorithms on IEEE CEC2014 and CEC2017 test suites. Experimental results indicate that the proposed algorithm is a more promising optimizer.

Towards accurate keyspace analysis of chaos-based image ciphers

In recent years, there has been a surge in new chaos-based cryptographic algorithms, many of which claim to have unusually large keyspaces. Although cryptographic primitives such as symmetric-key ciphers should have a secret keyspace large enough to resist brute force attacks, simply increasing the size of a secret key may not lead to improved security margins. An n-bit key may not necessarily have a keyspace of 2n-1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$2^n-1$$\\end{document} due to the key scheduling algorithm or how the key is used. In this paper, we cryptanalyse several chaos-based algorithms from the perspective of their key schedules. Our numerical analysis is based on the known-plaintext attack model, Kerckhoff’s principle and considers the number representations used for real number computation. Our analysis reveals that the actual security margins for these ciphers are significantly lower, some by a factor of over 2100\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$2^{100}$$\\end{document} than what was claimed. We then provide accurate keyspace estimates for these ciphers. Finally, we highlight alternative solutions for how secret keys can be used in the context of chaos-based cryptography and propose a simple key schedule as a proof of concept. Despite its simplicity, the proposed key schedule not only ensures that the keyspace matches the key length but also passes both the NIST and ENT statistical test suites, making it a viable option for generating secure cryptographic keys. Our work contributes towards addressing one of the fundamental problems in chaos-based cryptography that limits its real-world impact and reputation within the cryptographic community.

Go-Explore Complex 3-D Game Environments for Automated Reachability Testing

Modern AAA video games feature huge game levels and maps which are increasingly hard for level testers to cover exhaustively. As a result, games often ship with catastrophic bugs such as the player falling through the floor or being stuck in walls. We propose an approach specifically targeted at reachability bugs in simulated 3D environments based on the powerful exploration algorithm, Go-Explore, which saves unique checkpoints across the map and then identifies promising ones to explore from. We show that when coupled with simple heuristics derived from the game's navigation mesh, Go-Explore finds challenging bugs and comprehensively explores complex environments <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">without the need for human demonstration or knowledge of the game dynamics</i> . Go-Explore vastly outperforms more complicated baselines including reinforcement learning with intrinsic curiosity in both covering the navigation mesh and number of unique positions across the map discovered. Finally, due to our use of parallel agents, our algorithm can fully cover a vast 1.5km x 1.5km game world within 10 hours on a single machine making it extremely promising for continuous testing suites.

Diversified Knowledge Transfer Strategy for Multitasking Particle Swarm Optimization.

Evolutionary multitasking optimization (EMTO) has capability of performing a population of individuals together by sharing their intrinsic knowledge. However, the existed methods of EMTO mainly focus on improving its convergence using parallelism knowledge belonging to different tasks. This fact may lead to the problem of local optimization in EMTO due to unexploited knowledge on behalf of the diversity. To address this problem, in this article, a diversified knowledge transfer strategy is proposed for multitasking particle swarm optimization algorithm (DKT-MTPSO). First, according to the state of population evolution, an adaptive task selection mechanism is introduced to manage the source tasks that contribute to the target tasks. Second, a diversified knowledge reasoning strategy is designed to capture the knowledge of convergence, as well as the knowledge associated with diversity. Third, a diversified knowledge transfer method is developed to expand the region of generated solutions guided by acquired knowledge with different transfer patterns so that the search space of tasks can be explored comprehensively, which is favor of EMTO alleviating local optimization. Finally, the performance of the proposed algorithm is evaluated in comparison with some other state-of-the-art EMTO algorithms on multiobjective multitasking benchmark test suits, and the practicality of the algorithm is verified in a real-world application study. The results of experiments demonstrate the superiority of DKT-MTPSO compared to other algorithms.

Unveiling Faulty User Sequences: A Model-Based Approach to Test Three-Tier Software Architectures

Context:When testing three-tiered architectures, strategies often rely on superficial information, e.g., black-box input. However, the correct behavior of software-intensive systems based on such architectural pattern also depends on the logic hidden behind the interface. Verifying the response process is thus often complex and requires ad-hoc strategies. Objective:We propose an approach to identify faults hidden behind the presentation layer. The model-based approach uses an architectural abstraction called managed component Data Flow Graph (mcDFG). The mcDFG is aware of the interactions between all layers of the architecture and guides the generation of tests based on different mcDFG coverage criteria to identify faults in the business logic. Method:To evaluate the approach viability, we consider a three-tiered web application and 32 faults. The fault detection capability is assessed by comparing a set of test suites created by following our method and a set of test suites developed by utilizing traditional testing strategies. Results:The collected data show that the proposed model-based approach is a viable option to identify faults hidden in the logic layer, as it can outperform standard strategies based solely on the presentation layer while keeping the number of test cases and number of interactions per test case low.

LS-14 test suite for long sequences

Random number sequences are used in many branches of science. Because of many techni- cal reasons and their practicality, pseudo random sequences are usually employed in place of true number sequences. Whether a sequence generated through a deterministic process is a pseudo random, in other words, random-looking sequence or it contains certain pat- terns, can be determined with the help of statistics and mathematics. Although, in the literature there are many statistical randomness tests for this purpose, there is no much work on test suites specialized for long sequences, that is sequences of length 1,000,000 bits or more. Most of the randomness tests for long sequences use some mathematical approximations to compute expected values of the random variables and hence their results contain some errors. Another approach to evaluate randomness criteria of long sequences is to partition the long sequence into a collection short sequences and evaluate the collection for the ran- domness using statistical goodness of fit tests. The main advantage of this approach is, as the individual sequences are short, there is no need to use mathematical approximations. On the other hand when the second approach is preferred, partition the long sequence into a collection of fixed length subsequences and this approach causes a loss of information in some cases. Hence the idea of dynamic partition should be included to perform a more reliable test suite. In this paper, we propose three new tests, namely the entire R2 run, dynamic saturation point, and dynamic run tests. Moreover, we introduce a new test suite, called LS-14, consisting of 14 tests to evaluate randomness of long sequences. As LS-14 employs all three approaches: testing the entire long sequence, testing the collection of fixed length partitions of it, and finally, testing the collection obtained by the dynamic partitions of it, the proposed LS-14 test suit differs from all existing suites. Mutual comparisons of all 14 tests in the LS-14 suite, with each other are computed. Moreover, results obtained from the proposed test suite and NIST SP800-22 suite are compared. Examples of sequences with certain patterns which are not observed by NIST SP800-22 suite but detected by the proposed test suite are given.

Hippopotamus optimization algorithm: a novel nature-inspired optimization algorithm

The novelty of this article lies in introducing a novel stochastic technique named the Hippopotamus Optimization (HO) algorithm. The HO is conceived by drawing inspiration from the inherent behaviors observed in hippopotamuses, showcasing an innovative approach in metaheuristic methodology. The HO is conceptually defined using a trinary-phase model that incorporates their position updating in rivers or ponds, defensive strategies against predators, and evasion methods, which are mathematically formulated. It attained the top rank in 115 out of 161 benchmark functions in finding optimal value, encompassing unimodal and high-dimensional multimodal functions, fixed-dimensional multimodal functions, as well as the CEC 2019 test suite and CEC 2014 test suite dimensions of 10, 30, 50, and 100 and Zigzag Pattern benchmark functions, this suggests that the HO demonstrates a noteworthy proficiency in both exploitation and exploration. Moreover, it effectively balances exploration and exploitation, supporting the search process. In light of the results from addressing four distinct engineering design challenges, the HO has effectively achieved the most efficient resolution while concurrently upholding adherence to the designated constraints. The performance evaluation of the HO algorithm encompasses various aspects, including a comparison with WOA, GWO, SSA, PSO, SCA, FA, GOA, TLBO, MFO, and IWO recognized as the most extensively researched metaheuristics, AOA as recently developed algorithms, and CMA-ES as high-performance optimizers acknowledged for their success in the IEEE CEC competition. According to the statistical post hoc analysis, the HO algorithm is determined to be significantly superior to the investigated algorithms. The source codes of the HO algorithm are publicly available at https://www.mathworks.com/matlabcentral/fileexchange/160088-hippopotamus-optimization-algorithm-ho.

Hybrid Framework To Exclude Similar and Faulty Test Cases In Regression Testing

Regression testing is a crucial phase in the software development lifecycle that makes sure that new changes/updates in the software system don’t introduce defects or don’t affect adversely the existing functionalities. However, as the software systems grow in complexity, the number of test cases in regression suite can become large which results into more testing time and resource consumption. In addition, the presence of redundant and faulty test cases may affect the efficiency of the regression testing process. Therefore, this paper presents a new Hybrid Framework to Exclude Similar &amp; Faulty Test Cases in Regression Testing (ETCPM) that utilizes automated code analysis techniques and historical test execution data to identity and exclude redundant, similar and faulty test cases from the given regression suite. Our experimental results clearly show the benefits of the ETCPM framework in terms of reduction in the testing time, optimization of the resource allocation, and improvement in the overall quality of regression test suite. ETCPM enables software development teams to achieve faster and reliable regression testing by intelligent exclusion of similar and fault test cases, which yields in reduction in the software delivery cycles and better end user satisfaction.

Botox Optimization Algorithm: A New Human-Based Metaheuristic Algorithm for Solving Optimization Problems.

This paper introduces the Botox Optimization Algorithm (BOA), a novel metaheuristic inspired by the Botox operation mechanism. The algorithm is designed to address optimization problems, utilizing a human-based approach. Taking cues from Botox procedures, where defects are targeted and treated to enhance beauty, the BOA is formulated and mathematically modeled. Evaluation on the CEC 2017 test suite showcases the BOA's ability to balance exploration and exploitation, delivering competitive solutions. Comparative analysis against twelve well-known metaheuristic algorithms demonstrates the BOA's superior performance across various benchmark functions, with statistically significant advantages. Moreover, application to constrained optimization problems from the CEC 2011 test suite highlights the BOA's effectiveness in real-world optimization tasks.

Cryptographic transistor for true random number generator with low power consumption.

We design a cryptographic transistor (cryptoristor)-based true random number generator (tRNG) with low power consumption and small footprint. This is the first attempt to use irregular and unpredictable operation-induced randomness of a cryptoristor as an entropy source. To extract discrete random numbers with a binary code from the cryptoristor, we developed a noise-coupling analog-to-digital converter. This converter not only converts analog signals to digital random bits but also improves the randomness of the entropy source with low power consumption. The randomness of the cryptoristor is attributed to the random carrier multiplications with the creation and stochastic carrier escape as destruction, which occurs iteratively as long as the input current is fed. The cryptoristor-based tRNG passed 15 randomness test suites of NIST Special Publication 800-22. It is robust to iterative operational stresses and to ambient temperature changes, making it an attractive option for hardware-based security solutions in the Internet of Things due to its low power consumption and small size.

Physical and mechanical properties of nano-modified polybenzoxazine nanocomposite laminates: Pre-flight tests before exposure to low Earth orbit

Polybenzoxazine (PBZ) formulations based on commercial components and modified with polyhedral oligomeric silsesquioxanes (POSS, 5 wt% and 10 wt%) have been scaled up for industrialisation, and their processing characteristics determined using thermal analysis and rheometry. The ultimate glass transition temperatures for the cured PBZ resins are increased by catalysis and by the incorporation of POSS. Carbon fibre reinforced polymer (CFRP) laminates (500 mm × 500 mm x 3 mm) were manufactured using resin transfer moulding and subjected to a suite of mechanical tests. The addition of 5 wt% POSS to the PBZ formulation leads to substantial improvements in both strength (tensile, flexural, short beam shear, and compression) and modulus (tensile, flexural, and compression). The CFRP specimens were subjected to a suite of pre-flight qualification tests prior to deployment in low Earth orbit. When the CFRP laminates were exposed to <1 × 10−5 mbar and 125 °C, the total mass loss recorded for each sample was <1 wt%. The introduction of POSS at both levels of incorporation improves erosion yield following exposure to high atomic oxygen flux and the ratio of solar absorptance (α) to emittance (ε) remained constant following simulated solar exposure. The outcome was that the three PBZ CFRP samples passed their pre-flight tests.

APSO-SL: An Adaptive Particle Swarm Optimization with State-Based Learning Strategy

Particle swarm optimization (PSO) has been extensively used to solve practical engineering problems, due to its efficient performance. Although PSO is simple and efficient, it still has the problem of premature convergence. In order to address this shortcoming, an adaptive particle swarm optimization with state-based learning strategy (APSO-SL) is put forward. In APSO-SL, the population distribution evaluation mechanism (PDEM) is used to evaluate the state of the whole population. In contrast to using iterations to just the population state, using the population spatial distribution is more intuitive and accurate. In PDEM, the population center position and best position for calculation are used for calculation, greatly reducing the algorithm’s computational complexity. In addition, an adaptive learning strategy (ALS) has been proposed to avoid the whole population’s premature convergence. In ALS, different learning strategies are adopted according to the population state to ensure the population diversity. The performance of APSO-SL is evaluated on the CEC2013 and CEC2017 test suites, and one engineering problem. Experimental results show that APSO-SL has the best performance compared with other competitive PSO variants.