Performance Testing Framework Research Articles

An enhanced distributed framework for real-time performance testing of large scale IoT dataset using big data analytic tools

<p>The demand for analyzing enormous IoT datasets is rising in parallel with the popularity of the IoT. There are considerable obstacles to effective processing and analysis due to the amount, velocity, and variety of IoT data. In this research, we present a distributed system that makes use of big data analytic tools like Apache Hive, Spark, and Hadoop to efficiently test the performance of massive IoT datasets. The framework addresses the lack of a comprehensive solution by providing a scalable and fault-tolerant architecture. We discuss the motivation behind real-time performance testing in the context of big data analytics for IoT datasets and highlight the need for a distributed framework. A literature review is conducted to explore existing performance testing frameworks, big data analytic tools, and approaches for performance testing big data analytics. The proposed framework’s key components, including dataset generation, test scenario specification, cluster configuration, performance metrics collection, analysis and visualization modules, and implementation details, including tool choices, are discussed. An experimental evaluation is conducted to validate the framework’s performance, and it is suggested to incorporate blockchain technology. Overall, the proposed framework offers a comprehensive solution for real-time performance testing of large-scale IoT datasets, providing organizations and researchers with a valuable tool to ensure efficient and reliable IoT data processing and analysis.</p>

Performance portable ice-sheet modeling with MALI

High-resolution simulations of polar ice sheets play a crucial role in the ongoing effort to develop more accurate and reliable Earth system models for probabilistic sea-level projections. These simulations often require a massive amount of memory and computation from large supercomputing clusters to provide sufficient accuracy and resolution; therefore, it has become essential to ensure performance on these platforms. Many of today’s supercomputers contain a diverse set of computing architectures and require specific programming interfaces in order to obtain optimal efficiency. In an effort to avoid architecture-specific programming and maintain productivity across platforms, the ice-sheet modeling code known as MPAS-Albany Land Ice (MALI) uses high-level abstractions to integrate Trilinos libraries and the Kokkos programming model for performance portable code across a variety of different architectures. In this article, we analyze the performance portable features of MALI via a performance analysis on current CPU-based and GPU-based supercomputers. The analysis highlights not only the performance portable improvements made in finite element assembly and multigrid preconditioning within MALI with speedups between 1.26 and 1.82x across CPU and GPU architectures but also identifies the need to further improve performance in software coupling and preconditioning on GPUs. We perform a weak scalability study and show that simulations on GPU-based machines perform 1.24–1.92x faster when utilizing the GPUs. The best performance is found in finite element assembly, which achieved a speedup of up to 8.65x and a weak scaling efficiency of 82.6% with GPUs. We additionally describe an automated performance testing framework developed for this code base using a changepoint detection method. The framework is used to make actionable decisions about performance within MALI. We provide several concrete examples of scenarios in which the framework has identified performance regressions, improvements, and algorithm differences over the course of 2 years of development.

PerformERL: a performance testing framework for erlang

The Erlang programming language is used to build concurrent, distributed, scalable and resilient systems. Every component of these systems has to be thoroughly tested not only for correctness, but also for performance. Performance analysis tools in the Erlang ecosystem, however, do not provide a sufficient level of automation and insight needed to be integrated in modern tool chains. In this paper, we present PerformERL : an extendable performance testing framework that combines the repeatability of load testing tools with the details on how the resources are internally used typical of the performance monitoring tools. These features allow PerformERL to be integrated in the early stages of testing pipelines, providing users with a systematic approach to identifying performance issues. This paper introduces the PerformERL framework, focusing on its features, design and imposed monitoring overhead measured through both theoretical estimates and trial runs on systems in production. The uniqueness of the features offered by PerformERL , together with its usability and contained overhead prove that the framework can be a valuable resource in the development and maintenance of Erlang applications.

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models.

Transcribe@Replay: A Web Requests Initiation Framework for Client-Side High Concurrency

As the E-Business prevails at an unprecedented rate, an urgent press is impelled for a testing tool aiming at the high client-side concurrency. In this paper, aiming at the client-side concurrent requests, a performance testing framework is put forward and a prototype is implemented as a motivating case study. First, an attempt is made and a scalable testing framework, namely Transcribe@Replay is proposed. It is composed of three components, e.g. Transcriber, Monitor, and a Replay. To emulate high concurrent requests in a scalable manner, the distributed architecture is adopted for the Replay infrastructure. Second, a prototype is designed as a motivating case study. The implementation of each component, including Transcriber, Monitor and Replay, are described in detail. And the optimization spaces are discussed in-depth, e.g. the transcription template file generation, tasks generation and dispatch for virtual users, persistent request for HTTP 1.0 and HTTP 1.1, and the communication protocol based on EPOLL, and etc. Third, an experiment is conducted and the request/response cycle is traced. Experimental results demonstrate the feasibility of the proposed Transcribe@Replay framework.

An MDE performance testing framework based on random model generation

The scalability of model-related operations (e.g., model transformations), when they are to be applied in industrial model-driven engineering, becomes an important issue. However, there is a lack of an automated performance testing framework for those operations, since the existing ones for ordinary programs are ill-suited. Such a framework is required to provide the function of creating and organizing test cases, and the ability of generating test input of large size automatically, because large scale models are not widely available, making it hard to test the performance and coverage of those operations without any bias. This paper proposes a performance testing framework, integrated with a random model generation algorithm, for model-related operations. The framework, based on a test model, can be used to specify and arrange test cases into test suites. And the model generation algorithm can generate a random model correctly and efficiently, according to the metamodel and user-defined constraints. Finally, we present two case studies, one experiment in randomness, and two experiments in generation efficiency to evaluate the framework and algorithm. Results show that the framework is competent to support performance testing of model-related operations, and the algorithm is random and efficient enough to generate test data for performance testing.

Unit testing performance with Stochastic Performance Logic

Unit testing is an attractive quality management tool in the software development process, however, practical obstacles make it difficult to use unit tests for performance testing. We present Stochastic Performance Logic, a formalism for expressing performance requirements, together with interpretations that facilitate performance evaluation in the unit test context. The formalism and the interpretations are implemented in a performance testing framework and evaluated in multiple experiments, demonstrating the ability to identify performance differences in realistic unit test scenarios.

Statistical Performance Comparisons of Computers

As a fundamental task in computer architecture research, performance comparison has been continuously hampered by the variability of computer performance. In traditional performance comparisons, the impact of performance variability is usually ignored (i.e., the means of performance observations are compared regardless of the variability), or in the few cases directly addressed with <inline-formula><tex-math>$t$</tex-math></inline-formula> -statistics without checking the number and normality of performance observations. In this paper, we formulate a performance comparison as a statistical task, and empirically illustrate why and how common practices can lead to incorrect comparisons. We propose a non-parametric hierarchical performance testing (HPT) framework for performance comparison, which is significantly more practical than standard <inline-formula><tex-math>$t$</tex-math></inline-formula> -statistics because it does not require to collect a large number of performance observations in order to achieve a normal distribution of sample mean. In particular, the proposed HPT can facilitate quantitative performance comparison, in which the performance speedup of one computer over another is statistically evaluated. Compared with the HPT, a common practice which uses geometric mean performance scores to estimate the performance speedup has errors of <inline-formula><tex-math>$8.0$ </tex-math></inline-formula> to <inline-formula><tex-math>$56.3$</tex-math></inline-formula> percent on SPEC CPU2006 or SPEC MPI2007, which demonstrates the necessity of using appropriate statistical techniques. This HPT framework has been implemented as an open-source software, and integrated in the PARSEC 3.0 benchmark suite.

Realization of an Execution System of Distributed Performance Testing Based on TTCN-3

An execution system of distributed performance testing is designed in this paper, which provides a mechanism of node communication, test script deployment, test scheduling, execution-driving and test result collection in distributed environment. A workload model is established, by which testers can describe the performance testing requirement. A performance testing framework is given, which simulates user behaviors in real environment based on virtual users so as to generate workload from the system under test (SUT). It can control the execution of virtual users by TTCN-3 standard interface. After executing the performance testing, test report is generated by extracting log. A method of generating performance test-case is studied by reusing functional test scripts. By executing performance testing on an online bookstore, this paper demonstrates the availability of the method of reusing TTCN-3 functional test scripts and the capability of distributed performance testing system established.

D-P2P-Sim+: A novel distributed framework for P2P protocols performance testing

In recent technologies like IoT (Internet of Things) and Web 2.0, a critical problem arises with respect to storing and processing the large amount of collected data. In this paper we develop and evaluate distributed infrastructures for storing and processing large amount of such data. We present a distributed framework that supports customized deployment of a variety of indexing engines over million-node overlays. The proposed framework provides the appropriate integrated set of tools that allows applications processing large amount of data, to evaluate and test the performance of various application protocols for very large scale deployments (multi million nodes–billions of keys). The key aim is to provide the appropriate environment that contributes in taking decisions regarding the choice of the protocol in storage P2P systems for a variety of big data applications. Using lightweight and efficient collection mechanisms, our system enables real-time registration of multiple measures, integrating support for real-life parameters such as node failure models and recovery strategies. Experiments have been performed at the PlanetLab network and at a typical research laboratory in order to verify scalability and show maximum re-usability of our setup. D-P2P-Sim+ framework is publicly available at http://code.google.com/p/d-p2p-sim/downloads/list.

Building a Distributed Testing Execution System Based on TTCN-3

In this paper, a distributed testing execution system is designed, which provides a mechanism of node communication, test script deployment, test scheduling, executor-driving and test result collection in distributed environment. A workload model is established, by which testers can describe the performance testing requirement. A performance testing framework is given, which simulates user behaviors in real environment based on virtual users so as to generate workload from the system under test (SUT). It can control the execution of virtual users by TTCN-3 standard interface. After executing the performance testing, test report is generated by extracting log. A method of generating performance test-case is studied by reusing functional test scripts. By executing performance testing on an online bookstore, this paper demonstrates the availability of the method of reusing TTCN-3 functional test scripts and the capability of distributed performance testing system that had been established.

Performance Testing Framework in a Heterogeneous and Hybrid Smart Grid Communication Network

Heterogeneous and hybrid smart grid communication network is comprised of different communication mediums and technologies. Performance evaluation is one of the main concerns for smart grid communication system. In any smart grid communication implementation, to determine the performance factor of the network, a testing of an end-to-end process flow is required. An effective and coordinated testing procedure plays a crucial role in evaluating the performance of smart grid communications. Therefore, this study proposes a testing framework which specifies the types of communication mediums and technologies, the evaluation criteria and software tools to carry out the testing. The proposed testing scheme is used as a guideline to analyze and assess the performance of smart grid communication system.

Model-driven generative framework for automated OMG DDS performance testing in the cloud

The Object Management Group's (OMG) Data Distribution Service (DDS) provides many configurable policies which determine end-to-end quality of service (QoS) of applications. It is challenging to predict the system's performance in terms of latencies, throughput, and resource usage because diverse combinations of QoS configurations influence QoS of applications in different ways. To overcome this problem, design-time formal methods have been applied with mixed success, but lack of sufficient accuracy in prediction, tool support, and understanding of formalism has prevented wider adoption of the formal techniques. A promising approach to address this challenge is to emulate system behavior and gather data on the QoS parameters of interest by experimentation. To realize this approach, which is preferred over formal methods due to their limitations in accurately predicting QoS, we have developed a model-based automatic performance testing framework with generative capabilities to reduce manual efforts in generating a large number of relevant QoS configurations that can be deployed and tested on a cloud platform. This paper describes our initial efforts in developing and using this technology.

Performance testing framework for smart grid communication network

Smart grid communication network is comprised of different communication mediums and technologies. Performance evaluation is one of the main concerns in smart grid communication system. In any smart grid communication implementation, to determine the performance factor of the network, a testing of an end-to-end process flow is required. Therefore, an effective and coordinated testing procedure plays a crucial role in evaluating the performance of smart grid communications. In this paper, a testing framework is proposed as a guideline to analyze and assess the performance of smart grid communication network.

Vice vs. Virtue Investing Around the World

In this paper, we empirically test the extent to which a portfolio of socially not responsible firms screened out of a market portfolio will trade at a discount. We create a set of global and domestic sin indexes consisting of a large number of publicly traded socially not responsible stocks around the world belonging to what we label as the Sextet of Sin: adult entertainment, alcohol, gambling, nuclear power, tobacco, and weapons. We compare their stock market performance directly with a set of virtue comparables consisting of the most important in-ternational socially responsible investment indexes. Employing a multi-factor performance measurement framework and recent boot-strap procedures for robust performance testing, we find no compelling evidence in the data that ethical and unethical screens lead to a sig-nificant difference in their financial performance.

Development of Integrated System for Virtual Design and Performance Testing of Automobile Seat

Most of traditional design processes of mechanical parts are regarded as sequential and discrete, since different kinds of softwas should be introduced. In this paper, we develop an integrated fram ework for virtual design and performance testing of an automobile seat. The system is composed of four modules, i.e. CAD, static analysis, dynamic analysis, and draft drawing module. In the CAD module, PRO/ENGINEER 3D seat model is created using parameters to be modified with the result of static and dynamic analysis. In the static analysis, headrest tere used in each design stage make it difficult to feedback their results to upstream process. These discrete processes may result in time loss and cost rise. In recent years, life cycle of product is reduced. To have competence with others, new concept design processt is simulated using ANSYS. In the dynamic analysis module, FMVSS201 test is simulated using DADS. Overall data flow is controlled by Motif. The advantage of the system is that even a novice can perform and review the whole design process, without a good hand at professional design/analysis S/W in each stage. The system also provides a virtual design space, where engineers in different development stage can access common data of design models. The concept could be applied to other fields and it could reduce time and money required in design process.