Conformance Testing Method Research Articles

CLOCIS: Cloud-Based Conformance Testing Framework for IoT Devices in the Future Internet

In recent years, the Internet of Things (IoT) has not only become ubiquitous in daily life but has also emerged as a pivotal technology across various sectors, including smart factories and smart cities. Consequently, there is a pressing need to ensure the consistent and uninterrupted delivery of IoT services. Conformance testing has thus become an integral aspect of IoT technologies. However, traditional methods of IoT conformance testing fall short of addressing the evolving requirements put forth by both industry and academia. Historically, IoT testing has necessitated a visit to a testing laboratory, implying that both the testing systems and testers must be co-located. Furthermore, there is a notable absence of a comprehensive method for testing an array of IoT standards, especially given their inherent heterogeneity. With a surge in the development of diverse IoT standards, crafting an appropriate testing environment poses challenges. To address these concerns, this article introduces a method for remote IoT conformance testing, underpinned by a novel conceptual architecture termed CLOCIS. This architecture encompasses an extensible approach tailored for a myriad of IoT standards. Moreover, we elucidate the methods and procedures integral to testing IoT devices. CLOCIS, predicated on this conceptual framework, is actualized, and to attest to its viability, we undertake IoT conformance testing and present the results. When leveraging CLOCIS, small and medium-sized enterprises (SMEs) and entities in the throes of IoT service development stand to benefit from a reduced time to market and cost-efficient testing procedures. Additionally, this innovation holds promise for IoT standardization communities, enabling them to champion their standards with renewed vigor.

Conformance testing of ontologies through ontology requirements

In recent years, several standard ontologies have been developed to maximise semantic interoperability in different domains; such standard ontologies ensure quality and integrity when describing a domain. Therefore, mechanisms to guarantee that developers build ontologies that conform to such standards are needed. However, while in fields such as Software Engineering or industry, conformance testing plays an essential role during product development, in the Ontology Engineering field there is a lack of techniques for this type of testing. This work introduces an ontology conformance testing method to analyse conformance between an ontology and a standard based on the standard requirements. Grounded on this method, the work also presents a minimum common knowledge identification method for analysing how a group of standards covers a particular domain and for identifying whether there are conflicts between them. This work has been validated by analysing the conformance between an ontology network and a set of standards on the Internet of Things domain, and by analysing the minimum common knowledge between such standards. This analysis shows that the conformance between ontologies and standards is mostly related to definition of classes. Furthermore, the analysis shows that although the analysed standards are related to the same domain, they are created to describe different areas of concern and, thus, there is a minimum overlap between them. Finally, it was concluded that the quality of the conformance analysis depends on the quality of the requirements specification: the more precise the requirements, the more precise the analysis between ontologies and standards.

Test Sequence Reduction of Wireless Protocol Conformance Testing to Internet of Things

Wireless communication protocols are indispensable in Internet of Things (IoT), which refer to rules and conventions that must be followed by both entities to complete wireless communication or service. Wireless protocol conformance testing concerns an effective way to judge whether a wireless protocol is carried out as expected. Starting from existing test sequence generation methods in conformance testing, an improved method based on overlapping by invertibility and multiple unique input/output (UIO) sequences is proposed in this paper. The method is accomplished in two steps: first, maximum-length invertibility-dependent overlapping sequences (IDOSs) are constructed, then a minimum-length rural postman tour covering the just constructed set of maximum-length IDOSs is generated and a test sequence is extracted from the tour. The soundness and effectiveness of the method are analyzed. Theory and experiment show that desirable test sequences can be yielded by the proposed method to reveal violations of wireless communication protocols in IoT.

Conformance testing with ioco proxy-testers: application to web service compositions deployed in clouds

This paper describes a conformance testing method which aims at passively testing the conformance of component-based systems. This paper addresses the problem of reaction sequence observation in implementation environments where the installation of testing tools is not possible. The originality of the method resides in the definition of a proxy-tester model from a specification and in the reformulation of the ioco test relation with proxy-tester properties. Initially, we define the proxy-tester model and we give the theoretical background to automatically generate proxy-testers from specifications modelled with input-output symbolic transition system (IOSTS). The second part of this paper describes the application of this method on web service compositions deployed in Clouds. We provide algorithms and passive tester architectures to collect traces from several concurrent composition instances running in parallel. Finally, we present the testing tool Cloud Paste and experiment results on two Clouds, Google AppEngine and Windows Azure.

Proxy-Monitor

This paper proposes a conformance testing method combining two well-known testing approaches, runtime verification and passive testing. Runtime verification addresses the monitoring of a system under test to check whether formal properties hold, while passive testing aims at checking the conformance of the system in the long-term. The method, proposed in this paper, checks whether an implementation conforms to its specification with reference to the ioco test relation. While passively checking if ioco holds, it also checks whether the implementation meets safety properties, which informally state that “nothing bad ever happens”. This paper also tackles the trace extraction problem, which is common to both runtime verification and passive testing. The authors define the notion of Proxy-monitors for collecting traces even when the implementation environment access rights are restricted. Then, they apply and specialise this approach on Web service compositions. A Web service composition deployed in different Clouds is experimented to assess the feasibility of the method.

An Open Conformance Test System towards the Standardization of Wireless Sensor Networks

The conformance testing on the wireless sensor networks (WSNs) is fundamental for the functional large-scale deployment and interconnection with the global Internet. The WSN protocols based on the Internet Protocol version 6 (IPv6) are the major trends of the future applications, which include IEEE 802.15.4, 6LoWPAN, and RPL. These protocols are not only diverse for various applications, but also volatile in the future. Moreover, sensor nodes are resource constrained, application related and lack the standardized test interface. Hence the corresponding conformance testing is seriously insufficient and needs urgently research. In this paper, the IPv6-based WSN protocols are analyzed and the conformance testing techniques and methods for IPv6-based WSNs are investigated towards the standardization of WSNs. A novel conformance test system for IPv6-based WSNs is designed and implemented, which is open, flexible, full featured, and practical. The conformance test system is suitable for the protocol evolution and the various hardware interfaces for the sensor nodes. The related outcomes will promote the standardization and commercialization of WSNs.

The Research of Synchronized Phasor Measurement Unit Testing and Evaluation

Phasor Measurement Unit (PMU) devices, based on the synchronized measurement technology, are basic building blocks of Wide Area Monitoring System (WAMS) and Wide Area Monitoring Analysis Protection-control (WAMAP). Similar to the Remote Terminal Unit (RTU) in Energy Management System (EMS), PMU is becoming very important in the WAMS or WAMAP. This paper introduces the architecture of PMU testing and evaluation (T&E) in our laboratory. The integrated evaluation schemes include testing environment, performance testing method, conformance testing method and evaluation.

Assessment of conformance and interoperability testing methods used for construction industry product models

This paper presents a review and assessment of conformance and interoperability testing methods for product data models used in the construction industry. Conformance testing methodologies, with varying degrees of rigor, have been developed and applied to ensure interoperability across product modeling software applications in other engineering and industry domains, such as, engineering and manufacturing of automotive and aerospace products. Current conformance testing and evaluation of interoperability for construction industry product modeling software do not necessarily apply those same principles and are usually done on an ad-hoc basis. Key principles are identified for improved methods and metrics for developing conformance and interoperability testing capabilities for the construction industry.

FSM-based conformance testing methods: A survey annotated with experimental evaluation

The development of test cases is an important issue for testing software, communication protocols and other reactive systems. A number of methods are known for the development of a test suite based on a formal specification given in the form of a finite state machine. In this paper, we overview and experiment with these methods to assess their complexity, applicability, completeness, fault detection capability, length and derivation time of their test suites. The experiments are conducted on randomly generated specifications and on two realistic protocols called the Simple Connection Protocol and the ITU-T V.76 Recommendation .

Coverage-guided test generation for continuous and hybrid systems

In this paper, we describe a formal framework for conformance testing of continuous and hybrid systems, using the international standard `Formal Methods in Conformance Testing' FMCT. We propose a n...

Coverage-guided test generation for continuous and hybrid systems

In this paper, we describe a formal framework for conformance testing of continuous and hybrid systems, using the international standard `Formal Methods in Conformance Testing' FMCT. We propose a novel test coverage measure for these systems, which is defined using the star discrepancy notion. This coverage measure is used to quantify the validation `completeness'. It is also used to guide input stimulus generation by identifying the portions of the system behaviors that are not adequately examined. We then propose a test generation method, which is based on a robotic motion planning algorithm and is guided by the coverage measure. This method was implemented in a prototype tool that can handle high dimensional systems (up to 100 dimensions).

BioAPl기반 시스템 모듈을 검증하기 위한 적합성시험 방법 연구

생체인식기술을 응용한 제품이 다양해지면서 상호운용성의 문제가 제기되어 국제표준화가 진행중인 BioAPl(Biometric Application Pro-gramming Interface)를 기반으로 한 생체인식 제품의 개발이 증대되고 있다. 그러나 BioAPl를 적용하여 개발된 제품에 대한 성능 측정 및 표준규격의 적합성 여부에 대한 신뢰성 있는 평가 도구가 마련되어 있지 않은 상태이다. 본 논문은 생체인식시스템이 BioAPl 규격의 요구사항 및 기준에 얼마나 만족하여 개발하였는가를 검증하기 위한 적합성시험 방법을 연구하였다. 제안된 적합성 시험기술의 첫 번째는 BioAPl명세서가 제공하는 각 함수들을 제대로 구현하였는가에 대한 확인과 사용여부를 평가하는 것이다. 이것은 응용에서 APl(Application Provider Inter-face)함수를 호출하면 프레임워크를 통해 BSP(Biometric Service Provider)를 실행하는 것으로 이때 파라미터와 선행함수의 선택이 필요하다. 두 번째는 BioAPl의 해당 테스트케이스를 분석하여 모듈관리, 핸들기능, 검증기능에 대한 시나리오를 평가하는 것이다. 실험은 BioAPl 컨소시엄에서 제공하는 샘플프로그램과 상용 지문검증시스템의 BSP를 사용하여 제안하는 적합성 평가 방법에 대한 실험을 수행하였다. 이에 따라 BioAPl를 기반으로 한 BSP들이 요구사항에 적합하게 개발되었는지를 판단할 수 있었다. Recently the biometric recognition technology is intensively studied and the standardization of the technology has been highly demanded for its commercialization. Currently many blometric recognition products are being developed based on the BioAPl(Biometric Application Program-ming Interface) specification. However, the reliable testing tools (or scenarios) to evaluate performance and conformance of the products are not shown yet. In this paper, a conformance testing method is presented, which verifies a biometric recognition system to meet the requirements of the BioAPl standard. Two different testing procedures are used in the proposed method. The first procedure evaluates that each functions offered in the BioAPl specification are correctly implemented and that the functions are actually used in the system. Through the Procedure, a BSP(Biometric Service Provider) system is executed on the framework of the BioAPl functions. It requires selection of parameters and prece-dent functions that should be executed first. The second procedure evaluates the abilities of module management, handling operations and ver-ification process by the analysis of the test cases. It tests the correctness of the system operation when a testing scenario is given. The proposed testing method is applied on a fingerprint verification BSP using the sample BSP provided by the BioAPl consortium. The experimental results shows the benefits of the proposed testing method.

FSM-based incremental conformance testing methods

The development of appropriate test cases is an important issue for conformance testing of protocol implementations and other reactive software systems. A number of methods are known for the development of a test suite based on a specification given in the form of a finite state machine. In practice, the system requirements evolve throughout the lifetime of the system and the specifications are modified incrementally. We adapt four well-known test derivation methods, namely, the HIS, W, Wp, and UIOv methods, for generating tests that would test only the modified parts of an evolving specification. Some application examples and experimental results are provided. These results show significant gains when using incremental testing in comparison with complete testing, especially when the modified part represents less than 20 percent of the whole specification.

A dynamic protocol conformance test method

Protocol conformance test is used to promote the interoperability of protocol implementations developed by venders. Non-interoperability between protocol implementations may be caused by ambiguity and/or misinterpretation of protocol specifications by vendors or by different implementations using different options in specifications. The conventional method used for protocol conformance test has been standardized by ISO/IEC JTC1 and ITU-T with the purpose of determining whether a protocol implementation conforms to its specification. However, the conventional method sometimes gives wrong test results because the test is based on static test sequences. This problem is caused by some failed transitions of a protocol’s finite state machine included in a test sequence, which have an effect on the test result of transitions to be tested. In this paper, an approach called dynamic conformance test method (DCTM) is proposed to solve this problem. DCTM dynamically selects different test sequences during testing depending on whether alternative paths without failed transitions exist. As a result, the fault coverage of DCTM is better than that of the conventional test method. DCTM has been implemented and applied to the TCP protocol in order to demonstrate its improvement in the fault coverage compared to that of the conventional method.

Implementation of Hierarchical Conformance Testing Platform for LAPB

Implementation of hierarchical conformance testing method for testing LAPB in this report ia discussed. UIO method of testing the state of a FSM has some disadvantages over the hierarchical method. While it is a compact way of checking that the state of the FSM is indeed what we need it to be, UIO method causes some special cases of errors to be masked. Under certain specific cases of errors, UIO testing actually traverses a different path while generating the expected output. This usually results in the final state of the UIO to be different, and depending on the next few test-transitions, the error may not be detected at all in such a situation. All this is because the transitions tested are only of single length. The hierarchical method of testing conformance uses path-lengths of two and more in an increasing order of hierarchy to completely test all possible behaviours of the FSM.

An extended-UIO-based method for protocol conformance testing

Verification of protocols is performed through conformance testing. The aim of this paper is to introduce the conformance test generation approach for protocols described by means of Finite State Machines. A functional fault model is adopted and the state discrimination is performed by applying an extended version of Unique Input Output Sequences (UIO), which, different from classical UIO, can always be found in any state. Both algorithms for efficient extended UIO sequence identification and for optimal test sequence generation are presented, together with the experimental results on different protocol descriptions.

Software evaluation using the 9241 evaluator

There is an increasing need for practical and comprehensive evaluation methods and tools for conformance testing with ISO standards.In this study, we focus on ISO9241 which is an important ergonomic standard. A brief description shows its content and structure. Practical evaluations include the amount of time and resources which must be managed in software projects,while comprehensive evaluations require that the context of use be considered during the evaluation of user interfaces. In order to complete a comprehensive evaluation of usability, it is necessary to use more than one evaluation method. Therefore, an overview of different evaluation approaches is given, describing their advantages and disadvantages. Finally, the ISO 9241 evaluator is presented in detail as an example of a practical expert-based evaluation method for conformance testing with the ISO 9241 standard, that can be integrated in a comprehensive evaluation approach.

Standardization of formal methods in conformance testing of communication protocols

This paper presents the work developed by the joint ISO/ITU-T working group on “Formal Methods in Conformance Testing” (FMCT) in order to obtain a future standard. It concerns several areas of conformance testing: terminology, testing concepts, test architecture, specification languages for tests and test generation methods. The influence of FMCT on testing methodology is also analysed and future trends in the area are addressed.

Testing methods for SDL systems

The aim of this tutorial is to present an overview of conformance testing methods for SDL systems. These methods can be classified into two main groups: those whose intent is the totally automated test generation from the SDL system specification and those that provide interactive test generation methods. We present some of the more representative methods illustrating their application by a common example. This example, the Inres protocol and service, will facilitate the comparison of the methods with respect to SDL and test notations, restrictions on the language, results of the application to the protocol and the tools supporting the methods.

Conformance testing of X-25 packet level

This paper explains a method and tool for conformance testing of X.25 packet level protocol, and DTE equipment based on X.25 protocols. The method uses standard test suites, and standardized TTCN (Tree and Tabular Combined Notation).