Abstract
This paper (1) summarizes the history of the RERS challenge for the analysis and verification of reactive systems, its profile and intentions, its relation to other competitions, and, in particular, its evolution due to the feedback of participants, and (2) presents the most recent development concerning the synthesis of hard benchmark problems. In particular, the second part proposes a way to tailor benchmarks according to the depths to which programs have to be investigated in order to find all errors. This gives benchmark designers a method to challenge contributors that try to perform well by excessive guessing.
Highlights
Competitions and challenges have provided a valuable contribution to the development of verification and analysis tools, and numerous events of this kind have evolved over the last decades [4,6,8,29,32,36,43]
A frequent argument against the use of generated benchmarks is the potential threat to the validity of profiling results that arises from their artificial nature. We address this threat in Rigorous Examination of Reactive Systems challenge (RERS) by using sets of linear temporal logic (LTL) properties for inducing structure or actual industrial system models at the core of our benchmark synthesis
The formal semantics of LTL is based on a satisfaction relation between infinite words and LTL formulas [3]: Definition 5 (Semantics of LTL) Let AP be an alphabet of atomic propositions and let (2AP)ω denote infinite sequences over sets A ⊆ AP
Summary
Competitions and challenges have provided a valuable contribution to the development of verification and analysis tools, and numerous events of this kind have evolved over the last decades [4,6,8,29,32,36,43]. 2) summarizes the history of RERS, its profile and intentions, its relation to other competitions, and, in particular, its evolution due to the feedback of participants. This comprises a discussion of experienced ‘oddities’, both at RERS and in relation to other competitions, as well as ways to overcome them. The proposed tailoring of benchmarks focusing on the depths to which programs have to be investigated in order to find all errors gives benchmark designer a way to challenge contributors who are claiming satisfaction without sufficient evidence
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: International Journal on Software Tools for Technology Transfer
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
