Abstract
Good test data is crucial for driving new developments in computer vision (CV), but two questions remain unanswered: which situations should be covered by the test data, and how much testing is enough to reach a conclusion? In this paper we propose a new answer to these questions using a standard procedure devised by the safety community to validate complex systems: the hazard and operability analysis (HAZOP). It is designed to systematically identify possible causes of system failure or performance loss. We introduce a generic CV model that creates the basis for the hazard analysis and—for the first time—apply an extensive HAZOP to the CV domain. The result is a publicly available checklist with more than 900 identified individual hazards. This checklist can be utilized to evaluate existing test datasets by quantifying the covered hazards. We evaluate our approach by first analyzing and annotating the popular stereo vision test datasets Middlebury and KITTI. Second, we demonstrate a clearly negative influence of the hazards in the checklist on the performance of six popular stereo matching algorithms. The presented approach is a useful tool to evaluate and improve test datasets and creates a common basis for future dataset designs.
Highlights
Many safety-critical systems depend on computer vision (CV) technologies to navigate or manipulate their environment and require a thorough safety assessment due to the evident risk to human lives (Matthias et al 2010)
We show that the CV-hazard and operability analysis (HAZOP) correctly identifies challenging situations and on the other hand, we provide a guideline for all researches to do their own analysis of test data
The goal is to show that the entries of the CV-HAZOP are meaningful and that the checklist is a useful tool to evaluate robustness of CV algorithms
Summary
Many safety-critical systems depend on CV technologies to navigate or manipulate their environment and require a thorough safety assessment due to the evident risk to human lives (Matthias et al 2010). This work presents a new way to facilitate a safety assessment process to overcome these problems: a standard method developed by the safety community is applied to the CV domain for the first time. A big problem when validating CV algorithms is the enormous set of possible test images. Validation tries to show that the algorithm can reliably solve the task at hand, even under difficult conditions Both use application specific datasets, their goals are different and benchmarking sets are not suited for validation. The main challenge for validation in CV is listing elements and relations which are known to be “difficult” for CV algorithms (comparable to optical illusions for humans). The impact of identified hazards on the output of multiple stereo vision algorithms is compared in Sect.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
