Abstract
The first formal attempt to standardize the test and evaluation of localization systems was the ISO/IEC 18305 standard. As reviewed in this paper, the ISO/IEC 18305 provides guidelines to evaluate an indoor localization system as a black box. Such evaluation is useful for end users, but not fully useful for either developers or researchers. As a result, both developers and researchers use their own evaluation methodologies, which have a set of weaknesses. Our objective is to compare the standard to the evaluation methodologies implemented by developers. We focus on pedestrian localization systems because they are the most numerous. In this work, we survey both competitions and individual works, between 2008–2018, with the ISO/IEC 18305 standard as a guideline. That is, we have reviewed the evaluation methodologies regarding the different aspects defined in the standard. We finalize with a summary of the shortcomings of the evaluation methodologies implemented by the developers, as well as the shortcomings of the standard. Our article ends with a summary of open issues that need to be addressed in order to foster the development of evaluation methodologies of localization systems. The main conclusion of our work is that the developers of localization systems are in need of guidelines to design evaluation methodologies, especially for the evaluation of system components. The standard fails to satisfy such requirement by proposing an evaluation strategy that focuses more on the end user than the developer.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.