Assessing tracker performance in a multi-sensor environment

Abstract

As new surveillance systems and sensor technologies are introduced to support enhancements to the air traffic control (ATC) infrastructure the need arises for a systematic evaluation of tracker algorithms. Ground automation systems use the surveillance data generated by multiple sensors in order to support critical ATC functions. To make the best use of the multi-sensor environment, tracker algorithms use different data fusion approaches with varying degrees of performance depending on how these algorithms are configured and fine tuned. Testing tracker performance, taking into account realistic sensor environments, requires an approach that includes four components: a) use of recorded live traffic to define realistic “ground truth” scenario data; b) target generation based on the desired sensor environment and with the inclusion of real-life target anomalies; c) definition of a set of ‘orthogonal’ metrics and associated statistics that covers all aspects of tracker performance that are of operational relevance such as accuracy, stability, lag, track loss and track swap; d) automated computation of the metrics that compare tracker outputs with the a priori known “ground truth” in order to quantify tracker performance for statistically significant samples. In addition to providing a direct quantitative assessment of tracker performance, these metrics and the approach presented here can be used to perform meaningful comparison and ranking of tracker algorithms, validation of new systems using a legacy reference system, parameter tuning on ATC surveillance applications, system and regression tests, and support the development of system specifications. The tracker evaluation approach described here was used in the test program for the Federal Aviation Administration's (FAA) En Route Automation Modernization (ERAM) system, specifically to test the accuracy of ERAM's Surveillance Data Processor (SDP) subsystem. Sample results and the tools developed to implement the test approach are described.