Analysis and optimization of a class of nonlinear detection filters

Abstract

It is well know that the matched filter is the optimal linear detection filter but, this allows for the possible existence of a nonlinear detection filter with better performance. This paper considers the class of nonlinear detection filters that are composed of a linear filter followed by an arbitrary point process. The result is general enough to include detection paradigm in which the signal model is not additive. The ROC (receiver operating characteristic) curve of a general point process is analyzed. This analysis reveals that nonreversibility and not nonlinearity of the point process is responsible for the improvement of the ROC curve. That is, an reversible point process, either linear or nonlinear, leaves the ROC curve unchanged. However, a nonreversible point process will alter the ROC curve. This result is used to define a canonical ROC curve which is then utilized to derive the optimal point process. Several simple forms of the point process are considered first then the general optimal point process is derived. The technique is illustrated with several examples. Results for the special case of unimodal signal densities receive particular attention.