Equalization for maximum likelihood detectors

Abstract

In this paper, the minimum mean-square error (MMSE) technique has been used to equalize the recording channel in order to facilitate the application of the Viterbi detector. The resulting performance has been compared with that of the optimal equalization system which yields the minimum probability of error at the output of the Viterbi detector. The results indicate that depending on the constraint used in the MMSE design, the amount of noise correlation varies significantly at the equalizer output, which in turn makes a large difference in the performance of the Viterbi detector. In particular, in the jitter-dominant channel where unconditioned channel noise samples are highly correlated, the monic constraint on the equalizer target response tends to whiten the noise samples at the equalizer output. This results in a significant performance improvement of the monic constraint upon the fixed-energy constraint as well as the popular partial response targets of the form (1-D)(1+D)/sup n/.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>