Noise, filters and detection limits

Abstract

The “noise” of Chromatographic baselines has been investigated in regard to the detector, the nature and extent of filtering or smoothing, and the methodologies of qualitative and quantitative assessment: all in order to clarify the role such factors play in the determination and interconversion of some common types of detection limits. This study scrutinizes baselines from the flame photometric detector in single-channel continuous and ten-channel multiplexing versions; it also examines baselines from flame ionization and electron-capture detectors. It makes use of finite impulse response and non-weighted moving-average digital smoothing, as well as three-pole analog filtering. Baseline fluctuations are quantified by the standard deviation derived from the common root-mean-square (RMS) calculation, or from the less common least-squares Gaussian fit; peak-to-peak noise ( N p-p) is estimated by procedures including or excluding presumed outliers. Individual results are expressed as the ratio of N p-p measurement and RMS calculation performed on the same data set. A wide variety of such ratios are then assembled from different detectors, filters, and smoothing conditions. They prove conclusively that -contrary to common belief—the conversion factor between the two types of measurements does vary: usually between 4 and 10, but occasionally even farther. Consequently, the conversion factor between the corresponding two types of detection limits varies as well. The N p-p/RMS ratio depends largely on the detector-of-origin, its condition, and the extent to which noise has been filtered. In contrast, the nature and sophistication of the filter hardly matters: either for the N p-p/RMS ratio or for the practical detection limit. This is because the slow undulations characteristic of heavily filtered baselines represent —at least in the detectors we used-dampened fast noise rather than aboriginally slow noise. Corresponding computer simulations, based on amplitudinally random noise smoothed by stationary boxcar or non-weighted moving-average filters, produce results strikingly similar to actual baselines. Simulated fast RMS noise correlates, as expected, with the square root (log-log slope = 1 2 ) of the filter's time constant. The corresponding slopes for experimental noise are usually close to 1 2 as well. Most importantly, though, the simulated N p-p/RMS ratio varies strongly with the extent of smoothing -thus mimicking and thereby explaining the behavior of the experimental ratio.