ABSTRACT Atmospheric airglow lines, in particular those from the OH molecule, are a major source of background radiation at near infrared wavelngths (1-2.3 microns). Because of this, several observatories around the world are investigating the potential for blocking these emission lines using instruments based on filters or diffraction gratings. This paper presents numerical calculations of the performance of such devices. A flexible computer program automates the evaluation of several OH blocking schemes and allows optimization of instrument parameters. A number of important conclusions arise from the study of filter-based and general grating (dispersive) configurations designed for the H (1.5-1.8 microns) and J (1.13-1.37 microns) photometric bands. 1. State-of-the-art multilayer dielectric bandpass filters tailored to isolate a relatively or completely emission-free region do not provide any improvement over the broadband H filter itself. Similarly, a high spectral resolution Fabry-Perot interferometer acting as a fixed filter gives disappointing performance. This is due to the shape of technically-feasible filter profiles and the strength and distribution of OH lines in the H band. 2. Several small gaps in the distribution of J band emission lines provide the opportunity for moderate (~0.5 mag) improvement in sensitivity using low cost dielectic bandpass filters. The broad wings in the transmission profile of Fabry-Perot interferometers makes them impractical to take advantage of these gaps. 3. A stack of nearly-ideal dielectric notch filters dsigned to remove the brightest individual OH lines gives poor performance in both the H and J photometric bands, and the performance degrades with increasing number of filters in series. Broader blocking filters centered on the brightest groupings of lines produce no improvement in sensitivity in the H band. Blocking all the J airglow lines leaves the equivalent of two bandpass filters at the short and long wavelength extremes of the band, and provides a moderate (~0.5 mag) increase in performance. 4. Dispersive instruments, such as those employing gratings, can provide ~1.3 magnitudes of improved sensitivity in the H band, and ~1.6 magnitudes in the J band. The improvement in sensitivity increases monotonically with the spectral resolution of the dispersive instrument, although significant gains are attainable with R ≥ 4000. All the OH lines should be blocked to give reasonable performance. It is important to understand and control diffraction and scattering in dispersive systems. Each additional percent of unblocked OH line radiation leads to a 10% decrease in sensitivity. 5. The H and J band airglow continuum is poorly understood, and few precise measurements exist in the literature. It is imperative to improve this situation before major instrumentation projects begin, since the range of published fluxes encompasses values that can make such instruments pointless.