Multiple‐harmonic electromagnetic waves in the ULF band have occasionally been observed in Earth's magnetosphere, both near the magnetic equator in the outer plasmasphere and in the plasma sheet boundary layer (PSBL) in Earth's magnetotail. Observations by the Cluster spacecraft of multiple‐harmonic electromagnetic waves with fundamental frequency near the local proton cyclotron frequency, Ωcp, were recently reported in the plasma sheet boundary layer by Broughton et al. (2008). A companion paper surveys the entire magnetotail passage of Cluster during 2003, and reports 35 such events, all in the PSBL, and all associated with elevated fluxes of counterstreaming ions and electrons. In this study we use observed pitch angle distributions of ions and electrons during a wave event observed by Cluster on 9 September 2003 to perform an instability analysis. We use a semiautomatic procedure for developing model distributions composed of bi‐Maxwellian components that minimizes the difference between modeled and observed distribution functions. Analysis of wave instability using the WHAMP electromagnetic plasma wave dispersion code and these model distributions reveals an instability near Ωcp and its harmonics. The observed and model ion distributions exhibit both beam‐like and ring‐like features which might lead to instability. Further instability analysis with simple beam‐like and ring‐like model distribution functions indicates that the instability is due to the ring‐like feature. Our analysis indicates that this instability persists over an enormous range in the effective ion beta (based on a best fit for the observed distribution function using a single Maxwellian distribution), β′, but that the character of the instability changes with β′. For β′ of order unity (for instance, the observed case with β′ ∼ 0.4), the instability is predominantly electromagnetic; the fluctuating magnetic field has components in both the perpendicular and parallel directions, but the perpendicular fluctuations are larger. If β′ is greatly decreased to about 5 × 10−4 (by increasing the magnetic field), the instability becomes electrostatic. On the other hand, if β′ is increased (by decreasing the magnetic field), the instability remains electromagnetic, but becomes predominantly compressional (magnetic fluctuations predominantly parallel) at β′ ∼ 2. The β′ dependence we observe here may connect various waves at harmonics of the proton gyrofrequency found in different regions of space.