The linear stability of a radially localized layer rotating about the cylindrical axis in a magnetized plasma is investigated using an eigenvalue analysis. The eigenvalue equation is solved numerically in a parameter regime characteristic of the Space Physics Simulation Chamber (SPSC) experiments [Amatucci et al., Phys. Rev. Lett. 77, 1978 (1996)] at the Naval Research Laboratory (NRL). Four types of instabilities are predicted. They are type-A and type-B Kelvin-Helmholtz instabilities, a transverse current-driven instability, and the inhomogeneous energy density driven instability (IEDDI). A quantitative comparison between theory and experiment indicates that an experimentally observed fluctuation in a rotating plasma layer is an IEDDI.