Kinetic instabilities in a dense plasma of a continuous electron cyclotron resonance (ECR) discharge in a mirror magnetic trap at the Gasdynamic Ion Source for Multipurpose Operation (GISMO) setup are studied. We experimentally define unstable regimes and corresponding plasma parameters, where the excitation of electromagnetic emission is observed, accompanied by the precipitation of energetic electrons from the magnetic trap. A comprehensive experimental study of the precipitating electron energy distribution and plasma electromagnetic emission spectra, together with theoretical estimates of the cyclotron instability increment proves that under the experimental conditions, the observed instability is related to the excitation of whistler-mode waves, which are a driver of losses of energetic electrons from the magnetic trap. The results of this study are important for the further development of the GISMO electron cyclotron resonance ion source facility and for the improvement of its parameters as an ion source. Also, this research on plasma kinetic instabilities is of fundamental interest and provides experimental tools to simultaneously study plasma electromagnetic activity and corresponding changes in a resonant electron energy distribution.