Abstract The modulation instability of whistler mode waves caused by thermal electron anisotropy is studied. Based on MHD equations, the nonlinear schrödinger equation (NLSE) that describes the nonlinear modulation of whistler waves is derived by Using the Krylov-Bogoliubov-Mitropolsky (KBM) method. The condition for wave modulation instability is obtained from the loss cone distribution function of thermal electron anisotropy, revealing that the nonlinear growth of the waves tends towards electron perpendicular temperature anisotropy. By setting up continuous background waves and introducing small ion low frequency perturbations, we find that the change in the amplitude of the modulated wave is related with wave number. This finding has been validated through simulations that align with our analytical results. Additionally, we also calculate the maximum amplitude of the wave with loss cone angle and times, which revealed that the electron vertical temperature anisotropy will lead to the modulation instability of the whistler wave. This further confirms the occurrence of the modulation instability of the whistler wave in laboratory plasmas and strengthens their credibility.