We study an indivisibility problem of the relative class numbers of CM fields. For prime p>3, Kohnen–Ono gave a lower bound of the number of the imaginary quadratic fields whose class numbers are prime to p by using modular forms of half-integral weight. We generalize their method to Hilbert modular forms and give a lower bound of the number of CM quadratic extensions K/F whose relative class numbers prime to p for totally real number field F which is Galois over Q and sufficiently large prime p. Combining the indivisibility result with the decomposition condition of p, we show a result on vanishing of relative Iwasawa invariants.