The size dependence of the longitudinal optical (LO) phonons in the excited state of CuBr quantum dots (QD's) in glass and NaBr crystals in the intermediate confinement regime was studied by means of persistent spectral hole burning spectroscopy. The phonon-exciton coupled states were clearly observed at a photon energy of about 2.993 eV when the LO phonon energy is close to the energy difference between the ground $1S$ and excited $1P$ states of CuBr QD's in glass. The energies of the LO phonons observed in smaller CuBr QD's in glass and NaBr crystals were determined to be about 18.6 and 17.6 meV, respectively, which are smaller than that of LO phonons in the bulk CuBr material. The energy softening of the LO phonons was explained in terms of the phonon renormalization.