Brillouin spectroscopy has been used to characterize shear acoustic phonons in materials. However, conventional instruments had slow acquisition times over 10 min per 1 mW of input optical power, and they required two objective lenses to form a 90° scattering geometry necessary for polarization coupling by shear phonons. Here, we demonstrate a confocal Brillouin microscope capable of detecting both shear and longitudinal phonons with improved speeds and with a single objective lens. Brillouin scattering spectra were measured from polycarbonate, fused quartz, and borosilicate in 1-10 s at an optical power level of 10 mW. The elastic constants, phonon mean free path and the ratio of the Pockels coefficients were determined at microscopic resolution.