Spontaneous Brillouin scattering in bulk crystalline solids is governed by the intrinsic selection rules locking the relative polarization of the excitation laser and the Brillouin signal. In this work, we independently manipulate the polarization of the two by employing polarization-sensitive optical resonances in elliptical micropillars to induce a wavelength-dependent rotation of the polarization states. Consequently, a polarization-based filtering technique allows us to measure acoustic phonons with frequencies difficult to access with standard Brillouin and Raman spectroscopies. This technique can be extended to other polarization-sensitive optical systems, such as plasmonic, photonic, or birefringent nanostructures, and finds applications in optomechanical, optoelectronic, and quantum optics devices.