We demonstrate a polarization-interference-based fiber vibration sensor by incorporating a Sagnac interferometer based on a polarization-diversity loop structure composed of a polarization beam splitter, polarization-maintaining photonic crystal fiber, and polarization controllers. A laser diode and a photodetector were used to implement a high-speed intensity-based sensor. By using a piezoelectric transducer, we performed single frequency vibration measurement over 1-3000 Hz, resulting in a cutoff frequency of ~2100 Hz, a phase shift per unit length and unit strain of ~3.48 mrad/(με · m), and a minimum detectable strain perturbation of ~0.46 nε/Hz <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sup> at 2000 Hz. Naturally, damped vibration response was also investigated using a metal cantilever, and the relationship between the sensor output and applied impulse was analyzed.