The laser Dopper vibrometer (LDV) has become a standard instrument to perform high spatial resolution 3D vibration measurements of complex structures. An important step in the LDV measurement procedure is the estimation of the pose of the instrument relative to the test object. Existing LDV pose estimation methods rely on the use of 3D range sensors to perform this task. These sensors are costly and often their spatial resolution and accuracy is too limited for an accurate LDV pose estimation. But most importantly, existing LDV pose estimation techniques are slow. In this paper, we present a novel LDV pose estimation method based on matching of a 2D camera image and a CAD model of the test object. In contrast to existing pose estimation techniques, our method is able to track the pose of the LDV in real-time (up to about 30 LDV positions per second). This means that our method also allows LDV pose estimation when either the LDV or the object is moving. Using the real-time LDV pose tracking we present a new LDV measurement methodology where we hold instrument in our hand while moving around the test structure. This allows us to perform full-field scanning and 3D velocity measurements with a low-cost single point LDV sensor. We show that the measurement accuracy of a stationary handheld LDV measurement is about 40 dB (which is about 10 dB lower than in the case of measurements on a tripod). Furthermore, we demonstrate that the 3D vibrations measurements using the handheld vibrometer have an error of about 20% compared to classical measurements performed by combining three LDV measurements on a tripod. We illustrate our method on vibration measurements of a composite bicycle frame.