Scanning laser Doppler vibrometry is a widely adopted method to measure the full-field out-of-plane vibrational response of materials in view of detecting defects or estimating stiffness parameters. Recent technological developments have led to performant 3D scanning laser Doppler vibrometers, which give access to both out-of-plane and in-plane vibrational velocity components. In the present study, the effect of using (i) the in-plane component; (ii) the out-of-plane component; and (iii) both the in-plane and out-of-plane components of the recorded vibration velocity on the inverse determination of the stiffness parameters is studied. Input data were gathered from a series of numerical simulations using a finite element model (COMSOL), as well as from broadband experimental measurements by means of a 3D infrared scanning laser Doppler vibrometer. Various materials were studied, including carbon epoxy composite and wood materials. The full-field vibrational velocity response is converted to the frequency-wavenumber domain by means of Fourier transform, from which complex wavenumbers are extracted using the matrix pencil decomposition method. To infer the orthotropic elastic stiffness tensor, an inversion procedure is developed by coupling the semi-analytical finite element (SAFE) as a forward method to the particle swarm optimizer. It is shown that accounting for the in-plane velocity component leads to a more accurate and robust determination of the orthotropic elastic stiffness parameters.