While most of researches on system identification of building structures are aimed at finding modal parameters first and identifying the corresponding physical parameters by using the transformation in terms of transfer functions and cross spectra, etc., direct physical parameter system identification methods have been proposed recently. Due to the problem of signal/noise (SN) ratios, the previous methods are restricted mostly to earthquake records or forced vibration data. In this paper, a theoretical investigation is performed on the influence of wind disturbances on stiffness identification of building structures using micro-tremor at limited floors. It is concluded that the influence of wind disturbances on stiffness identification of building structures using micro-tremor at limited floors is restricted in case of using time-series data for low-rise buildings and does not cause serious problems.