Accounting for uncertainties in seismic site response is crucial to improving the performance of one-dimensional (1D) ground response analyses (GRAs) at downhole array recording sites. In addition to site effects, uncertainties in 1D-GRAs can also be contributed from the seismic source and/or path. Though often representing not more than one percent of the distance (path) from the source, site conditions are known to have an enormous influence on ground shaking. In this study, we focus on the site shear-wave velocity (VS) structure, which is the main ingredient for estimating the variability of site response. As such, VS can manifest aleatory uncertainties related to the effects of small-scale spatial heterogeneities within the near surface, thus VS can substantially modify ground shaking during earthquakes. We apply a novel VS randomization approach to propagate the small-scale heterogeneities of VS to estimate seismic site response within a non-stationary probabilistic framework. The randomization approach generates samples of VS profiles that are used to perform several 1D-GRAs and obtain an averaged site response and related variability. The proposed method is implemented on data recorded at two downhole array sites with different subsurface soil conditions: a soft soil site on Treasure Island (California, United States of America) and a rock outcrop site in Cadarache (South-East France). We show that synthetic surface-to-borehole transfer functions from 1D-GRAs provide an acceptable fit to the empirical transfer functions from low-motion earthquake records and succeed in reproducing most of the site-specific seismic response variability. The remaining mismatch between transfer functions is likely due to insufficient precision on the seismic bedrock and the impedance contrast. The variability in site response is discussed with emphasis on the role of VS small-scale heterogeneities, attenuation, and input motion incidence angle in ground motion variability for the site and soil conditions at both locations.