In this paper, a frequency-labeled adaptive sparse grid collocation (FL-ASGC) method is proposed to study the uncertainty quantification (UQ) of the frequency response of general viscoelastic damping structures. The advantage of the proposed method is that, on the one hand, it has a good approximation ability for both smooth and non-smooth responses so that it can robustly predict the variability of frequency responses under various damping levels. On the other hand, by evaluating the contribution of each interpolation node to the response at all frequencies, it can adaptively select samples and their accompanying analysis frequency domains, thereby minimizing the computational costs of probabilistic surrogate modeling. To obtain the deterministic solution of each sample, the layer-wise finite element method combined with the direct frequency response (DFR) method is employed. Two numerical examples with different damping materials are provided to test the proposed method. Uncertainties that obey normal distribution in material and geometry are considered. The results show the proposed method can accurately, efficiently, and robustly predict the variability of the frequency response of general viscoelastic damping structures.