Frequency- and temperature-dependent properties of viscoelastic materials have a significant effect on vibration characteristics of constrained layer damping (CLD) structures. In this article, parametrized level set method (PLSM) is extended to solve the topology optimization problem of CLD plates, in which the viscoelastic material with frequency- and temperature-dependent characteristics is employed as damping core. The single/weighted modal loss factors, which are solved by an iterative method, are defined as objective functions, and topology optimization problem is formulated based on the finite element model for CLD plates which are described by combining parametric level set (PLS) functions. The sensitivities of objective functions with respect to expansion coefficients in PLS are calculated by the use of adjoint vector method. As the optimized results with complex constant shear modulus for viscoelastic core indicates, the proposed method is effective for topology optimization of CLD structures and insensitive to initial design. Numerical examples with the consideration of frequency- and temperature-dependent characteristics for “soft” and “hard” viscoelastic core are implemented. Through comparing the optimal results with different models for viscoelastic core, the influences of frequency- and temperature-dependent characteristics for viscoelastic core on natural frequencies, modal loss factors and optimized layouts are analyzed. Similarity indices are defined for distinguishing the difference among optimized layouts and initial design induced by modal strain energy contour. The optimal design dependence on layer thicknesses is further discussed with additional mass of CLD materials unchanged.