An efficient identification method for modal testing of viscoelastic composite materials is demonstrated in this paper, through the analytical–experimental transfer function method. The procedure for the identification of analytical–experimental transfer functions is carried out using a genetic algorithm (GA) by minimizing the difference between the measured response from tests and the calculated response, which is a function of the modal parameters. The analytical transfer functions provide a sub-structuring process to identify modes, as a function of damped natural frequencies and loss factors of a complex structure and it is insensitive to experimental noise as well as the modal coupling effect. The proposed method is verified with the calculation of the elastic modulus and modal properties of a cantilever steel beam with the FEM and compared with the identification results of the proposed algorithm. The effectiveness of the proposed method is demonstrated by investigating the static and dynamic behavior of epoxy cantilever beam specimens reinforced with silica nanoparticles. Analytical–experimental transfer functions accurately identified the viscoelastic and dynamic response of the studied specimens, while the results indicated that the inclusion of nanosilica particles increased the stiffness of the epoxy network and the damping response of the reinforced specimens is improved.