The dynamic characterization of structures by means of modal parameters offers many valuable insights into the vibrational behavior of these structures. However, modal parameter estimation has traditionally required expert knowledge and cumbersome manual effort such as, for example, the selection of poles from a stabilization diagram. Automated approaches which replace the user inputs with a set of rules depending on the input data set have been developed to address this shortcoming. This paper presents an alternative approach based on Bayesian optimization. This way, the possible solution space for the modal parameter estimation is kept as widely open as possible while ensuring a high accuracy of the final modal model. The proposed approach was validated on both a synthetic test data set and experimental modal analysis data of a machine tool. Furthermore, it was benchmarked against a similar tool from a well-known numerical computation software application.