Optimal design of tuned mass damper (TMD) parameters should be implemented before device installation to achieve the desired vibration reduction effects for tall and quite slender buildings. Although various optimization algorithms have been developed for searching TMD parameters, there has been very limited exploration of machine learning-based optimization approach in this domain. In this paper, two machine learning-based algorithms are proposed to optimize the natural frequency and damping ratio of a TMD device for seismically excited building. The first algorithm investigates the single-objective optimization of a TMD parameters on a linear building structure. It is based on physics-enhanced generative and adversarial network (GAN) architecture. A generative network randomly generates the primary TMD parameters within the specified parameter ranges, while an adversarial network guides the evolution of parameters to acquire updated parameters, a physical evaluation network assesses the performance of the updated and primary parameters, subsequently providing training data for both the generated network and adversarial network. Furthermore, a second physics-enhanced machine learning algorithm is proposed for the multi-objective optimization of a TMD parameters on a nonlinear building structure under various seismic excitations. The physical evaluation network is extended to consider multiple objective functions, in which the multiple objective functions are integrated through a kind of decomposition scheme, i.e., weighted sum approach. Then, a better set of parameters can be determined and a Pareto optimal solution is obtained. To verify the effectiveness of the two proposed algorithms, single-objective optimization of a TMD parameters for a linear shear-type structure under seismic excitation and a multiple-objectives optimization of a TMD parameters for a seismically excited nonlinear moment-resisting frame (MRF) are conducted and the optimizations are validated by comparing with those by the conventional particle swarm optimization (PSO) approach.