With the trend toward taller and more flexible building structures, a mass-damper shaking table system has been considered as means for vibration suppression to external excitation and disturbances in recent years. However, there are few researches on the control of nonlinear structure using active mass damper (AMD) under earthquake excitation, especially for high-rise building. In this work, a multilayer feedforward neural network with the modified Newton method, similar to BFGS algorithm, is developed for vibration suppression of a building structure. The benchmark tests show that the modified Newton method is superior to many conventional ones: steepest descent, steepest descent with adaptive learning rate, conjugate gradient, and Newton-based methods and is suitable to small network in engineering applications. Experimental results show that an AMD system combined with the modified Newton method remains effective for building structure vibration suppression under free vibration, forced vibration and Ji-Ji Earthquake (Sep. 21, 1999) excitation.